After five years of work, in July 2021 the NIST/OSAC Subcommittee on Fire and Explosion Investigations released its strategic vision document, Strengthening Fire and Explosion Investigation in the United States: A Strategic Vision for Moving Forward. This document is in response to the 2009 National Academy of Science (NAS) report, Strengthening Forensic Science in the United States: A Path Forward. This NAS report formed the basis for the formation of OSAC and explicitly identified fire investigation as a forensic science. The NAS report also identified weaknesses within fire investigation as a forensic science.
In response to the NAS report, the NIST/OSAC Subcommittee on Fire and Explosion Investigations set out to create a strategic vision document to identify and map improvements in fire and explosion investigations. The document details a path forward to improve the quality of fire investigations by listing a series of specific recommendations. Some of the most impactful recommendations for both public and private sector fire investigation professionals included competency and proficiency testing for all fire investigators.
Of all of the Chapters within the document, perhaps Chapter 8, “Strengthening Oversight of Fire and Explosion Investigation Practices” carried the most impactful recommendations. The chapter addresses many topics, concluding that the fire investigation community is better served by individuals who are certified, while still being subjected to competency and proficiency testing, and working out of accredited FIUs. The FIUs are recognized as a critical element, and it is recommended that they be formed and managed in accordance with the new NFPA 1321 standard, Standard for Fire Investigation Units, become accredited by a third party, maintain a quality assurance program, and implement independent technical review processes.
NFPA 1321 and Proficiency Testing
The NIST/OSAC recommendations can be found throughout NFPA 1321. Importantly, the current NFPA 1321 draft contains language in Chapter 7 which states, “Management shall evaluate the technical competencies of personnel prior to assigning them to conduct independent work.” In addition, the document states, “Management shall develop and implement a method for the annual evaluation of the proficiency of FIU personnel by an experienced and competent evaluator.” Lastly, the document states, “Management shall document the competency and proficiency of FIU personnel.” Currently, it is expected that NFPA 1321 will be published in the summer of 2024.
Proficiency Testing for Fire Investigation
Proficiency testing in forensic disciplines is widespread. The definition of proficiency testing varies depending on the discipline, but most embrace something along the lines of a quality assurance measure used to monitor performances and quality through the examination and interpretation of a prepared sample whose results have previously characterized. Basically, it is the evaluation of a participant against pre-established criteria, used to identify the effectiveness of the actual procedure, training of the individual, and/or equipment used.
Proficiency testing in fire investigation is a quality assurance measure used by an organization to monitor the performance and quality of fire/explosion investigations. This is realized through annual testing of individual fire investigators who are tasked to examine and interpret data set(s) which have been previously characterized. Proficiency testing specifically provides a means to evaluate the skills and abilities of individual fire investigators to perform analyses, tasks, and/or entire fire/explosion investigations. Proficiency testing requires fire investigators to apply recommended and clearly defined methodologies, as found in NFPA 921 and other relevant texts, during these analyses, tasks or investigations. This includes collecting, documenting, characterizing, and/or analyzing various data used to draw conclusions. The proficiency test should be designed to reflect the knowledge, skills and training necessary to fulfill the fire investigator’s responsibilities (i.e. Job Performance Requirements as detailed in NFPA 1033) as well as their ability to draw accurate conclusions. Areas reflecting a fire investigator’s knowledge, skills and training may include, but are not limited to:
• Fire pattern analysis
• Timeline analysis
• Safety assessments
• Documentation
• Report writing
• Evidence processing
• Origin determination
• Cause determination
The fire investigator proficiency test may assess the application of a single methodology, several methodologies, or even an entire fire/explosion investigation. Proficiency tests should be developed and administered by a competent, knowledgeable party capable of making necessary assessments.
Proficiency testing is an important part of forensic service provider’s quality assurance program. It can evaluate individual proficiency, and also be used to evaluate whether an FIU’s operations are effective.
Competency testing, on the other hand, is when a newly trained fire investigator demonstrates specialized knowledge, skills and abilities necessary to conduct examinations prior to performing the actual work. It is a comprehensive test of technical skills as part of basic training and is not designed to necessarily address the agency’s overall quality system or performance, procedures, etc. Fire investigators undergo competency testing upon completing basic or fundamental training courses prior to actively working on actual fire investigations.
The Mechanics of Proficiency Testing for Fire Investigation
Proficiency testing may be realized in several different forms. The Bureau of Alcohol, Tobacco, Firearms and Explosives (ATF) National Response Team, which conducts annual proficiency testing, typically tests a methodology or several methodologies to conclude a certain part of the fire or explosion investigation. In the past participants had to conduct arc mapping, identify electrical safety issues, and sift post blast debris. Others have advocated for reviewing an entire actual fire investigation as a proficiency test. The nature of proficiency testing, regardless of the scope, is the focus on using recommended methodologies to obtain an accurate result. For fire investigation, the proficiency test will likely be an “open proficiency test” where they know they are being evaluated. In other forensic disciplines, such as fingerprint analysis, participants may be given a “blind proficiency test” where they are given known samples to compare under the guise of an actual case. In this instance, the participant believes they are looking at legitimate samples from an actual investigation, as it is submitted to them under normal procedures.
The mechanics of proficiency testing include oversight by competent third party (in this case likely another competent, experienced fire investigator). Though the outcome is predefined in the proficiency test, it is not shared with the participant. The investigator is provided with data, and then must utilize training and education to properly conduct a variety of investigative tasks to arrive at proper (known) conclusions. The methods used by the fire investigator should be based on recommended methodologies found in NFPA 921 and other relevant texts.
Proficiency testing is documented in a report which records scoring and ultimately whether the participant passed the test (it can be pass/fail). Should a participant fail, they may have an opportunity to take the proficiency test a second time. The results of all proficiency tests, passes or failures, are recorded and typically sent along to the participant’s chain of command. Sometimes these reports are not shared with the participants. It is important to realize that the documentation of the results is preserved forever and in certain jurisdictions may be required to be produced during the discovery phase in legal hearings. For example, in New York State, the current criminal procedure (NYS CPL 245.20) states that within 20 days of arraignment, the prosecution must provide, “Expert opinion evidence, including the name, business address, current curriculum vitae, a list of publications, and a list of proficiency tests and results administered or taken within the past ten years…”
The administration of a proficiency test falls to a competent third party who has the knowledge and experience to appropriately evaluate the participant. Though not required, some competent third parties may be accredited to ISO 17043, Conformity Assessment – General Requirements for Proficiency Testing, which outlines the requirements for the competence and impartiality of proficiency testing providers (as well as the consistent operation of proficiency testing schemes).
Conclusion
The impact of proficiency testing in fire investigation has the potential to be quite consequential. It is anticipated that proficiency testing will improve fire investigation and provide a tool for management to assess quality assurance or their standard operating procedures. The individual fire investigator must understand what is required to remain proficient (up to date training and education) and realize that these tests may be the subject of scrutiny during legal hearings.
