Inconclusives, errors, and error rates in forensic firearms analysis:Three statistical perspectives

Forensic Science International Synergy, Journal Year: 2022, Volume and Issue: 5, P. 100273 - 100273

Published: Jan. 1, 2022

Error rates that have been published in recent open black box studies of forensic firearms examiner performance very low, typically below one percent. These low error challenged, however, as not properly taking into account the categories, "Inconclusive", examiners can reach comparing a pair bullets or cartridges. challenges themselves challenged; how to consider inconclusives and their effect on is currently matter sharp debate. We review several viewpoints put forth, then examine impact from three fresh statistical perspectives: (a) an ideal perspective using objective measurements combined with algorithms, (b) basic sampling theory practice, (c) standards experimental design human studies. Our conclusions vary perspective: be simple errors (or, other hand, simply correct at least well justified); need counted bring doubt assessments rates; are potential errors, more explicitly, necessarily equivalent casework mask casework. From all these perspectives, it impossible read out trustworthy estimates those which carried date. At most, reasonable bounds rates. much larger than nominal reported To get straightforward, sound requires challenging but critical improvement A proper study-one yields direct, rates-will require new measures blind proficiency testing embedded ordinary

Language: Английский

---

Misuse of statistical method results in highly biased interpretation of forensic evidence in

Law Probability and Risk, Journal Year: 2024, Volume and Issue: 23(1)

Published: Jan. 1, 2024

Since the National Academy of Sciences released their report outlining paths for improving reliability, standards, and policies in forensic sciences (NAS, 2009), there has been heightened interest evaluating scientific validity science disciplines.Guyll et al. (2023) seek to evaluate cartridge-case comparisons.They conducted an experiment test accuracy firearms examiners.They then describe how triers fact such as a judge or jury criminal case, who are initially unbiased have not yet seen any evidence, should apply results case at hand.Specifically, Guyll use Bayes' rule calculate posterior probability that cartridge found crime scene was fired from reference gun (often linked defendant), given decision examiner.A key input this calculation is prior odds gun, which set 1 claim be unbiased.However, we explain below, typically highly biased against defendant can lead judges jurors trials grossly misunderstand interpret evidence.It imperative address erroneous statistical argument (2023), being presented by prosecution ongoing homicide (DC Superior Court, 2023).We discuss some other aspects study design analysis well.Our focus on specific issues exhaustive.

Language: Английский

---

Repeatability and reproducibility of comparison decisions by firearms examiners

Journal of Forensic Sciences, Journal Year: 2023, Volume and Issue: 68(5), P. 1721 - 1740

Published: July 2, 2023

Abstract In a comprehensive study to assess various aspects of the performance qualified forensic firearms examiners, volunteer examiners compared both bullets and cartridge cases fired from three different types firearms. They rendered opinions on each comparison according Association Firearm & Tool Mark Examiners (AFTE) Range Conclusions, as Identification, Inconclusive (A, B, or C), Elimination, Unsuitable. this part study, sets used previously characterize overall accuracy were blindly resubmitted repeatability (105 examiners; 5700 comparisons cases) reproducibility (191 bullets, 193 cases; 5790 comparisons) examinations. Data gathered using prevailing AFTE also recategorized into two hypothetical scoring systems. Consistently positive differences between observed agreement expected indicate that exceed chance agreement. When averaged over cases, decisions (involving all five levels Range) was 78.3% for known matches 64.5% nonmatches. Similarly 67.3%% 36.5% For reproducibility, many disagreements definitive inconclusive category. Examiner are reliable trustworthy in sense identifications unlikely when comparing non‐matching items, eliminations they matching items.

Language: Английский

---

Shining a Light on Forensic Black-Box Studies

Statistics and Public Policy, Journal Year: 2023, Volume and Issue: 10(1)

Published: May 23, 2023

Forensic science plays a critical role in the United States criminal justice system. For decades, many feature-based fields of forensic science, such as firearm and toolmark identification, developed outside scientific community's purview. The results these studies are widely relied on by judges nationwide. However, this reliance is misplaced. Black-box to date suffer from inappropriate sampling methods high rates missingness. Current black-box ignore both problems arriving at error rate estimates presented courts. We explore impact each type limitation using available data court materials. show that rely non-representative samples examiners. Using case study popular ballistics study, we find evidence may commit fewer errors than wider population which they came. also missingness non-ignorable. recent latent print ignoring likely systematic underestimates rates. Finally, offer concrete steps overcome limitations.

Language: Английский

---

Three-Way ROCs for Forensic Decision Making

Statistics and Public Policy, Journal Year: 2023, Volume and Issue: 10(1)

Published: July 21, 2023

Firearm examiners use a comparison microscope to judge whether bullets or cartridge cases were fired by the same gun. Examiners can reach one of three possible conclusions: Identification (a match), Elimination (not Inconclusive. Numerous error rate studies report that firearm commit few errors when they conduct these examinations. However, also many inconclusive judgments (>50%), and how score responses is controversial. There have recently been Signal Detection Theory (SDT) primers in this domain. Unfortunately, analyses rely on hypothetical data fail address response issue adequately. This article reports an SDT analysis using from large study practicing examiners. First, we demonstrate problem relying traditional two-way model, which either drops combines responses; addition lacking ecological validity, approach leads implausible results. Second, introduce readers three-way model. We forensic firearms While statistically complicated, it well suited evaluate performance for any domain decision categories exist.

Language: Английский

---

A call for open science in forensics

Proceedings of the National Academy of Sciences, Journal Year: 2024, Volume and Issue: 121(24)

Published: May 23, 2024

The modern canon of open science consists five “schools thought” that justify unfettered access to the fruits scientific research: i) public engagement, ii) democratic right access, iii) efficiency knowledge gain, iv) shared technology, and v) better assessment impact. Here, we introduce a sixth school: due process. Due process under law includes “discovery” by defendant potentially exculpatory evidence held prosecution. When such is scientific, becomes Constitutional mandate for science. To illustrate significance this new school, present case study from forensics, which centers on federally funded investigation reports summary statistics indicating identification decisions made forensic firearms examiners are highly accurate. Because growing concern about validity methods, larger community called release complete analyzable dataset independent audit verification. Those in possession data opposed three years while were used prosecutors gain admissibility criminal trials. paint an incomplete picture hint at flaws experimental design analysis. Under circumstances, withholding underlying proceeding violates Following successful open-science model drug testing through “clinical trials,” place strict requirements timing release, argue registered “forensic trials” ensure transparency accountability.

Language: Английский

---

Comments on: A Re-analysis of Repeatability and Reproducibility in the Ames-USDOE-FBI Study, by Dorfman and Valliant

Statistics and Public Policy, Journal Year: 2023, Volume and Issue: 10(1)

Published: March 13, 2023

Language: Английский

---

Methodological problems in every black-box study of forensic firearm comparisons

Law Probability and Risk, Journal Year: 2024, Volume and Issue: 23(1)

Published: Jan. 1, 2024

Abstract Reviews conducted by the National Academy of Sciences (2009) and President’s Council Advisors on Science Technology (2016) concluded that field forensic firearm comparisons has not been demonstrated to be scientifically valid. Scientific validity requires adequately designed studies examiner performance in terms accuracy, repeatability, reproducibility. Researchers have performed “black-box” with goal estimating these measures. As statisticians expertise experimental design, we a literature search such date then evaluated design statistical analysis methods used each study. Our conclusion is all our methodological flaws are so grave they render invalid, is, incapable establishing scientific firearms examination. Notably, error rates among examiners, both collectively individually, remain unknown. Therefore, statements about common origin bullets or cartridge cases based examination “individual” characteristics do basis. We provide some recommendations for future studies.

Language: Английский

---

Commentary on: Monson KL, Smith ED, Peters EM. Repeatability and reproducibility of comparison decisions by firearms examiners. J Forensic Sci. 2023;68(5):1721–40. https://doi.org/10.1111/1556‐4029.15318

Journal of Forensic Sciences, Journal Year: 2023, Volume and Issue: 68(6), P. 2214 - 2215

Published: Oct. 10, 2023

Language: Английский

---

Authors' response

Journal of Forensic Sciences, Journal Year: 2023, Volume and Issue: 68(6), P. 2216 - 2218

Published: Oct. 10, 2023

Language: Английский

---