Inconclusive decisions and error rates in forensic science

Forensic Science International Synergy, Journal Year: 2024, Volume and Issue: 8, P. 100472 - 100472

Published: Jan. 1, 2024

In recent years, there has been discussion and controversy relating to the treatment of inconclusive decisions in forensic feature comparison disciplines when considering reliability examination methods results. this article, we offer a brief review various viewpoints suggestions that have recently put forth, followed by solution believe addresses decisions. We consider issues context method conformance performance as two distinct concepts, both which are necessary for determination reliability. Method relates an assessment whether outcome is result analyst's adherence procedures define method. reflects capacity discriminate between different propositions interest (e.g., mated non-mated comparisons). then discuss implications these science community.

Language: Английский

---

Accuracy of comparison decisions by forensic firearms examiners

Journal of Forensic Sciences, Journal Year: 2022, Volume and Issue: 68(1), P. 86 - 100

Published: Oct. 1, 2022

Abstract This black box study assessed the performance of forensic firearms examiners in United States. It involved three different types and 173 volunteers who performed a total 8640 comparisons both bullets cartridge cases. The overall false‐positive error rate was estimated as 0.656% 0.933% for cases, respectively, while false negatives 2.87% 1.87% respectively. majority errors were made by limited number examiners. Because chi‐square tests independence strongly suggest that probabilities are not same each examiner, these maximum‐likelihood estimates based on beta‐binomial probability model do depend an assumption equal examiner‐specific rates. Corresponding 95% confidence intervals (0.305%, 1.42%) (0.548%, 1.57%) positives (1.89%, 4.26%) (1.16%, 2.99%) results this consistent with prior studies, despite its comprehensive design challenging specimens.

Language: Английский

---

Inconclusives, errors, and error rates in forensic firearms analysis:Three statistical perspectives

Forensic Science International Synergy, Journal Year: 2022, Volume and Issue: 5, P. 100273 - 100273

Published: Jan. 1, 2022

Error rates that have been published in recent open black box studies of forensic firearms examiner performance very low, typically below one percent. These low error challenged, however, as not properly taking into account the categories, "Inconclusive", examiners can reach comparing a pair bullets or cartridges. challenges themselves challenged; how to consider inconclusives and their effect on is currently matter sharp debate. We review several viewpoints put forth, then examine impact from three fresh statistical perspectives: (a) an ideal perspective using objective measurements combined with algorithms, (b) basic sampling theory practice, (c) standards experimental design human studies. Our conclusions vary perspective: be simple errors (or, other hand, simply correct at least well justified); need counted bring doubt assessments rates; are potential errors, more explicitly, necessarily equivalent casework mask casework. From all these perspectives, it impossible read out trustworthy estimates those which carried date. At most, reasonable bounds rates. much larger than nominal reported To get straightforward, sound requires challenging but critical improvement A proper study-one yields direct, rates-will require new measures blind proficiency testing embedded ordinary

Language: Английский

---

Planning, design and logistics of a decision analysis study: The FBI/Ames study involving forensic firearms examiners

Forensic Science International Synergy, Journal Year: 2022, Volume and Issue: 4, P. 100221 - 100221

Published: Jan. 1, 2022

This paper describes design and logistical aspects of a decision analysis study to assess the performance qualified firearms examiners working in accredited laboratories United States terms accuracy (error rate), repeatability, reproducibility decisions involving comparisons fired bullets cartridge cases. The purpose was validate current practice forensic discipline firearms/toolmarks (F/T) examination. It elicited error rate data by counting number false positive negative conclusions. Preceded experimental design, decisions, logistics described herein, testing ultimately administered 173 qualified, practicing F/T public private crime laboratories. first round evaluated accuracy, while two subsequent rounds repeatability examiner project expands on previous studies many challenging executing recommended double-blind format.

Language: Английский

---

Validity of forensic cartridge-case comparisons

Proceedings of the National Academy of Sciences, Journal Year: 2023, Volume and Issue: 120(20)

Published: May 8, 2023

This article presents key findings from a research project that evaluated the validity and probative value of cartridge-case comparisons under field-based conditions. Decisions provided by 228 trained firearm examiners across US showed forensic comparison is characterized low error rates. However, inconclusive decisions constituted over one-fifth all rendered, complicating evaluation technique's ability to yield unambiguously correct decisions. Specifically, restricting only conclusive identification elimination yielded true-positive true-negative rates exceeding 99%, but incorporating inconclusives caused these values drop 93.4% 63.5%, respectively. The asymmetric effect on two occurred because were rendered six times more frequently for different-source than same-source comparisons. Considering value, which decision's usefulness determining comparison's ground-truth state, predicted their corresponding states with near perfection. Likelihood ratios (LRs) further greatly increase odds state matching asserted decision. Inconclusive also possessed predicting status having LR indicating they status. study manipulated difficulty using models produce dissimilar markings. model chosen being difficult received comparisons, resulting in lower rate compared less model. Relatedly, exhibited strongly predictive

Language: Английский

---

The false promise of firearms examination validation studies: Lay controls, simplistic comparisons, and the failure to soundly measure misidentification rates

Journal of Forensic Sciences, Journal Year: 2024, Volume and Issue: 69(4), P. 1334 - 1349

Published: April 29, 2024

Abstract Several studies have recently attempted to estimate practitioner accuracy when comparing fired ammunition. But whether this research has included sufficiently challenging comparisons dependent upon expertise for accurate conclusions regarding source remains largely unexplored in the literature. Control groups of lay people comprise one means vetting question, assessing comparison samples were at least enough distinguish between experts and novices. This article therefore utilizes such a group, specifically 82 attorneys, as post hoc control juxtaposes their performance on set cartridge case images from commonly cited study (Duez et al. J Forensic Sci. 2018;63:1069–1084) with that original participant pool professionals. Despite lacking kind formalized training experience common latter, our participants displayed an ability, generally, cases by same versus different guns 327 they performed. And while rates lagged substantially behind those professionals same‐source comparisons, different‐source was essentially indistinguishable trained examiners. indicates although we vetted may provide useful information about professional performing it little offer terms measuring examiners' ability guns. If similar issues pervade other studies, then there is reason rely false‐positive generated.

Language: Английский

---

The Hawthorne effect in studies of firearm and toolmark examiners

Journal of Forensic Sciences, Journal Year: 2025, Volume and Issue: unknown

Published: April 10, 2025

Abstract The Hawthorne effect refers to the tendency of individuals behave differently when they know are being studied. In forensic science domain, concerns have been raised about “strategic examiner,” where examiner uses different decision thresholds depending on whether in a test situation or working an actual case. blind testing conducted by Houston Forensic Science Center (“HFSC”) firearms examination presents unique opportunity hypothesis that rate inconclusive calls differs for discovered vs. undiscovered tests firearm examination. Over 5 years, 529 item comparisons were filtered into casework at HFSC. items was 56.4%, while 39.3%. Thus, percentage 43.5% higher among than items. This pattern results held bullet (83% 59%) and cartridge case (29% 20%) both same‐source different‐source comparisons. These findings corroborate examiners tested demonstrate necessity if research goal is evaluate performance conducting casework.

Language: Английский

---

Repeatability and reproducibility of comparison decisions by firearms examiners

Journal of Forensic Sciences, Journal Year: 2023, Volume and Issue: 68(5), P. 1721 - 1740

Published: July 2, 2023

Abstract In a comprehensive study to assess various aspects of the performance qualified forensic firearms examiners, volunteer examiners compared both bullets and cartridge cases fired from three different types firearms. They rendered opinions on each comparison according Association Firearm & Tool Mark Examiners (AFTE) Range Conclusions, as Identification, Inconclusive (A, B, or C), Elimination, Unsuitable. this part study, sets used previously characterize overall accuracy were blindly resubmitted repeatability (105 examiners; 5700 comparisons cases) reproducibility (191 bullets, 193 cases; 5790 comparisons) examinations. Data gathered using prevailing AFTE also recategorized into two hypothetical scoring systems. Consistently positive differences between observed agreement expected indicate that exceed chance agreement. When averaged over cases, decisions (involving all five levels Range) was 78.3% for known matches 64.5% nonmatches. Similarly 67.3%% 36.5% For reproducibility, many disagreements definitive inconclusive category. Examiner are reliable trustworthy in sense identifications unlikely when comparing non‐matching items, eliminations they matching items.

Language: Английский

---

How to make better forensic decisions

Proceedings of the National Academy of Sciences, Journal Year: 2022, Volume and Issue: 119(38)

Published: Sept. 13, 2022

Much of forensic practice today involves human decisions about the origins patterned sensory evidence, such as tool marks and fingerprints discovered at a crime scene. These are made by trained observers who compare evidential pattern to an exemplar produced suspected source evidence. The decision consists determination whether two patterns similar enough have come from same source. Although comparison disciplines for decades played valued role in criminal investigation prosecution, extremely high personal societal costs failure-the conviction innocent people-has elicited calls caution development better practices. been heard scientific community involved study information processing, which has begun offer much-needed perspectives on measurement, discrimination, classification context. Here I draw well-established theoretical empirical approach science illustrate vulnerabilities contemporary suggest specific strategies improvement.

Language: Английский

---

Shining a Light on Forensic Black-Box Studies

Statistics and Public Policy, Journal Year: 2023, Volume and Issue: 10(1)

Published: May 23, 2023

Forensic science plays a critical role in the United States criminal justice system. For decades, many feature-based fields of forensic science, such as firearm and toolmark identification, developed outside scientific community's purview. The results these studies are widely relied on by judges nationwide. However, this reliance is misplaced. Black-box to date suffer from inappropriate sampling methods high rates missingness. Current black-box ignore both problems arriving at error rate estimates presented courts. We explore impact each type limitation using available data court materials. show that rely non-representative samples examiners. Using case study popular ballistics study, we find evidence may commit fewer errors than wider population which they came. also missingness non-ignorable. recent latent print ignoring likely systematic underestimates rates. Finally, offer concrete steps overcome limitations.

Language: Английский

---