When you know, you know, and that’s the problem
Open and blind proficiency tests
Laboratories of all types routinely participate in what’s called proficiency testing. Clinical, pathology, public health, engineering, and other types of laboratories test their performance for the analyses or measurements they provide. In proficiency testing, one or more samples are sent around between a number of laboratories; the samples have a known correct answer. Each laboratory has their employees complete the test according to a given set of instructions and the laboratory’s protocol. The employees report their results to their quality assurance administrator, who, in turn, sends them to the testing agency or provider. The agency or provider then sends out a report to each laboratory telling them how their employees did on the test compared to the other participants. Based on the employees’ results, the laboratory management might identify staff that need additional training, identify weaknesses in protocols and practices, and look for ways to improve the laboratory’s quality overall. Based on a government report, 98% of forensic laboratories participate in proficiency testing.
The key to proficiency tests is standardization: Units and measures and methods need to be defined and applied properly in order to get the correct results. What “correct” is differs from test to test, however, especially for forensic laboratories. Most clinical laboratories receive a narrow range of fairly pristine samples. In forensic science, anything from a single fiber to an entire city block can be evidence. If an illicit substance proficiency test sample was, for example, a small baggie of powdered baby formula with a known amount of cocaine. The laboratory would get one baggie per employee who conducts drug tests and have them follow the laboratory’s protocols. If the actual amount of the cocaine was 0.5% of the total weight of the sample, then the results need to be very close (plus or minus a certain value) to that amount. The laboratory would indicate what methods and instrumentation were used (based on what they have). If the proficiency test was a shoe print, however, there’s no measure of “plus or minus.” Shoe print comparison is a qualitative process and the range of answers and their wording varies from laboratory to laboratory. This can make the evaluation of “correct” answers difficult, let alone inconclusive results (a topic for a future posting). Nevertheless, accredited laboratories participate in proficiency testing and some are required to do so by law at least annually.
Proficiency testing allows laboratories to evaluate their performance and find out if it is satisfactory for the intended purpose. Comparison of results among laboratories allows them to evaluate their employees, protocols, and instrumentation. Proficiency testing is a critical aspect of quality assurance programs, especially in forensic science. So what’s the big deal?
You know it’s a test
Most proficiency tests are open, meaning that examiners are aware that they are being tested. As anyone who’s taken a test, you know you act differently when it’s “test time.” If a forensic examiner fails a proficiency test, they may be taken off of casework, demoted, or receive some other punitive measure (by the way, people don’t fail, systems do). Forensic examiners “may be unusually diligent and cautious when they know they are being observed and tested.” Who wouldn’t? Critics have concerns about how well proficiency tests represent forensic examiners’ actual practice. They cite the Hawthorne effect, which posits that people behave differently when they are aware they are being monitored. The argument, and it’s a good one, is that the consequences of the test-takers’ performance affect what they do.
And they’re too easy
Other than the Hawthorne effect, others have said that forensic proficiency tests are too easy and some in the profession agree. One proficiency test provider admitted that the company was under commercial pressure to make the tests easier, thus making sure sales continued. Part of this is that crime scene samples are messy and adulterated while the proficiency test samples are clean and of higher quality.
Once the tests are distributed to forensic examiners, the concept is for the examiners to conduct the tests separately; sometimes this doesn’t happen. Collusion, discussion, and data left behind on instruments can all change the way the examiners perform on the test (I’ve personally heard examiners discuss proficiency test results before submitting them). Thus, as a true measure of examiner “proficiency” (and there’s a whole big discussion to be had about what that word means), some critics feel that proficiency testing in forensic science is often worthless.
Why this matters
But that’s not the point of proficiency testing: It’s supposed to be a central tool for assessing quality in a forensic laboratory. Therefore, for the proficiency testing process to be useful and beneficial, the testing samples must be treated like regular samples. Otherwise, the proficiency being tested is not the actual, daily proficiency of the employee or the laboratory. Therefore, the correct environment for a proficiency test should replicate the normal environment for routine testing. Doing that in a clinical laboratory where the samples all come in the same way and are anonymized for patient privacy is one thing. How does a forensic laboratory pretend that proficiency tests are regular cases with messy evidence from a plurality of submitters?
Blinders on
Blind proficiency tests are made to look like real cases that have been submitted by a law enforcement agency. They need to be created and packaged in a way that does not tip the examiners to the fact that they are completing a test. As you might imagine, in Forensic Land, this can get complicated. A clinical technician is not going to call a physician and ask, “Did you draw this blood from the left arm or the right arm?” or “Did the patient have any contact with their spouse the night of the blood draw?”. Context in forensic science is important and examiners often are in contact with the submitting agency to clarify where pieces of evidence came from. The challenges to blind proficiency testing range from the creation of realistic test cases and packaging, the costs, the logistics of getting the case into the laboratory “normally” and tracking it as a blind proficiency test. These are problems solved by creativity and funding. The real problem is getting the personnel to accept that they will be blind proficiency tested.
Management, yes; employees, meh
Upper management is usually thrilled with the idea of blind proficiency testing. The examiners, well, not so much. Examiners who have never worked in a laboratory that used blind proficiency testing have the worst opinions about the process. They got their hackles up when asked whether it would increase their diligence in their casework. It seems they already believed that they were already quite diligent in their work, which most people might think. But, forensic science is a notoriously defensive profession and takes to criticism like sodium and potassium take to water. Examiners who have worked in laboratories with blind proficiency testing, however, have a more positive take on it as significantly more necessary for accurate assessment of examiner’s proficiency. No examiners who worked in a laboratory with blind testing described such testing as unnecessary or without benefit.
What to do?
Forensic laboratory management needs to adopt and promote a culture of continuous improvement where finding problems, calling them out, and openly saying “We fixed it!” is a good thing. This is closer to a science mindset than a policing one (I’m just going to leave that there for now). Many other industries follow the find it-fix it approach, like aviation and medicine. The forensic enterprise’s stakeholders could provide momentum for the profession as well:
Quality assurance staff
Laboratory management
Law enforcement
Prosecutors
Defense attorneys
Judges
Professional associations (like the Association of Forensic Quality Assurance Managers and the American Society of Crime Laboratory Directors)
Accreditation bodies
Proficiency test providers
Researchers at academic institutions
The solution is systemic and much larger than just saying, “Oh, we do blind proficiency testing.” It is, however, a good start. Changing the culture of forensic science requires deliberate thinking about what sort of ‘culture’ will be conducive to producing whatever it is we want from forensic science. So far, only a few have put their thinking caps on.