Aug 13 2019
Under the Microscope – Kevin Cheng
Reproducibility is one of the main principles of the scientific method. Reproducibility in forensic DNA interpretation typically relates to the reported match statistic. The development of probabilistic genotyping (PG) solutions using semi-continuous and continuous models was partly in a response to intra and inter laboratory differences to improve reproducibility of the results. However, there are still some aspects of DNA profile interpretation that are not controlled by PG software, and variability is to be expected.
In his presentation at ISHI, Kevin Cheng (Scientist Developer at the Institute of Environmental and Science Research (ESR)) will present on a collaborative exercise of 174 participants, across 42 laboratories, that aimed to define sources of variation. We sat down with Kevin and asked him to explain the challenges that exist with reproducibility in mixture interpretation, his biggest takeaways from the exercise, and his advice for those who are new to the field of forensics.
Kevin, thanks for chatting with us today! What are the challenges that exist with reproducibility in mixture interpretation?
We define reproducibility as the variation between results under changed conditions of measurement, for example by different analysts or different methods. The result we are interested in is typically the match statistic.
The development of probabilistic genotyping (PG) solutions was partly in a response to intra and inter laboratory studies that demonstrated some variation in match statistic within and between laboratories.
PG removes many of the subjective decisions when interpreting DNA profiles however, there are still some aspects of DNA profile interpretation that are not controlled by PG software, and variability is to be expected. At each step in the DNA analysis prior to interpretation, some variability may be introduced that can impact the reproducibility of the statistic after interpretation. Therefore, the challenges of reproducibility in mixture interpretation stems from the differences in standard operating procedures.
Differences in interpretation of the same sample between laboratories are expected as they use different kits, different capillary electrophoresis models, different analytical thresholds, different allele frequencies, and theta values etc. Differences within the same laboratory may also be encountered due to some analyst decisions such as the number of contributors and setting propositions within the LR assignment.
What inter-laboratory studies have been done so far to explore reproducibility? What was the result of these studies?
Recently a number of inter-laboratory studies conducted using PG solutions have been reported such as the use of LRmix and LRmix Studio in the Euroforgen-NoE (Prieto et. al, 2014) and GHEP-ISFG (Barrio et. al, 2018) collaborative exercises, and the use of STRmix™ in the study by Stuart Cooper et. al (2015). These studies highlighted that differences in LR were due to assigning the number of contributors, laboratory specific operating procedures when analyzing DNA profiles (e.g. utilizing different stutter filter thresholds), and differences in laboratory specific parameters (e.g. drop-in and dropout parameters).
Can you briefly describe the study that you’ll be presenting at ISHI? What is the goal?
In this collaborative exercise of 174 participants, across 42 laboratories, we aimed to explore the sources of variation in the LR due to analysis and interpretation methods. Two complex mixtures from the PROVEDIt dataset (Alfonse et. al, 2018) were analyzed by each participant and propositions based on the provided case circumstances were determined by each participant. We restricted some known sources of variation by requiring all participants to use the same allele frequency and theta values. We show a convergence of results within, and between, laboratories. The reproducibility of the LR reported in this exercise can be attributed to the use of PG software and supports the ongoing transition of forensic laboratories to probabilistic genotyping.
How did this case study come to fruition? How did you become involved?
We noted that variability in the LR between analysis within a lab and between labs was being raised as a concern in court. We wanted to explore the variability within the community of STRmix users. The study was initiated by Drs Bright and Buckleton and they invited me in the early stages of this study to assist with the data collation.
Did any surprises arise during the course of the study?
From my point of view, the biggest surprise was the impact that differences in the local data analysis settings (such as smoothing and normalization) had on the LR due to variation in the peak heights. The differences in analysis methods between laboratories is not something I have previously considered. At the extreme, this resulted in some peaks being observed just above or just below the analytical threshold. As expected, these differences in peak heights will impact the assigned LR.
What has been your biggest takeaway from this experience?
My biggest takeaway from this study is that whilst variation between labs can be minimized it can never be eliminated. I think it is also equally as important for forensic scientists to remember to review their analysis methods prior to analyzing a profile, just to make sure none of the settings have been changed.
What tips would you give to someone who is just starting out in the field of forensics, or what is the best advice that you’ve received?
For someone starting out in the field of forensics as a scientist, it is important to remember that in forensic science, any decision that is made can impact someone’s life. Therefore, if you are ever uncertain or do not know something, it is important to ask and learn from your peers and mentors. It is equally as important to have your work peer-reviewed by another scientist.
As we celebrate the 30th anniversary of ISHI, do you have any predictions for what the future holds or do you have any fond memories of using older technologies/techniques?
Slooten (2017) published a paper on identifying common donors in DNA mixtures. Subsequently, this theory was applied by Bright et al. (2019). I predict that in the future this theory may be used on a regular basis to find potential links between cases. However, to the best of my knowledge, this has yet to be applied to real casework. Therefore, I look forward to reading about it first application(s) in casework.
What’s one thing that others may not know about you?
This will be my first time attending ISHI, so I hope to meet more people interested in the same field and get to know more about everyone’s research.
WOULD YOU LIKE TO SEE MORE ARTICLES LIKE THIS? SUBSCRIBE TO THE ISHI BLOG BELOW!