Chapter 12. DNA testing was not safeguarded against error

Each of us is genetically unique, and there are many cases in which it is convenient to make use of our genetic individuality: for parentage analysis, identification of victims, and identification of criminals. DNA provides one of the most specific methods of "typing" a person, but many features of ideal data are being violated when evidence has been gathered for criminal prosecution.

Motivation

Someone has committed a violent crime, and some blood was left at the crime scene. The blood type (presumed to be that of the assailant) is AB. The suspect also has blood type AB. Is this fact sufficient evidence of guilt? No, for several reasons. One reason is that AB is too common in the general population to warrant any conclusions as to the guilt of the suspect. If we have more information such as: the genotype of the blood at the scene is both AB and X1, and the suspect is both of these, it is now perhaps more likely that the suspect is the source of the crime-scene blood. However, it is necessary to know how common is the combination of AB and X1 in the population of possible assailants. If this combination is very common, then we still do not have much more information than we had with the AB blood type alone. We want enough information to know that the chance of finding a random person with that genotype is small.

What is DNA typing?

DNA typing is a method in which our genetic material (DNA) is converted into a series of bands that, in principle, ultimately distinguish each of us from nearly everyone else on earth. DNA is easily recovered from sperm, urine, blood, and hair, so that criminals often unwittingly leave their DNA at crime scenes, and the DNA of victims is even sometimes carried away on the clothes of their assailants. By using DNA, we are thus often able to place individuals at crime scenes, and in the case of rape, are able to identify the man who "provided" the sperm.

Recent numbers. By 1990, DNA technology had been used in over 2000 court cases in the U.S., encompassing 49 states and Washington D.C. The October 12, 1991 Austin American Statesman reported that Williamson County's first use of DNA typing had just resulted in the conviction of a rape suspect, who was sentenced to 99 years in prison. Not all DNA typing has led to convictions, however. From any attempt to match a DNA fingerprint between suspect and forensic sample, three outcomes are possible. For the U.S. up to 1990, these outcomes (and their frequencies) were: (i) exclusion of the suspect (37%), (ii) inability to resolve the DNA fingerprint (20-25%), and declaration of a match (40%). DNA typing was cast into the national spotlight for good, however, with the O.J. Simpson trial in the fall of 1994.

The typing process

The emphasis in this chapter is on ideal data, but we cannot address this subject without explaining the model used in DNA typing. DNA typing involves identifying the lengths of specific pieces of our chromosomes, which are made of DNA. As its final product, the method produces a bar-code pattern, and it is the number and positions of the bars that are used to identify us. In essence, our genome characterizes us with something akin to a Social Security number, except that the digits are replaced with bars along an axis. The method by which we obtain the bar-code involves five basic steps:

• 1) Obtain a DNA sample from tissue (blood, hair, feces, saliva, urine, sperm), either directly from a person or from a crime scene.
• 2) Cut the DNA into small pieces at certain chromosome landmarks (so the DNA is cut at the same positions, regardless of the sample). Although these pieces are cut at the same landmarks in everyone, the lengths of the pieces vary enormously from person to person (and even vary between the two copies present in each person). It is this variation that makes DNA typing useful.
• 3) Force the cut DNA to move through a gel - a flat piece of expensive jello, so that short pieces separate from long ones over the length of the gel. The only purpose of this step is to sort the pieces by their length. Once the pieces are spread out across the gel (short ones at one end, progressively larger ones toward the other end), they are kept in place.
• 4) Visualize specific pieces according to the landmarks they possess; in DNA jargon, a probe is used to highlight specific fragments, and a picture is taken (usually on X-ray film). The positions of bands (the bar-code) tell us the lengths of the pieces of DNA containing the landmark, thus telling us the band size.

If two DNA bar-codes appear to be identical when compared for many different probes, then the likelihood that both samples came from the same person is greatly increased in most circumstances. To calculate the probability that two different people would have the same bar-code, we need to know the frequency in the population of the different bands in the bar-code (plus some correction factors that needn't concern us here). So to make use of a match, it is also necessary to have a large database of DNA profiles of different people so that we can estimate the frequencies of the different bands. Of course, if the bar-codes of two profiles differ, then we know that they did not come from the same person.

Limitations. The bar code is a physical model of the sizes of specific pieces of a person's DNA. The position of each band on the film represents the distance that the piece of DNA moved after it was put in the gel, and there are several factors that can influence how far the DNA moves in addition to its length. There are many sources of possible error in addition to the ones noted previously:

• Sample degradation can cause pieces of DNA to appear shorter than they actually are. It is a problem with many forensic applications, because the DNA is often recovered from the crime scene hours or days after the sample is deposited.
• The extent to which a piece of DNA moves in a gel depends on vagaries of an electrical current, salt concentrations, and lots of other factors that cannot be controlled exactly. As a consequence, two DNA fragments the same don't always move similarly in a gel, and the result is that the band size will vary even though the fragment sizes are the same. Conversely, two pieces of DNA of slightly different sizes can give indistinguishable band sizes.

With this background, we can now consider whether the data gathered for U.S. criminal prosecution fulfills the features of ideal data.

Ideal data?

DNA typing is still a new technology, and its introduction to the courts in the U.S. has been hotly contested by some scientists for various reasons. Relevant to this chapter was the discovery that the DNA typing process itself was not meeting ideal data criteria. In contrast to DOT drug testing, here is no set of standards for performing DNA typing: the labs doing DNA typing work do not have to meet certification standards, they are not subject to required blind testing procedures, and at least initially, they were reluctant to release information about how they conducted their work. The remainder of this chapter explains more about the nature of DNA typing and which features of ideal data are relevant to the work. We foreshadow these findings in the following template:

Explicit protocols

The DNA typing process is one horrendous set of protocols, as the model suggests. In fact, there are chapters of lab books devoted to the different ways that this procedure can be undertaken. No one could hope to undertake even part of this method without relying on highly explicit biochemical methods that have been worked out for decades. And, by and large, the labs do adhere to such protocols. But protocols apply to each step in the process, and deviations from protocols can take many forms: mislabeling samples, contaminations, losing or ruining the DNA, failing to add certain liquids or salts at different parts of the process, and so on. The debates in the courts have been revealing that the explicit protocols that a lab claims to adopt have in fact sometimes been violated in important respects.

Some violations of a protocol will ruin the typing process, whereby the lab simply lacks any bar code at the final step. These violations are obviously unintentional and any lab that commits them even occasionally will soon be out of business. But many violations of protocol aren't so blatant, yet they can have a big impact on the conclusions reached. In some cases, a protocol can be violated in such subtle ways that it makes the difference between a match being declared versus a mismatch being declared. The most obvious such violation applies to the protocol by which a match between two bar codes is declared to exist or not exist. By eye, the bar codes may appear to match when in fact they don't match by rigid criteria. A lab failing to adhere to its protocol can then reach any conclusion it wishes. And an absolutely fatal violation of protocol occurs when samples are mislabeled or mixed up. This single mistake can free a guilty person or convict an innocent one. The possibility of sample mixup is itself a function of the protocol for processing and handling the specimens, and a good protocol can virtually eliminate the possibility of sample mixup.

Replication

The forensic sample of DNA may often be so small that the lab can only type it once. (Multiple probes can always be used on a single DNA specimen, but our concern here is that the sample may be small enough that all of it must be used on a single gel.) In this case, the most basic feature of replication cannot be employed: the forensic sample cannot be processed twice. Any errors that arise in the gel cannot be discovered, because there is no second chance. (Newer technology may enable replication from small samples, however, and permit replication of the entire typing process.)

But replication is not totally absent from the typing process. Labs that perform DNA typing undertake replicate runs of samples that they have in abundance, and from this kind of replication they come to understand how much error is present in the typical typing procedure. So the main concern about the lack of replication concerns small forensic samples that can be processed and analyzed only once.

Standards are used in all typing procedures at one basic level. When the DNA is "size-fractionated" on a gel, the only way to know how far fragments of a particular size have moved through the gel is to include fragments of known length on the gel (these pieces of known size are called "size" standards). So size standards are used to calibrate the bar codes of the forensic and suspect samples. And if something funny has happened during the process, the standards will reveal it by exhibiting an anomalous pattern.

Other kinds of standards should be used as well, but often aren't. There is one kind of probe, for example, that can be used to identify male DNA by the presence of a band; female DNA does not show any band. When using this probe, it is essential that a sample of known male DNA be present, to be sure that the probe worked the way it should; this sample of known male DNA serves as a comparison for any female DNA that might be present. And sample standards ought to be included with the suspect's samples when DNA is sent to a lab for typing; these standards would help detect a sample mixup and detect a laboratory's inability to correctly type a sample.

Randomization is relevant to only a limited extent. For many aspects of DNA typing, no choice needs to be made between otherwise equivalent samples or people. However, any choice of which standards to use should be made randomly. As there has not been a systematic effort to include sample standards in DNA typing procedures, there has not even been an opportunity to apply randomization.

Blind designs are not used and in fact have been deliberately avoided by such agencies as the FBI. The samples are clearly identified as to whether they come from the suspect, victim, or crime scene, and the full criminal history of the suspect is known at the time as well. And in scoring band positions, the standard method is to look at all samples on a gel at once, rather than scoring band positions for each sample in isolation (blindly) from other samples.

The failure to include blind procedures makes it easy for a lab to "produce" a match between suspect and forensic sample when a match does not exist. Merely consider how you would respond to the accusation that a formerly convicted mass murderer was the suspect in another murder versus the accusation that a member of the Mormon Tabernacle Choir was the prime suspect. Knowledge of this evidence could subconsciously influence the care you devoted to preventing sample mixup and could influence your acceptance of the results.

Case histories in DNA typing : The Castro case

At its inception, DNA typing was heralded as a method of identifying culprits with virtual certainty. Yet soon after this honeymoon phase, a few scientists discovered that the typing labs were so sloppy that they failed to live up to even the most elementary standards observed throughout the scientific community. The most publicized exposition of sloppy DNA typing was authored by Eric Lander in the scientific journal Nature (1989, Vol 339: 501). Lander had volunteered to evaluate DNA typing evidence in a pre-trial hearing (known as a Frye hearing), in which the admissibility of the DNA evidence was being considered. The crime was the murder of a woman and her daughter, and the accused murderer was a neighbor, Jose Castro. The only evidence to connect him with this crime was a small blood spot on his watch, which had been analyzed by the lab Lifecodes. The lab report declared that the blood DNA on his watch matched the blood DNA of the deceased mother. Lifecodes had analyzed this DNA with 4 probes and declared the probability of a random match to be 1 in 100 million. Their lab report failed to identify any abnormalities or deviations from their published protocols.

A careful inspection of the lab report by Lander and the defense noted some rather astounding discrepancies between what the report claimed and what had actually been done. The four most extreme examples were:

• i) Unspecified protocol. The number of bands in the DNA profiles (i.e. the number of bands in the bar code) differed between watch DNA and mother DNA for one of the probes used (5 versus 3 bands). Of the 5 bands in the watch DNA, two matched the bands observed in the mother's DNA, which was the reason the match was declared. The presence of extra bands could mean that the watch blood was contaminated with additional DNA, or it could mean that the blood did not belong to the mother. The lab's failure to indicate how it declared a match in this case amounts to an unspecified protocol and, at the very least, should have influenced how they calculated the probability of a random match.
• ii) Violation of protocol. Although the lab had declared a match between the watch DNA and mother DNA, the actual positions of the bands differed slightly between the two samples. The lab had interpreted these differences as technical artifacts, but the differences between the two bar codes had nonetheless exceeded the labs own standards for declaring a match. The lab had in fact judged the match by "eyeball," rather than by numbers. This procedure amounts to a violation of protocol in declaring a match.
• iii) Unspecified protocol. The daughter DNA revealed an excess of bands for one probe: two bands were expected, yet 4 were observed. The significance of this result is somewhat subtle. It could mean that the daughter DNA was contaminated, but that possibility is unlikely because the DNA would have been obtained from relatively pure tissues from the body. At a minimum, this error amounts to the use of an unspecified protocol, and again, it should have influenced the calculations of a random match probability.
• iv) Lack of standards. An attempt to identify the sex of the watch DNA had not included a male DNA standard, so there was no way to determine if the test had correctly demonstrated that the watch DNA was female. In an incredible series of exchanges on the witness stand, Lifecodes changed its explanation for the lack of a male standard 3 times.

Overall, Lifecodes' explanation for discrepancies between their report and the actual data were filled with ad hoc speculations and assumptions that had not been tested. The arrogance displayed by the lab was clearly shocking to Lander, as it has been to many of the people reading his article. While violations of protocol are inevitable at some level, the violations noted above made the difference between a match being declared at odds of 1 in 100 million versus the failure to declare a match at all. Even granting Lifecodes the most liberal interpretation of their violations, their calculation of the random-match probability should have been substantially altered from 1 in 100,000,000. In fact, during these hearings, a meeting of experts who had testified on both sides was convened out of court to discuss these matters (excepting the senior representative from Lifecodes). This "bipartisan" group of scientists concluded that the data were not reliable enough to declare a match or mismatch. Note that this conclusion did not indicate that the watch blood was not the mother's - only that they couldn't tell. Nonetheless, the prosecution pressed ahead with the DNA evidence, against the advice of its own witnesses. The judge threw out the DNA evidence, but Castro later confessed to the murders.

Widespread violations of ideal data

The Castro case may be unique in some respects, but it has been commonplace for prosecution agencies to ignore what scientists would regard as standard procedures in gathering data. Here is the text of two letters sent from the Chicago Police Department to the FBI, requesting DNA typing. All names are omitted from our text; where names were included in the letters, a description is given in square brackets [].

Letter 1: From Chicago Police Crime Lab to F.B.I. DNA Laboratory Division, 10 August, 1989

Dear [name of Director of F.B.I. lab],

I am writing to request that DNA typing be performed on several items of serological evidence. The names of the people involved are: [name of female victim] F/W (the victim) and [name of male suspect] M/B (the suspect). The evidence I am sending you consists of the following:

• Blood standard from [name of victim]
• Blood standard from [name of suspect]
• Extract from swab
• Extract from victim's pants
• Extract from victim's bra

All three of these extracts were found to be semen/spermatozoa positive and the two extracts from the clothing were found to have ABO, PGM and PEP A activity consistent with that of the suspect. I am also enclosing a copy of my laboratory report stating these results.

The facts of the case are that on 25 May 1989, the victim was grabbed from behind, pulled into the woods and sexually assaulted. The victim never got a good look at her offender and therefore is not able to make a positive I.D. of the suspect. The suspect [name] had just been released from the ILLINOIS DEPARTMENT OF CORRECTIONS after serving time for the same type of crime in the same area. At this time the suspect has not been charged.

Thank you very much for your assistance in this matter. Please feel free to contact me if you need more information.

Letter 2: From Chief of Detective Division, Chicago Dept. of Police to F.B.I. DNA lab

Dear [name, Commanding Officer, F.B.I. DNA lab],

In early January, 1990, detectives assigned to the Chicago Police Department's Detective Division, Area Three Violent Crimes Unit were assigned to investigate the particularly brutal Aggravated Criminal Sexual Assault, Robbery and Kidnapping of one [name of victim], recorded under Chicago Police Department Records Division Number N-005025. On January 10, 1990, one [name of suspect] M/N/31 years, FBI [#], C.P.D. Record Number [#], was arrested and charged with this and other offenses.

Blood and saliva samples of the offender and victim were obtained and tendered to Technician [name of technician] of the C.P.D. Criminalistics Unit. A sexual assault kit (Vitullo Kit) was also completed and submitted for the victim.

If any additional information is needed, kindly contact Detective [name], star [#], Area Three Violent Crimes Unit, 3900 South California, Chicago, Illinois 60632, Telephone #(312)-744-8280, or the office of the undersigned.

The most striking feature of these letters is the absence of blind testing. Information about the suspect is contained in the letters (even including alleged crimes). Furthermore, no standards are included, which would offer quality control assurances as well as guard against sample mixup. As far as we know, these letters are typical. The law does not require blind testing or standards, and the police units may not even recognize the possible consequences of omitting these design features.