Correspondence

Radiographic Abnormalities and Asbestos Exposure: Libby, Montana

Peipins et al. (2003) described the study conducted by the Agency for Toxic Substances and Disease Registry (ATSDR 2001, 2002a) to analyze the association between the prevalence of radiographic abnormalities and asbestos exposure pathways for residents of Libby, Montana. Although Peipins et al. presented many detailed results, they failed to explicitly state the obvious conclusion of their analysis: excess risk of asbestos-related disease for Libby residents is a consequence of occupational exposure, and risk associated with low-level exposure is negligible. This finding is extremely important for guiding future public health assessments of exposure to vermiculite from the Libby mine and exposure to amphibole asbestos in general.

Peipins et al. (2003) made a clear case for the importance of occupational exposure. They reported the following statistically significant factors for predicting pleural abnormalities: being a former mine worker; being older; having been a household contact of a mine worker; and being male. These results associate pleural abnormalities with high occupational exposure groups--mine workers and household contacts of mine workers. Only one environmental exposure pathway was statistically significant--played in vermiculite pile, which may involve exposures as high as occupational exposures.

The ATSDR data (ATSDR 2001, 2002a) indicate that 17.8% of the 6,668 subjects with X-ray films had pleural abnormalities. This percentage appears large in comparison to 6.7% of subjects with pleural abnormalities in the internal control group (no exposure). However, occupational exposures inflate the difference. I reanalyzed the data by forming three exposure groups: mine workers, residents with other occupational or domestic exposure, and residents with no occupational or domestic exposure (the "environmental exposure only" group) (Price B. In press). Pleural abnormalities were recorded for 51% of mine workers, 19.9% of residents with other occupational or domestic exposure, and 9.1% of residents with no occupational or domestic exposure.

These results again demonstrate the importance of occupational exposure for pleural abnormalities. The percentage for the "environmental exposure only" group is close to the percentage for the internal controls. Other confounding factors mentioned by Peipins et al. (2003) could further reduce the difference. One of those factors is the false positive bias due to subpleural fat, which would be expected to be greater in the environmental exposure group than in the mine worker group (Price B. In press).

Peipins et al. (2003) implied that asbestos exposure was high in Libby:

Air sampling in downtown Libby in 1975 and at several points in the 1980s detected levels well above the Occupational Safety and Health Administration's [OSHA] occupational limit of 0.1 fiber/cm³ over 8 hr of exposure (Atkinson et al. 1982; Dixon et al. 1985; U.S. Department of Labor 1994).

However, the cited references do not contain data for samples collected in 1975. Dixon et al. (1985) reported that a total of four stationary samples were collected, each for 2 hr (Atkinson et al. 1982). These samples, analyzed by two laboratories, produced the following measurements, respectively: 0.08 and 0.50 fiber/cm³, 0.10 and 0.02 fiber/cm³, 0.03 fiber/cm³ and none detected, and 0.03 and 0.02 fiber/cm³. Comparison of these 2-hr sampling results to OSHA's 8-hr limit is not meaningful. Also, because the measurements vary considerably between laboratories, the largest measurement is not a reliable estimate of exposure in Libby.

Recently, the U.S. Environmental Protection Agency (U.S. EPA 2001, 2002) measured airborne asbestos levels in Libby and estimated exposure for Libby residents engaged in activities that disturbed household dust, vermiculite attic insulation, and soil. The results, stated as lifetime average daily exposure, ranged from 0.00007 to 0.005 fiber/cm³. Risk levels calculated by the U.S. EPA for these exposures were between 1 10^-6 and 1 10^-4, the acceptable range defined by the U.S. EPA (1989) for regulatory decisions.

The study described by Peipins et al. (2003) is one of a number of studies addressing asbestos exposure and risk in Libby (Amandus and Wheeler 1987; ATSDR 2000, 2001, 2002a, 2002b; McDonald 2001; McDonald et al. 1986; McDonald JC. Unpublished data; U.S. EPA 2001, 2002). The perception that mine workers' disease rates apply to all Libby residents gained credibility through the ATSDR (2000) asbestosis mortality study: "... mortality in Libby resulting from asbestosis was approximately 40-60 times higher than expected." Later, the ATSDR (2002b) explained that the excess mortality was a consequence of occupational exposure. As noted by Peipins et al. (2003), the ATSDR observed a total of 12 asbestosis deaths: 11 males who were previously employed at the Libby mine and 1 female who was a household contact of a mine worker.

The results presented by Peipins et al. (2003) and results from other studies of asbestos in Libby indicate that occupational exposure--not low-level environmental exposure--is the most significant risk factor for asbestos-related disease. Peipins et al. should have stated that conclusion explicitly, taking the first step toward correcting misperceptions about asbestos disease in Libby and, more generally, the risk of disease associated with low-level exposures to amphibole asbestos.

The author declares he has no competing financial interests.

Bertram Price
Price Associates, Inc.
White Plains, New York
E-mail: bprice@priceassociatesinc.com

References

Amandus HE, Wheeler R. 1987. The morbidity and mortality of vermiculite miners and millers exposed to tremolite-actinolite: Part II. Mortality. Am J Ind Med 11(1):15-26.

Atkinson GR, Rose D, Thomas K, Chatfield EJ, Going JE. 1982. Collection, Analysis and Characterization of Vermiculite Samples for Fiber Content and Asbestos Contamination. EPA Contract 68-01-5915. Washington, DC:U.S. Environmental Protection Agency.

ATSDR. 2000. Health Consultation, Mortality from Asbestosis in Libby, Montana. Atlanta, GA:Agency for Toxic Substances and Disease Registry. Available: http://www.atsdr.cdc.gov/HAC/PHA/libby/lib_toc.html [accessed 7 January 2004].

------. 2001. Year 2000 Medical Testing of Individuals Potentially Exposed to Asbestiform Minerals Associated with Vermiculite in Libby, Montana: A Report to the Community. Atlanta, GA:Agency for Toxic Substances and Disease Registry.

------. 2002a. Preliminary Findings of Libby, Montana Asbestos Medical Testing (Combined Testing, 2000 and 2001). Atlanta, GA:Agency for Toxic Substances and Disease Registry.

------. 2002b. Mortality in Libby, Montana, 1979-1998. Atlanta, GA:Agency for Toxic Substances and Disease Registry.

Dixon GH, Doria J, Freed JR, Wood P, May I, Chambers T, et al. 1985. Exposure Assessment for Asbestos-Contaminated Vermiculite. EPA-560/5-85- 013. Washington, DC:U.S. Environmental Protection Agency.

McDonald C. 2001. Carcinogenicity of fibrous tremolite in workplace and general environments. Presented at the Asbestos Health Effects Conference, 24 May 2001, Oakland, CA. Available: http://www.epa.gov/swerrims/ahec/summary.htm [accessed 12 January 2004].

McDonald JC, McDonald AD, Armstrong B, Sebastien P. 1986. Cohort study of mortality of vermiculite miners exposed to tremolite. Br J Ind Med 43(7):436-444.

Peipins LA, Lewin M, Campolucci S, Lybarger JA, Miller A, Middleton D, et al. 2003. Radiographic abnormalities and exposure to asbestos-contaminated vermiculite in the community of Libby, Montana, USA. Environ Health Perspect 111:1753-1759; doi:10.1289/ehp.6346 [Online 2 July 2003]

Price B. In press. Asbestos Exposure and Health Risks: Libby Montana. J Regul Toxicol Pharmacol.

U.S. Department of Labor. 1994. Occupational Exposure to Asbestos. 29CFR Parts 1910, 1915, 1926. Final Rule. Fed Reg 59:40964-41162.

U.S. EPA. 1989. Risk Assessment Guidance for Superfund, Part B. EPA/540/1-89/002. Washington, DC:U.S. Environmental Protection Agency. Available: http://www.epa.gov/superfund/programs/risk/ragsb/index.htm [accessed 12 January 2004].

U.S. EPA. 2001. Memorandum from Chris Weis, U.S. EPA, to Paul Peronard, U.S. EPA. Amphibole Mineral Fibers in Source Materials in Residential and Commercial Area of Libby Pose an Imminent and Substantial Endangerment to Public Health. Washington, DC:U.S. Environmental Protection Agency, 20 December 2001.

U.S. EPA. 2002. Memorandum from Chris Weis, U.S. EPA, to Paul Peronard, U.S. EPA. Revised Screening Risk Estimates. Washington, DC:US Environmental Protection Agency, 22 November 2002.

Radiographic Abnormalities: Response from Peipins et al.

[citation in pubmed]

We appreciate Price's interest in our article (Peipins et al. 2003). We stated clearly that being a former W.R. Grace (WRG) worker was a significant risk factor for both pleural and interstitial abnormalities. We also noted that only age was more strongly associated with these outcomes in multivariate analyses and that these results were not unexpected.

However, we disagree with Price's statement that the obvious conclusion of our analysis is that risk associated with low-level environmental exposure is negligible. Such a conclusion ignores key results. For example, we found that playing in the vermiculite piles and longer duration of residence in Libby, Montana, were associated with pleural abnormalities, even after controlling for occupational and domestic exposures (Peipins et al. 2003). We also found that the prevalence of pleural abnormalities increased with increasing number of exposure pathways, even after we removed WRG workers from the analysis. This suggests a cumulative effect from multiple exposures that exclude working in the mine.

Price incorrectly labels our "no-apparent-exposure" group as an "internal control group." We did not have an internal no-exposure group (Peipins et al. 2003). Our no-apparent-exposure group consisted of participants who responded "no" to the exposure pathways listed in the questionnaire and who were likely exposed via ambient air and other pathways not assessed by our screening questionnaire. The rate of 6.7% for the no-apparent-exposure group in our analysis and the rate of 9.1% given by Price in his letter are considerably higher than the prevalence rates of pleural abnormalities found in published studies of other nonoccupationally exposed populations in the United States, which range from 0.2% among blue-collar workers in North Carolina (Castellan et al. 1985) to 2.3% among patients at Veterans Affairs hospitals in New Jersey (Miller and Zurlo 1996). Of note, these studies did not exclude family contacts of workers or domestic exposures (Castellan et al. 1985; Anderson et al. 1979).

When assessing subpleural fat as a confounding factor, we found former WRG workers to have higher body mass indexes (BMIs) than those who were not former WRG workers. We controlled for subpleural fat by including BMI in both our multivariate analyses and our pathways analyses. Therefore, the associations between environmental exposures, as well as occupational and domestic exposure, and pleural abnormalities remained when controlled for BMI.

In regard to Price's comments on past exposures in Libby, sampling performed by WRG in 1975 showed markedly elevated ambient air asbestos concentrations in downtown Libby [U.S. Environmental Protection Agency (EPA 2002)]. These findings are consistent with the limited ambient air samples collected by the U.S. EPA (Dixon et al. 1985; Atkinson et al. 1982). Although Price points out that the variation in detectable laboratory results ranged from 0.02 to 0.5 fiber/cm³, depending on the laboratory, it is clear that the ambient air concentrations in Libby easily approached, if not exceeded, occupational 8-hr limits. In a cross-sectional study of workers at an Ohio fertilizer plant that processed vermiculite from Libby, Montana, Lockey et al. (1984) found that workers with daily time-weighted-average exposures of 0.031-0.415 fiber/cm³, similar to the ambient air concentrations reported in Libby, had significantly elevated radiographic pleural changes and pleuritic chest pain.

Price asserts that Agency for Toxic Substances and Disease Registry (ATSDR) mortality studies conducted for the Libby area have created a false perception of the community's asbestos-related mortality experience. Results from ATSDR's mortality study (ATSDR 2002) revealed significantly elevated rates of mesothelioma, asbestosis, and lung cancer when compared with the Montana and U.S. populations. Workers were included in the determination of asbestos-related mortality in Libby, as is done as a matter of practice throughout the nation to determine comparative standardized mortality rates. Nevertheless, there were several deaths found that did not appear to be occupationally related. Notably, one of the three mesothelioma cases identified for inclusion in our study (Peipins et al. 2003) did not occur among former mine workers (ATSDR 2002). Additionally, Lincoln County, Montana, had the highest age-adjusted asbestosis mortality rate in the United States for 1988-1997, even when compared to other counties that contain large asbestos exposed workforces (Castellan R. Unpublished data).

On the basis of our results, we conclude that both occupational and environmental risk factors are important predictors of asbestos-related radiographic abnormalities in this community. We thank Price for his comments and hope that this letter provides additional insights to these issues.

The authors declare they have no competing financial interests.

Lucy A. Peipins
Centers for Disease Control and Prevention
Atlanta, Georgia

Michael Lewin
Sharon Campolucci
Jeffrey A. Lybarger
Vikas Kapil
Dan Middleton
Agency for Toxic Substances and Disease Registry
Atlanta, Georgia
E-mail: mlewin@cdc.gov

Aubrey Miller
Christopher Weis
U.S. Environmental Protection Agency Denver, Colorado

Michael Spence
Montana Department of Health and Human Services
Helena, Montana

Brad Black
Lincoln County Department of Environmental Health
Libby, Montana

References

Anderson H, Lilis R, Daum S, Selikoff IJ. 1979. Asbestosis among household contacts of asbestos factory workers. Ann NY Acad Sci 330:387-399.

ATSDR. 2002. Mortality in Libby, Montana, 1979-1998. Atlanta, GA:Agency for Toxic Substances and Disease Registry

Castellan RM, Sanderson WT, Petersen MR. 1985. Prevalence of radiographic appearance of pneumoconiosis in an unexposed blue collar population. Am Rev Respir Dis 131:684-686.

Dixon GH, Doria J, Freed JR, Wood P, May I, Chambers T, et al. 1985. Exposure Assessment for Asbestos-Contaminated Vermiculite. EPA-560/5-85-013. Washington, DC:U.S. Environmental Protection Agency.

Lockey JE, Brooks SM, Jarabek AM, Khoury PR, McKay RT, Carson A, et al. 1984. Pulmonary changes after exposure to vermiculite contaminated with fibrous tremolite. Am Rev Respir Dis 129:952-958.

Miller JA, Zurlo JV. Asbestos plaques in a typical Veteran's hospital population. 1996. Am J Ind Med 6:726-729.

U.S. EPA. 2002. Libby Site Administrative Record. Denver, CO:EPA Record Center, Superfund Reference Desk. Available: http://www.epa.gov/region8superfund/libby/qdocs020610.html [accessed 9 January 2004].

Overstating the Consequences of Radiographic Abnormalities

[citation in pubmed]

"Radiographic Abnormalities and Exposure to Asbestos-Contaminated Vermiculite in the Community of Libby, Montana, USA" by Peipins et al. (2003) is the first journal publication by the Agency for Toxic Substances and Disease Registry (ATSDR) of the results of their multiyear medical testing program of Libby residents. Peipins et al.'s (2003) conclusion that 18% of study participants had pleural abnormalities has received wide attention and has lead to understandable concern among Libby residents and health professionals. We, as principals of Health Network America (HNA) and administrators of the Libby Medical Plan (LMP), are in the unique position of having participated in the peer review of applicants and members of the LMP. The LMP is a health benefit program for the people who lived in and around Libby and developed an asbestos-related condition. The peer reviewers include two board-certified radiologists who specialize in chest radiography and/or pneumoconiosis and are certified B-readers, and a third board-certified radiologist who specializes in interpretation of pleural disease on chest computed tomography (CT) scans. Although the review process is ongoing, some of our preliminary observations are relevant because they include many cases reported as "abnormal" by the ATSDR. In this letter we seek to communicate the more serious issues raised by this review process.

The basic ATSDR study design included a three-view chest X ray (posterior-anterior, left and right, and obliques) on all participants over 18 years of age, with the X rays to be interpreted by three B-readers. If two of the B-readers identified a pleural or interstitial abnormality, this would be regarded as a positive response by the ATSDR. If only one of the initial two B-readers identified an abnormality, the third B-reader also performed an interpretation.

Several problems are raised by this design. First, all B-readers were aware that the X rays were of Libby residents, opening the door to reader bias. Second, the B-readers knew the sequence in which they reviewed the films. B-readers 1 and 2 knew they were always the first or second reader. B-reader 3, then, not only knew the X rays were from Libby but also had the reports of B-readers 1 and 2 prior to making his interpretation. Although control films or a control group would have been useful in resolving these issues, they were not used.

Peipins et al. (2003) bolstered their findings with the assertion that "if two out of three B-readers indicated...," implying agreement between these readers. The HNA's review (HNA. Unpublished data) reveals that in many instances this was not the case. For example, if one reader found a potential pleural plaque on the right chest wall and the second reader recorded a possible pleural plaque on the left chest wall or diaphragm, this was apparently recorded as a positive finding of an abnormality by two B-readers. These discrepancies were not reported by the ATSDR to study participants or by Peipins et al. (2003).

Table 1.

Perhaps the most troubling issue in the study by Peipins et al. (2003) is the misreading of plaques or thickening when only pleural fat was present. While acknowledging the confounding influence of obesity and pleural fat in determining pleural disease, Peipins et al. (2003) failed to scientifically account for this. The ATSDR (2001) reported that the body mass indexes (BMIs) of 67% of the 7,307 participants were 25, indicating overweight, and 32% of these were obese (BMI of 30). We independently verified the true incidence of overweight and obesity by calculating BMIs of the LMP participants: 89% were overweight and 54% were obese (Table 1).

Peipins et al. (2003) conceded that

a heavier BMI can make it more difficult to distinguish between pleural abnormalities and subpleural or extraplural fat ...

This difficulty was clearly present in their study. As part of the HNA review, X rays and CT scans of study participants were sent for peer review as described above. Although this review is continuing, it is clear that in many cases, participants coded as positive for pleural changes either had no visible asbestos-related changes on their X rays or they had subpleural fat that was misdiagnosed as pleural thickening or plaques.

As a result of the study bias, nonconformity of the B-reader reports, and not accounting for high BMIs and pleural fat, the study by Peipins et al. (2003) markedly overstated the consequences of asbestos exposure in Libby, Montana.

The authors are employees of Health Network America, which is the plan administrator for the Libby Medical Plan. Medical evidence in support of applications is peer-reviewed by independent medical professionals to determine eligibility. Health Network America is paid a fee for its services that is independent of admission or nonadmission of applicants to the plan.

J. Jay Flynn
Stephen Kardos
Lin Yan
Health Network America, Inc.
West Eatontown, New Jersey
E-mail: jflynn@healthnetworkamerica.com

References

ATSDR. 2001. Year 2000 Medical Testing of Individuals Potentially Exposed To Asbestoform Minerals Associated with Vermiculite in Libby, Montana A Report to the Community. Atlanta, GA:Agency for Toxic Substances and Disease Registry. Available: http://www.atsdr.cdc.gov/asbestos/doc_phl_testreport.html [accessed 12 January 2004].

Overstating the Consequences: Peipins et al.'s Response

[citation in pubmed]

Flynn et al. raise concerns about differences in chest radiograph interpretation and bias among B-readers in our study (Peipins et al. 2003). Furthermore, they suggest that an internal review by Health Network America (HNA) found misdiagnosis in many cases, with subpleural fat being miscoded as asbestos-related pleural changes.

In the Libby, Montana, screening program, all films were reviewed by at least two experienced B-readers, with a third B-reader functioning as a "tiebreaker" to settle disagreements. Participants were categorized as "positive" if two B-readers reported any pleural abnormality, and as "indeterminate" if only one B-reader reported an abnormality. For clinical purposes, participants in both the "positive" and the "indeterminate" categories were notified and encouraged to follow up with their personal physician. The three B-readers are respected experts in the field and are distinguished members of academic institutions. Furthermore, the design of the screening program employed is similar to that used in previous studies of asbestos-related radiographic abnormalities (Rogan et al. 1987, 2000).

Although the B-readers in the screening program were aware that the X rays were from the Libby area, they had no other information about occupational or environmental exposure pathways and were blinded to the identities of individuals who were screened. This is important because a clear exposure-response relationship was documented between the presence of pleural abnormalities and the number of exposure pathways reported by participants. Only 6.7% of the group with "no apparent exposure" had pleural abnormalities, compared to 10.8% of the group reporting one to three exposure pathways, 14.4% of the group reporting four to five exposure pathways, and 23.7% of the group reporting six or more pathways. This trend remained significant even after controlling for body mass index in the multivariate analysis.

We recently evaluated preliminary data from high resolution chest computed tomography (CT) scans conducted on 353 Libby medical screening participants with "indeterminate" chest radiographs. Pleural abnormalities were identified in 98 persons (28% of all tested) whose chest radiographs were classified as "indeterminate" (i.e., only one out of three B-readers noted an abnormality). This suggests that the results of the Libby screening program may have actually underestimated the number of abnormal findings.

In summary, we disagree with Flynn et al.'s assertion that these findings can be explained by study bias issues related to chest radiograph B-readings or by misinterpretation of pleural fat. Rather, more recent high-resolution CT scanning results suggest that our estimates of pleural abnormalities in this population may be conservative and may actually underestimate the true prevalence of these abnormalities seen on chest radiographs.

The authors declare they have no competing financial interests.

Lucy A. Peipins
Centers for Disease Control and Prevention
Atlanta, Georgia

Michael Lewin
Sharon Campolucci
Jeffrey A. Lybarger
Vikas Kapil
Dan Middleton
Agency for Toxic Substances and Disease Registry
Atlanta, Georgia
E-mail: mlewin@cdc.gov

Aubrey Miller
Christopher Weis
U.S. Environmental Protection Agency Denver, Colorado

Michael Spence
Montana Department of Health and Human Services
Helena, Montana

Brad Black
Lincoln County Department of Environmental Health Libby, Montana

References

Rogan WJ, Ragan NB, Dinse GE. 2000. X-ray evidence of increased asbestos exposure in the US population from NHANES I and NHANES II, 1973-1978. National Health Examination Survey. Cancer Causes Control 11:441-449.

Rogan WJ, Gladen BC, Ragan NB, Anderson HA. 1987. US prevalence of occupational pleural thickening. A look at chest X-rays from the first National Health and Nutrition Examination Survey. Am J Epidemiol 126:893-900. .

Expert Witnesses Need to Know About the New Risks

[citation in pubmed]

Professionals should not enter into providing witness or consulting services without considering their personal liabilities. I would like to provide some recent changes and liability issues for the consideration of expert witnesses and consultants.

Times change, as does the law. Under recent court decisions, an expert witness whose work fails to meet professional standards may find himself/herself being hauled back into court as a defendant in a malpractice claim. If the expert knows of possible personal weaknesses, hiring additional experts to cover these weak areas is a feasible solution.

Historically, witnesses could not be sued for defamation on the basis of their testimony in court. The law granted this immunity to encourage candid testimony, or as one court put it, "to ensure that the path to truth is left as free and unobstructed as possible."

In 1999, however, the Supreme Court of Pennsylvania carved a large exception out of the immunity doctrine for expert witnesses. In the case of LLMD of Michigan, Inc. v. Jackson-Cross Co. (1999), the court held that a client could sue his expert witness for negligence if the expert fails to exercise the care and skill common to his profession in forming his opinions on the client's case.

In 2002, the Supreme Court of Appeals in West Virginia took the issue further in Davis v. Wallace (2002) when it suggested that an expert witness could be sued for negligence not only by his own client but also by the opposing party against whom the expert testifies. Experts need to keep up with their fields of interest in the era of rapid new information. This can be done by reading, attending meetings, and talking with other experts. Publishing articles in peer-reviewed journals can help document and prove expertise to the judge. Keeping good records, which display knowledge through good writing skills, is critical; many decisions are based on written documents. Continuing education courses are available to improve the expert's practice and avoid liability problems. Also, all communications should be carefully edited before they are sent.

Scientists who serve as expert witnesses in federal lawsuits should be prepared to justify their theories and methods used in each case in a Daubert hearing. This is a layman's shorthand reference to the 1993 U.S. Supreme Court decision in Daubert v. Merrell Dow Pharmaceuticals, Inc. (1993). The Daubert case radically altered the rules for expert witnesses in federal court. For most of the twentieth century, federal courts were supposed to permit expert witnesses to offer scientific evidence at trial only if the scientific principles involved were generally accepted within their field. In the Daubert case, the Supreme Court held that the general acceptance test was too restrictive and was not compatible with modern rules of evidence.

On one hand, the Daubert case (Daubert v. Merrell Dow Pharmaceuticals, Inc. 1993) opened the courthouse door to novel scientific evidence. On the other hand, this case vested trial judges with a gatekeeping role to weed out "irrational pseudoscientific assertions," or to separate cutting-edge principles from "junk science" (Daubert v. Merrell Dow Pharmaceuticals, Inc. 1993). As part of their gatekeeping function, judges are required to make a preliminary assessment to determine whether the expert's theory or technique has been and can be tested, whether it has been subject to peer review and publication, and whether the theory or technique has a known or potential rate of error.

Courts have not established hard and fast rules on the manner in which they will decide Daubert objections. The process could be as simple as the lawyers submitting documents to the judge, or it could involve a form of mini-trial in which the court hears testimony from the expert and other experts who attempt to validate or discredit the expert's theories and practices at issue.

What should scientists do, knowing that their work as expert witnesses may be rejected by a judge and knowing that their own clients may sue them if the case turns out badly? Some experts may decide that it is not worth the liability exposure and may decline invitations to serve as an expert witness. But doing good, reliable, legally defensible work is the best way a scientist can survive a Daubert challenge.

The author declares he has no competing financial interests.

Henry G. Nowicki
PACS Testing, Consulting, and Training
Pittsburgh, Pennsylvania
E-mail: hnpacs@aol.com

References

Daubert v. Merrell Dow Pharmaceuticals, Inc. 1993. Case No. 92-102. U.S. Supreme Court, Washington, DC.

Davis v. Wallace. 2002. Case No. 29966. Civil Action 99-C-146. West Virginia Supreme Court of Appeals, Charleston, WV.

LLMD of Michigan, Inc. v. Jackson-Cross Co. 1999. Case No. 559, Pa.297, 740 A.2d 186. Pennsylvania Supreme Court of Appeals, Harrisburg, PA. .

Validation of (Q)SARs Models

[citation in pubmed]

From a practitioner's point of view (but not having been part of the workshop), I feel compelled to comment on "Summary of a Workshop on Regulatory Acceptance of (Q)SARs for Human Health and Environmental Endpoints" by Jaworska et al. (2003).

There are a variety of quantitative structure-activity relationships [(Q)SARs] models available for a variety of purposes, and, as stated by Jaworska et al. (2003), predictive power is a critical issue in evaluating any model. Regrettably, the accompanying articles by Eriksson et al. (2003) and Cronin et al. (2003a, 2003b) fail to mention any of the recent publications on the application of probabilistic neural networks (PNNs) for the modeling of toxicity endpoints. Highly effective PNN models have been demonstrated for the fathead minnow (Kaiser and Niculescu 1999), the waterflea Daphnia magna (Kaiser and Niculescu 2001a), the ciliate Tetrahymena pyriformis (Niculescu et al. 2000), the Microtox bacterium Vibrio fischeri (Kaiser and Niculescu. In press), and estrogen receptor binding affinity (Kaiser and Niculescu 2001b). Indeed, Moore et al. (2003) have shown that fathead minnow PNN has superior performance in essentially all aspects when compared to the other methods. Other types of neural networks have similarly been shown to be robust and to provide optimal predictions (e.g., Burden and Winkler 1999). Furthermore, commercially available programs using PNN methodology have recently become available for the estimation of several toxicologic endpoints, such as fathead minnow 96-hr median lethal concentrations (LC₅₀) (TerraBase, Inc. 2002), rat and mouse intravenous LD₅₀ (TerraBase, Inc. 2003a), and estrogen receptor binding affinity (TerraBase, Inc. 2003b).

Although representativity or domain of a model are good concepts in theory, they are difficult to define or use in practice. Moreover, the statistical descriptors of a model's performance--such as goodness of fit, specificity, sensitivity, transparency, and similarity--are often misleading because the applied data set(s) for many (Q)SARs are narrow, skewed, or otherwise nonrepresentative of the chemical world existing in reality. In most cases, a model user cannot ascertain whether a particular model may or may not be used for a particular compound and end point to be estimated. Without tests of comparative performance, this conundrum exists for users of most models. Even for quite similar compounds, model outputs can vary by several orders of magnitude between both models and measured values. For example, predictions of octanol/water partition coefficients (a physical property) for a small set of quite similar compounds by commonly used models show a large divergence of values (Vrakas et al. 2003). Therefore, the (only) proof of model accuracy is in the testing of each model's performance against a broad spectrum of measured data, which are not part of the training set of each model. In practice, this means that performance of a model should be the driving force for its acceptability in the regulatory world, not its statistics.

Regular scrutiny of performance has been commonplace in other areas. For example, the performance of Canadian environmental analytical laboratories is regularly checked with round robin testing. The predictive power of carcinogenicity and mutagenicity models has been evaluated in several rounds of testing, with the biological testing subsequent to the models' predictions. There is a great need for such comparative testing of the usefulness of various existing (Q)SAR models. The valiant performance testing of several toxicity-prediction (Q)SARs models by Moore et al. (2003) shows some surprising results and further gives credence to this thought. Indeed, Jaworska et al. (2003) also stress the need for an independent organization to validate data and models irrespective of any model's claims.

The author is the director of research and a principal of TerraBase, Inc.

Klaus L.E. Kaiser
TerraBase, Inc.
Hamilton, Ontario, Canada
E-mail: mail@terrabase-inc.com

References

Burden FR, Winkler DA. 1999. Robust QSAR models using Bayesian regularized neural networks. J Med Chem 42:3183-3187.

Cronin MT, Jaworska JS, Walker JD, Comber MH, Watts CD, Worth AP. 2003a. Use of QSARs in international decision-making frameworks to predict health effects of chemical substances. Environ Health Perspect 111:1391-1401.

Cronin MTD, Walker JD, Jaworska JS, Comber MH, Watts CD, Worth AP. 2003b. Use of QSARs in international decision-making frameworks to predict ecologic effects and environmental fate of chemical substances. Environ Health Perspect 111:1376-1390.

Eriksson L, Jaworska J, Worth AP, Cronin MTD, McDowell RM, Gramatica P. 2003. Methods for reliability and uncertainty assessment and for applicability evaluations of classification- and regression-based QSARs. Environ Health Perspect 111:1361-1375.

Jaworska JS, Comber M, Auer C, Van Leeuwen CJ. 2003. Summary of a workshop on regulatory acceptance of (Q)SARs for human health and environmental endpoints. Environ Health Perspect 111:1358-1360.

Kaiser KLE, Niculescu SP. 1999. Using probabilistic neural networks to model the toxicity of chemicals to the fathead minnow (Pimephales promelas): a study based on 865 compounds. Chemosphere 38:3237-3245.

------. 2001a. Modeling the acute toxicity of chemicals to Daphnia magna: a probabilistic neural network approach. Environ Toxicol Chem 20:420-431.

------. 2001b. On the PNN modelling of estrogen receptor binding data for carboxylic acid esters and organochlorine compounds. Water Qual Res J Canada 36:619-630.

------. In press. Neural network modeling of Vibrio fischeri toxicity data with structural physico-chemical parameters and molecular indicator variables. In: QSARs for Predicting Ecological Effects of Chemicals (Walker JD, ed). Pensacola, FL:SETAC Press.

Moore DRJ, Breton RL, MacDonald DB. 2003. A comparison of model performance for six quantitative structure-activity relationship packages that predict acute toxicity to fish. Environ Technol Chem 22:1799-1809.

Niculescu SP, Kaiser KLE, Schultz TW. 2000. Modeling the toxicity of chemicals to Tetrahymena pyriformis using molecular fragment descriptors and probabilistic neural networks. Arch Environ Toxicol Chem 39:289-298.

TerraBase, Inc. 2002. TerraQSAR - FHM. Fish Toxicity Computation Program. Available: http://www.terrabase-inc.com//tq-4fhm.htm [accessed 7 January 2004].

------. 2003a. TerraQSAR - RMIV. Rat / Mouse intravenous LD50 Computation Program. Available: http://www.terrabase-inc.com//tq-4rmiv.htm [accessed 7 January 2004].

------. 2003b. TerraQSAR - E2-RBA. Estrogen Receptor Binding Affinity Computation Program. Available: http://www.terrabase-inc.com//tq-4e2rba.htm [accessed 7 January 2004.

Vrakas D, Tsantili-Kakoulidou A, Hadjipavlou-Litina D. 2003. Exploring the consistency of logP estimation for substituted coumarins. QSAR Combin Sci 22:622-629. .

Validation of (Q)SARs Models: Jaworska et al.'s Response

[citation in pubmed]

We warmly welcome and agree with the comments from Kaiser. In writing the reports that comprised the Mini-Monograph on "Regulatory Acceptance of (Q)SARS" published in EHP and also during the workshop sponsored by the International Council of Chemical Associations and the European Chemical Industry Council and held in Setubal, Portugal, in March 2002 (Jaworska et al. 2003), it was never our stated or underlying intention to make a thorough review of (Q)SARs for toxicologic endpoints and fate. This has been more than adequately achieved in recent editions of Environmental Toxicology and Chemistry (2003) and the Journal of Molecular Structure-THEOCHEM (THEOCHEM 2003). This is firmly stated in the papers themselves (Cronin et al. 2003a, 2003b; Eriksson et al. 2003). Instead, the purpose of these articles was to review the use of (Q)SARs to predict toxicity and fate by regulatory bodies. However, it should be understood, that such reviews will be needed, and are ongoing, in order for (Q)SARs to find a more extensive and transparent use in regulatory schemes and thus achieve their stated goal of reducing animal testing. In general, only predictive techniques used by regulatory bodies are mentioned, although some other work to illustrate various other techniques was cited to provide examples. We did state that the probabilistic neural network (PNN) provided the best predictions for fish toxicity on the basis of an external test set (Cronin et al. 2003a), but like the vast majority of examples of (Q)SARs in the literature, the other endpoints were not included. With regard to the article by Eriksson et al. (2003), the scope was clearly defined in the title; that is, it provided an assessment of regression- and projection-based methods. There are also other approaches to (Q)SAR, such as support vectors machines; however, these were not included because they are not used broadly in all (Q)SAR disciplines. Further, these techniques are not in frequent use by regulatory bodies.

We agree with Kaiser that the performance of a model should be a driving force and is more important than statistics. Appropriate measures of predictivity or performance are enshrined in the cornerstones of current attempts at validation. As part of the validation process, the domain of applicability should also be defined. This will allow a user to know whether a prediction is likely to be valid. It should also be noted that Eriksson et al. (2003) explain that representativity and homogeneity are basic requirements for (Q)SAR modeling, regardless of the statistical method used to develop the model. Eriksson et al. (2003) also noted that these concepts were easy to check and accomplish. Provided the basic conditions of representativity and homogeneity are not violated, statistical descriptors of a model's performance are not misleading. Indeed, failure to comply with representativity and homogeneity will result in the statistical assessment of model performance becoming inappropriate.

The reality is that simple and transparent models are (on the whole) favored by regulatory bodies over more opaque approaches, even if it means losing some statistical fit. We understand that problems could arise in some PNN approaches that have used large numbers of descriptors, some with no relevance to the mechanism of action (Cronin and Schultz 2001). A further issue with PNNs is the fact that they do not provide the same numerical solution to a problem when repeated.

The goal of the Setubal workshop was to encourage further development of the (Q)SAR sciences toward practical use and application. It is important, that as these models develop into more efficient tools for society to understand the fate and effect of chemicals, without extensive use of animals and other resources, they be seen to be valid and suitable for the purpose. To achieve this goal, there are a number of ongoing attempts to validate (Q)SARs (e.g., at the European Union and the Organisation for Economic Co-operation and Development level). These will attempt to weigh all the evidence and produce scientifically sound and usable validation techniques. Interested parties are encouraged to join in these efforts. More information is available on the European Commission Joint Research Centre website (JRC 2003).

The authors declare they have no competing financial interests.

Joanna S. Jaworska
Procter & Gamble Eurocor
Strombeek-Bever, Belgium
E-mail: jaworska.j@pg.com

Mark Cronin
Liverpool John Moores University
Liverpool, United Kingdom

Lennart Eriksson
Umetrics
Umea�, Sweden

Mike Comber
ExxonMobil
Brussels, Belgium

References

ETC. 2003. Environ Toxicol Chem 22(8):1651-1936.

Cronin M, Schultz TW. 2001. Development of quantitative structure-activity relationships for the toxicity of aromatic compounds to Tetrahymena pyriformis: comparative assessment of the methodologies. Chem Res Toxicol 14:1284-1295.

Cronin MTD, Walker JD, Jaworska JS, Comber MHI, Watts CD, Worth AP. 2003a. Use of QSARs in international decision-making frameworks to predict ecological effects and environmental fate of chemical substances. Environ Health Perspect 111:1376-1390.

Cronin MTD, Jaworska JS, Walker JD, Comber MHI, Watts CD, Worth AP. 2003b. Use of QSARs in international decision-making frameworks to predict human health effects. Environ Health Perspect 111:1391-1401.

Jaworska JS, Comber M, Auer C, Van Leeuwen CJ. 2003. Summary of the workshop on regulatory acceptance of (Q)SARs. Environ Health Perspect 111:1358-1360.

JRC. 2003. QSARS. Ispra, Italy:European Commission, Joint Research Centre. Available: http://ecb.jrc.it/QSAR/ [accessed 2 Dec 2003].

Erriksson L, Jaworska J, Worth AP, Cronin MTD, McDowell RM, Gramatica P. 2002. Methods for reliability and uncertainty assessment and for applicability evaluations of classification and regression-based QSARs. Environ Health Perspect 111:1361-1375.

THEOCHEM. 2003. J Mol Structure 622(1-2):1-184..

Correction

In the January Forum article "Tidal Turbines: Wave of the Future?" [EHP 112:A26 (2004)], Bj�rn Bekken was incorrectly quoted as saying, "Once we pass peak production of fossil fuels, the difference between supply and demand decreases day by day and energy prices will go up." What Bekken in fact said was that the difference between supply and demand increases day by day. EHP regrets the error.

[Table of Contents]

Last Updated: January 21, 2004