|
FROM: Morgan Hoff & Timothy Brand
members of West Valley Citizen Air Watch 10377 Vista Knoll Blvd.,
Cupertino, CA 95014 (408)733-5570
Critique of Dames & Moore Report "Analysis of Emissions
Test Results and Residual By-Products from Facilities Using Tires as a Fuel
Supplement" IWM-C5064
DATE: January 12, 1998
D&M General Critique
Simply stated, D&M is junk science. The data is sloppy and scientifically
inadmissible. The method used to analyze the data is bad science and the authors donit
admit or perhaps never realized that their conclusions were clearly not statistically
significant. Due to its many severe faults in data and methodology the Dames & Moore
report "Analysis of Emissions Test Results O" should be set aside as simply not
credible. Overview Simply stated, the purpose of the Dames & Moore report
"Analysis of Emissions Test Results and Residual By-Products from Facilities Using
Tires as a Fuel Supplement" IWM-C5064 (D&M) is to answer two questions:
1. Can one prove with statistical significance that the emissions from industrial fires
do not increase when tires are added as a fuel supplement? Typical industrial fires would
be associated with cement kilns, industrial boilers, lime kilns.
2. Can one prove with statistical significance that adding tires as a fuel supplement to
industrial fires is safe from the viewpoint of health effects?
A good way to answer the two questions above would be to collect data from industrial
fires at several facilities, confirm that the only variable was the use of tires as a fuel
supplement and individually determine for each facility what impact the addition of tires
had on emissions and health risks. This is not what D&M did.
Instead D&M collected the data and did the converse. On a chemical by chemical basis,
across all the reporting facilities, D&M created two groups. They were
"baseline" i.e., without tires and "with tires as a fuel supplement".
As an example, one data set might be "CO emissions at all facilities, without
tires" and the complement would be "CO emissions at all facilities, but with
tires as a supplement". D&M compared the range and mean of the "with"
and "without" case for each chemical, reached statistically insignificant
conclusions but failed to report the lack of significance.
A partial critique follows. There are many, many other problems in D&M that are not
enumerated below.
The Data is Scientifically Inadmissible. The data comes from diverse types of facilities
burning diverse fuels but that is an observation, not an indictment. D&M table 4-1 is
the only place in the report we can find information on the facilities. The table lists 27
unique facilities but 29 combinations of facilities and fuels so we will refer to it below
as 29 facilities. Here are just a few facts that should cause one to seriously question
the data from these facilities.
S Between 41% and 62% (12 to 18 of 29) of the facilities are not in California and
consequently not subject to the Californiais tough pollution regulations and monitoring.
The inclusion of "dirty" facilities hides the facts and trends by expanding the
emission ranges. The 12 to 18 facilities are in New Jersey, Illinois, Michigan, Kansas,
Idaho, Texas, Florida, Washington, Wisconsin and New York. (footnote#1)
S In 7% (2 of 29) of the facilities the primary fuel is "unknown".
S In a different 7% (2 of 29) of the facilities the primary fuel is listed as some form of
tires ("Whole TDF" or "Chip TDF"). This is supposed to be a study
comparing emissions and health risks with and without tires as a fuel supplement. What
sense does it make to compare "the burning of tires" to "the burning of
tires WITH TIRES". There will be no difference. Is this a typo in the table or an
example of ignorance?
S 65% ( 19 of 29) of the facilities do not list enough information to calculate the
percent of tires in the fuel mix.. This information is necessary to correlate the input (%
of tires in fuel mix) to the output (change in emissions or health risk). D&M attempts
a correlation without this necessary information.
S On the flip side, only 14% (4) of the facilities are in California
AND
appear to include enough information to calculate the percent of tires in the fuel mix.
S Consequently the other 86% of the facilities should be tossed out of the study or
carefully scrutinized for appropriateness prior to inclusion. But currently that 86% is
included in the study and represents a large source of misinformation.
S Even if one sees no reason to reject the non-California inputs, the 65% of facilities
for which one canit calculate the percent of tires in the fuel mix should be excluded. So
at a minimum, the data from two out of three facilities should be discarded.
S There is no reason to believe that the data is representative or unbiased. It is a
voluntary submission from facilities with a self interest in lowering fuel costs by making
the way smooth for tire burning in California. There is no guarantee that the most
damaging information was submitted.
S The facilities are not reasonably identified. D&M table 4-1 "Facilities
Responding to Information Request" doesnit include the city, county or state of the
facilities.
S There is no link in the report between a given facility and its data in the bar charts.
For example, there is no identification of facility "C01" or any other facility
on D&M figure 4-4 "Difference of TDF versus Baseline for Carbon Monoxide".
There is no facility identification on any figure showing emissions. It serves to make the
results untraceable and unverifiable. The report is a dead end. One can not, from this
report, answer the simple question "Did emissions increase on a facility by facility
basis when TDF was used?" Is this muddying of the waters intentional or simply bad
science?
The Method Used to Analyze the Data Is Bad Science
S D&M says that it normalized the emission data among the sources but there is no
evidence of normalizing. D&M confuses the picture by comparing the same emission
chemical across all facilities without normalizing. Here is what D&M did to build a
data set for CO emissions across the 29 facilities:
It put all of the "per plant CO emission data without tires as a supplement" in
one set and all of the "per plant CO emission data with tires as a supplement"
in another set.
The facilities in D&M table 4-1 range in capacity by a factor of 30 to 1 (footnote #2)
but all the data was combined un-normalized. This fact is borne out by the units shown for
what is supposed to be normalized emissions. In all cases the units are lb/hour, that is,
all of the emissions from each plant. They are not scaled to the size of the facility as
would be indicated by emission units of (lb/hour) per Mbtu/hr. D&M takes the smoke
stack emission from plant XYZ and a plant that is 30 times as large consuming 30 times the
fuel and emitting more and just puts them in the same set. Then it calculates the mean and
standard deviation and is surprised that the range of emissions across the facilities is
1,000:1 or 10,000:1.
S Dirty and/or large facilities, those with high per plant emission rates, swamp out the
others. The mean and extremely large ranges are driven by the big emission numbers. This
hides trends.
S The data is terribly sloppy and this is why the conclusions in D&M are not
statistically significant or supportable and should be rejected.
S D&M misapplies statistical analysis. Here is an example from page 25.
"Mean NOx emission rates showed a decrease when using TDF with values of 243
lb/hr and 192 lb/hr for baseline and TDF test results, respectivelyOOO.."
Consolidating the numbers in that quote and including range data from D&M Table 4-6 we
get this table.
Without TDF With TDF
Compound Units Max Min Mean Standard Deviation Max Min
Mean Standard Deviation
NOx lb/hr 972 19.6 243 215 606 8.3 192 152
The above standard deviations are a huge fraction of their respective mean.
The standard score for the difference of these means is 1.04 (footnote#3), corresponding
to a 70% confidence level. That means there is a 30% chance that the two means, NOx
emissions with and without TDF, are from the same population. Translated, there is a 1 in
3 chance that there was no reduction in NOx and so the conclusion is statistically not
significant. A rule of thumb is that a 99% confidence level is considered highly
significant, a 95% confidence level is considered probably significant and a level less
than 95% is not significant. The 70% confidence level mentioned above is thus clearly not
significant. D&M reached a conclusion "Mean NOx emission rates showed a decrease
when using TDFO" but after doing the calculations it failed to report that the
difference was clearly not statistically significant. Sloppy data led to large ranges and
large standard deviations while sloppy analysis led to conclusions without statistical
significance. D&M is full of such lapses from sound scientific methodology.
S It is very important to properly understand this last point. It does not say the data
shows no statistically significant difference between NOx emissions with and without tires
as a fuel supplement, that there is no difference in the NOx emissions for the two cases
and therefore it is safe to burn tires. It instead says that the data is so sloppy that
using it we can not determine with any reasonable statistical significance whether the NOx
emissions went up or whether they went down.
If these statistical terms, the mean and standard deviation, are confusing, think of the
NOx data in terms of a collection of physical balls of differing weights. The "no
tire" balls group has a mean weight of 243 lbs and a standard deviation of 215 lbs.
That is equivalent to saying that we have 95% confidence the "no tire" balls
weigh the mean + 2*standard deviations = 243 +430 lb or between -187 lb and 673 lb.
Similarly the "tire" balls weigh 192 + 304 lb or between -112 and 496 lb.
Immediately we see there is a huge overlap in weights between the two groups and it is
very difficult to tell the two groups apart. Secondly we recognize that what we know about
the ballsi weights is ridiculously imprecise and that is the reason for our confusion.
This is exactly what the statistical values, the mean and standard deviation, tell us
about the NOx and other emissions.
The data is so imprecise that we can not responsibly use it to distinguish the "no
tires" and "tires" cases. We can not responsibly use it to judge whether
the emissions went up, went down or didnit change when we added tires to the fire. The
burden of proof is to show that adding tires to industrial fires does not increase the
emissions or health risk. The data is simply not good enough to prove that adding tires to
an industrial fire, like a cement kiln, is safe.
Response to the Four Results of D&M, p ES-3. D&M Result #1 "The analysis is
based on air emission measurements at 28 facilities out of which cement kilns located in
California comprised the largest data set." S Refer to critique of data above. A
larger data set is facilities not in California.
D&M Result #2 "The change in health risk due to toxic air emissions from
facilities using tires as a fuel supplement is not significantly different from the same
facilities using typical fuels, such as coal." S This statement is the key
justification for the California Integrated Waste Management Boardis (CIWMB) vigorous
pursuit of tire burning, but as explained above it is not supported by the data.
S The D&M conclusions were developed by wrong headed analysis and lack statistical
significance so they are meaningless. S A more accurate and succinct conclusion would have
stated that the study was inconclusive because the TDF impact was less than the large
uncertainties in the data sets collected.
D&M Result #3 "Greater variability in health risk occurs between different
facilities, combustion practices, and air emission control systems [than the variability
for a each pollutant across all facilities with and without tires as a fuel
supplement]."
S This statement only goes to highlight the extremely sloppy data and the
inappropriateness of using it in the first place.
D&M Result #4 "This comparison of health does not include the incremental health
risk if the tires are landfilled or stockpiled. This incremental health risk would be
associated with the use of typical fuels for landfill operation if the tires were
landfilled or the risk of catastrophic fires if stockpiled."
S This observation has no numbers or probabilities to give it weight. The probability of a
catastrophic fire is not addressed. The unspecified "incremental health risks"
are alluded to without supporting data and there is no concept of the size of these
hypothetical problems. Furthermore, the report doesnit acknowledge techniques such a
baling which radically reduce water pooling, pest populations and the likelihood of a
stockpile fire.
Conclusion Regarding D&M
S Due to its many severe faults the Dames & Moore report "Analysis of Emissions
Test Results O" should be set aside as not credible.
---------------------------
1 The range in this percentage exists because D&M Table 4-1 does not identify the
state where the facility is located. We investigated to identify the locale of each
facility and were able to identify the state for all but six facilities.
2 This is based on the D&M Table 4-1 Heat Input field and the Production Rate field.
3 standard score of difference of means = difference in means / standard deviation of the
difference of the means {this next footnote is damaged below here because it includes
equiations that don't readily translate into text}
= where mean(X1) is the mean of sample set 1 having N1 elements and is the standard
deviation of the set.
= (243 - 192) /
= 1.04 which corresponds to a 70% confidence level for a two-tailed test.
|