Dear
Editor:
We
are pleased to see continued interest in the International Stormwater BMP
Database. The January/February issue of Stormwater
included an article titled “Use of the International Stormwater BMP Database.”
The article presented opinions on how data from the BMP Database should be
analyzed and pointed out a number of issues that we are currently working on. We
would like to take this opportunity to provide some broad perspective on the
project, as well as provide a few selected comments on the
article.
The
primary mission of the BMP Database is to improve the monitoring and reporting
information on BMPs via recommended standardized monitoring and reporting, and
data storage protocols. The intention was and still is to create a versatile
database that is flexible enough to support broad analyses, as well as more
microscopic (regional, BMP subsets, etc.) analyses now and in the future.
Secondarily, the team also provides periodic broad national-level summaries of
BMP performance based on the data available to the project. The intent of these
summaries is to conduct analyses in the same manner across all information in
the database to give perspective on the overall state of the practice and extent
of data available.
It
is important to note that the underlying data are available for anyone to
download, analyze, and interpret as one sees fit in terms of BMP classifications
and design attributes, analyzing only facilities that fit one’s description of
“common design” as referenced in the article, or any other way one wants to
analyze the data. Indeed, many researchers have conducted thorough analyses of
the information for their own purposes, screening and categorizing data as they
deemed appropriate for their analyses. We have always strongly encouraged
researchers and others to conduct their own analyses and have done so ourselves
in our own practices when working with others to develop useful project- or
region-specific performance information. However, we do feel it is useful to
provide a “broad and generalized” analysis on a national basis to give the
research community an overview of where we are at a national level in our
efforts to collect and understand the available data. When taking such a broad
perspective, one can make any number of arguments about why BMPs should or
should not be grouped, studies to leave in or out, regions or climate areas to
include or not, etc.
When
analyzing BMP performance, we strongly recommend that one consider not only
performance data, but also the unit processes that are included within BMPs.
Best professional practice dictates that most projects or efforts that seek to
explore the fundamental relationships between design and performance of BMP
systems should utilize a combination of statistically rigorous empirical data
analysis (from information sources such as the database), modeling (particularly
hydrologic and hydraulic), unit process engineering, and expert opinion rooted
in on-the-ground experience. In particular, the database team this year is
currently working to better facilitate use of the data by researchers who are
interested in conducting analyses of the relationships between performance and
unit processes/design characteristics.
There
are a few specific points in the article that we would like to
address:
1.
We strongly disagree with the article’s recommendation of not including studies
“in which the performance is clearly substandard.” The purpose of the BMP
Database is to document the full range of performance in the hope that a better
understanding of the relationship between BMP design and performance can be
gleaned. Fundamentally, the database is an empirical tool. The team is very
concerned with introducing any bias, either positive or negative. Taking steps
to exclude data based on performance results could undermine the intent of the
tool. Of greater concern to the database team is the likelihood of biasing the
data in any way that could potentially limit its utility to the community in the
future.
Of
course, when doing a more detailed analysis for a local or regional BMP
selection or design standards, if a BMP clearly was not designed to local or
regional requirements/conditions, then it would be appropriate to exclude it
from analysis that is trying to predict performance based on local conditions.
These types of specific analyses are not appropriate to present in the
“generalized analyses” conducted by the team to date. Many researchers have
effectively used the database to conduct project and regional specific analyses.
An exception to these general statements is that when evidence becomes available
indicating that a previously unidentified issue exists with data collection or
reporting, the project team will consider removing the study, provided that such
a request is properly documented. This, however, is a different situation than
removing a study or BMP that doesn’t “perform well.”
2.
With regard to percent removal, we stand by our recent editorial published in
Stormwater
(“15 Reasons You Should Think Twice Before Using Percent Removal to Assess BMP
Performance,” January/February 2008). We would point out that in our generalized
analyses, we clearly show whether a BMP demonstrates a statistically significant
difference between influent and effluent, which eliminates the issue of relying
solely on effluent quality, when in fact, the BMP did not have an observed
effect. We again do not feel it is appropriate to exclude a BMP from the
database just because influent concentrations are at some defined “irreducible”
concentration for some pollutants. It is just as important to understand where
BMPs will likely not have an effect as where they will.
3.
We believe that there are many arguments and appropriate reasons for grouping
and not grouping BMPs when conducting specific project or regional analyses that
may change depending on what one is interested in learning. For example, the
article makes a case for grouping some BMPs together in general analyses (wet
ponds and wetlands). Many others would likely argue that the unit processes
differences amongst these BMP types would be substantial enough that one should
not group them. For example, wetlands by their nature tend to be shallower
systems with increased substrate and vegetation contact, microbial activities,
and shorter depths for settling, etc. that would be very different than a deep
wet pond where rooted vegetation cannot be established. Additionally, for most
ponds, the ratio of substrate area-to-pond volume is much less, and the time for
a particle to settle to the bottom is greater. This would be expected to
substantially affect the performance for such pollutants as sediments,
nutrients, and oils and greases, amongst others. So, if one were interested in
pollutants that could be affected by these significant unit process (and design)
differences between wet ponds and wetlands, the suggested grouping in the
article would be inappropriate.
Our
point is that one should not consider generalized groupings (or ungroupings) of
BMPs for summary or detailed analyses without careful consideration of the
pollutant types and forms, the unit processes (designs) that would be expected
to affect those differences, climatic differences, etc. With this said, the
project team recognizes that there are some cases where researchers may
inadvertently enter studies into a BMP category that may not best represent
primary unit processes and design characteristics. In such cases, it may be
appropriate to shift the study itself to a more appropriate
category.
4.
We agree with a number of suggestions for how one could parse the BMP design and
performance information better. Unfortunately, much of that parsing would
currently be problematic because it would not leave enough data in most
categories to enable analysis, as the January article points out. Our hope is
that as the BMP Database grows, we (and others) can further parse out data based
on unit processes included in BMPs, climate and regional conditions, etc. with
the ultimate goal of improving BMP selection and design.
The
BMP Database team began working early last summer on a number of the issues
raised in the article, including categorization of BMP types, reviewing and
updating the reporting and performance information required/requested, protocols
for low-impact development/distributed controls, and updating the BMP monitoring
and reporting guidance. Prior to the publication of the January article, a
one-day intensive peer review with national experts was held in Denver to help
guide future revisions and enhancements to the database.
The
project team is excited about forthcoming changes and appreciates the
investments of the Water Environment Research Foundation, American Society of
Civil Engineers–Environmental and Water Research Institute, the US Environmental
Protection Agency, and the Federal Highway Administration during 2009, which
will enable implementation of these enhancements. We also continually seek entry
of new data sets into the database so that robust conclusions regarding BMP
performance can be drawn as the database continues to grow. Please see our Web
site for more information on submitting studies to the database (www.bmpdatabase.org).
We look forward to reporting our progress to your readers via articles and
posting to the Web site and appreciate the ongoing support and dialogue on this
project.
Sincerely,
The
International Stormwater BMP Database Team
Eric
Strecker, P.E., PI
Geosyntec
Consultants
Jonathan
Jones, P.E., PI
Wright
Water Engineers Inc.
Jane
Clary
Wright
Water Engineers Inc.
Marcus
Quigley, P.E.
Geosyntec
Consultants