Performance Evaluation Data Structures
Various data structures containing tabular statistics and information about evaluation conditions can also be constructed. These data structures can be constructed by running the following line of code:
evaluation.calculate_metrics()
This will construct the statistics dataframes evaluation.stats_df
and evaluation.avg_stats_df
and
populate the deployment dictionary evaluation.deploy_dict
with details about the evaluation.
More detail about each of these data structures is provided below.
Statistics DataFrames
Sensor vs. FRM/FEM Statistics: evaluation.stats_df
A pandas DataFrame containing statistics for the sensor vs. FRM/FEM linearity (R2),
bias (slope and intercept), RMSE, N (number of sensor-FRM/FEM data point pairs), as well
as the minimum, maximum, and the mean sensor concentration. Data are presented for all
averaging intervals specified by evaluation.eval_param_averaging
.
evaluation.stats_df
is saved as a comma-separated value file at
/data/eval_stats/[sensor_name]/[sensor_name]_[parameter]_vs_[reference_name]_stats_df_YYMMDD.csv
where [sensor_name]
is the name of the sensor, [parameter]
is the SDFS parameter name,
[reference_name]
is the name of the reference monitor, and YYMMDD
is the date the figure was compiled.
Below is an example of evaluation.stats_df
for the Toco Toucan Quickstart Guide example:
Sensor Name |
Sensor_Number |
Sensor_Serial |
Averaging Interval |
Param |
Reference |
R$^2$ |
Slope |
Intercept |
Sensor RMSE |
N |
Sensor_Min |
Sensor_Max |
Sensor_Mean |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Toco_Toucan |
1 |
RT01 |
1-hour |
PM25 |
T-API T640X at 16.67 LPM |
0.5216234756550462 |
0.8191825319244019 |
-1.676243556875662 |
3.5281941408285844 |
710.0 |
1.1283333333333334 |
13.763333333333332 |
4.710611124421121 |
Toco_Toucan |
2 |
RT02 |
1-hour |
PM25 |
T-API T640X at 16.67 LPM |
0.5540957508355946 |
0.8254680178697646 |
-1.7550479793461324 |
3.508767319754394 |
712.0 |
1.145 |
12.465 |
4.678385416666664 |
Toco_Toucan |
3 |
RT03 |
1-hour |
PM25 |
T-API T640X at 16.67 LPM |
0.5638579786715556 |
0.7508014694014582 |
-1.692978020396487 |
3.931586688145356 |
712.0 |
0.8816666666666666 |
11.37 |
4.158528206928842 |
Metric Average: |
0.5465257350540654 |
0.7984840063985416 |
-1.7080898522060937 |
3.6561827162427782 |
711.3333333333334 |
1.0516666666666667 |
12.532777777777776 |
4.515841582672209 |
|||||
Toco_Toucan |
1 |
RT01 |
24-hour |
PM25 |
T-API T640X at 16.67 LPM |
0.6228184308540052 |
0.862652058229343 |
-2.056301654583035 |
3.3090455977487983 |
29.0 |
2.0139680333564973 |
7.857430555555556 |
4.65241859578438 |
Toco_Toucan |
2 |
RT02 |
24-hour |
PM25 |
T-API T640X at 16.67 LPM |
0.6579780523926115 |
0.9185089802725335 |
-2.4919265748675032 |
3.3008919655600906 |
29.0 |
1.9179166666666667 |
8.139861111111111 |
4.6511848769809605 |
Toco_Toucan |
3 |
RT03 |
24-hour |
PM25 |
T-API T640X at 16.67 LPM |
0.5775702631554632 |
0.8405996738864467 |
-2.389694167854771 |
3.8135986597907343 |
29.0 |
1.1636111111111112 |
7.641319444444444 |
4.14752787996533 |
Metric Average: |
0.6194555821340266 |
0.8739202374627744 |
-2.3126407991017697 |
3.4745120743665407 |
29.0 |
1.698498603711425 |
7.879537037037037 |
4.483710450910223 |
Sensor vs. Intersensor Average Statistics: evaluation.avg_stats_df
A pandas DataFrame containing statistics relating the sensor vs. intersensor average
linearity (R2), bias (slope and intercept), RMSE, N (number of concurrent
sensor measurements during which all sensors in the testing group reported values), as well as the
minimum, maximum, and the mean sensor concentration. Data are presented for all
averaging intervals specified by evaluation.eval_param_averaging
.
evaluation.avg_stats_df
is saved as a comma-separated value file at
/data/eval_stats/[sensor_name]/[sensor_name]_[parameter]_vs_[reference_name]_avg_stats_df_YYMMDD.csv
where [sensor_name]
is the name of the sensor, [parameter]
is the SDFS parameter name,
[reference_name]
is the name of the reference monitor, and YYMMDD
is the date the figure was compiled.
Below is an example of evaluation.stats_df
for the Toco Toucan Quickstart Guide example:
Sensor Name |
Sensor_Number |
Sensor_Serial |
Averaging Interval |
Param |
Reference |
R$^2$ |
Slope |
Intercept |
Sensor RMSE |
N |
Sensor_Min |
Sensor_Max |
Sensor_Mean |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Toco_Toucan |
1 |
RT01 |
1-hour |
PM25 |
T-API T640X at 16.67 LPM |
0.5216234756550462 |
0.8191825319244019 |
-1.676243556875662 |
3.5281941408285844 |
710.0 |
1.1283333333333334 |
13.763333333333332 |
4.710611124421121 |
Toco_Toucan |
2 |
RT02 |
1-hour |
PM25 |
T-API T640X at 16.67 LPM |
0.5540957508355946 |
0.8254680178697646 |
-1.7550479793461324 |
3.508767319754394 |
712.0 |
1.145 |
12.465 |
4.678385416666664 |
Toco_Toucan |
3 |
RT03 |
1-hour |
PM25 |
T-API T640X at 16.67 LPM |
0.5638579786715556 |
0.7508014694014582 |
-1.692978020396487 |
3.931586688145356 |
712.0 |
0.8816666666666666 |
11.37 |
4.158528206928842 |
Metric Average: |
0.5465257350540654 |
0.7984840063985416 |
-1.7080898522060937 |
3.6561827162427782 |
711.3333333333334 |
1.0516666666666667 |
12.532777777777776 |
4.515841582672209 |
|||||
Toco_Toucan |
1 |
RT01 |
24-hour |
PM25 |
T-API T640X at 16.67 LPM |
0.6228184308540052 |
0.862652058229343 |
-2.056301654583035 |
3.3090455977487983 |
29.0 |
2.0139680333564973 |
7.857430555555556 |
4.65241859578438 |
Toco_Toucan |
2 |
RT02 |
24-hour |
PM25 |
T-API T640X at 16.67 LPM |
0.6579780523926115 |
0.9185089802725335 |
-2.4919265748675032 |
3.3008919655600906 |
29.0 |
1.9179166666666667 |
8.139861111111111 |
4.6511848769809605 |
Toco_Toucan |
3 |
RT03 |
24-hour |
PM25 |
T-API T640X at 16.67 LPM |
0.5775702631554632 |
0.8405996738864467 |
-2.389694167854771 |
3.8135986597907343 |
29.0 |
1.1636111111111112 |
7.641319444444444 |
4.14752787996533 |
Metric Average: |
0.6194555821340266 |
0.8739202374627744 |
-2.3126407991017697 |
3.4745120743665407 |
29.0 |
1.698498603711425 |
7.879537037037037 |
4.483710450910223 |
Deployment Dictionary: evaluation.deploy_dict
The deployment dictionary evaluation.deploy_dict
contains descriptive statistics and textual information about
the deployment, including details about the testing agency, deployment site,
sensors tested, and site conditions during the evaluation.
The top level organizes details by deployment group, testing information, and testing location. A deployment group is defined as a collection of sensors that are collocated and concurrently recording data during a consecutive timeframe.
Expanding the deployment group field, three subfields are listed detailing sensor descriptions (sensors), statistics pertaining to the evaluation parameter (PM25), and statistics describing the meteorological site conditions during the deployment (Meteorological Conditions). Timestamps for the start, end, and duration of the evaluation are also listed (derived from sensor dataset timestamps).
The parameter statistics subfield contains categories for sensor-sensor (inter-sensor)
precision, error relative to FRM/FEM, and reference (FRM/FEM) measurement statistics.
The example shown below is for a scenario where the evaluation parameter is PM25
(PM2.5). Statistics are presented at sampling frequencies indicated for the
evaluation parameter via the eval_param_averaging instance attribute.
The Meteorological Conditions subfield contains statistics pertaining to temperature, relative humidity, and other parameters measured by instruments at the testing site. These measurements are independent of air sensor meteorological measurements collected by internal sensing components.
The deployment dictionary evaluation.deploy_dict
is saved as a JSON file to
/data/eval_stats/[sensor_name]/[sensor_name]_[parameter]_Evaluation_YYMMDD.json
where [sensor_name]
is the name of the sensor, [parameter]
is the SDFS parameter name,
[reference_name]
is the name of the reference monitor, and YYMMDD
is the date the figure was compiled.
Below is an example of evaluation.deploy_dict
for the Toco Toucan Quickstart Guide example:
{
"sensortoolkit Version": "0.7.4b2",
"Date of Analysis": "2022-02-04 15:12:41 PM",
"Sensor Name": "Toco_Toucan",
"Sensor Firmware Version": "Unspecified",
"Deployment Groups": {
"Group 1": {
"sensors": {
"1": {
"serial_id": "RT01",
"deploy_issues": "False",
"recording_interval": "1-minute",
"uptime_1-hour": 97.268,
"uptime_24-hour": 96.667
},
"2": {
"serial_id": "RT02",
"deploy_issues": "False",
"recording_interval": "1-minute",
"uptime_1-hour": 97.541,
"uptime_24-hour": 96.667
},
"3": {
"serial_id": "RT03",
"deploy_issues": "False",
"recording_interval": "1-minute",
"uptime_1-hour": 97.541,
"uptime_24-hour": 96.667
}
},
"eval_start": "2019-08-01T12:11:00+0000",
"eval_end": "2019-08-31T23:59:00+0000",
"eval_duration": "30 days 11:48:00",
"PM25": {
"Precision": {
"cv_1-hour": 12.951,
"std_1-hour": 0.585,
"n_1-hour": 712,
"cv_24-hour": 6.874,
"std_24-hour": 0.308,
"n_24-hour": 29
},
"Error": {
"rmse_1-hour": 3.654,
"nrmse_1-hour": 46.897,
"rmse_24-hour": 3.483,
"nrmse_24-hour": 44.881
},
"Reference": {
"reference_name": "T-API T640X at 16.67 LPM",
"conc_min_1-hour": 3.3,
"conc_max_1-hour": 15.3,
"n_exceed_conc_goal_1-hour": 0,
"conc_min_24-hour": 5.663,
"conc_max_24-hour": 11.046,
"n_exceed_conc_goal_24-hour": 0
}
},
"Meteorological Conditions": {
"Temperature": {
"instrument_name": "Philips Pw9762/02",
"min_1-hour": 14.3,
"max_1-hour": 37.7,
"n_exceed_target_criteria_1-hour": 0,
"n_measurement_pairs_1-hour": 728.0,
"min_24-hour": 21.146,
"max_24-hour": 28.904,
"n_exceed_target_criteria_24-hour": 0,
"n_measurement_pairs_24-hour": 31.0
},
"Relative Humidity": {
"instrument_name": "Hygrothermograph Elec or Mach Avg",
"min_1-hour": 24.0,
"max_1-hour": 97.0,
"n_exceed_target_criteria_1-hour": 144,
"n_measurement_pairs_1-hour": 728.0,
"min_24-hour": 59.917,
"max_24-hour": 87.583,
"n_exceed_target_criteria_24-hour": 0,
"n_measurement_pairs_24-hour": 31.0
}
}
}
},
"Testing Organization": {
"Deployment name": "[Name of Deployment]",
"Org name": [
"[Organization name line 1]",
"[Organization name line 2]"
],
"Website": {
"website name": "[Website Name]",
"website link": "[Website Link]"
},
"Contact email": "[Contact Email]",
"Contact phone": "[Contact Phone Number]"
},
"Testing Location": {
"Site name": "[Site Name]",
"Site address": "[Site Address]",
"Site lat": "[Site Latitude]",
"Site long": "[Site Longitude]",
"Site AQS ID": "[Site AQS ID]"
}
}
Note
n_exceed_conc_goal
(1-hour/24-hour) for the evaluation parameter reference
is the number of averaging intervals during which the reference concentration
exceeds the EPA’s recommended elevated concentration value for either PM2.5
(>25 μg/m3) or O3 (> 60 ppbv) testing.
This term as it relates to the Meteorological_Conditions
subcategories for
temperature and relative humidity indicate the number of intervals during which
conditions exceeded the manufacturer’s recommended operating range.