sensortoolkit.ingest._sensor_import
This module contains wrapper methods for importing and loading sensor data.
These methods call on other methods both within this module and in parallel
modules. For instance, recorded sensor datasets are located by the
sensor_import()
method, which subsequently calls methods in the
_standard_ingest.py
module for converting these datasets to SDFS format.
sensor_import()
can also be used to load previously processed (SDFS
formatted) sensor datasets by calling methods in the
_processed_data_loader.py
module.
Important
The AirSensor.sensor_setup()
method can be used to import sensor data from
a wide range of formatting schemes. However, there may be circumstances in
which the setup method and underlying generalized ingestion methodology is not
able to handle certain sensor datasets.
For example, devices that record multiple datasets simulaneously will require custom ingestion methods. Datasets for the PurpleAir PA-II that are obtained from the ThingSpeak API may fall into this category, as the API service may provide separate datasets for each internal PM sensor (channels A and B).
For these circumstances, users should create custom functions for importing sensor data. When attempting to load sensor data via the AirSensor.load_data() method, users should pass the custom ingestion function to load_data().
Example:
# Your custom ingestion function
def custom_ingestion_method(path_to_data_file, sensor_serial_id):
# Load data from the file path for the specified sensor unit
data = pandas.read_csv(path_to_data_file)
# Other steps you may need to take to convert the data into SDFS format
# ...
return data
# Assuming you have created a AirSensor object named 'sensor'
# Pass the custom ingest function to the ingest_method attribute
sensor.load_data(load_raw_data=True,
write_to_file=True,
ingest_method=custom_ingestion_method)
If users come across a circumstance where the standard_ingest()
method is
not successfully able to import sensor data, users are recommended to create
a custom ingestion method.
- @Author:
- Samuel Frederick, NSSC Contractor (ORAU)U.S. EPA / ORD / CEMM / AMCD / SFSB
- Created:
Wed Dec 4 08:57:18 2019
- Last Updated:
Wed Jul 14 10:22:15 2021
Functions
Concatenate pandas DataFrame with DateTimeIndex to the specified time period (bdate, edate). |
|
Wrapper for ingestion modules. |
|
Import recorded or processed sensor data. |