sensortoolkit.ingest._sensor_import

This module contains wrapper methods for importing and loading sensor data. These methods call on other methods both within this module and in parallel modules. For instance, recorded sensor datasets are located by the sensor_import() method, which subsequently calls methods in the _standard_ingest.py module for converting these datasets to SDFS format. sensor_import() can also be used to load previously processed (SDFS formatted) sensor datasets by calling methods in the _processed_data_loader.py module.

Important

The AirSensor.sensor_setup() method can be used to import sensor data from a wide range of formatting schemes. However, there may be circumstances in which the setup method and underlying generalized ingestion methodology is not able to handle certain sensor datasets.

For example, devices that record multiple datasets simulaneously will require custom ingestion methods. Datasets for the PurpleAir PA-II that are obtained from the ThingSpeak API may fall into this category, as the API service may provide separate datasets for each internal PM sensor (channels A and B).

For these circumstances, users should create custom functions for importing sensor data. When attempting to load sensor data via the AirSensor.load_data() method, users should pass the custom ingestion function to load_data().

Example:

# Your custom ingestion function
def custom_ingestion_method(path_to_data_file, sensor_serial_id):

    # Load data from the file path for the specified sensor unit
    data = pandas.read_csv(path_to_data_file)

    # Other steps you may need to take to convert the data into SDFS format
    # ...

    return data

# Assuming you have created a AirSensor object named 'sensor'
# Pass the custom ingest function to the ingest_method attribute
sensor.load_data(load_raw_data=True,
                 write_to_file=True,
                 ingest_method=custom_ingestion_method)

If users come across a circumstance where the standard_ingest() method is not successfully able to import sensor data, users are recommended to create a custom ingestion method.


@Author:
Samuel Frederick, NSSC Contractor (ORAU)
U.S. EPA / ORD / CEMM / AMCD / SFSB
Created:

Wed Dec 4 08:57:18 2019

Last Updated:

Wed Jul 14 10:22:15 2021

Functions

concat_dataset

Concatenate pandas DataFrame with DateTimeIndex to the specified time period (bdate, edate).

ingest_wrapper

Wrapper for ingestion modules.

sensor_import

Import recorded or processed sensor data.