sensortoolkit.qc._persistent_values.persistent_values

persistent_values(df, param, tolerance=3, freq='H', invalidate=False)[source]

Flag data points where consecutive timestamp parameter values repeat.

Values persisting for N or greater consecutive timestamps will be flagged (N is the integer value set for the tolerance). If invalidate is true, corresponding values will be set null (np.nan).

Parameters
  • df (pandas DataFrame) – Dataset containing parameter data to check for repeating values.

  • param (str) – The name of the parameter to check for repeating values.

  • tolerance (int, optional) – The number of consecutive entries for repeated/persistent values required to flag a data point. Defaults to 3.

  • freq (TYPE, optional) – The sampling frequency or averaging interval of the passed dataset, expressed as a pandas offset alias (see a list here https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#offset-aliases). Defaults to ‘H’ for 1-hour averaged datasets.

  • invalidate (bool, optional) – If True, repeated entries will be set null (np.nan). Defaults to False.

Returns

Modified dataset with flagged entries for repeated data entries.

Return type

df (pandas DataFrame)