Common submodule¶
This module provides routines that are used throughout the library.
rainforest.common.io_data
: read various MCH data formats
rainforest.common.retrieve_data
: retrieve MCH files (COSMO, polar, Cartesian) from storage
rainforest.common.radarprocessing
: radar processing routines
rainforest.common.constants
: define constants used throughout the code
rainforest.common.graphics
: creating plots for QPE realization and evaluation
rainforest.common.lookup
: computing and reading lookup tables that are used for interpolation and geolocalization of data
rainforest.common.lookup
: various small utilities used throughout the code
rainforest.common.io_data module¶
Functions to read MeteoSwiss products
Daniel Wolfensberger MeteoSwiss/EPFL daniel.wolfensberger@epfl.ch December 2019
-
rainforest.common.io_data.
read_cart
(cart_file)¶ Generic function that reads a Cartesian radar file, either in gif or binary format (ELDES or as generated by the qpe module of this library) and converts its data to floating numbers
- Parameters
cart_file (str) – Full path of the file to read as a a string
- Returns
- Return type
The cartesian data in a numpy array
-
rainforest.common.io_data.
read_gif
(gif_file)¶ Reads a Cartesian radar file in gif format
- Parameters
gif_file (str) – Full path of the file to read as a a string
- Returns
- Return type
The cartesian data in a numpy array
-
rainforest.common.io_data.
read_polar
(polar_files, physic_value=True)¶ Reads a polar radar metranet file as a pyart radar instance, if multiple files corresponding to multiple elevations (sweeps) are provided they are merged into a single pyart instance
- Parameters
polar_files (str or list of strings) –
Can be either: - Full path of the file to read as a a string - string with wildcard pointing to the files to read,
e.g. …../MLA192711055.*
a list of full filepaths
- Returns
sweepnumbers (list) – list of sweeps numbers corresponding to all files that were read
merged (a pyart Radar instance)
-
rainforest.common.io_data.
read_station_data
(gauge_file)¶ Reads gauge data as given by the Climap software
- Parameters
gauge_file (str) – Full path of the file to read as a a string
- Returns
- Return type
The cartesian data in a numpy array
-
rainforest.common.io_data.
read_status
(status_file, add_wet_radome=False)¶ Reads a radar xml status file
- Parameters
fname (str) – Full path of the status xml file to be read
add_wet_Radome (boolean (optional)) – For older files, there is not information about the wet radome. If this is true, the script will estimate the wet radome precipitation as a 3 x 3 mean of the RZC product at the given time (as is done for more recent files)
- Returns
- Return type
The status as a python dict
-
rainforest.common.io_data.
read_vpr
(xml_file, radar=None)¶ Reads a vpr xml file and returns an interpolator that can be used to obtain the vpr correction at any altitude
- Parameters
vpr_file (str) –
Full path of the file to read radar : char (optional)
the radar is required to know the vpr reference height, it needs to be either ‘A’,’D’,’L’,’P’ or ‘W’, if not specified, it will be inferred from the vpr filename
- Returns
An interpolator that returns the vpr correction as a function of
the height, the correction is a multiplicative factor
-
rainforest.common.io_data.
read_xls
(xls_file)¶ Reads an excel file such as those used for CPC vlaidation
- Parameters
fname (str) – Full path of the excel file to be read
- Returns
- Return type
The excel file as a Pandas dataframe
-
rainforest.common.io_data.
save_gif
(gif_file, precip)¶ Saves a precipitation map in float to a gif file (AQC format)
- Parameters
gif_file (str) – Full path of the file to write as a a string
precip (2D array) – 2D array containing the precipitation intensities
rainforest.common.retrieve_data module¶
Functions to retrieve MeteoSwiss products from the archives
Daniel Wolfensberger MeteoSwiss/EPFL daniel.wolfensberger@epfl.ch December 2019
-
rainforest.common.retrieve_data.
get_COSMO_T
(time, sweeps=None, radar=None)¶ Retrieves COSMO temperature data from the CSCS repository, and interpolates them to the radar gates, using precomputed lookup tables
- Parameters
time (datetime.datetime instance) – the time at which to get the COSMO data in datetime format
sweeps (list of integers) – specify which sweeps (elevations) need to be retrieved in the form of a list, if not specified, all 20 will be retrieved
radar (list of chars) – list of radars for which to retrieve COSMO data, if not specified all 5 radars will be used (‘A’,’L’,’D’,’W’,’P’)
- Returns
T_at_radar – A dict containing the temperature at the radar gates, in the following form: dict[radar][‘T’][sweep_number]
- Return type
dict
-
rainforest.common.retrieve_data.
get_COSMO_variables
(time, variables, sweeps=None, radar=None, tmp_folder='/tmp/', cleanup=True)¶ Retrieves COSMO data from the CSCS repository, and interpolates them to the radar gates, using precomputed lookup tables This is a more generic but much slower function than the previous one, as it reads all COSMO variables directly from the GRIB files
- Parameters
time (datetime.datetime instance) – the time at which to get the COSMO data in datetime format
variables (list of strings) – List of COSMO variables to retrieve, ex. P, T, QV, QR, RH, etc…
sweeps (list of integers (optional)) – specify which sweeps (elevations) need to be retrieved in the form of a list, if not specified, all 20 will be retrieved
= list of chars (optional) (radar) – list of radars for which to retrieve COSMO data, if not specified all 5 radars will be used (‘A’,’L’,’D’,’W’,’P’)
= str (optional) (tmp_folder) – Directory where to store the extracted files
= boolean (optional) (cleanup) – If true all extracted files will be deleted before returning the output (recommended)
- Returns
A dict containing the COSMO variables at the radar gates, in the following
form (dict[radar][variables][sweep_number])
-
rainforest.common.retrieve_data.
retrieve_CPCCV
(time, stations)¶ Retrieves cross-validation CPC data for a set of stations from the xls files prepared by Yanni
- Parameters
time (datetime.datetime instance) – starting time of the time range
stations (list of str) – list of weather stations at which to retrieve the CPC.CV data
- Returns
A numpy array corresponding at the CPC.CV estimations at every specified
station
-
rainforest.common.retrieve_data.
retrieve_prod
(folder_out, start_time, end_time, product_name, pattern=None, pattern_type='shell', sweeps=None)¶ Retrieves radar data from the CSCS repository for a specified time range, unzips them and places them in a specified folder
- Parameters
folder_out (str) – directory where to store the unzipped files
start_time (datetime.datetime instance) – starting time of the time range
end_time (datetime.datetime instance) – end time of the time range
product_name (str) – name of the product, as stored on CSCS, e.g. RZC, CPCH, MZC, BZC…
pattern (str) – pattern constraint on file names, can be used for products which contain multiple filetypes, f.ex CPCH folders contain both rda and gif files, if only gifs are wanted : file_type = ‘*.gif’
pattern_type (either 'shell' or 'regex' (optional)) – use ‘shell’ for standard shell patterns, which use * as wildcard use ‘regex’ for more advanced regex patterns
sweeps (list of int (optional)) – For polar products, specifies which sweeps (elevations) must be retrieved, if not specified all available sweeps will be retrieved
- Returns
- Return type
A list containing all the filepaths of the retrieved files
rainforest.common.radarprocessing module¶
Created on Mon Dec 16 17:38:52 2019
@author: wolfensb
-
class
rainforest.common.radarprocessing.
Radar
(radname, polfiles, statusfile=None, vprfile=None)¶ Bases:
object
A class that contains polar radar data and performs some pre-processing before adding that data to the database or computing the QPE product The different elevations are stored in a dictionary rather as in a single pyart radar instance as this was found to be faster in practice
Creates an Radar class instance
- Parameters
radname (char) – Name of the radar, either ‘A’,’D’,’L’,’P’ or ‘W’
polfiles (list of str) – List of full filepaths of the radar files for a given timestemp, one file for every elevation, typically obtained with the retrieve_prod function in the common submodule
statusfile (str (optional)) – Full path of the status file that corresponds to this particular radar and timestep, used to compute noise estimates
vprfile (str(optional)) – Full path of the vpr xml file that corresponds to this particular radar and timestep, used to compute VPR correction
-
add_cosmo_data
(cosmo_data)¶ Adds COSMO data to the radar instance
- Parameters
cosmo_data (dict) –
dict of COSMO data at polar coordinates obtained from the get_COSMO_variables function in the common submodule this dict must have the following format
dic[variable][sweep]
-
compute_hydro
()¶ Computes the hydrometeor classification using Nikola Besic’ algorithm, all necessary fields ZH, ZDR, RHOHV, KDP, T (COSMO) must be available
-
compute_kdp
(dscfg)¶ Computes KDP using the simple moving least-square algorithm
- Parameters
dscfg (dict) – dictionary containing the following fields RMIN: RMAX: RWIND: ZMIN: ZMAX:
-
compute_noise
()¶ Computes a noise estimate from a status file
-
correct_attenuation
()¶ Corrects for attenuation using the ZPHI algorithm (Testud et al.) using the COSMO temperature to identify liquid precipitation
-
get_field
(sweep, field_name)¶ Gets a radar variable at given elevation (sweep)
- Parameters
sweep (int) – Sweep number from 1 to 20
field_name (str) – name of the variable, e.g. ZH, ZDR, RHOHV, SW, …
-
snr_mask
(snr_threshold)¶ Masks the radar data at low SNR
- Parameters
snr_threshold (float) – Minimal SNR to consider in dB
-
visib_mask
(min_visib, max_visib_corr)¶ Masks the radar data at low visibility and corrects the reflectivity for visibility
- Parameters
min_visib (int) – Minimal visibility below which the data is masked
max_visib_corr (float) – Maximum visibility correction factor, the visibility correction is 100/VISIB (with VISIB in %) and can be thresholded with this parameter. This is usually set to 2 at MeteoSwiss
-
rainforest.common.radarprocessing.
hydroClass_single
(radars, zh, zdr, kdp, rhohv, temp, weights=array([1., 1., 1., 0.75, 0.5 ]))¶ Computes the hydrometeor classes for columnar data, note that all input arrays except weights must have the same length
- Parameters
radars (ndarray of char) – Array of radar IDs, (‘A’,’D’,’L’,’P’,’W’)
zh (ndarray) – Array of radar reflectivity in dBZ
zdr (ndarray) – Array of diff. reflectivity in dB
kdp (ndarray) – Array of specific phase shift on propagation in deg / km
rhohv (ndarray) – Array of copolar correlation coefficient
temp (ndarray) – Array of radar temperature in Celcius
weights (ndarray (optional)) – The weight of every input feature, zh, zdr, kdp, rhohv, temp in the hydrometeor classification
- Returns
The hydrometeor classes as ndarray with values from 0 to 8, corresponding to
the classes – 0 : no data 1 : aggregates (AG) 2 : light rain (LR) 3 : ice crystals (CR) 4 : rimed particles (RP) 5 : rain (RN) 6 : vertically aligned ice (VI) 7 : wet snow (WS) 8 : melting hail (MH) 9: dry hail / high density graupel (IH/HDG)
rainforest.common.constants module¶
Set of constants regarding MeteoSwiss radars and QPE
-
rainforest.common.constants.
MODE
(x)¶
rainforest.common.graphics module¶
Set of functions to display QPE precipitation data and verification scores
-
class
rainforest.common.graphics.
MidpointNormalize
(vmin=None, vmax=None, transition=None, clip=False)¶ Bases:
matplotlib.colors.Normalize
Normalizing that is linear up to a certain transition value, logarithmic afterwards
If vmin or vmax is not given, they are initialized from the minimum and maximum value respectively of the first input processed. That is, __call__(A) calls autoscale_None(A). If clip is True and the given value falls outside the range, the returned value will be 0 or 1, whichever is closer. Returns 0 if:
vmin==vmax
Works with scalars or arrays, including masked arrays. If clip is True, masked values are set to 1; otherwise they remain masked. Clipping silently defeats the purpose of setting the over, under, and masked colors in the colormap, so it is likely to lead to surprises; therefore the default is clip = False.
-
class
rainforest.common.graphics.
QPE_cmap
¶ Bases:
matplotlib.colors.LinearSegmentedColormap
Colormap that uses purple tones for small values, and then a transition from blue to red for values above the transition
Create color map from linear mapping segments
segmentdata argument is a dictionary with a red, green and blue entries. Each entry should be a list of x, y0, y1 tuples, forming rows in a table. Entries for alpha are optional.
Example: suppose you want red to increase from 0 to 1 over the bottom half, green to do the same over the middle half, and blue over the top half. Then you would use:
cdict = {'red': [(0.0, 0.0, 0.0), (0.5, 1.0, 1.0), (1.0, 1.0, 1.0)], 'green': [(0.0, 0.0, 0.0), (0.25, 0.0, 0.0), (0.75, 1.0, 1.0), (1.0, 1.0, 1.0)], 'blue': [(0.0, 0.0, 0.0), (0.5, 0.0, 0.0), (1.0, 1.0, 1.0)]}
Each row in the table for a given color is a sequence of x, y0, y1 tuples. In each sequence, x must increase monotonically from 0 to 1. For any input value z falling between x[i] and x[i+1], the output value of a given color will be linearly interpolated between y1[i] and y0[i+1]:
row i: x y0 y1 / / row i+1: x y0 y1
Hence y0 in the first row and y1 in the last row are never used.
See also
LinearSegmentedColormap.from_list
Static method; factory function for generating a smoothly-varying LinearSegmentedColormap.
makeMappingArray
For information about making a mapping array.
-
rainforest.common.graphics.
plot_crossval_stats
(stats, output_folder)¶ Plots the results of a crossvalidation intercomparion as performed in the rf.py module
- Parameters
stats (dict) – dictionary containing the result statistics as obtained in the rf.py:model_intercomparison function
output_folder (str) – where to store the plots
-
rainforest.common.graphics.
qpe_plot
(data, subplots=None, figsize=None, vmin=0.04, vmax=120, transition=10, ch_border=True, xlim=None, ylim=None, cbar_orientation='horizontal', **kwargs)¶ Plots one or multiple QPE realizations using a special colormap, that shows a clear transition between low and high precipitation intensities, for low precipitation it is linear whereas for high precipitation it is logarithmic
If multiple QPE realizations are given, they will be displayed as subplots
- Parameters
data (list of numpy arrays or numpy array) – the set of QPE realizations to display
subplots (2-element tuple (optional)) – Tuple indicating the number of subplots in each direction, the product of its elements must be equal to the number of QPE realizations If not provided, the default will be (1,n) where n is the number of realizations
figsize (2-element tuple (optional)) – Tuple indicating the size of the figure in inches in both directions (w,h)
vmin (float (optional)) – Minimum value of precipitation to display, values below will be blank
vmax (float (optional)) – Maximum value of precipitation to display, values below above will be shown with the color corresponding to vmax
transition (float (optional)) – The transition value from which to change colormap and switch from linear to logarithmic scale
ch_border (bool (optiona)) – Whether to overlay the shapefile of the Swiss borders
xlim (2 element tuple (optional)) – limits of the plots in the west-east direction (in Swiss coordinates)
ylim (2 element tuple (optional)) – limits of the plots in the south-north direction (in Swiss coordinates)
cbar_orientation (str (optional)) – colorbar orientation, either ‘horizontal’ or ‘vertical’
**kwargs – All additional arguments that can be passed to imshow
- Returns
- Return type
Nothing
-
rainforest.common.graphics.
qpe_scatterplot
(qpe_est, ref, title_prefix='', figsize=(10, 7.5))¶ Plots the results of multiple QPE models as a function of the reference gauge measurement
- Parameters
qpe_est (dict of arrays) – Every value in the dictionary is a set of QPE estimates, every key is a model
ref (np.ndarray) – contains the reference observations (gauge), must have the same shape as any element in qpe_est
- title_prefix: str (optional)
a prefix for the suptitle (global titl
- figsize: 2-element tuple (optional)
Tuple indicating the size of the figure in inches in both directions (w,h)
- Returns
- Return type
Nothing
-
rainforest.common.graphics.
score_plot
(scores, title_prefix='', figsize=(10, 5))¶ Plots a series of QPE verification scores in the form of stacked barplots, for different ranges of precipitation
IMPORTANT: the scores dictionary must have the following structure
scores[model][precip_range][score]
for example
scores[‘RF_dualpol’][‘0.0-2.0’][‘ME’] = -0.27
you can get such a dictionary with the perfscores function in common.utils i.e. scores[‘RZC’] = perfscores(…)
- Parameters
scores (dict of dict of dict of scores) – the set of scores to display
title_prefix (str (optional)) – a prefix for the suptitle (global title)
figsize (2-element tuple (optional)) – Tuple indicating the size of the figure in inches in both directions (w,h)
- Returns
- Return type
Nothing
rainforest.common.lookup module¶
Functions to read and compute lookup tables
The available lookup tables are
qpebias_station : dict with the bias correction of RZC at every station
cosmo1_to_rad : dict which maps COSMO grid to polar radar grid it has keys [sweep][coord_type], coord_type is ‘idx_0’ : first index of COSMO grids, ‘idx_1’: second, ‘idx_3’: third, mask’ is 1 for points that fall outside of COSMO domain. This lookup table is valid only for COSMO data stored in /store/s83/owm/COSMO-1/
cosmo2_to_rad : same thing but for COSMO 2 data
cosmo1T_to_rad : same thing but for netCDF files of COSMO1 temperature extracted for MDR and stored in /store/s83/owm/COSMO-1/ORDERS/MDR/
cosmo1T_to_rad : same thing but for netCDF files of COSMO2 temperature extracted for MDR and stored in /store/msrad/cosmo/cosmo2/data/
station_to_rad : maps the SMN stations to radar coordinates, it is an extraction of the more generic but less convenient qpegrid_to_rad table It is list of 3 elements, first element is a dict with keys [station][sweep][ncode] and gives the polar gates (azimuth_idx, range_idx) that fall within a Cartesian pixel at a given radar elevation and for a given station neighbour (00 = station location, -1-1 = one to the south-west, 22 = two km to north and 2 km to east) second element is a dict giving distance from every station to the radar third element is a dict of keys [station][sweep] giving the height above ground of the radar observations above that station at a given elevation (sweep) MISSING KEYS IMPLY NO RADAR VISIBILITY
cartcoords_rad : gives the Cartesian (Swiss LV03) coordinates of all polar gates. It is a dict that gives for every sweep a 3D array of shape ncoords x nazimuth x nrange, ncoords is 3, first slice is Swiss Y coordinate (west to east), second is Swiss X-coordinate (south to north) and last is Swiss Z coordinate (altitude)
qpegrid_to_rad : maps the radar polar data to any gridpoint of the Swiss QPE grid. It is simply a 2D array with 5 columns | sweep | azimuth_idx | range_idx | Swiss Y coord | Swiss X coord|
station_to_qpegrid : maps every station to the corresponding QPE gridpoint it is a dict of keys [station][ncode] and gives the index of every neighbour of every station in the QPE 2D grid 640 x 710 pixels
visibility_rad : gives the (static) visibility of every polar gate for a given sweep number in the form of a 2D field of size nazimuth x nrange
-
rainforest.common.lookup.
calc_lookup
(lookup_type, radar=None)¶ Calculates a lookup table and stores it in the /data/lookup_data folder
- Parameters
lookup_type (str) –
the lookup table type, must be one of
qpebias_station
cosmo1_to_rad
cosmo2_to_rad
cosmo1T_to_rad
cosmo2T_to_rad
station_to_rad
cartcoords_rad
qpegrid_to_rad
station_to_qpegrid
visibility_rad
radar (char or list of chars (optional)) – the radar for which to retrieve the lookup table, needed only if the lookup_type contains the term ‘rad’, must be either ‘A’, ‘D’, ‘L’, ‘W’ or ‘P’
-
rainforest.common.lookup.
get_lookup
(lookup_type, radar=None)¶ Read a lookup table from the /data/lookup_data folder
- Parameters
lookup_type (str) –
the lookup table type, must be one of
qpebias_station
cosmo1_to_rad
cosmo2_to_rad
cosmo1T_to_rad
cosmo2T_to_rad
station_to_rad
cartcoords_rad
qpegrid_to_rad
station_to_qpegrid
visibility_rad
radar (char or list of chars (optional)) – the radar for which to retrieve the lookup table, needed only if the lookup_type contains the term ‘rad’, must be either ‘A’, ‘D’, ‘L’, ‘W’ or ‘P’
- Returns
lut – The lookup table in the form of a python dict
- Return type
dict
rainforest.common.utils module¶
Set of functions that can be useful
Daniel Wolfensberger MeteoSwiss/EPFL daniel.wolfensberger@epfl.ch December 2019
-
rainforest.common.utils.
LV03toWGS84
(east, north, heights)¶ Converts a set of WGS84, lat/lon/heights to Swiss CH1903 coordinates
- Parameters
east (ndarray) – Easterly Swiss coordinates (CHY)
north (ndarray) – northerly Swiss coordinates (CHX)
heights (ndarray) – heights a.s.l in WGS84 coordinates
- Returns
- Return type
lat, lon and height coordinates in WGS84
-
rainforest.common.utils.
aggregate_multi
(array_3d, agg_operators)¶ Aggregates a 3D numpy array alongs its first axis, using different aggregation operators
- Parameters
array_3d (ndarray) – 3D numpy array, of shape (N x M x L)
agg_operators (ndarray of integers) – array of aggregation operators as defined in the constants.py file, must have the same length as the first dimension of array_3D (N)
- Returns
- Return type
An aggregated array of size M x L
-
rainforest.common.utils.
chunks
(l, n)¶ Divides a list l into n sublists of similar sizes
-
rainforest.common.utils.
dict_flatten
(mydict)¶ Flattens a nested dictionary
-
rainforest.common.utils.
envyaml
(filename)¶ Reads a yaml configuration file while parsing environment variables. Environment variables must be written as ${NAME_OF_VAR} in the yaml file
- Parameters
filename (str) – path of the input yaml file
- Returns
the yaml content in the form of a python dict
- Return type
dict
-
rainforest.common.utils.
get_qpe_files
(input_folder, t0=None, t1=None, time_agg=None, list_models=None)¶ Gets the list of all qpe files in a folder (as saved by qpe_compute) and separates them by qpe type and timestep
- Parameters
input_folder (str) – main directory where the qpe files are saved, it contains one subfolder for every qpe model (type) that was used
t0 (datetime (optional)) – Starting time of the period to retrieve, will be used to filter files, if not provided starting time will be time of first file
t1 (datetime (optional)) – End time of the period to retrieve, will be used to filter files, if not provided end time will be time of last file
time_agg (minutes (optional)) – Will aggregate all files to a reference time in minutes (e.g. use 10 to put together all files that correspond to a single gauge measurement)
list_models ((optional)) – List of qpe types to retrieve , if not provided all folders in input_folder will be used
- Returns
A dictionary where every key is a QPE model and every value is a list
with all files in chronological order
-
rainforest.common.utils.
hex_to_rgb
(value)¶
-
rainforest.common.utils.
idx_cart
(x, y)¶ Returns the Cartesian index of a set of coordinates x and y
-
rainforest.common.utils.
nanadd_at
(a, indices, b)¶ Replaces nans by zero in call to np.add.at
-
rainforest.common.utils.
nearest_time
(dt, reference)¶ Gets the nearest earlier reference timestep to a given datetime, for ex. if dt = 1 Jan 2020 10:12, and reference is 10 it will return
1 Jan 2020 10:10, or if dt = 1 Jan 2020 10:59 and reference is 60 it will return 1 Jan 2020 10:00
- Parameters
dt (datetime) – The datetime to check
reference (int) – The reference timestep in minutes
- Returns
- Return type
The closest earlier timestep in datetime format
-
rainforest.common.utils.
nested_dict_gen
(d)¶ The generator for the previous function
-
rainforest.common.utils.
nested_dict_values
(d)¶ Extracts all values from a nested dictionary
-
rainforest.common.utils.
perfscores
(est_data, ref_data, bounds=None, array=False)¶ Computes a set of precipitation performance scores, on different data ranges. The scores are
scatter: 0.5 * (Qw84(x) - Qw16(x)), where Qw is a quantile weighted by ref_data / sum(ref_data) and x is est_data / ref_data in dB scale
RMSE: root mean square error (linear error)
bias: (ME/mean(ref_data) + 1) in dB
ED: the energy distance which is a measure of the distance between two distributions (https://en.wikipedia.org/wiki/Energy_distance)
- Parameters
est_data (ndarray) – array of estimates (ex. precip from QPE)
ref_data (ndarray) – array of reference (ex. precip from gauge)
bounds (list (optional)) – list of bounds on ref_data for which to compute the error metrics, by default all data will be used (unbounded), note that even if you prescribe bounds the scores for the overall data will always be added in the output
array (boolean (optional)) – Whether or not to convert the output dict to a numpy array
- Returns
all_metrics – a dictionary containing all the scores, organized in the following way all_metrics[bound][score]
- Return type
dict or ndarray
-
rainforest.common.utils.
quantile
(data, weights, quantile)¶ Weighted quantile of an array with respect to the last axis.
- Parameters
data (ndarray) – Input array.
weights (ndarray) – Array with the weights. It must have the same size of the last axis of data.
quantile (float) – Quantile to compute. It must have a value between 0 and 1.
- Returns
quantile – The output value.
- Return type
float
-
rainforest.common.utils.
quantile_1D
(data, weights, quantile)¶ Compute the weighted quantile of a 1D numpy array.
- Parameters
data (ndarray) – Input array (one dimension).
weights (ndarray) – Array with the weights of the same size of data.
quantile (float) – Quantile to compute. It must have a value between 0 and 1.
- Returns
quantile_1D – The output value.
- Return type
float
-
rainforest.common.utils.
read_df
(pattern, dbsystem='dask', sqlContext=None)¶ Reads a set of data contained in a folder as a spark or dask DataFrame
- Parameters
pattern (str) – Unix style wildcard pattern pointing to the files, for example /store/msrad/folder/*.csv will read all csv files in that folder
dbsystem (str) – Either “dask” if you want a Dask DataFrame or “spark” if you want a spark dataframe
sqlContext (sqlContext instance) – sqlContext to use, required only if dbystem = ‘spark’
- Returns
- Return type
A spark or dask DataFrame instance
-
rainforest.common.utils.
read_task_file
(task_file)¶ Reads a database processing task file
-
rainforest.common.utils.
rename_fields
(data)¶ Rename pyart fields from pyrad names to simpler names, according to the dictionary PYART_NAMES_MAPPING in the constants.py module
-
rainforest.common.utils.
round_to_hour
(dt)¶ Returns the sweep number of a polar file based on its name
-
rainforest.common.utils.
split_by_time
(files_rad)¶ Separate a list of files by their timestamp
-
rainforest.common.utils.
stack_uneven
(arrays, fill_value=nan)¶ Fits mmltiple into a single numpy array, even if they are different sizes, assigning a fill_value to parts with no data
- Parameters
arrays (list of np arrays) – list of numpy array to stack, they can have different dimensions
fill_value ((float, optional)) – the fill value with which to fill the missing pixels
- Returns
a np.ndarray with size N x M, where N is the sum of the number of
rows of all arrays and M is the maximal number of columns in all arrays
-
rainforest.common.utils.
sweepnumber_fromfile
(fname)¶ Returns the sweep number of a polar file based on its name
-
rainforest.common.utils.
timefromfilename
(fname)¶ Returns the datetime of a file based on its name
-
rainforest.common.utils.
timestamp_from_datestr
(datestr)¶
-
rainforest.common.utils.
timestamp_from_datetime
(dt)¶
-
rainforest.common.utils.
wgs84toCH1903
(lat, lon, heights)¶ Converts a set of WGS84, lat/lon/heights to Swiss CH1903 coordinates, east, north and height
- Parameters
lat (ndarray) – latitudes in decimal format (degrees)
lon (ndarray) – longitudes in decimal format (degrees)
heights (ndarray) – heights a.s.l in WGS84 coordinates
- Returns
- Return type
east, north and height coordinates in CHLV190
rainforest.common.wgs84_ch1903 module¶
-
class
rainforest.common.wgs84_ch1903.
GPSConverter
¶ Bases:
object
GPS Converter class which is able to perform convertions between the CH1903 and WGS84 system.
-
CHtoWGSheight
(y, x, h)¶
-
CHtoWGSlat
(y, x)¶
-
CHtoWGSlng
(y, x)¶
-
DecToSexAngle
(dec)¶
-
LV03toWGS84
(east, north, height)¶ Convert LV03 to WGS84 Return a array of double that contain lat, long, and height
-
SexAngleToSeconds
(dms)¶
-
SexToDecAngle
(dms)¶
-
WGS84toLV03
(latitude, longitude, ellHeight)¶ Convert WGS84 to LV03 Return an array of double that contaign east, north, and height
-
WGStoCHh
(lat, lng, h)¶
-
WGStoCHx
(lat, lng)¶
-
WGStoCHy
(lat, lng)¶
-