mdf4reader module documentation¶
Measured Data Format file reader module for version 4.x.
Platform and python version¶
With Unix and Windows for python 2.6+ and 3.2+
Author: | Aymeric Rateau |
---|
Created on Thu Dec 10 12:57:28 2013
Dependencies¶
- Python >2.6, >3.2 <http://www.python.org>
- Numpy >1.6 <http://numpy.scipy.org>
- bitarray to parse bits in not aligned bytes
- Sympy to convert channels with formula if needed
- zlib to uncompress data block if needed
Attributes¶
- PythonVersion : float
- Python version currently running, needed for compatibility of both python 2.6+ and 3.2+
mdf4reader module¶
-
class
mdfreader.mdf4reader.
DATA
() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object's (key, value) pairs dict(iterable) -> new dictionary initialized as if via: d = {} for k, v in iterable: d[k] = v dict(**kwargs) -> new dictionary initialized with the name=value pairs in the keyword argument list. For example: dict(one=1, two=2)¶ Bases:
dict
Methods
addRecord
(record)Adds a new record in DATA class dict. clear
()copy
()fromkeys
($type, iterable[, value])Returns a new dict with keys from iterable and values equal to value. get
(k[,d])items
()keys
()load
(record, info[, nameList, sortedFlag, vlsd])Reads data block from record definition pop
(k[,d])If key is not found, d is returned if given, otherwise KeyError is raised popitem
()2-tuple; but raise KeyError if D is empty. read
(channelSet, info, filename)Reads data block readRecord
(recordID, info, buf[, channelSet])read record from a buffer setdefault
(k[,d])update
([E, ]**F)If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k] values
()-
addRecord
(record)¶ Adds a new record in DATA class dict.
Parameters: record class
channel group definition listing record channel classes
-
fid
¶
-
load
(record, info, nameList=None, sortedFlag=True, vlsd=False)¶ Reads data block from record definition
Parameters: record class
channel group definition listing record channel classes
info class
contains blocks
nameList : list of str, optional
list of channel names
sortedFlag : bool, optional
flag to know if data block is sorted (only one Channel Group in block) or unsorted (several Channel Groups identified by a recordID). As unsorted block can contain CG records in random order, block is processed iteratively, not in raw like sorted -> much slower reading
vlsd : bool
indicate a sd block, compressed (DZ) or not (SD)
Returns: numpy recarray of data
-
pointerTodata
¶
-
read
(channelSet, info, filename)¶ Reads data block
Parameters: channelSet : set of str
set of channel names
info : info object
contains blocks structures
filename
name of file ot read
-
readRecord
(recordID, info, buf, channelSet=None)¶ read record from a buffer
Parameters: recordID : int
record identifier
info class
contains blocks
buf : str
buffer of data from file to be converted to channel raw data
channelSet : set of str
setof channel names to be read
-
type
¶
-
-
mdfreader.mdf4reader.
DATABlock
(record, info, parent_block, channelSet=None, nrecords=None, sortedFlag=True, vlsd=False)¶ DATABlock converts raw data into arrays
Parameters: record : class
record class instance describing a channel group record
parent_block : class
MDFBlock class containing at least parent block header
channelSet : set of str, optional
defines set of channels to only read, can be slow but saves memory, for big files
nrecords: int, optional
number of records to read
sortedFlag : bool, optional
flag to know if data block is sorted (only one Channel Group in block) or unsorted (several Channel Groups identified by a recordID). As unsorted block can contain CG records in random order, block is processed iteratively, not in raw like sorted -> much slower reading
vlsd : bool
indicate a sd block, compressed (DZ) or not (SD)
Returns: a recarray containing the channels data
Notes
This function will read DTBlock, RDBlock, DZBlock (compressed), RDBlock (VLSD), sorted or unsorted
-
mdfreader.mdf4reader.
equalizeStringLength
(buf)¶ Makes all strings in a list having same length by appending spaces strings.
Parameters: buf : list of str Returns: list of str elements all having same length
-
mdfreader.mdf4reader.
formulaConv
(vect, formula)¶ apply formula conversion to data
Parameters: vect : numpy 1D array
raw data to be converted to physical value
cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict
Returns: converted data to physical value
-
mdfreader.mdf4reader.
linearConv
(vect, cc_val)¶ apply linear conversion to data
Parameters: vect : numpy 1D array
raw data to be converted to physical value
cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict
Returns: converted data to physical value
-
class
mdfreader.mdf4reader.
mdf4
(fileName=None, channelList=None, convertAfterRead=True, filterChannelNames=False, noDataLoading=False, compression=False, convertTables=False, metadata=2)¶ Bases:
mdfreader.mdf.mdf_skeleton
mdf file reader class from version 4.0 to 4.1.1
Attributes
fileName (str) file name MDFVersionNumber (int) mdf file version number masterChannelList (dict) Represents data structure: a key per master channel with corresponding value containing a list of channels One key or master channel represents then a data group having same sampling interval. multiProc (bool) Flag to request channel conversion multi processed for performance improvement. One thread per data group. convertAfterRead (bool) flag to convert raw data to physical just after read filterChannelNames (bool) flag to filter long channel names from its module names separated by ‘.’ file_metadata (dict) file metadata with minimum keys : author, organisation, project, subject, comment, time, date Methods
read4( fileName=None, info=None, multiProc=False, channelList=None, convertAfterRead=True) Reads mdf 4.x file data and stores it in dict _getChannelData4(channelName) Returns channel numpy array _convertChannel4(channelName) converts specific channel from raw to physical data according to CCBlock information _convertAllChannel4() Converts all channels from raw data to converted data according to CCBlock information -
apply_invalid_bit
(channel_name)¶ Mask data of channel based on its invalid bit definition if there is
Parameters: channel_name : str
Name of channel
-
getChannelName4
(name, path)¶ finds mdf channel name from name and path
Parameters: name : str
channel name
path: str
source path or name, or channel group name, source name or path
-
read4
(fileName=None, info=None, multiProc=False, channelList=None, convertAfterRead=True, compression=False, metadata=2)¶ Reads mdf 4.x file data and stores it in dict
Parameters: fileName : str, optional
file name
info : mdfinfo4.info4 class
info4 class containing all MDF Blocks
multiProc : bool
flag to activate multiprocessing of channel data conversion
channelList : list of str, optional
list of channel names to be read If you use channelList, reading might be much slower but it will save you memory. Can be used to read big files
convertAfterRead : bool, optional
flag to convert channel after read, True by default If you use convertAfterRead by setting it to false, all data from channels will be kept raw, no conversion applied. If many float are stored in file, you can gain from 3 to 4 times memory footprint To calculate value from channel, you can then use method .getChannelData()
compression : bool, optional
flag to activate data compression with blosc
metadata: int, optional, default = 2
Reading metadata has impact on performance, especially for mdf 4.x using xml. 2: minimal metadata reading (mostly channel blocks) 1: used for noDataLoading 0: all metadata reading, including Source Information, Attachment, etc..
-
write4
(fileName=None, compression=False)¶ Writes simple mdf 4.1 file
Parameters: fileName : str, optional
Name of file If file name is not input, written file name will be the one read with appended ‘_new’ string before extension
compression : bool
flag to store data compressed
Notes
All channels will be converted to physical data, so size might be bigger than original file
-
-
mdfreader.mdf4reader.
rationalConv
(vect, cc_val)¶ apply rational conversion to data
Parameters: vect : numpy 1D array
raw data to be converted to physical value
cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict
Returns: converted data to physical value
-
mdfreader.mdf4reader.
readUnsorted
(record, info, parent_block, channelSet=None)¶
-
mdfreader.mdf4reader.
read_sdblock
(signal_data_type, sdblock, sdblock_length)¶ Reads vlsd channel from its SD Block bytes
Parameters: signal_data_type : int
sdblock : bytes
SD Block bytes
sdblock_length: int
SD Block data length (header not included)
Returns: array
-
class
mdfreader.mdf4reader.
record
(dataGroup, channelGroup)¶ Bases:
list
Methods
addChannel
(info, channelNumber)add a channel in class append
(object)clear
()copy
()count
(value)extend
(iterable)generate_chunks
()calculate data split index
(value, [start, [stop]])Raises ValueError if the value is not present. initialise_recarray
(info, channelSet, nrecords)Initialise recarray insert
L.insert(index, object) – insert object before index loadInfo
(info)gathers records related from info class pop
([index])Raises IndexError if list is empty or index is out of range. readRecordBuf
(buf, info[, channelSet])read stream of record bytes readSortedRecord
(fid, info[, channelSet])reads record, only one channel group per datagroup read_all_channels_sorted_record
(fid)reads all channels from file using numpy fromstring, chunk by chunk read_channels_from_bytes
(bita, info[, …])reads stream of record bytes using dataRead module if available otherwise bitarray read_channels_from_bytes_fallback
(bita, info)reads stream of record bytes using bitarray in case no dataRead available read_not_all_channels_sorted_record
(fid, …)reads channels from file listed in channelSet remove
(value)Raises ValueError if the value is not present. reverse
L.reverse() – reverse IN PLACE sort
([key, reverse])-
CANOpen
¶
-
CGrecordLength
¶
-
Flags
¶
-
MLSD
¶
-
VLSD
¶
-
VLSD_CG
¶
-
addChannel
(info, channelNumber)¶ add a channel in class
Parameters: info : mdfinfo4.info4 class
channelNumber : int
channel number in mdfinfo4.info4 class
-
byte_aligned
¶
-
channelGroup
¶
-
channelNames
¶
-
dataGroup
¶
-
dataRecordName
¶
-
generate_chunks
()¶ calculate data split
Returns: (nrecord_chunk, chunk_size)
-
initialise_recarray
(info, channelSet, nrecords, dtype=None, channels_indexes=None)¶ Initialise recarray
Parameters: info: info class
channelSet : set of str, optional
set of channel to read
nrecords: int
number of records
dtype: numpy dtype, optional
channels_indexes: list of int, optional
Returns: rec : numpy recarray
contains a matrix of raw data in a recarray (attributes corresponding to channel name)
-
invalid_channel
¶
-
loadInfo
(info)¶ gathers records related from info class
Parameters: info : mdfinfo4.info4 class
-
master
¶
-
numberOfRecords
¶
-
numpyDataRecordFormat
¶
-
readRecordBuf
(buf, info, channelSet=None)¶ read stream of record bytes
Parameters: buf : stream
stream of bytes read in file
info class
contains blocks structure
channelSet : set of str, optional
set of channel to read
Returns: rec : dict
returns dictionary of channel with its corresponding values
-
readSortedRecord
(fid, info, channelSet=None)¶ reads record, only one channel group per datagroup
Parameters: fid :
file identifier
pointer
position in file of data block beginning
channelSet : set of str, optional
set of channel to read
Returns: rec : numpy recarray
contains a matrix of raw data in a recarray (attributes corresponding to channel name)
Notes
If channelSet is None, read data using numpy.core.records.fromfile that is rather quick. However, in case of large file, you can use channelSet to load only interesting channels or only one channel on demand, but be aware it might be much slower.
-
read_all_channels_sorted_record
(fid)¶ reads all channels from file using numpy fromstring, chunk by chunk
Parameters: fid :
file identifier
Returns: rec : numpy recarray
contains a matrix of raw data in a recarray (attributes corresponding to channel name)
-
read_channels_from_bytes
(bita, info, channelSet=None, nrecords=None, dtype=None, channels_indexes=None)¶ reads stream of record bytes using dataRead module if available otherwise bitarray
Parameters: bita : stream
stream of bytes
info: info class
channelSet : set of str, optional
set of channel to read
nrecords: int
number of records
dtype: numpy dtype
channels_indexes: list of int
Returns: rec : numpy recarray
contains a matrix of raw data in a recarray (attributes corresponding to channel name)
-
read_channels_from_bytes_fallback
(bita, info, channelSet=None, nrecords=None, dtype=None, channels_indexes=None)¶ reads stream of record bytes using bitarray in case no dataRead available
Parameters: bita : stream
stream of bytes
info: info class
channelSet : set of str, optional
set of channel to read
nrecords: int
number of records
dtype: numpy dtype
channels_indexes: list of int
Returns: rec : numpy recarray
contains a matrix of raw data in a recarray (attributes corresponding to channel name)
-
read_not_all_channels_sorted_record
(fid, info, channelSet)¶ reads channels from file listed in channelSet
Parameters: fid :
file identifier
info: info class
channelSet : set of str, optional
set of channel to read
Returns: rec : numpy recarray
contains a matrix of raw data in a recarray (attributes corresponding to channel name)
-
recordID
¶
-
recordIDCFormat
¶
-
recordIDsize
¶
-
recordLength
¶
-
recordToChannelMatching
¶
-
-
mdfreader.mdf4reader.
textToTextConv
(vect, cc_ref)¶ apply text to text conversion to data
Parameters: vect : numpy 1D array
raw data to be converted to physical value
cc_ref : cc_ref from mdfinfo4.info4 conversion block (‘CCBlock’) dict
Returns: converted data to physical value
-
mdfreader.mdf4reader.
textToValueConv
(vect, cc_val, cc_ref)¶ apply text to value conversion to data
Parameters: vect : numpy 1D array
raw data to be converted to physical value
cc_val : cc_val from mdfinfo4.info4 conversion block (‘CCBlock’) dict
cc_ref : cc_ref from mdfinfo4.info4 conversion block (‘CCBlock’) dict
Returns: converted data to physical value
-
mdfreader.mdf4reader.
valueRangeToTextConv
(vect, cc_val, cc_ref)¶ apply value range to text conversion to data
Parameters: vect : numpy 1D array
raw data to be converted to physical value
cc_val : cc_val from mdfinfo4.info4 conversion block (‘CCBlock’) dict
cc_ref : cc_ref from mdfinfo4.info4 conversion block (‘CCBlock’) dict
Returns: converted data to physical value
-
mdfreader.mdf4reader.
valueRangeToValueTableConv
(vect, cc_val)¶ apply value range to value table conversion to data
Parameters: vect : numpy 1D array
raw data to be converted to physical value
cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict
Returns: converted data to physical value
-
mdfreader.mdf4reader.
valueToTextConv
(vect, cc_val, cc_ref)¶ apply value to text conversion to data
Parameters: vect : numpy 1D array
raw data to be converted to physical value
cc_val : cc_val from mdfinfo4.info4 conversion block (‘CCBlock’) dict
cc_ref : cc_ref from mdfinfo4.info4 conversion block (‘CCBlock’) dict
Returns: converted data to physical value
-
mdfreader.mdf4reader.
valueToValueTableWInterpConv
(vect, cc_val)¶ apply value to value table with interpolation conversion to data
Parameters: vect : numpy 1D array
raw data to be converted to physical value
cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict
Returns: converted data to physical value
-
mdfreader.mdf4reader.
valueToValueTableWOInterpConv
(vect, cc_val)¶ apply value to value table without interpolation conversion to data
Parameters: vect : numpy 1D array
raw data to be converted to physical value
cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict
Returns: converted data to physical value