mdf4reader module documentation

Measured Data Format file reader module for version 4.x.

Platform and python version

With Unix and Windows for python 2.6+ and 3.2+

Author:Aymeric Rateau

Created on Thu Dec 10 12:57:28 2013

Dependencies

Attributes

PythonVersion : float
Python version currently running, needed for compatibility of both python 2.6+ and 3.2+

mdf4reader module

class mdfreader.mdf4reader.DATA() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object's (key, value) pairs dict(iterable) -> new dictionary initialized as if via: d = {} for k, v in iterable: d[k] = v dict(**kwargs) -> new dictionary initialized with the name=value pairs in the keyword argument list. For example: dict(one=1, two=2)

Bases: dict

Methods

addRecord(record) Adds a new record in DATA class dict.
clear()
copy()
fromkeys($type, iterable[, value]) Returns a new dict with keys from iterable and values equal to value.
get(k[,d])
items()
keys()
load(record, info[, nameList, sortedFlag, vlsd]) Reads data block from record definition
pop(k[,d]) If key is not found, d is returned if given, otherwise KeyError is raised
popitem() 2-tuple; but raise KeyError if D is empty.
read(channelSet, info, filename) Reads data block
readRecord(recordID, info, buf[, channelSet]) read record from a buffer
setdefault(k[,d])
update([E, ]**F) If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]
values()
addRecord(record)

Adds a new record in DATA class dict.

Parameters:

record class

channel group definition listing record channel classes

fid
load(record, info, nameList=None, sortedFlag=True, vlsd=False)

Reads data block from record definition

Parameters:

record class

channel group definition listing record channel classes

info class

contains blocks

nameList : list of str, optional

list of channel names

sortedFlag : bool, optional

flag to know if data block is sorted (only one Channel Group in block) or unsorted (several Channel Groups identified by a recordID). As unsorted block can contain CG records in random order, block is processed iteratively, not in raw like sorted -> much slower reading

vlsd : bool

indicate a sd block, compressed (DZ) or not (SD)

Returns:

numpy recarray of data

pointerTodata
read(channelSet, info, filename)

Reads data block

Parameters:

channelSet : set of str

set of channel names

info : info object

contains blocks structures

filename

name of file ot read

readRecord(recordID, info, buf, channelSet=None)

read record from a buffer

Parameters:

recordID : int

record identifier

info class

contains blocks

buf : str

buffer of data from file to be converted to channel raw data

channelSet : set of str

setof channel names to be read

type
mdfreader.mdf4reader.DATABlock(record, info, parent_block, channelSet=None, nrecords=None, sortedFlag=True, vlsd=False)

DATABlock converts raw data into arrays

Parameters:

record : class

record class instance describing a channel group record

parent_block : class

MDFBlock class containing at least parent block header

channelSet : set of str, optional

defines set of channels to only read, can be slow but saves memory, for big files

nrecords: int, optional

number of records to read

sortedFlag : bool, optional

flag to know if data block is sorted (only one Channel Group in block) or unsorted (several Channel Groups identified by a recordID). As unsorted block can contain CG records in random order, block is processed iteratively, not in raw like sorted -> much slower reading

vlsd : bool

indicate a sd block, compressed (DZ) or not (SD)

Returns:

a recarray containing the channels data

Notes

This function will read DTBlock, RDBlock, DZBlock (compressed), RDBlock (VLSD), sorted or unsorted

mdfreader.mdf4reader.equalizeStringLength(buf)

Makes all strings in a list having same length by appending spaces strings.

Parameters:buf : list of str
Returns:list of str elements all having same length
mdfreader.mdf4reader.formulaConv(vect, formula)

apply formula conversion to data

Parameters:

vect : numpy 1D array

raw data to be converted to physical value

cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict

Returns:

converted data to physical value

mdfreader.mdf4reader.linearConv(vect, cc_val)

apply linear conversion to data

Parameters:

vect : numpy 1D array

raw data to be converted to physical value

cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict

Returns:

converted data to physical value

class mdfreader.mdf4reader.mdf4(fileName=None, channelList=None, convertAfterRead=True, filterChannelNames=False, noDataLoading=False, compression=False, convertTables=False, metadata=2)

Bases: mdfreader.mdf.mdf_skeleton

mdf file reader class from version 4.0 to 4.1.1

Attributes

fileName (str) file name
MDFVersionNumber (int) mdf file version number
masterChannelList (dict) Represents data structure: a key per master channel with corresponding value containing a list of channels One key or master channel represents then a data group having same sampling interval.
multiProc (bool) Flag to request channel conversion multi processed for performance improvement. One thread per data group.
convertAfterRead (bool) flag to convert raw data to physical just after read
filterChannelNames (bool) flag to filter long channel names from its module names separated by ‘.’
file_metadata (dict) file metadata with minimum keys : author, organisation, project, subject, comment, time, date

Methods

read4( fileName=None, info=None, multiProc=False, channelList=None, convertAfterRead=True) Reads mdf 4.x file data and stores it in dict
_getChannelData4(channelName) Returns channel numpy array
_convertChannel4(channelName) converts specific channel from raw to physical data according to CCBlock information
_convertAllChannel4() Converts all channels from raw data to converted data according to CCBlock information
apply_invalid_bit(channel_name)

Mask data of channel based on its invalid bit definition if there is

Parameters:

channel_name : str

Name of channel

getChannelName4(name, path)

finds mdf channel name from name and path

Parameters:

name : str

channel name

path: str

source path or name, or channel group name, source name or path

read4(fileName=None, info=None, multiProc=False, channelList=None, convertAfterRead=True, compression=False, metadata=2)

Reads mdf 4.x file data and stores it in dict

Parameters:

fileName : str, optional

file name

info : mdfinfo4.info4 class

info4 class containing all MDF Blocks

multiProc : bool

flag to activate multiprocessing of channel data conversion

channelList : list of str, optional

list of channel names to be read If you use channelList, reading might be much slower but it will save you memory. Can be used to read big files

convertAfterRead : bool, optional

flag to convert channel after read, True by default If you use convertAfterRead by setting it to false, all data from channels will be kept raw, no conversion applied. If many float are stored in file, you can gain from 3 to 4 times memory footprint To calculate value from channel, you can then use method .getChannelData()

compression : bool, optional

flag to activate data compression with blosc

metadata: int, optional, default = 2

Reading metadata has impact on performance, especially for mdf 4.x using xml. 2: minimal metadata reading (mostly channel blocks) 1: used for noDataLoading 0: all metadata reading, including Source Information, Attachment, etc..

write4(fileName=None, compression=False)

Writes simple mdf 4.1 file

Parameters:

fileName : str, optional

Name of file If file name is not input, written file name will be the one read with appended ‘_new’ string before extension

compression : bool

flag to store data compressed

Notes

All channels will be converted to physical data, so size might be bigger than original file

mdfreader.mdf4reader.rationalConv(vect, cc_val)

apply rational conversion to data

Parameters:

vect : numpy 1D array

raw data to be converted to physical value

cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict

Returns:

converted data to physical value

mdfreader.mdf4reader.readUnsorted(record, info, parent_block, channelSet=None)
mdfreader.mdf4reader.read_sdblock(signal_data_type, sdblock, sdblock_length)

Reads vlsd channel from its SD Block bytes

Parameters:

signal_data_type : int

sdblock : bytes

SD Block bytes

sdblock_length: int

SD Block data length (header not included)

Returns:

array

class mdfreader.mdf4reader.record(dataGroup, channelGroup)

Bases: list

Methods

addChannel(info, channelNumber) add a channel in class
append(object)
clear()
copy()
count(value)
extend(iterable)
generate_chunks() calculate data split
index(value, [start, [stop]]) Raises ValueError if the value is not present.
initialise_recarray(info, channelSet, nrecords) Initialise recarray
insert L.insert(index, object) – insert object before index
loadInfo(info) gathers records related from info class
pop([index]) Raises IndexError if list is empty or index is out of range.
readRecordBuf(buf, info[, channelSet]) read stream of record bytes
readSortedRecord(fid, info[, channelSet]) reads record, only one channel group per datagroup
read_all_channels_sorted_record(fid) reads all channels from file using numpy fromstring, chunk by chunk
read_channels_from_bytes(bita, info[, …]) reads stream of record bytes using dataRead module if available otherwise bitarray
read_channels_from_bytes_fallback(bita, info) reads stream of record bytes using bitarray in case no dataRead available
read_not_all_channels_sorted_record(fid, …) reads channels from file listed in channelSet
remove(value) Raises ValueError if the value is not present.
reverse L.reverse() – reverse IN PLACE
sort([key, reverse])
CANOpen
CGrecordLength
Flags
MLSD
VLSD
VLSD_CG
addChannel(info, channelNumber)

add a channel in class

Parameters:

info : mdfinfo4.info4 class

channelNumber : int

channel number in mdfinfo4.info4 class

byte_aligned
channelGroup
channelNames
dataGroup
dataRecordName
generate_chunks()

calculate data split

Returns:(nrecord_chunk, chunk_size)
hiddenBytes
initialise_recarray(info, channelSet, nrecords, dtype=None, channels_indexes=None)

Initialise recarray

Parameters:

info: info class

channelSet : set of str, optional

set of channel to read

nrecords: int

number of records

dtype: numpy dtype, optional

channels_indexes: list of int, optional

Returns:

rec : numpy recarray

contains a matrix of raw data in a recarray (attributes corresponding to channel name)

invalid_channel
loadInfo(info)

gathers records related from info class

Parameters:info : mdfinfo4.info4 class
master
numberOfRecords
numpyDataRecordFormat
readRecordBuf(buf, info, channelSet=None)

read stream of record bytes

Parameters:

buf : stream

stream of bytes read in file

info class

contains blocks structure

channelSet : set of str, optional

set of channel to read

Returns:

rec : dict

returns dictionary of channel with its corresponding values

readSortedRecord(fid, info, channelSet=None)

reads record, only one channel group per datagroup

Parameters:

fid :

file identifier

pointer

position in file of data block beginning

channelSet : set of str, optional

set of channel to read

Returns:

rec : numpy recarray

contains a matrix of raw data in a recarray (attributes corresponding to channel name)

Notes

If channelSet is None, read data using numpy.core.records.fromfile that is rather quick. However, in case of large file, you can use channelSet to load only interesting channels or only one channel on demand, but be aware it might be much slower.

read_all_channels_sorted_record(fid)

reads all channels from file using numpy fromstring, chunk by chunk

Parameters:

fid :

file identifier

Returns:

rec : numpy recarray

contains a matrix of raw data in a recarray (attributes corresponding to channel name)

read_channels_from_bytes(bita, info, channelSet=None, nrecords=None, dtype=None, channels_indexes=None)

reads stream of record bytes using dataRead module if available otherwise bitarray

Parameters:

bita : stream

stream of bytes

info: info class

channelSet : set of str, optional

set of channel to read

nrecords: int

number of records

dtype: numpy dtype

channels_indexes: list of int

Returns:

rec : numpy recarray

contains a matrix of raw data in a recarray (attributes corresponding to channel name)

read_channels_from_bytes_fallback(bita, info, channelSet=None, nrecords=None, dtype=None, channels_indexes=None)

reads stream of record bytes using bitarray in case no dataRead available

Parameters:

bita : stream

stream of bytes

info: info class

channelSet : set of str, optional

set of channel to read

nrecords: int

number of records

dtype: numpy dtype

channels_indexes: list of int

Returns:

rec : numpy recarray

contains a matrix of raw data in a recarray (attributes corresponding to channel name)

read_not_all_channels_sorted_record(fid, info, channelSet)

reads channels from file listed in channelSet

Parameters:

fid :

file identifier

info: info class

channelSet : set of str, optional

set of channel to read

Returns:

rec : numpy recarray

contains a matrix of raw data in a recarray (attributes corresponding to channel name)

recordID
recordIDCFormat
recordIDsize
recordLength
recordToChannelMatching
mdfreader.mdf4reader.textToTextConv(vect, cc_ref)

apply text to text conversion to data

Parameters:

vect : numpy 1D array

raw data to be converted to physical value

cc_ref : cc_ref from mdfinfo4.info4 conversion block (‘CCBlock’) dict

Returns:

converted data to physical value

mdfreader.mdf4reader.textToValueConv(vect, cc_val, cc_ref)

apply text to value conversion to data

Parameters:

vect : numpy 1D array

raw data to be converted to physical value

cc_val : cc_val from mdfinfo4.info4 conversion block (‘CCBlock’) dict

cc_ref : cc_ref from mdfinfo4.info4 conversion block (‘CCBlock’) dict

Returns:

converted data to physical value

mdfreader.mdf4reader.valueRangeToTextConv(vect, cc_val, cc_ref)

apply value range to text conversion to data

Parameters:

vect : numpy 1D array

raw data to be converted to physical value

cc_val : cc_val from mdfinfo4.info4 conversion block (‘CCBlock’) dict

cc_ref : cc_ref from mdfinfo4.info4 conversion block (‘CCBlock’) dict

Returns:

converted data to physical value

mdfreader.mdf4reader.valueRangeToValueTableConv(vect, cc_val)

apply value range to value table conversion to data

Parameters:

vect : numpy 1D array

raw data to be converted to physical value

cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict

Returns:

converted data to physical value

mdfreader.mdf4reader.valueToTextConv(vect, cc_val, cc_ref)

apply value to text conversion to data

Parameters:

vect : numpy 1D array

raw data to be converted to physical value

cc_val : cc_val from mdfinfo4.info4 conversion block (‘CCBlock’) dict

cc_ref : cc_ref from mdfinfo4.info4 conversion block (‘CCBlock’) dict

Returns:

converted data to physical value

mdfreader.mdf4reader.valueToValueTableWInterpConv(vect, cc_val)

apply value to value table with interpolation conversion to data

Parameters:

vect : numpy 1D array

raw data to be converted to physical value

cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict

Returns:

converted data to physical value

mdfreader.mdf4reader.valueToValueTableWOInterpConv(vect, cc_val)

apply value to value table without interpolation conversion to data

Parameters:

vect : numpy 1D array

raw data to be converted to physical value

cc_val : mdfinfo4.info4 conversion block (‘CCBlock’) dict

Returns:

converted data to physical value