cis_interface package¶
Subpackages¶
- cis_interface.communication package
- Subpackages
- Submodules
- cis_interface.communication.AsciiFileComm module
- cis_interface.communication.AsciiTableComm module
- cis_interface.communication.AsyncComm module
- cis_interface.communication.ClientComm module
- cis_interface.communication.CommBase module
- cis_interface.communication.ErrorComm module
- cis_interface.communication.FileComm module
- cis_interface.communication.IPCComm module
- cis_interface.communication.PandasFileComm module
- cis_interface.communication.PickleFileComm module
- cis_interface.communication.RMQAsyncComm module
- cis_interface.communication.RMQComm module
- cis_interface.communication.RPCComm module
- cis_interface.communication.ServerComm module
- cis_interface.communication.ZMQComm module
- Module contents
- Subpackages
- cis_interface.dataio package
- cis_interface.drivers package
- Subpackages
- Submodules
- cis_interface.drivers.AsciiFileInputDriver module
- cis_interface.drivers.AsciiFileOutputDriver module
- cis_interface.drivers.AsciiTableInputDriver module
- cis_interface.drivers.AsciiTableOutputDriver module
- cis_interface.drivers.CMakeModelDriver module
- cis_interface.drivers.ClientDriver module
- cis_interface.drivers.ClientRequestDriver module
- cis_interface.drivers.ClientResponseDriver module
- cis_interface.drivers.CommDriver module
- cis_interface.drivers.ConnectionDriver module
- cis_interface.drivers.Driver module
- cis_interface.drivers.FileInputDriver module
- cis_interface.drivers.FileOutputDriver module
- cis_interface.drivers.GCCModelDriver module
- cis_interface.drivers.IODriver module
- cis_interface.drivers.IPCCommDriver module
- cis_interface.drivers.InputCommDriver module
- cis_interface.drivers.InputDriver module
- cis_interface.drivers.MakeModelDriver module
- cis_interface.drivers.MatInputDriver module
- cis_interface.drivers.MatOutputDriver module
- cis_interface.drivers.MatlabModelDriver module
- cis_interface.drivers.ModelDriver module
- cis_interface.drivers.OutputCommDriver module
- cis_interface.drivers.OutputDriver module
- cis_interface.drivers.PandasFileInputDriver module
- cis_interface.drivers.PandasFileOutputDriver module
- cis_interface.drivers.PickleFileInputDriver module
- cis_interface.drivers.PickleFileOutputDriver module
- cis_interface.drivers.PythonModelDriver module
- cis_interface.drivers.RMQAsyncClientDriver module
- cis_interface.drivers.RMQAsyncCommDriver module
- cis_interface.drivers.RMQAsyncServerDriver module
- cis_interface.drivers.RMQClientDriver module
- cis_interface.drivers.RMQCommDriver module
- cis_interface.drivers.RMQInputDriver module
- cis_interface.drivers.RMQOutputDriver module
- cis_interface.drivers.RMQServerDriver module
- cis_interface.drivers.RPCCommDriver module
- cis_interface.drivers.RPCDriver module
- cis_interface.drivers.ServerDriver module
- cis_interface.drivers.ServerRequestDriver module
- cis_interface.drivers.ServerResponseDriver module
- cis_interface.drivers.ZMQCommDriver module
- Module contents
- Subpackages
- cis_interface.examples package
- cis_interface.interface package
- cis_interface.serialize package
Submodules¶
cis_interface.backwards module¶
This module allows for backward compatibility.
cis_interface.command_line module¶
cis_interface.config module¶
This module imports the configuration for cis_interface.
Todo
Remove reference to environment variables for accessing config options.
-
class
cis_interface.config.
CisConfigParser
(defaults=None, dict_type=<class 'collections.OrderedDict'>, allow_no_value=False, *, delimiters=('=', ':'), comment_prefixes=('#', ';'), inline_comment_prefixes=None, strict=True, empty_lines_in_values=True, default_section='DEFAULT', interpolation=<object object>, converters=<object object>)[source]¶ Bases:
configparser.ConfigParser
Config parser that returns None if option not provided on get.
-
cis_interface.config.
cfg_environment
(env=None, cfg=None)[source]¶ Set environment variables based on config options.
Parameters: - env (dict, optional) – Dictionary of environment variables that should be updated. Defaults to os.environ.
- cfg (
cis_interface.config.CisConfigParser
, optional) – Config parser with options that should be used to update the environment. Defaults tocis_interface.config.cis_cfg
.
-
cis_interface.config.
cfg_logging
(cfg=None)[source]¶ Set logging levels from config options.
Parameters: cfg ( cis_interface.config.CisConfigParser
, optional) – Config parser with options that should be used to update the environment. Defaults tocis_interface.config.cis_cfg
.
cis_interface.platform module¶
This module handle platform compatibility issues.
cis_interface.runner module¶
This module provides tools for running models using cis_interface.
-
class
cis_interface.runner.
CisRunner
(modelYmls, namespace, host=None, rank=0, cis_debug_level=None, rmq_debug_level=None, cis_debug_prefix=None)[source]¶ Bases:
cis_interface.tools.CisClass
This class handles the orchestration of starting the model and IO drivers, monitoring their progress, and cleaning up on exit.
Parameters: - modelYmls (list) – List of paths to yaml files specifying the models that should be run.
- namespace (str) – Name that should be used to uniquely identify any RMQ exchange.
- host (str, optional) – Name of the host that the models will be launched from. Defaults to None.
- rank (int, optional) – Rank of this set of models if run in parallel. Defaults to 0.
- cis_debug_level (str, optional) – Level for CiS debug messages. Defaults to environment variable ‘CIS_DEBUG’.
- rmq_debug_level (str, optional) – Level for RabbitMQ debug messages. Defaults to environment variable ‘RMQ_DEBUG’.
- cis_debug_prefix (str, optional) – Prefix for CiS debug messages. Defaults to namespace.
-
namespace
¶ str – Name that should be used to uniquely identify any RMQ exchange.
-
host
¶ str – Name of the host that the models will be launched from.
-
rank
¶ int – Rank of this set of models if run in parallel.
-
modeldrivers
¶ dict – Model drivers associated with this run.
-
inputdrivers
¶ dict – Input drivers associated with this run.
-
outputdrivers
¶ dict – Output drivers associated with this run.
-
serverdrivers
¶ dict – The addresses associated with different server drivers.
-
interrupt_time
¶ float – Time of last interrupt signal.
-
error_flag
¶ bool – True if one or more models raises an error.
..todo:: namespace, host, and rank do not seem strictly necessary.
-
all_drivers
¶ iterator – For all drivers.
-
closeChannels
(force_stop=False)[source]¶ Stop IO drivers and join the threads.
Parameters: force_stop (bool, optional) – If True, the terminate method is used to stop the drivers. Otherwise, the stop method is used. The stop method will try to exit gracefully while terminate will exit as quickly as possible. Defaults to False.
-
createDriver
(yml)[source]¶ Create a driver instance from the yaml information.
Parameters: yml (yaml) – Yaml object containing driver information. Returns: An instance of the specified driver. Return type: object
-
createInputDriver
(yml)[source]¶ Create an input driver instance from the yaml information.
Parameters: yml (yaml) – Yaml object containing driver information. Returns: An instance of the specified driver. Return type: object
-
createModelDriver
(yml)[source]¶ Create a model driver instance from the yaml information.
Parameters: yml (yaml) – Yaml object containing driver information. Returns: An instance of the specified driver. Return type: object
-
createOutputDriver
(yml)[source]¶ Create an output driver instance from the yaml information.
Parameters: yml (yaml) – Yaml object containing driver information. Returns: An instance of the specified driver. Return type: object
-
do_client_exits
(model)[source]¶ Perform exits for IO drivers associated with a client model.
Parameters: model (dict) – Dictionary of model parameters including any associated IO drivers.
-
do_model_exits
(model)[source]¶ Perform exits for IO drivers associated with a model.
Parameters: model (dict) – Dictionary of model parameters including any associated IO drivers.
-
io_drivers
(model=None)[source]¶ Return the input and output drivers for one or all models.
Parameters: model (str, optional) – Name of a model that I/O drivers should be returned for. Defaults to None and all I/O drivers are returned. Returns: Access to list of I/O drivers. Return type: iterator
-
loadDrivers
()[source]¶ Load all of the necessary drivers, doing the IO drivers first and adding IO driver environmental variables back tot he models.
-
cis_interface.runner.
get_runner
(models, **kwargs)[source]¶ Get runner for a set of models, getting run information from the environment.
Parameters: - models (list) – List of yaml files containing information on the models that should be run.
- **kwargs – Additonal keyword arguments are passed to CisRunner.
Returns: Runner for the provided models.
Return type: Raises: Exception
– If config option ‘namespace’ in ‘rmq’ section not set.
cis_interface.scanf module¶
Small scanf implementation.
Python has powerful regular expressions but sometimes they are totally overkill when you just want to parse a simple-formatted string. C programmers use the scanf-function for these tasks (see link below).
This implementation of scanf translates the simple scanf-format into regular expressions. Unlike C you can be sure that there are no buffer overflows possible.
- For more information see
- Original code from:
- http://code.activestate.com/recipes/502213-simple-scanf-implementation/
Modified original to make the %f more robust, as well as added %* modifier to skip fields.
Version: 1.3.3
- Releases:
- 1.0
- 2010-10-11
- Initial release
- 1.1
- 2010-10-13
- Changed regex from ‘match’ (only matches at beginning of line) to ‘search’ (matches anywhere in line)
- Bugfix - ignore cast for skipped fields
- 1.2
- 2013-05-30
- Added ‘collapseWhitespace’ flag (defaults to True) to take the search string and replace all whitespace with regex string to match repeated whitespace. This enables better matching in log files where the data has been formatted for easier reading. These cases have variable amounts of whitespace between the columns, depending on the number of characters in the data itself.
- 1.3
- 2016-01-18
- Add ‘extractdata’ function.
- 1.3.1
- 2016-06-23
- Release to PyPi, now including README.md
- Updates by Meagan Lang:
- 2018-04-12
- Added ability to parse components of field specifiers and cope with whitespace padding.
-
cis_interface.scanf.
scanf
(format, s=None, collapseWhitespace=True)[source]¶ - scanf supports the following formats:
- %c One character %5c 5 characters %d, %i int value %7d, %7i int value with length 7 %f float value %o octal value %X, %x hex value %s string terminated by whitespace
Examples: >>> scanf(“%s - %d errors, %d warnings”, “/usr/sbin/sendmail - 0 errors, 4 warnings”) (‘/usr/sbin/sendmail’, 0, 4) >>> scanf(“%o %x %d”, “0123 0x123 123”) (66, 291, 123)
If the parameter s is a file-like object, s.readline is called. If s is not specified, stdin is assumed.
The function returns a tuple of found values or None if the format does not match.
-
cis_interface.scanf.
scanf_compile
(format, collapseWhitespace=True)[source]¶ Translate the format into a regular expression
For example: >>> format_re, casts = _scanf_compile(‘%s - %d errors, %d warnings’) >>> print format_re.pattern (S+) - ([+-]?d+) errors, ([+-]?d+) warnings
Translated formats are cached for faster reuse
cis_interface.tools module¶
This modules offers various tools.
-
class
cis_interface.tools.
CisClass
(name, uuid=None, workingDir=None, timeout=60.0, sleeptime=0.01, **kwargs)[source]¶ Bases:
logging.LoggerAdapter
Base class for CiS classes.
Parameters: - name (str) – Class name.
- uuid (str, optional) – Unique ID for this instance. Defaults to None and is assigned.
- workingDir (str, optional) – Working directory. If not provided, the current working directory is used.
- timeout (float, optional) – Maximum time (in seconds) that should be spent waiting on a process. Defaults to 60.
- sleeptime (float, optional) – Time that class should sleep for when sleep is called. Defaults to 0.01.
- **kwargs – Additional keyword arguments are assigned to the extra_kwargs dictionary.
-
name
¶ str – Class name.
-
uuid
¶ str – Unique ID for this instance.
-
sleeptime
¶ float – Time that class should sleep for when sleep called.
-
longsleep
¶ float – Time that the class will sleep for when waiting for longer tasks to complete (10x longer than sleeptime).
-
timeout
¶ float – Maximum time that should be spent waiting on a process.
-
workingDir
¶ str – Working directory.
-
errors
¶ list – List of errors.
-
extra_kwargs
¶ dict – Keyword arguments that were not parsed.
-
sched_out
¶ obj – Output from the last scheduled task with output.
-
logger
¶ logging.Logger – Logger object for this object.
-
suppress_special_debug
¶ bool – If True, special_debug log messages are suppressed.
-
as_str
(obj)[source]¶ Return str version of object if it is not already a string.
Parameters: obj (object) – Object that should be turned into a string. Returns: String version of provided object. Return type: str
-
check_timeout
(key=None, key_level=0)[source]¶ Check timeout for the calling function/method.
Parameters: - key (str, optional) – Key for timeout that should be checked. Defaults to None and is set by the calling class and function/method (See timeout_key).
- key_level (int, optional) – Positive integer indicating the level of the calling class and function/method that should be used to key the timeout. 0 is the class and function/method that called start_timeout. Higher values use classes and function/methods further up in the stack. Defaults to 0.
Raises: KeyError
– If there is not a timeout registered for the specified key.
-
error
¶ Log an error level message.
-
exception
¶ Log an exception level message.
-
get_timeout_key
(key_level=0, key_suffix=None)[source]¶ Return a key for a given level in the stack, relative to the function calling get_timeout_key.
Parameters: - key_level (int, optional) – Positive integer indicating the level of the calling class and function/method that should be used to key the timeout. 0 is the class and function/method that is 2 steps higher in the stack. Higher values use classes and function/methods further up in the stack. Defaults to 0.
- key_suffix (str, optional) – String that should be appended to the end of the generated key. Defaults to None and is ignored.
Returns: Key identifying calling object and method.
Return type:
-
name
Name of the class object.
-
print_encoded
(msg, *args, **kwargs)[source]¶ Print bytes to stdout, encoding if possible.
Parameters:
-
raise_error
(e)[source]¶ Raise an exception, logging it first.
Parameters: e (Exception) – Exception to raise. Raises: The provided exception.
-
sched_task
(t, func, args=[], kwargs={}, store_output=False)[source]¶ Schedule a task that will be executed after a certain time has elapsed.
Parameters: - t (float) – Number of seconds that should be waited before task is executed.
- func (object) – Function that should be executed.
- args (list, optional) – Arguments for the provided function. Defaults to [].
- kwargs (dict, optional) – Keyword arguments for the provided function. Defaults to {}.
- store_output (bool, optional) – If True, the output from the scheduled task is stored in self.sched_out. Otherwise, it is not stored. Defaults to False.
-
sleep
(t=None)[source]¶ Have the class sleep for some period of time.
Parameters: t (float, optional) – Time that class should sleep for. If not provided, the attribute ‘sleeptime’ is used.
-
special_debug
¶ Log debug level message contingent of supression flag.
-
start_timeout
(t=None, key=None, key_level=0, key_suffix=None)[source]¶ Start a timeout for the calling function/method.
Parameters: - t (float, optional) – Maximum time that the calling function should wait before timeing out. If not provided, the attribute ‘timeout’ is used.
- key (str, optional) – Key that should be associated with the timeout that is created. Defaults to None and is set by the calling class and function/method (See get_timeout_key).
- key_level (int, optional) – Positive integer indicating the level of the calling class and function/method that should be used to key the timeout. 0 is the class and function/method that called start_timeout. Higher values use classes and function/methods further up in the stack. Defaults to 0.
- key_suffix (str, optional) – String that should be appended to the end of the generated key. Defaults to None and is ignored.
Raises: KeyError
– If the key already exists.
-
stop_timeout
(key=None, key_level=0, key_suffix=None, quiet=False)[source]¶ Stop a timeout for the calling function method.
Parameters: - key (str, optional) – Key for timeout that should be stopped. Defaults to None and is set by the calling class and function/method (See timeout_key).
- key_level (int, optional) – Positive integer indicating the level of the calling class and function/method that should be used to key the timeout. 0 is the class and function/method that called start_timeout. Higher values use classes and function/methods further up in the stack. Defaults to 0.
- key_suffix (str, optional) – String that should be appended to the end of the generated key. Defaults to None and is ignored.
- quiet (bool, optional) – If True, error message on timeout exceeded will be debug log. Defaults to False.
Raises: KeyError
– If there is not a timeout registered for the specified key.
-
timeout_key
¶ str – Key identifying calling object and method.
-
class
cis_interface.tools.
CisPopen
(cmd_args, forward_signals=True, **kwargs)[source]¶ Bases:
subprocess.Popen
Uses Popen to open a process without a buffer. If not already set, the keyword arguments ‘bufsize’, ‘stdout’, and ‘stderr’ are set to 0, subprocess.PIPE, and subprocess.STDOUT respectively. This sets the output stream to unbuffered and directs both stdout and stderr to the stdout pipe. In addition this class overrides Popen.kill() to allow processes to be killed with CTRL_BREAK_EVENT on windows.
Parameters: - args (list, str) – Shell command or list of arguments that should be run.
- forward_signals (bool, optional) – If True, flags will be set such that signals received by the spawning process will be forwarded to the child process. If False, the signals will not be forwarded. Defaults to True.
- **kwargs – Additional keywords arguments are passed to Popen.
-
class
cis_interface.tools.
CisThread
(name=None, target=None, args=(), kwargs=None, daemon=False, group=None, **cis_kwargs)[source]¶ Bases:
threading.Thread
,cis_interface.tools.CisClass
Thread for CiS that tracks when the thread is started and joined.
-
lock
¶ threading.RLock – Lock for accessing the sockets from multiple threads.
-
start_event
¶ threading.Event – Event indicating that the thread was started.
-
terminate_event
¶ threading.Event – Event indicating that the thread should be terminated. The target must exit when this is set.
-
main_terminated
¶ bool – True if the main thread has terminated.
-
terminate
(no_wait=False)[source]¶ Set the terminate event and wait for the thread to stop.
Parameters: no_wait (bool, optional) – If True, terminate will not block until the thread stops. Defaults to False and blocks. Raises: AssertionError
– If no_wait is False and the thread has not stopped after the timeout.
-
wait
(timeout=None, key=None)[source]¶ Wait until thread finish to return using sleeps rather than blocking.
Parameters:
-
was_started
¶ bool – True if the thread was started. False otherwise.
-
was_terminated
¶ bool – True if the thread was terminated. False otherwise.
-
-
class
cis_interface.tools.
CisThreadLoop
(*args, **kwargs)[source]¶ Bases:
cis_interface.tools.CisThread
Thread that will run a loop until the terminate event is called.
-
on_main_terminated
(dont_break=False)[source]¶ Actions performed when 1st main terminated.
Parameters: dont_break (bool, optional) – If True, the break flag won’t be set. Defaults to False.
-
was_break
¶ bool – True if the break flag was set.
-
-
class
cis_interface.tools.
TimeOut
(max_time, key=None)[source]¶ Bases:
object
Class for checking if a period of time has been elapsed.
Parameters: -
max_time
¶ float – Maximum period of time that should elapsed before ‘is_out’ returns True.
-
start_time
¶ float – Result of time.time() at start.
-
key
¶ str – Key that was used to register the timeout.
-
elapsed
¶ float – Total time that has elapsed since the start.
-
is_out
¶ bool – True if there is not any time remaining. False otherwise.
-
-
cis_interface.tools.
eval_kwarg
(x)[source]¶ If x is a string, eval it. Otherwise just return it.
Parameters: x (str, obj) – String to be evaluated as an object or an object. Returns: Result of evaluated string or the input object. Return type: obj
-
cis_interface.tools.
get_default_comm
()[source]¶ Get the default comm that should be used for message passing.
-
cis_interface.tools.
is_ipc_installed
()[source]¶ Determine if the IPC libraries are installed.
Returns: True if the IPC libraries are installed, False otherwise. Return type: bool
-
cis_interface.tools.
is_zmq_installed
(check_c=True)[source]¶ Determine if the libczmq & libzmq libraries are installed.
Returns: True if both libraries are installed, False otherwise. Return type: bool
-
cis_interface.tools.
locate_path
(fname, basedir='/')[source]¶ Find the full path to a file using where on Windows.
-
cis_interface.tools.
popen_nobuffer
(*args, **kwargs)[source]¶ Uses Popen to open a process without a buffer. If not already set, the keyword arguments ‘bufsize’, ‘stdout’, and ‘stderr’ are set to 0, subprocess.PIPE, and subprocess.STDOUT respectively. This sets the output stream to unbuffered and directs both stdout and stderr to the stdout pipe.
Parameters: - args (list, str) – Shell command or list of arguments that should be run.
- forward_signals (bool, optional) – If True, flags will be set such that signals received by the spawning process will be forwarded to the child process. If False, the signals will not be forwarded. Defaults to True.
- **kwargs – Additional keywords arguments are passed to Popen.
Returns: Process that was started.
Return type:
cis_interface.units module¶
cis_interface.yamlfile module¶
-
cis_interface.yamlfile.
load_yaml
(fname)[source]¶ Parse a yaml file defining a run.
Parameters: fname (str) – Path to the yaml file. Returns: Contents of yaml file. Return type: dict
-
cis_interface.yamlfile.
parse_component
(yml, ctype, yamldir, existing=None)[source]¶ Parse a yaml entry for a component, adding it to the list of existing components.
Parameters: - yml (dict) – YAML dictionary for a component.
- ctype (str) – Component type. This can be ‘input’, ‘output’, ‘model’, or ‘connection’.
- yamldir (str) – Full path to directory containing the yaml this component was in.
- existing (dict, optional) – Dictionary of existing components. Defaults to empty dict.
Raises: TypeError
– If yml is not a dictionary.ValueError
– If dtype is not ‘input’, ‘output’, ‘model’, or ‘connection’.RuntimeError
– If the yml dictionary is missing a required keyword.ValueError
– If the component already exists.RuntimeError
– If ‘kwargs’ is an entry in the yml.
Returns: All components identified.
Return type:
-
cis_interface.yamlfile.
parse_connection
(yml, yamldir, existing)[source]¶ Parse a yaml entry for a connection between I/O channels.
Parameters: Raises: RuntimeError
– If the yml dictionary is missing a required keyword.AssertionError
– If the ‘input’ or ‘output’ entry is not a string.RuntimeError
– If the ‘input’ entry is not a model output or file.RuntimeError
– If neither the ‘input’ or ‘output’ entries correspond to model I/O channels.ValueError
– If the ‘input’ is a file and ‘read_meth’ entry is not ‘all’, ‘line’, ‘table’, or ‘table_array’.ValueError
– If the ‘output’ is a file and ‘write_meth’ entry is not ‘all’, ‘line’, ‘table’, or ‘table_array’.
Returns: Updated log of all entries.
Return type:
-
cis_interface.yamlfile.
parse_io
(yml, yamldir, existing)[source]¶ Parse a yaml entry for an I/O channel.
Parameters: Raises: RuntimeError
– If the yml dictionary is missing a required keyword.Returns: Updated log of all entries.
Return type:
-
cis_interface.yamlfile.
parse_model
(yml, yamldir, existing)[source]¶ Parse a yaml entry for a model.
Parameters: Raises: RuntimeError
– If the yml dictionary is missing a required keyword.Returns: Updated log of all entries.
Return type:
-
cis_interface.yamlfile.
parse_yaml
(files)[source]¶ Parse list of yaml files.
Parameters: files (str, list) – Either the path to a single yaml file or a list of yaml files. Raises: RuntimeError
– If one of the I/O channels is not initialized with driver information.Returns: Dictionary of information parsed from the yamls. Return type: dict
Module contents¶
This package provides a framework for integrating models across languages such that they can be run simultaneously, passing input back and forth.