Formatted I/O¶
In addition to passing raw strings, the cis_interface framework also has support for formatting/processing messages from/to language native objects.
Scalars¶
The format of messages containing scalar variables (strings, integers, and
floating point numbers) can be specified using a C-style format string
(See C-Style Format Strings for details). In
the example below, the format string "%6s\t%d\t%f\n"
indicates that each
message will contain a 6 character string, an integer, and a float. In addition
to providing a format string, the C API requires the use of different functions
for initializing channels and sending/receiving to/from them.
Model Code:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 | # Import classes for input/output channels
from cis_interface.interface.CisInterface import CisInput, CisOutput
# Initialize input/output channels
in_channel = CisInput('inputA', '%6s\t%d\t%f\n')
out_channel = CisOutput('outputA', '%6s\t%d\t%f\n')
# Loop until there is no longer input or the queues are closed
while True:
# Receive input from input channel
# If there is an error, the flag will be False
flag, msg = in_channel.recv()
if not flag:
print("Model A: No more input.")
break
name, count, size = msg
# Print received message
print('Model A: %s, %d, %f' % (name, count, size))
# Send output to output channel
# If there is an error, the flag will be False
flag = out_channel.send(name, count, size)
if not flag:
print("Model A: Error sending output.")
break
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 | # Import classes for input/output channels
from cis_interface.interface.CisInterface import CisInput, CisOutput
# Initialize input/output channels
in_channel = CisInput('inputB', '%6s\t%d\t%f\n')
out_channel = CisOutput('outputB', '%6s\t%d\t%f\n')
# Loop until there is no longer input or the queues are closed
while True:
# Receive input from input channel
# If there is an error, the flag will be False
flag, msg = in_channel.recv()
if not flag:
print("Model B: No more input.")
break
name, count, size = msg
# Print received message
print('Model B: %s, %d, %f' % (name, count, size))
# Send output to output channel
# If there is an error, the flag will be False
flag = out_channel.send(name, count, size)
if not flag:
print("Model B: Error sending output.")
break
|
The same YAML can be used as was used for the example from
Getting started with the modification that
the files are now read/written line-by-line (read_meth: line
and
write_meth: line
) rather than all at once:
Model YAML:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | models:
- name: python_modelA
driver: PythonModelDriver
args: ./src/formatted_io1_modelA.py
inputs: inputA
outputs: outputA
- name: python_modelB
driver: PythonModelDriver
args: ./src/formatted_io1_modelB.py
inputs: inputB
outputs: outputB
connections:
- input: outputA # Connection between model A output & model B input
output: inputB
- input: ./Input/input.txt # Connection between file and model A input
output: inputA
read_meth: line
- input: outputB # Connection between model B output and file
output: ./output.txt
write_meth: line
|
Tables by Row¶
Tables can also be passed in a similar manner. For input from a table, the format
string does not need to be provided and will be determined by the source of the
table. There are different API classes/functions for I/O from/to table channels
versus standard channels in each language. (e.g. CisInput
vs.
CisAsciiTableInput
in Python)
Model Code:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | # Import classes for input/output channels
from cis_interface.interface.CisInterface import (
CisAsciiTableInput, CisAsciiTableOutput)
# Initialize input/output channels
in_channel = CisAsciiTableInput('inputA')
out_channel = CisAsciiTableOutput('outputA', '%6s\t%d\t%f\n')
# Loop until there is no longer input or the queues are closed
while True:
# Receive input from input channel
# If there is an error, the flag will be False
flag, msg = in_channel.recv()
if not flag:
print("Model A: No more input.")
break
name, count, size = msg
# Print received message
print('Model A: %s, %d, %f' % (name, count, size))
# Send output to output channel
# If there is an error, the flag will be False
flag = out_channel.send(name, count, size)
if not flag:
print("Model A: Error sending output.")
break
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | # Import classes for input/output channels
from cis_interface.interface.CisInterface import (
CisAsciiTableInput, CisAsciiTableOutput)
# Initialize input/output channels
in_channel = CisAsciiTableInput('inputB')
out_channel = CisAsciiTableOutput('outputB', '%6s\t%d\t%f\n')
# Loop until there is no longer input or the queues are closed
while True:
# Receive input from input channel
# If there is an error, the flag will be False
flag, msg = in_channel.recv()
if not flag:
print("Model B: No more input.")
break
name, count, size = msg
# Print received message
print('Model B: %s, %d, %f' % (name, count, size))
# Send output to output channel
# If there is an error, the flag will be False
flag = out_channel.send(name, count, size)
if not flag:
print("Model B: Error sending output.")
break
|
The read_meth: table
and write_meth: table
options in the YAML, tell the
cis_interface framework that the file should be read/written as a table
row-by-row including verification that each row conforms with the table.
Model YAML:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | models:
- name: python_modelA
driver: PythonModelDriver
args: ./src/formatted_io2_modelA.py
inputs: inputA
outputs: outputA
- name: python_modelB
driver: PythonModelDriver
args: ./src/formatted_io2_modelB.py
inputs: inputB
outputs: outputB
connections:
- input: outputA # Connection between model A output & model B input
output: inputB
- input: ./Input/input.txt # Connection between file and model A input
output: inputA
read_meth: table
- input: outputB # Connection between model B output and file
output: ./output.txt
write_meth: table
|
Tables as Arrays¶
Tables can also be passed as arrays. In Python and Matlab, this is done using
separate classes/functions. In C and C++, this is done by passing 1 to the as_array
arguments of the table API classes/functions.
Model Code:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | # Import classes for input/output channels
from cis_interface.interface.CisInterface import (
CisAsciiArrayInput, CisAsciiArrayOutput)
# Initialize input/output channels
in_channel = CisAsciiArrayInput('inputA')
out_channel = CisAsciiArrayOutput('outputA', '%6s\t%d\t%f\n')
# Loop until there is no longer input or the queues are closed
while True:
# Receive input from input channel
# If there is an error, the flag will be False
flag, arr = in_channel.recv()
if not flag:
print("Model A: No more input.")
break
# Print received message
print('Model A: (%d rows)' % len(arr))
for i in range(len(arr)):
print(' %s, %d, %f' % tuple(arr[i]))
# Send output to output channel
# If there is an error, the flag will be False
flag = out_channel.send(arr)
if not flag:
print("Model A: Error sending output.")
break
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | # Import classes for input/output channels
from cis_interface.interface.CisInterface import (
CisAsciiArrayInput, CisAsciiArrayOutput)
# Initialize input/output channels
in_channel = CisAsciiArrayInput('inputB')
out_channel = CisAsciiArrayOutput('outputB', '%6s\t%d\t%f\n')
# Loop until there is no longer input or the queues are closed
while True:
# Receive input from input channel
# If there is an error, the flag will be False
flag, arr = in_channel.recv()
if not flag:
print("Model B: No more input.")
break
# Print received message
print('Model B: (%d rows)' % len(arr))
for i in range(len(arr)):
print(' %s, %d, %f' % tuple(arr[i]))
# Send output to output channel
# If there is an error, the flag will be False
flag = out_channel.send(arr)
if not flag:
print("Model B: Error sending output.")
break
|
The YAML only differs in that read_meth: table_array
and
write_meth: table_array
for the connections to files to indicate that the files
should be interpreted as tables and that the tables should be read/written in their
entirety as arrays.
Model YAML:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | models:
- name: python_modelA
driver: PythonModelDriver
args: ./src/formatted_io3_modelA.py
inputs: inputA
outputs:
name: outputA
field_names: name,count,size
- name: python_modelB
driver: PythonModelDriver
args: ./src/formatted_io3_modelB.py
inputs: inputB
outputs:
name: outputB
field_names: name,count,size
connections:
- input: outputA # Connection between model A output & model B input
output: inputB
- input: ./Input/input.txt # Connection between file and model A input
output: inputA
read_meth: table_array
- input: outputB # Connection between model B output and file
output: ./output.txt
write_meth: table_array
|
Tables as Pandas Data Frames¶
In Python, tables can also be passed as Pandas data frames.
Model Code:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | # Import classes for input/output channels
from cis_interface.interface.CisInterface import (
CisPandasInput, CisPandasOutput)
# Initialize input/output channels
in_channel = CisPandasInput('inputA')
out_channel = CisPandasOutput('outputA')
# Loop until there is no longer input or the queues are closed
while True:
# Receive input from input channel
# If there is an error, the flag will be False
flag, frame = in_channel.recv()
if not flag:
print("Model A: No more input.")
break
# Print received message
nrows = len(frame.index)
print('Model A: (%d rows)' % len(frame.index))
print(frame)
# Send output to output channel
# If there is an error, the flag will be False
flag = out_channel.send(frame)
if not flag:
print("Model A: Error sending output.")
break
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | # Import classes for input/output channels
from cis_interface.interface.CisInterface import (
CisPandasInput, CisPandasOutput)
# Initialize input/output channels
in_channel = CisPandasInput('inputB')
out_channel = CisPandasOutput('outputB')
# Loop until there is no longer input or the queues are closed
while True:
# Receive input from input channel
# If there is an error, the flag will be False
flag, frame = in_channel.recv()
if not flag:
print("Model B: No more input.")
break
# Print received message
nrows = len(frame.index)
print('Model B: (%d rows)' % nrows)
print(frame)
# Send output to output channel
# If there is an error, the flag will be False
flag = out_channel.send(frame)
if not flag:
print("Model B: Error sending output.")
break
|
The YAML specifies read_meth: pandas
and write_meth: pandas
for the
connections to files to indicate that the files should be interpreted as CSV
tables using Pandas.
Model YAML:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | models:
- name: python_modelA
driver: PythonModelDriver
args: ./src/formatted_io4_modelA.py
inputs: inputA
outputs:
name: outputA
field_names: name,count,size
- name: python_modelB
driver: PythonModelDriver
args: ./src/formatted_io4_modelB.py
inputs: inputB
outputs:
name: outputB
field_names: name,count,size
connections:
- input: outputA # Connection between model A output & model B input
output: inputB
- input: ./Input/input.txt # Connection between file and model A input
output: inputA
read_meth: pandas
- input: outputB # Connection between model B output and file
output: ./output.txt
write_meth: pandas
|
As Pandas data frames are a Python specific construction, they cannot be used within models written in other languages. However, the files can still be read using Pandas. The data format returned to models on the receiving end of sent Pandas data frames will receive arrays in the proper native data type. In addition, a model written in Python can receive any array sent by another model (whether or not it is Python) as a Pandas data frame.