Formatted I/O

In addition to passing raw strings, the cis_interface framework also has support for formatting/processing messages from/to language native objects.

Scalars

The format of messages containing scalar variables (strings, integers, and floating point numbers) can be specified using a C-style format string (See C-Style Format Strings for details). In the example below, the format string "%6s\t%d\t%f\n" indicates that each message will contain a 6 character string, an integer, and a float. In addition to providing a format string, the C API requires the use of different functions for initializing channels and sending/receiving to/from them.

Model Code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# Import classes for input/output channels
from cis_interface.interface.CisInterface import CisInput, CisOutput

# Initialize input/output channels
in_channel = CisInput('inputA', '%6s\t%d\t%f\n')
out_channel = CisOutput('outputA', '%6s\t%d\t%f\n')

# Loop until there is no longer input or the queues are closed
while True:

    # Receive input from input channel
    # If there is an error, the flag will be False
    flag, msg = in_channel.recv()
    if not flag:
        print("Model A: No more input.")
        break
    name, count, size = msg

    # Print received message
    print('Model A: %s, %d, %f' % (name, count, size))

    # Send output to output channel
    # If there is an error, the flag will be False
    flag = out_channel.send(name, count, size)
    if not flag:
        print("Model A: Error sending output.")
        break
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# Import classes for input/output channels
from cis_interface.interface.CisInterface import CisInput, CisOutput

# Initialize input/output channels
in_channel = CisInput('inputB', '%6s\t%d\t%f\n')
out_channel = CisOutput('outputB', '%6s\t%d\t%f\n')

# Loop until there is no longer input or the queues are closed
while True:

    # Receive input from input channel
    # If there is an error, the flag will be False
    flag, msg = in_channel.recv()
    if not flag:
        print("Model B: No more input.")
        break
    name, count, size = msg

    # Print received message
    print('Model B: %s, %d, %f' % (name, count, size))

    # Send output to output channel
    # If there is an error, the flag will be False
    flag = out_channel.send(name, count, size)
    if not flag:
        print("Model B: Error sending output.")
        break

(Example in other languages)

The same YAML can be used as was used for the example from Getting started with the modification that the files are now read/written line-by-line (read_meth: line and write_meth: line) rather than all at once:

Model YAML:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
models:
  - name: python_modelA
    driver: PythonModelDriver
    args: ./src/formatted_io1_modelA.py
    inputs: inputA
    outputs: outputA

  - name: python_modelB
    driver: PythonModelDriver
    args: ./src/formatted_io1_modelB.py
    inputs: inputB
    outputs: outputB

connections:
  - input: outputA  # Connection between model A output & model B input
    output: inputB
  - input: ./Input/input.txt  # Connection between file and model A input
    output: inputA
    read_meth: line
  - input: outputB  # Connection between model B output and file
    output: ./output.txt
    write_meth: line

(Example in other languages)

Tables by Row

Tables can also be passed in a similar manner. For input from a table, the format string does not need to be provided and will be determined by the source of the table. There are different API classes/functions for I/O from/to table channels versus standard channels in each language. (e.g. CisInput vs. CisAsciiTableInput in Python)

Model Code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# Import classes for input/output channels
from cis_interface.interface.CisInterface import (
    CisAsciiTableInput, CisAsciiTableOutput)

# Initialize input/output channels
in_channel = CisAsciiTableInput('inputA')
out_channel = CisAsciiTableOutput('outputA', '%6s\t%d\t%f\n')

# Loop until there is no longer input or the queues are closed
while True:

    # Receive input from input channel
    # If there is an error, the flag will be False
    flag, msg = in_channel.recv()
    if not flag:
        print("Model A: No more input.")
        break
    name, count, size = msg

    # Print received message
    print('Model A: %s, %d, %f' % (name, count, size))

    # Send output to output channel
    # If there is an error, the flag will be False
    flag = out_channel.send(name, count, size)
    if not flag:
        print("Model A: Error sending output.")
        break
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# Import classes for input/output channels
from cis_interface.interface.CisInterface import (
    CisAsciiTableInput, CisAsciiTableOutput)

# Initialize input/output channels
in_channel = CisAsciiTableInput('inputB')
out_channel = CisAsciiTableOutput('outputB', '%6s\t%d\t%f\n')

# Loop until there is no longer input or the queues are closed
while True:

    # Receive input from input channel
    # If there is an error, the flag will be False
    flag, msg = in_channel.recv()
    if not flag:
        print("Model B: No more input.")
        break
    name, count, size = msg

    # Print received message
    print('Model B: %s, %d, %f' % (name, count, size))

    # Send output to output channel
    # If there is an error, the flag will be False
    flag = out_channel.send(name, count, size)
    if not flag:
        print("Model B: Error sending output.")
        break

(Example in other languages)

The read_meth: table and write_meth: table options in the YAML, tell the cis_interface framework that the file should be read/written as a table row-by-row including verification that each row conforms with the table.

Model YAML:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
models:
  - name: python_modelA
    driver: PythonModelDriver
    args: ./src/formatted_io2_modelA.py
    inputs: inputA
    outputs: outputA

  - name: python_modelB
    driver: PythonModelDriver
    args: ./src/formatted_io2_modelB.py
    inputs: inputB
    outputs: outputB

connections:
  - input: outputA  # Connection between model A output & model B input
    output: inputB
  - input: ./Input/input.txt  # Connection between file and model A input
    output: inputA
    read_meth: table
  - input: outputB  # Connection between model B output and file
    output: ./output.txt
    write_meth: table

(Example in other languages)

Tables as Arrays

Tables can also be passed as arrays. In Python and Matlab, this is done using separate classes/functions. In C and C++, this is done by passing 1 to the as_array arguments of the table API classes/functions.

Model Code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# Import classes for input/output channels
from cis_interface.interface.CisInterface import (
    CisAsciiArrayInput, CisAsciiArrayOutput)

# Initialize input/output channels
in_channel = CisAsciiArrayInput('inputA')
out_channel = CisAsciiArrayOutput('outputA', '%6s\t%d\t%f\n')

# Loop until there is no longer input or the queues are closed
while True:

    # Receive input from input channel
    # If there is an error, the flag will be False
    flag, arr = in_channel.recv()
    if not flag:
        print("Model A: No more input.")
        break

    # Print received message
    print('Model A: (%d rows)' % len(arr))
    for i in range(len(arr)):
        print('   %s, %d, %f' % tuple(arr[i]))

    # Send output to output channel
    # If there is an error, the flag will be False
    flag = out_channel.send(arr)
    if not flag:
        print("Model A: Error sending output.")
        break
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# Import classes for input/output channels
from cis_interface.interface.CisInterface import (
    CisAsciiArrayInput, CisAsciiArrayOutput)

# Initialize input/output channels
in_channel = CisAsciiArrayInput('inputB')
out_channel = CisAsciiArrayOutput('outputB', '%6s\t%d\t%f\n')

# Loop until there is no longer input or the queues are closed
while True:

    # Receive input from input channel
    # If there is an error, the flag will be False
    flag, arr = in_channel.recv()
    if not flag:
        print("Model B: No more input.")
        break

    # Print received message
    print('Model B: (%d rows)' % len(arr))
    for i in range(len(arr)):
        print('   %s, %d, %f' % tuple(arr[i]))

    # Send output to output channel
    # If there is an error, the flag will be False
    flag = out_channel.send(arr)
    if not flag:
        print("Model B: Error sending output.")
        break

(Example in other languages)

The YAML only differs in that read_meth: table_array and write_meth: table_array for the connections to files to indicate that the files should be interpreted as tables and that the tables should be read/written in their entirety as arrays.

Model YAML:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
models:
  - name: python_modelA
    driver: PythonModelDriver
    args: ./src/formatted_io3_modelA.py
    inputs: inputA
    outputs:
      name: outputA
      field_names: name,count,size

  - name: python_modelB
    driver: PythonModelDriver
    args: ./src/formatted_io3_modelB.py
    inputs: inputB
    outputs:
      name: outputB
      field_names: name,count,size

connections:
  - input: outputA  # Connection between model A output & model B input
    output: inputB
  - input: ./Input/input.txt  # Connection between file and model A input
    output: inputA
    read_meth: table_array
  - input: outputB  # Connection between model B output and file
    output: ./output.txt
    write_meth: table_array

(Example in other languages)

Tables as Pandas Data Frames

In Python, tables can also be passed as Pandas data frames.

Model Code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# Import classes for input/output channels
from cis_interface.interface.CisInterface import (
    CisPandasInput, CisPandasOutput)

# Initialize input/output channels
in_channel = CisPandasInput('inputA')
out_channel = CisPandasOutput('outputA')

# Loop until there is no longer input or the queues are closed
while True:

    # Receive input from input channel
    # If there is an error, the flag will be False
    flag, frame = in_channel.recv()
    if not flag:
        print("Model A: No more input.")
        break

    # Print received message
    nrows = len(frame.index)
    print('Model A: (%d rows)' % len(frame.index))
    print(frame)

    # Send output to output channel
    # If there is an error, the flag will be False
    flag = out_channel.send(frame)
    if not flag:
        print("Model A: Error sending output.")
        break
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# Import classes for input/output channels
from cis_interface.interface.CisInterface import (
    CisPandasInput, CisPandasOutput)

# Initialize input/output channels
in_channel = CisPandasInput('inputB')
out_channel = CisPandasOutput('outputB')

# Loop until there is no longer input or the queues are closed
while True:

    # Receive input from input channel
    # If there is an error, the flag will be False
    flag, frame = in_channel.recv()
    if not flag:
        print("Model B: No more input.")
        break

    # Print received message
    nrows = len(frame.index)
    print('Model B: (%d rows)' % nrows)
    print(frame)

    # Send output to output channel
    # If there is an error, the flag will be False
    flag = out_channel.send(frame)
    if not flag:
        print("Model B: Error sending output.")
        break

(Example in other languages)

The YAML specifies read_meth: pandas and write_meth: pandas for the connections to files to indicate that the files should be interpreted as CSV tables using Pandas.

Model YAML:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
models:
  - name: python_modelA
    driver: PythonModelDriver
    args: ./src/formatted_io4_modelA.py
    inputs: inputA
    outputs:
      name: outputA
      field_names: name,count,size

  - name: python_modelB
    driver: PythonModelDriver
    args: ./src/formatted_io4_modelB.py
    inputs: inputB
    outputs:
      name: outputB
      field_names: name,count,size

connections:
  - input: outputA  # Connection between model A output & model B input
    output: inputB
  - input: ./Input/input.txt  # Connection between file and model A input
    output: inputA
    read_meth: pandas
  - input: outputB  # Connection between model B output and file
    output: ./output.txt
    write_meth: pandas

(Example in other languages)

As Pandas data frames are a Python specific construction, they cannot be used within models written in other languages. However, the files can still be read using Pandas. The data format returned to models on the receiving end of sent Pandas data frames will receive arrays in the proper native data type. In addition, a model written in Python can receive any array sent by another model (whether or not it is Python) as a Pandas data frame.