ALPHA
This module defines a simple add-on system to extend ryvencore’s functionalities. Some default add-ons are provided in the addons.default package, and additional add-ons can be added and registered in the Session.
get_state()
and set_state()
)data
dict when it’s serializedself.get_addon('your_addon')
in your nodesAdd-on access is blocked during loading (deserialization), so nodes should not access any
add-ons during the execution of Node.__init__
or Node.set_data
.
This prevents inconsistent states. Nodes are loaded first, then the add-ons.
Therefore, the add-on should be sufficiently isolated and self-contained.
your_addons
for you addons or use ryvencore’s addon directoryYourAddon.py
in your_addons
YourAddon(ryvencore.AddOn)
that defines your add-on’s functionalityaddon = YourAddon()
at the end of the modulesession.register_addon_dir('path/to/your_addons')
See ryvencore.addons.default
for examples.
This file defines the Data
type, which must be used to pass data between nodes.
It should be subclassed to define custom data types. In particular, serialization
and deserialization must be implemented for each respective type. Types that are
pickle serializable by default can be used directly with Data(my_data)
.
Data
(value=None, load_from=None)[source]¶Bases: object
Base class for data objects.
Subclass this class and implement serialization and deserialization accordingly to send data to other nodes.
In case of large data sets being shared, you might want to leave serialization empty, which means the graph will not enter the same state when you reload it, which is fine as long as your nodes are built appropriately e.g. such that you can quickly regenerate that state by updating the root node.
Be careful when consuming complex input data: modification can lead to undesired
effects. In particular, if you share some data object D
with successor nodes
N1
and N2
, and N1
changes D
directly, then N2
will see the change as well, because they look at the same Data object:
>>> import ryvencore as rc
>>>
>>> class Producer(rc.Node):
... init_outputs = [rc.NodeOutputType()]
...
... def push_data(self, D):
... self.D = D
... self.update()
...
... def update_event(self, inp=-1):
... self.set_output_val(0, self.D)
>>>
>>> class Consumer(rc.Node):
... init_inputs = [rc.NodeInputType()]
...
... def update_event(self, inp=-1):
... p = self.input(0).payload
... p.append(4)
... print(p)
>>>
>>> def build_and_run(D):
... s = rc.Session()
... f = s.create_flow('main')
... producer = f.create_node(Producer)
... consumer1 = f.create_node(Consumer)
... consumer2 = f.create_node(Consumer)
... f.connect_nodes(producer.outputs[0], consumer1.inputs[0])
... f.connect_nodes(producer.outputs[0], consumer2.inputs[0])
... producer.push_data(D)
>>>
>>> build_and_run(rc.Data([1, 2, 3]))
[1, 2, 3, 4]
[1, 2, 3, 4, 4]
This can be useful for optimization when sharing large data, but might not
be what you want.
To avoid this you might want to make sure to copy D
when its payload is
consumed:
>>> class MyListData(rc.Data):
... @property
... def payload(self):
... return self._payload.copy()
>>>
>>> build_and_run(MyListData([1, 2, 3]))
[1, 2, 3, 4]
[1, 2, 3, 4]
payload
¶This module defines the abstract flow, managing node, edges, etc. Flow execution is implemented by FlowExecutor class.
A flow is a directed, usually but not necessarily acyclic multi-graph of nodes and edges (connections between nodes). The nodes are the computational units and the edges define the flow of data between them. The fundamental operations to perform on a flow are:
There are a couple of different modes / algorithms for executing a flow.
Data Flow
In the normal data flow mode, data is simply forward propagated on change. Specifically, this means the following:
A node output may have 0 or more outgoing connections/edges. When a node’s output value is updated, the new value is propagated to all connected nodes’ inputs. If there are multiple edges, the order of activation is undefined.
A node input may have 0 or 1 incoming connections/edges. When a node’s input receives new data, the node’s update event is invoked.
A flow execution is started once some node’s update event is invoked (either
by direct invocation through node.update()
, or by receiving input data), or
some node’s output value is updated.
A node can consume inputs and update outputs at any time.
Assumptions:
- no non-terminating feedback loops.
Data Flow with Optimization
Since the naive implementation of the above specification can be highly inefficient in some cases, a more advanced algorithm can be used. This algorithm ensures that, during a flow execution, each edge is updated at most once. It should implement the same semantics as the data flow algorithm, but with a slightly tightened assumption:
- no feedback loops / cycles in the graph
- nodes never modify their ports (inputs, outputs) during execution (update event)
The additional work required for this at the beginning of a flow execution is based on a DP algorithm running in \(\mathcal{O}(|V|+ |E|)\) time, where \(|V|\) is the number of nodes and \(|E|\) is the number of edges. However, when there are multiple consecutive executions without any subsequent changes to the graph, this work does not need to be repeated and execution is very fast.
Execution Flow
The special exec mode uses an additional type of connection (edge): the execution connection. While pure data flows are the more common use case, some applications call for a slightly different paradigm. You can think of the exec mode as e.g. UnrealEngine’s blueprint system.
In exec mode, calling node.exec_output(index)
has a similar effect as calling
node.set_output_val(index, val)
in data mode,
but without any data being propagated, so it’s just a trigger signal.
Pushing output data, however, does not cause updates in successor nodes.
When a node is updated (it received an update event through an exec connection), once it
needs input data (it calls self.input(index)
), if that input is connected to some
predecessor node P, then P receives an update event with inp=-1
, during which
it should push the output data.
Therefore, data is not forward propagated on change (node.set_output_val(index, value)
),
but generated on request (backwards,
node.input()
-> pred.update_event()
-> pred.set_output_val()
-> return).
The exec mode is still somewhat experimental, because the data mode is the far more common use case. It is not yet clear how to best implement the exec mode in a way that is both efficient and easy to use.
Assumptions:
- no non-terminating feedback loops with exec connections
Flow
(session, title)[source]¶Bases: Base
Manages all abstract flow components (nodes, edges, executors, etc.) and exposes methods for modification.
add_node
(node: Node)[source]¶Places the node object in the graph, Stores it, and causes the node’s
Node.place_event()
to be executed. Flow.create_node()
automatically
adds the node already, so no need to call this manually.
remove_node
(node: Node)[source]¶Removes a node from the flow without deleting it. Can be added again
with Flow.add_node()
.
check_connection_validity
(p1: NodeOutput, p2: NodeInput) bool [source]¶Checks whether a considered connect action is legal.
connect_nodes
(p1: NodeOutput, p2: NodeInput, silent: bool = False) Optional[Tuple[NodeOutput, NodeInput]] [source]¶Connects nodes or disconnects them if they are already connected.
TODO: change this; rather introduce disconnect_nodes()
instead
add_connection
(c: Tuple[NodeOutput, NodeInput], silent: bool = False)[source]¶Adds an edge between two node ports.
connected_inputs
(out: NodeOutput) List[NodeInput] [source]¶Returns a list of all connected inputs to the given output port.
connected_output
(inp: NodeInput) Optional[NodeOutput] [source]¶Returns the connected output port to the given input port, or
None
if it is not connected.
Node
(params)[source]¶Bases: Base
Base class for all node blueprints. Such a blueprint is made by subclassing this class and registering that subclass in the session. Actual node objects are instances of it. The node’s static properties are static attributes. Refer to python’s static class attributes behavior.
title
= ''¶the node’s title
tags
: List[str] = []¶a list of tag strings, often useful for searching etc.
version
: str = None¶version tag, use it!
init_inputs
: List[NodeInputType] = []¶list of node input types determining the initial inputs
init_outputs
: List[NodeOutputType] = []¶initial outputs list, see init_inputs
identifier
: str = None¶unique node identifier string. if not given it will set it to the class name when registering in the session
legacy_identifiers
: List[str] = []¶a list of compatible identifiers, useful if you change the class name (and hence the identifier) to provide backward compatibility to load old projects that rely on the old identifier
identifier_prefix
: str = None¶becomes part of the identifier if set; can be useful for grouping nodes
initialize
()[source]¶Called by the Flow. This method
It does not crash on exception when loading user_data, as this is not uncommon when developing nodes.
update
(inp=-1)[source]¶Activates the node, causing an update_event()
if block_updates
is not set.
For performance-, simplicity-, and maintainability-reasons activation is now
fully handed over to the operating FlowExecutor
, and not managed decentralized
in Node, NodePort, and Connection anymore.
input
(index: int) Optional[Data] [source]¶Returns the data residing at the data input of given index.
Do not call on exec inputs.
exec_output
(index: int)[source]¶Executes an exec output, causing activation of all connections.
Do not call on data outputs.
set_output_val
(index, data: Data)[source]¶Sets the value of a data output causing activation of all connections in data mode.
update_event
(inp=-1)[source]¶VIRTUAL
Gets called when an input received a signal or some node requested data of an output in exec mode. Implement this in your node class, this is the place where the main processing of your node should happen.
place_event
()[source]¶VIRTUAL
Called once the node object has been fully initialized and placed in the flow.
When loading content, place_event()
is executed before connections are built,
so updating output values here will not cause any other nodes to be updated during loading.
Notice that this method gets executed every time the node is added to the flow, which can happen multiple times for the same object (e.g. due to undo/redo operations).
remove_event
()[source]¶VIRTUAL
Called when the node is removed from the flow; useful for stopping threads and timers etc.
additional_data
() dict [source]¶VIRTUAL
additional_data()
/load_additional_data()
is almost equivalent to
get_state()
/set_state()
,
but it turned out to be useful for frontends to have their own dedicated version,
so get_state()
/set_state()
stays clean for all specific node subclasses.
load_additional_data
(data: dict)[source]¶VIRTUAL
For loading the data returned by additional_data()
.
get_state
() dict [source]¶VIRTUAL
If your node is stateful, implement this method for serialization. It should return a JSON compatible
dict that encodes your node’s state. The dict will be passed to set_state()
when the node is loaded.
set_state
(data: dict, version)[source]¶VIRTUAL
Opposite of get_state()
, reconstruct any custom internal state here.
create_input
(label: str = '', type_: str = 'data', add_data={}, insert: Optional[int] = None)[source]¶Creates and adds a new input at the end or index insert
if specified.
create_output
(label: str = '', type_: str = 'data', insert: Optional[int] = None)[source]¶Creates and adds a new output at the end or index insert
if specified.
Session
(gui: bool = False)[source]¶Bases: Base
The Session is the top level interface to your project. It mainly manages flows, nodes, and add-ons and provides methods for serialization and deserialization of the project.
load_addons
(location: str)[source]¶Loads all addons from the given location. location
can be an absolute path to any readable directory.
See ryvencore.AddOn
.
register_nodes
(node_classes: List)[source]¶Registers a list of Nodes which then become available in the flows. Do not attempt to place nodes in flows that haven’t been registered in the session before.
unregister_node
(node_class)[source]¶Unregisters a node which will then be removed from the available list. Existing instances won’t be affected.
create_flow
(title: Optional[str] = None, data: Optional[Dict] = None) Flow [source]¶Creates and returns a new flow. If data is provided the title parameter will be ignored.
rename_flow
(flow: Flow, title: str) bool [source]¶Renames an existing flow and returns success boolean.
flow_title_valid
(title: str) bool [source]¶Checks whether a considered title for a new flow is valid (unique) or not.
load
(data: Dict) List[Flow] [source]¶Loads a project and raises an exception if required nodes are missing (not registered).