fireworks.user_objects.firetasks package¶
Subpackages¶
- fireworks.user_objects.firetasks.tests package
- Submodules
- fireworks.user_objects.firetasks.tests.test_dataflow_tasks module
- fireworks.user_objects.firetasks.tests.test_fileio_tasks module
- fireworks.user_objects.firetasks.tests.test_filepad_tasks module
- fireworks.user_objects.firetasks.tests.test_script_task module
- fireworks.user_objects.firetasks.tests.test_templatewriter_task module
- Module contents
Submodules¶
fireworks.user_objects.firetasks.dataflow_tasks module¶
This module includes dataflow firetask tasks
-
class
fireworks.user_objects.firetasks.dataflow_tasks.
CommandLineTask
(*args, **kwargs)¶ Bases:
fireworks.core.firework.FireTaskBase
A Firetask to execute external commands in a shell
- Required params:
- command_spec (dict): a dictionary specification of the command (see below for details)
- Optional params:
- inputs ([str]): list of labels, one for each input argument
- outputs ([str]): list of labels, one for each output argument
- chunk_number (int): the serial number of the firetask when it is part of a series generated by a ForeachTask
- command_spec = {
- ‘command’: [str], # mandatory label_1: dict_1, # optional label_2: dict_2, # optional …
} The ‘command’ is a representation of the command as to be used with subprocess package. The optional keys label_1, label_2, etc. are the actual labels used in the inputs and outputs. The dictionaries dict_1, dict_2, etc. have the following schema: {
- ‘binding’: {
- prefix: str or None, separator: str or None
}, ‘source’: {
- ‘type’: ‘path’ or ‘data’ or ‘identifier’
- or ‘stdin’ or ‘stdout’ or ‘stderr’ or None,
‘value’: str or int or float
}, ‘target’: {
- ‘type’: ‘path’ or ‘data’ or ‘identifier’
- or ‘stdin’ or ‘stdout’ or ‘stderr’ or None,
‘value’: str
}
}
- If the ‘type’ in the ‘source’ field is ‘data’ the ‘value’ can be of
types ‘str’, ‘int’ and ‘float’.
- When a str is found instead of dict for some ‘source’, for example
‘source’: ‘string’, ‘string’ is replaced with spec[‘string’] which must be available and of the schema of the ‘source’.
- When a str is found instead of dict for some label, for example
label: ‘string’, ‘string’ is replaced with spec[‘string’] which can be a dictionary with this schema or a list of such dictionaries.
-
command_line_tool
(command, inputs=None, outputs=None)¶ This function composes and executes a command from provided specifications.
- Required parameters:
- command ([str]): the command as to be passed to subprocess.Popen
- Optional parameters:
- inputs ([dict, [dict]]): list of the specifications for inputs; multiple inputs may be passed in one list of dictionaries
- outputs ([dict]): list of the specifications for outputs
Returns: - ‘target’: {
- ‘type’: ‘path’ or ‘data’ or ‘identifier’
- or ‘stdin’ or ‘stdout’ or ‘stderr’ or None
’value’: str
}
If outputs is None then an empty list is returned.
Return type: - list of target dictionaries for each output
-
optional_params
= ['inputs', 'outputs', 'chunk_number']¶
-
required_params
= ['command_spec']¶
-
run_task
(fw_spec)¶
-
class
fireworks.user_objects.firetasks.dataflow_tasks.
ForeachTask
(*args, **kwargs)¶ Bases:
fireworks.core.firework.FireTaskBase
This firetask branches the workflow creating parallel fireworks using FWAction: one firework for each element or each chunk from the split list. Each firework in this generated list contains the firetask specified in the task dictionary. If the number of chunks is specified the split list will be divided into this number of chunks and each chunk will be processed by one of the generated child fireworks.
- Required params:
- task (dict): a dictionary version of the firetask
- split (str): a label of an input list; it must be available both in the inputs list of the specified task and in the spec.
- Optional params:
- number of chunks (int): if provided the split input list will be divided into this number of sublists and each will be processed by a separate child firework
-
optional_params
= ['number of chunks']¶
-
required_params
= ['task', 'split']¶
-
run_task
(fw_spec)¶
-
class
fireworks.user_objects.firetasks.dataflow_tasks.
ImportDataTask
(*args, **kwargs)¶ Bases:
fireworks.core.firework.FireTaskBase
Update the spec with data from file in a nested dictionary at a position specified by a mapstring = maplist[0]/maplist[1]/… i.e. spec[maplist[0]][maplist[1]]… = data
-
optional_params
= []¶
-
required_params
= ['filename', 'mapstring']¶
-
run_task
(fw_spec)¶
-
-
class
fireworks.user_objects.firetasks.dataflow_tasks.
JoinDictTask
(*args, **kwargs)¶ Bases:
fireworks.core.firework.FireTaskBase
combines specified spec fields into a dictionary
-
optional_params
= ['rename']¶
-
required_params
= ['inputs', 'output']¶
-
run_task
(fw_spec)¶
-
-
class
fireworks.user_objects.firetasks.dataflow_tasks.
JoinListTask
(*args, **kwargs)¶ Bases:
fireworks.core.firework.FireTaskBase
combines specified spec fields into a list.
-
required_params
= ['inputs', 'output']¶
-
run_task
(fw_spec)¶
-
fireworks.user_objects.firetasks.fileio_tasks module¶
-
class
fireworks.user_objects.firetasks.fileio_tasks.
ArchiveDirTask
(*args, **kwargs)¶ Bases:
fireworks.core.firework.FiretaskBase
Wrapper around shutil.make_archive to make tar archives.
Parameters: -
optional_params
= ['format']¶
-
required_params
= ['base_name']¶
-
run_task
(fw_spec)¶
-
-
class
fireworks.user_objects.firetasks.fileio_tasks.
CompressDirTask
(*args, **kwargs)¶ Bases:
fireworks.core.firework.FiretaskBase
Compress all files in a directory.
Parameters: -
optional_params
= ['compression', 'dest', 'ignore_errors']¶
-
run_task
(fw_spec)¶
-
-
class
fireworks.user_objects.firetasks.fileio_tasks.
DecompressDirTask
(*args, **kwargs)¶ Bases:
fireworks.core.firework.FiretaskBase
Decompress all files in a directory. Autodetects gz, bz2 and z file extensions.
Parameters: -
optional_params
= ['dest', 'ignore_errors']¶
-
run_task
(fw_spec)¶
-
-
class
fireworks.user_objects.firetasks.fileio_tasks.
FileDeleteTask
(*args, **kwargs)¶ Bases:
fireworks.core.firework.FiretaskBase
A Firetask to delete files:
- Required params:
- files_to_delete: ([str]) Filenames to delete
- Optional params:
- dest: (str) Shared path for files
- ignore_errors (bool): Whether to ignore errors. Defaults to True.
-
required_params
= ['files_to_delete']¶
-
run_task
(fw_spec)¶
-
class
fireworks.user_objects.firetasks.fileio_tasks.
FileTransferTask
(*args, **kwargs)¶ Bases:
fireworks.core.firework.FiretaskBase
A Firetask to Transfer files. Note that
- Required params:
- mode: (str) - move, mv, copy, cp, copy2, copytree, copyfile, rtransfer
- files: ([str]) or ([(str, str)]) - list of source files, or dictionary containing
- ‘src’ and ‘dest’ keys
- dest: (str) destination directory, if not specified within files parameter (else optional)
- Optional params:
- server: (str) server host for remote transfer
- user: (str) user to authenticate with on remote server
- key_filename: (str) optional SSH key location for remote transfer
- max_retry: (int) number of times to retry failed transfers; defaults to 0 (no retries)
- retry_delay: (int) number of seconds to wait between retries; defaults to 10
-
fn_list
= {'move': <function move>, 'mv': <function move>, 'copy': <function copy>, 'cp': <function copy>, 'copy2': <function copy2>, 'copytree': <function copytree>, 'copyfile': <function copyfile>}¶
-
optional_params
= ['server', 'user', 'key_filename', 'max_retry', 'retry_delay']¶
-
required_params
= ['mode', 'files']¶
-
run_task
(fw_spec)¶
-
class
fireworks.user_objects.firetasks.fileio_tasks.
FileWriteTask
(*args, **kwargs)¶ Bases:
fireworks.core.firework.FiretaskBase
A Firetask to write files:
- Required params:
- files_to_write: ([{filename:(str), contents:(str)}]) List of dicts with filenames
- and contents
- Optional params:
- dest: (str) Shared path for files
-
required_params
= ['files_to_write']¶
-
run_task
(fw_spec)¶
fireworks.user_objects.firetasks.filepad_tasks module¶
-
class
fireworks.user_objects.firetasks.filepad_tasks.
AddFilesTask
(*args, **kwargs)¶ Bases:
fireworks.core.firework.FiretaskBase
A Firetask to add files to the filepad.
- Required params:
- paths (list/str): either list of paths or a glob pattern string.
- Optional params:
- identifiers ([str]): list of identifiers, one for each file in the paths list
- directory (str): path to directory where the pattern matching is to be done.
- filepad_file (str): path to the filepad db config file
- compress (bool): whether or not to compress the file before inserting to gridfs
- metadata (dict): metadata to store along with the file, stored in ‘metadata’ key
-
optional_params
= ['identifiers', 'directory', 'filepad_file', 'compress', 'metadata']¶
-
required_params
= ['paths']¶
-
run_task
(fw_spec)¶
-
class
fireworks.user_objects.firetasks.filepad_tasks.
DeleteFilesTask
(*args, **kwargs)¶ Bases:
fireworks.core.firework.FiretaskBase
A Firetask to delete files from the filepad
- Required params:
- identifiers ([str]): identifiers of files to delete
- Optional params:
- filepad_file (str): path to the filepad db config file
-
optional_params
= ['filepad_file']¶
-
required_params
= ['identifiers']¶
-
run_task
(fw_spec)¶
-
class
fireworks.user_objects.firetasks.filepad_tasks.
GetFilesTask
(*args, **kwargs)¶ Bases:
fireworks.core.firework.FiretaskBase
A Firetask to fetch files from the filepad and write it to specified directory(current working directory if not specified)
- Required params:
- identifiers ([str]): identifiers of files to fetch
- Optional params:
- filepad_file (str): path to the filepad db config file
- dest_dir (str): destination directory, default is the current working directory
- new_file_names ([str]): if provided, the retrieved files will be renamed
-
optional_params
= ['filepad_file', 'dest_dir', 'new_file_names']¶
-
required_params
= ['identifiers']¶
-
run_task
(fw_spec)¶
-
fireworks.user_objects.firetasks.filepad_tasks.
get_fpad
(fpad_file)¶
fireworks.user_objects.firetasks.script_task module¶
This module includes tasks to integrate scripts and python functions
-
class
fireworks.user_objects.firetasks.script_task.
PyTask
(*args, **kwargs)¶ Bases:
fireworks.core.firework.FiretaskBase
Runs any python function! Extremely powerful, which allows you to essentially run any accessible method on the system. The optional inputs and outputs lists may contain spec keys to add to args list and to make the function output available in the curent and in children fireworks.
- Required parameters:
- func (str): Fully qualified python method. E.g., json.dump, or shutil
- .copy, or some other function that is not part of the standard library!
- Optional parameters:
- args (list): List of args. Default is empty.
- kwargs (dict): Dictionary of keyword args. Default is empty.
- auto_kwargs (bool): If True, all other params not starting with ‘_’ are supplied as keyword args
- stored_data_varname (str): Whether to store the output in FWAction. If this is a string that does not evaluate to False, the output of the function will be stored as FWAction(stored_data={stored_data_varname: output}). The name is deliberately long to avoid potential name conflicts.
- inputs ([str]): a list of keys in spec which will be used as inputs; the generated arguments list will be appended to args
- outputs ([str]): a list of spec keys that will be used to pass the function’s outputs to child fireworks
- chunk_number (int): a serial number of the Firetask within a group of Firetasks generated by a ForeachTask
-
optional_params
= ['args', 'kwargs', 'auto_kwargs', 'stored_data_varname', 'inputs', 'outputs', 'chunk_number']¶
-
required_params
= ['func']¶
-
run_task
(fw_spec)¶
-
class
fireworks.user_objects.firetasks.script_task.
ScriptTask
(*args, **kwargs)¶ Bases:
fireworks.core.firework.FiretaskBase
Runs a user-defined script
-
classmethod
from_str
(shell_cmd, parameters=None)¶
-
required_params
= ['script']¶
-
run_task
(fw_spec)¶
-
classmethod
fireworks.user_objects.firetasks.templatewriter_task module¶
-
class
fireworks.user_objects.firetasks.templatewriter_task.
TemplateWriterTask
(*args, **kwargs)¶ Bases:
fireworks.core.firework.FiretaskBase
Task to write templated files via Jinja2 library Required parameters:
- template_file: (str) - path to template file
- context: (dict) - variable replacements for the template file
- output_file: (str) - output file
- Optional parameters:
- append: (bool) - append to output file (instead of overwrite)
- template_dir: (str) - directory in which to find the template file
-
run_task
(fw_spec)¶
fireworks.user_objects.firetasks.unittest_tasks module¶
-
class
fireworks.user_objects.firetasks.unittest_tasks.
ExportTestSerializer
(a)¶ Bases:
fireworks.utilities.fw_serializers.FWSerializable
-
__init__
(a)¶
-
classmethod
from_dict
(m_dict)¶
-
to_dict
()¶
-
-
class
fireworks.user_objects.firetasks.unittest_tasks.
TestSerializer
(a, m_date)¶ Bases:
fireworks.utilities.fw_serializers.FWSerializable
-
__init__
(a, m_date)¶
-
classmethod
from_dict
(m_dict)¶
-
to_dict
(*args, **kwargs)¶
-