core Package
firework Module
This module contains some of the most central FireWorks classes:
- A FireTaskBase defines the contract for tasks that run within a FireWork (FireTasks)
- A FWAction encapsulates the output of a FireTask and tells FireWorks what to do next after a job completes
- A FireWork defines a workflow step and contains one or more FireTasks.
- A Launch describes the run of a FireWork on a computing resource.
-
class fireworks.core.firework.FWAction(command, stored_data=None, mod_spec=None)
Bases: fireworks.utilities.fw_serializers.FWSerializable
- A FWAction encapsulates the output of a FireTask (it is returned by a FireTask after the FireTask completes). The
- FWAction allows a user to store rudimentary output data as well as return commands that alter the workflow.
-
__init__(command, stored_data=None, mod_spec=None)
Parameters: |
- command – (str) an item from the list of FWAction.commands
- stored_data – (dict) any output data to store. Intended to be brief, not store a ton of data.
- mod_spec – description of how to modify the Workflow according to a set of rules (see tutorial docs)
|
-
commands = ['CONTINUE', 'DEFUSE', 'MODIFY', 'DETOUR', 'CREATE', 'PHOENIX', 'BREAK']
-
classmethod from_dict(*args, **kwargs)
-
to_dict(*args, **kwargs)
-
class fireworks.core.firework.FireTaskBase(parameters=None)
Bases: fireworks.utilities.fw_serializers.FWSerializable
FireTaskBase is used as an abstract class that defines a computing task (FireTask). All FireTasks
should inherit from FireTaskBase.
-
__init__(parameters=None)
Parameters: | parameters – (dict) Parameters that control the FireTask’s operation (custom depending on the FireTask |
type)
-
classmethod from_dict(*args, **kwargs)
-
run_task(fw_spec)
This method gets called when the FireTask is run. It can take in a FireWork spec,
perform some task using that data, and then return an output in the form of a FWAction.
Parameters: | fw_spec – (dict) a FireWork spec |
Returns: | (FWAction) |
-
to_dict(*args, **kwargs)
-
class fireworks.core.firework.FireWork(tasks, spec=None, launches=None, state='WAITING', created_on=None, fw_id=-1)
Bases: fireworks.utilities.fw_serializers.FWSerializable
-
STATE_RANKS = {'RUNNING': 5, 'WAITING': 1, 'RESERVED': 3, 'FIZZLED': 4, 'CANCELED': 6, 'READY': 2, 'COMPLETED': 7, 'DEFUSED': 0}
-
__init__(tasks, spec=None, launches=None, state='WAITING', created_on=None, fw_id=-1)
Parameters: |
- tasks – (list) a list of FireTasks to run in sequence
- spec – (dict) specification of the job to run. Used by the FireTask
- fw_id – (int) the FW’s database id (negative numbers will be re-assigned dynamically when they are
entered in the database through the LaunchPad.
- launches – (list) a list of Launch objects of this FireWork
- state – (str) the state of the FW (e.g. WAITING, RUNNING, COMPLETED, CANCELED)
|
-
classmethod from_dict(*args, **kwargs)
-
to_db_dict()
-
to_dict(*args, **kwargs)
-
class fireworks.core.firework.Launch(state, launch_dir, fworker=None, host=None, ip=None, action=None, state_history=None, launch_id=None, fw_id=None)
Bases: fireworks.utilities.fw_serializers.FWSerializable, object
-
__init__(state, launch_dir, fworker=None, host=None, ip=None, action=None, state_history=None, launch_id=None, fw_id=None)
Parameters: |
- state – (str) the state of the Launch (e.g. RUNNING, COMPLETED)
- launch_dir – (str) the directory where the Launch takes place
- fworker – (FWorker) The FireWorker running the Launch
- host – (str) the hostname where the launch took place (set automatically if None)
- ip – (str) the IP address where the launch took place (set automatically if None)
- action – (FWAction) the output of the Launch
- state_history – (list) a history of all states of the Launch and when they occurred
- launch_id – (int) launch_id set by the LaunchPad
- fw_id – (int) id of the FireWork this Launch is running
|
-
classmethod from_dict(*args, **kwargs)
-
last_pinged
Returns: | (datetime) the time the Launch last pinged a heartbeat that it was still running |
-
reservedtime_secs
Returns: | (int) number of seconds the Launch was queued |
-
runtime_secs
Returns: | (int) the number of seconds that the Launch ran for |
-
set_reservation_id(reservation_id)
Adds the job_id to the reservation
Parameters: | reservation_id – the id of the reservation (e.g., queue reservation) |
-
state
Returns: | (str) The current state of the Launch. |
-
time_end
Returns: | (datetime) the time the Launch was COMPLETED or FIZZLED |
-
time_reserved
Returns: | (datetime) the time the Launch was RESERVED in the queue |
-
time_start
Returns: | (datetime) the time the Launch started RUNNING |
-
to_db_dict()
-
to_dict(*args, **kwargs)
-
touch_history()
Updates the update_at field of the state history of a Launch. Used to ping that a Launch is still alive.
fw_config Module
A set of global constants for FireWorks (Python code as a config file)
-
fireworks.core.fw_config.FWConfig(*args, **kwargs)
-
fireworks.core.fw_config.singleton(class_)
fworker Module
This module contains classes relevant for a FireWorker (worker computing resource)
-
class fireworks.core.fworker.FWorker(name='anonymous worker', category='general worker', query=None, params=None)
Bases: fireworks.utilities.fw_serializers.FWSerializable
-
__init__(name='anonymous worker', category='general worker', query=None, params=None)
Parameters: |
- name – the name of the resource, should be unique
- category – a String describing the computing resource, does not need to be unique
- query – a dict query that restricts the type of FireWork this resource will run
- params – further descriptions of this resource
|
-
classmethod from_dict(*args, **kwargs)
-
to_dict(*args, **kwargs)
launchpad Module
The LaunchPad manages the FireWorks database.
-
class fireworks.core.launchpad.LaunchPad(host='localhost', port=27017, name='fireworks', username=None, password=None, logdir=None, strm_lvl=None)
Bases: fireworks.utilities.fw_serializers.FWSerializable
The LaunchPad manages the FireWorks database.
-
__init__(host='localhost', port=27017, name='fireworks', username=None, password=None, logdir=None, strm_lvl=None)
Parameters: |
- host –
- port –
- name –
- username –
- password –
- logdir –
- strm_lvl –
|
-
add_wf(wf)
Parameters: | wf – a Workflow object. |
-
detect_fizzled(expiration_secs=14400, fix=False)
-
detect_unreserved(expiration_secs=1209600, fix=False)
-
classmethod from_dict(d)
-
get_fw_by_id(fw_id)
Given a FireWork id, give back a FireWork object
Parameters: | fw_id – FireWork id (int) |
Returns: | FireWork object |
-
get_fw_ids(query=None, sort=False)
Return all the fw ids that match a query,
:param query: a dict representing a Mongo query
-
get_launch_by_id(launch_id)
Given a Launch id, return details of the Launch
Parameters: | launch_id – launch id |
Returns: | Launch object |
-
get_new_fw_id()
Checkout the next FireWork id
-
get_new_launch_id()
Checkout the next Launch id
-
get_wf_by_fw_id(fw_id)
Given a FireWork id, give back the Workflow containing that FireWork
:param fw_id:
:return: A Workflow object
-
maintain()
-
mark_fizzled(launch_id)
-
reset(password, require_password=True)
Create a new FireWorks database. This will overwrite the existing FireWorks database! To safeguard against accidentally erasing an existing database, a password must be entered.
Parameters: |
- password – A String representing today’s date, e.g. ‘2012-12-31’
- require_password – Whether a password is required to reset the DB. Highly recommended to leave this set to True, otherwise you are inviting disaster!
|
-
run_exists()
Checks to see if the database contains any FireWorks that are ready to run
:return: (T/F)
-
to_dict()
Note: usernames/passwords are exported as unencrypted Strings!
-
unreserve(launch_id)
rocket Module
TODO: add docs
-
class fireworks.core.rocket.Rocket(launchpad, fworker, fw_id)
The Rocket fetches a workflow step from the FireWorks database and executes it.
-
__init__(launchpad, fworker, fw_id)
Parameters: |
- launchpad – A LaunchPad object for interacting with the FW database
- fworker – A FWorker object describing the computing resource
|
-
run()
Run the rocket (actually check out a job from the database and execute it)
-
fireworks.core.rocket.ping_launch(launchpad, launch_id, stop_event)
rocket_launcher Module
-
fireworks.core.rocket_launcher.launch_rocket(launchpad, fworker=None, logdir=None, strm_lvl=None, fw_id=None)
Run a single rocket in the current directory
:param launchpad: a LaunchPad object
:param fworker: a FWorker object
-
fireworks.core.rocket_launcher.rapidfire(launchpad, fworker=None, m_dir=None, logdir=None, strm_lvl=None, nlaunches=0, sleep_time=60, max_loops=-1)
Keeps running Rockets in m_dir until we reach an error. Automatically creates subdirectories for each Rocket.
Usually stops when we run out of FireWorks from the LaunchPad.
Parameters: |
- launchpad – a LaunchPad object
- fworker – a FWorker object
- m_dir – the directory in which to loop Rocket running
- nlaunches – 0 means ‘until completion’, -1 means ‘infinity’
|
workflow Module
-
class fireworks.core.workflow.Workflow(fireworks, links_dict=None, metadata={})
Bases: fireworks.utilities.fw_serializers.FWSerializable
-
class Links
Bases: dict, fireworks.utilities.fw_serializers.FWSerializable
-
classmethod from_dict(m_dict)
-
nodes
-
parent_links
-
to_db_dict()
-
to_dict()
-
Workflow.__init__(fireworks, links_dict=None, metadata={})
Parameters: |
- fireworks – a list of FireWork objects
- links_dict – A dict representing workflow links
- metadata – metadata for this Workflow
|
-
Workflow.apply_action(action, fw_id)
-
classmethod Workflow.from_FireWork(fw)
-
classmethod Workflow.from_dict(m_dict)
-
classmethod Workflow.from_tarfile(tar_filename)
-
Workflow.refresh(fw_id, updated_ids=None)
-
Workflow.root_fw_ids
-
Workflow.to_db_dict()
-
Workflow.to_dict()
-
Workflow.to_tarfile(f_name='fwf.tar', f_format='json')