midgard.dev.plugins
Set up a plug-in architecture for Midgard
Description:
In order to be able to add models, parsers, data sources etc without needing to
hardcode names, but rather pick them from configuration files, we use a simple
plug-in architecture. The plug-in mechanism is based on the different plug-ins
registering themselves using the register
decorator:
from midgard.dev import plugins
@plugins.register
def simple_model(rundate, tech, dset):
...
Plug-ins are registered based on the name of the module (file) they are defined
in, as well as the package (directory) which contains them. Typically all
plug-ins of a given type are collected in a package, e.g. models, techniques,
parsers, etc. To list all plug-ins in a package use names
:
> from midgard.dev import plugins
> plugins.names('midgard.models')
['model_one', 'model_three', 'model_two']
If the optional parameter config_key
is given, then only plug-ins listed in
the corresponding section in the current configuration file is listed. For
instance, if the configuration file contains a line saying
ham_models = model_three, model_one
then we can list only the ham_models
as follows:
> from midgard.dev import plugins
> plugins.names('midgard.models', config_key='ham_models')
['model_one', 'model_three']
Note that the plug-ins by default are sorted alphabetically.
To run the plug-ins, use either call_all
or call_one
. The former calls all
plug-ins and returns a dictionary containing the result from each plug-in. As
with names
the optional parameter config_key
may be given:
> from midgard.dev import plugins
> plugins.call_all('midgard.models', config_key='ham_models', arg_to_plugin='hello')
{'model_three': <result from model_three>, 'model_one': <result from model_one>}
Arguments to the plug-ins should be passed as named arguments to call_all
.
Similarly, one plug-in may be called explicitly using call_one
:
> from midgard.dev import plugins
> plugins.call_one('midgard.models', plugin_name='model_one', arg_to_plugin='hello')
<result from model_one>
There may be more than one function in each plug-in that is decorated by
register
. In this case, the default behaviour is that only the first function
will be called. To call the other registered functions one should use the
list_parts
function to get a list of these functions and call them explicitly
using the part
optional parameter to call_one
:
> from midgard.dev import plugins
> plugins.list_parts('midgard.techniques', plugin_name='vlbi')
['read', 'edit', 'calculate', 'estimate', 'write_result'])
> for part in plugins.list_parts('midgard.techniques', plugin_name='vlbi'):
... plugins.call_one('midgard.techniques', plugin_name='vlbi', part=part, ...)
Plugin
Plugin(name:str, function:Callable, file_path:pathlib.Path, sort_value:int)
Information about a plug-in
Args:
name
: Name of the plug-in.function
: The plug-in.file_path
: Path to the source code of the plug-in, may be used to add the source as a dependency.sort_value
: Value used when sorting plug-ins in order to control the order they are called.
add_alias()
add_alias(package_name:str, alias:str) -> None
Add alias to plug-in package
This allows one package of plug-ins to be spread over several directories
Args:
package_name
: Name of package containing plug-ins.directory
: Additional plug-in directory.
call()
call(package_name:str, plugin_name:str, part:Union[str, NoneType]=None, prefix:Union[str, NoneType]=None, plugin_logger:Union[Callable[[str], NoneType], NoneType]=None, **plugin_args:Any) -> Any
Call one plug-in
If the plug-in is not part of the package an UnknownPluginError is raised.
If there are several functions registered in a plug-in and part
is not
specified, then the first function registered in the plug-in will be
called.
Args:
package_name
: Name of package containing plug-ins.plugin_name
: Name of the plug-in, i.e. the module containing the plug-in.part
: Name of function to call within the plug-in (optional).prefix
: Prefix of the plug-in name, used if the plug-in name is not found (optional).plugin_logger
: Function used for logging (optional).plugin_args
: Named arguments passed on to the plug-in.
Returns:
Return value of the plug-in.
call_all()
call_all(package_name:str, plugins:Union[List[str], NoneType]=None, part:Union[str, NoneType]=None, prefix:Union[str, NoneType]=None, plugin_logger:Union[Callable[[str], NoneType], NoneType]=None, **plugin_args:Any) -> Dict[str, Any]
Call all plug-ins in a package
If plugins
is given, it should be a list of names of plug-ins. If a
plug-in listed in the plugins
-list or in the config file does not exist,
an UnknownPluginError is raised.
If plugins
is not given, all available plugins will be called. Do note,
however, that this will import all python files in the package.
Args:
package_name
: Name of package containing plug-ins.plugins
: List of plug-in names that should be used (optional).part
: Name of function to call within the plug-ins (optional).prefix
: Prefix of the plug-in names, used for a plug-in if it is not found (optional).plugin_logger
: Function used for logging (optional).plugin_args
: Named arguments passed on to all the plug-ins.
Returns:
Dictionary of all results from the plug-ins.
doc()
doc(package_name:str, plugin_name:str, part:Union[str, NoneType]=None, prefix:Union[str, NoneType]=None, long_doc:bool=True, include_details:bool=False) -> str
Document one plug-in
If the plug-in is not part of the package an UnknownPluginError is raised.
If there are several functions registered in a plug-in and part
is not
specified, then the first function registered in the plug-in will be
documented.
Args:
package_name
: Name of package containing plug-ins.plugin_name
: Name of the plug-in, i.e. the module containing the plug-in.part
: Name of function to call within the plug-in (optional).prefix
: Prefix of the plug-in name, used if the plug-in name is unknown (optional).long_doc
: Whether to return the long doc-string or the short one-line string (optional).include_details
: Whether to include development details like parameters and return values (optional).
Returns:
Documentation of the plug-in.
doc_all()
doc_all(package_name:str, plugins:Union[Iterable[str], NoneType]=None, prefix:Union[str, NoneType]=None, long_doc:bool=True, include_details:bool=False) -> Dict[str, str]
Call all plug-ins in a package
If plugins
is given, it should be a list of names of plug-ins. If a
plug-in listed in the plugins
-list does not exist, an UnknownPluginError
is raised.
If plugins
is not given, all available plugins will be called. Do note,
however, that this will import all python files in the package.
Args:
package_name (String): Name of package containing plug-ins. plugins (Tuple): List of plug-ins that should be used (optional). prefix (String): Prefix of the plug-in names, used if any of the plug-ins are unknown (optional). long_doc (Boolean): Whether to return the long doc-string or the short one-line string (optional). include_details (Boolean): Whether to include development details like parameters and return values (optional).
Returns:
Dict
: Dictionary of all results from the plug-ins.
exists()
exists(package_name:str, plugin_name:str) -> bool
Check whether or not a plug-in exists in a package
Tries to import the given plug-in.
Args:
package_name
: Name of package containing plug-ins.plugin_name
: Name of the plug-in (module).
Returns:
True if plug-in exists, False otherwise.
load()
load(package_name:str, plugin_name:str, prefix:Union[str, NoneType]=None) -> str
Load one plug-in from a package
First tries to load the plugin with the given name. If that fails, it tries to load {prefix}_{plugin_name} instead.
Args:
package_name
: Name of package containing plug-ins.plugin_name
: Name of the plug-in (module).prefix
: Prefix of the plug-in name, used if the plug-in name is unknown (optional).
Returns:
Actual name of plug-in (with or without prefix).
names()
names(package_name:str, plugins:Union[Iterable[str], NoneType]=None, prefix:Union[str, NoneType]=None) -> List[str]
List plug-ins in a package
If plugins
is given, it should be a list of names of plug-ins. If a
plug-in listed in the plugins
-list does not exist, an UnknownPluginError
is raised.
If plugins
is not given, all available plugins will be listed. Do note,
however, that this will import all python files in the package.
Args:
package_name
: Name of package containing plug-ins.plugins
: List of plug-ins that should be used (optional).prefix
: Prefix of the plug-in names, used if any of the plug-in names are unknown (optional).
Returns:
List of strings with names of plug-ins.
parts()
parts(package_name:str, plugin_name:str, prefix:Union[str, NoneType]=None) -> List[str]
List all parts of one plug-in
Args:
package_name
: Name of package containing plug-ins.plugin_name
: Name of the plug-in.prefix
: Prefix of the plug-in name, used if the plug-in name is unknown (optional).
Returns:
List
: Strings with names of parts.
register()
register(func:Callable, name:Union[str, NoneType]=None, sort_value:int=0) -> Callable
Decorator used to register a plug-in
Plug-ins are registered based on the name of the module (file) they are defined in, as well as the package (directory) which contains them. Typically all plug-ins of a given type are collected in a package, e.g. models, techniques, parsers, etc. The path to the source code file is also stored. This is used to be able to add the source code as a dependency file when the plug-in is called.
If name
is given, the plug-in is registered based on this name instead of
the name of the module. The name of the module is still registered as a
part that can be used to distinguish between similar plug-ins in different
files (see for instance how session
is used in midgard.pipelines
).
Args:
func
: The function that is being registered.name
: Alternative name of plug-in. Used byregister_named
.sort_value
: The value used when sorting plug-ins. Used byregister_ordered
.
Returns:
The function that is being registered.
register_named()
register_named(name:str) -> Callable
Decorator used to register a named plug-in
This allows for overriding the name used to register the plug-in. See
register
for more details.
Args:
name
: Name used for plug-in instead of module name.
Returns:
Decorator that registers a named function.
register_ordered()
register_ordered(sort_value:int) -> Callable
Decorator used to register a plug-in with a specific sort order
The sort value should be a number. Lower numbers are sorted first, higher numbers last. Plug-ins without an explicit sort_order gets the sort value of 0.
Args:
sort_value
: The value used when sorting plug-ins.
Returns:
Decorator that registers an ordered function.