midgard.dev.plugins

Set up a plug-in architecture for Midgard

Description:

In order to be able to add models, parsers, data sources etc without needing to hardcode names, but rather pick them from configuration files, we use a simple plug-in architecture. The plug-in mechanism is based on the different plug-ins registering themselves using the register decorator:

from midgard.dev import plugins

@plugins.register
def simple_model(rundate, tech, dset):
    ...

Plug-ins are registered based on the name of the module (file) they are defined in, as well as the package (directory) which contains them. Typically all plug-ins of a given type are collected in a package, e.g. models, techniques, parsers, etc. To list all plug-ins in a package use names:

> from midgard.dev import plugins
> plugins.names('midgard.models')
['model_one', 'model_three', 'model_two']

If the optional parameter config_key is given, then only plug-ins listed in the corresponding section in the current configuration file is listed. For instance, if the configuration file contains a line saying

ham_models = model_three, model_one

then we can list only the ham_models as follows:

> from midgard.dev import plugins
> plugins.names('midgard.models', config_key='ham_models')
['model_one', 'model_three']

Note that the plug-ins by default are sorted alphabetically.

To run the plug-ins, use either call_all or call_one. The former calls all plug-ins and returns a dictionary containing the result from each plug-in. As with names the optional parameter config_key may be given:

> from midgard.dev import plugins
> plugins.call_all('midgard.models', config_key='ham_models', arg_to_plugin='hello')
{'model_three': <result from model_three>, 'model_one': <result from model_one>}

Arguments to the plug-ins should be passed as named arguments to call_all.

Similarly, one plug-in may be called explicitly using call_one:

> from midgard.dev import plugins
> plugins.call_one('midgard.models', plugin_name='model_one', arg_to_plugin='hello')
<result from model_one>

There may be more than one function in each plug-in that is decorated by register. In this case, the default behaviour is that only the first function will be called. To call the other registered functions one should use the list_parts function to get a list of these functions and call them explicitly using the part optional parameter to call_one:

> from midgard.dev import plugins
> plugins.list_parts('midgard.techniques', plugin_name='vlbi')
['read', 'edit', 'calculate', 'estimate', 'write_result'])
> for part in plugins.list_parts('midgard.techniques', plugin_name='vlbi'):
...   plugins.call_one('midgard.techniques', plugin_name='vlbi', part=part, ...)

Plugin

Plugin(name:str, function:Callable, file_path:pathlib.Path, sort_value:int)

Information about a plug-in

Args:

add_alias()

add_alias(package_name:str, alias:str) -> None

Add alias to plug-in package

This allows one package of plug-ins to be spread over several directories

Args:

call()

call(package_name:str, plugin_name:str, part:Union[str, NoneType]=None, prefix:Union[str, NoneType]=None, plugin_logger:Union[Callable[[str], NoneType], NoneType]=None, **plugin_args:Any) -> Any

Call one plug-in

If the plug-in is not part of the package an UnknownPluginError is raised.

If there are several functions registered in a plug-in and part is not specified, then the first function registered in the plug-in will be called.

Args:

Returns:

Return value of the plug-in.

call_all()

call_all(package_name:str, plugins:Union[List[str], NoneType]=None, part:Union[str, NoneType]=None, prefix:Union[str, NoneType]=None, plugin_logger:Union[Callable[[str], NoneType], NoneType]=None, **plugin_args:Any) -> Dict[str, Any]

Call all plug-ins in a package

If plugins is given, it should be a list of names of plug-ins. If a plug-in listed in the plugins-list or in the config file does not exist, an UnknownPluginError is raised.

If plugins is not given, all available plugins will be called. Do note, however, that this will import all python files in the package.

Args:

Returns:

Dictionary of all results from the plug-ins.

doc()

doc(package_name:str, plugin_name:str, part:Union[str, NoneType]=None, prefix:Union[str, NoneType]=None, long_doc:bool=True, include_details:bool=False) -> str

Document one plug-in

If the plug-in is not part of the package an UnknownPluginError is raised.

If there are several functions registered in a plug-in and part is not specified, then the first function registered in the plug-in will be documented.

Args:

Returns:

Documentation of the plug-in.

doc_all()

doc_all(package_name:str, plugins:Union[Iterable[str], NoneType]=None, prefix:Union[str, NoneType]=None, long_doc:bool=True, include_details:bool=False) -> Dict[str, str]

Call all plug-ins in a package

If plugins is given, it should be a list of names of plug-ins. If a plug-in listed in the plugins-list does not exist, an UnknownPluginError is raised.

If plugins is not given, all available plugins will be called. Do note, however, that this will import all python files in the package.

Args:

package_name (String): Name of package containing plug-ins. plugins (Tuple): List of plug-ins that should be used (optional). prefix (String): Prefix of the plug-in names, used if any of the plug-ins are unknown (optional). long_doc (Boolean): Whether to return the long doc-string or the short one-line string (optional). include_details (Boolean): Whether to include development details like parameters and return values (optional).

Returns:

exists()

exists(package_name:str, plugin_name:str) -> bool

Check whether or not a plug-in exists in a package

Tries to import the given plug-in.

Args:

Returns:

True if plug-in exists, False otherwise.

load()

load(package_name:str, plugin_name:str, prefix:Union[str, NoneType]=None) -> str

Load one plug-in from a package

First tries to load the plugin with the given name. If that fails, it tries to load {prefix}_{plugin_name} instead.

Args:

Returns:

Actual name of plug-in (with or without prefix).

names()

names(package_name:str, plugins:Union[Iterable[str], NoneType]=None, prefix:Union[str, NoneType]=None) -> List[str]

List plug-ins in a package

If plugins is given, it should be a list of names of plug-ins. If a plug-in listed in the plugins-list does not exist, an UnknownPluginError is raised.

If plugins is not given, all available plugins will be listed. Do note, however, that this will import all python files in the package.

Args:

Returns:

List of strings with names of plug-ins.

parts()

parts(package_name:str, plugin_name:str, prefix:Union[str, NoneType]=None) -> List[str]

List all parts of one plug-in

Args:

Returns:

register()

register(func:Callable, name:Union[str, NoneType]=None, sort_value:int=0) -> Callable

Decorator used to register a plug-in

Plug-ins are registered based on the name of the module (file) they are defined in, as well as the package (directory) which contains them. Typically all plug-ins of a given type are collected in a package, e.g. models, techniques, parsers, etc. The path to the source code file is also stored. This is used to be able to add the source code as a dependency file when the plug-in is called.

If name is given, the plug-in is registered based on this name instead of the name of the module. The name of the module is still registered as a part that can be used to distinguish between similar plug-ins in different files (see for instance how session is used in midgard.pipelines).

Args:

Returns:

The function that is being registered.

register_named()

register_named(name:str) -> Callable

Decorator used to register a named plug-in

This allows for overriding the name used to register the plug-in. See register for more details.

Args:

Returns:

Decorator that registers a named function.

register_ordered()

register_ordered(sort_value:int) -> Callable

Decorator used to register a plug-in with a specific sort order

The sort value should be a number. Lower numbers are sorted first, higher numbers last. Plug-ins without an explicit sort_order gets the sort value of 0.

Args:

Returns:

Decorator that registers an ordered function.