lenstronomy.Sampling.Pool package¶
Submodules¶
lenstronomy.Sampling.Pool.multiprocessing module¶
this file is taken from schwimmbad (https://github.com/adrn/schwimmbad) and an explicit fork by Aymeric Galan to replace the multiprocessing with the multiprocess dependence as for multi-threading, multiprocessing is not supporting dill (only pickle) which is required.
The class also extends with a is_master()
definition
- class lenstronomy.Sampling.Pool.multiprocessing.MultiPool(processes=None, initializer=None, initargs=(), **kwargs)[source]¶
Bases:
multiprocess.pool.Pool
A modified version of
multiprocessing.pool.Pool
that has better behavior with regard toKeyboardInterrupts
in themap()
method. (Original author: Peter K. G. Williams)- processesint, optional
The number of worker processes to use; defaults to the number of CPUs.
- initializercallable, optional
If specified, a callable that will be invoked by each worker process when it starts.
- initargsiterable, optional
Arguments for
initializer
; it will be called asinitializer(*initargs)
.- kwargs:
Extra arguments passed to the
multiprocessing.pool.Pool
superclass.
- map(func, iterable, chunksize=None, callback=None)[source]¶
Equivalent to the built-in
map()
function andmultiprocessing.pool.Pool.map()
, without catchingKeyboardInterrupt
.- funccallable
A function or callable object that is executed on each element of the specified
tasks
iterable. This object must be picklable (i.e. it can’t be a function scoped within a function or alambda
function). This should accept a single positional argument and return a single object.- iterableiterable
A list or iterable of tasks. Each task can be itself an iterable (e.g., tuple) of values or data to pass in to the worker function.
- callbackcallable, optional
An optional callback function (or callable) that is called with the result from each worker run and is executed on the master process. This is useful for, e.g., saving results to a file, since the callback is only called on the master thread.
- resultslist
A list of results from the output of each
worker()
call.
- wait_timeout = 3600¶
lenstronomy.Sampling.Pool.pool module¶
this file is taken from schwimmbad (https://github.com/adrn/schwimmbad) and an explicit fork by Aymeric Galan to replace the multiprocessing with the multiprocess dependence as for multi-threading, multiprocessing is not supporting dill (only pickle) which is required.
Tests show that the MPI mode works with Python 3.7.2 but not with Python 3.7.0 on a specific system due to mpi4py dependencies and configurations.
Contributions by: - Peter K. G. Williams - Júlio Hoffimann Mendes - Dan Foreman-Mackey - Aymeric Galan - Simon Birrer
Implementations of four different types of processing pools:
MPIPool: An MPI pool.
MultiPool: A multiprocessing for local parallelization.
SerialPool: A serial pool, which uses the built-in map function
- lenstronomy.Sampling.Pool.pool.choose_pool(mpi=False, processes=1, **kwargs)[source]¶
Extends the capabilities of the schwimmbad.choose_pool method.
It handles the use_dill parameters in kwargs, that would otherwise raise an error when processes > 1. Any thread in the returned multiprocessing pool (e.g. processes > 1) also default
The requirement of schwimmbad relies on the master branch (as specified in requirements.txt). The ‘use_dill’ functionality can raise if not following the requirement specified.
Choose between the different pools given options from, e.g., argparse.
- mpibool, optional
Use the MPI processing pool,
MPIPool
. By default,False
, will use theSerialPool
.- processesint, optional
Use the multiprocessing pool,
MultiPool
, with this number of processes. By default,processes=1
, will use them:class:~schwimmbad.serial.SerialPool.
Any additional kwargs are passed in to the pool class initializer selected by the arguments.