Huey’s API¶
Most end-users will interact with the API using the two decorators:
The API documentation will follow the structure of the huey API, starting with
the highest-level interfaces (the decorators) and eventually discussing the
lowest-level interfaces, the BaseQueue
and BaseDataStore
objects.
Function decorators and helpers¶
-
class
Huey
(name[, result_store=True[, events=True[, store_none=False[, always_eager=False[, store_errors=True[, blocking=False[, **storage_kwargs]]]]]]])¶ Huey executes tasks by exposing function decorators that cause the function call to be enqueued for execution by the consumer.
Typically your application will only need one Huey instance, but you can have as many as you like – the only caveat is that one consumer process must be executed for each Huey instance.
Parameters: - name – the name of the huey instance or application.
- result_store (bool) – whether the results of tasks should be stored.
- events (bool) – whether events should be emitted by the consumer.
- store_none (bool) – Flag to indicate whether tasks that return
None
should store their results in the result store. - always_eager (bool) – Useful for testing, this will execute all tasks immediately, without enqueueing them.
- store_errors (bool) – whether task errors should be stored.
- blocking (bool) – whether the queue will block (if False, then the queue will poll).
- storage_kwargs – arbitrary kwargs to pass to the storage implementation.
Example usage:
from huey import RedisHuey, crontab huey = RedisHuey('my-app') @huey.task() def slow_function(some_arg): # ... do something ... return some_arg @huey.periodic_task(crontab(minute='0', hour='3')) def backup(): # do a backup every day at 3am return
-
task
([retries=0[, retry_delay=0[, retries_as_argument=False[, include_task=False]]]])¶ Function decorator that marks the decorated function for processing by the consumer. Calls to the decorated function will do the following:
- Serialize the function call into a message suitable for storing in the queue
- Enqueue the message for execution by the consumer
- If a
result_store
has been configured, return anTaskResultWrapper
instance which can retrieve the result of the function, orNone
if not using a result store.
Note
Huey can be configured to execute the function immediately by instantiating it with
always_eager = True
– this is useful for running in debug mode or when you do not wish to run the consumer.Here is how you might use the
task
decorator:# assume that we've created a huey object from huey import RedisHuey huey = RedisHuey() @huey.task() def count_some_beans(num): # do some counting! return 'Counted %s beans' % num
Now, whenever you call this function in your application, the actual processing will occur when the consumer dequeues the message and your application will continue along on its way.
Without a result store:
>>> res = count_some_beans(1000000) >>> res is None True
With a result store:
>>> res = count_some_beans(1000000) >>> res <huey.api.TaskResultWrapper object at 0xb7471a4c> >>> res() 'Counted 1000000 beans'
Parameters: - retries (int) – number of times to retry the task if an exception occurs
- retry_delay (int) – number of seconds to wait between retries
- retries_as_argument (boolean) – whether the number of retries should be passed in to the decorated function as an argument.
- include_task (boolean) – whether the task instance itself should be
passed in to the decorated function as the
task
argument.
Return type: decorated function
The return value of any calls to the decorated function depends on whether the
Huey
instance is configured with aresult_store
. If a result store is configured, the decorated function will return anTaskResultWrapper
object which can fetch the result of the call from the result store – otherwise it will simply returnNone
.The
task
decorator also does one other important thing – it adds a special function onto the decorated function, which makes it possible to schedule the execution for a certain time in the future:-
{decorated func}.schedule(args=None, kwargs=None, eta=None, delay=None, convert_utc=True)
Use the special
schedule
function to schedule the execution of a queue task for a given time in the future:import datetime # get a datetime object representing one hour in the future in_an_hour = datetime.datetime.now() + datetime.timedelta(seconds=3600) # schedule "count_some_beans" to run in an hour count_some_beans.schedule(args=(100000,), eta=in_an_hour) # another way of doing the same thing... count_some_beans.schedule(args=(100000,), delay=(60 * 60))
Parameters: - args – arguments to call the decorated function with
- kwargs – keyword arguments to call the decorated function with
- eta (datetime) – the time at which the function should be executed
- delay (int) – number of seconds to wait before executing function
- convert_utc – whether the
eta
ordelay
should be converted from local time to UTC, defaults toTrue
. If you are running your consumer inlocaltime
mode, you should probably specifyFalse
here.
Return type: like calls to the decorated function, will return an
TaskResultWrapper
object if a result store is configured, otherwise returnsNone
-
{decorated func}.call_local
Call the
@task
-decorated function without enqueueing the call. Or, in other words,call_local()
provides access to the actual function.>>> count_some_beans.call_local(1337) 'Counted 1337 beans'
-
{decorated func}.task_class
Store a reference to the task class for the decorated function.
>>> count_some_beans.task_class tasks.queuecmd_count_beans
-
periodic_task
(validate_datetime)¶ Function decorator that marks the decorated function for processing by the consumer at a specific interval. Calls to functions decorated with
periodic_task
will execute normally, unliketask()
, which enqueues tasks for execution by the consumer. Rather, theperiodic_task
decorator serves to mark a function as needing to be executed periodically by the consumer.Note
By default, the consumer will execute
periodic_task
functions. To disable this, run the consumer with-n
or--no-periodic
.The
validate_datetime
parameter is a function which accepts a datetime object and returns a boolean value whether or not the decorated function should execute at that time or not. The consumer will send a datetime to the function every minute, giving it the same granularity as the linux crontab, which it was designed to mimic.For simplicity, there is a special function
crontab()
, which can be used to quickly specify intervals at which a function should execute. It is described below.Here is an example of how you might use the
periodic_task
decorator and thecrontab
helper:from huey import crontab from huey import RedisHuey huey = RedisHuey() @huey.periodic_task(crontab(minute='*/5')) def every_five_minutes(): # this function gets executed every 5 minutes by the consumer print("It's been five minutes")
Note
Because functions decorated with
periodic_task
are meant to be executed at intervals in isolation, they should not take any required parameters nor should they be expected to return a meaningful value. This is the same regardless of whether or not you are using a result store.Parameters: validate_datetime – a callable which takes a datetime
and returns a boolean whether the decorated function should execute at that time or notReturn type: decorated function Like
task()
, the periodic task decorator adds several helpers to the decorated function. These helpers allow you to “revoke” and “restore” the periodic task, effectively enabling you to pause it or prevent its execution.-
{decorated_func}.revoke([revoke_until=None[, revoke_once=False]])
Prevent the given periodic task from executing. When no parameters are provided the function will not execute again.
This function can be called multiple times, but each call will overwrite the limitations of the previous.
Parameters: - revoke_until (datetime) – Prevent the execution of the task until the
given datetime. If
None
it will prevent execution indefinitely. - revoke_once (bool) – If
True
will only prevent execution the next time it would normally execute.
# skip the next execution every_five_minutes.revoke(revoke_once=True) # pause the command indefinitely every_five_minutes.revoke() # pause the command for 24 hours every_five_minutes.revoke(datetime.datetime.now() + datetime.timedelta(days=1))
- revoke_until (datetime) – Prevent the execution of the task until the
given datetime. If
-
{decorated_func}.is_revoked([dt=None])
Check whether the given periodic task is revoked. If
dt
is specified, it will check if the task is revoked for the given datetime.Parameters: dt (datetime) – If provided, checks whether task is revoked at the given datetime
-
{decorated_func}.restore()
Clears any revoked status and run the task normally
If you want access to the underlying task class, it is stored as an attribute on the decorated function:
-
{decorated_func}.task_class
Store a reference to the task class for the decorated function.
-
-
revoke
(task[, revoke_until=None[, revoke_once=False]])¶ Prevent the given task from being executed by the consumer after it has been enqueued. To understand this method, you need to know a bit about how the consumer works. When you call a function decorated by the
Huey.task()
method, calls to that function will enqueue a message to the consumer indicating which task to execute, what the parameters are, etc. If the task is not scheduled to execute in the future, and there is a free worker available, the task starts executing immediately. Otherwise if workers are busy, it will wait in line for the next free worker.When you revoke a task, when the worker picks up the revoked task to start executing it, it will instead just throw it away and get the next available task. So, revoking a task only has affect between the time you call the task and the time the worker actually starts executing the task.
Note
When the revoked task is a periodic task, this affects the task as a whole. When the task is a normal task, the revocation action only applies to the given task instance.
This function can be called multiple times, but each call will overwrite any previous revoke settings.
Parameters: - revoke_until (datetime) – Prevent the execution of the task until the
given datetime. If
None
it will prevent execution indefinitely. - revoke_once (bool) – If
True
will only prevent execution the next time it would normally execute.
- revoke_until (datetime) – Prevent the execution of the task until the
given datetime. If
-
restore
(task)¶ Takes a previously revoked task and un-revokes it.
-
revoke_by_id
(task_id[, revoke_until=None[, revoke_once=False]])¶ Exactly the same as
Huey.revoke()
, except it accepts a task ID instead of the task instance itself.
-
restore_by_id
(task_id)¶ Exactly the same as
Huey.restore()
, except it accepts a task ID instead of the task instance itself.
-
is_revoked
(task[, dt=None])¶ Returns a boolean indicating whether the given task is revoked. If the
dt
parameter is specified, then the result will indicate whether the task is revoked at that particular datetime.
-
result
(task_id[, blocking=False[, timeout=None[, backoff=1.15[, max_delay=1.0[, revoke_on_timeout=False[, preserve=False]]]]]])¶ Attempt to retrieve the return value of a task. By default,
result()
will simply check for the value, returningNone
if it is not ready yet. If you want to wait for a value, you can specifyblocking=True
. This will loop, backing off up to the providedmax_delay
, until the value is ready or thetimeout
is reached. If thetimeout
is reached before the result is ready, aDataStoreTimeout
exception will be raised.Warning
By default the result store will delete a task’s return value after the value has been successfully read (by a successful call to the
result()
orTaskResultWrapper.get()
methods). If you need to use the task result multiple times, you must specifypreserve=True
when calling these methods.Parameters: - task_id – the task’s unique identifier.
- blocking (bool) – whether to block while waiting for task result
- timeout – number of seconds to block (if
blocking=True
) - backoff – amount to backoff delay each iteration of loop
- max_delay – maximum amount of time to wait between iterations when attempting to fetch result.
- revoke_on_timeout (bool) – if a timeout occurs, revoke the task, thereby preventing it from running if it is has not started yet.
- preserve (bool) – see the above warning. When set to
True
, this parameter ensures that the task result should be preserved after having been successfully retrieved.
-
pending
([limit=None])¶ Return all unexecuted tasks currently in the queue.
-
scheduled
([limit=None])¶ Return all unexecuted tasks currently in the schedule.
-
all_results
()¶ Return a mapping of task-id to pickled result data for all executed tasks whose return values have not been automatically removed.
-
crontab
(month='*', day='*', day_of_week='*', hour='*', minute='*')¶ Convert a “crontab”-style set of parameters into a test function that will return
True
when a givendatetime
matches the parameters set forth in the crontab.Acceptable inputs:
- “*” = every distinct value
- “*/n” = run every “n” times, i.e. hours=’*/4’ == 0, 4, 8, 12, 16, 20
- “m-n” = run every time m..n
- “m,n” = run on m and n
Return type: a test function that takes a datetime
and returns a boolean
TaskResultWrapper¶
-
class
TaskResultWrapper
(huey, task)¶ Although you will probably never instantiate an
TaskResultWrapper
object yourself, they are returned by any calls totask()
decorated functions (provided that huey is configured with a result store). TheTaskResultWrapper
talks to the result store and is responsible for fetching results from tasks.Once the consumer finishes executing a task, the return value is placed in the result store, allowing the producer to retrieve it.
Note
By default, the data is removed from the result store after being read, but this behavior can be disabled.
Getting results from tasks is very simple:
>>> from main import count_some_beans >>> res = count_some_beans(100) >>> res # what is "res" ? <huey.queue.TaskResultWrapper object at 0xb7471a4c> >>> res() # Fetch the result of this task. 'Counted 100 beans'
What happens when data isn’t available yet? Let’s assume the next call takes about a minute to calculate:
>>> res = count_some_beans(10000000) # let's pretend this is slow >>> res.get() # Data is not ready, so None is returned. >>> res() is None # We can omit ".get", it works the same way. True >>> res(blocking=True, timeout=5) # Block for up to 5 seconds Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/charles/tmp/huey/src/huey/huey/queue.py", line 46, in get raise DataStoreTimeout huey.exceptions.DataStoreTimeout >>> res(blocking=True) # No timeout, will block until it gets data. 'Counted 10000000 beans'
-
get
([blocking=False[, timeout=None[, backoff=1.15[, max_delay=1.0[, revoke_on_timeout=False[, preserve=False]]]]]])¶ Attempt to retrieve the return value of a task. By default,
get()
will simply check for the value, returningNone
if it is not ready yet. If you want to wait for a value, you can specifyblocking=True
. This will loop, backing off up to the providedmax_delay
, until the value is ready or thetimeout
is reached. If thetimeout
is reached before the result is ready, aDataStoreTimeout
exception will be raised.Warning
By default the result store will delete a task’s return value after the value has been successfully read (by a successful call to the
result()
orTaskResultWrapper.get()
methods). If you need to use the task result multiple times, you must specifypreserve=True
when calling these methods.Note
Instead of calling
.get()
, you can simply call theTaskResultWrapper
object directly. Both methods accept the same parameters.Parameters: - blocking (bool) – whether to block while waiting for task result
- timeout – number of seconds to block (if
blocking=True
) - backoff – amount to backoff delay each iteration of loop
- max_delay – maximum amount of time to wait between iterations when attempting to fetch result.
- revoke_on_timeout (bool) – if a timeout occurs, revoke the task, thereby preventing it from running if it is has not started yet.
- preserve (bool) – see the above warning. When set to
True
, this parameter ensures that the task result should be preserved after having been successfully retrieved.
-
revoke
()¶ Revoke the given task. Unless it is in the process of executing, it will be revoked and the task will not run.
in_an_hour = datetime.datetime.now() + datetime.timedelta(seconds=3600) # run this command in an hour res = count_some_beans.schedule(args=(100000,), eta=in_an_hour) # oh shoot, I changed my mind, do not run it after all res.revoke()
-
restore
()¶ Restore the given task. Unless it has already been skipped over, it will be restored and run as scheduled.
-