Usage¶
Usage is best described by example.
Simple Usage¶
import asyncio
from aiohttp import ClientSession
from arq import Actor, BaseWorker, concurrent
class Downloader(Actor):
async def startup(self):
self.session = ClientSession(loop=self.loop)
@concurrent
async def download_content(self, url):
async with self.session.get(url) as response:
content = await response.read()
print(f'{url}: {content.decode():.80}...')
return len(content)
async def shutdown(self):
self.session.close()
class Worker(BaseWorker):
shadows = [Downloader]
async def download_lots():
d = Downloader()
for url in ('https://facebook.com', 'https://microsoft.com', 'https://github.com'):
await d.download_content(url)
await d.close()
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(download_lots())
(This script is complete, it should run “as is” both to enqueue jobs and run them)
To enqueue the jobs, simply run the script:
python demo.py
To execute the jobs, either after running demo.py
or before/during:
arq demo.py
For details on the arq CLI:
arq --help
Startup & Shutdown coroutines¶
The startup
and shutdown
coroutines are provided as a convenient way to run logic as actors start and finish,
however it’s important to not that these methods are not called by default when actors are initialised or closed.
They are however called when the actor was started and closed on the worker, eg. in “shadow” mode, see above.
In other words: if you need these coroutines to be called when using an actor in your code; that’s your responsibility.
For example, in the above example there’s no need for self.session
when using the actor in “default” mode,
eg. called with python demo.py
, so neither startup
or shutdown
are called.
Usage with aiohttp¶
Assuming you have Downloader
already defined as per above.
from aiohttp import web
async def start_job(request):
data = await request.post()
# this will enqueue the download_content job
await request.app['downloader'].download_content(data['url'])
raise web.HTTPFound(location='/wherever/')
async def shutdown(app):
await app['downloader'].close()
def create_app():
app = web.Application()
...
app.router.add_post('/start-job/', start_job)
app['downloader'] = Downloader()
# use aiohttp's on_shutdown trigger to close downloader
app.on_shutdown.append(shutdown)
return app
if __name__ == '__main__':
app = create_app()
web.run_app(app, port=8000)
(Won’t run as Downloader
is not defined)
For a full example arq usage with aiohttp and docker see the demo app.
Health checks¶
arq will automatically record some info about it’s current state in redis every health_check_interval
seconds,
see arq.worker.BaseWorker.health_check_interval
. That key/value will expire after health_check_interval + 1
seconds so you can be sure if the variable exists arq is alive and kicking (technically you can be sure it
was alive and kicking health_check_interval
seconds ago).
You can run a health check with the CLI (assuming you’re using the above example):
arq --check demo.py
The command will output the value of the health check if found;
then exit 0
if the key was found and 1
if it was not.
A health check value takes the following form:
Feb-20_11:02:40 j_complete=0 j_failed=0 j_timedout=0 j_ongoing=0 q_high=0 q_dft=0 q_low=0
Where the items have the following meaning:
j_complete
the number of jobs completedj_failed
the number of jobs which have failed eg. raised an exceptionj_timedout
the number of jobs which have timed out, eg. exceededarq.worker.BaseWorker.timeout_seconds
and been cancelledj_ongoing
the number of jobs currently being performedq_*
the number of pending jobs in each queue
Cron Jobs¶
Functions can be scheduled to be run periodically at specific times
from arq import Actor, cron
class FooBar(Actor):
@cron(hour={9, 12, 18}, minute=12)
async def foo(self):
print('run foo job at 9.12am, 12.12pm and 6.12pm')
See arq.main.cron()
for details on the available arguments and how how cron works.
Usage roughly shadows cron except None
is equivalent on *
in crontab.
As per the example sets can be used to run at multiple of the given unit.
Note that second
defaults to 0
so you don’t in inadvertently run jobs every second and microsecond
defaults to 123456
so you don’t inadvertently run jobs every microsecond and so arq avoids enqueuing jobs
at the top of a second when the world is generally slightly busier.
Multiple Queues¶
Functions can be assigned to different queues, by default arq defines three queues:
HIGH_QUEUE
, DEFAULT_QUEUE
and LOW_QUEUE
which are prioritised by the worker in that order.
from arq import Actor, concurrent
class RegistrationEmail(Actor):
@concurrent
async def email_standard_user(self, user_id):
send_user_email(user_id)
@concurrent(Actor.HIGH_QUEUE)
async def email_premium_user(self, user_id):
send_user_email(user_id)
(Just a snippet, won’t run “as is”)
Direct Enqueuing¶
Functions can we enqueued directly whether or no they’re decorated with @concurrent
.
from arq import Actor
class FooBar(Actor):
async def foo(self, a, b, c):
print(a + b + c)
async def main():
foobar = FooBar()
await foobar.enqueue_job('foo', 1, 2, c=48, queue=Actor.LOW_QUEUE)
await foobar.enqueue_job('foo', 1, 2, c=48) # this will be queued in DEFAULT_QUEUE
await foobar.close()
(This script is almost complete except for loop.run_until_complete(main())
as above to run main
,
you would also need to define a worker to run the jobs)
See arq.main.Actor.enqueue_job()
for more details.
Worker Customisation¶
Workers can be customised in numerous ways, this is preferred to command line arguments as it’s easier to document and record.
from arq import BaseWorker
class Worker(BaseWorker):
# execute jobs from both Downloader and FooBar above
shadows = [Downloader, FooBar]
# allow lots and lots of jobs to run simultaniously, default 50
max_concurrent_tasks = 500
# force the worker to close quickly after a termination signal is received, default 6
shutdown_delay = 2
# jobs may not take more than 10 seconds, default 60
timeout_seconds = 10
# number of seconds between health checks, default 60
health_check_interval = 30
def logging_config(self, verbose):
conf = super().logging_config(verbose)
# alter logging setup to set arq.jobs level to WARNING
conf['loggers']['arq.jobs']['level'] = 'WARNING'
return conf
(This script is more-or-less complete,
provided Downloader
and FooBar
are defined and imported it should run “as is”)
See arq.worker.BaseWorker()
for more customisation options.
For more information on logging see arq.logs.default_log_config()
.