Python Celery - Weekly Celery Tutorials and How-tos: Auto-reload Celery on code changes

For a long time, my most frustrating developer experience with Celery was the lack of worker restart on code changes.

For example, Gunicorn supports a --reload argument. This setting causes workers to be restarted whenever your application code changes. Which is almost indispensable even when you are really disciplined TDD disciple.

Unfortunately, Celery does not suport such a reload option. Celery once had an --autoreload option but it was deprecated in version 3.1.0 or so - though the documents suggested otherwise for a long time. Which only added to the frustration and confusion.

As of any recent Celery version, we are on our own when we want to avoid manual worker restarts after code changes.

In this article I am showing you a simple workaround to get your Celery worker restarted on code changes. This will simplify your Celery development workflow and save you many Ctrl+C-Arrow-Up-Enter round trips.

watchdog and watchmedo

watchdog is a Python library to monitor file system events. When you create, edit, change or delete a file or a directory, watchdog raises an event that you can catch and handle. It runs on all big operating systems, including Linux, Max OS and Windows and works on Python 2.7 and 3.4+.

In addition to the actual Python API, watchdog comes with a utility script called watchmedo. We can run watchmedo on the command line to monitor a folder for file events.

Whenever an event is raised, watchmedo restarts another command. Which is precisely what we are after in order to restart our Celery worker on code (Python file) changes.

Install the watchdog package via pip:

> pip install watchdog

And confirm that watchmedo is available as a utility script on the command line.

> watchmedo --help

usage: watchmedo [-h] [--version]
                 {tricks-from,tricks,tricks-generate-yaml,generate-tricks-yaml,log,shell-command,auto-restart}
                 ...

positional arguments:
  {tricks-from,tricks,tricks-generate-yaml,generate-tricks-yaml,log,shell-command,auto-restart}
    tricks-from (tricks)
                        Subcommand to execute tricks from a tricks
                        configuration file. :param args: Command line argument
                        options.
    tricks-generate-yaml (generate-tricks-yaml)
                        Subcommand to generate Yaml configuration for tricks
                        named on the command line. :param args: Command line
                        argument options.
    log                 Subcommand to log file system events to the console.
                        :param args: Command line argument options.
    shell-command       Subcommand to execute shell commands in response to
                        file system events. :param args: Command line argument
                        options.
    auto-restart        Subcommand to start a long-running subprocess and
                        restart it on matched events. :param args: Command
                        line argument options.

optional arguments:
  -h, --help            show this help message and exit
  --version             show program's version number and exit

Copyright 2011 Yesudeep Mangalapilly <yesudeep@gmail.com>.
Copyright 2012 Google, Inc.

Licensed under the terms of the Apache license, version 2.0. Please see
LICENSE in the source code for more information.

Auto-restart Celery

watchmedo supports an auto-restart argument. With the auto-restart argument, watchmedo takes control of a long-running subprocess and restarts it on matched file system events. Have a look at watchmedo auto-restart --help for details.

Usually, I declare my Celery worker as app in a dedicated worker.py module and start the Celery worker with the celery worker command:

celery worker --app=worker.app --concurrency=1 --loglevel=INFO

Let’s modify the celery worker command to put watchmedo in the driver’s seat. We want watchmedo to restart the celery worker command on code-changes.

watchmedo auto-restart --directory=./ --pattern=*.py --recursive -- celery worker --app=worker.app --concurrency=1 --loglevel=INFO

Here, watchmedo monitors the current directory (--directory) and its subdirectories (--recursive) for changes in any of the .py (--pattern) files.

Whenever that happens, it kills the current celery worker and spins up a new one (celery worker --app=worker.app --concurrency=1 --loglevel=INFO).

Note -- before the command argument which tells watchmedo to not interpret the Celery arguments.

Happy Celery coding!

from Planet Python
via read more