What now ?¶

If you are a developer and want to leverage Circus in your own project, write plugins or hooks, go to fordevs.

If you are an ops and want to manage your processes using Circus, go to forops.

Contributions and Feedback¶

More on contributing: contribs.

Useful Links:

Documentation index¶

Installing Circus¶

Circus is a Python package which is published on PyPI - the Python Package Index.

The simplest way to install it is to use pip, a tool for installing and managing Python packages:

$ pip install circus

Or download the archive on PyPI, extract and install it manually with:

$ python setup.py install

If you want to try out Circus, see the examples.

If you are using debian or any debian based distribution, you also can use the ppa to install circus, it's at https://launchpad.net/~roman-imankulov/+archive/circus

zc.buildout¶

We provide a zc.buildout configuration, you can use it by simply running the bootstrap script, then calling buildout:

$ python bootstrap.py
$ bin/buildout

More on Requirements¶

Circus works with:

When you install circus, the latest versions of the Python dependencies will be pulled out for you.

You can also install them manually using the pip-requirements.txt file we provide:

$ pip install -r pip-requirements.txt

If you want to run the Web console you will need to install circus-web:

$ pip install circus-web

Tutorial¶

Step-by-step tutorial¶

The examples directory in the Circus repository contains many examples to get you started, but here's a full tutorial that gives you an overview of the features.

We're going to supervise a WSGI application.

Installation¶

Circus is tested on Mac OS X and Linux with the latest Python 2.6, 2.7, 3.2 and 3.3. To run a full Circus, you will also need libzmq, libevent & virtualenv.

On Debian-based systems:

$ sudo apt-get install libzmq-dev libevent-dev python-dev python-virtualenv

Create a virtualenv and install circus, circus-web and chaussette in it

$ virtualenv /tmp/circus
$ cd /tmp/circus
$ bin/pip install circus
$ bin/pip install circus-web
$ bin/pip install chaussette

Once this is done, you'll find a plethora of commands in the local bin dir.

Usage¶

Chaussette comes with a default Hello world app, try to run it:

$ bin/chaussette

You should be able to visit http://localhost:8080 and see hello world.

Stop Chaussette and add a circus.ini file in the directory containing:

[circus]
statsd = 1
httpd = 1
[watcher:webapp]
cmd = bin/chaussette --fd $(circus.sockets.web)
numprocesses = 3
use_sockets = True
[socket:web]
host = 127.0.0.1
port = 9999

This config file tells Circus to bind a socket on port 9999 and run 3 chaussettes workers against it. It also activates the Circus web dashboard and the statistics module.

Save it & run it using circusd:

$ bin/circusd --daemon circus.ini

Now visit http://127.0.0.1:9999, you should see the hello world app. The difference now is that the socket is managed by Circus and there are several web workers that are accepting connections against it.

NOTE:

You can also visit http://localhost:8080/ and enjoy the Circus web dashboard.

Interaction¶

Let's use the circusctl shell while the system is running:

$ bin/circusctl
circusctl 0.7.1
circusd-stats: active
circushttpd: active
webapp: active
(circusctl)

You get into an interactive shell. Type help to get all commands:

(circusctl) help
Documented commands (type help <topic>):
========================================
add     get            list         numprocesses  quit     rm      start   stop
decr    globaloptions  listen       numwatchers   reload   set     stats
dstats  incr           listsockets  options       restart  signal  status
Undocumented commands:
======================
EOF  help

Let's try basic things. Let's list the web workers processes and add a new one:

(circusctl) list webapp
13712,13713,13714
(circusctl) incr webapp
4
(circusctl) list webapp
13712,13713,13714,13973

Congrats, you've interacted with your Circus! Get off the shell with Ctrl+D and now run circus-top:

$ bin/circus-top

This is a top-like command to watch all your processes' memory and CPU usage in real time.

Hit Ctrl+C and now let's quit Circus completely via circus-ctl:

$ bin/circusctl quit
ok

Next steps¶

You can plug your own WSGI application instead of Chaussette's hello world simply by pointing the application callable.

Chaussette also comes with many backends like Gevent or Meinheld.

Read https://chaussette.readthedocs.org/ for all options.

Why should I use Circus instead of X ?¶

Coming from Supervisor¶

Supervisor is a very popular solution in the Python world and we're often asked how Circus compares with it.

If you are coming from Supervisor, this page tries to give an overview of how the tools differ.

Differences overview¶

Supervisor & Circus have the same goals - they both manage processes and provide a command-line script — respectively supervisord and circusd — that reads a configuration file, forks new processes and keep them alive.

Circus has an extra feature: the ability to bind sockets and let the processes it manages use them. This "pre-fork" model is used by many web servers out there, like Apache or Unicorn. Having this option in Circus can simplify a web app stack: all processes and sockets are managed by a single tool.

Both projects provide a way to control a running daemon via another script. respectively supervisorctl and circusctl. They also both have events and a way to subscribe to them. The main difference is the underlying technology: Supervisor uses XML-RPC for interacting with the daemon, while Circus uses ZeroMQ.

Circus & Supervisor both have a web interface to display what's going on. Circus' is more advanced because you can follow in real time what's going on and interact with the daemon. It uses web sockets and is developed in a separate project (circus-web.)

There are many other subtle differences in the core design, we might list here one day… In the meantime, you can learn more about circus internals in design.

Configuration¶

Both systems use an ini-like file as a configuration.

Here's a small example of running an application with Supervisor. In this case, the application will be started and restarted in case it crashes

[program:example]
command=npm start
directory=/home/www/my-server/
user=www-data
autostart=true
autorestart=true
redirect_stderr=True

In Circus, the same configuration is done by:

[watcher:example]
cmd=npm start
working_dir=/home/www/my-server/
user=www-data
stderr_stream.class=StdoutStream

Notice that the stderr redirection is slightly different in Circus. The tool does not have a tail feature like in Supervisor, but will let you hook any piece of code to deal with the incoming stream. You can create your own stream hook (as a Class) and do whatever you want with the incoming stream. Circus provides some built-in stream classes like StdoutStream, FileStream, WatchedFileStream, or TimedRotatingFileStream.

Circus for Ops¶

WARNING:

The first step to manage a Circus daemon is to write its configuration file. See configuration. If you are deploying a web stack, have a look at sockets.

Circus can be deployed using Python 2.6, 2.7, 3.2 or 3.3 - most deployments out there are done in 2.7. To learn how to deploy Circus, check out deployment.

To manage a Circus daemon, you should get familiar with the list of commands you can use in circusctl. Notice that you can have the same help online when you run circusctl as a shell.

We also provide circus-top, see cli and a nice web dashboard. see circushttpd.

Last, to get the most out of Circus, make sure to check out how to use plugins and hooks. See plugins and hooks.

Ops documentation index¶

Configuration¶

Circus can be configured using an ini-style configuration file.

Example:

[circus]
check_delay = 5
endpoint = tcp://127.0.0.1:5555
pubsub_endpoint = tcp://127.0.0.1:5556
include = \*.more.config.ini
umask = 002
[watcher:myprogram]
cmd = python
args = -u myprogram.py $(circus.wid) $(CIRCUS.ENV.VAR)
warmup_delay = 0
numprocesses = 5
# hook
hooks.before_start = my.hooks.control_redis
# will push in test.log the stream every 300 ms
stdout_stream.class = FileStream
stdout_stream.filename = test.log
# optionally rotate the log file when it reaches 1 gb
# and save 5 copied of rotated files
stdout_stream.max_bytes = 1073741824
stdout_stream.backup_count = 5
[env:myprogram]
PATH = $PATH:/bin
CAKE = lie
[plugin:statsd]
use = circus.plugins.statsd.StatsdEmitter
host = localhost
port = 8125
sample_rate = 1.0
application_name = example
[socket:web]
host = localhost
port = 8080

circus - single section¶

version: 1
disable_existing_loggers: false
formatters:


  simple:


    format: '%(asctime)s - %(name)s - [%(levelname)s] %(message)s'
handlers:


  logfile:


    class: logging.FileHandler


    filename: logoutput.txt


    level: DEBUG


    formatter: simple
loggers:


  circus:


    level: DEBUG


    handlers: [logfile]


    propagate: no
root:


  level: DEBUG


  handlers: [logfile]

watcher:NAME - as many sections as you want¶

socket:NAME - as many sections as you want¶

Once a socket is created, the ${circus.sockets.NAME} string can be used in the command (cmd or args) of a watcher. Circus will replace it by the FD value. The watcher must also have use_sockets set to True otherwise the socket will have been closed and you will get errors when the watcher tries to use it.

Example:

[watcher:webworker]
cmd = chaussette --fd $(circus.sockets.webapp) chaussette.util.bench_app
use_sockets = True
[socket:webapp]
host = 127.0.0.1
port = 8888

plugin:NAME - as many sections as you want¶

Circus comes with a few pre-shipped plugins but you can also extend them easily by developing your own.

env or env[:WATCHERS] - as many sections as you want¶

Section responsible for delivering environment variable to run processes.

Example:

[watcher:worker1]
cmd = ping 127.0.0.1
[watcher:worker2]
cmd = ping 127.0.0.1
[env]
CAKE = lie

The variable CAKE will propagated to all watchers defined in config file.

WATCHERS can be a comma separated list of watcher sections to apply this environment to. if multiple env sections match a watcher, they will be combine in the order they appear in the configuration file. later entries will take precedence.

Example:

[watcher:worker1]
cmd = ping 127.0.0.1
[watcher:worker2]
cmd = ping 127.0.0.1
[env:worker1,worker2]
PATH = /bin
[env:worker1]
PATH = $PATH
[env:worker2]
CAKE = lie

worker1 will be run with PATH = $PATH (expanded from the environment circusd was run in) worker2 will be run with PATH = /bin and CAKE = lie

It's possible to use wildcards as well.

Example:

[watcher:worker1]
cmd = ping 127.0.0.1
[watcher:worker2]
cmd = ping 127.0.0.1
[env:worker*]
PATH = /bin

Both worker1 and worker2 will be run with PATH = /bin

Using environment variables¶

When writing your configuration file, you can use environment variables defined in the env section or in os.environ itself.

You just have to use the circus.env. prefix.

Example:

[watcher:worker1]
cmd = $(circus.env.shell)
[watcher:worker2]
baz = $(circus.env.user)
bar = $(circus.env.yeah)
sup = $(circus.env.oh)
[socket:socket1]
port = $(circus.env.port)
[plugin:plugin1]
use = some.path
parameter1 = $(circus.env.plugin_param)
[env]
yeah = boo
[env:worker2]
oh = ok

If a variable is defined in several places, the most specialized value has precedence: a variable defined in env:XXX will override a variable defined in env, which will override a variable defined in os.environ.

environment substitutions can be used in any section of the configuration in any section variable.

Formatting the commands and arguments with dynamic variables¶

As you may have seen, it is possible to pass some information that are computed dynamically when running the processes. Among other things, you can get the worker id (WID) and all the options that are passed to the Process. Additionally, it is possible to access the options passed to the Watcher which instanciated the process.

NOTE:

For instance, if you want to access some variables that are contained in the environment, you would need to do it with a setting like this:

cmd = "make-me-a-coffee --sugar $(CIRCUS.ENV.SUGAR_AMOUNT)"

This works with both cmd and args.

Important:

Stream configuration¶

Simple stream class like QueueStream and StdoutStream don't have specific attributes but some other stream class may have some:

FileStream¶

NOTE:

Example:

[watcher:myprogram]
cmd = python -m myapp.server
stdout_stream.class = FileStream
stdout_stream.filename = test.log
stdout_stream.time_format = %Y-%m-%d %H:%M:%S
stdout_stream.max_bytes = 1073741824
stdout_stream.backup_count = 5

WatchedFileStream¶

NOTE:

Example:

[watcher:myprogram]
cmd = python -m myapp.server
stdout_stream.class = WatchedFileStream
stdout_stream.filename = test.log
stdout_stream.time_format = %Y-%m-%d %H:%M:%S

TimedRotatingFileStream¶

NOTE:

Example:

[watcher:myprogram]
cmd = python -m myapp.server
stdout_stream.class = TimedRotatingFileStream
stdout_stream.filename = test.log
stdout_stream.time_format = %Y-%m-%d %H:%M:%S
stdout_stream.utc = True
stdout_stream.rotate_when = H
stdout_stream.rotate_interval = 1

FancyStdoutStream¶

Example:

[watcher:myprogram]
cmd = python -m myapp.server
stdout_stream.class = FancyStdoutStream
stdout_stream.color = green
stdout_stream.time_format = %Y/%m/%d | %H:%M:%S

Commands¶

At the epicenter of circus lives the command systems. circusctl is just a zeromq client, and if needed you can drive programmaticaly the Circus system by writing your own zmq client.

All messages are JSON mappings.

For each command below, we provide a usage example with circusctl but also the input / output zmq messages.

circus-ctl commands¶

Add a watcher¶

This command add a watcher dynamically to a arbiter.

ZMQ Message¶

{


    "command": "add",


    "properties": {


        "cmd": "/path/to/commandline --option"


        "name": "nameofwatcher"


        "args": [],


        "options": {},


        "start": false


    }
}

A message contains 2 properties:

The response return a status "ok".

Command line¶

$ circusctl add [--start] <name> <cmd>

Options¶

Decrement the number of processes in a watcher¶

This comment decrement the number of processes in a watcher by <nbprocess>, 1 being the default.

ZMQ Message¶

{


    "command": "decr",


    "propeties": {


        "name": "<watchername>"


        "nb": <nbprocess>


        "waiting": False


    }
}

The response return the number of processes in the 'numprocesses` property:

{ "status": "ok", "numprocesses": <n>, "time", "timestamp" }

Command line¶

$ circusctl decr <name> [<nb>] [--waiting]

Options¶

Get circusd stats¶

You can get at any time some statistics about circusd with the dstat command.

ZMQ Message¶

To get the circusd stats, simply run:

{


    "command": "dstats"
}

The response returns a mapping the property "infos" containing some process informations:

{


  "info": {


    "children": [],


    "cmdline": "python",


    "cpu": 0.1,


    "ctime": "0:00.41",


    "mem": 0.1,


    "mem_info1": "3M",


    "mem_info2": "2G",


    "nice": 0,


    "pid": 47864,


    "username": "root"


  },


  "status": "ok",


  "time": 1332265655.897085
}

Command Line¶

$ circusctl dstats

Get the value of specific watcher options¶

This command can be used to query the current value of one or more watcher options.

ZMQ Message¶

{


    "command": "get",


    "properties": {


        "keys": ["key1, "key2"]


        "name": "nameofwatcher"


    }
}

A request message contains two properties:

The response object has a property options which is a dictionary of option names and values.

eg:

{


    "status": "ok",


    "options": {


        "graceful_timeout": 300,


        "send_hup": True,


    },


    time': 1332202594.754644
}

Command line¶

$ circusctl get <name> <key1> <key2>

Get the arbiter options¶

This command return the arbiter options

ZMQ Message¶

{


    "command": "globaloptions",


    "properties": {


        "key1": "val1",


        ..


    }
}

A message contains 2 properties:

The response return an object with a property "options" containing the list of key/value returned by circus.

eg:

{


    "status": "ok",


    "options": {


        "check_delay": 1,


        ...


    },


    time': 1332202594.754644
}

Command line¶

$ circusctl globaloptions

Options¶

Options Keys are:

Increment the number of processes in a watcher¶

This comment increment the number of processes in a watcher by <nbprocess>, 1 being the default

ZMQ Message¶

{


    "command": "incr",


    "properties": {


        "name": "<watchername>",


        "nb": <nbprocess>,


        "waiting": False


    }
}

The response return the number of processes in the 'numprocesses` property:

{ "status": "ok", "numprocesses": <n>, "time", "timestamp" }

Command line¶

$ circusctl incr <name> [<nb>] [--waiting]

Options¶

Create shell into circusd process¶

This command is only useful if you have the ipython package installed.

Command Line¶

$ circusctl ipython

Get list of watchers or processes in a watcher¶

ZMQ Message¶

To get the list of all the watchers:

{


    "command": "list",
}

To get the list of active processes in a watcher:

{


    "command": "list",


    "properties": {


        "name": "nameofwatcher",


    }
}

The response return the list asked. the mapping returned can either be 'watchers' or 'pids' depending the request.

Command line¶

$ circusctl list [<name>]

Subscribe to a watcher event¶

ZMQ¶

At any moment you can subscribe to a circus event. Circus provides a PUB/SUB feed on which any clients can subscribe. The subscriber endpoint URI is set in the circus.ini configuration file.

Events are pubsub topics:

All events messages are in a json struct.

Command line¶

The client has been updated to provide a simple way to listen on the events:

circusctl listen [<topic>, ...]

Example of result:¶

$ circusctl listen tcp://127.0.0.1:5556
watcher.refuge.spawn: {u'process_id': 6, u'process_pid': 72976,


                       u'time': 1331681080.985104}
watcher.refuge.spawn: {u'process_id': 7, u'process_pid': 72995,


                       u'time': 1331681086.208542}
watcher.refuge.spawn: {u'process_id': 8, u'process_pid': 73014,


                       u'time': 1331681091.427005}

Get the list of sockets¶

ZMQ Message¶

To get the list of sockets:

{


    "command": "listsockets",
}

The response return a list of json mappings with keys for fd, name, host and port.

Command line¶

$ circusctl listsockets

Get the number of processes¶

Get the number of processes in a watcher or in a arbiter

ZMQ Message¶

{


    "command": "numprocesses",


    "propeties": {


        "name": "<watchername>"


    }
}

The response return the number of processes in the 'numprocesses` property:

{ "status": "ok", "numprocesses": <n>, "time", "timestamp" }

If the property name isn't specified, the sum of all processes managed is returned.

Command line¶

$ circusctl numprocesses [<name>]

Options¶

Get the number of watchers¶

Get the number of watchers in a arbiter

ZMQ Message¶

{


    "command": "numwatchers",
}

The response return the number of watchers in the 'numwatchers` property:

{ "status": "ok", "numwatchers": <n>, "time", "timestamp" }

Command line¶

$ circusctl numwatchers

Get the value of all options for a watcher¶

This command returns all option values for a given watcher.

ZMQ Message¶

{


    "command": "options",


    "properties": {


        "name": "nameofwatcher",


    }
}

A message contains 1 property:

The response object has a property options which is a dictionary of option names and values.

eg:

{


    "status": "ok",


    "options": {


        "graceful_timeout": 300,


        "send_hup": True,


        ...


    },


    time': 1332202594.754644
}

Command line¶

$ circusctl options <name>

Options¶

Options Keys are:

Quit the arbiter immediately¶

When the arbiter receive this command, the arbiter exit.

ZMQ Message¶

{


    "command": "quit",


    "waiting": False
}

The response return the status "ok".

If waiting is False (default), the call will return immediately after calling stop_signal on each process.

If waiting is True, the call will return only when the stop process is completely ended. Because of the graceful_timeout option, it can take some time.

Command line¶

$ circusctl quit [--waiting]

Reload the arbiter or a watcher¶

This command reloads all the process in a watcher or all watchers. This will happen in one of 3 ways:

ZMQ Message¶

{


    "command": "reload",


    "properties": {


        "name": '<name>",


        "graceful": true,


        "sequential": false,


        "waiting": False


    }
}

The response return the status "ok". If the property graceful is set to true the processes will be exited gracefully.

If the property name is present, then the reload will be applied to the watcher.

Command line¶

$ circusctl reload [<name>] [--terminate] [--waiting]


                            [--sequential]

Options¶

Reload the configuration file¶

This command reloads the configuration file, so changes in the configuration file will be reflected in the configuration of circus.

ZMQ Message¶

{


    "command": "reloadconfig",


    "waiting": False
}

The response return the status "ok". If the property graceful is set to true the processes will be exited gracefully.

Command line¶

$ circusctl reloadconfig [--waiting]

Restart the arbiter or a watcher¶

This command restart all the process in a watcher or all watchers. This funtion simply stop a watcher then restart it.

ZMQ Message¶

{


    "command": "restart",


    "properties": {


        "name": "<name>",


        "waiting": False,


        "match": "[simple|glob|regex]"


    }
}

The response return the status "ok".

If the property name is present, then the reload will be applied to the watcher.

If waiting is False (default), the call will return immediately after calling stop_signal on each process.

If waiting is True, the call will return only when the restart process is completely ended. Because of the graceful_timeout option, it can take some time.

The match parameter can have the value simple for string compare, glob for wildcard matching (default) or regex for regex matching.

Command line¶

$ circusctl restart [name] [--waiting] [--match=simple|glob|regex]

Options¶

Remove a watcher¶

This command removes a watcher dynamically from the arbiter. The watchers are gracefully stopped by default.

ZMQ Message¶

{


    "command": "rm",


    "properties": {


        "name": "<nameofwatcher>",


        "nostop": False,


        "waiting": False


    }
}

The response return a status "ok".

If nostop is True (default: False), the processes for the watcher will not be stopped - instead the watcher will just be forgotten by circus and the watcher processes will be responsible for stopping themselves. If nostop is not specified or is False, then the watcher processes will be stopped gracefully.

If waiting is False (default), the call will return immediately after starting to remove and stop the corresponding watcher.

If waiting is True, the call will return only when the remove and stop process is completely ended. Because of the graceful_timeout option, it can take some time.

Command line¶

$ circusctl rm <name> [--waiting] [--nostop]

Options¶

Set a watcher option¶

ZMQ Message¶

{


    "command": "set",


    "properties": {


        "name": "nameofwatcher",


        "options": {


            "key1": "val1",


            ..


        }


        "waiting": False


    }
}

The response return the status "ok". See the command Options for a list of key to set.

Command line¶

$ circusctl set <name> <key1> <value1> <key2> <value2> --waiting

Send a signal¶

This command allows you to send a signal to all processes in a watcher, a specific process in a watcher or its children.

ZMQ Message¶

To send a signal to all the processes for a watcher:

{


    "command": "signal",


    "property": {


        "name": <name>,


        "signum": <signum>
}

To send a signal to a process:

{


    "command": "signal",


    "property": {


        "name": <name>,


        "pid": <processid>,


        "signum": <signum>
}

An optional property "children" can be used to send the signal to all the children rather than the process itself:

{


    "command": "signal",


    "property": {


        "name": <name>,


        "pid": <processid>,


        "signum": <signum>,


        "children": True
}

To send a signal to a process child:

{


    "command": "signal",


    "property": {


        "name": <name>,


        "pid": <processid>,


        "signum": <signum>,


        "child_pid": <childpid>,
}

It is also possible to send a signal to all the children of the watcher:

{


    "command": "signal",


    "property": {


        "name": <name>,


        "signum": <signum>,


        "children": True
}

Lastly, you can send a signal to the process and its children, with the recursive option:

{


    "command": "signal",


    "property": {


        "name": <name>,


        "signum": <signum>,


        "recursive": True
}

Command line¶

$ circusctl signal <name> [<pid>] [--children]


        [--recursive] <signum>

Options:¶

Start the arbiter or a watcher¶

This command starts all the processes in a watcher or all watchers.

ZMQ Message¶

{


    "command": "start",


    "properties": {


        "name": '<name>",


        "waiting": False,


        "match": "[simple|glob|regex]"


    }
}

The response return the status "ok".

If the property name is present, the watcher will be started.

If waiting is False (default), the call will return immediately after calling start on each process.

If waiting is True, the call will return only when the start process is completely ended. Because of the graceful_timeout option, it can take some time.

The match parameter can have the value simple for string compare, glob for wildcard matching (default) or regex for regex matching.

Command line¶

$ circusctl restart [name] [--waiting] [--match=simple|glob|regex]

Options¶

Get process infos¶

You can get at any time some statistics about your processes with the stat command.

ZMQ Message¶

To get stats for all watchers:

{


    "command": "stats"
}

To get stats for a watcher:

{


    "command": "stats",


    "properties": {


        "name": <name>


    }
}

To get stats for a process:

{


    "command": "stats",


    "properties": {


        "name": <name>,


        "process": <processid>


    }
}

Stats can be extended with the extended_stats hook but extended stats need to be requested:

{


    "command": "stats",


    "properties": {


        "name": <name>,


        "process": <processid>,


        "extended": True


    }
}

The response retun an object per process with the property "info" containing some process informations:

{


  "info": {


    "children": [],


    "cmdline": "python",


    "cpu": 0.1,


    "ctime": "0:00.41",


    "mem": 0.1,


    "mem_info1": "3M",


    "mem_info2": "2G",


    "nice": 0,


    "pid": 47864,


    "username": "root"


  },


  "process": 5,


  "status": "ok",


  "time": 1332265655.897085
}

Command Line¶

$ circusctl stats [--extended] [<watchername>] [<processid>]

Get the status of a watcher or all watchers¶

This command start get the status of a watcher or all watchers.

ZMQ Message¶

{


    "command": "status",


    "properties": {


        "name": '<name>",


    }
}

The response return the status "active" or "stopped" or the status / watchers.

Command line¶

$ circusctl status [<name>]

Options¶

Example¶

$ circusctl status dummy
active
$ circusctl status
dummy: active
dummy2: active
refuge: active

Stop watchers¶

This command stops a given watcher or all watchers.

ZMQ Message¶

{


    "command": "stop",


    "properties": {


        "name": "<name>",


        "waiting": False,


        "match": "[simple|glob|regex]"


    }
}

The response returns the status "ok".

If the name property is present, then the stop will be applied to the watcher corresponding to that name. Otherwise, all watchers will get stopped.

If waiting is False (default), the call will return immediatly after calling stop_signal on each process.

If waiting is True, the call will return only when the stop process is completly ended. Because of the graceful_timeout option, it can take some time.

The match parameter can have the value simple for string compare, glob for wildcard matching (default) or regex for regex matching.

Command line¶

$ circusctl stop [name] [--waiting] [--match=simple|glob|regex]

Options¶

CLI tools¶

circus-top¶

circus-top is a top-like console you can run to watch live your running Circus system. It will display the CPU, Memory usage and socket hits if you have some.

Example of output:

-----------------------------------------------------------------------
circusd-stats


 PID                 CPU (%)             MEMORY (%)
14252                 0.8                 0.4


                      0.8 (avg)           0.4 (sum)
dummy


 PID                 CPU (%)             MEMORY (%)
14257                 78.6                0.1
14256                 76.6                0.1
14258                 74.3                0.1
14260                 71.4                0.1
14259                 70.7                0.1


                      74.32 (avg)         0.5 (sum)
----------------------------------------------------------------------

circus-top is a read-only console. If you want to interact with the system, use circusctl.

circusctl¶

circusctl can be used to run any command listed in commands . For example, you can get a list of all the watchers, you can do

$ circusctl list

Besides supporting a handful of options you can also specify the endpoint circusctl should use using the CIRCUSCTL_ENDPOINT environment variable.

The Web Console¶

Circus comes with a Web Console that can be used to manage the system.

The Web Console lets you:

NOTE:

[circus]
statsd = True

The web console is its own package, you need to install:

$ pip install circus-web

To enable the console, add a few options in the Circus ini file:

[circus]
httpd = True
httpd_host = localhost
httpd_port = 8080

httpd_host and httpd_port are optional, and default to localhost and 8080.

If you want to run the web app on its own, just run the circushttpd script:

$ circushttpd
Bottle server starting up...
Listening on http://localhost:8080/
Hit Ctrl-C to quit.

By default the script will run the Web Console on port 8080, but the --port option can be used to change it.

Using the console¶

Once the script is running, you can open a browser and visit http://localhost:8080. You should get this screen: [image]

The Web Console is ready to be connected to a Circus system, given its endpoint. By default the endpoint is tcp://127.0.0.1:5555.

Once you hit Connect, the web application will connect to the Circus system.

With the Web Console logged in, you should get a list of watchers, and a real-time status of the two Circus processes (circusd and circusd-stats).

You can click on the status of each watcher to toggle it from Active (green) to Inactive (red). This change is effective immediatly and let you start & stop watchers.

If you click on the watcher name, you will get a web page for that particular watcher, with its processes:

On this screen, you can add or remove processes, and kill existing ones.

Last but not least, you can add a brand new watcher by clicking on the Add Watcher link in the left menu: .SS Running behind Nginx

Nginx can act as a proxy and security layer in front of circus-web.

NOTE:

Nginx >= 1.3.13¶

As of Nginx>=1.3.13 websocket support is built-in, so there is no need to combine Nginx with Varnish or HAProxy. An example Nginx config with websocket support:

upstream circusweb_server {


  server 127.0.0.1:8080;
}
server {


 listen   80;


 server_name  _;


 location / {


   proxy_pass http://circusweb_server;


   proxy_http_version 1.1;


   proxy_set_header Upgrade $http_upgrade;


   proxy_set_header Connection "upgrade";


   proxy_set_header Host $host;


   proxy_set_header X-Real-IP $remote_addr;


   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;


   proxy_set_header X-Forwarded-Proto http;


   proxy_redirect off;


  }


 location ~/media/\*(.png|.jpg|.css|.js|.ico)$ {


   alias /path_to_site-packages/circusweb/media/;


  }
}

Nginx < 1.3.13¶

Nginx versions < 1.3.13 do not have websocket support built-in.

To provide websocket support for circus-web when using Nginx < 1.3.13, you can combine Nginx with Varnish or HAProxy. That is, Nginx in front of circus-web, with Varnish or HAProxy in front of Nginx.

The example below shows the combined Nginix and Varnish configuration required to proxy circus-web and provide websocket support.

Nginx configuration:

upstream circusweb_server {


  server 127.0.0.1:8080;
}
server {


 listen   8001;


 server_name  _;


 location / {


    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;


    proxy_set_header Host $http_host;


    proxy_redirect off;


    proxy_pass http://circusweb_server;


  }


 location ~/media/\*(.png|.jpg|.css|.js|.ico)$ {


   alias /path_to_site-packages/circusweb/media/;


  }
}

If you want more Nginx configuration options, see http://wiki.nginx.org/HttpProxyModule.

Varnish configuration:

backend default {


    .host = "127.0.0.1";


    .port = "8001";
}
backend socket {


    .host = "127.0.0.1";


    .port = "8080";


    .connect_timeout = 1s;


    .first_byte_timeout = 2s;


    .between_bytes_timeout = 60s;
}
sub vcl_pipe {


     if (req.http.upgrade) {


         set bereq.http.upgrade = req.http.upgrade;


     }
}
sub vcl_recv {


    if (req.http.Upgrade ~ "(?i)websocket") {


        set req.backend = socket;


      return (pipe);


    }
}

In the Varnish configuration example above two backends are defined. One serving the web console and one serving the socket connections. Web console requests are bound to port 8001. The Nginx 'server' directive should be configured to listen on port 8001.

Websocket connections are upgraded and piped directly to the circushttpd process listening on port 8080 by Varnish. i.e. bypassing the Nginx proxy.

Ubuntu¶

Since the version 13.10 (Saucy), Ubuntu includes Nginx with websocket support in its own repositories. For older versions, you can install Nginx>=1.3.13 from the official Nginx stable PPA, as so:

sudo apt-get install python-software-properties
sudo add-apt-repository ppa:nginx/stable
sudo apt-get update
sudo apt-get install nginx
nginx -v

Password-protect circushttpd¶

As explained in the Security page, running circushttpd is pretty unsafe. We don't provide any security in Circus itself, but you can protect your console at the NGinx level, by using http://wiki.nginx.org/HttpAuthBasicModule

Example:

location / {


    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;


    proxy_set_header Host $http_host;


    proxy_set_header X-Forwarded-Host: $http_host;


    proxy_set_header X-Forwarded-Proto: $scheme;


    proxy_redirect off;


    proxy_pass http://127.0.0.1:8080;


    auth_basic            "Restricted";


    auth_basic_user_file  /path/to/htpasswd;
}

The htpasswd file contains users and their passwords, and a password prompt will pop when you access the console.

You can use Apache's htpasswd script to edit it, or the Python script they provide at: http://trac.edgewall.org/browser/trunk/contrib/htpasswd.py

However, there's no native support for the combined use of HTTP Authentication and WebSockets (the server will throw HTTP 401 error codes). A workaround is to disable such authentication for the socket.io server.

Example (needs to be added before the previous rule):

location /socket.io {


    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;


    proxy_set_header Host $http_host;


    proxy_set_header X-Forwarded-Host: $http_host;


    proxy_set_header X-Forwarded-Proto: $scheme;


    proxy_redirect off;


    proxy_pass http://127.0.0.1:8080;
}

Of course that's just one way to protect your web console, you could use many other techniques.

Extending the web console¶

We picked bottle to build the webconsole, mainly because it's a really tiny framework that doesn't do much. By having a look at the code of the web console, you'll eventually find out that it's really simple to understand.

Here is how it's split:

If you want to add a feature in the web console you can reuse the code that's existing. A few tools are at your disposal to ease the process:

The StatsNamespace class is responsible for managing the websocket communication on the server side. Its documentation should help you to understand what it does.

Working with sockets¶

Circus can bind network sockets and manage them as it does for processes.

The main idea is that a child process that's created by Circus to run one of the watcher's command can inherit from all the opened file descriptors.

That's how Apache or Unicorn works, and many other tools out there.

Goal¶

The goal of having sockets managed by Circus is to be able to manage network applications in Circus exactly like other applications.

For example, if you use Circus with Chaussette -- a WGSI server, you can get a very fast web server running and manage "Web Workers" in Circus as you would do for any other process.

Splitting the socket managment from the network application itself offers a lot of opportunities to scale and manage your stack.

Design¶

The gist of the feature is done by binding the socket and start listening to it in circusd:

import socket
sock = socket.socket(FAMILY, TYPE)
sock.bind((HOST, PORT))
sock.listen(BACKLOG)
fd = sock.fileno()

Circus then keeps track of all the opened fds, and let the processes it runs as children have access to them if they want.

If you create a small Python network script that you intend to run in Circus, it could look like this:

import socket
import sys
fd = int(sys.argv[1])   # getting the FD from circus
sock = socket.fromfd(fd, FAMILY, TYPE)
# dealing with one request at a time
while True:


    conn, addr = sock.accept()


    request = conn.recv(1024)


    .. do something ..


    conn.sendall(response)


    conn.close()

Then Circus could run like this:

[circus]
check_delay = 5
endpoint = tcp://127.0.0.1:5555
pubsub_endpoint = tcp://127.0.0.1:5556
stats_endpoint = tcp://127.0.0.1:5557
[watcher:dummy]
cmd = mycoolscript $(circus.sockets.foo)
use_sockets = True
warmup_delay = 0
numprocesses = 5
[socket:foo]
host = 127.0.0.1
port = 8888

$(circus.sockets.foo) will be replaced by the FD value once the socket is created and bound on the 8888 port.

NOTE:

((circus.sockets.foo))

Real-world example¶

Chaussette is the perfect Circus companion if you want to run your WSGI application.

Once it's installed, running 5 meinheld workers can be done by creating a socket and calling the chaussette command in a worker, like this:

[circus]
endpoint = tcp://127.0.0.1:5555
pubsub_endpoint = tcp://127.0.0.1:5556
stats_endpoint = tcp://127.0.0.1:5557
[watcher:web]
cmd = chaussette --fd $(circus.sockets.web) --backend meinheld mycool.app
use_sockets = True
numprocesses = 5
[socket:web]
host = 0.0.0.0
port = 8000

We did not publish benchmarks yet, but a Web cluster managed by Circus with a Gevent or Meinheld backend is as fast as any pre-fork WSGI server out there.

Using built-in plugins¶

Circus comes with a few built-in plugins. This section presents these plugins and their configuration options.

Statsd¶

FullStats¶

RedisObserver¶

HttpObserver¶

ResourceWatcher¶

Example:

[circus]
; ...
[watcher:program]
cmd = sleep 120
[plugin:myplugin]
use = circus.plugins.resource_watcher.ResourceWatcher
watcher = program
min_cpu = 10
max_cpu = 70
min_mem = 0
max_mem = 20

Watchdog¶

Flapping¶

Options can be overriden in the watcher section using a flapping. prefix. For instance, here is how you would configure a specific max_retry value for nginx:

[watcher:nginx]
cmd = /path/to/nginx
flapping.max_retry = 2
[watcher:myscript]
cmd = ./my_script.py
; ... other watchers
[plugin:flapping]
use = circus.plugins.flapping.Flapping
max_retry = 5

CommandReloader¶

Deployment¶

Although the Circus daemon can be managed with the circusd command, it's easier to have it start on boot. If your system supports Upstart, you can create this Upstart script in /etc/init/circus.conf.

start on filesystem and net-device-up IFACE=lo
stop on runlevel [016]
respawn
exec /usr/local/bin/circusd /etc/circus/circusd.ini

This assumes that circusd.ini is located at /etc/circus/circusd.ini. After rebooting, you can control circusd with the service command:

# service circus start/stop/restart

If your system supports systemd, you can create this systemd unit file under /etc/systemd/system/circus.service.

[Unit]
Description=Circus process manager
After=syslog.target network.target nss-lookup.target
[Service]
Type=simple
ExecReload=/usr/bin/circusctl reload
ExecStart=/usr/bin/circusd /etc/circus/circus.ini
Restart=always
RestartSec=5
[Install]
WantedBy=default.target

A reboot isn't required if you run the daemon-reload command below:

# systemctl --system daemon-reload

Then circus can be managed via:

# systemctl start/stop/status/reload circus

Recipes¶

This section will contain recipes to deploy Circus. Until then you can look at Pete's Puppet recipe or at Remy's Chef recipe

Papa Process Kernel¶

One problem common to process managers is that you cannot restart the process manager without restarting all of the processes it manages. This makes it difficult to deploy a new version of Circus or new versions of any of the libraries on which it depends.

If you are on a Unix-type system, Circus can use the Papa process kernel. When used, Papa will create a long-lived daemon that will serve as the host for any processes and sockets you create with it. If circus is shutdown, Papa will maintain everything it is hosting.

Setup¶

Start by installing the papa and setproctitle modules:

pip install papa
pip install setproctitle

The setproctitle module is optional. It will be used if present to rename the Papa daemon for top and ps to something like "papa daemon from circusd". If you do not install the setproctitle module, that title will be the command line of the process that launched it. Very confusing.

Once Papa is installed, add use_papa=true to your critical processes and sockets. Generally you want to house all of the processes of your stack in Papa, and none of the Circus support processes such as the flapping and stats plugins.

[circus]
loglevel = info
[watcher:nginx]
cmd = /usr/local/nginx/sbin/nginx -p /Users/scottmax/Source/service-framework/Common/conf/nginx -c /Users/scottmax/Source/service-framework/Common/conf/nginx/nginx.conf
warmup_delay = 3
graceful_timeout = 10
max_retry = 5
singleton = true
send_hup = true
stop_signal = QUIT
stdout_stream.class = FileStream
stdout_stream.filename = /var/logs/web-server.log
stdout_stream.max_bytes = 10000000
stdout_stream.backup_count = 10
stderr_stream.class = FileStream
stderr_stream.filename = /var/logs/web-server-error.log
stderr_stream.max_bytes = 1000000
stderr_stream.backup_count = 10
active = true
use_papa = true
[watcher:logger]
cmd = /my_service/env/bin/python logger.py run
working_dir = /my_service
graceful_timeout = 10
singleton = true
stop_signal = INT
stdout_stream.class = FileStream
stdout_stream.filename = /var/logs/logger.log
stdout_stream.max_bytes = 10000000
stdout_stream.backup_count = 10
stderr_stream.class = FileStream
stderr_stream.filename = /var/logs/logger.log
stderr_stream.max_bytes = 1000000
stderr_stream.backup_count = 10
priority = 50
use_papa = true
[watcher:web_app]
cmd = /my_service/env/bin/uwsgi --ini uwsgi-live.ini --socket fd://$(circus.sockets.web) --stats 127.0.0.1:809$(circus.wid)
working_dir = /my_service/web_app
graceful_timeout=10
stop_signal = QUIT
use_sockets = True
stdout_stream.class = FileStream
stdout_stream.filename = /var/logs/web_app.log
stdout_stream.max_bytes = 10000000
stdout_stream.backup_count = 10
stderr_stream.class = FileStream
stderr_stream.filename = /var/logs/web_app.log
stderr_stream.max_bytes = 1000000
stderr_stream.backup_count = 10
hooks.after_spawn = examples.uwsgi_lossless_reload.children_started
hooks.before_signal = examples.uwsgi_lossless_reload.clean_stop
hooks.extended_stats = examples.uwsgi_lossless_reload.extended_stats
priority = 40
use_papa = true
[socket:web]
path = /my_service/sock/uwsgi
use_papa = true
[plugin:flapping]
use = circus.plugins.flapping.Flapping
window = 10
priority = 1000

NOTE:

Design Goal¶

Papa is designed to be very minimalist in features and requirements. It does:

It does not:

Papa requires no third-party libraries so it can run on just the standard Python library. It can make use of the setproctitle package but that is only used for making the title prettier for ps and top and is not essential.

The functionality has been kept to a minimum so that you should never need to restart the Papa daemon. As much of the functionality has been pushed to the client library as possible. That way you should be able to deploy a new copy of Papa for new client features without needing to restart the Papa daemon. Papa is meant to be a pillar of stability in a changing sea of 3rd party libraries.

Operation¶

Most things remain unchanged whether you use Papa or not. You can still start and stop processes. You can still get status and stats for processes. The main thing that changes is that when you do circusctl quit, all of the Papa processes are left running. When you start circusd back up, those processes are recovered.

NOTE:

Logging¶

While Circus is shut down, Papa will store up to 2M of output per process. Then it will start dumping the oldest data. When you restart Circus, this cached output will be quickly retrieved and sent to the output streams. Papa requires that receipt of output be acknowledged, so you should not lose any output during a shutdown.

Not only that, but Papa saves the timestamp of the output. Circus has been enhanced to take advantage of timestamp data if present. So if you are writing the output to log files or somewhere, your timestamps should all be correct.

Problems¶

If you use the incr or decr command to change the process count for a watcher, this will be reset to the level specified in the INI file when circusd is restarted.

Also, I have experienced problems with the combination of copy_env and virtualenv. You may note that the INI sample above circumvents this issue with explicit paths.

Telnet Interface¶

Papa has a basic command-line interface that you can access through telnet:

telnet localhost 20202
help

Circus for developers¶

Using Circus as a library¶

Circus provides high-level classes and functions that will let you manage processes in your own applications.

For example, if you want to run four processes forever, you could write:

from circus import get_arbiter
myprogram = {"cmd": "python myprogram.py", "numprocesses": 4}
arbiter = get_arbiter([myprogram])
try:


    arbiter.start()
finally:


    arbiter.stop()

This snippet will run four instances of myprogram and watch them for you, restarting them if they die unexpectedly.

To learn more about this, see library

Extending Circus¶

It's easy to extend Circus to create a more complex system, by listening to all the circusd events via its pub/sub channel, and driving it via commands.

That's how the flapping feature works for instance: it listens to all the processes dying, measures how often it happens, and stops the incriminated watchers after too many restarts attempts.

Circus comes with a plugin system to help you write such extensions, and a few built-in plugins you can reuse. See plugins.

You can also have a more subtile startup and shutdown behavior by using the hooks system that will let you run arbitrary code before and after some processes are started or stopped. See hooks.

Last but not least, you can also add new commands. See addingcmds.

Developers Documentation Index¶

Circus Library¶

The Circus package is composed of a high-level get_arbiter() function and many classes. In most cases, using the high-level function should be enough, as it creates everything that is needed for Circus to run.

You can subclass Circus' classes if you need more granularity than what is offered by the configuration.

The get_arbiter function¶

get_arbiter() is just a convenience on top of the various circus classes. It creates a arbiter (class Arbiter) instance with the provided options, which in turn runs a single Watcher with a single Process.

Example:

from circus import get_arbiter
arbiter = get_arbiter([{"cmd": "myprogram", "numprocesses": 3}])
try:


    arbiter.start()
finally:


    arbiter.stop()

Classes¶

Circus provides a series of classes you can use to implement your own process manager:

Example:

>>> from circus.process import Process
>>> process = Process('Top', 'top', shell=True)
>>> process.age()
3.0107998847961426
>>> process.info()
'Top: 6812  N/A tarek Zombie N/A N/A N/A N/A N/A'
>>> process.status
1
>>> process.stop()
>>> process.status
2
>>> process.info()
'No such process (stopped?)'

Writing plugins¶

Circus comes with a plugin system which lets you interact with circusd.

NOTE:

A Plugin is composed of two parts:

Each plugin is run as a separate process under a custom watcher.

A few examples of some plugins you could create with this system:

Circus itself comes with a few built-in plugins.

The CircusPlugin class¶

Circus provides a base class to help you implement plugins: circus.plugins.CircusPlugin

When initialized by Circus, this class creates its own event loop that receives all circusd events and pass them to handle_recv(). The data received is a tuple containing the topic and the data itself.

handle_recv() must be implemented by the plugin.

The call() and cast() methods can be used to interact with circusd if you are building a Plugin that actively interacts with the daemon.

handle_init() and handle_stop() are just convenience methods you can use to initialize and clean up your code. handle_init() is called within the thread that just started. handle_stop() is called in the main thread just before the thread is stopped and joined.

Writing a plugin¶

Let's write a plugin that logs in a file every event happening in circusd. It takes one argument which is the filename.

The plugin may look like this:

from circus.plugins import CircusPlugin
class Logger(CircusPlugin):


    name = 'logger'


    def __init__(self, *args, **config):


        super(Logger, self).__init__(*args, **config)


        self.filename = config.get('filename')


        self.file = None


    def handle_init(self):


        self.file = open(self.filename, 'a+', buffering=1)


    def handle_stop(self):


        self.file.close()


    def handle_recv(self, data):


        watcher_name, action, msg = self.split_data(data)


        msg_dict = self.load_message(msg)


        self.file.write('%s %s::%r\n' % (action, watcher_name, msg_dict))

That's it ! This class can be saved in any package/module, as long as it can be seen by Python.

For example, Logger may be found in a plugins module within a myproject package.

Async requests¶

In case you want to make any asynchronous operations (like a Tornado call or using periodicCall) make sure you are using the right loop. The loop you always want to be using is self.loop as it gets set up by the base class. The default loop often isn't the same and therefore code might not get executed as expected.

Trying a plugin¶

You can run a plugin through the command line with the circus-plugin command, by specifying the plugin fully qualified name:

$ circus-plugin --endpoint tcp://127.0.0.1:5555 --pubsub tcp://127.0.0.1:5556 --config filename:circus-events.log myproject.plugins.Logger
[INFO] Loading the plugin...
[INFO] Endpoint: 'tcp://127.0.0.1:5555'
[INFO] Pub/sub: 'tcp://127.0.0.1:5556'
[INFO] Starting

Another way to run a plugin is to let Circus handle its initialization. This is done by adding a [plugin:NAME] section in the configuration file, where NAME is a unique name for your plugin:

[plugin:logger]
use = myproject.plugins.Logger
filename = /var/myproject/circus.log

use is mandatory and points to the fully qualified name of the plugin.

When Circus starts, it creates a watcher with one process that runs the pointed class, and pass any other variable contained in the section to the plugin constructor via the config mapping.

You can also programmatically add plugins when you create a circus.arbiter.Arbiter class or use circus.get_arbiter(), see library.

Performances¶

Since every plugin is loaded in its own process, it should not impact the overall performances of the system as long as the work done by the plugin is not doing too many calls to the circusd process.

Hooks¶

Circus provides hooks that can be used to trigger actions upon watcher events. Available hooks are:

Example¶

A typical use case is to control that all the conditions are met for a process to start. Let's say you have a watcher that runs Redis and a watcher that runs a Python script that works with Redis. With Circus you can order the startup by using the priority option:

[watcher:queue-worker]
cmd = python -u worker.py
priority = 1
[watcher:redis]
cmd = redis-server
priority = 2

With this setup, Circus will start Redis first and then it will start the queue worker. But Circus does not really control that Redis is up and running. It just starts the process it was asked to start. What we miss here is a way to control that Redis is started and fully functional. A function that controls this could be:

import redis
import time
def check_redis(*args, **kw):


    time.sleep(.5)  # give it a chance to start


    r = redis.StrictRedis(host='localhost', port=6379, db=0)


    r.set('foo', 'bar')


    return r.get('foo') == 'bar'

This function can be plugged into Circus as an before_start hook:

[watcher:queue-worker]
cmd = python -u worker.py
hooks.before_start = mycoolapp.myplugins.check_redis
priority = 1
[watcher:redis]
cmd = redis-server
priority = 2

Once Circus has started the redis watcher, it will start the queue-worker watcher, since it follows the priority ordering. Just before starting the second watcher, it will run the check_redis function, and in case it returns False will abort the watcher starting process.

Hook signature¶

A hook must follow this signature:

def hook(watcher, arbiter, hook_name, **kwargs):


    ...


    # If you don't return True, the hook can change


    # the behavior of circus (depending on the hook)


    return True

Where watcher is the Watcher class instance, arbiter the Arbiter one, hook_name the hook name and kwargs some additional optional parameters (depending on the hook type).

The after_spawn hook adds the pid parameters:

def after_spawn(watcher, arbiter, hook_name, pid, **kwargs):


    ...


    # If you don't return True, circus will kill the process


    return True

Where pid is the PID of the corresponding process.

Likewise, before_signal and after_signal hooks add pid and signum:

def before_signal_hook(watcher, arbiter, hook_name, pid, signum, **kwargs):


    ...


    # If you don't return True, circus won't send the signum signal


    # (SIGKILL is always sent)


    return True

Where pid is the PID of the corresponding process and signum is the corresponding signal.

You can ignore those but being able to use the watcher and/or arbiter data and methods can be useful in some hooks.

Note that hooks are called with named arguments. So use the hook signature without changing argument names.

The extended_stats hook has its own additional parameters in kwargs:

def extended_stats_hook(watcher, arbiter, hook_name, pid, stats, **kwargs):


    ...

Where pid is the PID of the corresponding process and stats the regular stats to be returned. Add your own stats into stats. An example is in examples/uwsgi_lossless_reload.py.

As a last example, here is a super hook which can deal with all kind of signals:

def super_hook(watcher, arbiter, hook_name, **kwargs):


    pid = None


    signum = None


    if hook_name in ('before_signal', 'after_signal'):


        pid = kwargs['pid']


        signum = kwargs['signum']


    ...


    return True

Hook events¶

Everytime a hook is run, its result is notified as an event in Circus.

There are two events related to hooks:

Adding new commands¶

We tried to make adding new commands as simple as possible.

You need to do three things:

Let's say we want to add a command which returns the number of watchers currently in use, we would do something like this (extensively commented to allow you to follow more easily):

from circus.commands.base import Command
from circus.exc import ArgumentError, MessageError
class NumWatchers(Command):


    """It is a good practice to describe what the class does here.


    Have a look at other commands to see how we are used to format


    this text. It will be automatically included in the documentation,


    so don't be affraid of being exhaustive, that's what it is made


    for.


    """


    # all the commands inherit from `circus.commands.base.Command`


    # you need to specify a name so we find back the command somehow


    name = "numwatchers"


    # Set waiting to True or False to define your default behavior


    # - If waiting is True, the command is run synchronously, and the client may get


    #   back results.


    # - If waiting is False, the command is run asynchronously on the server and the client immediately


    #   gets back an 'ok' response


    #


    #   By default, commands are set to waiting = False


    waiting = True


    # options


    options = [('', 'optname', default_value, 'description')]


    properties = ['foo', 'bar']


    # properties list the command arguments that are mandatory. If they are


    # not provided, then an error will be thrown


    def execute(self, arbiter, props):


        # the execute method is the core of the command: put here all the


        # logic of the command and return a dict containing the values you


        # want to return, if any


        return {"numwatchers": arbiter.numwatchers()}


    def console_msg(self, msg):


        # msg is what is returned by the execute method.


        # this method is used to format the response for a console (it is


        # used for instance by circusctl to print its messages)


        return "a string that will be displayed"


    def message(self, *args, **opts):


        # message handles console input.


        # this method is used to map console arguments to the command


        # options. (its is used for instance when calling the command via


        # circusctl)


        # NotImplementedError will be thrown if the function is missing


        numArgs = 1


        if not len(args) == numArgs:


            raise ArgumentError('Invalid number of arguments.')


        else:


            opts['optname'] = args[0]


        return self.make_message(**opts)


    def validate(self, props):


        # this method is used to validate that the arguments passed to the


        # command are correct. An ArgumentError should be thrown in case


        # there is an error in the passed arguments (for instance if they


        # do not match together.


        # In case there is a problem wrt their content, a MessageError


        # should be thrown. This method can modify the content of the props


        # dict, it will be passed to execute afterwards.

Use cases examples¶

This chapter presents a few use cases, to give you an idea on how to use Circus in your environment.

Running a WSGI application¶

Running a WSGI application with Circus is quite interesting because you can watch & manage your web workers using circus-top, circusctl or the Web interface.

This is made possible by using Circus sockets. See whycircussockets.

Let's take an example with a minimal Pyramid application:

from pyramid.config import Configurator
from pyramid.response import Response
def hello_world(request):


    return Response('Hello %(name)s!' % request.matchdict)
config = Configurator()
config.add_route('hello', '/hello/{name}')
config.add_view(hello_world, route_name='hello')
application = config.make_wsgi_app()

Save this script into an app.py file, then install those projects:

$ pip install Pyramid
$ pip install chaussette

Next, make sure you can run your Pyramid application using the chaussette console script:

$ chaussette app.application
Application is <pyramid.router.Router object at 0x10a4d4bd0>
Serving on localhost:8080
Using <class 'chaussette.backend._waitress.Server'> as a backend

And check that you can reach it by visiting http://localhost:8080/hello/tarek

Now that your application is up and running, let's create a Circus configuration file:

[circus]
check_delay = 5
endpoint = tcp://127.0.0.1:5555
pubsub_endpoint = tcp://127.0.0.1:5556
stats_endpoint = tcp://127.0.0.1:5557
[watcher:webworker]
cmd = chaussette --fd $(circus.sockets.webapp) app.application
use_sockets = True
numprocesses = 3
[socket:webapp]
host = 127.0.0.1
port = 8080

This file tells Circus to bind a socket on port 8080 and run chaussette workers on that socket -- by passing its fd.

Save it to server.ini and try to run it using circusd

$ circusd server.ini
[INFO] Starting master on pid 8971
[INFO] sockets started
[INFO] circusd-stats started
[INFO] webapp started
[INFO] Arbiter now waiting for commands

Make sure you still get the app on http://localhost:8080/hello/tarek.

Congrats ! you have a WSGI application running 3 workers.

You can run the circushttpd or the cli, and enjoy Circus management.

Running a Django application¶

Running a Django application is done exactly like running a WSGI application. Use the PYTHONPATH to import the directory the project is in, the directory that contains the directory that has settings.py in it (with Django 1.4+ this directory has manage.py in it) :

[socket:dwebapp]
host = 127.0.0.1
port = 8080
[watcher:dwebworker]
cmd = chaussette --fd $(circus.sockets.dwebapp) dproject.wsgi.application
use_sockets = True
numprocesses = 2
[env:dwebworker]
PYTHONPATH = /path/to/parent-of-dproject

If you need to pass the DJANGO_SETTINGS_MODULE for a backend worker for example, you can pass that also though the env configation option:

[watcher:dbackend]
cmd = /path/to/script.py
numprocesses=3
[env:dbackend]
PYTHONPATH = /path/to/parent-of-dproject
DJANGO_SETTINGS_MODULE=dproject.settings

See http://chaussette.readthedocs.org for more about chaussette.

Design decisions¶

Overall architecture¶

[image]

Circus is composed of a main process called circusd which takes care of running all the processes. Each process managed by Circus is a child process of circusd.

Processes are organized in groups called watchers. A watcher is basically a command circusd runs on your system, and for each command you can configure how many processes you want to run.

The concept of watcher is useful when you want to manage all the processes running the same command -- like restart them, etc.

circusd binds two ZeroMQ sockets:

NOTE:

Another process called circusd-stats is run by circusd when the option is activated. circusd-stats's job is to publish CPU/Memory usage statistics in a dedicated PUB/SUB channel.

This specialized channel is used by circus-top and circus-httpd to display a live stream of the activity.

circus-top is a console script that mimics top to display all the CPU and Memory usage of the processes managed by Circus.

circus-httpd is the web managment interface that will let you interact with Circus. It displays a live stream using web sockets and the circusd-stats channel, but also let you interact with circusd via its REQ/REP channel.

Last but not least, circusctl is a command-line tool that let you drive circusd via its REQ/REP channel.

You can also have plugins that subscribe to circusd's PUB/SUB channel and let you send commands to the REQ/REP channel like circusctl would.

Security¶

Circus is built on the top of the ZeroMQ library and comes with no security at all in its protocols. However, you can run a Circus system on a server and set up an SSH tunnel to access it from another machine.

This section explains what Circus does on your system when you run it, and ends up describing how to use an SSH tunnel.

You can also read http://www.zeromq.org/area:faq#toc5

TCP ports¶

By default, Circus opens the following TCP ports on the local host:

These ports allow client apps to interact with your Circus system, and depending on how your infrastructure is organized, you may want to protect these ports via firewalls or configure Circus to run using IPC ports.

Here's an example of running Circus using only IPC entry points:

[circus]
check_delay = 5
endpoint = ipc:///var/circus/endpoint
pubsub_endpoint = ipc:///var/circus/pubsub
stats_endpoint = ipc:///var/circus/stats

When Configured using IPC, the commands must be run from the same box, but no one can access them from outside, unlike using TCP. The commands must also be run as a user that has write access to the ipc socket paths. You can modify the owner of the endpoint using the endpoint_owner config option. This allows you to run circusd as the root user, but allow non-root processes to send commands to circusd. Note that when using endpoint_owner, in order to prevent non-root processes from being able to start arbitrary processes that run with greater privileges, the add command will enforce that new Watchers must run as the endpoint_owner user. Watcher definitions in the local config files will not be restricted this way.

Of course, if you activate the Web UI, the 8080 port will still be open.

circushttpd¶

When you run circushttpd manually, or when you use the httpd option in the ini file like this:

[circus]
check_delay = 5
endpoint = ipc:///var/circus/endpoint
pubsub_endpoint = ipc:///var/circus/pubsub
stats_endpoint = ipc:///var/circus/stats
httpd = 1

The web application will run on port 8080 and will let anyone accessing the web page manage the circusd daemon.

That includes creating new watchers that can run any command on your system !

Do not make it publicly available

If you want to protect the access to the web panel, you can serve it behind Nginx or Apache or any proxy-capable web server, that can take care of the security.

User and Group Permissions¶

By default, all processes started with Circus will be running with the same user and group as circusd. Depending on the privileges the user has on the system, you may not have access to all the features Circus provides.

For instance, some statistics features on a running processes require extended privileges. Typically, if the CPU usage numbers you get using the stats command are N/A, it means your user can't access the proc files. This will be the case by default under Mac OS X.

You may run circusd as root to fix this, and set the uid and gid values for each watcher to get all the features.

But beware that running circusd as root exposes you to potential privilege escalation bugs. While we're doing our best to avoid any bugs, running as root and facing a bug that performs unwanted actions on your system may be dangerous.

The best way to prevent this is to make sure that the system running Circus is completely isolated (like a VM) or to run the whole system under a controlled user.

SSH tunneling¶

Clients can connect to a circusd instance by creating an SSH tunnel. To do so, pass the command line option --ssh followed by user@address, where user is the user on the remote server and address is the server's address as seen by the client. The SSH protocol will require credentials to complete the login.

If circusd as seen by the SSH server is not at the default endpoint address localhost:5555 then specify the circusd address using the option --endpoint

Secured setup example¶

Setting up a secured Circus server can be done by:

Contributing to Circus¶

Circus has been started at Mozilla but its goal is not to stay only there. We're trying to build a tool that's useful for others, and easily extensible.

We really are open to any contributions, in the form of code, documentation, discussions, feature proposal etc.

You can start a topic in our mailing list : http://tech.groups.yahoo.com/group/circus-dev/

Or add an issue in our bug tracker

Fixing typos and enhancing the documentation¶

It's totally possible that your eyes are bleeding while reading this half-english half-french documentation, don't hesitate to contribute any rephrasing / enhancement on the form in the documentation. You probably don't even need to understand how Circus works under the hood to do that.

Adding new features¶

New features are of course very much appreciated. If you have the need and the time to work on new features, adding them to Circus shouldn't be that complicated. We tried very hard to have a clean and understandable API, hope it serves the purpose.

You will need to add documentation and tests alongside with the code of the new feature. Otherwise we'll not be able to accept the patch.

How to submit your changes¶

We're using git as a DVCS. The best way to propose changes is to create a branch on your side (via git checkout -b branchname) and commit your changes there. Once you have something ready for prime-time, issue a pull request against this branch.

We are following this model to allow to have low coupling between the features you are proposing. For instance, we can accept one pull request while still being in discussion for another one.

Before proposing your changes, double check that they are not breaking anything! You can use the tox command to ensure this, it will run the testsuite under the different supported python versions.

Please use : http://issue2pr.herokuapp.com/ to reference a commit to an existing circus issue, if any.

Avoiding merge commits¶

Avoiding merge commits allows to have a clean and readable history. To do so, instead of doing "git pull" and letting git handling the merges for you, using git pull --rebase will put your changes after the changes that are commited in the branch, or when working on master.

That is, for us core developers, it's not possible anymore to use the handy github green button on pull requests if developers didn't rebased their work themselves or if we wait too much time between the request and the actual merge. Instead, the flow looks like this:

git remote add name repo-url
git fetch name
git checkout feature-branch
git rebase master
# check that everything is working properly and then merge on master
git checkout master
git merge feature-branch

Discussing¶

If you find yourself in need of any help while looking at the code of Circus, you can go and find us on irc at #circus-tent on irc.freenode.org (or if you don't have any IRC client, use the webchat)

You can also start a thread in our mailing list - http://tech.groups.yahoo.com/group/circus-dev

Frequently Asked Questions¶

Here is a list of frequently asked questions about Circus:

How does Circus stack compare to a classical stack?¶

In a classical WSGI stack, you have a server like Gunicorn that serves on a port or an unix socket and is usually deployed behind a web server like Nginx: [image]

Clients call Nginx, which reverse proxies all the calls to Gunicorn.

If you want to make sure the Gunicorn process stays up and running, you have to use a program like Supervisord or upstart.

Gunicorn in turn watches for its processes ("workers").

In other words you are using two levels of process managment. One that you manage and control (supervisord), and a second one that you have to manage in a different UI, with a different philosophy and less control over what's going on (the wsgi server's one)

This is true for Gunicorn and most multi-processes WSGI servers out there I know about. uWsgi is a bit different as it offers plethoras of options.

But if you want to add a Redis server in your stack, you will end up with managing your stack processes in two different places.

Circus' approach on this is to manage processes and sockets.

A Circus stack can look like this: [image]

So, like Gunicorn, Circus is able to bind a socket that will be proxied by Nginx. Circus don't deal with the requests but simply binds the socket. It's then up to a web worker process to accept connections on the socket and do the work.

It provides equivalent features than Supervisord but will also let you manage all processes at the same level, wether they are web workers or Redis or whatever. Adding a new web worker is done exactly like adding a new Redis process.

Benches¶

We did a few benches to compare Circus & Chaussette with Gunicorn. To summarize, Circus is not adding any overhead and you can pick up many different backends for your web workers.

See:

How to troubleshoot Circus?¶

By default, circusd keeps its logging to stdout rather sparse. This lack of output can make things hard to troubleshoot when processes seem to be having trouble starting.

To increase the logging circusd provides, try increasing the log level. To see the available log levels just use the --help flag.

$ circus --log-level debug test.ini

One word of warning. If a process is flapping and the debug log level is turned on, you will see messages for each start attempt. It might be helpful to configure the app that is flapping to use a warmup_delay to slow down the messages to a manageable pace.

[watcher:webapp]
cmd = python -m myapp.wsgi
warmup_delay = 5

By default, stdout and stderr are captured by the circusd process. If you are testing your config and want to see the output in line with the circusd output, you can configure your watcher to use the StdoutStream class.

[watcher:webapp]
cmd = python -m myapp.wsgi
stdout_stream.class = StdoutStream
stderr_stream.class = StdoutStream

If your application is producing a traceback or error when it is trying to start up you should be able to see it in the output.

Changelog history¶

0.12.1 - 2015-08-05¶

0.12 - 2015-06-02¶

This release brings Python 3.4, Tornado 4 and Windows support, among several exciting features and fixes.

The Windows support is still experimental, and does not handle streams.

Major changes:

More changes:

0.11.1 - 2014-05-22¶

0.11 - 2014-05-21¶

This release is not introducing a lot of features, and focused on making Circus more robust & stable.

Major changes/fixes:

More changes:

0.10 - 2013-11-04¶

Major changes:

More changes:

0.9.3 - 2013-09-04¶

0.9.2 - 2013-07-17¶

0.9 - 2013-07-16¶

0.8.1 - 2013-05-28¶

0.8 - 2013-05-24¶

0.7.1 - 2013-05-02¶

0.7 - 2013-04-08¶

0.6 - 2012-12-18¶

0.5.2 - 2012-07-26¶

0.5.1 - 2012-07-11¶

0.5 - 2012-07-06¶

0.4 - 2012-06-12¶

0.3.4 - 2012-05-30¶

0.3.3 - 2012-05-29¶

0.3.2 - 2012-05-24¶

0.3.1 - 2012-04-18¶

0.3 - 2012-04-18¶

0.2 - 2012-04-04¶

0.1 - 2012-03-20¶

man pages¶

circusd man page¶

Synopsis¶

circusd [options] [config]

Description¶

circusd is the main process of the Circus architecture. It takes care of running all the processes. Each process managed by Circus is a child process of circusd.

Arguments¶

Options¶

See also¶

circus (1), circusctl (1), circusd-stats (1), circus-plugin (1), circus-top (1).

Full Documentation is available at http://circus.readthedocs.org

circusctl man page¶

Synopsis¶

circusctl [options] command [args]

Description¶

circusctl is front end to control the Circus daemon. It is designed to help the administrator control the functionning of the Circud circusd daemon.

Commands¶

Options¶

See Also¶

circus (1), circusd (1), circusd-stats (1), circus-plugin (1), circus-top (1).

Full Documentation is available at http://circus.readthedocs.org

circus-plugin man page¶

Synopsis¶

circus-plugin [options] [plugin]

Description¶

circus-plugin allows to launch a plugin from a running Circus daemon.

Arguments¶

Options¶

See also¶

circus (1), circusd (1), circusctl (1), circusd-stats (1), circus-top (1).

Full Documentation is available at http://circus.readthedocs.org

circus-top man page¶

Synopsis¶

circus-top [options]

Description¶

circus-top is a top-like command to display the Circus daemon and processes managed by circus.

Options¶

See also¶

circus (1), circusctl (1), circusd (1), circusd-stats (1), circus-plugin (1).

Full Documentation is available at http://circus.readthedocs.org

circusd-stats man page¶

Synopsis¶

circusd-stats [options]

Description¶

circusd-stats runs the stats aggregator for Circus.

Options¶

See also¶

circus (1), circusd (1), circusctl (1), circus-plugin (1), circus-top (1).

Full Documentation is available at http://circus.readthedocs.org

Glossary: Circus-specific terms¶

Copyright¶

Circus was initiated by Tarek Ziade and is licenced under APLv2

Benoit Chesneau was an early contributor and did many things, like most of the circus.commands work.

Licence¶

Copyright 2012 - Mozilla Foundation
Copyright 2012 - Benoit Chesneau
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at


    http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Contributors¶

See the full list at https://github.com/circus-tent/circus/blob/master/CONTRIBUTORS.txt