rest_cherrypy

Authentication

Authentication is performed by passing a session token with each request. Tokens are generated via the Login URL.

The token may be sent in one of two ways: as a custom header or as a session cookie. The latter is far more convenient for clients that support cookies.

• Include a custom header named X-Auth-Token.

  For example, using curl:

  curl -sSk https://localhost:8000/login \
      -H 'Accept: application/x-yaml' \
      -d username=saltdev \
      -d password=saltdev \
      -d eauth=pam

  Copy the token value from the output and include it in subsequent requests:

  curl -sSk https://localhost:8000 \
      -H 'Accept: application/x-yaml' \
      -H 'X-Auth-Token: 697adbdc8fe971d09ae4c2a3add7248859c87079'\
      -d client=local \
      -d tgt='*' \
      -d fun=test.ping

• Sent via a cookie. This option is a convenience for HTTP clients that automatically handle cookie support (such as browsers).

  For example, using curl:

  # Write the cookie file:
  curl -sSk https://localhost:8000/login \
        -c ~/cookies.txt \
        -H 'Accept: application/x-yaml' \
        -d username=saltdev \
        -d password=saltdev \
        -d eauth=auto
  
  # Read the cookie file:
  curl -sSk https://localhost:8000 \
        -b ~/cookies.txt \
        -H 'Accept: application/x-yaml' \
        -d client=local \
        -d tgt='*' \
        -d fun=test.ping

  Another example using the requests library in Python:

  >>> import requests
  >>> session = requests.Session()
  >>> session.post('http://localhost:8000/login', json={
      'username': 'saltdev',
      'password': 'saltdev',
      'eauth': 'auto',
  })
  <Response [200]>
  >>> resp = session.post('http://localhost:8000', json=[{
      'client': 'local',
      'tgt': '*',
      'fun': 'test.arg',
      'arg': ['foo', 'bar'],
      'kwarg': {'baz': 'Baz!'},
  }])
  >>> resp.json()
  {u'return': [{
      ...snip...
  }]}

Usage

This interface directly exposes Salt's Python API. Everything possible at the CLI is possible through the Python API. Commands are executed on the Salt Master.

The root URL (/) is RPC-like in that it accepts instructions in the request body for what Salt functions to execute, and the response contains the result of those function calls.

For example:

% curl -sSi https://localhost:8000         -H 'Content-type: application/json'         -d '[{
        "client": "local",
        "tgt": "*",
        "fun": "test.ping"
    }]'
HTTP/1.1 200 OK
Content-Type: application/json
[...snip...]

{"return": [{"jerry": true}]}

The request body must be an array of commands. Use this workflow to build a command:

1. Choose a client interface.

2. Choose a function.

3. Fill out the remaining parameters needed for the chosen client.

The client field is a reference to the main Python classes used in Salt's Python API. Read the full Client APIs documentation, but in short:

• "local" uses LocalClient which sends commands to Minions. Equivalent to the salt CLI command.

• "runner" uses RunnerClient which invokes runner modules on the Master. Equivalent to the salt-run CLI command.

• "wheel" uses WheelClient which invokes wheel modules on the Master. Wheel modules do not have a direct CLI equivalent but they typically manage Master-side resources such as state files, pillar files, the Salt config files, and the key wheel module exposes similar functionality as the salt-key CLI command.

Most clients have variants like synchronous or asynchronous execution as well as others like batch execution. See the full list of client interfaces.

Each client requires different arguments and sometimes has different syntax. For example, LocalClient requires the tgt argument because it forwards the command to Minions and the other client interfaces do not. LocalClient also takes arg (array) and kwarg (dictionary) arguments because these values are sent to the Minions and used to execute the requested function there. RunnerClient and WheelClient are executed directly on the Master and thus do not need or accept those arguments.

Read the method signatures in the client documentation linked above, but hopefully an example will help illustrate the concept. This example causes Salt to execute two functions -- the test.arg execution function using LocalClient and the test.arg runner function using RunnerClient; note the different structure for each command. The results for both are combined and returned as one response.

% curl -b ~/cookies.txt -sSi localhost:8000         -H 'Content-type: application/json'         -d '
[
    {
        "client": "local",
        "tgt": "*",
        "fun": "test.arg",
        "arg": ["positional arg one", "positional arg two"],
        "kwarg": {
            "keyword arg one": "Hello from a minion",
            "keyword arg two": "Hello again from a minion"
        }
    },
    {
        "client": "runner",
        "fun": "test.arg",
        "keyword arg one": "Hello from a master",
        "keyword arg two": "Runners do not support positional args"
    }
]
'
HTTP/1.1 200 OK
[...snip...]
{
  "return": [
    {
      "jerry": {
        "args": [
          "positional arg one",
          "positional arg two"
        ],
        "kwargs": {
          "keyword arg one": "Hello from a minion",
          "keyword arg two": "Hello again from a minion",
          [...snip...]
        }
      },
      [...snip; other minion returns here...]
    },
    {
      "args": [],
      "kwargs": {
        "keyword arg two": "Runners do not support positional args",
        "keyword arg one": "Hello from a master"
      }
    }
  ]
}

One more example, this time with more commonly used functions:

curl -b /tmp/cookies.txt -sSi localhost:8000         -H 'Content-type: application/json'         -d '
[
    {
        "client": "local",
        "tgt": "*",
        "fun": "state.sls",
        "kwarg": {
            "mods": "apache",
            "pillar": {
                "lookup": {
                    "wwwdir": "/srv/httpd/htdocs"
                }
            }
        }
    },
    {
        "client": "runner",
        "fun": "cloud.create",
        "provider": "my-ec2-provider",
        "instances": "my-centos-6",
        "image": "ami-1624987f",
        "delvol_on_destroy", true
    }
]
'
HTTP/1.1 200 OK
[...snip...]
{
  "return": [
    {
      "jerry": {
        "pkg_|-install_apache_|-httpd_|-installed": {
            [...snip full state return here...]
        }
      }
      [...snip other minion returns here...]
    },
    {
        [...snip full salt-cloud output here...]
    }
  ]
}

Content negotiation

This REST interface is flexible in what data formats it will accept as well as what formats it will return (e.g., JSON, YAML, urlencoded).

• Specify the format of data in the request body by including the Content-Type header.

• Specify the desired data format for the response body with the Accept header.

We recommend the JSON format for most HTTP requests. urlencoded data is simple and cannot express complex data structures -- and that is often required for some Salt commands, such as starting a state run that uses Pillar data. Salt's CLI tool can reformat strings passed in at the CLI into complex data structures, and that behavior also works via salt-api, but that can be brittle and since salt-api can accept JSON it is best just to send JSON.

Here is an example of sending urlencoded data:

curl -sSik https://localhost:8000 \
    -b ~/cookies.txt \
    -d client=runner \
    -d fun='jobs.lookup_jid' \
    -d jid='20150129182456704682'

curl -ksi http://localhost:8000/login \
-H "Accept: application/json" \
-d username='myapiuser' \
--data-urlencode password='1234+' \
-d eauth='pam'

Long-Running HTTP Connections

The CherryPy server is a production-ready, threading HTTP server written in Python. Because it makes use of a thread pool to process HTTP requests it is not ideally suited to maintaining large numbers of concurrent, synchronous connections. On moderate hardware with default settings it should top-out at around 30 to 50 concurrent connections.

That number of long-running, synchronous Salt processes is also not ideal. Like at the CLI, each Salt command run will start a process that instantiates its own LocalClient, which instantiates its own listener to the Salt event bus, and sends out its own periodic saltutil.find_job queries to determine if a Minion is still running the command. Not exactly a lightweight operation.

Timeouts

In addition to the above resource overhead for long-running connections, there are the usual HTTP timeout semantics for the CherryPy server, any HTTP client being used, as well as any hardware in between such as proxies, gateways, or load balancers. rest_cherrypy can be configured not to time-out long responses via the expire_responses setting, and both LocalClient and RunnerClient have their own timeout parameters that may be passed as top-level keywords:

curl -b /tmp/cookies.txt -sSi localhost:8000         -H 'Content-type: application/json'         -d '
[
    {
        "client": "local",
        "tgt": "*",
        "fun": "test.sleep",
        "kwarg": {"length": 30},
        "timeout": 60
    },
    {
        "client": "runner",
        "fun": "test.sleep",
        "kwarg": {"s_time": 30},
        "timeout": 60
    }
]
'

Best Practices

Given the performance overhead and HTTP timeouts for long-running operations described above, the most effective and most scalable way to use both Salt and salt-api is to run commands asynchronously using the local_async, runner_async, and wheel_async clients.

Running asynchronous jobs results in being able to process 3x more commands per second for LocalClient and 17x more commands per second for RunnerClient, in addition to much less network traffic and memory requirements. Job returns can be fetched from Salt's job cache via the /jobs/<jid> endpoint, or they can be collected into a data store using Salt's Returner system.

The /events endpoint is specifically designed to handle long-running HTTP connections and it exposes Salt's event bus which includes job returns. Watching this endpoint first, then executing asynchronous Salt commands second, is the most lightweight and scalable way to use rest_cherrypy while still receiving job returns in real-time. But this requires clients that can properly handle the inherent asynchronicity of that workflow.

Performance Tuning

The thread_pool and socket_queue_size settings can be used to increase the capacity of rest_cherrypy to handle incoming requests. Keep an eye on RAM usage as well as available file handles while testing changes to these settings. As salt-api is a thin wrapper around Salt's Python API, also keep an eye on the performance of Salt when testing.

Future Plans

Now that Salt uses the Tornado concurrency library internally, we plan to improve performance in the API by taking advantage of existing processes and event listeners and to use lightweight coroutines to facilitate more simultaneous HTTP connections and better support for synchronous operations. That effort can be tracked in issue 26505, but until that issue is closed rest_cherrypy will remain the officially recommended REST API.

salt-api using the CherryPy server

The default configuration is to run this module using salt-api to start the Python-based CherryPy server. This server is lightweight, multi-threaded, encrypted with SSL, and should be considered production-ready. See the section above for performance expectations.

Using a WSGI-compliant web server

This module may be deployed on any WSGI-compliant server such as Apache with mod_wsgi or Nginx with FastCGI, to name just two (there are many).

Note, external WSGI servers handle URLs, paths, and SSL certs directly. The rest_cherrypy configuration options are ignored and the salt-api daemon does not need to be running at all. Remember Salt authentication credentials are sent in the clear unless SSL is being enforced!

An example Apache virtual host configuration:

<VirtualHost *:80>
    ServerName example.com
    ServerAlias *.example.com

    ServerAdmin [email protected]

    LogLevel warn
    ErrorLog /var/www/example.com/logs/error.log
    CustomLog /var/www/example.com/logs/access.log combined

    DocumentRoot /var/www/example.com/htdocs

    WSGIScriptAlias / /path/to/salt/netapi/rest_cherrypy/wsgi.py
</VirtualHost>