Chef Infra Client Security

[edit on GitHub]

All communication with the Chef Infra Server must be authenticated using the Chef Infra Server API, which is a REST API that allows requests to be made to the Chef Infra Server. Only authenticated requests will be authorized. Most of the time, and especially when using knife, Chef Infra Client, or the Chef Infra Server web interface, the use of the Chef Infra Server API is transparent. In some cases, the use of the Chef Infra Server API requires more detail, such as when making the request in Ruby code, with a knife plugin, or when using cURL.

Authentication

The authentication process ensures the Chef Infra Server responds only to requests made by trusted users. Public key encryption is used by the Chef Infra Server. When a node and/or a workstation is configured to run Chef Infra Client, both public and private keys are created. The public key is stored on the Chef Infra Server, while the private key is returned to the user for safe keeping. (The private key is a .pem file located in the .chef directory or in /etc/chef.)

Both Chef Infra Client and knife use the Chef Infra Server API when communicating with the Chef Infra Server. The chef-validator uses the Chef Infra Server API, but only during the first Chef Infra Client run on a node.

Each request to the Chef Infra Server from those executables sign a special group of HTTP headers with the private key. The Chef Infra Server then uses the public key to verify the headers and verify the contents.

chef-validator

Every request made by Chef Infra Client to the Chef Infra Server must be an authenticated request using the Chef Infra Server API and a private key. When Chef Infra Client makes a request to the Chef Infra Server, Chef Infra Client authenticates each request using a private key located in /etc/chef/client.pem.

However, during the first Chef Infra Client run, this private key does not exist. Instead, Chef Infra Client attempts to use the private key assigned to the chef-validator, located in /etc/chef/validation.pem. (If, for any reason, the chef-validator is unable to make an authenticated request to the Chef Infra Server, the initial Chef Infra Client run will fail.)

During the initial Chef Infra Client run, Chef Infra Client registers itself with the Chef Infra Server using the private key assigned to the chef-validator, after which Chef Infra Client will obtain a client.pem private key for all future authentication requests to the Chef Infra Server.

After the initial Chef Infra Client run has completed successfully, the chef-validator is no longer required and may be deleted from the node. Use the delete_validation recipe found in the chef-client cookbook (https://github.com/chef-cookbooks/chef-client) to remove the chef-validator.

During a Chef Infra Client Run

As part of every Chef Infra Client run, Chef Infra Client authenticates to the Chef Infra Server using an RSA private key and the Chef Infra Server API.

authentication_protocol_version

The authentication_protocol_version option in the client.rb file is used to determine the authentication protocol that communicates with Chef Infra Server. For example, specify protocol version 1.3 to enable support for SHA-256 algorithms:

knife[:authentication_protocol_version] = '1.3'

Note that authentication protocol 1.3 is only supported on Chef Server versions 12.4.0 and above.

SSL Certificates

Chef Server 12 and later enables SSL verification by default for all requests made to the server, such as those made by knife and Chef Infra Client. The certificate that is generated during the installation of the Chef Infra Server is self-signed, which means the certificate is not signed by a trusted certificate authority (CA) that ships with Chef Infra Client. The certificate generated by the Chef Infra Server must be downloaded to any machine from which knife and/or Chef Infra Client will make requests to the Chef Infra Server.

For example, without downloading the SSL certificate, the following knife command:

knife client list

responds with an error similar to:

ERROR: SSL Validation failure connecting to host: chef-server.example.com ...
ERROR: OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3 ...

This is by design and will occur until a verifiable certificate is added to the machine from which the request is sent.

/.chef/trusted_certs

The /.chef/trusted_certs directory stores trusted SSL certificates used to access the Chef Infra Server:

• On each workstation, this directory is the location into which SSL certificates are placed after they are downloaded from the Chef Infra Server using the knife ssl fetch subcommand
• On every node, this directory is the location into which SSL certificates are placed when a node has been bootstrapped with Chef Infra Client from a workstation

SSL_CERT_FILE

Use the SSL_CERT_FILE environment variable to specify the location for the SSL certificate authority (CA) bundle that is used by Chef Infra Client.

A value for SSL_CERT_FILE is not set by default. Unless updated, the locations in which Chef Infra will look for SSL certificates are:

• Chef Infra Client: /opt/chef/embedded/ssl/certs/cacert.pem
• ChefDK: /opt/chefdk/embedded/ssl/certs/cacert.pem
• Chef Workstation: /opt/chef-workstation/embedded/ssl/certs/cacert.pem

Keeping the default behavior is recommended. To use a custom CA bundle, update the environment variable to specify the path to the custom CA bundle. If (for some reason) SSL certificate verification stops working, ensure the correct value is specified for SSL_CERT_FILE.

client.rb Settings

Use following client.rb settings to manage SSL certificate preferences:

Knife Subcommands

The Chef Infra Client includes two knife commands for managing SSL certificates:

• Use knife ssl check to troubleshoot SSL certificate issues
• Use knife ssl fetch to pull down a certificate from the Chef Infra Server to the /.chef/trusted_certs directory on the workstation.

After the workstation has the correct SSL certificate, bootstrap operations from that workstation will use the certificate in the /.chef/trusted_certs directory during the bootstrap operation.

knife ssl check

Run the knife ssl check subcommand to verify the state of the SSL certificate, and then use the response to help troubleshoot issues that may be present.

Verified

If the SSL certificate can be verified, the response to

knife ssl check

is similar to:

Connecting to host chef-server.example.com:443
Successfully verified certificates from 'chef-server.example.com'

Unverified

If the SSL certificate cannot be verified, the response to

knife ssl check

is similar to:

Connecting to host chef-server.example.com:443
ERROR: The SSL certificate of chef-server.example.com could not be verified
Certificate issuer data:
  /C=US/ST=WA/L=S/O=Corp/OU=Ops/CN=chef-server.example.com/emailAddress=[email protected]

Configuration Info:

OpenSSL Configuration:
* Version: OpenSSL 1.0.2u  20 Dec 2019
* Certificate file: /opt/chef-workstation/embedded/ssl/cert.pem
* Certificate directory: /opt/chef-workstation/embedded/ssl/certs
Chef SSL Configuration:
* ssl_ca_path: nil
* ssl_ca_file: nil
* trusted_certs_dir: "/Users/grantmc/Downloads/chef-repo/.chef/trusted_certs"

TO FIX THIS ERROR:

If the server you are connecting to uses a self-signed certificate,
you must configure chef to trust that certificate.

By default, the certificate is stored in the following location on the
host where your Chef Infra Server runs:

  /var/opt/opscode/nginx/ca/SERVER_HOSTNAME.crt

Copy that file to your trusted_certs_dir (currently:

  /Users/grantmc/Downloads/chef-repo/.chef/trusted_certs)

using SSH/SCP or some other secure method, then re-run this command to
confirm that the certificate is now trusted.

knife ssl fetch

Run the knife ssl fetch to download the self-signed certificate from the Chef Infra Server to the /.chef/trusted_certs directory on a workstation.

Verify Checksums

The SSL certificate that is downloaded to the /.chef/trusted_certs directory should be verified to ensure that it is, in fact, the same certificate as the one located on the Chef Infra Server. This can be done by comparing the SHA-256 checksums.

1. View the checksum on the Chef Infra Server:

   ssh [email protected] sudo sha256sum /var/opt/opscode/nginx/ca/chef-server.example.com.crt
   

   The response is similar to:

   <ABC123checksum>  /var/opt/opscode/nginx/ca/chef-server.example.com.crt
   

2. View the checksum on the workstation:

   gsha256sum .chef/trusted_certs/chef-server.example.com.crt
   

   The response is similar to:

   <ABC123checksum>  .chef/trusted_certs/chef-server.example.com.crt
   

3. Verify that the checksum values are identical.