Notes on osquery and Security Onion Infrastructure
This initial section is designed to provide reader's with a brief overview of the technologies that are relevant to this post.
osquery
osquery is an open-source, cross-platform tool that allows you to query computers as if they were an SQL database. This tool enables cyber security professionals to gain insights into computers by writing SQL-based queries to retrieve data on running processes, loaded kernel modules, open network connections, browser plugins, hardware events, and more. This versatility and flexibility make osquery a valuable asset for system monitoring, incident response, security detection, and many other applications in diverse environments.
Salt
SALT Stack, more commonly referred to as Salt, is a Python-based open-source configuration management software and remote execution engine.
For the purpose of this post, be aware that Security Onion uses Salt to manage the configuration of its Docker containers. In order to override the Security Onion defaults, we must copy the default init.sls files that describe docker's configuration to their equivalent local file path. We then modify the copied init.sls files in this local location, and these changes will be applied by Salt.
Docker
Docker is an open-source platform that allows developers to automate the deployment, scaling, and management of applications. Security Onion uses docker containers to run each of its applications. These containers can be networked together and can be accessed similar to ssh'ing into a server.
Nginx
Nginx is an open-source software that functions as a web server, reverse proxy, load balancer, HTTP cache, and SSL offloader. In Security Onion, it is used as a reverse proxy and SSL offloader to handle secure access to several services.
In relation to osquery, it is used to handle SSL sessions and offloading. The so-nginx container listens on port 8090 to establish and handle the encrypted communication for deployed osquery agents. It then routes the traffic using grpc over port 8080 to the so-fleet container.
Port Forward Issue
When you install osquery using the provided download from Security Onion, it includes a wrapper service called 'so-launcher'. This service is configured through a flag file. If you're port forwarding, you need to set the hostname field in this flag file to the IP address of the Firewall.
However, a problem arises after the agent has been port forwarded and initiates a TLS handshake with the Nginx server. The agent recognizes that the IP it is attempting to connect to and the Nginx server are not the same. This mismatch leads to a TLS handshake error and the connection is terminated.
The problem lies within the certificate that Nginx presents to the deployed agents.
Technical breakdown
This section provide a more in-depth look at the issue. For the solution, please see the next sections.
For this explanation, assume the following IPs:
SO Manger IP: 10.10.78.10
Firewall IP: 10.10.77.30
In this instance, the osquery communication is failing in the TLS Handshake phase, which we will now take a closer look at.
In the TLS handshake, the first step after establishing a TCP connection is the "ClientHello" message from the client (osquery agent, in this case) to the server (Nginx). In this "ClientHello" message, the client specifies the TLS version it is running, the cipher suites it has available. If the client knows the server's hostname, it can also includes a Server Name Indication (SNI) in the "ClientHello" message.
Following the "ClientHello" message, the Nginx server replies with a "ServerHello" message, selecting the highest mutually supported TLS version and a cipher suite. After that, the server sends its digital certificate (containing the public key and its details, like the Common Name (CN)) to the client in a "Certificate" message.
The CN in a certificate is the entity's name that the certificate belongs to and is validated by the Certificate Authority (CA). In this case, osquery is provided with Security Onion's ca.crt as its roots.pem file. If using an IP address as the CN, the client expects to connect to that specific IP. In this case, the CN of the Nginx server's certificate is the SO Manager's IP, which does not match the IP the osquery agent is attempting to connect to.
If an agent tries to connect to a hostname of 10.10.77.30 (Firewall IP), but the Nginx server's SSL certificate CN is 10.10.78.10 (SO Manager IP), then the client will consider the certificate invalid because the CN (10.10.78.10) does not match the IP address (10.10.77.30) the client is trying to connect to.
This mismatch triggers the client to abort the TLS handshake process, leading to a failure to establish a secure session and an application error stating it expected 10.10.77.30, but the cert provided 10.10.78.10.
Port Forward Fix
To solve the problem, we need to create a new certificate that includes the SO Manager IP and the Firewall IP. This is accomplished by adding the Firewall IP as a Subject Alternative Name to the new certificate.
Osquery agents within Security Onion are set up to authenticate using Security Onion's root ca.crt certificate. Noting this, we can leverage the existing ca.crt and ca.key to generate a new certificate that Nginx can utilize. By creating an intermediary certificate from the root certificates, we can keep using the root ca.crt for authentication. This strategy perserves communication between the server and all current agents, while also accommodating any new agents we roll out in the future.
Step-by-Step Fix
NOTE: These instructions are performed on the SO Manager:
Create New Certificates
- Switch to root
sudo su
2. Create a folder to stage certs
mkdir ~/osquery-certs && cd ~/osquery-certs
3. Create a config file for the cert.
NOTE: Update the CN and IP.1 fields to the be the Security Onion Manager IP (or wherever the nginx container is running). Update the IP.2 field to the IP of Firewall you are forwarding through.
cat << EOF > cert.conf
[ req ]
default_bits = 2048
prompt = no
default_md = sha256
distinguished_name = dn
[ dn ]
CN = 10.10.78.10
[ req_ext ]
subjectAltName = @alt_names
[ alt_names ]
IP.1 = 10.10.78.10
IP.2 = 10.10.77.30
EOF
4. Create a new private key
openssl genpkey -algorithm RSA -out osquery-server.key
New Command Explanations:
openssl: Invokes the OpenSSL toolkit.genpkey: Used to generate a private key.-algorithm RSA: Specifies the RSA algorithm for key generation.-out osquery-server.key: File to which the generated private key will be written.
5. Create a Certificate Signing Request (CSR)
openssl req -new -key osquery-server.key -out osquery-server.csr -config cert.conf -reqexts req_ext
New Command Explanations:
req: Initiates a new CSR-new: Specifies a new CSR being createdkey: They private key to use for the CSR-config: The configuration file for the CSR-reqexts: Includes extensions specified in config file
6. Create the Key
openssl x509 -req -in osquery-server.csr -CA /etc/pki/ca.crt -CAkey /etc/pki/ca.key -CAc
reateserial -out osquery-server.crt -days 3650 -extensions req_ext -extfile cert.conf
New Command Explanations:
x509: Indicates public key certificate format-req: Specifies a CSR will be used for input-in: The CSR to use.-CA: The Certificate Authority cert used to sign the new cert-CAkey: The private key of the Certificate Authority used to sign the new cert-CAcreateserial: Automatically create a serial number file if one does not exist.-days 3650: Days for which the certificate is valid
7. Copy the new key and cert to /etc/pki
cp osquery-server.{crt,key} /etc/pki/
Update the Nginx Configuration
NOTE: This post assumes Nginx has not been modified. If there is an existing /opt/so/saltstack/local/salt/nginx/etc/nginx.conf file, modify the existing one.
- Copy over the
nginx.conffile for modification
cp /opt/so/saltstack/default/salt/nginx/etc/nginx.conf /opt/so/saltstack/local/salt/nginx/etc/
2. Update owner of nginx.conf
chown socore:socore /opt/so/saltstack/local/salt/nginx/etc/nginx.conf
3. Modify the new nginx.conf
vim /opt/so/saltstack/local/salt/nginx/etc/nginx.conf
# Look for a server (should be line 50) with the following 'listen' value:
# listen 8090 ssl http2 default_server;
# Modify the 'ssl_certificate' and 'ssl_certificate_key' values to the following:
ssl_certificate "/etc/pki/nginx/osquery-server.crt";
ssl_certificate_key "/etc/pki/nginx/osquery-server.key";
# Save and close the file
Update the Nginx Docker Configuration
Copy over the Nginx init.sls file and modify the docker bind mounts to include the new certificates
cp /opt/so/saltstack/default/salt/nginx/init.sls /opt/so/saltstack/local/salt/nginx/
- Update the owner of
init.sls
chown socore:socore /opt/so/saltstack/local/salt/nginx/etc/init.sls
2. Edit the copied init.sls file
vim /opt/so/saltstack/local/salt/nginx/init.sls
# Look for the 'so-nginx:' config line (should be line 88).
# Under the 'binds' config (line 92), add the two following entries directly under the "/etc/pki/managerssl.key:/etc/pki/nginx/server.key:ro" line. This tells docker to make these files available inside the container.
- /etc/pki/osquery-server.key:/etc/pki/nginx/osquery-server.key:ro
- /etc/pki/osquery-server.crt:/etc/pki/nginx/osquery-server.crt:ro
3. Apply the new state
# Apply new state
salt-call state.apply nginx
# If files already exist and you are trying to update them, do a restart instead
so-restart nginx
Verify Changes
To verify the changes actually took place, we can check the following locations inside the container.
1. Remote into the container
# Remote into docker container. NOTE: we need to use 'sh', not 'bash'.
docker exec -it so-nginx /bin/sh
# You should be dropped in with just a '#' terminal symbol.
2. Verify the new certificate files.
# Check for files
ls /etc/pki/nginx
# You should see osquery-server.crt and osquery-server.key
3. Verify the updated configuration
# Check the nginx.conf file
grep osquery /etc/nginx/nginx.conf
# You should see the two osquery-server.crt and osquery-server.key lines. Alternatively use less or vi to look at the file.
4. Exit the container
# To exit the container, simply type exit
exit
Conclusion
If all went well, you should now be able to port forward the default Security Onion osquery agents through a firewall. Because we used the root ca.crt and ca.key, we perserved communication between the server and all current agents, while also accommodating any new agents we roll out in the future.