Skip navigation
All Places > Products > RSA NetWitness Platform > Blog > 2020 > August
2020

By default, NetWitness Endpoint 11.x creates a self-signed Certificate Authority during its initial installation, and uses this CA to generate certificates for the endpoint agent and the local reverse proxy that handles all incoming agent communications. Because all these certificates are generated from the same CA chain, they automatically trust each other and enable seamless, easy, and secure communications between agents and the endpoint server.

 

But what if this self-signed CA cannot be used within your organization? For a number of very valid reasons, many orgs might not allow software using a self-signed certificate, and may instead be required to use their own trusted CAs. If this is the case, we have a couple options - an easy way, and a hard way.

 

This blog covers the hard way.

 

Everything that we do in the hard way must occur after the Endpoint Log Hybrid host has been fully installed and provisioned. This means you'll need to complete the entire host installation before moving on to this process.

 

There are 2 primary requirements for the hard way:

  • you must be able to create a server certificate and private key capable Server Authentication
  • you must be able to create a client certificate and private key capable of Client Authentication
    • this client certificate must have Common Name (CN) value of rsa-nw-endpoint-agent

 

I won't be going into details on how to generate these certificates and keys - your org should have some kind of process in place for this. And since the certificates and keys generated from that process can output in a number of different formats, I won't be going into details on how to convert or reformat them. There are numerous guides, documents, and instructions online to help with that.

 

Once we have our server and client certificates and keys, make sure to also grab the CA chain used to generate them (at the very least, both certs need to have a common Root or Intermediate CA to be part of the same trusted chain). This should hopefully be available through the same process used to create the certs and keys. If not, we can also export CA chains from websites - if you do this, make sure it is the same chain used to create your certificates and keys.

 

The endstate format that we'll need for everything will be PEM. The single server and/or client cert should look like this:

-----BEGIN CERTIFICATE-----
MIIFODCCAyCgAwIBAgICEAEwDQYJKoZIhvcNAQELBQAwEjEQMA4GA1UEAwwHUm9v
dC1jYTAeFw0yMDA4MDUyMDE0MTdaFw0zMDA4MDMyMDE0MTdaMCUxIzAhBgNVBAMM
....snip....
-----END CERTIFICATE-----

 

The private key should look like this:

-----BEGIN PRIVATE KEY-----
MIIJQwIBADANBgkqhkiG9w0BAQEFAASCCS0wggkpAgEAAoICAQCuUtxhFPb+FtWD
mQyIELpYVW7isU2KA7ur6ZhWDnKI6pD1POYHfyftO6MhxYsaRrwQ+XxhRJhyT/Ht
....snip....
-----END PRIVATE KEY-----

 

And the Certificate Chain should look this (one BEGIN-END block per CA certificate in the chain...also, it will help to simplify the rest of the process if this chain only includes CA certificates):

-----BEGIN CERTIFICATE-----
MIIFODCCAyCgAwIBAgICEAEwDQYJKoZIhvcNAQELBQAwEjEQMA4GA1UEAwwHUm9v
dC1jYTAeFw0yMDA4MDUyMDE0MTdaFw0zMDA4MDMyMDE0MTdaMCUxIzAhBgNVBAMM
....snip....
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFBzCCAu+gAwIBAgIJAK5iXOLV5WZQMA0GCSqGSIb3DQEBCwUAMBIxEDAOBgNV
BAMMB1Jvb3QtY2EwHhcNMjAwODA1MTk1MTMxWhcNMzAwODAzMTk1MTMxWjASMRAw
....snip....
-----END CERTIFICATE-----

 

We want to make sure we have each of these PEM files for both the server and client certs and key we generated. Once we have these, we can proceed to the next set of steps.

 

The rest of this process will assume that all of these certificates, keys, and chains are staged on the Endpoint Log Hybrid host.

Every command we run from this point forward occurs on the Endpoint Log Hybrid.

We end up replacing a number of different files on this host, so you should also consider backup all the affected files before running the following commands.

 

For the server certificates:

  • # cp /path/to/server/certificate.pem /etc/pki/nw/web/endpoint-web-server-cert.pem
  • # cp /path/to/server/key.pem /etc/pki/nw/web/endpoint-web-server-key.pem
  • # cat /path/to/server/certificate.pem > /etc/pki/nw/web/endpoint-web-server-cert.chain
  • # cat /path/to/ca/chain.pem >> /etc/pki/nw/web/endpoint-web-server-cert.chain
  • # openssl crl2pkcs7 -nocrl -certfile /path/to/server/certificate.pem -certfile /path/to/ca/chain.pem -out /etc/pki/nw/web/endpoint-web-server-cert.p7b
  • # cp /path/to/ca/chain.pem /etc/pki/nw/nwe-trust/truststore.pem
  • # cp /path/to/ca/chain.pem /etc/pki/nw/nwe-ca/customrootca-cert.pem
  • # echo "/etc/pki/nw/nwe-ca/customrootca-cert.pem" > /etc/pki/nw/nwe-trust/truststore.p12.idx
  • # echo "/etc/pki/nw/nwe-ca/customrootca-cert.pem" > /etc/pki/nw/nwe-trust/truststore.pem.idx

 

The end results, with all the files we modified and replaced, should be:

 

Once we're confident we've completed these steps, run:

  • # systemctl restart nginx

 

We can verify that everything so far has worked by browsing to https://<endpoint_server_IP_or_FQDN> and checking the certificate presented by the server:

 

If this matches our server certificate and chain, then we can move on to the client certificates. If not, then we need to go back and figure out which step we did wrong.

 

For the client certificates:

  • openssl pkcs12 -export -out client.p12 -in /path/to/client/certificate.pem -inkey /path/to/client/key.pem -certfile /path/to/ca/chain.pem

 

...enter a password for the certificate bundle, and then SCP this client.p12 bundle onto a windows host. We'll come back to it in just a moment.

 

In the NetWitness UI, browse to Admin/Services --> Endpoint-Server --> Config --> Agent Packager tab. Change or validate any of the configurations you need, and then click "Generate Agent Packager." The Certificate Password field here is required to download the packager, but we won't be using the OOTB client certificate at all so don't stress about the password.

 

Unzip this packager onto the same windows host that has the client.p12 bundle we generated previously. Next, browse to the AgentPackager\config directory, replace the OOTB client.p12 file with the our custom-made client.p12 bundle, move back up up one directory, and run the AgentPackager.exe.

 

If our client.p12 bundle has been created correctly, then in the window that opens, we will be prompted for a password. This is the password we used when we ran the openssl pkcs12 command above, not the password we used in the UI to generate the packager. If they happen to be the same, fantastic....

 

We'll want to verify that the Client certificate and Root CA certificate thumbprints here match with our custom generated certificates.

 

With our newly generated agent installers, it is now time to test them. Pick a host in your org, run the appropriate agent installer, and then verify that you see the agent showing up in your UI at Investigate/Hosts.

 

If it does appear, congratulations! Make sure to record all these changes, and be ready to repeat them when certificates expire and agent installers need upgrading/updating.

 

If it doesn't, a couple things to check:

  • first, give it a couple minutes...it's not going to show up instantly
  • go back through all these steps and double-check that everything is correct
  • check the c:\windows\temp directory for a log file with the same name as your endpoint agent; e.g.: NWEAgent.log....if there are communication errors between the agent/host and the endpoint server, this log will likely have relevant troubleshooting details
  • if the agent log file has entries showing both "AgentCert" and "KnownServerCert" values, check that these thumbprints match the Client and Root CA certificate thumbprints from the AgentPackager output

    • ...I was not able to consistently reproduce this issue, but it is related to how the certs and keys are bundled together in the client.p12
    • ...when this happened to me, I imported my custom p12 bundle into the Windows MMC Certificates snap-in, and then exported it (make sure that the private key gets both imported and exported, as well as all the CAs in the chain), then re-ran my AgentPackger with this exported client.p12, and it fixed the error
    • ... ¯\_(ツ)_/¯
  • from a cmd prompt on the host, run c:\windows\system32\<service name of the agent>.exe /testnet
  • check the NGINX access log on the Endpoint Log Hybrid; along with the agent log file on the endpoint, this can show whether the agent and/or server are communication properly
    # tail -f /var/log/nginx/access.log

By default, NetWitness Endpoint 11.x creates a self-signed Certificate Authority during its initial installation, and uses this CA to generate certificates for the endpoint agent and the local reverse proxy that handles all incoming agent communications. Because all these certificates are generated from the same CA chain, they automatically trust each other and enable seamless, easy, and secure communications between agents and the endpoint server.

 

But what if this self-signed CA cannot be used within your organization? For a number of very valid reasons, many orgs might not allow software using a self-signed certificate, and may instead be required to use their own trusted CAs. If this is the case, we have a couple options - an easy way, and a hard way.

 

This blog covers the easy way.

 

The only real requirement for the easy way is that we are able to create an Intermediate CA certificate and its private key from our CA chain (or use an existing pair), and that this Intermediate CA is allowed to generate an additional, subordinate CA under it.

 

For my testing, "Root-ca" was my imaginary company's Root CA, and I created "My Company Intermediate CA" for use in my 11.4 Endpoint Log Hybrid.

 

(I'm no expert in certificates, but I can say that all the Intermediate CAs I created that had explicit extendedKeyUsage extensions failed. The only Intermediate CAs I could get to work included "All" of the Intended Purposes. If you know more about CAs and the specific extendedKeyUsage extensions needed for a CA to be able to create subordinate CAs, I'd be interested to know what they are.)

 

Once we have an Intermediate CA certificate and its private key, we need to make sure they are in PEM format. There are a number of ways to convert and check keys and certificates, and a whole bunch of resources online to help with this this, so I won't cover any of the various conversion commands or methods here. 

 

If the CA certificate looks like this, then it is most likely in the correct format:

-----BEGIN CERTIFICATE-----
MIIFODCCAyCgAwIBAgICEAEwDQYJKoZIhvcNAQELBQAwEjEQMA4GA1UEAwwHUm9v
dC1jYTAeFw0yMDA4MDUyMDE0MTdaFw0zMDA4MDMyMDE0MTdaMCUxIzAhBgNVBAMM
....snip....
-----END CERTIFICATE-----

 

And if the private key looks like this, then it is most likely in the correct format:

-----BEGIN PRIVATE KEY-----
MIIJQwIBADANBgkqhkiG9w0BAQEFAASCCS0wggkpAgEAAoICAQCuUtxhFPb+FtWD
mQyIELpYVW7isU2KA7ur6ZhWDnKI6pD1POYHfyftO6MhxYsaRrwQ+XxhRJhyT/Ht
....snip....
-----END PRIVATE KEY-----

 

Our last step in this process has to occur at a very specific point during the endpoint log hybrid's installation - after we have run the nwsetup-tui command and the host has been enabled within the NetWitness UI, but before we install the Endpoint Log Hybrid services:

  • on the endpoint host, create directory /etc/pki/nw/nwe-ca
  • place the CA certificate and CA private key files in this directory and name them nwerootca-cert.pem and nwerootca-key.pem, respectively

 

The basis for this process comes directly from the "Configure Multiple Endpoint Log Hybrid Hosts" step in the Post Installation tasks guide (https://community.rsa.com/docs/DOC-101660#NetWitne), if we want a bit more context or details on when this step should occur and how to do it properly.

 

Once we've done this, we can now install the Endpoint Log Hybrid services on the host.

 

I suggest you watch the installation log file on the endpoint server, because if the Intermediate CA does not have all the necessary capabilities, the installation will fail and this log file can help us identify which step (if my own experience is any guide, then it will most likely fail during the attempt to create the subordinate Endpoint Intermediate CA --> /etc/pki/nw/nwe-ca/esca-cert.pem):

# tail -f /var/log/netwitness/config-management/chef-solo.log

 

If all goes well, we'll be able to check that our endpoint-server is using our Intermediate CA by browsing to https://<endpoint_server_IP_or_FQDN> and checking the certificate presented by the server:

 

And our client.p12 certificate bundle within the agentPackager will be generated from the same chain:

 

And that's it!

 

Any agent packages we generate from this point forward will use the client.p12 certificates generated from our CA. Likewise, all agent-server communications will be encrypted with the certificates generated from our CA.

Thank you for joining us for the July 22nd NetWitness Webinar covering Data Carving using Logs as presented by Leonard Chvilicek. An edited recording is available below, with the Zoom link to the original webinar recording.

 

 

https://Dell.zoom.us/rec/share/9ddSC-v1qVxITbeS5hreSJY6AZnFeaa8hyEe-fYKxEvYejAP3hl67DCXZUjQGil6

Password: V0.*h5#v

Filter Blog

By date: By tag: