on Feb 5, 2020This guide describes the shell utility nw-shell that can be used to troubleshoot the operations of NetWitness Platform management services like security-server, investigate-server, and correlation-server. It is the equivalent of the NwConsole utility used to interact with NetWitness Platform capture services like Decoders and Concentrators. The shell utility is independent of the business logic of the service, and works the same with most NetWitness Platform services.
Features
The nw-shell utility implements the following features:
- Supports secure connections to the local NetWitness Platform service instances.
- Supports navigation of the service tree to explore its operational state (for more information, see Tree View) .
- Provides an intuitive display of configuration, metrics, and health-check information to help with troubleshooting.
- Supports scripting to automate simple administration tasks in field deployments.
- Supports Linux, OSX and Windows terminals.
Installation
The nw-shell utility can be installed with the rsa-nw-shell RPM as shown here:
$ sudo yum install rsa-nw-shell $ /usr/local/bin/nw-shell
RSA
RSA NetWitness Shell. Version: 4.0.0
See "help" to list available commands, "help connect" to get started.
offline »Usage
The primary goal of nw-shell is to help a human operator explore the runtime state of a NetWitness Platform service. It is, essentially, an interactive program that invokes APIs on running NetWitness Platform services. Each NetWitness Platform service includes a system API that exposes its runtime state as a logical tree. The shell leverages the Tree API structure to present a hierarchical view of a service that is similar to a file system view. Users can navigate the tree by using the cd command to access directories, and can view or modify the corresponding configuration at the location, or invoke API methods and view the current state of components in the node.
The set of commands available to users at a given time depend on the current shell context, for example, their placement inside the logical tree. Certain commands, however, are always available, and we begin with a description of those commands.
System Commands
Shell system commands operate on the shell itself, instead of the nodes of a connected service. These commands are always available.
- Use the
help <command>to explore the available commands. It is always available. nw-shellsupports tab completion of a command, and the applicable parameter names wherever possible. For example, pressing the Tab key after typingecompletes the command toexit.nw-shellalso supports a non-interactive mode where it executes scripts from a provided file, used by specifying the absolute filepath prefixed by the@argument. See NW11NWSHELL,SHELLShell for details.
Available Commands
Built-In Commands
clear: Clear the shell screen.
exit, quit: Exit the shell.
help: Display help about available commands.
history: Display or save the history of previously run commands
script: Read and execute commands from a file. stacktrace: Display the full stacktrace of the last error.
Context Commands
* cd: Change the current node. Usage: cd <path>
connect: Connect to a service. One of --service or --port must be specified. Usage: connect [--service <service>[.<id>]] [--broker
amqp://localhost/rsa/system] [--host localhost] [--port] [--insecure false]
* where: Which service shell is connected to?
Token Commands
login: Authenticate to a service. Usage: login [connect-parameters]
login-insecure: Authenticate to a service providing user and password on the command prompt. The password is recorded in the
shell history so this command must be used with care.Usage: login-insecure --user <user> --password <password> [connect-parameters]
* logout: Clear the authentication context: logout
* whoami: Who am I?
Tree Node Commands
* json: Print the current node as a JSON string
* show: Pretty print the current node
Tree Node List Commands
* config: Summarize configuration of the current subtree
* health: Summarize health of the current subtree
* ls: List the children of the current node. Usage: ls [<filter>] [--values] [--types]
* lsv: Shorthand for ls --values. Usage: lsv [<filter>] [--types]
* method: Summarize methods of the current subtree
* metrics: Summarize metrics of the current subtree
* snapshot: Snapshot the current subtree
Tree Node Method Commands
* invoke: Invokes the method that exists on the current method type node. Usage: invoke [argument] [--file jsonfile]
Tree Node Value Commands
* get: Get the value of the current node
* set: Set the value of the current node. Usage: set <new-value> Commands marked with (*) are currently unavailable.
Type `help <command>` to learn more.
Help, History Command Usage
offline » help connect
connect - Connect to a service. One of --service or --port must be specified. Usage: connect [--service <service>[.<id>]]
[--broker amqp://localhost/rsa/system] [--host localhost] [--port] [--insecure false]
...
...
offline » history
help
help connect
history
offline »
You can navigate previously-typed commands in nw-shell by using the Up and Down arrow keys, which can help minimize typing by recalling previously-executed commands.
Authentication Commands
As an administration and monitoring tool, it is important that nw-shell authenticates users before handing them control over a running service. The following commands manage the shell authentication context:
The login command can be used to authenticate the user and establish an identity. Service operators can log in once (against the NetWitness Security Server) and then use the token to connect to multiple services and perform administration based on the roles assigned. This single-sign-on workflow is simplified by separating the login and connect commands to allow the operator to authenticate once (using login) and then switch services seamlessly using connect.
In general, the login command takes the same parameters as connect (described below) to specify the service that performs the credential validation. For example:
offline » login
user: admin
password: ********
admin@offline » connect --service respond-server
INFO: Connected to respond-server (23e1dab7-0658-41a5-bb1e-d716a37d5ea5)
admin@respond-server:Folder:/rsa »
admin@respond-server:Folder:/rsa » connect --service investigate-server
INFO: Connected to investigate-server 2f21db20-4b50-48cf-8f7f-c0be0d1d1d12)
admin@investigate-server:Folder:/rsa »The current logged-in identity can be confirmed at any point using the whoami command. For example:
admin@offline » whoami
The shell security context can be cleared by using the logout command.
security-server:Folder:/rsa » logout
security-server:Folder:/rsa » whoami
You are not logged in.Some operations, such as setting certain configuration properties or invoking a method, require a certain RBAC. To perform such privileged operations, you need to authenticate with an identity that has the necessary permissions. For example:
security-server:Configuration » /rsa/logging/operations/max-file-count » get 10
security-server:Configuration:/rsa/logging/operations/max-file-count » set 15
ERROR: Failed to set the node value: Access is denied
security-server:Configuration:/rsa/logging/operations/max-file-count » login
user: admin
password: **********
admin@security-server:Configuration:/rsa/logging/operations/max-file-count » set 15
security-server:Configuration:/rsa/logging/operations/max-file-count » get 15
Context-Changing Commands
The following commands change the current shell context.
Connecting to a Service
The shell can connect to services over AMQP or HTTP(S), however, AMQP is the preferred and default option.
To connect to a specific instance of a service, name and serviceId needs to be supplied in the format:connect --service {service-name}.{serivceId}
For example, foo-server.d6a55b48-6103-46bd-9ead-3b4d589b302b.
If the service identifier is skipped, it is assumed to be any, for example, connect --service foo-server will connect with any service named foo-server that is connected to the AMQP broker.
Change Node
The cd command can be used to change the current node. Just like cd on a file system shell, it takes relative or absolute paths as input and changes the current context to the node at that path.
offline » cd Command 'cd' was found but is not currently available because you are not connected to any service.
offline » connect
security-server:Folder:/rsa » cd log
security-server:Folder:/rsa/logging » cd ../security
security-server:Component:/rsa/security » cd /rsa/security/fips-mode
security-server:Configuration:/rsa/security/fips-mode » cd
security-server:Folder:/rsa »The shell prompt summarizes and presents the user’s current context. It starts off with offline, and once the shell is connected to a service, it displays the service name it is connected to, and the type and the path of the current node. Once the user is authenticated, the userId is included in the prompt.
Invoking cd before connecting to a service reminds the user that some commands work only in certain contexts.
Note: Some commands are available only in certain contexts. For example, cd works only when the shell is online. The Help command lists all the commands, but commands marked with (*) are unavailable in the given context.
Certain node-specific commands are enabled only when the current node is of a certain type. For example, method nodes support a command invoke, which is not enabled for any other node type.
Node Display Commands
The following commands are available with all node types. They do not take any arguments, and display the node details for the user to review.
For example:
security-server:Configuration:/rsa/security/authentication/token-lifetime » show
security-server:Configuration » /rsa/security/authentication/token-lifetime » json
{
"path" : "/rsa/security/authentication/token-lifetime",
"type" : "Configuration",
"value" : "10 HOURS",
"parent" : {
"path" : "/rsa/security/authentication",
"type" : "Component"
},
"attributes" : {
"defaultValue" : 10,
"valueType" : "com.rsa.asoc.launch.api.helpers.Seconds",
"description" : "The time-to-live on a token."
}
}The output of json dumps the API payload and may contain more details than those shown by show.
Node Value Commands
Configuration, Metric and Health nodes have values. Their current values can be obtained using the get command.
For example:
security-server:Gauge:/rsa/process/hostname » get
"hostxyz.corp.emc.com"
security-server:Gauge:/rsa/process/hostname » cd /rsa/transport/http/secure
security-server:Configuration:/rsa/transport/http/secure » get
trueThe value of a Configuration node can be changed by invoking the set command. For example:
admin@security-server:Configuration:/rsa/security/pki/tls-protocols » show
admin@security-server:Configuration:/rsa/security/pki/tls-protocols » set '["SSLv3"]'
admin@security-server:Configuration:/rsa/security/pki/tls-protocols » value
[
"SSLv3"
]Node List Commands
Folder, Component and Method nodes can contain other nodes as their children. You can use the ls command to list the children nodes, their types, and where available, their current values.
For example:
security-server:Component:/rsa/process » ls
jvm Component
modules Component
current-time-utc Gauge
current-time-utc-pretty Gauge
fips140-mode Gauge
hostname Gauge
mode Gauge
service-id Gauge
service-name Gauge
status Gauge
uptime Gauge
uptime-pretty Gauge
version Gauge
version-full Gauge
version-raw Gauge
ready Method
shutdown Method
security-server:Component:/rsa/process » ls comp
jvm Component
modules Component
security-server:Component:/rsa/process » ls name:uptime
uptime Gauge
uptime-pretty Gauge
security-server:Component:/rsa/process » ls name:uptime --values
uptime Gauge 3713854
uptime-pretty Gauge 1 hour 1 minute 53 seconds
security-server:Component:/rsa/process » lsv
jvm
modules
current-time-utc 1483978142782
current-time-utc-pretty 2017-01-09T16:09:02.782Z
fips140-mode true
hostname hostxyz.corp.emc.com
mode Normal
service-id 1fb7572a-4d87-497e-a4da-802819c10a72
service-name no-op-server
status Running
uptime 512064
uptime-pretty 8 minutes 32 seconds
version 0.0
version-full 0.0.0.0
version-raw 0.0.0.0
ready
shutdownListing commands are not available on nodes that do not have children (for example, configuration or metric nodes):
security-server:Component:/rsa/process » cd hostname
security-server:Gauge:/rsa/process/hostname » ls
Command 'ls' exists but is not currently available because you are not logged in.Note: Values displayed in ls and lsv may be truncated for presentation purposes. Use get on the value node to get the complete (unaltered) value of a node.
Method Node Commands
The method command shows all the available methods and all the sub-folders under the current node.
security-server:Component:/rsa/process » method /rsa/process/ready /rsa/process/shutdown
On a node that is of Method type , the invoke command will invoke the method. For example:
security-server:Method:/rsa/process/ready » invoke
{
"ready" : true,
"serviceId" : "84f36740-5ae7-409f-b14f-b17e98703983",
"marketingVersion" : "0.0"}
If the method signature takes input, then invoke <input> will invoke the method with given parameters.
security-server:Method:/rsa/health/get » show
Method:/rsa/health/get » invoke 'rsa.health.checks.security-pki'
[
{
"name" : "rsa.security.pki.pki-health",
"status" : "Unhealthy",
"details" : {
"Reason" : "Using a self-signed certificate" }
}
] For inline method execution with input payload, in cases where the input contains any special characters such as \, \n, \r, \t , ", the input needs to be properly escaped with a backslash (\).
In cases where method takes a complex payload, payload can be supplied with a file reference. In this case, payload does not need any special escaping.
Method:/rsa/configuration/collections/register » invoke --file /tmp/jsonfile.txtRun State of a Service
Once nw-shell is connected to a service, commands like health, metrics, config or snapshot can be used to retrieve the current state of the service at the current node, and all the sub-folders under the current node.
Health
The health command lists the health of the components, and the all the sub-folders, under the current node.
admin@security-server:Component:/rsa/process » health
/rsa/process/jvm/memory-health Healthy
/rsa/process/modules/module-health Healthy
Metrics
The metrics command lists all of the available metrics, and all the sub-folders, under the current node.
admin@respond-server:Component:/rsa/tree/node » metrics
/rsa/tree/node/get/invoked 14
/rsa/tree/node/get/timer 5666541.077918259
/rsa/tree/node/list/invoked 12
/rsa/tree/node/list/timer 6652400.379371729
In the above example, /rsa/tree/node/get/invoked 14 shows that the method get is invoked 14 times, and timer shows that it took on average 5666541.077918259 nano seconds to process the get request.
Using cd into the timer node shows more details, such as rate of requests, for the current node for per-minute and five minutes. For example:
admin@respond-server:Timer:/rsa/tree/node/get/timer » show
Config
The config command shows all the configuration properties in the current node and all the sub-folders under the current node.
admin@respond-server:Component:/rsa/logging » config
/rsa/logging/audit/max-file-count 10
/rsa/logging/audit/max-file-size 10 MB
/rsa/logging/forward/categories [Audit]
/rsa/logging/forward/destination SYSLOG_UDP
/rsa/logging/forward/enabled true
/rsa/logging/forward/host localhost
/rsa/logging/forward/port 50514
/rsa/logging/forward/secure false
/rsa/logging/levels
/rsa/logging/operations/max-file-count 10
/rsa/logging/operations/max-file-size 10 MB
Snapshot
The snapshot command combines metrics, configuration, and health of the current state of the service node and all the sub-folders under the current node. This command is very useful for troubleshooting purposes. When opening a support case, take a snapshot dump at the root node \rsa of a service that might be having issues, and attach it to the case.
admin@security-server:Component:/rsa/security » snapshot
/rsa/security/account/external/get-all/invoked 16
/rsa/security/account/external/get-all/timer NaN
/rsa/security/account/external/get/failed 6
/rsa/security/account/external/get/invoked 6
/rsa/security/account/external/get/timer NaN
/rsa/security/account/force-password-change/invoke 1
/rsa/security/account/force-password-change/timer NaN
/rsa/security/account/get/invoked 123
/rsa/security/account/get/timer 902163.0
/rsa/security/account/password-policy/cannot-inclu false
/rsa/security/account/password-policy/min-chars 8
/rsa/security/account/password-policy/min-lower-ch 0
/rsa/security/account/password-policy/min-non-lati 0
/rsa/security/account/password-policy/min-numeric- 0
/rsa/security/account/password-policy/min-special- 0
----
Scripting
nw-shell supports non-interactive executions that can take commands from files. For example:
> cat /tmp/foo.script
connect --service security-server
cd /rsa/security/pki/ciphers
get
> time nw-shell @/tmp/foo.script
INFO: Connected to security-server (52c7b92c-23d5-4b3e-9973-31d8b1b27ec4)
[
"TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256",
"TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA",
"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
"TLS_DHE_RSA_WITH_AES_128_CBC_SHA"
]
nw-shell 5.89s user 0.33s system 305% cpu 2.032 totalnw-shell also has a built-in script which can be used to execute scripts in the shell once the shell is launched.
offline » script --file /tmp/foo.script
INFO: Connected to security-server (52c7b92c-23d5-4b3e-9973-31d8b1b27ec4)
[
"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
"TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256",
"TLS_DHE_RSA_WITH_AES_128_GCM_SHA256",
"TLS_RSA_WITH_AES_128_CBC_SHA256"
]
Note: nw-shell is primarily meant to help a human to explore the runtime state of a NetWitness Platform service. For most other purposes, it is usually best to use the service published APIs.
Troubleshooting Commands
The nw-shell utility provides troubleshooting commands to help recover from misconfigurations that may cause the service to work improperly.
fix-keystore
When a service’s keystore is corrupt or its SSL trust is broken with the rest of NetWitness Platform, you can use the fix-keystore command to fix this issue. This command reads the keystore file of the services running on the host and attempts to reestablish trust with the NetWitness Platform CA. For example, the following is the command to repair the respond-server 's keystore:
>> fix-keystore --service respond-server
This command repairs the keystore keystore.p12 that is located at /etc/netwitness/respond-server. It does not modify the service’s original keystore. The repaired keystore can be found with the suffix
.good, for example, keystore.p12.good. If no fixes can be made to the keystore, this command does not write the new keystore.p12.good file.
Once the keystore repair is successful, the system admin can restore the service’s communication by renaming keystore.p12.good to keystore.p12 and restart the service.
print-keystore
The print-keystore command prints the requested NetWitness Platform Service’s keystore certificates to the given output file in JSON format. Unless verbose is enabled, only a few key attributes of the certificates are printed to the file. For example, the following command prints the respond-server's keystore certificates.
>> print-keystore respond-server outputFile /tmp/crts
With verbose enabled:
>> print-keystore respond-server outputFile /tmp/crts verbose
reconstruct-keystore
The reconstruct-keystore command can be used when a service's keystore needs to be restored to its original clean state.
The following command reconstructs the respond-server's keystore to the new file /etc/netwitness/respond- server/keystore.p12.new (it does not modify the original keystore).
>> reconstruct-keystore --service respond-server
To restore the service back to its healthy state, restore keystore.p12.new to keystore.p12, and restart the service.
Advanced Customization
The following JVM system properties can be used to customize certain aspects of the nw-shell presentation:
These can be specified using the JAVA_OPTS environment variable. For example:
> export JAVA_OPTS="-Dtimeout=100s -Dconsole.width=40 -Dconsole.prompt=%s%s%s>" > export JAVA_OPTS="-Dconsole.colors=false" > nw-shell
The API timeout is a time unit which can use typical time unit values such as 30 MINUTES, 30 mins, 30 m, 5 seconds, 5 sec, 5s, 5, 8000 milliseconds, or 8000 ms (ms = milliseconds).
The value of console.prompt must contain placeholders (for example, %s) for three strings which are replaced, in order, by:
- The node type
- A separator (
:) - The node path
Setting console.colors to false will not turn off all use of colors, it only disables colors used by nw-shell in its output. The underlying libraries that the program uses may still output some text in color.
Table of Contents > shell