000029690 - Annodb does not start up properly in RSA Web Threat Detection 5.x

Document created by RSA Customer Support Employee on Jan 31, 2017Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000029690
Applies ToRSA Product Set: Web Threat Detection
RSA Product/Service Type: Forensics
RSA Version/Condition: 5.x
Platform: Linux
O/S Version: Red Hat Enterprise Linux 6.x / CentOS 6.x
IssueWhen working on the server that houses Annodb, errors similar to the following are reported:
 

Feb 23 11:54:18 <server name> annodb[<PID>]:MainThread:INFO:Silver Tail AnnoDb version 5.0.2.3
Feb 23 11:54:18 <server name> annodb[<PID>]:MainThread:INFO:AnnoDb out--SERVICE_NAME} starting database.....PGDATA=/var/opt/silvertail/srv/annodb, PGPORT=7078 MAX_WAIT=5<< AnnoDb postmaster already running pg_ctl returns 0<<<
Feb 23 11:54:18 <server name> annodb[<PID>]:MainThread:INFO:AnnoDb out--SERVICE_NAME} get database status pg_ctl: no server running
Feb 23 11:54:18 <server name> annodb[<PID>]:MainThread:ERROR:AnnoDb not running.
<<<<<<<<<<<<<PID will increase at this time>>>>>>>>>>>>>>>>
Feb 23 11:54:18 <server name> annodb[<PID>]:MainThread:INFO:Silver Tail AnnoDb version 5.0.2.3
Feb 23 11:54:18 <server name> annodb[<PID>]:MainThread:INFO:AnnoDb out--SERVICE_NAME} starting database.....PGDATA=/var/opt/silvertail/srv/annodb, PGPORT=7078 MAX_WAIT=5<< AnnoDb postmaster already running pg_ctl returns 0<<<
Feb 23 11:54:18 <server name> annodb[<PID>]:MainThread:INFO:AnnoDb out--SERVICE_NAME} get database status pg_ctl: no server running
Feb 23 11:54:18 <server name> annodb[<PID>]:MainThread:ERROR:AnnoDb not running.
<<<<<<<<<<<<<PID will increase at this time>>>>>>>>>>>>>>>>
Feb 23 11:54:19 <server name> annodb[<PID>]:MainThread:INFO:Silver Tail AnnoDb version 5.0.2.3
Feb 23 11:54:19 <server name> annodb[<PID>]:MainThread:INFO:AnnoDb out--SERVICE_NAME} starting database.....PGDATA=/var/opt/silvertail/srv/annodb, PGPORT=7078 MAX_WAIT=5<< AnnoDb postmaster already running pg_ctl returns 0<<<
Feb 23 11:54:19 <server name> annodb[<PID>]:MainThread:INFO:AnnoDb out--SERVICE_NAME} get database status pg_ctl: no server running
Feb 23 11:54:19 <server name> annodb[<PID>]:MainThread:ERROR:AnnoDb not running.
<<<<<<<<<<<<<PID will increase at this time>>>>>>>>>>>>>>>>
Feb 23 11:54:19 <server name> annodb[<PID>]:MainThread:INFO:Silver Tail AnnoDb version 5.0.2.3
Feb 23 11:54:19 <server name> annodb[<PID>]:MainThread:INFO:AnnoDb out--SERVICE_NAME} starting database.....PGDATA=/var/opt/silvertail/srv/annodb, PGPORT=7078 MAX_WAIT=5<< AnnoDb postmaster already running pg_ctl returns 0<<<
Feb 23 11:54:19 <server name> annodb[<PID>]:MainThread:INFO:AnnoDb out--SERVICE_NAME} get database status pg_ctl: no server running
Feb 23 11:54:19 <server name> annodb[<PID>]:MainThread:ERROR:AnnoDb not running.
<<<<<<<<<<<<<PID will increase at this time>>>>>>>>>>>>>>>>
Feb 23 11:54:20 <server name> annodb[<PID>]:MainThread:INFO:Silver Tail AnnoDb version 5.0.2.3
Feb 23 11:54:20 <server name> annodb[<PID>]:MainThread:INFO:AnnoDb out--SERVICE_NAME} starting database.....PGDATA=/var/opt/silvertail/srv/annodb, PGPORT=7078 MAX_WAIT=5<< AnnoDb postmaster already running pg_ctl returns 0<<<
Feb 23 11:54:20 <server name> annodb[<PID>]:MainThread:INFO:AnnoDb out--SERVICE_NAME} get database status pg_ctl: no server running
Feb 23 11:54:20 <server name> annodb[<PID>]:MainThread:ERROR:AnnoDb not running.

 
CauseThis is caused by the AnnoDb server not starting because of the server not shutting down properly. Not shutting down properly will leave the file "postmaster.pid" in the /var/opt/silvertail/srv/annodb/ directory.
This file is specifically looked for when starting the service. If that file is found it will error with this message:  ${SERVICE_NAME} postmaster already running
This error message can be seen in the example output above.  This is written into the start up script as a precaution.
 
Resolution

In most cases, simply removing the postmaster.pid file from the /var/opt/silvertail/srv/annodb/ directory will allow the service to start. However, if the postmaster.pid file is there because AnnoDb never stopped properly then there may be some stuck processes or possibly some processes that are still busy.
To quickly check for any of those services, issue the following command:  ps -ef | grep anno
If you see any processes returned other then the grep command, AnnoDb has not yet shut down properly.
If you see no AnnoDb processes, you can also run this command to look for any lingering postgres processes:  ps -ef | grep postgres
These processes are either started by postgres or lingering processes from a pervious startup. In this situation, attempt to stop these processes by doing a server restart, if possible.
This should clean up any lingering postgres processes and the AnnoDb service should start properly on the next boot, if the postmaster.pid file was removed properly.

Attachments

    Outcomes