503 Error with message 'No record found for selection of trigger' in 11.4.1.0

[RichardB]

I just had an interesting upgrade experience. On our test environment I did an upgrade from NetWitness 11.4.0.1 to 11.4.1.0 and got stuck with the UI not coming up. After some investigation the culprit showed in /var/lib/netwitness/uax/logs/sa.log :

2020-04-15 12:47:18,355 [main] WARN  org.springframework.web.context.support.XmlWebApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'org.springframework.scheduling.quartz.SchedulerFactoryBean#0'; nested exception is org.springframework.scheduling.SchedulingException: Could not start Quartz Scheduler; nested exception is org.quartz.SchedulerConfigException: Failure occured during job recovery. [See nested exception: org.quartz.JobPersistenceException: Couldn't retrieve trigger: No record found for selection of Trigger with key: 'DEFAULT.8c0550a9-a782-40f4-a66d-32c69e7080b5' and statement: SELECT * FROM QRTZ_SIMPLE_TRIGGERS WHERE SCHED_NAME = 'org.springframework.scheduling.quartz.SchedulerFactoryBean#0' AND TRIGGER_NAME = ? AND TRIGGER_GROUP = ? [See nested exception: java.lang.IllegalStateException: No record found for selection of Trigger with key: 'DEFAULT.8c0550a9-a782-40f4-a66d-32c69e7080b5' and statement: SELECT * FROM QRTZ_SIMPLE_TRIGGERS WHERE SCHED_NAME = 'org.springframework.scheduling.quartz.SchedulerFactoryBean#0' AND TRIGGER_NAME = ? AND TRIGGER_GROUP = ?]]

Searching on the RSA Community site let to 000017462 - 503 Error with message 'No record found for selection of trigger' in RSA Security Analyt...  which looked promising but is very outdated. It seems a newer version of H2 is being used that now stores data in platform.mv.db (instead of the old platform.h2.db).

I was able this fix this by replacing the first steps of DOC-47175 with:

1. systemctl stop jetty
2. cd /var/lib/netwitness/uax/db
3. cp platform.mv.db platform.mv.db.backup
4. wget https://repo1.maven.org/maven2/com/h2database/h2/1.4.199/h2-1.4.199.jar
5. java -cp h2-1.4.199.jar org.h2.tools.Shell -url jdbc:h2:file:./platform

The SQL part of DOC-47175 is still the same:

1. sql> DELETE FROM QRTZ_TRIGGERS WHERE JOB_NAME = '<triggerName>';
2. sql> DELETE FROM QRTZ_JOB_DETAILS WHERE JOB_NAME = '<triggerName>';
3. sql > quit

After a "systemctl start jetty" all is fine again.

[JohnSnider]

This was probably no due to the upgrade, but due to an old job (custom feed, subscription deployment, etc.  that was not completely removed at some time in the past, and the restart of jetty during the upgrade was still looking for a now non-existent job. I have seen this error has occur on just about every version of NW platform from time to time and you found the correct fix.  I agree the article needs to be updated, and I see about getting that done.

[RichardB]

I totally agree this error was not caused by the upgrade, but would have happened at any time we restarted jetty.

Interestingly enough the jettyuax package is still at 11.4.0.0 but I guess the root.war which belongs to rsa-nw-legacy-web-server-11.4.1.0 is more important for the functioning of the UI. A problem with the root.war was my first guess until I found the SchedulingException in the logs.