000038205 - Repairing corrupt MongoDB in RSA NetWitness Platform

Document created by RSA Customer Support Employee on Dec 10, 2019
Version 1Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000038205
Applies ToRSA Product Set: NetWitness Platform
RSA Product/Service Type: Admin Server, ESA, MongoDB
RSA Version/Condition: 11.X
Platform: CentOS
O/S Version: 7
IssueDue to unforeseen circumstances (e.g: unexpected shutdown), MongoDB on the NetWitness Admin Server / ESA appliance can become corrupt.
The most obvious symptoms of corruption include:
  1. mongod service failing to start:

    [root@nwsvr~]# systemctl status mongod
    ● mongod.service - High-performance, schema-free document-oriented database
       Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
       Active: failed (Result: start-limit) since Thu 2019-12-05 03:14:05 UTC; 44s ago
      Process: 2854 ExecStart=/usr/bin/mongod $OPTIONS (code=exited, status=14)
      Process: 2851 ExecStartPre=/bin/bash -c echo never > /sys/kernel/mm/transparent_hugepage/defrag (code=exited, status=0/SUCCESS)
      Process: 2848 ExecStartPre=/bin/bash -c echo never > /sys/kernel/mm/transparent_hugepage/enabled (code=exited, status=0/SUCCESS)
      Process: 2845 ExecStartPre=/usr/bin/chown -R mongod:mongod /var/run/mongodb (code=exited, status=0/SUCCESS)
      Process: 2843 ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb (code=exited, status=0/SUCCESS)
     Main PID: 1729 (code=exited, status=0/SUCCESS)

    Dec 05 03:14:05 nwsvr systemd[1]: mongod.service: control process exited, code=exited status=14
    Dec 05 03:14:05 nwsvr systemd[1]: Failed to start High-performance, schema-free document-oriented database.
    Dec 05 03:14:05 nwsvr systemd[1]: Unit mongod.service entered failed state.
    Dec 05 03:14:05 nwsvr systemd[1]: mongod.service failed.
    Dec 05 03:14:05 nwsvr systemd[1]: mongod.service holdoff time over, scheduling restart.
    Dec 05 03:14:05 nwsvr systemd[1]: start request repeated too quickly for mongod.service
    Dec 05 03:14:05 nwsvr systemd[1]: Failed to start High-performance, schema-free document-oriented database.
    Dec 05 03:14:05 nwsvr systemd[1]: Unit mongod.service entered failed state.
    Dec 05 03:14:05 nwsvr systemd[1]: mongod.service failed.


    AND:
     
  2. /var/log/mongodb/mongod.log being flooded with repeated messages similar to these:

    2019-12-05T04:07:54.321+0000 I CONTROL  [main] ***** SERVER RESTARTED *****
    2019-12-05T04:07:54.404+0000 I CONTROL  [initandlisten] MongoDB starting : pid=17354 port=27017 dbpath=/var/netwitness/mongo 64-bit host=nwsvr
    2019-12-05T04:07:54.404+0000 I CONTROL  [initandlisten] db version v3.6.4
    2019-12-05T04:07:54.404+0000 I CONTROL  [initandlisten] git version: d0181a711f7e7f39e60b5aeb1dc7097bf6ae5856
    2019-12-05T04:07:54.404+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
    2019-12-05T04:07:54.404+0000 I CONTROL  [initandlisten] allocator: tcmalloc
    2019-12-05T04:07:54.404+0000 I CONTROL  [initandlisten] modules: none
    2019-12-05T04:07:54.404+0000 I CONTROL  [initandlisten] build environment:
    2019-12-05T04:07:54.404+0000 I CONTROL  [initandlisten]     distmod: rhel70
    2019-12-05T04:07:54.404+0000 I CONTROL  [initandlisten]     distarch: x86_64
    2019-12-05T04:07:54.404+0000 I CONTROL  [initandlisten]     target_arch: x86_64
    2019-12-05T04:07:54.404+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "0.0.0.0", port: 27017, ssl: { CAFile: "/etc/pki/nw/trust/truststore.pem", PEMKeyFile: "/etc/pki/nw/mongo/mongod-combined.pem", allowConnectionsWithoutCertificates: true, allowInvalidCertificates: false, allowInvalidHostnames: true, disabledProtocols: "TLS1_0,TLS1_1", mode: "preferSSL", sslCipherConfig: "HIGH:!EXPORT:!aNULL@STRENGTH:!kRSA" } }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, security: { authorization: "enabled" }, setParameter: { internalQueryExecMaxBlockingSortBytes: "134217728", opensslDiffieHellmanParameters: "/etc/pki/nw/mongo/dhparam2048.pem" }, storage: { dbPath: "/var/netwitness/mongo", journal: { enabled: true }, wiredTiger: { engineConfig: { cacheSizeGB: 16.0 } } }, systemLog: { destination: "file", logAppend: true, logRotate: "reopen", path: "/var/log/mongodb/mongod.log" } }
    2019-12-05T04:07:54.405+0000 I -        [initandlisten] Detected data files in /var/netwitness/mongo created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
    2019-12-05T04:07:54.405+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=16384M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),cache_cursors=false,log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
    2019-12-05T04:07:55.052+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1575518875:52721][17354:0x7f3478cfab00], file:WiredTiger.wt, WT_CURSOR.insert: read checksum error for 20480B block at offset 942080: block header checksum of 0 doesn't match expected checksum of 662513522
    2019-12-05T04:07:55.052+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1575518875:52886][17354:0x7f3478cfab00], file:WiredTiger.wt, WT_CURSOR.insert: {942080, 20480, 662513522}: (chunk 1 of 20): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
    <snipped>
    2019-12-05T04:07:55.056+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1575518875:56262][17354:0x7f3478cfab00], file:WiredTiger.wt, WT_CURSOR.insert: {942080, 20480, 662513522}: (chunk 20 of 20): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
    2019-12-05T04:07:55.056+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1575518875:56485][17354:0x7f3478cfab00], file:WiredTiger.wt, WT_CURSOR.insert: WiredTiger.wt: encountered an illegal file format or internal value: (__wt_block_read_off, 302)
    2019-12-05T04:07:55.056+0000 E STORAGE  [initandlisten] WiredTiger error (-31804) [1575518875:56504][17354:0x7f3478cfab00], file:WiredTiger.wt, WT_CURSOR.insert: the process must exit and restart: WT_PANIC: WiredTiger library panic
    2019-12-05T04:07:55.056+0000 F -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
    2019-12-05T04:07:55.056+0000 F -        [initandlisten] 

    ***aborting after fassert() failure


    2019-12-05T04:07:55.149+0000 F -        [initandlisten] Got signal: 6 (Aborted).

     0x55b88862a861 0x55b888629a79 0x55b888629f5d 0x7f34777766d0 0x7f34773d0277 0x7f34773d1968 0x55b886d8b425 0x55b886e582ce 0x55b886ec4ba1 0x55b886d2827e 0x55b886d285a7 0x55b886f73997 0x55b886f73ad5 0x55b886eec7a3 0x55b886ef2e71 0x55b886f160a6 0x55b886f861b7 0x55b886f2f006 0x55b886edbf9a 0x55b886f53ae4 0x55b886edcf77 0x55b886e6c597 0x55b886e6923c 0x55b886e3b95e 0x55b886e1ff04 0x55b887014b67 0x55b886d240e7 0x55b886dff18c 0x55b886d8d0e9 0x7f34773bc445 0x55b886deea9f
    ----- BEGIN BACKTRACE -----
    {"backtrace":[{"b":"55B8863FC000","o":"222E861","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55B8863FC000","o":"222DA79"},{"b":"55B8863FC000","o":"222DF5D"},{"b":"7F3477767000","o":"F6D0"},{"b":"7F347739A000","o":"36277","s":"gsignal"},{"b":"7F347739A000","o":"37968","s":"abort"},{"b":"55B8863FC000","o":"98F425","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55B8863FC000","o":"A5C2CE"},{"b":"55B8863FC000","o":"AC8BA1"},{"b":"55B8863FC000","o":"92C27E","s":"__wt_err"},{"b":"55B8863FC000","o":"92C5A7","s":"__wt_panic"},{"b":"55B8863FC000","o":"B77997","s":"__wt_block_read_off"},{"b":"55B8863FC000","o":"B77AD5","s":"__wt_bm_read"},{"b":"55B8863FC000","o":"AF07A3","s":"__wt_bt_read"},{"b":"55B8863FC000","o":"AF6E71","s":"__wt_page_in_func"},{"b":"55B8863FC000","o":"B1A0A6","s":"__wt_row_search"},{"b":"55B8863FC000","o":"B8A1B7","s":"__wt_btcur_insert"},{"b":"55B8863FC000","o":"B33006"},{"b":"55B8863FC000","o":"ADFF9A"},{"b":"55B8863FC000","o":"B57AE4","s":"__wt_log_scan"},{"b":"55B8863FC000","o":"AE0F77","s":"__wt_txn_recover"},{"b":"55B8863FC000","o":"A70597","s":"__wt_connection_workers"},{"b":"55B8863FC000","o":"A6D23C","s":"wiredtiger_open"},{"b":"55B8863FC000","o":"A3F95E","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"55B8863FC000","o":"A23F04"},{"b":"55B8863FC000","o":"C18B67","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"55B8863FC000","o":"9280E7"},{"b":"55B8863FC000","o":"A0318C","s":"_ZN5mongo11mongoDbMainEiPPcS1_"},{"b":"55B8863FC000","o":"9910E9","s":"main"},{"b":"7F347739A000","o":"22445","s":"__libc_start_main"},{"b":"55B8863FC000","o":"9F2A9F"}],"processInfo":{ "mongodbVersion" : "3.6.4", "gitVersion" : "d0181a711f7e7f39e60b5aeb1dc7097bf6ae5856", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-862.11.6.el7.x86_64", "version" : "#1 SMP Tue Aug 14 21:49:04 UTC 2018", "machine" : "x86_64" }, "somap" : [ { "b" : "55B8863FC000", "elfType" : 3, "buildId" : "37115F864F27162060BAC158CC814CBD48741B71" }, { "b" : "7FFC4ED60000", "elfType" : 3, "buildId" : "2562E90AE9BCFB0D02589A7C3B20B07AC58D6A74" }, { "b" : "7F34788D5000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "2BDC2B6FF0B2C204CCE34D139A9EADA0272EB070" }, { "b" : "7F3478662000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "9112A44B60B06C5FE236C7B1EF38A8D74C16AC0C" }, { "b" : "7F34782A7000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "C4F0A80793B40F2E40596C49987D21EE7B9C4742" }, { "b" : "7F347809F000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "D33989EC31EFE745EB0D3B68A92D19E77D7DDFDA" }, { "b" : "7F3477E9B000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "5CDB5A56336E7E2BD14FFA189411E44A834AFCD8" }, { "b" : "7F3477B99000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "F4CAE74047F9AA2D5A71FDEC67C4285D75753EBA" }, { "b" : "7F3477983000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "531AA1391EA4E1489D5EF11AA5DC2FFD9E2BDFEE" }, { "b" : "7F3477767000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "F4C04BCE85D2D269D0A2AF4972FC69805B50345B" }, { "b" : "7F347739A000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "CB4B7554D1ADBEF2F001142DD6F0A5139FC9AA69" }, { "b" : "7F3478AEE000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D266B1F6650927E18108323BCCA8F7B68E68EB92" }, { "b" : "7F347714D000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "16FE0DC6CEFC5F444BC876516D02EFE9CC2D432F" }, { "b" : "7F3476E65000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "D1CD1B94855A85FBC735C745DB39BC096F7D8CC3" }, { "b" : "7F3476C61000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "A3832734347DCA522438308C9F08F45524C65C9B" }, { "b" : "7F3476A2E000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "A20F715C514B3EA873F4CC77D585A50CB670E266" }, { "b" : "7F3476818000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "EA8E45DC8E395CC5E26890470112D97A1F1E0B65" }, { "b" : "7F34765C7000", "path" : "/lib64/libbwrap.so.1", "elfType" : 3, "buildId" : "C8B57E5EDB72A0EB6BF8E4ABFD978CCC0CF7A12A" }, { "b" : "7F34763B9000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "9F82B06CE44724A8ACE827A2C95A9A76619EA314" }, { "b" : "7F34761B5000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7F3475F8E000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "A88379F56A51950A33198890D37F5F8AEE71F8B4" }, { "b" : "7F3475D2C000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "9CA3D11F018BEEB719CDB34BE800BF1641350D0A" }, { "b" : "7F3475ACE000", "path" : "/usr/lib64/bwrap-1.2.2.2/libcryptocme.so", "elfType" : 3, "buildId" : "2CDC228D1BF20A18FB4BC1BCD9ECBE8F6212AC9F" }, { "b" : "7F34758CA000", "path" : "/usr/lib64/bwrap-1.2.2.2/libccme_error_info.so", "elfType" : 3, "buildId" : "848E659A077F1D14A8AD96533D18A2BF2E012877" }, { "b" : "7F34756C6000", "path" : "/usr/lib64/bwrap-1.2.2.2/libccme_aux_entropy.so", "elfType" : 3, "buildId" : "FF51846DA67E57C6BC3BACF69B084C9D45DF5CD4" }, { "b" : "7F347542E000", "path" : "/usr/lib64/bwrap-1.2.2.2/libccme_base.so", "elfType" : 3, "buildId" : "A31E5C5B44F05A120D258AA19A59658E4EBCDE7D" }, { "b" : "7F34751D0000", "path" : "/usr/lib64/bwrap-1.2.2.2/libccme_asym.so", "elfType" : 3, "buildId" : "F44DA44D4B1515200C589C0A77DFBD2E76668BC0" }, { "b" : "7F3474F0B000", "path" : "/usr/lib64/bwrap-1.2.2.2/libccme_ecc_accel_fips.so", "elfType" : 3, "buildId" : "3980C59BA73A4E76F4CD807578C600E91D2D4F18" }, { "b" : "7F3474C6C000", "path" : "/usr/lib64/bwrap-1.2.2.2/libccme_ecc.so", "elfType" : 3, "buildId" : "7365F5D8E99D2A82BF26EE72B63C80137FC03D9D" }, { "b" : "7F34749DE000", "path" : "/usr/lib64/bwrap-1.2.2.2/libccme_base_non_fips.so", "elfType" : 3, "buildId" : "65ADC3FB0F2E7E7733198E983BA78215440DACA1" }, { "b" : "7F3474721000", "path" : "/usr/lib64/bwrap-1.2.2.2/libccme_ecc_accel_non_fips.so", "elfType" : 3, "buildId" : "50EEBC9B0FF5EE82BA77E82895F1AD474A21D490" }, { "b" : "7F34744CA000", "path" : "/usr/lib64/bwrap-1.2.2.2/libccme_ecc_non_fips.so", "elfType" : 3, "buildId" : "439D6B449A34920838E74828E88D420CBC1BAC3E" } ] }}
     mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55b88862a861]
     mongod(+0x222DA79) [0x55b888629a79]
     mongod(+0x222DF5D) [0x55b888629f5d]
     libpthread.so.0(+0xF6D0) [0x7f34777766d0]
     libc.so.6(gsignal+0x37) [0x7f34773d0277]
     libc.so.6(abort+0x148) [0x7f34773d1968]
     mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55b886d8b425]
     mongod(+0xA5C2CE) [0x55b886e582ce]
     mongod(+0xAC8BA1) [0x55b886ec4ba1]
     mongod(__wt_err+0x9D) [0x55b886d2827e]
     mongod(__wt_panic+0x33) [0x55b886d285a7]
     mongod(__wt_block_read_off+0x547) [0x55b886f73997]
     mongod(__wt_bm_read+0x135) [0x55b886f73ad5]
     mongod(__wt_bt_read+0x203) [0x55b886eec7a3]
     mongod(__wt_page_in_func+0x19F1) [0x55b886ef2e71]
     mongod(__wt_row_search+0x856) [0x55b886f160a6]
     mongod(__wt_btcur_insert+0x1027) [0x55b886f861b7]
     mongod(+0xB33006) [0x55b886f2f006]
     mongod(+0xADFF9A) [0x55b886edbf9a]
     mongod(__wt_log_scan+0xCE4) [0x55b886f53ae4]
     mongod(__wt_txn_recover+0x417) [0x55b886edcf77]
     mongod(__wt_connection_workers+0x37) [0x55b886e6c597]
     mongod(wiredtiger_open+0x196C) [0x55b886e6923c]
     mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x89E) [0x55b886e3b95e]
     mongod(+0xA23F04) [0x55b886e1ff04]
     mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x637) [0x55b887014b67]
     mongod(+0x9280E7) [0x55b886d240e7]
     mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0x86C) [0x55b886dff18c]
     mongod(main+0x9) [0x55b886d8d0e9]
     libc.so.6(__libc_start_main+0xF5) [0x7f34773bc445]
     mongod(+0x9F2A9F) [0x55b886deea9f]
    -----  END BACKTRACE  -----


     
ResolutionThe mongod --repair command can be used to attempt a recover on the corrupt MongoDB:
  1. Stop the mongod service:
    systemctl stop mongod
     
  2. Run the mongod --repair command:
    mongod --dbpath /var/netwitness/mongo --repair
     
  3. Once the repair has finished running, change the dbpath owner back to mongod:
    chown -R mongod:mongod /var/netwitness/mongo
     
  4. Start the mongod service:
    systemctl start mongod
     
Verify that you are now able to access the NetWitness Web UI, as well as the pages that access the MongoDB, e.g: Health & Wellness, Respond, etc.

NOTE: If the repair in step 3 above fails with the following messages, chances are you have hit known issues with the repair functionality in MongoDB versions 3.02 - 4.02 (NetWitness 11.X uses version 3.6.4) as documented in Improved mongod --repair option for WiredTiger :
 

2019-12-05T03:13:51.242+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1575515631:242064][2733:0x7fbf82945b00], file:WiredTiger.wt, WT_CURSOR.insert: WiredTiger.wt: encountered an illegal file format or internal value: (__wt_block_read_off, 302)
2019-12-05T03:13:51.242+0000 E STORAGE  [initandlisten] WiredTiger error (-31804) [1575515631:242077][2733:0x7fbf82945b00], file:WiredTiger.wt, WT_CURSOR.insert: the process must exit and restart: WT_PANIC: WiredTiger library panic
2019-12-05T03:13:51.242+0000 F -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2019-12-05T03:13:51.242+0000 F -        [initandlisten]


***aborting after fassert() failure


In this situation, the solution would be:

1. Stop the mongod service:
systemctl stop mongod

2. Download and install MongoDB 4.0.3 or newer on another machine, e.g: Windows. The current release is 4.2.1 as at the time of this writing.

3. Copy the entire /var/netwitness/mongo folder from the NetWitness appliance to this new machine.

4. On the new machine, run the mongod --repair command, pointing to the corrupt database folder. For example, if you have copied it to C:\mongo on a Windows machine:
C:\Program Files\MongoDB\Server\4.2\bin> mongod --dbpath C:\mongo --repair

5. After the repair process completes, on the NetWitness appliance, rename the existing /var/netwitness/mongo folder to make way for the repaired MongoDB folder from the previous step:
mv /var/netwitness/mongo /var/netwitness/mongo-corrupt
Perhaps a better idea would be to move the folder out of the appliance and then delete it if disk space is a concern.

6. Copy the repaired MongoDB folder back to /var/netwitness/mongo.

7. Change /var/netwitness/mongo owner to mongod:
chown -R mongod:mongod /var/netwitness/mongo

8. Start the mongod service:
systemctl start mongod

Verify that you are now able to access the NetWitness Web UI, as well as the pages that access the MongoDB, e.g: Health & Wellness, Respond etc.


If you are unsure of any of the steps above or experience any issues, contact RSA Support and quote this article number for further assistance.
 

Attachments

    Outcomes