000038205 - Repairing corrupt MongoDB in RSA NetWitness Platform

1. mongod service failing to start:
   

   
   [root@nwsvr~]# systemctl status mongod
   ● mongod.service - High-performance, schema-free document-oriented database
      Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
      Active: failed (Result: start-limit) since Thu 2019-12-05 03:14:05 UTC; 44s ago
     Process: 2854 ExecStart=/usr/bin/mongod $OPTIONS (code=exited, status=14)
     Process: 2851 ExecStartPre=/bin/bash -c echo never > /sys/kernel/mm/transparent_hugepage/defrag (code=exited, status=0/SUCCESS)
     Process: 2848 ExecStartPre=/bin/bash -c echo never > /sys/kernel/mm/transparent_hugepage/enabled (code=exited, status=0/SUCCESS)
     Process: 2845 ExecStartPre=/usr/bin/chown -R mongod:mongod /var/run/mongodb (code=exited, status=0/SUCCESS)
     Process: 2843 ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb (code=exited, status=0/SUCCESS)
    Main PID: 1729 (code=exited, status=0/SUCCESS)
   
   Dec 05 03:14:05 nwsvr systemd[1]: mongod.service: control process exited, code=exited status=14
   Dec 05 03:14:05 nwsvr systemd[1]: Failed to start High-performance, schema-free document-oriented database.
   Dec 05 03:14:05 nwsvr systemd[1]: Unit mongod.service entered failed state.
   Dec 05 03:14:05 nwsvr systemd[1]: mongod.service failed.
   Dec 05 03:14:05 nwsvr systemd[1]: mongod.service holdoff time over, scheduling restart.
   Dec 05 03:14:05 nwsvr systemd[1]: start request repeated too quickly for mongod.service
   Dec 05 03:14:05 nwsvr systemd[1]: Failed to start High-performance, schema-free document-oriented database.
   Dec 05 03:14:05 nwsvr systemd[1]: Unit mongod.service entered failed state.
   Dec 05 03:14:05 nwsvr systemd[1]: mongod.service failed.

   
   
   AND:
    
2. /var/log/mongodb/mongod.log being flooded with repeated messages similar to these:
   

   
   2019-12-05T04:07:54.321+0000 I CONTROL  [main] ***** SERVER RESTARTED *****
   2019-12-05T04:07:54.404+0000 I CONTROL  [initandlisten] MongoDB starting : pid=17354 port=27017 dbpath=/var/netwitness/mongo 64-bit host=nwsvr
   2019-12-05T04:07:54.404+0000 I CONTROL  [initandlisten] db version v3.6.4
   2019-12-05T04:07:54.404+0000 I CONTROL  [initandlisten] git version: d0181a711f7e7f39e60b5aeb1dc7097bf6ae5856
   2019-12-05T04:07:54.404+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
   2019-12-05T04:07:54.404+0000 I CONTROL  [initandlisten] allocator: tcmalloc
   2019-12-05T04:07:54.404+0000 I CONTROL  [initandlisten] modules: none
   2019-12-05T04:07:54.404+0000 I CONTROL  [initandlisten] build environment:
   2019-12-05T04:07:54.404+0000 I CONTROL  [initandlisten]     distmod: rhel70
   2019-12-05T04:07:54.404+0000 I CONTROL  [initandlisten]     distarch: x86_64
   2019-12-05T04:07:54.404+0000 I CONTROL  [initandlisten]     target_arch: x86_64
   2019-12-05T04:07:54.404+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "0.0.0.0", port: 27017, ssl: { CAFile: "/etc/pki/nw/trust/truststore.pem", PEMKeyFile: "/etc/pki/nw/mongo/mongod-combined.pem", allowConnectionsWithoutCertificates: true, allowInvalidCertificates: false, allowInvalidHostnames: true, disabledProtocols: "TLS1_0,TLS1_1", mode: "preferSSL", sslCipherConfig: "HIGH:!EXPORT:!aNULL@STRENGTH:!kRSA" } }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, security: { authorization: "enabled" }, setParameter: { internalQueryExecMaxBlockingSortBytes: "134217728", opensslDiffieHellmanParameters: "/etc/pki/nw/mongo/dhparam2048.pem" }, storage: { dbPath: "/var/netwitness/mongo", journal: { enabled: true }, wiredTiger: { engineConfig: { cacheSizeGB: 16.0 } } }, systemLog: { destination: "file", logAppend: true, logRotate: "reopen", path: "/var/log/mongodb/mongod.log" } }
   2019-12-05T04:07:54.405+0000 I -        [initandlisten] Detected data files in /var/netwitness/mongo created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
   2019-12-05T04:07:54.405+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=16384M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),cache_cursors=false,log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
   2019-12-05T04:07:55.052+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1575518875:52721][17354:0x7f3478cfab00], file:WiredTiger.wt, WT_CURSOR.insert: read checksum error for 20480B block at offset 942080: block header checksum of 0 doesn't match expected checksum of 662513522
   2019-12-05T04:07:55.052+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1575518875:52886][17354:0x7f3478cfab00], file:WiredTiger.wt, WT_CURSOR.insert: {942080, 20480, 662513522}: (chunk 1 of 20): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
   <snipped>
   2019-12-05T04:07:55.056+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1575518875:56262][17354:0x7f3478cfab00], file:WiredTiger.wt, WT_CURSOR.insert: {942080, 20480, 662513522}: (chunk 20 of 20): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
   2019-12-05T04:07:55.056+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1575518875:56485][17354:0x7f3478cfab00], file:WiredTiger.wt, WT_CURSOR.insert: WiredTiger.wt: encountered an illegal file format or internal value: (__wt_block_read_off, 302)
   2019-12-05T04:07:55.056+0000 E STORAGE  [initandlisten] WiredTiger error (-31804) [1575518875:56504][17354:0x7f3478cfab00], file:WiredTiger.wt, WT_CURSOR.insert: the process must exit and restart: WT_PANIC: WiredTiger library panic
   2019-12-05T04:07:55.056+0000 F -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
   2019-12-05T04:07:55.056+0000 F -        [initandlisten] 
   
   ***aborting after fassert() failure
   
   
   2019-12-05T04:07:55.149+0000 F -        [initandlisten] Got signal: 6 (Aborted).
   
    0x55b88862a861 0x55b888629a79 0x55b888629f5d 0x7f34777766d0 0x7f34773d0277 0x7f34773d1968 0x55b886d8b425 0x55b886e582ce 0x55b886ec4ba1 0x55b886d2827e 0x55b886d285a7 0x55b886f73997 0x55b886f73ad5 0x55b886eec7a3 0x55b886ef2e71 0x55b886f160a6 0x55b886f861b7 0x55b886f2f006 0x55b886edbf9a 0x55b886f53ae4 0x55b886edcf77 0x55b886e6c597 0x55b886e6923c 0x55b886e3b95e 0x55b886e1ff04 0x55b887014b67 0x55b886d240e7 0x55b886dff18c 0x55b886d8d0e9 0x7f34773bc445 0x55b886deea9f
   ----- BEGIN BACKTRACE -----
   {"backtrace":[{"b":"55B8863FC000","o":"222E861","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55B8863FC000","o":"222DA79"},{"b":"55B8863FC000","o":"222DF5D"},{"b":"7F3477767000","o":"F6D0"},{"b":"7F347739A000","o":"36277","s":"gsignal"},{"b":"7F347739A000","o":"37968","s":"abort"},{"b":"55B8863FC000","o":"98F425","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55B8863FC000","o":"A5C2CE"},{"b":"55B8863FC000","o":"AC8BA1"},{"b":"55B8863FC000","o":"92C27E","s":"__wt_err"},{"b":"55B8863FC000","o":"92C5A7","s":"__wt_panic"},{"b":"55B8863FC000","o":"B77997","s":"__wt_block_read_off"},{"b":"55B8863FC000","o":"B77AD5","s":"__wt_bm_read"},{"b":"55B8863FC000","o":"AF07A3","s":"__wt_bt_read"},{"b":"55B8863FC000","o":"AF6E71","s":"__wt_page_in_func"},{"b":"55B8863FC000","o":"B1A0A6","s":"__wt_row_search"},{"b":"55B8863FC000","o":"B8A1B7","s":"__wt_btcur_insert"},{"b":"55B8863FC000","o":"B33006"},{"b":"55B8863FC000","o":"ADFF9A"},{"b":"55B8863FC000","o":"B57AE4","s":"__wt_log_scan"},{"b":"55B8863FC000","o":"AE0F77","s":"__wt_txn_recover"},{"b":"55B8863FC000","o":"A70597","s":"__wt_connection_workers"},{"b":"55B8863FC000","o":"A6D23C","s":"wiredtiger_open"},{"b":"55B8863FC000","o":"A3F95E","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"55B8863FC000","o":"A23F04"},{"b":"55B8863FC000","o":"C18B67","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"55B8863FC000","o":"9280E7"},{"b":"55B8863FC000","o":"A0318C","s":"_ZN5mongo11mongoDbMainEiPPcS1_"},{"b":"55B8863FC000","o":"9910E9","s":"main"},{"b":"7F347739A000","o":"22445","s":"__libc_start_main"},{"b":"55B8863FC000","o":"9F2A9F"}],"processInfo":{ "mongodbVersion" : "3.6.4", "gitVersion" : "d0181a711f7e7f39e60b5aeb1dc7097bf6ae5856", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-862.11.6.el7.x86_64", "version" : "#1 SMP Tue Aug 14 21:49:04 UTC 2018", "machine" : "x86_64" }, "somap" : [ { "b" : "55B8863FC000", "elfType" : 3, "buildId" : "37115F864F27162060BAC158CC814CBD48741B71" }, { "b" : "7FFC4ED60000", "elfType" : 3, "buildId" : "2562E90AE9BCFB0D02589A7C3B20B07AC58D6A74" }, { "b" : "7F34788D5000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "2BDC2B6FF0B2C204CCE34D139A9EADA0272EB070" }, { "b" : "7F3478662000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "9112A44B60B06C5FE236C7B1EF38A8D74C16AC0C" }, { "b" : "7F34782A7000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "C4F0A80793B40F2E40596C49987D21EE7B9C4742" }, { "b" : "7F347809F000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "D33989EC31EFE745EB0D3B68A92D19E77D7DDFDA" }, { "b" : "7F3477E9B000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "5CDB5A56336E7E2BD14FFA189411E44A834AFCD8" }, { "b" : "7F3477B99000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "F4CAE74047F9AA2D5A71FDEC67C4285D75753EBA" }, { "b" : "7F3477983000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "531AA1391EA4E1489D5EF11AA5DC2FFD9E2BDFEE" }, { "b" : "7F3477767000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "F4C04BCE85D2D269D0A2AF4972FC69805B50345B" }, { "b" : "7F347739A000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "CB4B7554D1ADBEF2F001142DD6F0A5139FC9AA69" }, { "b" : "7F3478AEE000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D266B1F6650927E18108323BCCA8F7B68E68EB92" }, { "b" : "7F347714D000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "16FE0DC6CEFC5F444BC876516D02EFE9CC2D432F" }, { "b" : "7F3476E65000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "D1CD1B94855A85FBC735C745DB39BC096F7D8CC3" }, { "b" : "7F3476C61000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "A3832734347DCA522438308C9F08F45524C65C9B" }, { "b" : "7F3476A2E000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "A20F715C514B3EA873F4CC77D585A50CB670E266" }, { "b" : "7F3476818000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "EA8E45DC8E395CC5E26890470112D97A1F1E0B65" }, { "b" : "7F34765C7000", "path" : "/lib64/libbwrap.so.1", "elfType" : 3, "buildId" : "C8B57E5EDB72A0EB6BF8E4ABFD978CCC0CF7A12A" }, { "b" : "7F34763B9000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "9F82B06CE44724A8ACE827A2C95A9A76619EA314" }, { "b" : "7F34761B5000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7F3475F8E000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "A88379F56A51950A33198890D37F5F8AEE71F8B4" }, { "b" : "7F3475D2C000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "9CA3D11F018BEEB719CDB34BE800BF1641350D0A" }, { "b" : "7F3475ACE000", "path" : "/usr/lib64/bwrap-1.2.2.2/libcryptocme.so", "elfType" : 3, "buildId" : "2CDC228D1BF20A18FB4BC1BCD9ECBE8F6212AC9F" }, { "b" : "7F34758CA000", "path" : "/usr/lib64/bwrap-1.2.2.2/libccme_error_info.so", "elfType" : 3, "buildId" : "848E659A077F1D14A8AD96533D18A2BF2E012877" }, { "b" : "7F34756C6000", "path" : "/usr/lib64/bwrap-1.2.2.2/libccme_aux_entropy.so", "elfType" : 3, "buildId" : "FF51846DA67E57C6BC3BACF69B084C9D45DF5CD4" }, { "b" : "7F347542E000", "path" : "/usr/lib64/bwrap-1.2.2.2/libccme_base.so", "elfType" : 3, "buildId" : "A31E5C5B44F05A120D258AA19A59658E4EBCDE7D" }, { "b" : "7F34751D0000", "path" : "/usr/lib64/bwrap-1.2.2.2/libccme_asym.so", "elfType" : 3, "buildId" : "F44DA44D4B1515200C589C0A77DFBD2E76668BC0" }, { "b" : "7F3474F0B000", "path" : "/usr/lib64/bwrap-1.2.2.2/libccme_ecc_accel_fips.so", "elfType" : 3, "buildId" : "3980C59BA73A4E76F4CD807578C600E91D2D4F18" }, { "b" : "7F3474C6C000", "path" : "/usr/lib64/bwrap-1.2.2.2/libccme_ecc.so", "elfType" : 3, "buildId" : "7365F5D8E99D2A82BF26EE72B63C80137FC03D9D" }, { "b" : "7F34749DE000", "path" : "/usr/lib64/bwrap-1.2.2.2/libccme_base_non_fips.so", "elfType" : 3, "buildId" : "65ADC3FB0F2E7E7733198E983BA78215440DACA1" }, { "b" : "7F3474721000", "path" : "/usr/lib64/bwrap-1.2.2.2/libccme_ecc_accel_non_fips.so", "elfType" : 3, "buildId" : "50EEBC9B0FF5EE82BA77E82895F1AD474A21D490" }, { "b" : "7F34744CA000", "path" : "/usr/lib64/bwrap-1.2.2.2/libccme_ecc_non_fips.so", "elfType" : 3, "buildId" : "439D6B449A34920838E74828E88D420CBC1BAC3E" } ] }}
    mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55b88862a861]
    mongod(+0x222DA79) [0x55b888629a79]
    mongod(+0x222DF5D) [0x55b888629f5d]
    libpthread.so.0(+0xF6D0) [0x7f34777766d0]
    libc.so.6(gsignal+0x37) [0x7f34773d0277]
    libc.so.6(abort+0x148) [0x7f34773d1968]
    mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55b886d8b425]
    mongod(+0xA5C2CE) [0x55b886e582ce]
    mongod(+0xAC8BA1) [0x55b886ec4ba1]
    mongod(__wt_err+0x9D) [0x55b886d2827e]
    mongod(__wt_panic+0x33) [0x55b886d285a7]
    mongod(__wt_block_read_off+0x547) [0x55b886f73997]
    mongod(__wt_bm_read+0x135) [0x55b886f73ad5]
    mongod(__wt_bt_read+0x203) [0x55b886eec7a3]
    mongod(__wt_page_in_func+0x19F1) [0x55b886ef2e71]
    mongod(__wt_row_search+0x856) [0x55b886f160a6]
    mongod(__wt_btcur_insert+0x1027) [0x55b886f861b7]
    mongod(+0xB33006) [0x55b886f2f006]
    mongod(+0xADFF9A) [0x55b886edbf9a]
    mongod(__wt_log_scan+0xCE4) [0x55b886f53ae4]
    mongod(__wt_txn_recover+0x417) [0x55b886edcf77]
    mongod(__wt_connection_workers+0x37) [0x55b886e6c597]
    mongod(wiredtiger_open+0x196C) [0x55b886e6923c]
    mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x89E) [0x55b886e3b95e]
    mongod(+0xA23F04) [0x55b886e1ff04]
    mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x637) [0x55b887014b67]
    mongod(+0x9280E7) [0x55b886d240e7]
    mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0x86C) [0x55b886dff18c]
    mongod(main+0x9) [0x55b886d8d0e9]
    libc.so.6(__libc_start_main+0xF5) [0x7f34773bc445]
    mongod(+0x9F2A9F) [0x55b886deea9f]
   -----  END BACKTRACE  -----

   
   
    

WARNING:
Before using repairDatabase, make a backup copy of the dbpath directory, if possible. 
If there is not enough space on the device, move it to another device that has enough space.

1. Stop the mongod service, even if it's not successfully running.
   
   

   
   systemctl stop mongod

   
   
2. 
   

   Confirm sufficient space for your backup of mongo:

   
   
   

   
   du -h /var/netwitness/mongo
   du -h /root

   
   
3. Backup your mongo dbpath directory to a location like, /root/mongo.tgz:
   

   
   tar czvf /root/mongo.tgz /var/netwitness/mongo

   
   
4.  Run the mongod --repair command:
   

   
   mongod --dbpath /var/netwitness/mongo --repair

   
   
5. Once the repair has finished running, change the dbpath owner back to mongod:
   

   
   chown -R mongod:mongod /var/netwitness/mongo

   
   
6. Start the mongod service:
   

   
   systemctl start mongod

   
   

2020-11-02T17:35:29.231+0000 I STORAGE [initandlisten] exception in initAndListen std::exception: boost::filesystem::remove: Permission denied: "/var/netwitness/mongo/_tmp/", terminating

2019-12-05T03:13:51.242+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1575515631:242064][2733:0x7fbf82945b00], file:WiredTiger.wt, WT_CURSOR.insert: WiredTiger.wt: encountered an illegal file format or internal value: (__wt_block_read_off, 302)
2019-12-05T03:13:51.242+0000 E STORAGE  [initandlisten] WiredTiger error (-31804) [1575515631:242077][2733:0x7fbf82945b00], file:WiredTiger.wt, WT_CURSOR.insert: the process must exit and restart: WT_PANIC: WiredTiger library panic
2019-12-05T03:13:51.242+0000 F -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2019-12-05T03:13:51.242+0000 F -        [initandlisten]


***aborting after fassert() failure

1. Stop the mongod service:
   
   

   
   systemctl stop mongod

   
   
2. Download and install MongoDB 4.0.3 or newer on another machine, e.g: Windows, CentOS. The current release is 4.2.1 at the time of this writing. You cannot use an existing NetWitness device for this operation.
3. Copy the entire /var/netwitness/mongo folder from the NetWitness appliance to this new machine.
4. On the new machine, run the mongod --repair command, pointing to the corrupt database folder. For example, if you have copied it to C:\mongo on a Windows machine:
   

   
   C:\Program Files\MongoDB\Server\4.2\bin> mongod --dbpath C:\mongo --repair

   
   
5. After the repair process completes, on the NetWitness appliance, rename the existing /var/netwitness/mongo folder to make way for the repaired MongoDB folder from the previous step:
   

   
   mv /var/netwitness/mongo /var/netwitness/mongo-corrupt

   
   Perhaps a better idea would be to move the folder out of the appliance and then delete it if disk space is a concern.
    
6. Copy the repaired MongoDB folder back to /var/netwitness/mongo.
7. Change /var/netwitness/mongo owner to mongod:
   

   
   chown -R mongod:mongod /var/netwitness/mongo

   
   
8. Restore SELinux Context to the folder, if for some reason you lot it:
   

   
   restorecon -vF /var/netwitness/mongo

   
   
9. Start the mongod service:
   

   
   systemctl start mongod