000029826 - User Search using API, searching beyond the 20000 results limit

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000029826
Applies ToRSA Product Set: Silver Tail
RSA Product/Service Type: Forensics - Silver Tail
RSA Version/Condition: 4.6
Platform: Other
Platform (Other): Red Hat Enterprise Linux Server release 6.5 (Santiago)
O/S Version: Other
Product Name: null
Product Description: null
IssueWTD UI has limitation of maximum 20,000 records and the only option to change the number returned is to check the box to limit to 100 results.
Using direct API calls it's possible to search and obtain results without limitations and with added flexibility
ResolutionFirst authenticate and store the cookie: 

# curl -k -c mycookies.txt -b mycookies.txt "https://10.148.150.31/login?user=admin&passwd=silvertail&app=ui";



Next Create the search ( this will return a jobid):

# curl -k -c mycookies.txt -b mycookies.txt "https://10.148.150.31/search?find_parts=yes&find=user,page,User%5c%20IP%5c%20Changed,ip.country,ip,host&indexes=page='/accounttransfer.php'+and+host='www.prymida.com'&annotate=yes&hour=2015.3.16.1&len=672&limit=5000000&queryid=admin18446744072688845000&searchname=test&version=2&async=true"



You can view the job status in the UI, or with API, in the API you will need the jobid as return from previous command:

# curl -k -c mycookies.txt -b mycookies.txt "https://10.148.150.31/search?cmd=status&search-job-id=20339ae09bca2374a292bdf7a3bb6925&noIdleReset=true



Once the job is complete, obtain the results and store in "output.txt" with:

# curl -k -c mycookies.txt -b mycookies.txt "https://10.148.150.31/search?search-job-id=20339ae09bca2374a292bdf7a3bb6925&cmd=page&firstrecord=0&lastrecord=199999" -o output.txt



As you will notice you can choose which records you want, eg the above retrieves the first 2 million records.



Delete the finished job via the UI or using the API with:

# curl -k -c mycookies.txt -b mycookies.txt "https://10.148.150.31/search?cmd=cancel&search-job-id=20339ae09bca2374a292bdf7a3bb6925&noIdleReset=true"



Anything available in the search UI can be included and as shown in this example I set the result limit to 5,000,000 records 



Note that the limitations here are that such large searches will have a large impact of memory and CPU while the run, You may want to run such jobs when there is low activity and just after the hourly batch.

Attachments

    Outcomes