Data Sourcing to BigData
- Community Thread
- Data Publication
- Forum Thread
- REST API
- RSA Archer
- RSA Archer Suite
- search api
Not really sure if Archer is a bottleneck here, but depends on how you processes the records before submitting to BigData, and also how BigData executes the source. You can try to use Transport DF to save the file in XML and feed directly to your BigData to see if it processes the file long enough.
But the best options are API integrations and Data Gateway, I would say:
BigData agent uses search API to get the data , Only problem is it takes 6 Hours for 30K records, which is too much
Main purpose of sourcing data in BigData is to keep history and to do trend analysis.
As per my understanding Data Gateway is used for real time update and reporting in Archer,
How about Data Publication? Is it better than API
You can try, but DP can run only 1 per day. And it does not bring data in format you need, you would then need to traverse and collect the pieces. I do not know why in your case 30k records takes 6 hours, that is something to be debugged, I would say.
May be because ExecuteSearch API run in sequence, Page by Page, Do you know how many parallel calls we can make to get multiple pages simultaneously. There were around 1250 Pages i guess
There is no explicit limitation I am aware of, it depends on your platform design/architecture and hardware.
So, you can try parallel searches for sure.
However, for each call you need to make sure to not override the session of the same user.
Depending on the total amount of data in the modules involved it may be quicker to export to a database through the Data Publication Service and query that directly.
Regarding Data Publication, check out and vote up Data Publication frequency. Also, when the Data Publication Silverlight pages were replaced in 6.8, several internal API calls are exposed that may be helpful.
To run a DPS now with ID of 1:
To get DPS list like the Manage Data Publications page:
Sample API result:
"Name": "DPS Test",
"Description": "This is my description",
"SolutionName": "Archer Toolbox",
"UserName": "Letterman, Jeff",