000027155 - Using AXM and Cleartrust with Web Crawler or Web Spider products

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 3Show Document
  • View in full screen mode

Article Content

Article Number000027155
Applies ToAccess Manager 6.0
All OS
 All web agents
IssueInformation about Web Crawlers and RSA Access Manager and Cleartrust

"Web Crawler" is a generic name for any script or process that can be used for automating maintenance tasks on a website, such as checking to see if links are correct or for validating HTML code.  Microsoft refers to them as "Web Spiders", and they are also known as web robots, ants, automatic indexes,  bots, worms and scutters.

More general description information on:


RSA does not make any recommendations on which 3rd party product to select, but the following notations are applicable:

1) With Cleartrust 5.5.3 and Access Manager 6x products, you should be able to use any web crawler as long as it supports http cookies. 
2) When you set the product up, RSA recommends it to be incorporated on a separate virtual host on your agent.
3) The webcrawler should be assigned its own Cleartrust/AXM user name in the Entitlements Manager.
4) Windows 2003 Sharepoint is supported for the IIS agent and offers a web crawler component.
5) The Sharepoint web crawler is not able to process http cookies. If you choose to use it, you must use non forms based authentication for the virtual host the web crawler is configured for.

Legacy Article IDa40668