Unblocking the Silktide Prospect spider
- Applies to:
In rare circumstances, Silktide Prospect may be blocked from downloading the pages of a website by an anti-bot filter.
This is usually because Silktide Prospect’s requests originate in the AWS (Amazon Web Services) cloud, which some filtering systems block. This article gives the necessary technical information to unblock Silktide Prospect’s requests in your filtering system. Note – it is only possible to unblock websites for which you control the filtering.
Unblocking Silktide Prospect
The best way to unblock Silktide Prospect requests is to look for the following HTTP request header which is part of every request Silktide Prospect makes:
X-TESTING-APPLICATION: Silktide Prospect 1.0
What doesn’t work
Silktide Prospect’s server IPs are not static and change daily, so the best way to unblock Silktide Prospect is via request data.
Unfortunately, user agent is also not ideal as in order to circumvent some filtering tools, Silktide Prospect uses a user agent copied from a real browser and is subject to change. The current user agent for Silktide Prospect is:
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36
If you need further assistance, please contact your Silktide Prospect account manager.