Skip to main content

Whitelisting the MirrorWeb crawler

Whitelisting the MirrorWeb crawler

H
Written by Harriet Christie

Summary

Whitelisting the MirrorWeb crawler

Prerequisites

  • Access to MirrorWeb Insight and relevant permissions for this workflow.

Step-by-Step Instructions

On occasion, the MirrorWeb crawlers may experience difficulty capturing certain websites or assets. This is usually due to security that is in place on the hosting web server.

All of our crawls are done using the following user agent string - if this is whitelisted by the web hosts then the crawl will be much more replete and comprehensive

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 +https://www.mirrorweb.com

or a variation thereof (customer depending). If your crawler uses a custom user agent, you will have been informed as part of your onboarding.

Did this answer your question?