Monitoring speed and frequency

Conductor Website Monitoring differs from other SEO auditing tools in that it monitors websites 24/7. Search engines check your website 24/7, so it’s essential that your auditing solution does the same.

Our monitoring is driven by smart algorithms that take a vast array of signals into account to determine when to crawl a specific page.

The speed with which Conductor Website Monitoring monitors and the overhead (or impact) on your website depends on your website’s size, your webserver’s capacity and the monitoring speed setting.

The relation between web server capacity and monitoring speed

In order to avoid performance issues on your website and to align with your business needs, before using Conductor Website Monitoring it is essential that you verify what crawl speed your infrastructure can support and adjust your settings accordingly.

Although it is important to assess webserver capacity before using the application, Conductor Website Monitoring helps predict server overload by continuously assessing your website’s performance. It does this through its throttling algorithm, which looks at the time it takes to get a response from the webserver and at the response itself. If the algorithm detects an issue (for example, slower speeds, increased errors, etc.), it may slow down crawling automatically until the impact on the webserver has been remedied.

The relation between website size and monitoring speed

Once you have verified webserver capacity, for larger websites you may decide to crawl at a faster speed in order to keep up to date on your website's changes as well as possible. For example, if your website has 200,000 pages, you may wish to set Conductor Website Monitoring to request a page every second.

For smaller websites, you may decide to crawl at a slower speed. So, if your website is only a couple of thousand pages, the optimal crawl frequency may be very low. For example, if your website has 2,000 pages, you may only need Conductor Website Monitoring to request a page once every 10 seconds.

Setting a different monitoring speed

You can speed up or slow down Conductor Website Monitoring's crawling by setting a different monitoring speed in the website settings in Conductor Website Monitoring.

To crawl faster than the default monitoring speed you also need to verify domain ownership by connecting with Google Search Console or installing the Cloudflare Worker. You can read more about this here: Website verification

Setting different monitoring speeds at different times of the day

If your website sees or may see significantly higher loads due to traffic at peak times, you should configure Conductor Website Monitoring to slow down, or even pause, monitoring during these times by using the Peak / Off-peak monitoring speed feature: