MB SuiteMB Suite

Allow the Site Audit crawler in your firewall

Updated 2026-11-08

In two lines: the Site Audit of MB Suite visits a site with two User-Agents, MBSuite-Crawler and MB-Suite-SiteAuditor. If the site's firewall or Cloudflare stops them, allow both: with only one, part of the audit stays blocked.

If you reached this page from your server logs, it's the MB Suite crawler: someone audits the site's technical SEO with MB Suite. It only reads public pages and files, the same ones a search engine reads.

What are its two User-Agents?

NameWhat it doesFull User-Agent
MBSuite-CrawlerReads the site's pages when the site stops the first attempt.Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36 MBSuite-Crawler/1.0
MB-Suite-SiteAuditorChecks robots.txt, sitemap.xml, llms.txt, the domain redirects (www and HTTPS), the 404 page, the directory listing, and the HTTP/2 version.Mozilla/5.0 (compatible; MB-Suite-SiteAuditor/1.0; +https://help.mb-suite.com/seo/site-auditor-crawler)

To read the pages, the crawler first comes in as a regular Chrome browser. If the site stops it (Cloudflare or another firewall), it tries again identifying itself as MBSuite-Crawler: that's the name your firewall has to let through. The domain checks always go out as MB-Suite-SiteAuditor. The crawler runs on Google Cloud servers, so the rule is set by User-Agent.

How do I know the crawler is being blocked?

  • The audit fails, or says it couldn't analyze some pages: the firewall stopped MBSuite-Crawler.
  • The report shows the Domain checks blocked notice, with the list of what it couldn't verify (robots.txt, sitemap.xml, the redirects…): the firewall stopped MB-Suite-SiteAuditor. Those checks don't lower the health score, and the notice's How to allow the crawler link brings you here.
  • If MB Suite detects Cloudflare on the site, the site's Configuration tab shows Your site uses Cloudflare protection, with these same steps and a button to copy each name.

How do I allow it in Cloudflare?

  • Log in to your domain's Cloudflare dashboard.
  • Go to Security → Security Rules and create a new rule.
  • Field User Agent, operator contains, value MBSuite-Crawler. Add a second condition with Or: field User Agent, operator contains, value MB-Suite-SiteAuditor.
  • As the action, pick Skip (All remaining custom rules).
  • Save the rule and audit again with Audit now, or wait for the next scheduled audit.
If you don't have access to the Cloudflare dashboard, send this page to whoever manages the domain, or to your hosting provider: everything they need is here.

And in another firewall?

In any other firewall or WAF (your hosting's, a security plugin, a server rule…), the idea is the same: a rule that lets through the requests whose User-Agent contains MBSuite-Crawler or MB-Suite-SiteAuditor, before the rules that block bots. If the panel has no rules by User-Agent, ask your hosting provider for it with the two names.

Allow both names. A rule with only MBSuite-Crawler lets the pages through but keeps the domain checks blocked; one with only MB-Suite-SiteAuditor does the opposite.

Still have questions?

Ask MIA, the MB Suite AI assistant: open it with the MB AI button (⌘J) in the top bar. MIA knows the section you're in, so you can ask it about what you see on screen — and it can also answer how-to questions about using MB Suite.