MBSuite-Crawler and MB-Suite-SiteAuditor. If the site's firewall or Cloudflare stops them, allow both: with only one, part of the audit stays blocked.If you reached this page from your server logs, it's the MB Suite crawler: someone audits the site's technical SEO with MB Suite. It only reads public pages and files, the same ones a search engine reads.
What are its two User-Agents?
| Name | What it does | Full User-Agent |
|---|---|---|
MBSuite-Crawler | Reads the site's pages when the site stops the first attempt. | Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36 MBSuite-Crawler/1.0 |
MB-Suite-SiteAuditor | Checks robots.txt, sitemap.xml, llms.txt, the domain redirects (www and HTTPS), the 404 page, the directory listing, and the HTTP/2 version. | Mozilla/5.0 (compatible; MB-Suite-SiteAuditor/1.0; +https://help.mb-suite.com/seo/site-auditor-crawler) |
To read the pages, the crawler first comes in as a regular Chrome browser. If the site stops it (Cloudflare or another firewall), it tries again identifying itself as MBSuite-Crawler: that's the name your firewall has to let through. The domain checks always go out as MB-Suite-SiteAuditor. The crawler runs on Google Cloud servers, so the rule is set by User-Agent.
How do I know the crawler is being blocked?
- The audit fails, or says it couldn't analyze some pages: the firewall stopped
MBSuite-Crawler. - The report shows the Domain checks blocked notice, with the list of what it couldn't verify (robots.txt, sitemap.xml, the redirects…): the firewall stopped
MB-Suite-SiteAuditor. Those checks don't lower the health score, and the notice's How to allow the crawler link brings you here. - If MB Suite detects Cloudflare on the site, the site's Configuration tab shows Your site uses Cloudflare protection, with these same steps and a button to copy each name.
How do I allow it in Cloudflare?
- Log in to your domain's Cloudflare dashboard.
- Go to Security → Security Rules and create a new rule.
- Field User Agent, operator contains, value
MBSuite-Crawler. Add a second condition with Or: field User Agent, operator contains, valueMB-Suite-SiteAuditor. - As the action, pick Skip (All remaining custom rules).
- Save the rule and audit again with Audit now, or wait for the next scheduled audit.
And in another firewall?
In any other firewall or WAF (your hosting's, a security plugin, a server rule…), the idea is the same: a rule that lets through the requests whose User-Agent contains MBSuite-Crawler or MB-Suite-SiteAuditor, before the rules that block bots. If the panel has no rules by User-Agent, ask your hosting provider for it with the two names.
MBSuite-Crawler lets the pages through but keeps the domain checks blocked; one with only MB-Suite-SiteAuditor does the opposite.Still have questions?
Ask MIA, the MB Suite AI assistant: open it with the MB AI button (⌘J) in the top bar. MIA knows the section you're in, so you can ask it about what you see on screen — and it can also answer how-to questions about using MB Suite.