Keep your clients’ websites open to discovery.
A changed robots.txt or an accidental noindex can put a page out of reach. Check the settings, see the evidence, and get the fix.
No sign-up for the first report. We read this page and its robots.txt.
Search and training are different choices.
- Search
- OAI-SearchBot allowed
- Training
- GPTBot restricted
- Meaning
- Your policy can allow search discovery while restricting training.
We report your published configuration. Only the crawler’s own logs can prove its visit.
Your discovery report
Crawler rules for this page
| Crawler | Purpose | Published policy |
|---|
One place to watch the small changes that matter.
A client updates a plugin, a staging rule reaches production, or a crawler policy changes. Your next report shows what changed and what to review.
- Specific evidencePublished robots.txt restrictions, indexing directives, page status and readable titles.
- Separate search from trainingSee search crawler rules without treating intentional training restrictions as errors.
- Clear limitsA network block stays “could not verify.” This is a technical configuration check, with no ranking or citation guarantees.
Daily checks for your client list.
Monitor up to ten website URLs. Daily checks, 30 days of report history, and email alerts when findings change. No hosting credentials needed.
Try a page first.
The public report is free. Sign in by email when you want monitoring, then subscribe through Polar. Cancel any time; monitoring continues until your paid period ends.
Taxes, where applicable, are shown at checkout. Ten monitored URLs means ten selected pages, not a crawl of every page on ten websites.