What the check covers
Loads /robots.txt and shows its status, content and Sitemap directives.
SEO
robots.txt availability and directive check
Load robots.txt, inspect its status and find Sitemap references or accidental crawler blocks.
How to read the result
Look for accidental blocks of important sections and invalid Sitemap addresses.
Limitations
robots.txt is advisory and does not protect confidential data.
Example
A target you can check
https://example.comUse your own public address to get a result relevant to your configuration.
Privacy
What is processed
The check processes the submitted public target on the NexIP server. Do not submit secrets, tokens or private addresses.
Privacy policyWhen to use it
Before launch and during incidents
Run the check after a configuration change, when diagnosing an outage, or before publishing a new service.
Questions about this check
Можно ли запускать инструмент через API?
Да, инструмент доступен через POST /api/v1/tools/robots-check при включённом API.
Сохраняется ли история проверки?
Да, если история инструментов включена в настройках NexIP CMS.
Можно ли поделиться результатом?
Да, после проверки можно создать публичную ссылку на сохранённый результат.
Need recurring checks and incident history?
Create a monitor and receive notifications when the state changes.