What the check covers
Loads /robots.txt and shows its status, content and Sitemap directives.
SEO
Check robots.txt online
Check robots.txt availability, crawler directives, Sitemap references and indexing restrictions.
How to read the result
Look for accidental blocks of important sections and invalid Sitemap addresses.
Limitations
robots.txt is advisory and does not protect confidential data.
Example
A target you can check
https://example.comUse your own public address to get a result relevant to your configuration.
Privacy
What is processed
The check processes the submitted public target on the NexIP server. Do not submit secrets, tokens or private addresses.
Privacy policyWhen to use it
Before launch and during incidents
Run the check after a configuration change, when diagnosing an outage, or before publishing a new service.
NexIP diagnostic hub
Continue diagnostics
Open the complete topic hub with related checks, guides and reference materials.
Open diagnostic hubQuestions about this check
Можно ли запускать инструмент через API?
Да, инструмент доступен через POST /api/v1/tools/robots-check при включённом API.
Сохраняется ли история проверки?
Да, если история инструментов включена в настройках NexIP CMS.
Можно ли поделиться результатом?
Да, после проверки можно создать публичную ссылку на сохранённый результат.
Related materials
What is robots.txtRelated glossary terms
robots.txtNeed recurring checks and incident history?
Create a monitor and receive notifications when the state changes.