Smart Knife

Crawler files

browser

Evaluate robots.txt rules against a crawler and analyze sitemap XML.

robots.txtSitemap