Crawling only the sitemap on google webmaster tools

by Shan Xue   Last Updated January 26, 2018 17:04 PM - source

So recently, our website has been hacked and we're trying to clean everything right now. But, when doing the "site:" search it still shows the cached japanese websites.

So we tried playing with robots.txt i.e.:

User-agent: *

Disallow:

Sitemap: http://www.website.com/sitemap.xml

But when I enter the bad URL in robots.txt tester, it still allow the URL that we don't want.

Is there any way that google only crawls the sitemap on robots.txt without manually entering all the bad links on the "Disallow"?



Answers 1


Google has never limited itself to crawling and indexing just URLs that are in the sitemap. Such functionality does not exist, and I doubt that it ever will.

Stephen Ostermiller
Stephen Ostermiller
January 26, 2018 17:02 PM

Related Questions




Effect of Submitted URLs being blocked by robots

Updated September 17, 2019 11:04 AM