.Google.com's John Mueller addressed an inquiry concerning why Google.com indexes web pages that are prohibited from crawling by robots.txt as well as why the it is actually safe to overlook the related Search Console files regarding those creeps.Robot Website Traffic To Concern Guideline URLs.The individual talking to the question recorded that crawlers were creating web links to non-existent inquiry specification URLs (? q= xyz) to webpages along with noindex meta tags that are likewise blocked in robots.txt. What cued the concern is that Google is actually crawling the hyperlinks to those web pages, receiving obstructed through robots.txt (without noticing a noindex robotics meta tag) at that point receiving shown up in Google.com Look Console as "Indexed, though shut out through robots.txt.".The person inquired the following concern:." But below is actually the major concern: why would certainly Google mark webpages when they can't also observe the content? What's the advantage because?".Google.com's John Mueller verified that if they can't crawl the page they can't find the noindex meta tag. He also creates an intriguing acknowledgment of the internet site: search driver, suggesting to ignore the results due to the fact that the "average" customers won't observe those end results.He created:." Yes, you are actually proper: if our experts can't creep the webpage, our team can't see the noindex. That said, if our experts can not creep the webpages, at that point there is actually not a great deal for us to mark. Therefore while you may view a number of those pages with a targeted website:- inquiry, the common user will not observe them, so I wouldn't fuss over it. Noindex is also great (without robots.txt disallow), it just implies the Links will definitely end up being actually crept (and also find yourself in the Search Console record for crawled/not indexed-- neither of these standings induce issues to the remainder of the site). The important part is that you do not produce them crawlable + indexable.".Takeaways:.1. Mueller's answer validates the constraints in operation the Site: hunt advanced search operator for analysis explanations. Some of those causes is actually due to the fact that it is actually certainly not connected to the regular hunt index, it's a distinct factor completely.Google's John Mueller commented on the web site hunt driver in 2021:." The short answer is that a web site: concern is actually certainly not meant to be total, neither used for diagnostics objectives.A website inquiry is actually a particular type of search that limits the end results to a particular website. It is actually generally only words web site, a digestive tract, and afterwards the web site's domain name.This question limits the results to a details website. It is actually not implied to become a detailed assortment of all the pages coming from that web site.".2. Noindex tag without using a robots.txt is alright for these sort of circumstances where a robot is actually linking to non-existent web pages that are acquiring found through Googlebot.3. Links with the noindex tag will certainly generate a "crawled/not recorded" item in Browse Console which those won't have an unfavorable impact on the rest of the web site.Review the question and answer on LinkedIn:.Why would certainly Google mark web pages when they can't also observe the content?Included Image by Shutterstock/Krakenimages. com.