.Google's John Mueller answered an inquiry regarding why Google indexes pages that are actually disallowed coming from crawling by robots.txt and also why the it's risk-free to neglect the relevant Explore Console reports about those crawls.Bot Web Traffic To Concern Parameter URLs.The person inquiring the inquiry chronicled that crawlers were actually producing web links to non-existent query parameter Links (? q= xyz) to pages along with noindex meta tags that are also blocked in robots.txt. What prompted the concern is actually that Google is crawling the web links to those web pages, receiving shut out through robots.txt (without noticing a noindex robotics meta tag) at that point receiving turned up in Google Search Console as "Indexed, though blocked out by robots.txt.".The person inquired the observing inquiry:." Yet below's the big concern: why would Google.com index webpages when they can not also view the material? What's the conveniences because?".Google.com's John Mueller affirmed that if they can not creep the web page they can't observe the noindex meta tag. He additionally helps make an appealing mention of the website: search operator, suggesting to disregard the outcomes considering that the "average" individuals will not view those end results.He composed:." Yes, you are actually appropriate: if our company can't creep the page, our company can not see the noindex. That said, if our experts can't crawl the web pages, at that point there is actually certainly not a lot for our company to mark. Thus while you might find some of those webpages with a targeted web site:- question, the common user will not see all of them, so I wouldn't bother it. Noindex is actually also alright (without robots.txt disallow), it only indicates the URLs will end up being actually crept (and wind up in the Look Console file for crawled/not recorded-- neither of these conditions cause issues to the remainder of the website). The fundamental part is that you don't make them crawlable + indexable.".Takeaways:.1. Mueller's answer confirms the constraints being used the Site: search advanced search operator for analysis factors. Among those reasons is actually given that it's certainly not hooked up to the routine search mark, it is actually a distinct point altogether.Google's John Mueller discussed the internet site search operator in 2021:." The brief response is actually that a website: query is not meant to be complete, neither used for diagnostics reasons.An internet site concern is actually a details sort of hunt that confines the outcomes to a certain internet site. It's essentially only the word site, a colon, and after that the website's domain.This inquiry confines the end results to a particular web site. It's certainly not indicated to become an extensive collection of all the webpages from that website.".2. Noindex tag without making use of a robots.txt is actually fine for these kinds of situations where a robot is actually linking to non-existent web pages that are actually obtaining found out through Googlebot.3. URLs along with the noindex tag are going to create a "crawled/not catalogued" item in Look Console which those will not have a negative result on the rest of the website.Go through the concern and also address on LinkedIn:.Why will Google.com index webpages when they can't even observe the content?Featured Image through Shutterstock/Krakenimages. com.