Robots.txt is a text file that tells search engine crawlers (bots) which parts of a website they can crawl (search) and which parts they should not crawl. This file is registered with Webmaster Tools and is collected periodically.
Normally, Robots.txt is rarely modified after it is set up, but if problems occur in Google Console, Bing Webmaster Tools, or Naver Search Advisor, you may need to control it through Robots.txt.
If you make a modification, it will not be applied immediately because it will be reflected at the time of crawling. Therefore, I would like to find out how to make it reflected immediately after modification.
Table of Contents
Google Search Console
Google Search Console is a useful tool for registering sitemaps and checking for issues on your site. When you first access it, it can be difficult to find where the Robots.txt settings are. If you look at the explanation below, you will easily find where to set them.
After you enter Google Search Console, you will see a menu on the left side. At the very bottom, you will see Settings. You can go there.
If you go into settings, there is a section called Crawling in the middle. Here, you can click Open report in robots.txt.
When you open the Robots.txt report, you can check the file, the verification date, and the status. If the verification date is not recent, you can request a recrawl by clicking the three dots icon on the right.
Since your Webmaster Tools have been updated to the latest settings, your changes will be reflected in Google’s search engine.
Bing Webmaster Tools
Once you’ve logged into Bing Webmaster Tools , you’ll find ‘Tools & Advanced Features’ on the left menu. From there, select ‘Robots.txt Tester’.
If you go into the Robots.txt tester, you can see the crawled settings. Here, if you click the ‘Get Latest’ button on the right, the current settings will be reflected.
Naver Search Advisor
You can find the Robots.txt settings by selecting ‘Verification’ from the left menu of Naver Search Advisor .
If you go into the Robots.txt settings, you can check the current settings and the date and time when crawling was performed. If the latest settings are not reflected, you can click the collection request button.
Search engines periodically re-collect your Robots.txt file, so any changes you make will take effect over time. However, if you encounter any errors or warnings, please promptly request a re-collect in Webmaster Tools using the method above.
▶ How to fix /feed/ duplicate pages in Google Search Console
▶ Comment-page-address: how to fix ‘alternative page’ error
▶ 404 Error: Key Things to Check When You Get a Sitemap Not Found
