February 2025posted on 02.26.2025GPTBot is OpenAI’s web crawler OpenAI's web crawler has a user-agent: User agent token: GPTBot Full user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot) Which means it can be restricted to crawl your site using robots.txt: User-agent: GPTBot Disallow: / Found in OpenAI's documentation.No reactions yet
GPTBot is OpenAI’s web crawler
OpenAI's web crawler has a user-agent:
Which means it can be restricted to crawl your site using robots.txt:
Found in OpenAI's documentation.