Module 6 · Lesson 1
AI crawlers: robots.txt, who's who and what to allow
AI crawlers: robots.txt, who's who and what to allow
What each crawler from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple and Meta does, and how to write a robots.txt that keeps you in AI answers without giving away more than you want.
This lesson is for subscribers
Lessons 1.1 and 1.2 are free. The subscription unlocks all 40 and your GEO lab.