robots.txt and AGENTS.md both fail open
Ten of 33 robots.txt cells let a blocked URL through and Codex clipped the back of a 34 KiB AGENTS.md. All 219 runs exited 0. Which parser gave your verdict?
Tags
ai-crawler
2 posts
AI Crawlers Don't Run Your JavaScript
GPTBot and ClaudeBot don't render JS. Here's a curl experiment proving client-side pages are invisible to AI search, plus how to fix it server-side.