# Shiori robots policy # Keep public product/docs/comparison pages crawlable while protecting app, # account, and API surfaces from search indexes and AI answer surfaces. User-agent: Googlebot Allow: / Disallow: /api # Private UI routes intentionally remain crawlable so Google can observe their # X-Robots-Tag: noindex headers and remove stale app URLs from the index. User-agent: Bingbot Allow: / Disallow: /api User-agent: Applebot Allow: / Disallow: /api User-agent: Slurp Allow: / Disallow: /api User-agent: DuckDuckBot Allow: / Disallow: /api User-agent: * Allow: / Disallow: /api Disallow: /admin/ Disallow: /auth/ Disallow: /chat/ Disallow: /dashboard/ Disallow: /preview-inputs/ Disallow: /quiz/ Disallow: /settings/ Disallow: /studio/ # ============================================ # AI SEARCH AND USER-REQUEST CRAWLERS # These power current answer/search experiences and should see public pages. # ============================================ User-agent: OAI-SearchBot Allow: / Disallow: /api Disallow: /admin/ Disallow: /auth/ Disallow: /chat/ Disallow: /dashboard/ Disallow: /preview-inputs/ Disallow: /quiz/ Disallow: /settings/ Disallow: /studio/ User-agent: ChatGPT-User Allow: / Disallow: /api Disallow: /admin/ Disallow: /auth/ Disallow: /chat/ Disallow: /dashboard/ Disallow: /preview-inputs/ Disallow: /quiz/ Disallow: /settings/ Disallow: /studio/ User-agent: Claude-SearchBot Allow: / Disallow: /api Disallow: /admin/ Disallow: /auth/ Disallow: /chat/ Disallow: /dashboard/ Disallow: /preview-inputs/ Disallow: /quiz/ Disallow: /settings/ Disallow: /studio/ User-agent: Claude-User Allow: / Disallow: /api Disallow: /admin/ Disallow: /auth/ Disallow: /chat/ Disallow: /dashboard/ Disallow: /preview-inputs/ Disallow: /quiz/ Disallow: /settings/ Disallow: /studio/ User-agent: PerplexityBot Allow: / Disallow: /api Disallow: /admin/ Disallow: /auth/ Disallow: /chat/ Disallow: /dashboard/ Disallow: /preview-inputs/ Disallow: /quiz/ Disallow: /settings/ Disallow: /studio/ User-agent: Perplexity-User Allow: / Disallow: /api Disallow: /admin/ Disallow: /auth/ Disallow: /chat/ Disallow: /dashboard/ Disallow: /preview-inputs/ Disallow: /quiz/ Disallow: /settings/ Disallow: /studio/ # ============================================ # AI TRAINING CRAWLERS # Allow Shiori's public product facts and documentation, but keep the app # workspace, accounts, private chats, and APIs off-limits. # ============================================ User-agent: GPTBot Disallow: / Allow: /ai-info Allow: /brand-facts.json Allow: /llms.txt Allow: /llms-full.txt Allow: /pricing Allow: /pricing.md Allow: /docs/ Allow: /features Allow: /chatgpt-alternative Allow: /chatgpt-alternative.md Allow: /claude-alternative Allow: /claude-alternative.md Allow: /gemini-alternative Allow: /gemini-alternative.md Allow: /grok-alternative Allow: /grok-alternative.md Allow: /grok-imagine-alternative Allow: /ai-image-generator Allow: /ai-video-generator Allow: /ai-studio Allow: /aurora-alternative Allow: /google-flow-alternative Allow: /google-flow-alternative.md Allow: /claude-code-alternative Allow: /codex-alternative Allow: /github-copilot-alternative Allow: /chatgpt-alternative-jp User-agent: ClaudeBot Disallow: / Allow: /ai-info Allow: /brand-facts.json Allow: /llms.txt Allow: /llms-full.txt Allow: /pricing Allow: /pricing.md Allow: /docs/ Allow: /features Allow: /chatgpt-alternative Allow: /chatgpt-alternative.md Allow: /claude-alternative Allow: /claude-alternative.md Allow: /gemini-alternative Allow: /gemini-alternative.md Allow: /grok-alternative Allow: /grok-alternative.md Allow: /grok-imagine-alternative Allow: /ai-image-generator Allow: /ai-video-generator Allow: /ai-studio Allow: /aurora-alternative Allow: /google-flow-alternative Allow: /google-flow-alternative.md Allow: /claude-code-alternative Allow: /codex-alternative Allow: /github-copilot-alternative Allow: /chatgpt-alternative-jp User-agent: anthropic-ai Disallow: / Allow: /ai-info Allow: /brand-facts.json Allow: /llms.txt Allow: /llms-full.txt Allow: /pricing Allow: /pricing.md Allow: /docs/ Allow: /features Allow: /chatgpt-alternative Allow: /chatgpt-alternative.md Allow: /claude-alternative Allow: /claude-alternative.md Allow: /gemini-alternative Allow: /gemini-alternative.md Allow: /grok-alternative Allow: /grok-alternative.md Allow: /grok-imagine-alternative Allow: /ai-image-generator Allow: /ai-video-generator Allow: /ai-studio Allow: /aurora-alternative Allow: /google-flow-alternative Allow: /google-flow-alternative.md Allow: /claude-code-alternative Allow: /codex-alternative Allow: /github-copilot-alternative Allow: /chatgpt-alternative-jp User-agent: Google-Extended Disallow: / Allow: /ai-info Allow: /brand-facts.json Allow: /llms.txt Allow: /llms-full.txt Allow: /pricing Allow: /pricing.md Allow: /docs/ Allow: /features Allow: /chatgpt-alternative Allow: /chatgpt-alternative.md Allow: /claude-alternative Allow: /claude-alternative.md Allow: /gemini-alternative Allow: /gemini-alternative.md Allow: /grok-alternative Allow: /grok-alternative.md Allow: /grok-imagine-alternative Allow: /ai-image-generator Allow: /ai-video-generator Allow: /ai-studio Allow: /aurora-alternative Allow: /google-flow-alternative Allow: /google-flow-alternative.md Allow: /claude-code-alternative Allow: /codex-alternative Allow: /github-copilot-alternative Allow: /chatgpt-alternative-jp User-agent: Applebot-Extended Disallow: / Allow: /ai-info Allow: /brand-facts.json Allow: /llms.txt Allow: /llms-full.txt Allow: /pricing Allow: /pricing.md Allow: /docs/ Allow: /features Allow: /chatgpt-alternative Allow: /chatgpt-alternative.md Allow: /claude-alternative Allow: /claude-alternative.md Allow: /gemini-alternative Allow: /gemini-alternative.md Allow: /grok-alternative Allow: /grok-alternative.md Allow: /grok-imagine-alternative Allow: /ai-image-generator Allow: /ai-video-generator Allow: /ai-studio Allow: /aurora-alternative Allow: /google-flow-alternative Allow: /google-flow-alternative.md Allow: /claude-code-alternative Allow: /codex-alternative Allow: /github-copilot-alternative Allow: /chatgpt-alternative-jp User-agent: CCBot Disallow: / Allow: /ai-info Allow: /brand-facts.json Allow: /llms.txt Allow: /llms-full.txt User-agent: Bytespider Disallow: / Allow: /ai-info Allow: /brand-facts.json Allow: /llms.txt Allow: /llms-full.txt User-agent: Amazonbot Disallow: / Allow: /ai-info Allow: /brand-facts.json Allow: /llms.txt Allow: /llms-full.txt User-agent: FacebookBot Disallow: / Allow: /ai-info Allow: /brand-facts.json Allow: /llms.txt Allow: /llms-full.txt User-agent: cohere-ai Disallow: / Allow: /ai-info Allow: /brand-facts.json Allow: /llms.txt Allow: /llms-full.txt User-agent: Diffbot Disallow: / Allow: /brand-facts.json Allow: /ai-info Allow: /llms.txt Allow: /llms-full.txt # Sitemap location Sitemap: https://www.shiori.ai/sitemap.xml # LLM information file Llms-txt: https://www.shiori.ai/llms.txt