2 files changed, 86 insertions, 5 deletions
diff --git a/docs/_html_extra/llms.txt b/docs/_html_extra/llms.txt
new file mode 100644
index 00000000..a8f5b002
--- /dev/null
+++ b/docs/_html_extra/llms.txt
@@ -0,0 +1,52 @@
+# VyOS
+
+> VyOS is a free, open-source network operating system based on Debian GNU/Linux.
+> It provides routing, firewall, and VPN functionality via a unified CLI using
+> `set`/`delete`/`show` command hierarchy. Current rolling release is 1.5.x (circinus).
+
+VyOS configuration follows a tree structure. All configuration commands start with
+`set` (to add/change) or `delete` (to remove). Changes are staged and applied with
+`commit`. The CLI hierarchy maps directly to the documentation structure.
+
+## Quick Start
+
+- [Quick Start Guide](https://docs.vyos.io/en/latest/quick-start.html): Minimal setup walkthrough
+- [CLI Overview](https://docs.vyos.io/en/latest/cli.html): Command-line interface usage
+
+## Configuration
+
+- [Firewall](https://docs.vyos.io/en/latest/configuration/firewall/index.html): Zone-based firewall, rules, groups
+- [Interfaces](https://docs.vyos.io/en/latest/configuration/interfaces/index.html): Ethernet, bonding, bridge, VLAN, tunnel, wireless
+- [Protocols](https://docs.vyos.io/en/latest/configuration/protocols/index.html): BGP, OSPF, IS-IS, static routing, MPLS
+- [VPN](https://docs.vyos.io/en/latest/configuration/vpn/index.html): IPsec, OpenVPN, WireGuard, L2TP, PPTP
+- [NAT](https://docs.vyos.io/en/latest/configuration/nat/index.html): Source NAT, destination NAT, NAT66
+- [System](https://docs.vyos.io/en/latest/configuration/system/index.html): DNS, NTP, syslog, users, task scheduler
+- [High Availability](https://docs.vyos.io/en/latest/configuration/highavailability/index.html): VRRP
+- [Load Balancing](https://docs.vyos.io/en/latest/configuration/loadbalancing/index.html): WAN and reverse proxy
+- [Containers](https://docs.vyos.io/en/latest/configuration/container/index.html): Podman-based container support
+- [PKI](https://docs.vyos.io/en/latest/configuration/pki/index.html): Certificate management
+- [Policy](https://docs.vyos.io/en/latest/configuration/policy/index.html): Route maps, prefix lists, access lists
+- [Traffic Policy](https://docs.vyos.io/en/latest/configuration/trafficpolicy/index.html): QoS and shaping
+- [Service](https://docs.vyos.io/en/latest/configuration/service/index.html): DHCP, DNS forwarding, SNMP, SSH, HTTPS API
+- [VRF](https://docs.vyos.io/en/latest/configuration/vrf/index.html): Virtual routing and forwarding
+
+## Operations
+
+- [Operational Commands](https://docs.vyos.io/en/latest/operation/index.html): Show, monitor, restart commands
+
+## Installation
+
+- [Installation Guide](https://docs.vyos.io/en/latest/installation/index.html): Bare metal, virtual, cloud deployments
+
+## Automation
+
+- [Automation](https://docs.vyos.io/en/latest/automation/index.html): Ansible, Terraform, HTTP API, NETCONF
+
+## Configuration Examples
+
+- [Blueprints](https://docs.vyos.io/en/latest/configexamples/index.html): Real-world topology examples
+
+## Optional
+
+- [Contributing](https://docs.vyos.io/en/latest/contributing/index.html): Development workflow
+- [VPP](https://docs.vyos.io/en/latest/vpp/index.html): Vector Packet Processing integration
diff --git a/docs/_html_extra/robots.txt b/docs/_html_extra/robots.txt
index 627cb18e..71a6990b 100644
--- a/docs/_html_extra/robots.txt
+++ b/docs/_html_extra/robots.txt
@@ -1,7 +1,36 @@
-User-agent: atlassian-bot
-Disallow: 
-
 User-agent: *
-Disallow: # Allow everything
+Allow: /
+
+# AI Training Crawlers
+User-agent: GPTBot
+Allow: /
+
+User-agent: ClaudeBot
+Allow: /
+
+User-agent: Google-Extended
+Allow: /
+
+User-agent: CCBot
+Allow: /
+
+User-agent: PerplexityBot
+Allow: /
+
+# AI Search/Retrieval
+User-agent: ChatGPT-User
+Allow: /
+
+User-agent: Claude-SearchBot
+Allow: /
+
+User-agent: Claude-User
+Allow: /
+
+User-agent: OAI-SearchBot
+Allow: /
+
+User-agent: Perplexity-User
+Allow: /
 
-Sitemap: https://docs.vyos.io/sitemap.xml
-\ No newline at end of file
+Sitemap: https://docs.vyos.io/sitemap.xml