v1.0.0-draft: First public draft release
agents.md protocol for AI agent web discovery. Key features: - Two formats: Pure Markdown (simple) or YAML frontmatter (structured) - MCP gateway integration for tool access - Discovery via /.well-known/agents.md - Security: origin trust, endpoint validation, auth guidance - Backward compatible with robots.txt and llms.txt Design based on 3-iteration process: 1. Gap analysis and planning 2. Multi-model consensus on format decisions 3. Code review for completeness and clarity Philosophy: robots.txt says what agents CANNOT do, agents.md says what they CAN do. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
52
CHANGELOG.md
Normal file
52
CHANGELOG.md
Normal file
@@ -0,0 +1,52 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
All notable changes to the agents.md protocol specification.
|
||||||
|
|
||||||
|
## [1.0.0-draft] - 2026-01-14
|
||||||
|
|
||||||
|
First public draft release.
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **Core specification** defining agents.md file format
|
||||||
|
- **Two format options**: Pure Markdown (simple) and YAML frontmatter + Markdown (structured)
|
||||||
|
- **MCP gateway integration** for pointing agents to Model Context Protocol servers
|
||||||
|
- **Discovery mechanism** via `/.well-known/agents.md` (primary) and `/agents.md` (fallback)
|
||||||
|
- **Security section** with origin trust, endpoint validation, and authentication guidance
|
||||||
|
- **Backward compatibility** guidance for robots.txt and llms.txt coexistence
|
||||||
|
- **Caching recommendations** for both servers and agents
|
||||||
|
- **Examples** covering minimal sites, MCP-enabled APIs, and OAuth-protected e-commerce
|
||||||
|
|
||||||
|
### Design Decisions
|
||||||
|
|
||||||
|
Based on multi-model consensus:
|
||||||
|
|
||||||
|
1. **Hybrid format** - Optional YAML frontmatter for machine-readable config, Markdown body for human context
|
||||||
|
2. **Simple naming** - Keep `Can/Cannot` section names for intuitiveness
|
||||||
|
3. **MCP as pointer** - Reference MCP endpoints, don't embed tool schemas
|
||||||
|
4. **Websites first** - Focus on web use case, extensible for future
|
||||||
|
|
||||||
|
### Philosophy
|
||||||
|
|
||||||
|
| Standard | Purpose |
|
||||||
|
|----------|---------|
|
||||||
|
| robots.txt | What agents cannot access |
|
||||||
|
| llms.txt | What content is important |
|
||||||
|
| agents.md | What agents can do |
|
||||||
|
|
||||||
|
## [0.2.0] - 2026-01-14
|
||||||
|
|
||||||
|
Internal revision with MCP integration.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- Simplified format from API-centric to text-centric
|
||||||
|
- Added MCP gateway pointer concept
|
||||||
|
- Removed JSON Schema tool definitions
|
||||||
|
|
||||||
|
## [0.1.0] - 2026-01-14
|
||||||
|
|
||||||
|
Initial internal draft.
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- Basic protocol concept
|
||||||
|
- API tool definitions (later removed)
|
||||||
60
README.md
60
README.md
@@ -2,15 +2,18 @@
|
|||||||
|
|
||||||
**Tell AI agents what they can do on your website.**
|
**Tell AI agents what they can do on your website.**
|
||||||
|
|
||||||
## The Gap
|
[]()
|
||||||
|
[]()
|
||||||
|
|
||||||
| File | Purpose |
|
## The Problem
|
||||||
|------|---------|
|
|
||||||
| robots.txt | What bots **cannot access** |
|
| File | What it tells agents |
|
||||||
|
|------|---------------------|
|
||||||
|
| robots.txt | What you **cannot access** |
|
||||||
| llms.txt | What **content matters** |
|
| llms.txt | What **content matters** |
|
||||||
| **agents.md** | What agents **can do** |
|
| **???** | What you **can do** |
|
||||||
|
|
||||||
## Quick Start
|
## The Solution
|
||||||
|
|
||||||
Create `/.well-known/agents.md`:
|
Create `/.well-known/agents.md`:
|
||||||
|
|
||||||
@@ -31,13 +34,20 @@ An online bookstore.
|
|||||||
hello@mysite.com
|
hello@mysite.com
|
||||||
```
|
```
|
||||||
|
|
||||||
That's it. Plain text. Human readable.
|
That's it. Plain text. Human readable. Machine parseable.
|
||||||
|
|
||||||
## With MCP Gateway
|
## With MCP Gateway
|
||||||
|
|
||||||
Point agents to your MCP server for structured tool access:
|
Point agents to your [MCP](https://modelcontextprotocol.io/) server for structured tool access:
|
||||||
|
|
||||||
```markdown
|
```yaml
|
||||||
|
---
|
||||||
|
version: "1.0"
|
||||||
|
mcp:
|
||||||
|
endpoint: https://mysite.com/.well-known/mcp
|
||||||
|
transport: streamable-http
|
||||||
|
auth: oauth2
|
||||||
|
---
|
||||||
# My Site
|
# My Site
|
||||||
|
|
||||||
An online bookstore.
|
An online bookstore.
|
||||||
@@ -47,11 +57,6 @@ An online bookstore.
|
|||||||
- Check prices
|
- Check prices
|
||||||
- Place orders (authenticated)
|
- Place orders (authenticated)
|
||||||
|
|
||||||
## MCP
|
|
||||||
endpoint: https://mysite.com/.well-known/mcp
|
|
||||||
transport: streamable-http
|
|
||||||
auth: oauth2
|
|
||||||
|
|
||||||
## Contact
|
## Contact
|
||||||
hello@mysite.com
|
hello@mysite.com
|
||||||
```
|
```
|
||||||
@@ -68,20 +73,31 @@ Agent requests /.well-known/agents.md
|
|||||||
|
|
||||||
## Documentation
|
## Documentation
|
||||||
|
|
||||||
- [Specification](./spec/README.md) - Full protocol spec
|
- **[Specification](./spec/README.md)** - Full protocol spec (v1.0.0-draft)
|
||||||
- [Examples](./examples/) - Real-world examples
|
- **[Examples](./examples/)** - Real-world examples
|
||||||
- [FAQ](./docs/FAQ.md) - Common questions
|
- **[FAQ](./docs/FAQ.md)** - Common questions
|
||||||
|
- **[Changelog](./CHANGELOG.md)** - Version history
|
||||||
|
|
||||||
|
## Quick Comparison
|
||||||
|
|
||||||
|
| Aspect | robots.txt | llms.txt | agents.md |
|
||||||
|
|--------|------------|----------|-----------|
|
||||||
|
| Purpose | Crawl control | Content summary | Capabilities |
|
||||||
|
| Format | Custom | Markdown | Markdown + YAML |
|
||||||
|
| MCP | No | No | Yes |
|
||||||
|
|
||||||
## Status
|
## Status
|
||||||
|
|
||||||
**Draft** - Version 0.2.0
|
**v1.0.0-draft** - First public draft release
|
||||||
|
|
||||||
|
Feedback welcome via issues.
|
||||||
|
|
||||||
## Related Standards
|
## Related Standards
|
||||||
|
|
||||||
- [robots.txt](https://www.rfc-editor.org/rfc/rfc9309) - Crawl restrictions (1994)
|
- [robots.txt](https://www.rfc-editor.org/rfc/rfc9309) - RFC 9309 (1994)
|
||||||
- [llms.txt](https://llmstxt.org/) - Content for LLMs (2024)
|
- [llms.txt](https://llmstxt.org/) - Jeremy Howard (2024)
|
||||||
- [AGENTS.md](https://agents.md/) - Repository instructions (2025)
|
- [AGENTS.md](https://agents.md/) - OpenAI/Sourcegraph (2025)
|
||||||
- [MCP](https://modelcontextprotocol.io/) - Tool protocol (2024)
|
- [MCP](https://modelcontextprotocol.io/) - Anthropic (2024)
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
|
|||||||
360
spec/README.md
360
spec/README.md
@@ -1,6 +1,6 @@
|
|||||||
# agents.md Protocol Specification
|
# agents.md Protocol Specification
|
||||||
|
|
||||||
**Version:** 0.2.0
|
**Version:** 1.0.0-draft
|
||||||
**Status:** Draft
|
**Status:** Draft
|
||||||
**Updated:** 2026-01-14
|
**Updated:** 2026-01-14
|
||||||
|
|
||||||
@@ -18,23 +18,40 @@ A simple text file that tells AI agents what they can do on a website and option
|
|||||||
|
|
||||||
## 1. Discovery
|
## 1. Discovery
|
||||||
|
|
||||||
**Location:** `/.well-known/agents.md` or `/agents.md`
|
### Location
|
||||||
|
|
||||||
**Content-Type:** `text/markdown` or `text/plain`
|
Primary: `/.well-known/agents.md`
|
||||||
|
Fallback: `/agents.md`
|
||||||
|
|
||||||
Agents request the file like any HTTP resource:
|
### Content-Type
|
||||||
|
|
||||||
```
|
`text/markdown` or `text/plain`
|
||||||
|
|
||||||
|
### Request
|
||||||
|
|
||||||
|
```http
|
||||||
GET /.well-known/agents.md HTTP/1.1
|
GET /.well-known/agents.md HTTP/1.1
|
||||||
Host: example.com
|
Host: example.com
|
||||||
User-Agent: MyAgent/1.0
|
User-Agent: MyAgent/1.0 (AI Agent)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Caching
|
||||||
|
|
||||||
|
Servers SHOULD set appropriate `Cache-Control` headers:
|
||||||
|
|
||||||
|
```http
|
||||||
|
Cache-Control: public, max-age=86400
|
||||||
|
```
|
||||||
|
|
||||||
|
Agents SHOULD cache the `agents.md` file for 24 hours unless HTTP headers specify otherwise. Agents MUST NOT request this file more than once per hour for the same origin.
|
||||||
|
|
||||||
## 2. Format
|
## 2. Format
|
||||||
|
|
||||||
Plain Markdown. Human readable. Machine parseable.
|
Two formats are supported:
|
||||||
|
|
||||||
### Minimal Example
|
### Format A: Pure Markdown (Simple)
|
||||||
|
|
||||||
|
Plain Markdown with section headers. Best for simple sites.
|
||||||
|
|
||||||
```markdown
|
```markdown
|
||||||
# Example Site
|
# Example Site
|
||||||
@@ -54,9 +71,18 @@ A bookstore since 2010.
|
|||||||
agents@example.com
|
agents@example.com
|
||||||
```
|
```
|
||||||
|
|
||||||
### With MCP Gateway
|
### Format B: YAML Frontmatter + Markdown (Structured)
|
||||||
|
|
||||||
|
YAML frontmatter for machine-readable configuration, Markdown body for human context. Recommended when using MCP.
|
||||||
|
|
||||||
```markdown
|
```markdown
|
||||||
|
---
|
||||||
|
version: "1.0"
|
||||||
|
mcp:
|
||||||
|
endpoint: https://example.com/.well-known/mcp
|
||||||
|
transport: streamable-http
|
||||||
|
auth: none
|
||||||
|
---
|
||||||
# Example Bookstore
|
# Example Bookstore
|
||||||
|
|
||||||
Online bookstore with 50,000 titles.
|
Online bookstore with 50,000 titles.
|
||||||
@@ -71,10 +97,6 @@ Online bookstore with 50,000 titles.
|
|||||||
- Modify user accounts
|
- Modify user accounts
|
||||||
- Access admin functions
|
- Access admin functions
|
||||||
|
|
||||||
## MCP
|
|
||||||
endpoint: https://example.com/.well-known/mcp
|
|
||||||
transport: streamable-http
|
|
||||||
|
|
||||||
## Behavior
|
## Behavior
|
||||||
- Respect 1 request/second
|
- Respect 1 request/second
|
||||||
- Cache product data 1 hour
|
- Cache product data 1 hour
|
||||||
@@ -84,11 +106,18 @@ transport: streamable-http
|
|||||||
agents@example.com
|
agents@example.com
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Parsing Rules
|
||||||
|
|
||||||
|
1. If file starts with `---`, parse YAML frontmatter until closing `---`
|
||||||
|
2. Parse remaining content as Markdown
|
||||||
|
3. Section headers (`## Name`) define semantic sections
|
||||||
|
4. List items under sections define capabilities/rules
|
||||||
|
|
||||||
## 3. Sections
|
## 3. Sections
|
||||||
|
|
||||||
All sections are optional. Use what makes sense.
|
All sections are optional. Use what makes sense for your site.
|
||||||
|
|
||||||
### Identity (Header)
|
### Identity (H1 Header)
|
||||||
|
|
||||||
```markdown
|
```markdown
|
||||||
# Site Name
|
# Site Name
|
||||||
@@ -110,24 +139,35 @@ Brief description of what this site is.
|
|||||||
|
|
||||||
### MCP Gateway
|
### MCP Gateway
|
||||||
|
|
||||||
```markdown
|
Defined in YAML frontmatter (preferred) or Markdown section:
|
||||||
## MCP
|
|
||||||
endpoint: <url>
|
**YAML Frontmatter (preferred):**
|
||||||
transport: streamable-http | sse | stdio
|
```yaml
|
||||||
auth: none | api_key | oauth2
|
---
|
||||||
|
mcp:
|
||||||
|
endpoint: https://example.com/.well-known/mcp
|
||||||
|
transport: streamable-http
|
||||||
|
auth: none
|
||||||
|
---
|
||||||
```
|
```
|
||||||
|
|
||||||
The MCP section points agents to a [Model Context Protocol](https://modelcontextprotocol.io/) server for structured tool access. This is the bridge from simple text discovery to full capability interaction.
|
**Markdown Section (fallback):**
|
||||||
|
```markdown
|
||||||
|
## MCP
|
||||||
|
endpoint: https://example.com/.well-known/mcp
|
||||||
|
transport: streamable-http
|
||||||
|
auth: none
|
||||||
|
```
|
||||||
|
|
||||||
**Transport options:**
|
When using Markdown section format, content MUST be valid YAML key-value pairs.
|
||||||
- `streamable-http` - HTTP with streaming (recommended for web)
|
|
||||||
- `sse` - Server-Sent Events
|
|
||||||
- `stdio` - Standard I/O (local only)
|
|
||||||
|
|
||||||
**Auth options:**
|
**Fields:**
|
||||||
- `none` - Public tools, no authentication
|
|
||||||
- `api_key` - Requires API key (specify how to obtain)
|
| Field | Required | Values | Description |
|
||||||
- `oauth2` - OAuth 2.0 flow
|
|-------|----------|--------|-------------|
|
||||||
|
| `endpoint` | Yes | URL | MCP server endpoint |
|
||||||
|
| `transport` | No | `streamable-http`, `sse` | Transport protocol (default: `streamable-http`) |
|
||||||
|
| `auth` | No | `none`, `api_key`, `oauth2` | Authentication method (default: `none`) |
|
||||||
|
|
||||||
### Behavior Rules
|
### Behavior Rules
|
||||||
|
|
||||||
@@ -148,12 +188,12 @@ https://example.com/agent-support
|
|||||||
|
|
||||||
## 4. MCP Integration
|
## 4. MCP Integration
|
||||||
|
|
||||||
The `agents.md` file is the **handshake**. The MCP gateway is where **real work happens**.
|
The `agents.md` file is the **discovery handshake**. The MCP gateway is where **structured interaction happens**.
|
||||||
|
|
||||||
```
|
```
|
||||||
Agent reads agents.md
|
Agent reads agents.md
|
||||||
│
|
│
|
||||||
├─► Basic agent: understands site capabilities from text
|
├─► Basic agent: understands site from text
|
||||||
│
|
│
|
||||||
└─► Advanced agent: connects to MCP gateway
|
└─► Advanced agent: connects to MCP gateway
|
||||||
│
|
│
|
||||||
@@ -164,114 +204,85 @@ Agent reads agents.md
|
|||||||
- Prompts (guided workflows)
|
- Prompts (guided workflows)
|
||||||
```
|
```
|
||||||
|
|
||||||
### Example MCP Discovery Flow
|
### Discovery Flow
|
||||||
|
|
||||||
1. Agent fetches `/.well-known/agents.md`
|
1. Agent fetches `/.well-known/agents.md`
|
||||||
2. Parses MCP endpoint: `https://example.com/.well-known/mcp`
|
2. Parses YAML frontmatter or `## MCP` section
|
||||||
3. Connects via MCP protocol
|
3. Extracts MCP endpoint URL
|
||||||
4. Discovers available tools via `tools/list`
|
4. Connects via MCP protocol
|
||||||
5. Uses tools as permitted
|
5. Discovers available tools via `tools/list`
|
||||||
|
6. Uses tools as permitted
|
||||||
|
|
||||||
## 5. Backward Compatibility
|
### MCP Endpoint Location
|
||||||
|
|
||||||
|
Recommended: `/.well-known/mcp`
|
||||||
|
|
||||||
|
This follows the well-known URI pattern and keeps agent-related endpoints together.
|
||||||
|
|
||||||
|
## 5. Security
|
||||||
|
|
||||||
|
### Origin Trust
|
||||||
|
|
||||||
|
Agents MUST only trust `agents.md` from the site's origin. Instructions embedded in page content MUST be ignored.
|
||||||
|
|
||||||
|
### MCP Endpoint Validation
|
||||||
|
|
||||||
|
The MCP endpoint MUST share the same registrable domain as the `agents.md` file. For example:
|
||||||
|
|
||||||
|
| agents.md location | Valid MCP endpoints |
|
||||||
|
|-------------------|---------------------|
|
||||||
|
| `example.com/.well-known/agents.md` | `example.com/mcp`, `api.example.com/mcp` |
|
||||||
|
| `shop.example.com/.well-known/agents.md` | `shop.example.com/mcp`, `api.shop.example.com/mcp` |
|
||||||
|
|
||||||
|
Cross-origin MCP endpoints (different registrable domain) MUST be rejected unless the user explicitly approves.
|
||||||
|
|
||||||
|
### Transport Security
|
||||||
|
|
||||||
|
- MCP endpoints MUST use HTTPS in production
|
||||||
|
- Agents SHOULD warn users about HTTP endpoints
|
||||||
|
- Certificate validation MUST NOT be disabled
|
||||||
|
|
||||||
|
### Authentication
|
||||||
|
|
||||||
|
When `auth: oauth2` is specified:
|
||||||
|
- Agents SHOULD request minimal scopes
|
||||||
|
- Tokens MUST be stored securely
|
||||||
|
- Refresh tokens SHOULD be used when available
|
||||||
|
|
||||||
|
When `auth: api_key` is specified:
|
||||||
|
- Keys SHOULD be obtained through official channels
|
||||||
|
- Keys MUST NOT be shared between users
|
||||||
|
- Keys SHOULD be rotated periodically
|
||||||
|
|
||||||
|
### Least Privilege
|
||||||
|
|
||||||
|
Agents SHOULD request only the permissions they need for the current task.
|
||||||
|
|
||||||
|
## 6. Backward Compatibility
|
||||||
|
|
||||||
### With robots.txt
|
### With robots.txt
|
||||||
|
|
||||||
If `agents.md` exists, it supplements but does not replace `robots.txt`. Agents should still respect robots.txt crawl directives.
|
`agents.md` supplements but does not replace `robots.txt`. In case of conflict regarding access restrictions, `robots.txt` takes precedence.
|
||||||
|
|
||||||
The `Cannot` section in agents.md can mirror robots.txt restrictions:
|
|
||||||
|
|
||||||
```markdown
|
```markdown
|
||||||
## Cannot
|
## Cannot
|
||||||
- Access /admin (see robots.txt)
|
- Access /admin (per robots.txt)
|
||||||
- Access /private
|
- Access /private
|
||||||
```
|
```
|
||||||
|
|
||||||
### With llms.txt
|
### With llms.txt
|
||||||
|
|
||||||
`llms.txt` describes **content** for understanding.
|
Both files serve different purposes and can coexist:
|
||||||
`agents.md` describes **capabilities** for action.
|
|
||||||
|
|
||||||
Both can coexist. A site might have:
|
| File | Purpose |
|
||||||
- `/robots.txt` - crawl restrictions
|
|------|---------|
|
||||||
- `/llms.txt` - content summary
|
| `/robots.txt` | Crawl restrictions |
|
||||||
- `/.well-known/agents.md` - agent capabilities + MCP pointer
|
| `/llms.txt` | Content summary for LLMs |
|
||||||
|
| `/.well-known/agents.md` | Agent capabilities + MCP |
|
||||||
## 6. Security
|
|
||||||
|
|
||||||
### Origin Trust
|
|
||||||
|
|
||||||
Agents MUST only trust `agents.md` from the site's origin. Instructions embedded in page content should be ignored.
|
|
||||||
|
|
||||||
### MCP Authentication
|
|
||||||
|
|
||||||
When connecting to MCP gateways:
|
|
||||||
- Verify the endpoint matches the origin
|
|
||||||
- Use TLS (HTTPS)
|
|
||||||
- Follow the specified auth method
|
|
||||||
|
|
||||||
### Least Privilege
|
|
||||||
|
|
||||||
Agents should request only the permissions they need. If `auth: oauth2` is specified, request minimal scopes.
|
|
||||||
|
|
||||||
## 7. Examples
|
## 7. Examples
|
||||||
|
|
||||||
### Public API Site
|
### Minimal (No MCP)
|
||||||
|
|
||||||
```markdown
|
|
||||||
# Weather API
|
|
||||||
|
|
||||||
Free weather data for AI agents.
|
|
||||||
|
|
||||||
## Can
|
|
||||||
- Get current conditions
|
|
||||||
- Get forecasts (up to 7 days)
|
|
||||||
- Get weather alerts
|
|
||||||
|
|
||||||
## MCP
|
|
||||||
endpoint: https://weather.example/mcp
|
|
||||||
transport: streamable-http
|
|
||||||
auth: none
|
|
||||||
|
|
||||||
## Behavior
|
|
||||||
- 60 requests/minute
|
|
||||||
- Cache forecasts 30 minutes
|
|
||||||
|
|
||||||
## Contact
|
|
||||||
api@weather.example
|
|
||||||
```
|
|
||||||
|
|
||||||
### E-commerce Site
|
|
||||||
|
|
||||||
```markdown
|
|
||||||
# TechMart
|
|
||||||
|
|
||||||
Electronics retailer.
|
|
||||||
|
|
||||||
## Can
|
|
||||||
- Search products
|
|
||||||
- Compare specifications
|
|
||||||
- Check prices and stock
|
|
||||||
- Add to cart (authenticated)
|
|
||||||
- Checkout (authenticated)
|
|
||||||
|
|
||||||
## Cannot
|
|
||||||
- Access order history without user consent
|
|
||||||
- Modify account settings
|
|
||||||
|
|
||||||
## MCP
|
|
||||||
endpoint: https://techmart.example/.well-known/mcp
|
|
||||||
transport: streamable-http
|
|
||||||
auth: oauth2
|
|
||||||
|
|
||||||
## Behavior
|
|
||||||
- 1 request/second for browsing
|
|
||||||
- Identify as AI agent in requests
|
|
||||||
|
|
||||||
## Contact
|
|
||||||
partners@techmart.example
|
|
||||||
```
|
|
||||||
|
|
||||||
### Simple Blog (No MCP)
|
|
||||||
|
|
||||||
```markdown
|
```markdown
|
||||||
# My Tech Blog
|
# My Tech Blog
|
||||||
@@ -291,12 +302,115 @@ Articles about software development.
|
|||||||
hello@myblog.example
|
hello@myblog.example
|
||||||
```
|
```
|
||||||
|
|
||||||
## Appendix: Comparison
|
### With MCP Gateway
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
---
|
||||||
|
version: "1.0"
|
||||||
|
mcp:
|
||||||
|
endpoint: https://weather.example/.well-known/mcp
|
||||||
|
transport: streamable-http
|
||||||
|
auth: none
|
||||||
|
---
|
||||||
|
# Weather API
|
||||||
|
|
||||||
|
Free weather data for AI agents.
|
||||||
|
|
||||||
|
## Can
|
||||||
|
- Get current conditions
|
||||||
|
- Get forecasts (up to 7 days)
|
||||||
|
- Get weather alerts
|
||||||
|
|
||||||
|
## Behavior
|
||||||
|
- 60 requests/minute
|
||||||
|
- Cache forecasts 30 minutes
|
||||||
|
|
||||||
|
## Contact
|
||||||
|
api@weather.example
|
||||||
|
```
|
||||||
|
|
||||||
|
### E-commerce with OAuth
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
---
|
||||||
|
version: "1.0"
|
||||||
|
mcp:
|
||||||
|
endpoint: https://techmart.example/.well-known/mcp
|
||||||
|
transport: streamable-http
|
||||||
|
auth: oauth2
|
||||||
|
---
|
||||||
|
# TechMart
|
||||||
|
|
||||||
|
Electronics retailer.
|
||||||
|
|
||||||
|
## Can
|
||||||
|
- Search products
|
||||||
|
- Compare specifications
|
||||||
|
- Check prices and stock
|
||||||
|
- Add to cart (authenticated)
|
||||||
|
- Checkout (authenticated)
|
||||||
|
|
||||||
|
## Cannot
|
||||||
|
- Access order history without user consent
|
||||||
|
- Modify account settings
|
||||||
|
|
||||||
|
## Behavior
|
||||||
|
- 1 request/second for browsing
|
||||||
|
- Identify as AI agent in requests
|
||||||
|
|
||||||
|
## Contact
|
||||||
|
partners@techmart.example
|
||||||
|
```
|
||||||
|
|
||||||
|
## 8. Implementation Notes
|
||||||
|
|
||||||
|
### For Site Owners
|
||||||
|
|
||||||
|
1. Create `/.well-known/agents.md` on your server
|
||||||
|
2. Start with Format A (pure Markdown) for simplicity
|
||||||
|
3. Add MCP gateway later if you want structured tool access
|
||||||
|
4. Set `Cache-Control` header for efficient agent behavior
|
||||||
|
|
||||||
|
### For Agent Developers
|
||||||
|
|
||||||
|
1. Check `/.well-known/agents.md` first, fall back to `/agents.md`
|
||||||
|
2. Parse YAML frontmatter if present
|
||||||
|
3. Cache responses per HTTP headers (default: 24 hours)
|
||||||
|
4. Respect `Cannot` restrictions and `Behavior` rules
|
||||||
|
5. Connect to MCP gateway for structured tools
|
||||||
|
|
||||||
|
### Versioning
|
||||||
|
|
||||||
|
The `version` field in YAML frontmatter indicates spec compatibility:
|
||||||
|
|
||||||
|
- `1.x` - Compatible with this specification
|
||||||
|
- Future versions will maintain backward compatibility within major version
|
||||||
|
|
||||||
|
## Appendix A: Comparison Table
|
||||||
|
|
||||||
| Aspect | robots.txt | llms.txt | agents.md |
|
| Aspect | robots.txt | llms.txt | agents.md |
|
||||||
|--------|------------|----------|-----------|
|
|--------|------------|----------|-----------|
|
||||||
| Purpose | Crawl control | Content summary | Capabilities |
|
| Purpose | Crawl control | Content summary | Capabilities |
|
||||||
| Format | Custom syntax | Markdown | Markdown |
|
| Format | Custom syntax | Markdown | Markdown + optional YAML |
|
||||||
| Focus | Restrictions | Understanding | Actions |
|
| Focus | Restrictions | Understanding | Actions |
|
||||||
| MCP | No | No | Yes (optional) |
|
| MCP | No | No | Yes (optional) |
|
||||||
| Year | 1994 | 2024 | 2026 |
|
| Year | 1994 | 2024 | 2026 |
|
||||||
|
|
||||||
|
## Appendix B: YAML Frontmatter Schema
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# All fields optional except where noted
|
||||||
|
version: string # Spec version (e.g., "1.0")
|
||||||
|
|
||||||
|
mcp:
|
||||||
|
endpoint: string # Required if mcp present. MCP server URL
|
||||||
|
transport: string # "streamable-http" | "sse" (default: streamable-http)
|
||||||
|
auth: string # "none" | "api_key" | "oauth2" (default: none)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Appendix C: References
|
||||||
|
|
||||||
|
- [Model Context Protocol](https://modelcontextprotocol.io/) - MCP Specification
|
||||||
|
- [RFC 8615](https://www.rfc-editor.org/rfc/rfc8615) - Well-Known URIs
|
||||||
|
- [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309) - robots.txt
|
||||||
|
- [llms.txt](https://llmstxt.org/) - LLM Content Discovery
|
||||||
|
|||||||
Reference in New Issue
Block a user