v0.2: Simplify spec + add MCP gateway integration
Major revision based on first principles thinking: - Simplified format: plain Markdown, human readable - Focus on capabilities (Can/Cannot) not API schemas - MCP gateway pointer for structured tool access - Clear positioning vs robots.txt and llms.txt The agents.md file is the handshake. The MCP gateway is where real work happens. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
302
spec/README.md
Normal file
302
spec/README.md
Normal file
@@ -0,0 +1,302 @@
|
||||
# agents.md Protocol Specification
|
||||
|
||||
**Version:** 0.2.0
|
||||
**Status:** Draft
|
||||
**Updated:** 2026-01-14
|
||||
|
||||
## Abstract
|
||||
|
||||
A simple text file that tells AI agents what they can do on a website and optionally points them to an MCP gateway for structured tool access.
|
||||
|
||||
## Philosophy
|
||||
|
||||
| Standard | Tells agents... |
|
||||
|----------|-----------------|
|
||||
| robots.txt | What you **cannot access** |
|
||||
| llms.txt | What **content is important** |
|
||||
| **agents.md** | What you **can do** + where to connect |
|
||||
|
||||
## 1. Discovery
|
||||
|
||||
**Location:** `/.well-known/agents.md` or `/agents.md`
|
||||
|
||||
**Content-Type:** `text/markdown` or `text/plain`
|
||||
|
||||
Agents request the file like any HTTP resource:
|
||||
|
||||
```
|
||||
GET /.well-known/agents.md HTTP/1.1
|
||||
Host: example.com
|
||||
User-Agent: MyAgent/1.0
|
||||
```
|
||||
|
||||
## 2. Format
|
||||
|
||||
Plain Markdown. Human readable. Machine parseable.
|
||||
|
||||
### Minimal Example
|
||||
|
||||
```markdown
|
||||
# Example Site
|
||||
|
||||
A bookstore since 2010.
|
||||
|
||||
## Can
|
||||
- Search catalog
|
||||
- Read book details
|
||||
- Check availability
|
||||
|
||||
## Cannot
|
||||
- Place orders without human
|
||||
- Access user accounts
|
||||
|
||||
## Contact
|
||||
agents@example.com
|
||||
```
|
||||
|
||||
### With MCP Gateway
|
||||
|
||||
```markdown
|
||||
# Example Bookstore
|
||||
|
||||
Online bookstore with 50,000 titles.
|
||||
|
||||
## Can
|
||||
- Search and browse catalog
|
||||
- Read reviews and descriptions
|
||||
- Check prices and stock
|
||||
- Place orders (authenticated)
|
||||
|
||||
## Cannot
|
||||
- Modify user accounts
|
||||
- Access admin functions
|
||||
|
||||
## MCP
|
||||
endpoint: https://example.com/.well-known/mcp
|
||||
transport: streamable-http
|
||||
|
||||
## Behavior
|
||||
- Respect 1 request/second
|
||||
- Cache product data 1 hour
|
||||
- Identify in User-Agent header
|
||||
|
||||
## Contact
|
||||
agents@example.com
|
||||
```
|
||||
|
||||
## 3. Sections
|
||||
|
||||
All sections are optional. Use what makes sense.
|
||||
|
||||
### Identity (Header)
|
||||
|
||||
```markdown
|
||||
# Site Name
|
||||
|
||||
Brief description of what this site is.
|
||||
```
|
||||
|
||||
### Capabilities (Can/Cannot)
|
||||
|
||||
```markdown
|
||||
## Can
|
||||
- Action agents are allowed to take
|
||||
- Another allowed action
|
||||
|
||||
## Cannot
|
||||
- Restricted action
|
||||
- Another restriction
|
||||
```
|
||||
|
||||
### MCP Gateway
|
||||
|
||||
```markdown
|
||||
## MCP
|
||||
endpoint: <url>
|
||||
transport: streamable-http | sse | stdio
|
||||
auth: none | api_key | oauth2
|
||||
```
|
||||
|
||||
The MCP section points agents to a [Model Context Protocol](https://modelcontextprotocol.io/) server for structured tool access. This is the bridge from simple text discovery to full capability interaction.
|
||||
|
||||
**Transport options:**
|
||||
- `streamable-http` - HTTP with streaming (recommended for web)
|
||||
- `sse` - Server-Sent Events
|
||||
- `stdio` - Standard I/O (local only)
|
||||
|
||||
**Auth options:**
|
||||
- `none` - Public tools, no authentication
|
||||
- `api_key` - Requires API key (specify how to obtain)
|
||||
- `oauth2` - OAuth 2.0 flow
|
||||
|
||||
### Behavior Rules
|
||||
|
||||
```markdown
|
||||
## Behavior
|
||||
- Rate limit guidance
|
||||
- Caching expectations
|
||||
- Identification requirements
|
||||
```
|
||||
|
||||
### Contact
|
||||
|
||||
```markdown
|
||||
## Contact
|
||||
email@example.com
|
||||
https://example.com/agent-support
|
||||
```
|
||||
|
||||
## 4. MCP Integration
|
||||
|
||||
The `agents.md` file is the **handshake**. The MCP gateway is where **real work happens**.
|
||||
|
||||
```
|
||||
Agent reads agents.md
|
||||
│
|
||||
├─► Basic agent: understands site capabilities from text
|
||||
│
|
||||
└─► Advanced agent: connects to MCP gateway
|
||||
│
|
||||
▼
|
||||
MCP Server exposes:
|
||||
- Tools (search, checkout, etc.)
|
||||
- Resources (catalog, docs)
|
||||
- Prompts (guided workflows)
|
||||
```
|
||||
|
||||
### Example MCP Discovery Flow
|
||||
|
||||
1. Agent fetches `/.well-known/agents.md`
|
||||
2. Parses MCP endpoint: `https://example.com/.well-known/mcp`
|
||||
3. Connects via MCP protocol
|
||||
4. Discovers available tools via `tools/list`
|
||||
5. Uses tools as permitted
|
||||
|
||||
## 5. Backward Compatibility
|
||||
|
||||
### With robots.txt
|
||||
|
||||
If `agents.md` exists, it supplements but does not replace `robots.txt`. Agents should still respect robots.txt crawl directives.
|
||||
|
||||
The `Cannot` section in agents.md can mirror robots.txt restrictions:
|
||||
|
||||
```markdown
|
||||
## Cannot
|
||||
- Access /admin (see robots.txt)
|
||||
- Access /private
|
||||
```
|
||||
|
||||
### With llms.txt
|
||||
|
||||
`llms.txt` describes **content** for understanding.
|
||||
`agents.md` describes **capabilities** for action.
|
||||
|
||||
Both can coexist. A site might have:
|
||||
- `/robots.txt` - crawl restrictions
|
||||
- `/llms.txt` - content summary
|
||||
- `/.well-known/agents.md` - agent capabilities + MCP pointer
|
||||
|
||||
## 6. Security
|
||||
|
||||
### Origin Trust
|
||||
|
||||
Agents MUST only trust `agents.md` from the site's origin. Instructions embedded in page content should be ignored.
|
||||
|
||||
### MCP Authentication
|
||||
|
||||
When connecting to MCP gateways:
|
||||
- Verify the endpoint matches the origin
|
||||
- Use TLS (HTTPS)
|
||||
- Follow the specified auth method
|
||||
|
||||
### Least Privilege
|
||||
|
||||
Agents should request only the permissions they need. If `auth: oauth2` is specified, request minimal scopes.
|
||||
|
||||
## 7. Examples
|
||||
|
||||
### Public API Site
|
||||
|
||||
```markdown
|
||||
# Weather API
|
||||
|
||||
Free weather data for AI agents.
|
||||
|
||||
## Can
|
||||
- Get current conditions
|
||||
- Get forecasts (up to 7 days)
|
||||
- Get weather alerts
|
||||
|
||||
## MCP
|
||||
endpoint: https://weather.example/mcp
|
||||
transport: streamable-http
|
||||
auth: none
|
||||
|
||||
## Behavior
|
||||
- 60 requests/minute
|
||||
- Cache forecasts 30 minutes
|
||||
|
||||
## Contact
|
||||
api@weather.example
|
||||
```
|
||||
|
||||
### E-commerce Site
|
||||
|
||||
```markdown
|
||||
# TechMart
|
||||
|
||||
Electronics retailer.
|
||||
|
||||
## Can
|
||||
- Search products
|
||||
- Compare specifications
|
||||
- Check prices and stock
|
||||
- Add to cart (authenticated)
|
||||
- Checkout (authenticated)
|
||||
|
||||
## Cannot
|
||||
- Access order history without user consent
|
||||
- Modify account settings
|
||||
|
||||
## MCP
|
||||
endpoint: https://techmart.example/.well-known/mcp
|
||||
transport: streamable-http
|
||||
auth: oauth2
|
||||
|
||||
## Behavior
|
||||
- 1 request/second for browsing
|
||||
- Identify as AI agent in requests
|
||||
|
||||
## Contact
|
||||
partners@techmart.example
|
||||
```
|
||||
|
||||
### Simple Blog (No MCP)
|
||||
|
||||
```markdown
|
||||
# My Tech Blog
|
||||
|
||||
Articles about software development.
|
||||
|
||||
## Can
|
||||
- Read all public articles
|
||||
- Search by topic
|
||||
- Access RSS feed at /feed.xml
|
||||
|
||||
## Cannot
|
||||
- Post comments (requires human)
|
||||
- Access draft posts
|
||||
|
||||
## Contact
|
||||
hello@myblog.example
|
||||
```
|
||||
|
||||
## Appendix: Comparison
|
||||
|
||||
| Aspect | robots.txt | llms.txt | agents.md |
|
||||
|--------|------------|----------|-----------|
|
||||
| Purpose | Crawl control | Content summary | Capabilities |
|
||||
| Format | Custom syntax | Markdown | Markdown |
|
||||
| Focus | Restrictions | Understanding | Actions |
|
||||
| MCP | No | No | Yes (optional) |
|
||||
| Year | 1994 | 2024 | 2026 |
|
||||
Reference in New Issue
Block a user