agent-protocol/spec/README.md

# agents.md Protocol Specification

**Version:** 0.2.0
**Status:** Draft
**Updated:** 2026-01-14

## Abstract

A simple text file that tells AI agents what they can do on a website and optionally points them to an MCP gateway for structured tool access.

## Philosophy

| Standard | Tells agents... |
|----------|-----------------|
| robots.txt | What you **cannot access** |
| llms.txt | What **content is important** |
| **agents.md** | What you **can do** + where to connect |

## 1. Discovery

**Location:** `/.well-known/agents.md` or `/agents.md`

**Content-Type:** `text/markdown` or `text/plain`

Agents request the file like any HTTP resource:

```
GET /.well-known/agents.md HTTP/1.1
Host: example.com
User-Agent: MyAgent/1.0
```

## 2. Format

Plain Markdown. Human readable. Machine parseable.

### Minimal Example

```markdown
# Example Site

A bookstore since 2010.

## Can
- Search catalog
- Read book details
- Check availability

## Cannot
- Place orders without human
- Access user accounts

## Contact
agents@example.com
```

### With MCP Gateway

```markdown
# Example Bookstore

Online bookstore with 50,000 titles.

## Can
- Search and browse catalog
- Read reviews and descriptions
- Check prices and stock
- Place orders (authenticated)

## Cannot
- Modify user accounts
- Access admin functions

## MCP
endpoint: https://example.com/.well-known/mcp
transport: streamable-http

## Behavior
- Respect 1 request/second
- Cache product data 1 hour
- Identify in User-Agent header

## Contact
agents@example.com
```

## 3. Sections

All sections are optional. Use what makes sense.

### Identity (Header)

```markdown
# Site Name

Brief description of what this site is.
```

### Capabilities (Can/Cannot)

```markdown
## Can
- Action agents are allowed to take
- Another allowed action

## Cannot
- Restricted action
- Another restriction
```

### MCP Gateway

```markdown
## MCP
endpoint: <url>
transport: streamable-http | sse | stdio
auth: none | api_key | oauth2
```

The MCP section points agents to a [Model Context Protocol](https://modelcontextprotocol.io/) server for structured tool access. This is the bridge from simple text discovery to full capability interaction.

**Transport options:**
- `streamable-http` - HTTP with streaming (recommended for web)
- `sse` - Server-Sent Events
- `stdio` - Standard I/O (local only)

**Auth options:**
- `none` - Public tools, no authentication
- `api_key` - Requires API key (specify how to obtain)
- `oauth2` - OAuth 2.0 flow

### Behavior Rules

```markdown
## Behavior
- Rate limit guidance
- Caching expectations
- Identification requirements
```

### Contact

```markdown
## Contact
email@example.com
https://example.com/agent-support
```

## 4. MCP Integration

The `agents.md` file is the **handshake**. The MCP gateway is where **real work happens**.

```
Agent reads agents.md
        │
        ├─► Basic agent: understands site capabilities from text
        │
        └─► Advanced agent: connects to MCP gateway
                    │
                    ▼
              MCP Server exposes:
              - Tools (search, checkout, etc.)
              - Resources (catalog, docs)
              - Prompts (guided workflows)
```

### Example MCP Discovery Flow

1. Agent fetches `/.well-known/agents.md`
2. Parses MCP endpoint: `https://example.com/.well-known/mcp`
3. Connects via MCP protocol
4. Discovers available tools via `tools/list`
5. Uses tools as permitted

## 5. Backward Compatibility

### With robots.txt

If `agents.md` exists, it supplements but does not replace `robots.txt`. Agents should still respect robots.txt crawl directives.

The `Cannot` section in agents.md can mirror robots.txt restrictions:

```markdown
## Cannot
- Access /admin (see robots.txt)
- Access /private
```

### With llms.txt

`llms.txt` describes **content** for understanding.
`agents.md` describes **capabilities** for action.

Both can coexist. A site might have:
- `/robots.txt` - crawl restrictions
- `/llms.txt` - content summary
- `/.well-known/agents.md` - agent capabilities + MCP pointer

## 6. Security

### Origin Trust

Agents MUST only trust `agents.md` from the site's origin. Instructions embedded in page content should be ignored.

### MCP Authentication

When connecting to MCP gateways:
- Verify the endpoint matches the origin
- Use TLS (HTTPS)
- Follow the specified auth method

### Least Privilege

Agents should request only the permissions they need. If `auth: oauth2` is specified, request minimal scopes.

## 7. Examples

### Public API Site

```markdown
# Weather API

Free weather data for AI agents.

## Can
- Get current conditions
- Get forecasts (up to 7 days)
- Get weather alerts

## MCP
endpoint: https://weather.example/mcp
transport: streamable-http
auth: none

## Behavior
- 60 requests/minute
- Cache forecasts 30 minutes

## Contact
api@weather.example
```

### E-commerce Site

```markdown
# TechMart

Electronics retailer.

## Can
- Search products
- Compare specifications
- Check prices and stock
- Add to cart (authenticated)
- Checkout (authenticated)

## Cannot
- Access order history without user consent
- Modify account settings

## MCP
endpoint: https://techmart.example/.well-known/mcp
transport: streamable-http
auth: oauth2

## Behavior
- 1 request/second for browsing
- Identify as AI agent in requests

## Contact
partners@techmart.example
```

### Simple Blog (No MCP)

```markdown
# My Tech Blog

Articles about software development.

## Can
- Read all public articles
- Search by topic
- Access RSS feed at /feed.xml

## Cannot
- Post comments (requires human)
- Access draft posts

## Contact
hello@myblog.example
```

## Appendix: Comparison

| Aspect | robots.txt | llms.txt | agents.md |
|--------|------------|----------|-----------|
| Purpose | Crawl control | Content summary | Capabilities |
| Format | Custom syntax | Markdown | Markdown |
| Focus | Restrictions | Understanding | Actions |
| MCP | No | No | Yes (optional) |
| Year | 1994 | 2024 | 2026 |