Bruno Sarlo
f2b3a14685
Major revision based on first principles thinking: - Simplified format: plain Markdown, human readable - Focus on capabilities (Can/Cannot) not API schemas - MCP gateway pointer for structured tool access - Clear positioning vs robots.txt and llms.txt The agents.md file is the handshake. The MCP gateway is where real work happens. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
112 lines
3.2 KiB
Markdown
112 lines
3.2 KiB
Markdown
# Frequently Asked Questions
|
|
|
|
## General
|
|
|
|
### Why not just use robots.txt?
|
|
|
|
robots.txt tells bots what they *cannot* do. agent.md tells AI agents what they *can* do. They're complementary:
|
|
|
|
- robots.txt: "Don't crawl /admin"
|
|
- agent.md: "You can search our catalog via this API"
|
|
|
|
### Why Markdown?
|
|
|
|
1. Human readable and editable
|
|
2. Widely supported in documentation tools
|
|
3. YAML frontmatter is a proven pattern
|
|
4. Renders nicely on GitHub and documentation sites
|
|
|
|
### Is this related to MCP (Model Context Protocol)?
|
|
|
|
Inspired by MCP, but designed for web discovery rather than local tool execution. The tool definition format is similar to make agent.md familiar to developers already using MCP.
|
|
|
|
### Why /.well-known/?
|
|
|
|
Following RFC 8615 for well-known URIs. This is the same pattern used by:
|
|
- `/.well-known/security.txt`
|
|
- `/.well-known/apple-app-site-association`
|
|
- `/.well-known/openid-configuration`
|
|
|
|
## Implementation
|
|
|
|
### Do I need to remove my robots.txt?
|
|
|
|
No. Keep your robots.txt for traditional crawlers. The `robots` section in agent.md can mirror or extend those rules for AI agents.
|
|
|
|
### Can I have different capabilities for different agents?
|
|
|
|
Yes. Use content negotiation based on User-Agent or implement OAuth2 scopes for fine-grained access control.
|
|
|
|
### How should agents authenticate?
|
|
|
|
Start simple:
|
|
1. No auth for public read-only tools
|
|
2. API keys for rate-limited or premium features
|
|
3. OAuth2 for user-specific actions
|
|
|
|
### What if my API changes?
|
|
|
|
Update your agent.md. Agents should re-fetch periodically (respect Cache-Control headers).
|
|
|
|
## Security
|
|
|
|
### Can malicious sites trick agents?
|
|
|
|
The protocol specifies that agents MUST only parse agent.md from the site's origin. Instructions in page content are ignored.
|
|
|
|
### How do I prevent abuse?
|
|
|
|
1. Use rate limits (per-tool and global)
|
|
2. Require API keys for sensitive operations
|
|
3. Use OAuth2 scopes for user actions
|
|
4. Monitor usage patterns
|
|
|
|
### Should I expose all my APIs?
|
|
|
|
No. Only expose what you want agents to use. Internal APIs, admin endpoints, and sensitive operations should not be in agent.md.
|
|
|
|
## Compatibility
|
|
|
|
### What about GraphQL APIs?
|
|
|
|
You can define tools that call GraphQL endpoints:
|
|
|
|
```yaml
|
|
tools:
|
|
- name: query_products
|
|
endpoint: "POST /graphql"
|
|
parameters:
|
|
type: object
|
|
properties:
|
|
query:
|
|
type: string
|
|
description: "GraphQL query (limited to Product type)"
|
|
```
|
|
|
|
### Can I use this with OpenAPI/Swagger?
|
|
|
|
Yes! You can generate agent.md from OpenAPI specs. We're working on tooling for this.
|
|
|
|
### What about WebSocket endpoints?
|
|
|
|
agent.md focuses on request-response patterns. For real-time features, document WebSocket endpoints in the Markdown section but define polling alternatives as tools.
|
|
|
|
## Adoption
|
|
|
|
### How do I tell if a site supports agent.md?
|
|
|
|
1. Check `/.well-known/agent.md`
|
|
2. Look for `<link rel="agent">` in HTML
|
|
3. Check for `Link` header in HTTP response
|
|
|
|
### What if the site doesn't have agent.md?
|
|
|
|
Fall back to:
|
|
1. Traditional web scraping (respecting robots.txt)
|
|
2. Looking for documented APIs
|
|
3. Using general browsing capabilities
|
|
|
|
### Who decides what goes in agent.md?
|
|
|
|
Site owners. This is an opt-in protocol. Sites choose what capabilities to expose.
|