Files

Bruno Sarlo f2b3a14685 v0.2: Simplify spec + add MCP gateway integration

Major revision based on first principles thinking:
- Simplified format: plain Markdown, human readable
- Focus on capabilities (Can/Cannot) not API schemas
- MCP gateway pointer for structured tool access
- Clear positioning vs robots.txt and llms.txt

The agents.md file is the handshake.
The MCP gateway is where real work happens.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

2026-01-14 09:01:56 -03:00

3.2 KiB

Raw Blame History

Frequently Asked Questions

General

Why not just use robots.txt?

robots.txt tells bots what they cannot do. agent.md tells AI agents what they can do. They're complementary:

robots.txt: "Don't crawl /admin"
agent.md: "You can search our catalog via this API"

Why Markdown?

Human readable and editable
Widely supported in documentation tools
YAML frontmatter is a proven pattern
Renders nicely on GitHub and documentation sites

Inspired by MCP, but designed for web discovery rather than local tool execution. The tool definition format is similar to make agent.md familiar to developers already using MCP.

Why /.well-known/?

Following RFC 8615 for well-known URIs. This is the same pattern used by:

/.well-known/security.txt
/.well-known/apple-app-site-association
/.well-known/openid-configuration

Implementation

Do I need to remove my robots.txt?

No. Keep your robots.txt for traditional crawlers. The robots section in agent.md can mirror or extend those rules for AI agents.

Can I have different capabilities for different agents?

Yes. Use content negotiation based on User-Agent or implement OAuth2 scopes for fine-grained access control.

How should agents authenticate?

Start simple:

No auth for public read-only tools
API keys for rate-limited or premium features
OAuth2 for user-specific actions

What if my API changes?

Update your agent.md. Agents should re-fetch periodically (respect Cache-Control headers).

Security

Can malicious sites trick agents?

The protocol specifies that agents MUST only parse agent.md from the site's origin. Instructions in page content are ignored.

How do I prevent abuse?

Use rate limits (per-tool and global)
Require API keys for sensitive operations
Use OAuth2 scopes for user actions
Monitor usage patterns

Should I expose all my APIs?

No. Only expose what you want agents to use. Internal APIs, admin endpoints, and sensitive operations should not be in agent.md.

Compatibility

What about GraphQL APIs?

You can define tools that call GraphQL endpoints:

tools:
  - name: query_products
    endpoint: "POST /graphql"
    parameters:
      type: object
      properties:
        query:
          type: string
          description: "GraphQL query (limited to Product type)"

Can I use this with OpenAPI/Swagger?

Yes! You can generate agent.md from OpenAPI specs. We're working on tooling for this.

What about WebSocket endpoints?

agent.md focuses on request-response patterns. For real-time features, document WebSocket endpoints in the Markdown section but define polling alternatives as tools.

Adoption

How do I tell if a site supports agent.md?

Check /.well-known/agent.md
Look for <link rel="agent"> in HTML
Check for Link header in HTTP response

What if the site doesn't have agent.md?

Fall back to:

Traditional web scraping (respecting robots.txt)
Looking for documented APIs
Using general browsing capabilities

Who decides what goes in agent.md?

Site owners. This is an opt-in protocol. Sites choose what capabilities to expose.

3.2 KiB Raw Blame History