Serfe AI Discovery

Expose published WordPress content as clean Markdown and machine-readable discovery signals for AI agents, LLMs, and semantic search engines.

Plugin Objective

Traditional SEO plugins optimize your website for search engines like Google and Bing. Serfe AI Discovery complements your SEO strategy by making your public content structured and discoverable for the Agentic Web (AI Agents, LLMs, and conversational engines).

By serving clean .md representations of your public content alongside machine-readable discovery endpoints (robots.txt, llms.txt, and RFC 9727 API Catalogs), the plugin bridges the gap between WordPress and AI consumers.

What It Does

• Clean Markdown Content: Serves structured Markdown with YAML front matter for eligible public URLs via .md suffix or HTTP content negotiation (Accept: text/markdown).
• RFC 8288 Link Headers: Automatically advertises alternate Markdown versions, verified llms.txt (rel="describedby"), and XML sitemaps (rel="related").
• Machine-Readable API Catalog: Exposes an optional RFC 9727 Linkset at /.well-known/api-catalog pointing to the WordPress REST API.
• Root Discovery Management: Provides safe, in-admin editing and validation for robots.txt and llms.txt. On Multisite, each site serves and stores its own virtual file without writing to the shared filesystem.
• Agent Readiness Checks: Offers bounded, read-only local diagnostics for AI and agent compatibility.

Quick Example

# Standard browser request:
https://example.com/sample-post/      → Returns HTML

# Explicit Markdown request:
https://example.com/sample-post.md    → Returns Markdown (Content-Type: text/markdown)

Content negotiation is also supported:

curl -H "Accept: text/markdown" https://example.com/sample-post/

Eligible HTML pages include a link to their Markdown alternative:

<link rel="alternate" type="text/markdown" href="https://example.com/sample-post.md">

Requirements

• WordPress: 6.2 or later. The integration workflow is configured to check WordPress 6.2, 6.3, 6.4, 6.5, 6.6, 6.7, 7.0, and 7.1.
• PHP: 7.4 or higher (tested across PHP 7.4, 8.1, 8.4, and 8.5).

Supported Content

Excluded by Default

• Drafts, private, password-protected, or non-public content.
• WooCommerce transactional pages (cart, checkout, account).
• Dynamic blocks, shortcodes, feeds, and embeds (only static, saved content is exposed).

Usage & Settings

Access the plugin from the WordPress admin menu under AI Discovery:

• Overview: High-level status of active features, discovery files, and recent checks.
• Markdown: Instant master toggle to enable/disable Markdown publication. Includes Advanced Settings to exclude specific post types or exact URL paths (one per line).
• Robots.txt & llms.txt:
  • Review and edit root discovery files directly.
  • Safe takeover mechanism with checksum verification when replacing virtual responses (e.g. from Yoast or Rank Math).
  • One-click automatic backup and rollback.
  • Read-only validator checking sitemap declarations and crawl policies.
• API Catalog: Enable or disable the RFC 9727 public API catalog linkset.
• Help & Support: Community support via GitHub and Serfe Professional Services (GEO Assessment consultation).

Agents Discovery

Under AI Discovery → Discovery Checks, administrators can run read-only checks that request the site's own public endpoints. Checks verify:

• Availability and basic response checks for robots.txt and llms.txt.
• XML Sitemap detection and Link headers.
• Markdown content negotiation (Accept: text/markdown).
• API Catalog and REST API availability.
• Optional MCP, OAuth Protected Resource, Agent Skills, and Agentic Commerce discovery resources.

Extending Readiness Signals

Developers can register custom endpoints (such as Model Context Protocol or Agentic Commerce) via WordPress filters:

// Register Model Context Protocol (MCP) endpoint
add_filter( 'serfe_agent_readiness_mcp', function () {
    return array(
        'endpoint'   => 'https://example.com/mcp',
        'health_url' => 'https://example.com/mcp/health', // Optional 2xx health check
    );
} );

// Register OAuth Protected Resource Metadata
add_filter( 'serfe_agent_readiness_oauth_protected_resource', function () {
    return array(
        'metadata_url' => 'https://example.com/.well-known/oauth-protected-resource',
    );
} );

// Register Agentic Commerce endpoint
add_filter( 'serfe_agent_readiness_agentic_commerce', function () {
    return array(
        'endpoint'   => 'https://example.com/agentic-commerce',
        'health_url' => 'https://example.com/agentic-commerce/health',
    );
} );

Installation

1. Download the source ZIP from the Serfe AI Discovery repository using Code → Download ZIP.
2. In WordPress, go to Plugins → Add New → Upload Plugin and upload the downloaded ZIP.
3. Activate the plugin.
4. Go to AI Discovery → Markdown and turn on the master toggle.
5. Visit any public post with .md appended to verify.

Safety & Routing

The plugin enforces security and routing boundaries for Markdown responses:

• Internal WordPress routes (/wp-admin, /wp-json, /wp-content, /wp-includes, /xmlrpc.php) are never routed as Markdown.
• Markdown responses include X-Robots-Tag: noindex, follow and estimated token metrics (X-Markdown-Tokens).
• HTML pages that advertise Markdown and negotiated Markdown responses include Vary: Accept.
• Markdown responses use Cache-Control: private, no-store so shared caches do not retain them.

Collaboration & Support

Contributions are welcome!

• Contributing: Read CONTRIBUTING.md for guidelines and local verification steps.
• Community Support: Check SUPPORT.md or open an issue.
• Security: Report vulnerabilities privately following instructions in SECURITY.md.

Development & Testing

Run the self-contained test suites locally:

# Run unit & compatibility tests
for t in tests/*.php; do php "$t"; done

# Install development dependencies
composer install

# WordPress Coding Standards (PHPCS)
vendor/bin/phpcs --standard=phpcs.xml.dist

# Static Analysis (PHPStan)
vendor/bin/phpstan analyse --configuration=phpstan.neon.dist --no-progress

For live HTTP checks against a running WordPress site:

tests/http-smoke.sh https://example.com/sample-post/

Version

For the latest published version and release notes, see GitHub Releases and CHANGELOG.md.

About Serfe

Serfe is a software development company that specializes in building high-quality, scalable web applications and integrations. We are committed to open-source and helping organizations embrace modern web standards, including the emerging Agentic Web.

Learn more at serfe.com.

License

This plugin is licensed under the GNU General Public License v2.0 or later. The Serfe name and logos remain subject to their trademark rights.