WP Cortex

WP Cortex is a memory layer for WordPress. It indexes site content into a local SQLite store (structured fields, FTS5 full-text search and vector embeddings) so that AI search and chat features can be built on top of it. Everything is stored locally; the only external call is the optional OpenAI embeddings request.

• Version: 0.2.0
• Author: Bojan
• License: GPL-2.0-or-later

Status: the indexing layer, the hybrid search service and the admin chat assistant are implemented. Frontend chat and MCP access are on the roadmap.

Features (current)

• Two isolated indexes
  • public.sqlite holds only published, publicly viewable, non-password-protected content, and only fields/sections flagged as public.
  • admin.sqlite holds everything eligible: drafts, pending, scheduled and private posts (configurable), plus Yoast SEO, ACF and custom meta data.
• Extractors: core post data, taxonomies, Yoast SEO, ACF, custom meta keys, media library files. Extensible through a filter.
• Media indexing (optional): attachments are indexed as post type attachment with title, caption, alt text, description, file URL and media fields (MIME type, file name and size, dimensions, audio/video duration, artist, album; EXIF credit, copyright and camera in the admin index only). Media inherit the status and password protection of the post they are attached to; unattached media count as published. Image alt_text is stored even when empty. Yoast fields are not indexed for media.
• Front-end-equivalent content: blocks (including dynamic ones) are rendered as an anonymous visitor, so the index matches what the public sees and logged-in-only content never leaks into it. Forms, navigation, scripts and styles are stripped.
• Heading-aware chunking: HTML is split at headings; consecutive short sections are packed together and long ones split at paragraph/sentence boundaries into chunks of roughly N characters with configurable overlap.
• OpenAI embeddings with reuse by chunk hash: unchanged text is never re-embedded, even across documents and scopes.
• Batch indexing with progress UI: resumable and cancellable (Cortex > Indexing).
• Incremental auto-sync via WP-Cron when content, terms or relevant meta change.
• Hybrid search service (WPCortex\Search\SearchService): FTS5 BM25 keyword search, brute-force cosine semantic search, merged with Reciprocal Rank Fusion and grouped per document. Structured filters on post type, status, modified date and any indexed field (eq, neq, contains, not_contains, empty, not_empty, missing, exists). Degrades to keyword search when embeddings are unavailable. Also offers get_document() and field_catalog().
• Admin chat assistant: a floating chat panel on every admin screen for administrators. An LLM (through the WordPress AI Client, any configured provider) answers questions about site content by calling tools over the admin index, shows the posts cited in its answer (as #ID) as cards below it and can open a post in the editor on request. Conversations are stored per user in the {prefix}wp_cortex_conversations table. Needs an AI provider API key under Settings > Connectors.
• Abilities (WordPress Abilities API, category wp-cortex, read-only, manage_options, admin index): wp-cortex/search-content (hybrid search with field, status, type, author and date filters), wp-cortex/find-duplicates (posts sharing a title, field value or text content), wp-cortex/get-document, wp-cortex/list-fields. Used by the chat agent as tools; not exposed over REST or MCP yet.
• WP-CLI commands (wp cortex index, wp cortex status, wp cortex search).

Requirements

• WordPress 7.0+
• PHP 8.1+
• PHP pdo_sqlite extension compiled with FTS5 (the plugin shows an admin notice and stays inactive if pdo_sqlite is missing)
• Optional: an OpenAI API key. Without one, indexing still works but no vectors are created. The key is read, in order, from the OPENAI_API_KEY environment variable, the OPENAI_API_KEY constant, or the key configured under Settings > Connectors.

Installation and quick start

1. Copy the plugin into wp-content/plugins/wp-cortex and activate it.
2. (Recommended) Define WP_CORTEX_DATA_DIR in wp-config.php with a path outside the web root.
3. Add an OpenAI API key under Settings > Connectors (or via env/constant).
4. Open Cortex > Settings, choose post types and sources, and save.
5. Open Cortex > Indexing and click Sync (or run wp cortex index).

Configuration

Settings are stored in the wp_cortex_settings option and edited under Cortex > Settings.

Data directory

By default the databases live in wp-content/uploads/wp-cortex-<random>/. The random directory name is generated once and stored in the wp_cortex_data_dir_name option. The directory receives index.php, .htaccess and web.config deny rules, but Nginx ignores .htaccess, so on Nginx the fallback relies only on the unguessable name. The Indexing screen warns when the fallback is used and checks over HTTP whether admin.sqlite is downloadable.

For real protection, put the data outside the web root:

define( 'WP_CORTEX_DATA_DIR', '/var/lib/wp-cortex' );

The directory must be writable by the web server user. Changing the location does not move existing databases; re-run an index.

Filters

• wp_cortex_extractors (Extractor[] $extractors): add, remove or reorder extractors. Non-Extractor entries are discarded.
• wp_cortex_openai_api_key (string $key): override the OpenAI API key.
• wp_cortex_chat_system_instruction (string $instruction, array $context): modify the chat assistant's system instruction. $context holds screen and post_id.

Indexing

• Sync walks all eligible posts, skips unchanged ones and removes documents that are no longer eligible.
• Rebuild deletes both database files first and indexes everything from scratch. Use it after changing chunk settings or when the schema is damaged.
• Skipping: for each scope a content hash is computed over the document columns, fields and chunk hashes. If it matches the stored hash (and, when embeddings are active, no chunk lacks a vector for the current embedding signature), the post is skipped.
• Embedding reuse: each chunk is hashed (title, section heading, text). Existing vectors with the same hash and signature are reused; only missing ones are sent to OpenAI, in one pass per batch (96 inputs per request, up to 3 attempts on 429/5xx).
• Stale removal: documents not seen by a run (deleted posts, changed status, deselected post types) are deleted when the run finishes. Posts that become ineligible are also removed immediately when processed.
• Auto-sync: on save, trash/untrash/delete, status transition, term changes and changes of _yoast_wpseo_* or configured meta keys (with media indexing also alt text, attachment metadata, attach/detach, and the media attached to a post whose status or password changes), post IDs are queued (option wp_cortex_sync_queue) and processed by a single WP-Cron event ~15 seconds later, 50 posts at a time. It pauses while a full run is active.
• Run state is kept in the wp_cortex_index_run option and shared by the admin UI, REST and WP-CLI; a lock option prevents concurrent batches (stale after 300 s).

WP-CLI

wp cortex index [--rebuild]
wp cortex status
wp cortex search <query> [--scope=<admin|public>] [--mode=<hybrid|keyword|semantic>] [--limit=<n>] [--type=<post_type>]

index runs a full Sync (or Rebuild with --rebuild) with a progress bar and prints a summary. status shows per-scope statistics and the last run. search prints matching documents (id, type, status, score, title, best heading); scope defaults to admin, mode to hybrid.

REST API

Namespace wp-cortex/v1. All routes require the manage_options capability (and the usual REST nonce/auth).

Data model

Both databases share one schema (PRAGMA user_version = 1; WAL mode, foreign keys on):

• documents: one row per object (object_type, object_id, subtype, status, title, url, excerpt, author, published_at, modified_at, content_hash, indexed_at, run_id), unique on (object_type, object_id).
• fields: structured key/value rows (document_id, source, name, value). Sources: core, taxonomy, yoast, acf, meta, media.
• chunks: document_id, position, heading, content, content_hash, embedding BLOB, embedding_model.
• chunks_fts: FTS5 virtual table (title, heading, content; tokenizer unicode61 remove_diacritics 2) with rowid = chunks.id.

Vectors are little-endian float32 BLOBs (PHP pack( 'g*' )), the same layout sqlite-vec uses.

Embedding signature: <model>:<dimensions> (for example text-embedding-3-small:1536) is stored in chunks.embedding_model. Changing the model or dimensions makes existing vectors stale and they are regenerated on the next run.

Yoast fields are stored even when empty, so questions such as "no meta description" are simple equality checks.

Architecture

wp-cortex.php            Bootstrap, autoloader, activation hooks
uninstall.php            Removes the data directory
src/
  Plugin.php             Wires services together
  Settings.php           Typed option access, sanitizing, embedding signature
  Storage/
    Storage.php          Data directory location and protection
    Database.php         SQLite connection, schema, CRUD, stats
  Indexing/
    Document.php         Normalized object with public/admin flags
    Chunker.php          Heading-aware chunking
    Indexer.php          Pipeline: extract, chunk, embed, write
    IndexRun.php         Resumable batch run state machine
    PostSync.php         Auto-sync queue and WP-Cron worker
    Extractors/          Core, Taxonomy, Yoast, Acf, Meta, Media extractors
  Embeddings/
    EmbeddingProvider.php  Provider interface
    OpenAIEmbeddings.php   OpenAI client
    VectorCodec.php        float32 BLOB pack/unpack
  Search/SearchService.php Hybrid search, document lookup, field catalog
  Abilities/Abilities.php  Abilities API category and read-only abilities
  Chat/
    ChatAgent.php          LLM tool-calling loop (WordPress AI Client)
    ModelCatalog.php       Provider model list (cached)
    ConversationStore.php  MySQL conversation table
  Rest/                    IndexController, ChatController
  Cli/Command.php          WP-CLI commands
  Admin/                   Menu, Settings page, Indexing page, Chat panel
assets/                  Admin JS (no build step) and CSS

Roadmap

None of the following is implemented yet.

• Frontend chat with content: block/widget with citations, backed by the public index.
• MCP access for external agents via the WordPress Abilities API and MCP Adapter: public tools use the public index; admin tools use the admin index and are capability-gated.
• Optional sqlite-vec acceleration for vector search.
• Media file contents: extracting text from PDFs and other documents in the media library.

License

GPL-2.0-or-later.