WP RAG FAQ

An AI-powered documentation chatbot for WordPress using Retrieval-Augmented Generation (RAG). Upload your docs once — visitors get accurate answers from your content only. No generic ChatGPT responses. No hallucination.

---

How It Works

Admin uploads PDF / TXT / MD
         │
         ▼
Text extracted → split into 512-token chunks (50-token overlap)
         │
         ▼
Each chunk → OpenAI Embeddings API → 1536-float vector
         │
         ▼
Vectors stored in WordPress MySQL (no external vector DB needed)
         │
         ─────────────── visitor asks a question ───────────────
         │
         ▼
Question embedded → cosine similarity against all stored vectors
         │
         ▼
Top 4 matching chunks retrieved
         │
         ▼
GPT-4o-mini answers using ONLY those chunks as context
         │
         ▼
Answer + source attribution shown in floating chat widget

---

Features

• Floating chat widget — appears on all frontend pages, fully customizable color and title
• Upload PDF, TXT, or Markdown files via drag & drop or click
• Paste content directly — no file needed
• Two-phase indexing — chunking is instant; embedding shows a live progress bar
• Source attribution — every answer shows which document it came from
• Zero external dependencies beyond OpenAI — no Pinecone, no Redis, no external vector DB
• Client owns API costs — plugin uses the site owner's own OpenAI API key
• Works on any WordPress host — stores everything in MySQL

---

Architecture Decisions

Why no vector database?

Most RAG tutorials use Pinecone or Weaviate. For a WordPress plugin that needs to run on shared hosting, that's a hard dependency most clients can't support. We store embeddings as JSON in MySQL and compute cosine similarity in PHP.

For the typical WordPress site (hundreds to a few thousand document chunks), MySQL is fast enough and removes the external service requirement entirely.

Chunking strategy

Setting	Value	Reason
Chunk size	512 tokens (~2048 chars)	Balances context preservation vs retrieval precision
Overlap	50 tokens (~200 chars)	Prevents answers being cut off at chunk boundaries
Break point	Sentence boundary (.)	Avoids splitting mid-sentence when possible

Two-phase upload

Uploading large documents made single PHP requests time out. The solution splits the process:

1. Phase 1 (instant) — extract text, chunk, save raw text to DB
2. Phase 2 (batched) — JS calls an embed endpoint 5 chunks at a time, shows progress bar

Each batch completes in ~5–10 seconds, well within any PHP timeout.

---

Installation

1. Download or clone this repository
2. Upload the wp-rag-faq folder to /wp-content/plugins/
3. Activate via Plugins → Installed Plugins
4. Go to RAG FAQ in the WordPress admin sidebar
5. Enter your OpenAI API key and save settings
6. Upload a document or paste content
7. Wait for the embedding progress bar to complete
8. The chat widget is now live on your site

---

Configuration

Setting	Description	Default
OpenAI API Key	Your sk-... key — stored in WP database, never exposed	—
Widget Title	Heading shown in the chat panel	Documentation Assistant
Placeholder Text	Input hint text	Ask me anything about our docs...
Widget Color	Brand color for the button and header	#2563eb
Enable Widget	Toggle chat widget on all frontend pages	On

---

File Structure

wp-rag-faq/
├── wp-rag-faq.php              # Plugin bootstrap, hooks, constants
├── README.md
├── readme.txt                  # WordPress.org format readme
├── includes/
│   ├── class-database.php      # MySQL table management, CRUD
│   ├── class-chunker.php       # Text chunking (512 tokens, 50 overlap)
│   ├── class-pdf-parser.php    # Pure-PHP PDF text extractor
│   ├── class-embeddings.php    # OpenAI embed + cosine similarity + GPT-4o-mini
│   ├── class-chat.php          # Frontend AJAX handler for visitor questions
│   └── class-admin.php         # Admin AJAX handlers (upload, paste, embed, delete)
├── admin/
│   ├── admin-page.php          # Admin UI (settings, upload, indexed sources table)
│   └── admin.js                # Upload flow, progress bar, drag & drop
└── assets/
    ├── chat-widget.css          # Floating widget styles (responsive, CSS-variable themed)
    └── chat-widget.js           # Chat UI — typing indicator, source attribution

---

Database Schema

Table: wp_rag_faq_chunks

Column	Type	Description
id	BIGINT	Auto-increment primary key
source_name	VARCHAR(255)	Original filename or paste title
chunk_index	INT	Position of chunk within the source
chunk_text	LONGTEXT	Raw text of this chunk
embedding	LONGTEXT	JSON array of 1536 floats (OpenAI vector)
embedded	TINYINT	0 = pending, 1 = embedded and searchable
created_at	DATETIME	Timestamp

---

API Usage

This plugin makes two types of OpenAI API calls:

Call	Model	When	Approx Cost
Embedding	text-embedding-3-small	On upload (per chunk)	~$0.02 / 1M tokens
Chat completion	gpt-4o-mini	On every visitor question	~$0.15 / 1M input tokens

All API calls originate from the WordPress server using the site owner's API key.

---

PDF Support

The plugin includes a pure-PHP PDF parser (class-pdf-parser.php) that handles:

• FlateDecode (zlib) compressed content streams
• Literal strings (Hello World)
• Hex strings <48656c6c6f>
• TJ arrays [(Hello) -250 (World)]
• Tj / T* / Td / TD / Tm operators
• Brute-force text scan fallback

Note: Scanned PDFs (image-only) cannot be parsed by any PHP-based parser. For scanned documents, use the Paste Content feature to manually add the text.

---

Requirements

• WordPress 6.0+
• PHP 8.0+
• MySQL 5.7+ / MariaDB 10.3+
• OpenAI API key
• PHP zlib extension (for PDF decompression — enabled on virtually all hosts)

---

Contributing

Pull requests welcome. For major changes, open an issue first.

---

Author

Bilal Mahmood — bilalmahmood.dev

Built at Computan — combining deep WordPress expertise with practical AI integration.

---

License

GPL-2.0+ — see LICENSE