Ozi AI Crawler — WordPress Plugin

Auto-crawl multi-source → AI rewrite (JSON-strict) → drip publish. A production-grade WordPress plugin for content acquisition pipelines, built with PHP 8.1+, Action Scheduler, and a hardened security model.

Features

Crawl & Extract

• Multi-source ingestion: RSS, sitemap, listing pages, JSON feeds
• Smart extraction: Symfony DOM Crawler + CSS selectors with Extraction Lab UI for visual rule building
• HTTP hardening: SSRF defense with DNS resolve-and-pin (defeats DNS rebinding), private IP blocklist, redirect re-validation
• Proxy pool with rotation, scoring, and circuit breaker

AI Rewrite Pipeline

• JSON-strict outputs validated via opis/json-schema (no free-form drift)
• Multi-provider support: OpenAI-compatible, Google Gemini (with responseSchema + systemInstruction), Anthropic
• Key pool with weighted random routing (§10.2), state machine (active/cooling/disabled), and optimistic CAS locking to prevent thundering herd
• Cascade rate limiting across 5 scopes: system → source → provider → model → key
• Retry with strict-reminder: re-prompt model on schema validation failure

4-Layer Deduplication

1. L1 — URL canonical (post-normalization)
2. L2 — Pre-rewrite content (Simhash of raw extracted text)
3. L3 — Post-rewrite content (Simhash + Hamming distance)
4. L4 — Image content (EXIF-stripped SHA-256)

Media Pipeline

• WebP conversion with quality/dimension controls
• EXIF-stable hashing (Imagick stripImage before SHA-256)
• Configurable disable of WordPress intermediate sizes / -scaled big-image threshold
• Inline image URL replacement during publish

Publish

• Drip scheduling with per-source rate caps
• Idempotent (skip if already published)
• Status tracking: draft → scheduled → published → failed

Security

• AES-256-GCM encryption for API keys, proxy creds (random IV per encryption, 16-byte tag)
• Encryption kill-switch: 503 on REST + admin notice when key is missing — never falls back to plaintext
• Capability gating (ozicrawler_manage) + nonce verification on all writes
• Audit log for security-relevant actions
• Secret stripping in DB logger

Operations

• Action Scheduler (highest-version-wins pattern, bundled)
• DI container for clean service wiring (28 services, full boot chain tested)
• Health checks + maintenance UI (clear queue, reset state, regenerate keys)
• Strauss vendor scoping (OziCrawler\Vendor\*) — no namespace conflicts with other plugins

Installation

From release zip (recommended for production)

1. Download ozicrawler-x.y.z.zip from Releases
2. WordPress admin → Plugins → Add New → Upload Plugin
3. Activate

From source (development)

git clone https://github.com/OziNetworkVN/ozi-ai-crawler-wp.git ozicrawler
cd ozicrawler
composer install

Symlink or copy the folder into wp-content/plugins/, then activate via the WP admin.

Quick start

1. Settings → API Keys: add at least one provider key (OpenAI / Gemini / Anthropic)
2. Settings → Model Profiles: configure the model to use for rewrite
3. Settings → Prompt Profiles: pick a built-in profile or create a custom one (JSON schema enforced)
4. Sources: add a source URL — pick crawler type, set extraction rules in the Lab
5. Queue: jobs auto-flow via Action Scheduler — monitor in real-time
6. Drafts: review & approve generated drafts before publish
7. Publishing: configure drip rate, then enable auto-publish

Architecture

┌─────────────┐   ┌──────────────┐   ┌──────────────┐   ┌────────────┐
│   Source    │──▶│   Extract    │──▶│   Rewrite    │──▶│  Publish   │
│  (crawl)    │   │  (DOM+CSS)   │   │  (AI+JSON)   │   │  (drip)    │
└─────────────┘   └──────────────┘   └──────────────┘   └────────────┘
       │                  │                  │                 │
       ▼                  ▼                  ▼                 ▼
   L1 dedupe         L2 dedupe         L3 dedupe         L4 dedupe
   (URL canon)       (pre-Simhash)     (post-Simhash)    (image hash)

All inter-stage transitions are jobs in Action Scheduler, making the entire pipeline crash-safe and resumable.

Key directories

See docs/ for the full specification (27 documents).

Development

Requirements

• PHP 8.1+ with ext-imagick (recommended), ext-curl, ext-mbstring
• Composer 2.x
• WordPress 6.2+ for runtime testing
• MySQL 5.7+ / MariaDB 10.3+

Setup

composer install

Tests

vendor/bin/phpunit                    # 102 unit + integration tests
vendor/bin/phpstan analyse            # level 5 static analysis

Build release zip

bash bin/build-release.sh
# Produces: build/ozicrawler-1.0.0.zip

The build pipeline:

1. Installs prod dependencies (composer install --no-dev)
2. Runs Strauss to scope vendor namespaces → OziCrawler\Vendor\*
3. Copies Action Scheduler separately (excluded from prefixing per spec §23.8.4)
4. Dumps optimized classmap autoloader
5. Zips the plugin folder

Why two vendor folders?

• vendor/ — Composer dependencies in original namespaces. Used for dev & tests (IDE autocomplete works normally).
• vendor-scoped/ — Same packages with OziCrawler\Vendor\* namespace prefix. Used in the release zip to avoid conflicts with other WordPress plugins that may ship different versions of Symfony, Opis, etc.

Both are git-ignored and regenerated by composer install + bin/build-release.sh.

Documentation

The docs/ folder contains the full design spec:

• 01-overview.md — Goals & non-goals
• 04-database-schema.md — Full DB schema
• 08-rewrite-pipeline.md — AI rewrite flow §8.5
• 10-key-pool-routing.md — Key pool & rate limiting §10
• 22-extensibility-hooks.md — Actions & filters for theme/plugin integration
• ...and 22 more

Security

If you discover a security issue, please email the maintainer directly rather than opening a public issue. SSRF, encryption, and rate-limit related findings are particularly appreciated.

License

GPL-2.0-or-later. See LICENSE.

Credits

Built by the OziCrawler team. Uses:

• Action Scheduler by WooCommerce
• Symfony DOM Crawler & CSS Selector
• Opis JSON Schema
• Strauss for vendor scoping