WP Manifestindependent plugin directory
manifest / ai / ozi-ai-crawler-wp

OziCrawlerWP

Ozi AI Crawler — WordPress plugin for auto-crawl multi-source → AI rewrite (JSON-strict) → drip publish. PHP 8.1+, Action Scheduler, hardened security.

by OziCrawler Team · github.com/ozinetworkvn/ozi-ai-crawler-wp · website

★ 1stars
0forks

Install

No release zip yet. The repository archive installs, but the folder name will carry the branch suffix and updates will not flow:

wp plugin install https://github.com/ozinetworkvn/ozi-ai-crawler-wp/archive/refs/heads/main.zip
Readme github.com/ozinetworkvn/ozi-ai-crawler-wp ↗

Ozi AI Crawler — WordPress Plugin

Auto-crawl multi-source → AI rewrite (JSON-strict) → drip publish. A production-grade WordPress plugin for content acquisition pipelines, built with PHP 8.1+, Action Scheduler, and a hardened security model.

Features

Crawl & Extract

  • Multi-source ingestion: RSS, sitemap, listing pages, JSON feeds
  • Smart extraction: Symfony DOM Crawler + CSS selectors with Extraction Lab UI for visual rule building
  • HTTP hardening: SSRF defense with DNS resolve-and-pin (defeats DNS rebinding), private IP blocklist, redirect re-validation
  • Proxy pool with rotation, scoring, and circuit breaker

AI Rewrite Pipeline

  • JSON-strict outputs validated via opis/json-schema (no free-form drift)
  • Multi-provider support: OpenAI-compatible, Google Gemini (with responseSchema + systemInstruction), Anthropic
  • Key pool with weighted random routing (§10.2), state machine (active/cooling/disabled), and optimistic CAS locking to prevent thundering herd
  • Cascade rate limiting across 5 scopes: system → source → provider → model → key
  • Retry with strict-reminder: re-prompt model on schema validation failure

4-Layer Deduplication

  1. L1 — URL canonical (post-normalization)
  2. L2 — Pre-rewrite content (Simhash of raw extracted text)
  3. L3 — Post-rewrite content (Simhash + Hamming distance)
  4. L4 — Image content (EXIF-stripped SHA-256)

Media Pipeline

  • WebP conversion with quality/dimension controls
  • EXIF-stable hashing (Imagick stripImage before SHA-256)
  • Configurable disable of WordPress intermediate sizes / -scaled big-image threshold
  • Inline image URL replacement during publish

Publish

  • Drip scheduling with per-source rate caps
  • Idempotent (skip if already published)
  • Status tracking: draft → scheduled → published → failed

Security

  • AES-256-GCM encryption for API keys, proxy creds (random IV per encryption, 16-byte tag)
  • Encryption kill-switch: 503 on REST + admin notice when key is missing — never falls back to plaintext
  • Capability gating (ozicrawler_manage) + nonce verification on all writes
  • Audit log for security-relevant actions
  • Secret stripping in DB logger

Operations

  • Action Scheduler (highest-version-wins pattern, bundled)
  • DI container for clean service wiring (28 services, full boot chain tested)
  • Health checks + maintenance UI (clear queue, reset state, regenerate keys)
  • Strauss vendor scoping (OziCrawler\Vendor\*) — no namespace conflicts with other plugins

Installation

From release zip (recommended for production)

  1. Download ozicrawler-x.y.z.zip from Releases
  2. WordPress admin → Plugins → Add New → Upload Plugin
  3. Activate

From source (development)

git clone https://github.com/OziNetworkVN/ozi-ai-crawler-wp.git ozicrawler
cd ozicrawler
composer install

Symlink or copy the folder into wp-content/plugins/, then activate via the WP admin.

Quick start

  1. Settings → API Keys: add at least one provider key (OpenAI / Gemini / Anthropic)
  2. Settings → Model Profiles: configure the model to use for rewrite
  3. Settings → Prompt Profiles: pick a built-in profile or create a custom one (JSON schema enforced)
  4. Sources: add a source URL — pick crawler type, set extraction rules in the Lab
  5. Queue: jobs auto-flow via Action Scheduler — monitor in real-time
  6. Drafts: review & approve generated drafts before publish
  7. Publishing: configure drip rate, then enable auto-publish

Architecture

┌─────────────┐   ┌──────────────┐   ┌──────────────┐   ┌────────────┐
│   Source    │──▶│   Extract    │──▶│   Rewrite    │──▶│  Publish   │
│  (crawl)    │   │  (DOM+CSS)   │   │  (AI+JSON)   │   │  (drip)    │
└─────────────┘   └──────────────┘   └──────────────┘   └────────────┘
       │                  │                  │                 │
       ▼                  ▼                  ▼                 ▼
   L1 dedupe         L2 dedupe         L3 dedupe         L4 dedupe
   (URL canon)       (pre-Simhash)     (post-Simhash)    (image hash)

All inter-stage transitions are jobs in Action Scheduler, making the entire pipeline crash-safe and resumable.

Key directories

Path Purpose
src/Security/ Encryption, SSRF, capabilities, nonces, sanitizers
src/Crawl/ + src/Extract/ Source fetching + DOM extraction
src/Rewrite/ AI provider abstraction, JSON validation, retry
src/Routing/ Key pool, rate limiter, weighted routing
src/Media/ WebP conversion, image hashing
src/Publish/ Drip scheduler, idempotent publisher
src/Queue/ Action Scheduler job handlers
src/Rest/ REST controllers (14 endpoints)
src/Admin/ Admin UI controllers + views (15 pages)
src/Observability/ Logger, audit, health

See docs/ for the full specification (27 documents).

Development

Requirements

  • PHP 8.1+ with ext-imagick (recommended), ext-curl, ext-mbstring
  • Composer 2.x
  • WordPress 6.2+ for runtime testing
  • MySQL 5.7+ / MariaDB 10.3+

Setup

composer install

Tests

vendor/bin/phpunit                    # 102 unit + integration tests
vendor/bin/phpstan analyse            # level 5 static analysis

Build release zip

bash bin/build-release.sh
# Produces: build/ozicrawler-1.0.0.zip

The build pipeline:

  1. Installs prod dependencies (composer install --no-dev)
  2. Runs Strauss to scope vendor namespaces → OziCrawler\Vendor\*
  3. Copies Action Scheduler separately (excluded from prefixing per spec §23.8.4)
  4. Dumps optimized classmap autoloader
  5. Zips the plugin folder

Why two vendor folders?

  • vendor/ — Composer dependencies in original namespaces. Used for dev & tests (IDE autocomplete works normally).
  • vendor-scoped/ — Same packages with OziCrawler\Vendor\* namespace prefix. Used in the release zip to avoid conflicts with other WordPress plugins that may ship different versions of Symfony, Opis, etc.

Both are git-ignored and regenerated by composer install + bin/build-release.sh.

Documentation

The docs/ folder contains the full design spec:

  • 01-overview.md — Goals & non-goals
  • 04-database-schema.md — Full DB schema
  • 08-rewrite-pipeline.md — AI rewrite flow §8.5
  • 10-key-pool-routing.md — Key pool & rate limiting §10
  • 22-extensibility-hooks.md — Actions & filters for theme/plugin integration
  • ...and 22 more

Security

If you discover a security issue, please email the maintainer directly rather than opening a public issue. SSRF, encryption, and rate-limit related findings are particularly appreciated.

License

GPL-2.0-or-later. See LICENSE.

Credits

Built by the OziCrawler team. Uses: