AI Taxonomy Audit

WP-CLI tool for human-in-the-loop taxonomy enrichment using LLM (Ollama, OpenAI, or OpenRouter).

Classifies WordPress posts against controlled vocabularies and generates WP-CLI commands for applying suggested terms. All suggestions require human review before application.

Features

• Three provider options: Ollama (local), OpenAI, or OpenRouter (access to 100+ models)
• Two classification modes: Benchmark (vocabulary-only) or Audit (benchmark + gap-filling suggestions)
• Three analysis dimensions: Validate existing terms, suggest additions, and identify vocabulary gaps
• Two-step classification: Context-aware conversation for higher accuracy
• Retry logic: Automatically corrects invalid term suggestions
• Confidence scoring: Filter results by confidence threshold
• CSV workflow: Export, review in spreadsheets, then apply approved changes
• Comparison view: See existing terms vs suggestions side-by-side with status indicators
• Gap analysis: Identify taxonomy health issues, unused terms, and coverage gaps
• Audit mode: Suggest new taxonomy terms that should exist (gap-filling)
• Pruning tools: Safely remove unused taxonomy terms with generated scripts
• Stratified sampling: Sample across date ranges and categories for representative analysis
• Provider comparison: Compare results between Ollama, OpenAI, and OpenRouter
• Run storage: Track analysis runs with metadata, compare runs over time
• Cost tracking: Estimate costs before running, track actual usage, compare provider pricing

Three Dimensions of Taxonomy Audit

When auditing taxonomy health, there are three distinct questions to answer:

Dimension 1: Are Existing Terms Correct?

"This post has 'climate-risk' applied — is that accurate?"

The LLM reviews the content and either confirms the existing term is appropriate or suggests it may not belong. In the CSV output, this is indicated by the status column:

• KEEP — Term is already applied AND the LLM agrees it should be

Dimension 2: What Terms Should Be Added?

"This post should have 'decarbonisation' but doesn't"

The LLM suggests terms from your existing vocabulary that should be applied but currently aren't. This is the primary output of benchmark mode:

• ADD — Term exists in vocabulary, LLM suggests it, but not currently applied

Dimension 3: What Vocabulary Gaps Exist?

"There should be a term called 'carbon-accounting' but it doesn't exist"

The LLM identifies concepts in the content that have no matching term in your vocabulary. This requires audit mode (--audit flag):

• NEW — Term doesn't exist in vocabulary (suggested for creation)

Summary: Modes and Dimensions

Mode	Dimension 1 (Confirm)	Dimension 2 (Add)	Dimension 3 (New)
Benchmark (default)	✅	✅	❌
Audit (--audit)	✅	✅	✅

Recommended Workflow

📖 See PLAYBOOK.md for the complete step-by-step guide.

Quick overview:

1. Run benchmark mode first → assess vocabulary coverage
2. Review CSV → check KEEP vs ADD status
3. Run audit mode → discover vocabulary gaps (NEW terms)
4. Run gap-analysis → comprehensive health report
5. Apply approved suggestions

Requirements

• PHP 8.0+
• WordPress 6.0+
• WP-CLI
• One of:
  • Ollama (local, free, private) — ollama.ai
  • OpenAI API key (cloud, paid, higher quality)
  • OpenRouter API key (access to many models) — openrouter.ai

Installation

1. Clone or copy the plugin to wp-content/plugins/ai-taxonomy-audit/
2. Run composer install in the plugin directory
3. Activate the plugin (optional — only needed for admin features)

Configuration

Add to wp-config.php:

// For OpenAI (required if using --provider=openai)
define( 'OPENAI_API_KEY', 'sk-...' );

// For OpenRouter (required if using --provider=openrouter)
define( 'OPENROUTER_API_KEY', 'sk-or-...' );

// Optional: Override default models
define( 'DGW_OPENAI_MODEL', 'gpt-4o-mini' );              // Default: gpt-4o-mini
define( 'DGW_OLLAMA_MODEL', 'qwen2.5:latest' );               // Default: qwen2.5:latest
define( 'DGW_OPENROUTER_MODEL', 'google/gemma-2-9b-it:free' ); // Default: gemma-2-9b-it:free

// Optional: Ollama server location
define( 'DGW_OLLAMA_BASE_URI', 'http://localhost:11434' );

Quick Start

# Check provider status
wp taxonomy-audit status

# Classify 10 posts using OpenAI
wp taxonomy-audit classify --provider=openai --limit=10

# Classify using local Ollama
wp taxonomy-audit classify --provider=ollama --limit=10

# Classify using OpenRouter (access to many models)
wp taxonomy-audit classify --provider=openrouter --model=deepseek/deepseek-chat --limit=10
wp taxonomy-audit classify --provider=ollama --model=gemma3:27b --limit=5 --format=csv

# Review the generated CSV, then apply approved suggestions
wp taxonomy-audit apply --file=output/suggestions-2024-01-28-120000.csv --approved-only

Commands

classify

Classify posts against taxonomy vocabularies.

wp taxonomy-audit classify [options]

Options:

Option	Default	Description
--post_type=<type>	post	Post type to classify
--post-ids=<ids>	—	Comma-separated post IDs
--limit=<n>	10	Maximum posts to process
--taxonomies=<list>	category,post_tag	Taxonomies to classify against
--provider=<name>	ollama	LLM provider: ollama, openai, or openrouter
--model=<name>	varies	Model to use
--format=<fmt>	csv	Output: csv, json, or terminal
--prefix=<cmd>	ddev wp	WP-CLI prefix for generated commands
--min-confidence=<n>	0.7	Minimum confidence threshold (0-1)
--audit	—	Enable audit mode: suggest new terms that should exist
--single-step	—	Use single API call instead of two-step conversation
--dry-run	—	Preview posts and estimate costs without calling LLM
--unclassified-only	—	Only process posts without terms
--sampling=<strategy>	sequential	Sampling strategy: sequential or stratified
--save-run	—	Save results to structured run for historical tracking
--run-notes=<notes>	—	Notes to attach to the run (requires --save-run)
--skos-context=<file>	—	Path to SKOS Turtle file for hierarchical vocabulary context

Examples:

# Classify with OpenAI (better results)
wp taxonomy-audit classify --provider=openai --model=gpt-4o-mini --limit=20

# Classify with OpenRouter (access to DeepSeek, Llama, Mistral, etc.)
wp taxonomy-audit classify --provider=openrouter --model=deepseek/deepseek-chat --limit=20

# Use single-step mode (faster, less accurate)
wp taxonomy-audit classify --provider=openai --single-step --limit=10

# Classify specific posts
wp taxonomy-audit classify --post-ids=123,456,789

# Classify against custom taxonomies
wp taxonomy-audit classify --taxonomies=topic,region,document_type

# Preview posts and estimate costs before running
wp taxonomy-audit classify --provider=openai --limit=50 --dry-run

# Output directly to terminal (copyable commands)
wp taxonomy-audit classify --format=terminal --limit=5

# Use stratified sampling (across dates and categories)
wp taxonomy-audit classify --limit=20 --sampling=stratified

# Save results to a structured run for historical tracking
wp taxonomy-audit classify --limit=100 --provider=openai --save-run --run-notes="Initial baseline"

# Enable audit mode to discover vocabulary gaps (suggests new terms)
wp taxonomy-audit classify --audit --provider=openai --limit=20 --format=csv

# Use SKOS context for hierarchical vocabulary (requires wp-to-file-graph)
wp taxonomy-audit classify --skos-context=vocab/category.skos.ttl --taxonomies=category --limit=20

SKOS Context:

When you provide a SKOS Turtle file via --skos-context, the LLM receives hierarchical vocabulary information:

• Broader/narrower relationships — helps LLM understand term specificity
• SKOS definitions — richer context than WordPress term descriptions
• Hierarchical prompt formatting — terms displayed as a tree, encouraging specific term selection

Generate SKOS files using wp-to-file-graph:

# Export taxonomy as SKOS
wp wptofile-graph skos category --output=vocab/category.skos.ttl

# Use in classification
wp taxonomy-audit classify \
    --taxonomies=category \
    --skos-context=vocab/category.skos.ttl \
    --provider=openai \
    --limit=20

Audit Mode:

In audit mode (--audit), the LLM will:

1. Classify content against your existing vocabulary (benchmark)
2. Suggest new terms that should exist but don't (gap-filling)

Output includes an in_vocabulary column:

• TRUE — term exists in your vocabulary
• FALSE — suggested new term (requires manual creation before applying)

export-vocab

Export taxonomy vocabulary for review.

wp taxonomy-audit export-vocab [options]

Options:

Option	Default	Description
--taxonomies=<list>	category,post_tag	Taxonomies to export
--format=<fmt>	json	Output: json or table
--file=<path>	—	Save to file

Examples:

# View vocabulary as table
wp taxonomy-audit export-vocab --format=table

# Export to JSON file
wp taxonomy-audit export-vocab --taxonomies=topic,category --file=vocabulary.json

apply

Apply taxonomy suggestions from reviewed CSV or JSON file.

wp taxonomy-audit apply --file=<path> [options]

Options:

Option	Default	Description
--file=<path>	required	CSV or JSON file path
--approved-only	—	Only apply rows marked approved
--dry-run	—	Preview without applying

Examples:

# Apply all suggestions
wp taxonomy-audit apply --file=suggestions.csv

# Apply only approved rows
wp taxonomy-audit apply --file=suggestions.csv --approved-only

# Preview what would be applied
wp taxonomy-audit apply --file=suggestions.csv --dry-run

generate-script

Generate shell script from suggestions file.

wp taxonomy-audit generate-script --file=<path> [options]

Options:

Option	Default	Description
--file=<path>	required	CSV or JSON file path
--output=<path>	—	Output script path
--prefix=<cmd>	ddev wp	WP-CLI command prefix
--approved-only	—	Only include approved rows

Examples:

# Generate script to stdout
wp taxonomy-audit generate-script --file=suggestions.csv

# Save to file
wp taxonomy-audit generate-script --file=suggestions.csv --output=apply-terms.sh

# Use different WP-CLI prefix
wp taxonomy-audit generate-script --file=suggestions.csv --prefix="lando wp"

status

Check LLM provider status and configuration.

wp taxonomy-audit status

stats

Show classification statistics for a post type.

wp taxonomy-audit stats [options]

Options:

Option	Default	Description
--post_type=<type>	post	Post type to analyze
--taxonomies=<list>	category,post_tag	Taxonomies to check

list

List saved suggestion files.

wp taxonomy-audit list [--format=<fmt>]

gap-analysis

Analyze taxonomy gaps between suggestions and vocabulary.

wp taxonomy-audit gap-analysis --suggestions=<path> [options]

Options:

Option	Default	Description
--suggestions=<path>	required	Path to suggestions JSON file
--taxonomies=<list>	category,post_tag	Taxonomies to analyze
--format=<fmt>	table	Output: table or json
--output=<path>	—	Save JSON report to file
--save-run=<run-id>	—	Add gap analysis to an existing run

Output includes:

• Suggested new terms: Terms the LLM suggested that don't exist in vocabulary
• Unused existing terms: Terms in vocabulary that were never suggested
• Ambiguous terms: Terms with low average confidence (may need clarification)
• Uncovered content: Posts without adequate taxonomy suggestions
• Health score: Overall taxonomy fitness (0-100)

Examples:

# Run gap analysis with table output
wp taxonomy-audit gap-analysis --suggestions=output/suggestions-2024-01-28.json

# Save report as JSON
wp taxonomy-audit gap-analysis --suggestions=output/suggestions.json --output=gap-report.json

# Analyze specific taxonomies
wp taxonomy-audit gap-analysis --suggestions=output/suggestions.json --taxonomies=topic,region

# Add gap analysis to an existing run
wp taxonomy-audit gap-analysis --suggestions=output/runs/2025-01-29T103000/suggestions.json --save-run=2025-01-29T103000

unused-terms

Find taxonomy terms with zero posts.

wp taxonomy-audit unused-terms [options]

Options:

Option	Default	Description
--taxonomies=<list>	category,post_tag	Taxonomies to check
--format=<fmt>	table	Output: table, json, or csv

Examples:

# Find unused terms
wp taxonomy-audit unused-terms

# Check specific taxonomies
wp taxonomy-audit unused-terms --taxonomies=category,topic

# Export as JSON
wp taxonomy-audit unused-terms --format=json