WP Manifestindependent plugin directory
manifest / unclassified / tokenizer-engine-for-wp

Tokenizer Engine for WP

Search, related content and internal links, with Elasticsearch-like text analysis and no search server.

by lalbert · github.com/lalbert/tokenizer-engine-for-wp · website

★ 0stars
6composer installs

Install

Download source zip No release zip yet

This is the repository archive. It installs, but the folder name will carry the branch suffix and updates will not flow.

Install straight from GitHub with WP-CLI:

wp plugin install https://github.com/lalbert/tokenizer-engine-for-wp/archive/refs/heads/main.zip

On Packagist as lalbert/tokenizer-engine-for-wp:

composer require lalbert/tokenizer-engine-for-wp
No built-in updater. New versions won't appear under Dashboard → Updates. Install new releases yourself, or let Git Updater track the repository.
Readme github.com/lalbert/tokenizer-engine-for-wp ↗

Search, related content and internal links for WordPress, with the text analysis of lalbert/tokenizer-engine (stemming, elisions, stop words per language, Elasticsearch-compatible analyzers). No search server needed.

Status: 0.3.

Français : README.fr.md (translation; this English documentation is the reference).

What it does

  • Index the post types and taxonomies you choose, with the fields you choose (title, content, excerpt, taxonomies, custom fields) and a weight each. Shortcodes, blocks and synced patterns are cleaned so that the index holds what visitors read.
  • Related content between any post types and taxonomies ("more like this"), stable over time, with pinned and excluded items in the editor: block "Related content", shortcode [tokenizer_engine_related], PHP and REST APIs.
  • Search ranked with BM25 and highlighted, used for the search of the site (option) or through the APIs.
  • Internal link suggestions in the block editor: passages mentioning the title of another post or term, linked in one click; suggested terms (categories, tags...) for a post.
  • Automatic internal links (option): in the stored content (API, WP-CLI, with a dry run) or at display time, with guardrails (maximum per content, one link per target, no link in headings, exclusions) and the marker data-te-link="auto".
  • Suggestions while typing in the search fields of the site (option).
  • A public API for developers: PHP, REST, WP-CLI and hooks.

Installation

With Composer, in a project that manages WordPress with Composer (Bedrock or similar): the package has the type wordpress-plugin, so composer/installers puts it in the plugins directory.

composer require lalbert/tokenizer-engine-for-wp

Allow the plugin of composer/installers if Composer asks (config.allow-plugins).

Without Composer for WordPress, clone the plugin and install its dependencies:

cd wp-content/plugins && git clone https://github.com/lalbert/tokenizer-engine-for-wp.git && cd tokenizer-engine-for-wp && composer install --no-dev

Then activate it, go to Settings > Tokenizer Engine, enable the post types and taxonomies to index, and save: the index is built in the background (WP-Cron), or now with wp tokenizer-engine-for-wp index rebuild.

WP-Cron runs on page views: on a site with little traffic, run it from a real cron job (wp cron event run --due-now) so that the index and related content stay up to date.

Sizes

Related content is recomputed for every indexed object after changes (grouped) and every night: this suits up to some thousands of objects. See the measures of the SQL index in the README of lalbert/tokenizer-engine ("When to use it") for the search.

Requirements: WordPress 6.5+, PHP 8.2+, MySQL or MariaDB. License: GPL-2.0-or-later.