Newspack Nodes

A node-graph runtime for composable services, built as WordPress plugin infrastructure. Nodes pass messages and sink into one another; underneath them the runtime is WordPress — settings and the active topology set in the options table, the cold-start safety net on WP-Cron, worker spawn and commands over the REST API, command sessions and SSE slot leases in the shared cache tier (memcached, else APCu). Worker liveness and reader position are the exception: they live on the runtime's own file tree, in lock dirs and durable logs. It is WordPress-internal, not a standalone PHP bus.

Why

The traditional WordPress plugin shape — singletons, hooks-as-coupling, monolithic worker classes — makes composition hard. Each plugin grows its own private bus, its own private worker lifecycle, its own private read/write paths. Sharing pieces between plugins means cut-paste-modify, not Lego.

Newspack Nodes is a different bet. The substrate gives you one contract: every node receives messages via fill( array $message ), and every node sinks into another node. That uniformity is what makes composition work — any node connects to any other node, fan-out is a Tee, transforms are Hooks, file I/O is a Tail or Log. New behavior is a new Node class with a new fill() body.

The runtime is independent of any application — but not of WordPress. It owns the substrate (Node, Message, Router, Topic, Partition, Worker, Fleet, Job_Worker, REPL) and ships nothing application-specific. The four stock topologies are topologies/job-worker.tsl, which drives the generic Job_Worker_Node (its application context arriving through the before_job filter and the after_job action); topologies/job-intake.tsl, which drains the large-write job ingress on substrate-only installs; topologies/settings-sync.tsl, the single-instance settings-sync control plane; and topologies/topic-probe.tsl, the per-worker consumer-stats sweep. But every part of the lifecycle underneath belongs to WordPress, so "application-independent" is the honest claim and "standalone runtime" is not. The event logger, newspack-event-logger-nodes, adds ten node classes of its own: seven that carry its pipelines and three service interpreters.

This is the Lego-bricks architecture in PHP and WordPress, on a platform with no resident daemon: a worker is an HTTP request that outlives its caller and hands its slot to a successor at ~595 seconds, under WordPress.com Atomic's 15-minute request cap.

The parts nothing else ships

Job queues exist. These don't, anywhere else in WordPress:

• A live topology console. A graph editor over the running fleet: see every node and edge with live message counts, rewire a graph, save it as a .tsl — from the browser. The stock files stay immutable, so an edit saves under a new name that includes the stock one.
• An attached REPL. wp nodes cli <worker>.p0 pivots into a live worker over IPC: inspect with dump_node, trace, and stats, rewire sinks, send test messages — no restart, no redeploy.
• Time-travel debugging. Readers checkpoint durable cursors, so a Consumer can pause, single-step, and seek back through the log's history while you watch downstream react.
• A Jobs dashboard. Runs and errors per handler, duration and queue latency per job identity, and the jobs Topic's backlog — replayed 24 hours deep from the durable jobstats log the workers already write.
• Errors as docs. A refusal names the argument or the call that satisfies it — Consumer source partition not initialized; call arguments() first — and help <NodeType> in the REPL renders any node's schema, arguments, and verbs from the class itself.
• An infra-free test suite. 8,600+ tests across PHP and JavaScript, with no containers, no database, no memcached server and no WordPress install.
• Written-down architecture. Twenty ADRs, each carrying the alternatives it rejected and the condition that would reopen it (architecture-decisions.md).

When NOT to use Nodes

Nodes is a runtime, and a runtime you don't need is overhead. Reach for the incumbent when it already fits:

• One background job, now and then — Action Scheduler is one call, probably already installed, and runs anywhere WordPress does. Don't install a node-graph runtime to send a welcome email.
• A scheduled task that tolerates drift — wp_schedule_event() is free. WP-Cron's known weakness (it fires on traffic, so quiet sites drift) is only worth solving when it is your problem.
• Request-scope glue — actions and filters compose fine at request scale; that's what they're for.

Nodes earns its keep when the shape of the problem is a pipeline: durable ordered logs you can replay, long-lived workers that hold state between messages, graphs you rewire in a topology file instead of code, and a REPL/dashboard view into all of it. The event logger — a firehose that fans out into routing and aggregation — is the native case.

And one honest middle case: newspack-cache-cozy uses Nodes for one Timer node that enqueues one job per interval — incumbent-shaped work — because the substrate is already installed for the event logger, and its warm render needs a cadence WP-Cron cannot hold: a minute event competing for a slot with the reconcile pass and every other scheduled task. The marginal cost is near zero and it solves one real weakness. That's the test: if Nodes is already there, a one-node use is fine; if it isn't, don't add a runtime for one job.

Learn it

Start with getting-started.md — the bundled example pipeline, running in about five minutes — then work through the docs/ set (mapped by reading order in docs/README.md):

• getting-started.md — zero to a running pipeline you can poke at by hand.
• writing-a-plugin.md — build the AI-newsletter example from an empty directory, one node at a time.
• writing-a-dashboard.md — add a React admin dashboard that reads the pipeline's live state.
• writing-a-real-plugin.md — take that toy to the production version, two method bodies away.
• writing-a-real-dashboard.md — the production realities of shipping a dashboard (console, DevTools overlay, release).
• writing-a-view-node.md — the one-page contract for a dashboard slice's terminal view node.
• architecture-guide.md — full substrate design: message format, node contracts, drain loop, REPL.
• architecture-decisions.md — the load-bearing ADRs and the conditions that would reopen them.
• API.md — the REST endpoints and their envelopes, command signing, the two SSE streams, and every newspack_nodes/* hook.
• cli.md — every wp nodes subcommand and the flows they combine into.
• troubleshooting.md — the REPL, worker health, log paths, and the failure modes we actually hit.
• sse-host-budget.md — what one SSE stream costs in php-fpm children, and what happens when they run out.
• stability.md — the frozen surfaces, what changing one costs, and what stays internal; no name is aliased.
• upgrading.md — each breaking change with its fix, for moving a consumer across substrate versions.
• tachikoma-lineage.md — the Perl this runtime varies from, file and symbol, and why each divergence was chosen.

The complete code lives in examples/example-ai-newsletter/.

Quick Start

You need PHP 8.2 or newer, WordPress 6.5 or newer, and a cache backend — Memcached, or APCu on a single web host. Workers spawn over the REST API, so the runtime itself needs no WP-CLI — the verbs below and the REPL do. Install as a standard WordPress plugin, then:

# Activate (no app — just the runtime).
wp plugin activate newspack-nodes

# Eight checks before wiring anything: cache backend, filesystem, ownership,
# the housekeeping cron, config keys, worker liveness, consumer lag and dead
# letters. Each miss names its degradation; a critical one exits non-zero.
wp nodes doctor

# List active workers (none until a topology is activated).
wp nodes status

# Open the bare REPL (local nodes only).
wp nodes cli

To get workers running, install an application plugin that registers a topology — one call, Topology_Registry::register_plugin( 'My_Namespace\\', __DIR__ . '/topologies' ) — then activate it with wp nodes activate <topology>. Registration only makes the .tsl discoverable and the plugin's *_Node classes resolvable to make_node; the active set is the topologies config key, which defaults to empty, so nothing spawns until a name lands in it. wp nodes scaffold plugin my-pipeline writes that plugin's starter files in the shapes writing-a-plugin.md teaches, and never overwrites an existing one. The bundled examples/example-ai-newsletter/ is the smallest complete example; newspack-event-logger-nodes is the production one.