Runex

Roadmap ​

Where the architecture goes next. Symmetric to changelog.md: the changelog is how the system got here, this is the target it converges to. Every item here is additive to the invariants in architecture.md — nothing below relaxes a layer boundary or an existing assertion. The "Assertions ledger" at the end makes that auditable.

The thesis: a stable substrate, hot-pluggable channels ​

runex's core provides capability, never channels.

  • A capability is a generic, stable, core-owned I/O power: read a file, write a file, make an HTTP request, call an MCP tool. It changes on runex's schedule (almost never).
  • A channel is one concrete external system reached through capabilities: a WeChat export, an Obsidian vault, iOS Health, a Notion workspace over MCP. Its shape changes on someone else's schedule, often.

The current adapter set is already the inbound half of this. The target generalizes it both ways and moves channel definitions out of the recompiled core:

A channel break — Notion renames a field, WeChat bumps its schema — must never require touching, recompiling, or restarting the engine core. It is a hot-swappable extension, the way an Emacs package is to the C core or a mod is to a game engine.

The core's job is to be the stable plug socket. The plugs are extensions.

The decision: unify at the trigger level, not the data level ​

A tempting unification — "adapter degrades to a discover-* kernel that returns item_json, plus one generic upsert-item kernel; the elif chain disappears" — is rejected. It looks like simplification but regresses the contract:

  • It demotes the typed CanonicalItem boundary (typed fields, RefByName lazy resolution, attachments, reactive-trigger field ordering) to opaque JSON shuttled between two kernels through the DSL.
  • The DSL becomes a dumb pipe moving Python→Python data — a direct violation of architecture.md's own principle that the DSL is a description language and computation lives in kernels, and of Invariant 4 (kernels are an effect-only escape hatch, not a data-plumbing ABI).
  • It regresses manifest(): DataSourceSpec (params, watch, ontologies) becomes invisible inside a JSON blob, undoing the Phase 0–5 machine-first surface.
  • The actual win it reaches for — collapsing the CLI dispatch chain — was already delivered by the datasource registry (ds_registry.get(key)), without dissolving any types.

Chosen shape — unify at the trigger boundary, keep data typed:

  1. The registry becomes bidirectional: every channel is a typed DataSourceSpec (inbound) and/or SinkSpec (outbound, the symmetric new half), self-described by a module-level SPEC, exactly as adapters are today. Construction knowledge stays with the extension; the registry validates and orchestrates.
  2. The engine holds a registry handle the way it holds kernels. The ECA layer gains a verb that names a registered channel by key and triggers it — it does not carry the channel's data through the DSL. Ingest and writeback become first-class ECA citizens: schedulable via (trigger …), (guard …)-able, manifest()-introspectable — without the data leaving typed Python.
  3. CanonicalItem and upsert_node_from_item stay exactly as the architecture doc describes them. The cascade is still the connection.

Net: the same "no double-track" goal, with the contract intact.

Channels as extensions: API-churn isolation ​

Why a layer at all instead of more code in core: every channel is owned by a third party who will change it without warning. Isolating each behind a typed spec that lives outside the core means a channel break is a contained, hot-swappable failure, not a core regression.

  • Hot-registration. At Ontology.open(), the registry also scans ~/.runex/extensions/{sources,sinks,kernels}/*.py and registers any module exposing a SPEC (channels) or a KERNELS dict (capabilities). Drop a file in, restart your one process — no core edit, no rebuild, no redeploy. This is the runtime-extensibility promise the architecture's "Fit" section already claims, made literal for I/O.
  • Capability stdlib, demarcated from the trusted core. The core ships a small, curated set of generic dirty kernels — read-file, write-file, http-request, mcp-call — installed by the bootstrap, never by the pure-core kernel installer. A channel extension composes these plus its own parsing into a typed spec. The core never names WeChat or Notion — the same rule as Invariant 2 ("no business names in framework code"), now extended to "no channel names in core."
  • Writeback is the missing symmetric half. Inbound is discover → CanonicalItem → upsert. Outbound is an ECA action, on a Store event, invoking a registered SinkSpec by key (writeback-notion, notify-feishu). Same registry, same trigger model, same manifest() surface — not an untyped notify-* kernel grab-bag.

Trust boundary (explicit, not implicit) ​

Hot-discovery means arbitrary Python executes at open(). This is acceptable only under runex's existing single-user-local model — the extension directory is owned by the same trust principal as the .db file (the Emacs init.el model). The moment a shared or product deployment loads an extension directory it does not own, this is remote code execution.

Stated as an invariant for the ledger below: the extension directory shares the .db's trust principal; capability kernels are the sole sanctioned external-I/O surface and are demarcated from the pure trusted core. A product on top must not widen this without an isolation story.

The separation point (executable criterion) ​

The capability/channel seam is the I/O boundary, drawn as a broker-injection point — the same pattern L3 already uses to build the DSL env (Engine._make_env injects call-kernel, store primitives, and read-primitives; guards get a restricted env), lifted one layer out to where the registry builds a channel's I/O env.

  • Capability — core-owned, bootstrap-installed, trusted, few, stable. The only code that performs a syscall / socket / subprocess. Invariant 4's "sanctioned external-I/O surface", made structural rather than conventional.
  • Channel — extension, hot-loaded, churny, many. A DataSourceSpec / SinkSpec that (a) declares which capabilities it consumes, (b) supplies channel-specific params, (c) holds the pure transform raw ↔ CanonicalItem. It performs zero direct I/O and imports no socket/db/http library; it receives capability handles by injection.

The line: does this unit perform a syscall/socket/subprocess, or only shape data? Touches the OS/network → capability. Only transforms bytes ↔ CanonicalItem → channel. Path algebra (Path(...), .name, .parts) is data-shaping and stays; Path I/O verbs (.read_text, .glob, open() move to a capability. No third bucket; fetch and interpret never share a unit.

Acceptance test — the dogfood gate made executable. After a channel is split, a suite gate greps its module for open( / .read_text / .read_bytes / subprocess / sqlite3 / httpx / requests. / .glob( / .iterdir(. Any hit ⇒ the seam leaked ⇒ reject. Behaviour is preserved iff the channel emits a byte-identical CanonicalItem stream before and after the split (existing adapter tests stay green). This gate lives in the suite the way tests/test_lint_gate.py makes ruff real.

The line is the I/O boundary, NOT the typing boundary. Both sides stay typed; CanonicalItem never degrades to opaque JSON (contrast the rejected data-level unification above). Capability signatures are typed too.

Capability set is derived, never designed. Ship only capabilities an existing adapter provably exercises. The six adapters exercise exactly four: file (obsidian/claude/codex), sqlite read-only (opencode), command (wechat), http (media-snapshot — NocoDB v3). http was held back until a real consumer existed and shipped with that consumer, not before — the rule working as intended, not an exception to it. mcp stays deferred: no channel dogfoods it yet, so it does not exist.

Inbound split: complete. Landed across three commits — file + obsidian/claude (the N→1 proof + seam gate); command + sqlite + wechat/opencode; http + codex + media-snapshot. All six adapters now declare requires and perform zero direct I/O; the seam gate enforces it for every one. The inbound capability/channel boundary is done.

Outbound slice: complete (dogfood gate satisfied). SinkSpec (symmetric to DataSourceSpec, in the same lazy registry, manifest().data.sinks); a node→CanonicalItem read projection (the typed mirror of upsert_node_from_item); HttpCapability.post_json; the real NocoDB writeback sink; and the ECA (writeback "key") effect verb — the engine holds configured sinks the kernel way, so the DSL names the sink and the engine does the typed projection; the node's data never crosses the DSL. (writeback) is guard-disallowed exactly like call-kernel. tests/ test_writeback_e2e.py round-trips it: one reactive field-set → projection → sink → NocoDB POST, with the HTTP capability swapped for a recording stub at the same injection seam. The roadmap's "one real churny channel must round-trip before generalising" gate is met. The last P1 — hot-discovery of the extension dir for kernels + sources + sinks — is also landed.

Staged path ​

PriItemWhyTouches
✅ doneSinkSpec + bidirectional registry; manifest().data.sinksClose the inbound-only asymmetry without an untyped outbound pathextended "Datasource registry" → "Channel registry"; symmetric SinkSpec/SinkAdapter
✅ doneECA channel-trigger verb — (writeback "key"); engine holds configured sinks the kernel way; guard-disallowedMake writeback a schedulable & guardable ECA citizen without data crossing the DSLextended "Trigger"/"Extending the engine"; node→CanonicalItem projection is the typed seam
✅ doneCapability layer + registry injection — all six adapters split; derived set is file/sqlite/command/http (mcp still deferred — no consumer)The stable plug socket: channels declare requires, the broker injects handles, channels do zero direct I/Oextended "Datasource registry"; made Invariant 4 structural; enforced by tests/test_capability_seam.py
✅ doneHot-discovery at open() / registry build — ~/.runex/extensions/{kernels,sources,sinks}/*.py; KERNELS and channel SPECs auto-registered"Extend without editing the core" made literal for both capabilities and channelsrunex/extensions.py; adapters/registry.py; tests/test_extension_discovery.py; single-user-local trust model
✅ doneFirst real SinkSpec dogfooded end-to-end — NocoDB 作品快照 writeback (POST /api/v3 records), reactive field-set round-tripValidated the symmetric path on a real external API before generalizingtests/test_writeback_e2e.py; gate satisfied
✅ doneExtension-spec doc: how to author a DataSourceSpec/SinkSpec/capability kernelUsers write channels without reading engine sourcedocs/extension-authoring.md; sibling to ontology-authoring.md
P2Durable external subscription ergonomics on top of the (already-shipped) cursor event streamLet an external process react to Store changes without polling gluenone — events(since=cursor) already exists
✅ doneGuard expression introspection — guard_rejected reactive events and dispatch() Result.data now carry a structured guard_trace list: each entry names the failing sub-expression, the value it produced, and a note (e.g. "always-false literal"). Surfaced during hh-zk.scm dogfood. evaluate_guard_trace walks the AST, re-evaluating sub-expressions safely (guards are pure); engine.guard_trace() exposes it; facade.dispatch() attaches it on rejection.An agent debugging a failing guard can now read the exact sub-expression and its value without reading engine source — closes the "silent guard failure" usability gap.dsl/eval.py (evaluate_guard_trace); ontology/engine.py (guard_trace()); ontology/reactive.py (reactive_guard_rejected payload); ontology/facade.py (dispatch() Result)
✅ doneAgent-friendly CLI output — search --json, query --json (with composite --tag/--field/--name/--links-to/--created-last filters), and inspect node --json all ship full get_node_full() payloads in one round-trip. ontology * commands have --json across the board via the Result envelope. All the features listed in the original discussion item are implemented; the roadmap entry was not updated when they shipped. Residual gap (small): --state is not a first-class query filter — agents must use --field __state__MachineName=state with knowledge of the internal naming convention. Not blocking; a --state sugar alias is the only open ergonomics item.The structural advantages of runex (typed relationships, FTS, authoritative state) are now accessible to agents via machine-readable JSON output without extra glue.already landed in cli.py — search, query, inspect node, all onto_* commands
discussionCLI-discoverable DSL vocabulary — ontology manifest exposes the runtime surface (supertags, machines, actions) but not the DSL grammar itself. An agent authoring a new .scm file from scratch cannot discover available guard/effect primitives (links-on, field-nonempty?, transition-on, for-each, …), their signatures, or valid syntax without reading docs/dsl-reference.md. The claim "agent defines business logic at runtime" currently requires the DSL grammar to be pre-loaded into the agent's context (e.g. via SKILL.md) rather than being introspectable from the tool itself. Fix direction: runex ontology primitives (or ontology dsl --help) that emits guard-safe primitives, effect primitives, and a minimal example snippet for each — machine-readable JSON and human-readable Rich table.Makes the "agent-native ontology runtime" claim self-sufficient without external documentation in the agent's context.new CLI command; output sourced from dsl-reference.md data or a structured registry
✅ donecreate-node DSL write primitive — (create-node SUPERTAG NAME) added to the effect write-primitive block; guard-disallowed. Enables reactive actions to synthesise new nodes without a natural key — the "observe → extract → create draft → user approves" loop. Shipped in commit 5908ed1.The reactive layer can now create, not only update-and-link. The draft → approved machine pattern on synthesised nodes gives the user an explicit review gate.ontology/engine.py _make_env write block; ontology/reactive.py guard-disallow list; docs/dsl-reference.md
✅ doneList primitives: count, most-recent, most-recent-by-state, first — Pure read primitives added, safe in guard and effect. (count list) → integer; (most-recent supertag) → node-id of most-recently-created node with that tag; (most-recent-by-state supertag state) → same, filtered by machine state; (first list) → head or nil. Shipped in commit 5908ed1. Validates the "Sprint Activation Mirror" dogfood scenario — a reactive action can now measure execution drift, not just detect presence.Enables "the system notices what you are NOT doing" — deviation-detection and accountability scenarios require measurable count/sort, not just boolean presence.ontology/engine.py read-primitive block; docs/dsl-reference.md
✅ doneBlocking kernel lint — register_kernel(..., blocking=True) flag + ontology check warning — Decision: do not make the bus async. Making ReactiveBus._on_commit async introduces shared-mutable-state races (the _depth counter is a mutable instance variable, not thread-local) and does not fix the real problem: action effects that mix slow I/O with non-idempotent writes (create-node) would race on concurrent bus wakeups. The root cause is action design, not bus threading. What shipped: register_kernel(name, fn, *, blocking=False) flag — when True, the kernel name is added to engine.blocking_kernels. list_kernels() and manifest() now return [{"name": k, "blocking": bool}] dicts. analyze_blocking_kernels() statically walks every action's effect AST for (call-kernel "NAME" ...) forms where NAME is in blocking_kernels, and emits structured warnings. ontology check now runs both conflict analysis and blocking-kernel lint, displaying WARNING lines and exiting 1 under --strict. Pattern doc shipped in docs/extension-authoring.md (Signal-Then-Work). Action redesign required for thought_auto_transcribe (runex-product): three-phase shape — Phase 1 action sets status field (fast/atomic); Phase 2 external daemon detects pending status, runs blocking kernel, writes result; Phase 3 action reacts to result field with LLM step.Agents cannot know which kernels block the bus from the manifest alone. The lint gate makes the constraint machine-checkable at design time; the pattern doc teaches the correct alternative. Without this, every new .scm author faces the same trap.ontology/engine.py (register_kernel + blocking_kernels); ontology/facade.py (analyze_blocking_kernels, list_kernels, manifest); cli.py (onto_check); docs/extension-authoring.md (Signal-Then-Work section)
discussionMulti-machine / cross-agent ontology consistency — vault-as-sync-layer is the current position — When multiple machines or agents (e.g. a laptop Claude Code session and a remote openclaw instance) need to share the same ontology state, the tempting path is to expose runex as a host API so all agents hit a single authoritative store. This is rejected for now: runex's reactive bus is synchronous and local; putting SQLite behind HTTP does not change its concurrency model, introduces a network single point of failure, and violates the single-user-local trust invariant. The current position: the vault is the sync layer. runex's SQLite store is a derived, queryable index of the vault files; if the vault is synced across machines (Obsidian Sync, Synology, iCloud, git), each machine runs its own runex instance ingesting the same files and converges to the same ontology state. This is eventually consistent through the file-sync layer, not through runex itself. What this leaves open: (1) write conflicts — if two machines write to the same vault file concurrently, the file-sync layer resolves it, not runex; (2) reactive firing — two machines may independently fire the same reactive action on the same event, which requires effects to be idempotent; (3) if a genuine multi-writer, strongly-consistent shared store is ever needed, runex's store layer would need to be replaced or fronted by a real networked database, which is a significant architectural shift. None of these are the right problems to solve until real multi-machine usage patterns are observed.Not an engine architecture problem — it is a system topology question about where runex sits relative to the user's sync infrastructure. The answer depends entirely on how users actually deploy across machines, which is unknown. Record the current position so future decisions have a baseline.No code change. Position: vault sync owns cross-machine consistency; runex stays local-first. Revisit when multi-machine dogfood surfaces concrete conflicts.
✅ doneontology preview dry-run dispatch — runex ontology preview <action> <node-id> evaluates the full guard+effect+reactive cascade but rolls back all writes before returning. The implementation changed Store._txn from BEGIN/COMMIT to SAVEPOINT _sp / RELEASE (nestable, outer behavior identical); facade.preview() wraps the dispatch in a SAVEPOINT _preview / ROLLBACK TO SAVEPOINT _preview, capturing all cascade events before the rollback. Result carries data.dry_run=true, data.ran, the full events list, and guard_trace if the guard rejected. Observed failure mode: agents ran ontology trace to preview, silently committing both transitions; when user said "execute", IllegalTransitionError because the states were already consumed.Agents can now safely preview cascades before committing. ontology trace semantics are unchanged (it still commits); ontology preview is the non-destructive introspection path.store.py (_txn SAVEPOINT); ontology/facade.py (preview()); cli.py (onto_preview command)

| discussion | Bidirectional file-sync — source-of-truth is unsettled — The two one-way paths now exist: FileWatcher → ingest → Store (inbound) and ECA (writeback "markdown-file") → MarkdownFileSink → .md (outbound, landed with feat/markdown-file-sink). Composing them naively produces an infinite loop: Sink writes a file, FileWatcher detects the mtime change, re-ingests, ECA fires again. Beyond the loop, true bidirectional sync between two independently-mutable replicas is a distributed consistency problem with no general solution — the literature's catalogue (last-write-wins, 3-way merge, CRDTs, OT) each resolve it by giving up something. The key constraint that makes the problem tractable here: write-actor identity is distinguishable. Human edits arrive via the file; machine edits (agent, pipeline, reactive rules) arrive via the Store. These actors do not contend over the same fields by design — human fields are title, body, created, tags; machine fields are _state__*, _ai_summary, reactive-derived links. Under this constraint the architecture is not symmetric bidirectional sync but human-sovereign + machine-assisted: files are the human's workspace (always trusted, always wins), the Store is the machine's workspace (writeback only when human has not concurrently modified). What needs to be built before this becomes real: (1) Echo suppression — MarkdownFileSink and FileWatcher must share a known_writes: dict[Path, float] so the Sink can register a just-written mtime and the Watcher skips it; without this the loop is live. (2) Sync-epoch fields — _sync_written_at and _sync_file_mtime stored on the node; the Sink updates them after each write, the Watcher updates them after each ingest; conflict = file.mtime > _sync_written_at AND tx_log has Store mutations since _sync_file_mtime. (3) StoreWatcher — a cursor-based tx_since() poller that detects external Store mutations on nodes with a 来源路径 field and schedules a writeback, the symmetric counterpart of FileWatcher. Two architectural paths remain open: Path A (learn from Obsidian) — flip the truth: files are authoritative, the Store is a derived read-only index; write operations are always file-first; the bidirectional problem disappears but the Store's write API is restructured around "write the file, then ingest." Path B (keep Store as truth, files as projection) — the current architecture's implicit bet; bidirectional sync is real but bounded by actor-identity asymmetry; requires the three build items above. No path is chosen. Neither has been dogfooded far enough to know which friction is more tolerable. Record the position so the decision has a baseline when real usage patterns surface. | The two directional paths composing into a loop is a structural gap, not a missing feature. Source-of-truth is a pre-requisite architectural choice, not something to discover after building sync plumbing. Record now so a future actor does not unknowingly build against the wrong assumption. | watcher.py (echo suppression); pipeline/ingest.py + sinks/markdown_file.py (sync-epoch fields); new StoreWatcher (cursor-based external-mutation detection). No code change in this entry. |

| ✅ done | Derived-node lifecycle — source deletion propagates to the Store — FileWatcher now tracks the full lifecycle of derived nodes: when a file that was previously ingested disappears from the filesystem, the watcher deletes the corresponding Store nodes (located via the 来源路径 field). This closes the coherence gap where a deleted source file left a stale ghost node in the Store indefinitely. The mechanism is discriminated by the source_path_field constructor kwarg (default "来源路径"; pass None to disable). Native nodes — those with no 来源路径 field — are structurally unreachable by this logic and are never touched. The two node kinds (derived / native) thus have the correct lifecycle: derived nodes are born and deleted with their source; native nodes live and die inside runex. What this is NOT: it does not change the write direction for derived nodes — the Store remains the write target; writes to a derived node's fields are valid and will be overwritten on the next re-ingest (consistent with "Store is a cache" semantics). The deeper question of whether writes to derived nodes should be file-first (Path A of the bidirectional sync discussion) remains open. | Derived nodes whose source disappears no longer accumulate as ghost nodes. The Store now accurately reflects the external source's existence state without any architectural change to the query or write paths. | store.py (find_nodes_by_field_value); watcher.py (FileWatcher.__init__ + _tick deletion sweep); tests/test_watcher_derived_cleanup.py (4 tests); tests/test_store.py (4 tests) |

| discussion | Dual-mode node identity — derived vs native, source_uri as the implicit discriminator — runex supports two fundamentally different node types in the same Store, and this is intentional, not a gap. Derived nodes come from an external source (an Obsidian .md file, a WeChat export, a Claude session); they carry a source_uri and can be rebuilt by re-running ingest — the Store is a disposable cache for them. Native nodes are created inside runex (via create-node DSL effect, direct CLI, or reactive synthesis) and have no external home; the Store IS their source of truth — losing it means losing them. This maps to the well-known Master Data / Derived Data distinction; Palantir Foundry has the same duality (some Objects are dataset projections, some are Foundry-native). The current discriminator is source_uri: present → derived, absent → native. This is a convention, not enforced — the system does not know whether a given node is reconstructable. Practical implication for backup strategy: a Store containing only derived nodes needs no backup (re-ingest rebuilds it); a Store with native nodes must be treated as a real database and backed up. create-node is not a gap: the DSL effect that creates a node with no external file was previously flagged as a potential design flaw under the "pure ontology overlay" framing. It is not — it serves the native-node use case correctly. The two modes coexist in the same Store; the user's responsibility is knowing which nodes belong to which category. Future formalization (deferred): if Store disaster-recovery or intelligent backup tooling is ever needed, the source_uri convention could be promoted to an explicit node_kind: derived | native field on nodes, allowing the engine to enumerate reconstructable vs irreplaceable nodes without convention knowledge. Not urgent; the current convention is sufficient for single-user use. | Clarifies that create-node producing "orphan" nodes is correct behaviour, not a gap to fix. Sets the right expectation for backup strategy — users should not assume the Store is always disposable. Prevents future confusion when a node disappears after a Store rebuild. | No code change. Convention: source_uri is the discriminator. Deferred formalization: explicit node_kind field when disaster-recovery tooling warrants it. |

Dogfood gate: no generalization of the channel/sink mechanism is merged until at least one real churny channel round-trips through it (P1), per the project's "validate value before a large refactor" discipline.

Assertions ledger ​

Every architecture.md assertion, and what this roadmap does to it. Nothing is broken; everything is preserved or extended additively.

architecture.md assertionStatus under this roadmap
Invariant 1 — L4→L3→L2→L1 one-way; removing L3 leaves L2+pipeline workingPreserved. Channels/sinks live at the L4/registry edge; capability kernels are L3 escape hatches. No new upward dependency.
Invariant 2 — no business names in framework codeExtended. Strengthened to "no channel names in core" — WeChat/Notion live only in extension specs, never in the engine.
Invariant 3 — __state__* write-protectedPreserved. Untouched.
Invariant 4 — read-primitive vs kernel boundary; no eval/untrusted import in the trusted pathPreserved & made explicit. Capability kernels are the only sanctioned external-I/O surface, bootstrap-installed and demarcated from the pure core; the trust-boundary section states the RCE edge a product must not widen.
Invariant 5 — reactive order deterministicPreserved. Channel triggers are ordinary actions under the same priority ASC, name ASC rule.
Invariant 6 — error-on-conflict fails closedPreserved. Untouched.
Contract is an L4 concern; manifest() is the single introspection surfaceExtended. sinks join datasources in manifest(); the channel-trigger verb is introspectable. No data crosses the DSL untyped.
CanonicalItem is the typed ingest boundary; cascade is the connectionPreserved verbatim. The rejected JSON-shuttle design is explicitly not taken precisely to keep this true.
DSL is a description language; computation lives in kernelsPreserved. The trigger-level (not data-level) unification is chosen specifically to honor this.
"Future direction": L1–L3 extractable as a generic substrateReinforced. A stable capability socket + hot-plug channel registry is exactly what makes the substrate generic; channels are the part that does not extract.