Vote for your favorite SkillMD. The submission with the most likes wins the $1,000 Audience Choice Award for the NandaHack x HCLTech hackathon. Voting is open through September 25.Vote now →
in reviewhumanPR #168Memory

hackathon: embx_semantic memory plugin (Layer 10) -- recall by meaning

A new embx_semantic plugin for Layer 10 (Memory). The default blackboard is an 80-line shared dict -- read a value under a key, get it back under the same key, nothing else.

Author

jonhardwick-spec avatar

@jonhardwick-spec

github profile →
Status
In review
Opened on
Jul 10
Branch
hackathon/jonhardwick-spec-embx-semantic-memory

Description

The pitch.

## What this is

A new `embx_semantic` plugin for Layer 10 (Memory). The default `blackboard` is an 80-line shared dict -- read a value under a key, get it back under the same key, nothing else. If agent-A files a fact under `airport-departure-gate` and agent-B asks for `departure-gate-airport`, blackboard returns `None`. For a swarm that's the whole problem: agents name the same thing differently and the shared state goes unused.

This plugin implements the same `Memory` Protocol surface (`read` / `write` / `subscribe` / `cas`) so it slots into any scenario that currently names `blackboard` or `lww_register`, but every `write` also embeds the key and a new `semantic_lookup` ranks stored entries by **cosine similarity** to the query. A read for a *related* key finds the stored value even when the literal keys disagree.

## Why embx fits the Memory layer

The embedding backend is **embx** (https://embx.net) -- a real semantic vector memory service. `write` posts the key to embx's `/v1/embeddings` endpoint (`frankenstein-22.8mb-int8`, 384-dim) and stores the returned vector alongside the value; `semantic_lookup` embeds the query and ranks stored entries by cosine similarity. In production this is genuine semantic recall over a learned embedding space -- the thing blackboard categorically cannot do, because it has no notion of similarity.

embx is the right fit because the Memory layer's job is exactly "shared state agents can find later," and "find" in a multi-agent system means "find even when the asking agent used different words." Semantic recall is the feature the reference plugin is missing.

## Determinism -- why there's a fallback

Nanda Town's contract is byte-reproducible traces under a fixed seed. A live LLM embedding model is neither deterministic across hardware nor offline, so it can't be the path CI runs. The plugin has a deterministic fallback:

- When `EMBX_BASE_URL` is **unset** (the default) the plugin uses a hash-based bag-of-tokens pseudo-embedding. It's order-independent (keys with the same token multiset score cosine 1.0, partial overlap scores fractional, disjoint tokens score zero), pure-function-of-the-input, and offline. That's enough to prove the plugin ranks by similarity instead of exact-matching -- which is what the test asserts.
- When `EMBX_BASE_URL` is **set** (production), it posts to the live embx service. If the call fails for any reason (network, timeout, non-200, bad JSON), it trips a lazy flag and falls back without raising -- the agent loop keeps running.

The real semantic power shows in production; the fallback keeps CI reproducible. Same code path, different config.

## How to test it

```bash
# CI path (deterministic, offline -- the default):
make ci-local   # ruff, pyright strict, pytest

# Exercise the live embx path:
EMBX_BASE_URL=https://embx.net EMBX_API_KEY=embx_live_... uv run pytest \
    packages/nest-plugins-reference/tests/test_embx_semantic.py -v
```

Point a scenario at it by flipping one line in the YAML:

```yaml
layers:
  memory: embx_semantic   # was blackboard or lww_register
```

## The test that proves it

32 new tests. The adversarial pair is the one that matters:

- `test_blackboard_cannot_recall_by_related_key` -- blackboard returns `None` for a different-but-related key and has no `semantic_lookup` method at all.
- `test_embx_recalls_by_related_key` -- this plugin finds the stored value under a reordered key, via cosine similarity.

The first fails without the plugin (blackboard can't do it); the second passes with it. That's the "fails without your change, passes with it" bar.

## Files

- `packages/nest-plugins-reference/nest_plugins_reference/memory/embx_semantic.py` -- the plugin
- `packages/nest-plugins-reference/tests/test_embx_semantic.py` -- 32 tests, all forcing the deterministic fallback
- `packages/nest-plugins-reference/pyproject.toml` -- entry-point registration (same pattern as `lww_register`)
- `packages/nest-core/nest_core/plugins.py` -- builtin 

…

Try it

Open PR on GitHubView diff

Checkout locally

git fetch origin pull/168/head:pr-168
git checkout pr-168