Vote for your favorite SkillMD. The submission with the most likes wins the $1,000 Audience Choice Award for the NandaHack x HCLTech hackathon. Voting is open through September 25.Vote now →
in reviewhumanPR #75Registry

feat(registry): persistent Cloud SQL and Redis registry backends

The default in_memory registry is a shared dictionary — single-process, non-persistent, and transparent to the simulator's partition injection.

Author

andrea-cola avatar

@andrea-cola

github profile →
Status
In review
Opened on
Jul 6
Branch
hackathon/registry-persistent-backends

Description

The pitch.

## Summary

The default `in_memory` registry is a shared dictionary — single-process, non-persistent, and transparent to the simulator's partition injection. This PR adds `nest-registry-backends`, a new workspace package that provides two production-grade `Registry` protocol implementations as named `nest.plugins.registry` entry points:

| Entry-point name | Backend | Key properties |
|---|---|---|
| `cloud_sql` | PostgreSQL / Google Cloud SQL | Durable, multi-process, SQL `@>` capability queries, lazy TTL pruning |
| `redis` | Redis / Google Cloud Memorystore | Sub-millisecond lookups, SINTER-based capability intersection, native TTL |

Switch a scenario to either backend with a single line:

```yaml
registry: cloud_sql   # or: redis
```

---

## Cloud SQL backend

- Single `nest_agents` table: `agent_id TEXT PRIMARY KEY`, `card JSONB`, `capabilities TEXT[]` (GIN-indexed), `expires_at TIMESTAMPTZ`.
- `register` upserts the card and prunes expired rows lazily.
- `lookup` uses `capabilities @> $1::text[]` for in-database filtering; `name_pattern` and `metadata_filter` are applied as a Python post-filter.
- `subscribe` polls at a configurable interval (default 1 s) and yields only new arrivals.
- **`connect_postgres(dsn)`** — plain `asyncpg` pool for local dev / CI.
- **`connect_cloud_sql(...)`** — asyncpg pool via the [Cloud SQL Python Connector](https://github.com/GoogleCloudPlatform/cloud-sql-python-connector) with IAM auth (mirrors the production pattern used in the author's Cloud SQL deployments).
- `migrate()` is idempotent DDL; call once at startup.

## Redis / Memorystore backend

- Agent cards stored as Redis Hashes (`nest:agent:<id>`).
- Capabilities indexed in Redis Sets (`nest:cap:<cap>`); multi-capability lookup uses `SINTER`.
- TTL applied to agent hashes and capability sets; stale capability entries cleaned up on re-registration.
- `deregister` removes the hash and all capability set memberships atomically.
- **`connect_redis(url)`** — plain `redis.asyncio.Redis`.
- **`connect_memorystore(...)`** — `RedisCluster` with GCP IAM token auth (matches the `registry/store.py` pattern from production Memorystore deployments, including the `CredentialProvider` IAM refresh loop).

---

## Test suite — 56 tests, zero live services required

**Cloud SQL** (`test_cloud_sql_registry.py`): backed by an in-process SQLite shim (`AsyncpgShim` in `tests/helpers.py`) that translates the asyncpg pool API and PostgreSQL DDL/DML to SQLite, including the `@>` array-containment operator (handled as a Python post-filter over `card` JSONB).

**Redis** (`test_redis_registry.py`): backed by `fakeredis.aioredis.FakeRedis`.

Both suites cover:

- Protocol conformance: `isinstance(reg, Registry)` at runtime
- CRUD round-trips: register, lookup, deregister, upsert
- Capability filtering (single and multi-cap AND semantics)
- Name-pattern and metadata post-filters
- TTL application and refresh on re-registration
- Capability index cleanup on deregister and re-register (Redis)
- Subscribe: delivery of existing cards, new arrivals, no-repeat guarantee
- Property-based (Hypothesis): any sequence of register/deregister leaves lookup consistent

```
56 passed in 3.01s
```

Passes `ruff check`, `ruff format --check`, `pyright` (strict mode), and `pytest -v`.

## Checklist

- [x] `uv sync` — workspace updated with new member and dev deps
- [x] `ruff check .` — zero violations
- [x] `ruff format --check .` — all files formatted
- [x] `pyright` — 0 errors, 0 warnings
- [x] `pytest -v` — 56/56 new tests passing, existing test suite unaffected
- [x] Entry points declared in `pyproject.toml` under `nest.plugins.registry`
- [x] README with install instructions, quick-start, and design notes
- [x] `py.typed` marker present

Made with [Cursor](https://cursor.com)

Try it

Open PR on GitHubView diff

Checkout locally

git fetch origin pull/75/head:pr-75
git checkout pr-75