skills

26 skillsgithubanonymous
all skillsmemory-curation

memory-curation

Use when a memory store looks bloated or noisy, when recall returns transcript junk instead of facts, before any destructive memory migration or rebuild, for periodic hygiene, or when asked to "remember this" durably. Covers audit, dedupe, prune, rebuild, verify, back up, and recall hygiene. Do NOT use to build session distillation — the session-memory-distiller plugin already does that automatically.

raw markdown

Body

# Memory curation (the distiller writes; this skill keeps the store honest)

## Find the store first — never hardcode its name

The RAG server resolves the memory dataset from the **identity**, not this skill: `X-Memory-Dataset` header → the caller's saved ★ binding → `MEMORY_DATASET` env → an explicit `dataset_name` argument, which applies **only when nothing above resolves**. An explicit name that **disagrees** with a resolved binding is **refused** (a prompt-injection guard, not a bug). So **discover the store; do not assume it**:

- `list_memories` with **no** `dataset_name` — its first line names the resolved store (`N memory(ies) in '<name>'`). That is the discovery call; `search_memory` / `add_memory` / `delete_memory` / `forget_session` resolve identically (just omit the argument).
- `get_dataset_info` is the **exception**: its `dataset_name` is required and is *not* binding-resolved — pass the discovered name. It is the source of the live `document_count`.
- Binding: `GET /api/access/selections` returns `memory_dataset`; `set_memory_dataset` sets the ★. Pass `dataset_name` **only** when the user names a *specific* store — a hardcoded name retargets the call or is refused.

## The distiller already exists — do not rebuild it

`@deepseek-ai/dsh-session-memory-distiller` is **live** in the profile's `cordis.patch.yml` (`provider`, `model`, `maxDistillTokens: 2048`, `distillTimeoutMs: 120000`, `rawTranscript: false`, `debounceMs: 45000`). **Cite it by package name, never by path** — the plugin moves, and a stale path is a false claim: `$DSH_PROFILE_DIR/node_modules/@deepseek-ai/<pkg>` is a symlink to the real source (any DSH plugin).

What it already does, with no help from you: on `user/message` | `assistant/message` | `assistant/chunk` | `tool/result` it debounces 45 s per session; on quiet or `session/disposed` it rebuilds the transcript and distills it into `[MEMORY INDEX | project: … | type: session-summary | topics: …]` plus `## What we did` / `## Key decisions & rationale` / `## Gotchas` / `## How to proceed`, then POSTs `add_memory` with `metadata.kind = "project-summary"`. On ANY distill failure it falls back to the raw transcript as `kind: "session_history"` — a session is never lost.

**Why map-reduce:** a DSH session can hold a 1M-token context but the distill model cannot, so long transcripts are split into ~200k-char windows, each compressed by its own call, then reduced. Constants: `MAX_HISTORY_CHARS` 8,000,000 · `MAP_CHUNK_CHARS` 200,000 · `MAX_MAP_CHUNK_CHARS` 700,000 · `MAX_MAP_CHUNKS` 24, adaptive so the WHOLE transcript is covered. **Superseded:** the single-call revision (`MAX_HISTORY_CHARS` 32,000) — describing that is stale.

**Do not write a distill pipeline, a session summarizer, or a "remember this session" script.** This skill owns the other half: audit, dedupe, prune, rebuild, verify, backup, recall hygiene. Changing the distiller itself is a separate job (installer script + `plugin-build/` in the memory-ops directory).

## First check when recall gets noisy

| Symptom | Cause | Action |
|---|---|---|
| Store fills with `session_history` transcript chunks; recall returns mid-sentence fragments | Distiller is falling back on every flush | Confirm `provider` + `model` are both set in the profile. Log line `no provider/model configured — every write will fall back to the raw transcript` is the tell |
| Nothing is written at all | Missing transport config | Distiller logs `disabled — need url + RAG_MEMORY_DATASET/PASSWORD in the dsh process env`; both are set in the DSH env file |
| Near-identical docs from a few sessions | Chunker bug — embedding dedup cannot see it | Text-ratio dedup, not cosine |
| Same content under different session ids | Session forks | Keep the longest fork per cluster (lossless) |

## Count before you touch anything

- **`list_memories` returns an ARBITRARY scroll sample, sorted by `memory_ts` afterward — it is NOT the true newest-N.** (`mcp_server.py`: `dm.scroll_documents(...)` then `rows.sort(key=_ts, reverse=True)`.) Never reason about counts or ordering from it; use it only to harvest `memory_id` values.
- Live count: `get_dataset_info` → `document_count` — **re-read it, never reuse this number**; it moves as the distiller writes.
- Full corpus: `GET /api/datasets/{name}/export` → a `.tar.gz` with `meta.json` (password hash stripped), `documents.jsonl` (`{"id","payload"}` NDJSON), and `files/`. Header `X-RAG-Api-Key`; dataset routes also need `X-Dataset-Password`. Sweep scripts read `documents.jsonl` — the only view with true generation, chunk-count, and text-ratio data.

## Back up before any destructive step

Non-negotiable — the pre-Aug-19 memory loss is the precedent. A rebuild with no export is data loss.

1. Export first (the tar.gz above). Keep the parsed corpus too — the memory-ops directory holds the curated cards, distilled per-project summaries, gap sessions, `fact-cards.json`, and the plan JSONs.
2. Restore: `POST /api/admin/datasets/import` (multipart `file`, or JSON `{"s3_uri": ...}`). **The export carries no vectors — a restore re-embeds**, so it is not a byte-identical rollback.
3. Run every script WITHOUT `--apply` first and diff its plan against the export.

## The three bloat root causes (reference incident, 2026-09-09: 1,044 docs, only ~196 curated)

1. **Sliding-window chunker bug.** During the Sep-5 mass flush the chunker advanced its cursor ~48 chars against a ~29.5k-char window; the three biggest sessions each stored 62–164 near-identical copies of the same block while declaring 3,126 / 1,168 / 809 chunks. Text ratio 0.9997–0.9999, but embedding cosine landed **just under 0.995** — ingest dedup passed them all.
2. **Mass re-flush after a harness/plugin update** — ~600 chunks in 10 minutes (Sep 5 16:06–16:16 UTC); count went 429 (Sep 3) → 1,043 (Sep 9).
3. **Session forks** — each fork gets a new session id and flushes its inherited transcript, creating cross-session duplicates. A fork's transcript is a superset of the parent's kept content, so **keeping the longest fork per cluster is lossless.**

After that sweep: 648 docs = 196 curated + 452 `session_history` across 300 sessions, zero cross-session near-dups; remaining 0.9+ pairs are within-session header/prompt repeats — benign, leave them.

## Dedup rules

- Ingest gate is cosine ≥ `RAG_DEDUP_THRESHOLD` (default `0.995`) — `add_memory` auto-skips near-dups, so a re-add can be a silent no-op. Always read `stored_ids`; never assume the count landed.
- **Cosine alone misses root cause #1.** Compare TEXT: normalize whitespace, then `difflib.SequenceMatcher` ratio ≥ ~0.999 on the body, grouped by `session_id` + `memory_ts` generation. Within a session keep the newest generation; across fork clusters keep the longest.
- `delete_memory` is **explicit-IDs-only by design** — you cannot delete what you have not listed. Split memories need every chunk id. Delete in a loop: list → match → delete → repeat until nothing is found.

## Tooling (the deployment's memory-ops directory)

Locate it before citing it (reference locations in the fenced section); the script NAMES are the recall value. **Sweep/analysis** — `analyze2.py` (generation, zombie, and near-dup analysis over `documents.jsonl`), `measure.py` (stored vs declared chunk counts per session), `verify_copies.py` (per-chunk text-ratio evidence). **Rebuild**:

| Script | What it does |
|---|---|
| `rebuild-via-gateway.py` | list all memory_ids → delete all → re-add the normalized set. Destructive; run from the operator's host terminal (it needs the admin key, not the MCP path) |
| `purge-transcripts.py` | purge transcript-fragment memories (dry-run default; `--apply` to delete) |
| `dedup-sweep.py` | content-duplicate memories (same body, different headers/sessions); keeps the newest. Idempotent |
| `sweep-untagged.py` | delete untagged docs only when a tagged twin exists; untagged docs with no twin are kept |
| `renormalize.py` | delete old-format docs and re-ingest the normalized set (needs the RAG API key + memory password; REST via `kubectl port-forward` to bypass the gateway) |
| `final-cleanup.py` | full clear-and-re-add cycle for a multi-generation store (dry-run default; `--apply`) |

Reuse instead of regenerating: the master index, the curated cards, `fact-cards.json`, and the plan JSONs (`reingest-plan-*.json`, `store-fixpoint.json`, `preserve-docs.json`).

## PROJECT MAP principle (settled 2026-10-02 — do not re-litigate)

The dataset is a **PROJECT MAP** — *what we worked on / how to access it / how to proceed* — built from distilled per-project summaries + atomic fact cards + a WHO/themes card. **Not raw transcripts.** Transcripts stay local (`$DSH_HOME/sessions/`) or in backup exports. Chunking: 8192 tokens default.

## Recall hygiene (what a good memory looks like)

- Every doc opens with `[MEMORY INDEX | project: <slug> | type: <type> | topics: a,b,c]` — the header is embedded, so topic queries hit it directly. Keep the slug one of the known project slugs.
- One fact per card; 3–8 lowercase topics; exact paths, digests, versions, error strings verbatim.
- Metadata `kind` values observed: `project-summary`, `fact-card`, `methodology`, `correction`, `note`; `session_history` is machine-written only (the distiller's fallback path).

## Verify after any migration

1. `get_dataset_info` `document_count` matches the plan (accounting for auto-skipped dupes).
2. Run 5–7 probe queries through `search_memory`; the right document must land at #1 (the reference rebuild scored 7/7 at rrf 1.0 — hold that bar). Then re-export and re-run the dup analysis: zero cross-session dups.

## Gotchas

- Gateway tools are reached through the `pcai_tools` **relay** (see the reference section), not by
  writing `mcp__<gateway>__<server>__<tool>` names — there are no `tools[...]` bindings for them and
  no `mcp_catalog`/`mcp_call` client tools. Discover the surface with `{action:"list"}`.
- `/tmp` does not persist between bash tool calls — make every verification one self-contained call.
- Never run `git checkout` / `restore` / `clean` / `stash` anywhere (see `guard-uncommitted-work`).
- `forget_session(session_id)` deletes ONLY that session's `session_history` docs — the tool for a session with sensitive material. It never touches curated memories.
- The store is typically password-protected (`has_password: true`); dataset routes need `X-Dataset-Password`. In-store deletion is irreversible; the export is the only recovery, and it restores by re-embedding.
- `lessons-loop` owns the **sql-lessons / sql-lessons-candidates** promotion gate — a different store and lifecycle; do not conflate it with memory-store curation. Before a destructive rebuild, apply `guard-uncommitted-work` to anything on disk and run `cross-validate` on the delete/keep plan when large.

## Deployment values to resolve locally

Resolve your own equivalents (see PORTABILITY.md) before using any of these.

- Memory dataset: **discover it, never hardcode it.** `list_memories` with no `dataset_name` names the
  resolved store on its first line; the name comes from your identity's ★ binding or
  `RAG_MEMORY_DATASET` in the DSH env file.
- Gateway MCP goes through the **relay**: call the `pcai_tools` tool DIRECTLY (a normal tool call,
  not inside `run_code`) with `{action:"describe", server:"mm-rag"}` to promote that server; its
  tools are then directly callable for the rest of the turn. `{action:"list"}` returns every server
  and tool with a one-line summary. Memory tools on `mm-rag`: `list_memories`, `search_memory`,
  `add_memory`, `delete_memory`, `forget_session`, `get_dataset_info`.
- Profile: `$DSH_PROFILE_DIR` (`$DSH_HOME/profiles/<name>`). The distiller resolves via
  `$DSH_PROFILE_DIR/node_modules/@deepseek-ai/dsh-session-memory-distiller`, which symlinks to the
  plugin's real source — cite it by package name, never by an absolute path (the plugin moves).
- Distiller config: `provider:` the Lead/main-agent route (`pcai-model-endpoint-flow`) — the route
  that IS distilled — and `model:` any model that route offers. **Do NOT "fix" this to
  `pcai-model-endpoint-subagent`.** That route sends `X-PCAI-Conversation: raw`, so nothing on it
  is ever due for distill; pointing the distiller there silently disables memory capture instead of
  cleaning it up. Provider and model ids rot — enumerate with `list_subagent_models({ provider })`
  before changing either.
- Memory-ops dirs: a sweep/analysis dir and a rebuild dir of your own (the rebuild dir also holds
  `install-mapreduce.fish` and `plugin-build/`).
- Rebuild key `PCAI_LLM_ADMIN_KEY`; `renormalize.py` also wants `RAG_API_KEY` + `RAG_MEMORY_PASSWORD`.
- Parsed corpus (2026-10-04): `curated/` 570 md, `distilled/` 39 md, `dsh-gap/` 81 md, `fact-cards.json`. 2026-10-02 rebuild scored 7/7 probes at rrf 1.0.