ThothII/docs/gestione-memory.md

# Memory management

This document describes how Memory currently works in ThothII: its conceptual model, lifecycle, persistence, semantic search, review gates, session display, and main technical limits.

## Architectural summary

A Memory item is domain knowledge that can be reused across questions. It is not a copy of one question's schema linking.

```mermaid
flowchart TB
    CLARIFY["F1 concept clarified"] --> REVIEW["F8 reviewer review"]
    REVIEW -->|"accepted"| REGISTRY["registry.jsonl"]
    REVIEW -->|"declined"| LOCAL["Session decision only"]
    REGISTRY --> VECTOR["Qdrant semantic index"]
    VECTOR --> FUTURE["Future F2 retrieval"]
    FUTURE --> PROPOSAL["Reviewer proposal"]
```

```text
F1: clarify a concept
        │
        ▼
concept_clarified decision in the session ledger
        │
        ▼
F8: reviewer decides whether to promote it
        │
        ├── global registry registry.jsonl
        └── Qdrant semantic index
                    │
                    ▼
             F2 in a future session
             search and proposal to the reviewer
```

The main invariant is `REUSABLE_TYPES = {"concept_clarified"}`: only clarified concepts can be generated, saved, searched, or proposed as Memory. Decisions such as `table_promoted`, `table_excluded`, and `column_promoted` remain local to the question.

## The three management levels

| Level | Content | Function |
| --- | --- | --- |
| Session ledger | `concept_clarified`, `memory_promoted`, `memory_promotion_declined` | Audit and state for one session |
| Global registry | `mem-XXXX` records in `registry.jsonl` | Current canonical Memory archive |
| Qdrant index | Embeddings and metadata derived from the registry | Semantic search |

The ledger contains provenance and human decisions. The global record contains reusable text. The vector index is a search projection, not the place where the workflow records decisions directly.

## What can become Memory

During F1, the workflow records clarifications as `concept_clarified` decisions. A clarification can express:

- definitions of production or organizational concepts;
- inclusion and exclusion criteria for a product line;
- formulas and calculation methods;
- interpretations of time periods;
- mappings to specific tables and columns;
- the meaning of flags, codes, or indicators.

The `MemoryRecord` model contains:

- `id`, such as `mem-0001`;
- the timestamp, session, and sequence of the original decision;
- `type`;
- `subject`;
- `detail`;
- `rationale`;
- `question_context`;
- `tables` e `concepts`.

For new Memory items, the type is always `concept_clarified` and `tables` starts empty. A table or column may appear in the explanation as a technical mapping, but it cannot be the Memory item's standalone concept.

Valid example:

> Ablation means a procedure with `ablazione_transcatetere = TRUE`, counted with `COUNT(DISTINCT cod_paz)` by year.

Invalid examples:

- `fact_cardioversione` as approved Memory;
- `dim_time` as rejected Memory;
- an "include this table" decision saved for future questions.

The workflow skill also documents this rule in [harness/.pi/skills/tht-sessione/SKILL.md](../harness/.pi/skills/tht-sessione/SKILL.md:217).

## Promotion at the end of the session: F8

At the end of the workflow, the `reviewer_memory_promote` gate runs a deterministic preview:

```text
tht memory promote --session <id> --preview --json
```

The preview:

1. reads the session's effective decisions;
2. considers only `concept_clarified`;
3. discards decisions already promoted;
4. discards sequences already declined in F8;
5. deduplicates equivalent content;
6. proposes at most five candidates.

The code applies filtering and deduplication in
[harness/tht/memory/core.py](../harness/tht/memory/core.py); the gate applies an additional defensive filter in
[harness/.pi/extensions/gate/memory/index.js](../harness/.pi/extensions/gate/memory/index.js).

The reviewer sees one preselected checklist. For each candidate:

- selected: `tht memory save-one` runs, followed by a `memory_promoted` record;
- deselected: records `memory_promotion_declined`;
- no candidates: F8 closes automatically.

The `memory_promoted` marker uses `detail: seq:N`, a reference to the original `concept_clarified` decision. The flow is in [harness/.pi/extensions/tht-gate.js](../harness/.pi/extensions/tht-gate.js:1691).

Promotion does not run in F2, and the model cannot invent F8 candidates. Direct promotion commands are also protected by the anti-bypass gate.

## Global persistence

### JSONL registry

The current registry is:

```text
<artifacts>/memory/registry.jsonl
```

The registry is written through a temporary file and `os.replace`, so replacement is atomic. Promotion is idempotent on the `session_id + decision_seq` pair: the same decision from the same session cannot create two global records.

### Qdrant

After promotion, `save-one` builds one `VectorRecord` and sends it to the Qdrant index. The indexed text includes:

- type and subject;
- detail;
- rationale;
- question context;
- any concepts and mappings.

The vector record uses the ID `memory:mem-XXXX`. Its metadata stores `subject`, `detail`, `rationale`, `tables`, `concepts`, and the `kind` discriminator. The content's SHA-256 hash prevents embedding and upsert work when the text has not changed.

This behavior is implemented in
[harness/tht/memory/core.py](../harness/tht/memory/core.py).

### Current canonical source

The JSONL registry remains the application's canonical source, while Qdrant is a derived but persistent index. The workflow does not record decisions directly in the vector database. It uses Qdrant as a searchable projection of the registry and effective ledger.

## Reuse in F2

In a future session, F2 runs:

```text
tht memory search "<question>" --session <id> --json
```

The command:

1. creates an embedding for the question;
2. searches the vector store for records with `kind=memory` only;
3. resolves each hit in the JSONL registry through its `ref`;
4. discards records missing from the registry;
5. discards every type other than `concept_clarified`;
6. excludes Memory already decided in the current session;
7. returns results ordered by similarity.

Memory is never applied automatically. The model must present it in one `reviewer_decide` choice:

- a selected Memory item is recorded as a new `concept_clarified` in the current session;
- the rationale must cite the original `mem-XXXX` ID;
- a deselected Memory item means "do not apply it now", not "delete it globally".

If F2 is reopened, a deselected Memory item can be proposed again. `memory_rejected` remains supported for legacy decisions and sessions, but it is not the normal behavior for current F2 deselection.

## Effective ledger, rollback, and reopening

The ledger is append-only. Reopening and withdrawing decisions do not delete earlier rows, but they change which decisions are effective.

Memory helpers use `effective_decisions()` to:

- exclude withdrawn decisions;
- ignore decisions from phases that became stale after a rollback;
- prevent promotion of clarifications that are no longer valid.

The effective view is defined in [harness/tht/phase.py](../harness/tht/phase.py:82).

## Display in the session summary

The "Memories" section of the summary is projected at runtime from the session ledger; it is not a direct copy of the global registry.

The projection:

- shows approved Memory first;
- then shows declined Memory;
- resolves `seq:N` to the original `concept_clarified`;
- hides markers whose original record is not `concept_clarified`;
- hides subjects that match schema-linking tables;
- hides standalone subjects shaped like `fact_*` or `dim_*`;
- keeps table and field references when they are part of the conceptual explanation.

The logic is in [harness/tht/session/store.py](../harness/tht/session/store.py:237). The frontend renders a structured list and treats `subject`, `detail`, and `rationale` as Markdown instead of showing raw Markdown.

The transformation happens on read. Historical sessions use the current layout and filters without rewriting their original artifacts.

## Enforced invariants

The protections are distributed across several boundaries:

1. `REUSABLE_TYPES` in the Python core;
2. the `memory search` command filter;
3. the F8 preview filter;
4. filtering and deduplication in the Pi gate;
5. exclusion of table Memory from the UI projection.

This prevents one prompt or component change from reintroducing tables as Memory.

## Remaining limits and risks

### The registry and index are not one transaction

Saving broadly follows this sequence:

```text
JSONL registry → Qdrant → memory_promoted marker in the ledger
```

If Qdrant is unavailable, the registry can contain Memory that is not yet searchable. The command reports that reindexing is required.

If the ledger marker fails after the vector database save, the Memory item can exist globally without a complete session audit. The gate returns a manual recovery command.

### Five-candidate limit

F8 proposes at most five Memory items. If a session produces more than five valid concepts, the extra items are not shown and the session can be finalized without promoting them.

### Deduplication is not global

Deduplication prevents duplicates within one proposal, and idempotency prevents the same decision from being promoted twice. There is no global merge of semantically similar Memory items from different sessions.

### Old physical records

Old `table_promoted` or `table_excluded` records may still exist in historical artifacts or indexes. The current code makes them unusable by filtering by type and does not show them in session projections. Physically removing them from the vector database remains a separate cleanup task.

## Final assessment

The current implementation matches the functional requirement: Memory is reusable conceptual knowledge, not a schema-linking choice.

The strongest part is the multilayer protection of the `concept_clarified` type. The main technical debt is the coexistence of the JSONL registry and Qdrant, with no single transaction spanning the global archive, semantic index, and session ledger.