ThothII/docs/gestione-memory.md

Memory management

This document describes how Memory currently works in ThothII: its conceptual model, lifecycle, persistence, semantic search, review gates, session display, and main technical limits.

Architectural summary

A Memory item is domain knowledge that can be reused across questions. It is not a copy of one question's schema linking.

flowchart TB
    CLARIFY["F1 concept clarified"] --> REVIEW["F8 reviewer review"]
    REVIEW -->|"accepted"| REGISTRY["registry.jsonl"]
    REVIEW -->|"declined"| LOCAL["Session decision only"]
    REGISTRY --> VECTOR["Qdrant semantic index"]
    VECTOR --> FUTURE["Future F2 retrieval"]
    FUTURE --> PROPOSAL["Reviewer proposal"]

F1: clarify a concept
        │
        ▼
concept_clarified decision in the session ledger
        │
        ▼
F8: reviewer decides whether to promote it
        │
        ├── global registry registry.jsonl
        └── Qdrant semantic index
                    │
                    ▼
             F2 in a future session
             search and proposal to the reviewer

The main invariant is REUSABLE_TYPES = {"concept_clarified"}: only clarified concepts can be generated, saved, searched, or proposed as Memory. Decisions such as table_promoted, table_excluded, and column_promoted remain local to the question.

The three management levels

Level	Content	Function
Session ledger	concept_clarified, memory_promoted, memory_promotion_declined	Audit and state for one session
Global registry	mem-XXXX records in registry.jsonl	Current canonical Memory archive
Qdrant index	Embeddings and metadata derived from the registry	Semantic search

The ledger contains provenance and human decisions. The global record contains reusable text. The vector index is a search projection, not the place where the workflow records decisions directly.

What can become Memory

During F1, the workflow records clarifications as concept_clarified decisions. A clarification can express:

• definitions of production or organizational concepts;
• inclusion and exclusion criteria for a product line;
• formulas and calculation methods;
• interpretations of time periods;
• mappings to specific tables and columns;
• the meaning of flags, codes, or indicators.

The MemoryRecord model contains:

• id, such as mem-0001;
• the timestamp, session, and sequence of the original decision;
• type;
• subject;
• detail;
• rationale;
• question_context;
• tables e concepts.

For new Memory items, the type is always concept_clarified and tables starts empty. A table or column may appear in the explanation as a technical mapping, but it cannot be the Memory item's standalone concept.

Valid example:

Ablation means a procedure with ablazione_transcatetere = TRUE, counted with COUNT(DISTINCT cod_paz) by year.

Invalid examples:

• fact_cardioversione as approved Memory;
• dim_time as rejected Memory;
• an "include this table" decision saved for future questions.

The workflow skill also documents this rule in harness/.pi/skills/tht-sessione/SKILL.md.

Promotion at the end of the session: F8

At the end of the workflow, the reviewer_memory_promote gate runs a deterministic preview:

tht memory promote --session <id> --preview --json

The preview:

1. reads the session's effective decisions;
2. considers only concept_clarified;
3. discards decisions already promoted;
4. discards sequences already declined in F8;
5. deduplicates equivalent content;
6. proposes at most five candidates.

The code applies filtering and deduplication in harness/tht/memory/core.py; the gate applies an additional defensive filter in harness/.pi/extensions/gate/memory/index.js.

The reviewer sees one preselected checklist. For each candidate:

• selected: tht memory save-one runs, followed by a memory_promoted record;
• deselected: records memory_promotion_declined;
• no candidates: F8 closes automatically.

The memory_promoted marker uses detail: seq:N, a reference to the original concept_clarified decision. The flow is in harness/.pi/extensions/tht-gate.js.

Promotion does not run in F2, and the model cannot invent F8 candidates. Direct promotion commands are also protected by the anti-bypass gate.

Global persistence

JSONL registry

The current registry is:

<artifacts>/memory/registry.jsonl

The registry is written through a temporary file and os.replace, so replacement is atomic. Promotion is idempotent on the session_id + decision_seq pair: the same decision from the same session cannot create two global records.

Qdrant

After promotion, save-one builds one VectorRecord and sends it to the Qdrant index. The indexed text includes:

• type and subject;
• detail;
• rationale;
• question context;
• any concepts and mappings.

The vector record uses the ID memory:mem-XXXX. Its metadata stores subject, detail, rationale, tables, concepts, and the kind discriminator. The content's SHA-256 hash prevents embedding and upsert work when the text has not changed.

This behavior is implemented in harness/tht/memory/core.py.

Current canonical source

The JSONL registry remains the application's canonical source, while Qdrant is a derived but persistent index. The workflow does not record decisions directly in the vector database. It uses Qdrant as a searchable projection of the registry and effective ledger.

Reuse in F2

In a future session, F2 runs:

tht memory search "<question>" --session <id> --json

The command:

1. creates an embedding for the question;
2. searches the vector store for records with kind=memory only;
3. resolves each hit in the JSONL registry through its ref;
4. discards records missing from the registry;
5. discards every type other than concept_clarified;
6. excludes Memory already decided in the current session;
7. returns results ordered by similarity.

Memory is never applied automatically. The model must present it in one reviewer_decide choice:

• a selected Memory item is recorded as a new concept_clarified in the current session;
• the rationale must cite the original mem-XXXX ID;
• a deselected Memory item means "do not apply it now", not "delete it globally".

If F2 is reopened, a deselected Memory item can be proposed again. memory_rejected remains supported for legacy decisions and sessions, but it is not the normal behavior for current F2 deselection.

Effective ledger, rollback, and reopening

The ledger is append-only. Reopening and withdrawing decisions do not delete earlier rows, but they change which decisions are effective.

Memory helpers use effective_decisions() to:

• exclude withdrawn decisions;
• ignore decisions from phases that became stale after a rollback;
• prevent promotion of clarifications that are no longer valid.

The effective view is defined in harness/tht/phase.py.

Display in the session summary

The "Memories" section of the summary is projected at runtime from the session ledger; it is not a direct copy of the global registry.

The projection:

• shows approved Memory first;
• then shows declined Memory;
• resolves seq:N to the original concept_clarified;
• hides markers whose original record is not concept_clarified;
• hides subjects that match schema-linking tables;
• hides standalone subjects shaped like fact_* or dim_*;
• keeps table and field references when they are part of the conceptual explanation.

The logic is in harness/tht/session/store.py. The frontend renders a structured list and treats subject, detail, and rationale as Markdown instead of showing raw Markdown.

The transformation happens on read. Historical sessions use the current layout and filters without rewriting their original artifacts.

Enforced invariants

The protections are distributed across several boundaries:

1. REUSABLE_TYPES in the Python core;
2. the memory search command filter;
3. the F8 preview filter;
4. filtering and deduplication in the Pi gate;
5. exclusion of table Memory from the UI projection.

This prevents one prompt or component change from reintroducing tables as Memory.

Remaining limits and risks

The registry and index are not one transaction

Saving broadly follows this sequence:

JSONL registry → Qdrant → memory_promoted marker in the ledger

If Qdrant is unavailable, the registry can contain Memory that is not yet searchable. The command reports that reindexing is required.

If the ledger marker fails after the vector database save, the Memory item can exist globally without a complete session audit. The gate returns a manual recovery command.

Five-candidate limit

F8 proposes at most five Memory items. If a session produces more than five valid concepts, the extra items are not shown and the session can be finalized without promoting them.

Deduplication is not global

Deduplication prevents duplicates within one proposal, and idempotency prevents the same decision from being promoted twice. There is no global merge of semantically similar Memory items from different sessions.

Old physical records

Old table_promoted or table_excluded records may still exist in historical artifacts or indexes. The current code makes them unusable by filtering by type and does not show them in session projections. Physically removing them from the vector database remains a separate cleanup task.

Final assessment

The current implementation matches the functional requirement: Memory is reusable conceptual knowledge, not a schema-linking choice.

The strongest part is the multilayer protection of the concept_clarified type. The main technical debt is the coexistence of the JSONL registry and Qdrant, with no single transaction spanning the global archive, semantic index, and session ledger.