mptyl/ThothII
1
0
Fork 0
Code Issues 25 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

feat: implement memory and evidence administration with guided repairs
Publish documentation / publish (push) Successful in 1m27s
Details

Browse Source 
Add PostgreSQL-backed memory, editable evidence with source review and activation, and human-approved archive repairs across the harness, API, and UI. Include migrations, deployment support, regression coverage, and validation documentation.

Refresh permissions from validated session roles so existing administrator logins can access newly deployed archive management features.
This commit is contained in:
Codex
2026-09-10 10:31:34 +02:00
parent 8fe526dd6e
commit 82e2c91f42
168 changed files with 11914 additions and 1772 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/gestione-memory.md
+208 -222
View File
@@ -1,244 +1,230 @@
# Memory management
This document describes how Memory currently works in ThothII: its conceptual model, lifecycle, persistence, semantic search, review gates, session display, and main technical limits.
Memory holds reusable knowledge for a workspace. PostgreSQL is the authoritative
archive; Qdrant contains a rebuildable search projection. The harness owns both
the administrative operations and the verification of retrieved results.
## Architectural summary
## Administration
A Memory item is domain knowledge that can be reused across questions. It is not a copy of one question's schema linking.
Open **Administration → Memory management**, immediately after Database management.
An administrator selects the workspace explicitly. No active session, DWH binding,
embedding service or Qdrant connection is required to browse and edit the archive.
PostgreSQL must be available.
The page provides a paginated list, text/ID search, stable sorting, and combined
filters for family, concepts, database, table, column, origin and update date.
Filtering happens across the entire archive before pagination. Each card has a
complete detail view, an editable form, cancellation of unsaved changes, and an
explicit deletion confirmation.
| Family | Content |
| --- | --- |
| Domain clarification | A reusable definition or interpretation, with scope and context. |
| SQL rule | Guidance for constructing SQL, with scope and rationale. |
| Solved question | A question, its approved SQL and context; used as a consultative exemplar. |
| Explained error | A correction and its rationale, to avoid repeating a known error. |
All cards have a stable `mem-<UUID>` identity, title, scope, origin and timestamps.
Manual cards have no invented source session or decision. Workflow cards retain
their source references when an administrator edits them. There is no editorial
revision history.
The form also manages concepts, structured database/schema/table/column
dependencies, and links to other cards with an explicit meaning. Links can only
connect cards in the same workspace. Card, dependency and outgoing-link changes
are committed together. Deleting a card removes its incident links and dependencies,
while keeping the other cards, Evidence and Catalog metadata.
## Save, failure and recovery
```mermaid
flowchart TB
CLARIFY["F1 concept clarified"] --> REVIEW["F8 reviewer review"]
REVIEW -->|"accepted"| REGISTRY["registry.jsonl"]
REVIEW -->|"declined"| LOCAL["Session decision only"]
REGISTRY --> VECTOR["Qdrant semantic index"]
VECTOR --> FUTURE["Future F2 retrieval"]
FUTURE --> PROPOSAL["Reviewer proposal"]
flowchart LR
EDIT["Admin or workflow"] --> SQL["PostgreSQL transaction"]
SQL --> CARD["Current card and links"]
SQL --> WORK["Pending projection"]
WORK --> Q["Qdrant"]
Q --> CHECK["Verify current card and projection"]
CARD --> CHECK
CHECK --> REVIEW["Recall for human review"]
```
```text
F1: clarify a concept
│
▼
concept_clarified decision in the session ledger
│
▼
F8: reviewer decides whether to promote it
│
├── global registry registry.jsonl
└── Qdrant semantic index
│
▼
F2 in a future session
search and proposal to the reviewer
```
The service serializes each workspace's mutations. It first commits content and
the projection operation in PostgreSQL, then propagates to Qdrant. A failed save
is different from **saved, index update incomplete**. In the latter case the
current content is already available in administration, while its previous vector
result is excluded from recall.
The main invariant is `REUSABLE_TYPES = {"concept_clarified"}`: only clarified concepts can be generated, saved, searched, or proposed as Memory. Decisions such as `table_promoted`, `table_excluded`, and `column_promoted` remain local to the question.
The **Pending index updates** section offers an explicit retry, including cleanup
for deleted cards. These operations survive a restart. Retry reads the current
archive state and cannot restore an earlier edit or a deleted card. Technical
revisions are internal consistency markers, not user-managed card statuses.
## The three management levels
Every recall hit must resolve to a current card in the requested workspace, with
a matching valid projection. The response is reconstructed from PostgreSQL, never
from an unverified Qdrant payload. Deleted, orphaned and stale points are excluded
for both domain Memory and solved-question exemplars. An unavailable archive is
an operational error, not a successful empty archive.
| Level | Content | Function |
Rebuilding Memory uses only authoritative cards. It does not import JSONL files,
historical session artifacts or old vector payloads. Reference preprocessing keeps
the Memory collection separate and does not migrate legacy Memory payloads.
## Hybrid recall and links
Memory uses dense embeddings and Qdrant BM25, fused with reciprocal rank fusion.
Both branches receive the same workspace, family and scope filters before candidate
selection. Indexed text includes content, scope, rationale, question, exemplar SQL,
concepts and qualified physical dependencies. Both indexing and querying use the
workspace language (`en` or `it`), including manual administration without a DWH binding.
The workflow CLI binds recall to its configured database and schema. Optional
`--filters` JSON can narrow the business `scope`, `concepts`, `table` and `column`.
Business scope is an exact string; every requested concept must be present. Physical
context matches a single structured dependency: database, schema, table and column
cannot be satisfied by unrelated entries. A card without dependencies is workspace-wide;
a database-only or table-only dependency also applies to descendants. Without a table
filter, table-specific knowledge in the selected database/schema remains discoverable.
Filters cannot override the configured database/schema. Free-text business scope is
not automatically interpreted or inferred from the question.
The core expands outgoing links from current, eligible search candidates. It applies
the same filters and workflow-family restrictions to destinations and intermediate
cards. Removed, pending, previously decided and out-of-scope cards cannot act as bridges.
Traversal allows two hops, at most 20 outgoing links per card (stable target-ID order),
200 distinct card lookups and 400 inspected links in total. Search requests
`min(100, max(20, 3 × top))` seeds; the result limit remains between 1 and 100.
All candidates are ranked together using `1 / (60 + seed rank)` for direct hits,
plus the strongest linked contribution, decayed by `0.5` per hop. Repeated paths
do not accumulate votes. Ties use card ID. The returned `score` is a ranking score,
not cosine similarity or a confidence estimate. `retrieval.path` shows the strongest
link path, or just the card ID for an exclusively direct result. PostgreSQL content,
links and eligibility are resolved under the workspace operation lock after search.
Migration `002_hybrid_projection.sql` marks the format of old dense projections as
incompatible without changing their authoritative cards. They appear in pending
updates and are excluded from recall until explicit retry or `memory index` succeeds.
Memory writes can add a missing BM25 sparse vector to their collection; an incompatible
existing vector configuration fails visibly and leaves recovery pending. Reading never
silently falls back to dense retrieval. Reference remains independently managed.
An explicit `memory index` can also recreate a missing Memory collection before
rebuilding its projections; it never imports records from another source.
## Workflow integration
During the workflow, Pi prepares reusable proposals in the session artifact
`memory_proposals.json`. Each proposal names effective approved source decisions,
the content and scope, its rationale, and any physical dependencies or links.
Unexplained failures, rejected options and simple table selections do not create
reusable knowledge. Exact existing cards are reused; semantic similarity alone
never authorizes replacement. Updates name the existing card and its current revision.
At F8, `reviewer_memory_promote` presents one editable Memory summary, including
the approved solved question. The reviewer chooses additions and updates, edits
their content, scope, dependencies and links, or declines everything. Approved SQL
is read only here: changing the solution requires returning to SQL review.
Only selected cards and their links are committed. Invalid links or stale updates
roll back the entire selection. Links between selected new cards are resolved
inside the same transaction. An explicit update preserves the existing identity
and origin, including manually authored content.
A durable review receipt makes repeated delivery idempotent and recovers the
gap between saving Memory and recording `memory_summary_reviewed` in the session
ledger. Retry never recreates a deleted card. The gate then closes F8 and finalizes;
finalization itself performs no automatic Memory writes. Pending indexing remains
visible and recoverable in administration.
F2 consumes domain clarifications and excludes already decided Memory. In F4,
F6 and F7, `memory rules` retrieves applicable SQL rules and explained errors.
Pi presents their use in the existing schema, CTE or SQL approval gate.
Exemplar search remains consultative. Retrieval never constitutes approval.
Persistent Memory/Evidence conflict repair remains part of the joint X1 increment.
## Physical schema changes
After a successful physical Catalog synchronization, the backend passes the exact
removed tables and columns, database, schema and run identity to Memory. Only cards
with matching structured dependencies are deleted, together with their incident
links and searchable projections. Global cards and objects outside the synchronized
scope survive. Manual Catalog cleanup and failed DWH scans never trigger this deletion.
The Catalog transaction records a pending `memory_cleanup` phase before committing
the physical change. If cleanup or indexing fails, the run retains the original
removals. Retry completes that same operation without rescanning the DWH or relying
on a new diff. A new synchronization is blocked until this cleanup is completed.
The Memory deletion receipt and vector tombstones make the operation repeatable
across restarts. The synchronization drawer reports deleted Memory cards and errors.
## API and commands
The administrative HTTP surface requires `memory.manage`, included in the existing
admin role. The backend checks workspace identity and passes its trusted principal
and a protected request snapshot to the harness. The harness independently checks
the principal; ordinary users cannot bypass administration through the CLI.
Production authentication and CSRF protections apply to the new routes.
| Method | Path below `/api/workspaces/:workspaceId/memory` | Operation |
| --- | --- | --- |
| Session ledger | `concept_clarified`, `memory_promoted`, `memory_promotion_declined` | Audit and state for one session |
| Global registry | `mem-XXXX` records in `registry.jsonl` | Current canonical Memory archive |
| Qdrant index | Embeddings and metadata derived from the registry | Semantic search |
| GET | root | Search, filter and paginate the archive |
| POST | root | Create a card |
| GET | `/:cardId` | Read a complete card |
| PUT | `/:cardId` | Save content, links and dependencies together |
| DELETE | `/:cardId` | Delete a card and incident links |
| GET | `/pending` | List incomplete projection operations |
| POST | `/:cardId/retry` | Retry current projection work, including deletion |
The ledger contains provenance and human decisions. The global record contains reusable text. The vector index is a search projection, not the place where the workflow records decisions directly.
## What can become Memory
During F1, the workflow records clarifications as `concept_clarified` decisions. A clarification can express:
- definitions of production or organizational concepts;
- inclusion and exclusion criteria for a product line;
- formulas and calculation methods;
- interpretations of time periods;
- mappings to specific tables and columns;
- the meaning of flags, codes, or indicators.
The `MemoryRecord` model contains:
- `id`, such as `mem-0001`;
- the timestamp, session, and sequence of the original decision;
- `type`;
- `subject`;
- `detail`;
- `rationale`;
- `question_context`;
- `tables` e `concepts`.
For new Memory items, the type is always `concept_clarified` and `tables` starts empty. A table or column may appear in the explanation as a technical mapping, but it cannot be the Memory item's standalone concept.
Valid example:
> Ablation means a procedure with `ablazione_transcatetere = TRUE`, counted with `COUNT(DISTINCT cod_paz)` by year.
Invalid examples:
- `fact_cardioversione` as approved Memory;
- `dim_time` as rejected Memory;
- an "include this table" decision saved for future questions.
The workflow skill also documents this rule in [harness/.pi/skills/tht-sessione/SKILL.md](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/harness/.pi/skills/tht-sessione/SKILL.md#L217).
## Promotion at the end of the session: F8
At the end of the workflow, the `reviewer_memory_promote` gate runs a deterministic preview:
CLI configuration remains a per-command option. Commands emit pure JSON:
```text
tht memory promote --session <id> --preview --json
tht memory list --filters '{"family":"sql_rule","page":1}' -c <runtime.yaml>
tht memory show <card-id> -c <runtime.yaml>
tht memory create --data <card.json> -c <runtime.yaml>
tht memory update <card-id> --data <card.json> -c <runtime.yaml>
tht memory delete <card-id> --yes -c <runtime.yaml>
tht memory pending -c <runtime.yaml>
tht memory retry <card-id> -c <runtime.yaml>
tht memory index -c <runtime.yaml>
tht memory search "<question>" --session <id> --json -c <runtime.yaml>
tht memory search "<question>" --filters '{"table":"orders","column":"id","scope":"Sales"}' --json -c <runtime.yaml>
tht memory solved-search "<question>" --json -c <runtime.yaml>
tht memory rules "<question>" --session <id> --json -c <runtime.yaml>
tht memory propose --session <id> --data <proposals.json> -c <runtime.yaml>
tht memory summary --session <id> --json -c <runtime.yaml>
tht memory solved-index <session-id> --json -c <runtime.yaml>
```
The preview:
`memory index` rebuilds both domain and solved-question projections.
`solved-index` only retries an existing authoritative source receipt.
The reviewer gate owns `memory review-apply`; Pi must not call it directly.
Migration `003_review_receipts.sql` adds durable review and physical-cleanup receipts.
The backend uses `memory admin --workspace <id> -c <protected-request.json>` so
administration does not materialize session or DWH configuration.
1. reads the session's effective decisions;
2. considers only `concept_clarified`;
3. discards decisions already promoted;
4. discards sequences already declined in F8;
5. deduplicates equivalent content;
6. proposes at most five candidates.
## Installation and storage
The code applies filtering and deduplication in
[harness/tht/memory/core.py](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/harness/tht/memory/core.py); the gate applies an additional defensive filter in
[harness/.pi/extensions/gate/memory/index.js](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/harness/.pi/extensions/gate/memory/index.js).
Memory shares the installation's existing PostgreSQL service, using its own
`thoth_memory` schema and versioned harness migration pack. The existing
`catalog-migrate` preparation service runs Catalog migrations followed by
`python -m tht.memory.migrate`. The core image includes both migration runners.
Ordinary API requests never migrate the schema.
The reviewer sees one preselected checklist. For each candidate:
Connection credentials come from the existing generated `THT_CATALOG_DB_HOST`,
`THT_CATALOG_DB_PORT`, `THT_CATALOG_DB_NAME`, `THT_CATALOG_RUNTIME_USER` and
`THT_CATALOG_RUNTIME_PASSWORD_FILE`. Migration uses the corresponding migrator
user/password file. Direct URL environments can use `THT_CATALOG_DATABASE_URL`
(runtime) and `THT_CATALOG_MIGRATOR_DATABASE_URL`; the harness also accepts
`THT_CATALOG_RUNTIME_DATABASE_URL`. Credentials are not authored in workspace YAML.
- selected: `tht memory save-one` runs, followed by a `memory_promoted` record;
- deselected: records `memory_promotion_declined`;
- no candidates: F8 closes automatically.
The migration grants the installation login membership in the restricted
`thoth_memory_runtime` role. Every repository transaction sets that role and a
workspace context. Forced row-level policies isolate cards, links, dependencies
and projection operations. This role has Memory DML and migration-status read
access, with no runtime DDL privilege. There are no cascading foreign keys to
Catalog or Evidence. A missing or incompatible schema returns a clear operational
error. Preparation is repeatable and checks migration checksums.
The `memory_promoted` marker uses `detail: seq:N`, a reference to the original `concept_clarified` decision. The flow is in [harness/.pi/extensions/tht-gate.js](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/harness/.pi/extensions/tht-gate.js#L1691).
Promotion does not run in F2, and the model cannot invent F8 candidates. Direct promotion commands are also protected by the anti-bypass gate.
## Global persistence
### JSONL registry
The current registry is:
```text
<artifacts>/memory/registry.jsonl
```
The registry is written through a temporary file and `os.replace`, so replacement is atomic. Promotion is idempotent on the `session_id + decision_seq` pair: the same decision from the same session cannot create two global records.
### Qdrant
After promotion, `save-one` builds one `VectorRecord` and sends it to the Qdrant index. The indexed text includes:
- type and subject;
- detail;
- rationale;
- question context;
- any concepts and mappings.
The vector record uses the ID `memory:mem-XXXX`. Its metadata stores `subject`, `detail`, `rationale`, `tables`, `concepts`, and the `kind` discriminator. The content's SHA-256 hash prevents embedding and upsert work when the text has not changed.
This behavior is implemented in
[harness/tht/memory/core.py](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/harness/tht/memory/core.py).
### Current canonical source
The JSONL registry remains the application's canonical source, while Qdrant is a derived but persistent index. The workflow does not record decisions directly in the vector database. It uses Qdrant as a searchable projection of the registry and effective ledger.
## Reuse in F2
In a future session, F2 runs:
```text
tht memory search "<question>" --session <id> --json
```
The command:
1. creates an embedding for the question;
2. searches the vector store for records with `kind=memory` only;
3. resolves each hit in the JSONL registry through its `ref`;
4. discards records missing from the registry;
5. discards every type other than `concept_clarified`;
6. excludes Memory already decided in the current session;
7. returns results ordered by similarity.
Memory is never applied automatically. The model must present it in one `reviewer_decide` choice:
- a selected Memory item is recorded as a new `concept_clarified` in the current session;
- the rationale must cite the original `mem-XXXX` ID;
- a deselected Memory item means "do not apply it now", not "delete it globally".
If F2 is reopened, a deselected Memory item can be proposed again. `memory_rejected` remains supported for legacy decisions and sessions, but it is not the normal behavior for current F2 deselection.
## Effective ledger, rollback, and reopening
The ledger is append-only. Reopening and withdrawing decisions do not delete earlier rows, but they change which decisions are effective.
Memory helpers use `effective_decisions()` to:
- exclude withdrawn decisions;
- ignore decisions from phases that became stale after a rollback;
- prevent promotion of clarifications that are no longer valid.
The effective view is defined in [harness/tht/phase.py](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/harness/tht/phase.py#L82).
## Display in the session summary
The "Memories" section of the summary is projected at runtime from the session ledger; it is not a direct copy of the global registry.
The projection:
- shows approved Memory first;
- then shows declined Memory;
- resolves `seq:N` to the original `concept_clarified`;
- hides markers whose original record is not `concept_clarified`;
- hides subjects that match schema-linking tables;
- hides standalone subjects shaped like `fact_*` or `dim_*`;
- keeps table and field references when they are part of the conceptual explanation.
The logic is in [harness/tht/session/store.py](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/harness/tht/session/store.py#L237). The frontend renders a structured list and treats `subject`, `detail`, and `rationale` as Markdown instead of showing raw Markdown.
The transformation happens on read. Historical sessions use the current layout and filters without rewriting their original artifacts.
## Enforced invariants
The protections are distributed across several boundaries:
1. `REUSABLE_TYPES` in the Python core;
2. the `memory search` command filter;
3. the F8 preview filter;
4. filtering and deduplication in the Pi gate;
5. exclusion of table Memory from the UI projection.
This prevents one prompt or component change from reintroducing tables as Memory.
## Remaining limits and risks
### The registry and index are not one transaction
Saving broadly follows this sequence:
```text
JSONL registry → Qdrant → memory_promoted marker in the ledger
```
If Qdrant is unavailable, the registry can contain Memory that is not yet searchable. The command reports that reindexing is required.
If the ledger marker fails after the vector database save, the Memory item can exist globally without a complete session audit. The gate returns a manual recovery command.
### Five-candidate limit
F8 proposes at most five Memory items. If a session produces more than five valid concepts, the extra items are not shown and the session can be finalized without promoting them.
### Deduplication is not global
Deduplication prevents duplicates within one proposal, and idempotency prevents the same decision from being promoted twice. There is no global merge of semantically similar Memory items from different sessions.
### Old physical records
Old `table_promoted` or `table_excluded` records may still exist in historical artifacts or indexes. The current code makes them unusable by filtering by type and does not show them in session projections. Physically removing them from the vector database remains a separate cleanup task.
## Final assessment
The current implementation matches the functional requirement: Memory is reusable conceptual knowledge, not a schema-linking choice.
The strongest part is the multilayer protection of the `concept_clarified` type. The main technical debt is the coexistence of the JSONL registry and Qdrant, with no single transaction spanning the global archive, semantic index, and session ledger.
See [the M1 specification](plans/2026-09-08-memory-m1-spec.md) and
[ADR 0018](adr/0018-use-postgres-for-memory-and-qdrant-for-retrieval.md) for the
approved scope and acceptance boundaries.
See [M2 implementation and validation](plans/2026-09-09-memory-m2-validation.md)
for the retrieval checks and real embedding test command.