feat: implement memory and evidence administration with guided repairs
Publish documentation / publish (push) Successful in 1m27s

Add PostgreSQL-backed memory, editable evidence with source review and activation, and human-approved archive repairs across the harness, API, and UI. Include migrations, deployment support, regression coverage, and validation documentation.

Refresh permissions from validated session roles so existing administrator logins can access newly deployed archive management features.
This commit is contained in:
Codex
2026-09-10 10:31:34 +02:00
parent 8fe526dd6e
commit 82e2c91f42
168 changed files with 11914 additions and 1772 deletions
@@ -0,0 +1,59 @@
---
status: accepted
date: 2026-09-08
---
# Use PostgreSQL for Memory and Qdrant for retrieval
The new Memory module uses the installation's existing PostgreSQL service as the
authority for Memory Cards, their links, and structured schema dependencies.
Qdrant holds a rebuildable search projection. This replaces the JSONL registry and
allows related administrative changes to be coordinated in one database transaction.
The owner accepted this direction in Q9 of the Memory design interview; implementation
is pending.
Memory uses its own tables and remains a separate module from the Metadata Catalog.
Sharing the PostgreSQL service does not transfer Memory ownership to Database
management or make Evidence and Memory one canonical domain.
## Retrieval and graph
The owner also accepted hybrid semantic/lexical Qdrant retrieval with scope filters
and explicit card links traversed in core. Both belong to the planned first version;
there is no dedicated graph database or external Memory framework.
This extends [ADR 0017](0017-separate-reference-vectors-from-runtime-memory.md):
the separate reference and memory collections remain, while Memory gains sparse
lexical indexing in addition to dense vectors. Memory projections become rebuildable
from the module's PostgreSQL authority. Preprocessing Clear still preserves Memory;
this decision does not add it to the preprocessing cleanup scope.
Links support discovery. Finding a card through a link does not approve its use.
The core proposes links with the cards for the same final review; Administration
provides manual creation, editing, and deletion. Deleting a card removes its incident
links without deleting the other linked cards.
## Considered options
- Retaining JSONL preserves the current storage format, but leaves coordinated
card/link/dependency mutations and concurrent administrative writes to application code.
- Using Qdrant as the sole authority is a viable alternative for record storage and
retrieval. PostgreSQL is preferred for the coordinated mutations of the new module,
with Qdrant reserved for its search projection.
- Adding a dedicated graph database would introduce another service; the selected
bounded traversal can be implemented in core over persisted links.
## Consequences
The module needs a persistence contract, PostgreSQL schema, and explicit propagation
of additions, updates, and deletions to Qdrant. Choosing PostgreSQL does not make this
propagation atomic across both systems: failures, retries, and invalidation of stale
search content must be handled and tested. An index rebuild uses the original card
content and cannot resurrect deleted cards.
The decision does not introduce Memory revision history or require compatibility with
existing development sessions. Evidence authoring and publication remain governed by
their own contract until the Evidence management project defines its evolution.
The [Memory management project](../plans/2026-09-08-memory-management.md) records the
approved behavior, scope, and integration work.
@@ -0,0 +1,152 @@
---
status: accepted
date: 2026-09-08
---
# Maintain local Evidence with external editors and manual consolidation
Evidence management must provide complete access to registered Evidence and a
maintenance path, as accepted in Q11–Q15 and revised during simplification. The owner subsequently
clarified that a context specialist writes drafts independently of the installation
and without PostgreSQL access; the system refines and stores them locally for use
and maintenance. Implementation is pending.
The storage mechanics of Q11 were revisited in the
[simplification review](../plans/2026-09-08-memory-evidence-simplification-review.md).
The original choice combined the workspace repository as Evidence authority with
application-managed Git writes. The owner accepted the separation of external draft
files/repositories from a durable local canonical file archive and removes
commit/push from normal CRUD. The management surface must expose discoverable,
editable Markdown for domain specialists, without JSONL editing. The owner chose
external editors for release 0: their preferred Mac/PC editor or vim/nano on the
server. No web editor or content form is required. The page keeps browsing,
filtering, and detail, with actual host file paths and maintenance instructions.
The owner also requested manual consolidation followed by manual Git diff, commit,
and push, relying on operator discipline rather than automation. The owner accepted
the remaining simplifications and requested explicit clarification of the core
format change and the simple terminal-based Git check. The original application Git writer
is not the final implementation prescription. A suggestion
to move Evidence authority into PostgreSQL was withdrawn after the clarification.
Memory remains a distinct module with its own PostgreSQL authority under
[ADR 0018](0018-use-postgres-for-memory-and-qdrant-for-retrieval.md).
## Manual consolidation and Git follow-up
The external-editor decision replaces the form's single Save with saving files and
running one consolidation command. The command checks expected structure, required
fields, types, identifiers, references, and provenance. It reports the affected file
and the necessary correction. Invalid input stops activation; valid input updates
derived metadata, the corpus, and the Evidence index. New and deleted files use the
same path. An inaccessible archive must not be interpreted as deleted Evidence.
Structural validation does not prove semantic correctness or rerun model refinement
to rewrite manually edited rules.
The core reads only successfully consolidated active content, never the editable
working files directly. Candidate validation and activation must preserve the last
valid corpus on failure, or report unavailability if its integrity cannot be
guaranteed. Unconsolidated edits must not mix new text with old retrieval data.
Consolidation reports full local success only once the updated content is available
to the core. If persistence succeeds but activation fails, the command reports the
partial outcome and can be rerun without duplicating units or losing edits.
The operation does not imply atomicity across
authoritative storage and Qdrant; their failure and recovery behavior requires
explicit implementation. An interrupted operation must not leave a partial corpus
advertised as ready for subsequent use.
The maintained archive is a persistent Git working tree containing the local
Evidence; it may reuse the workspace repository. Draft sources and curated units
remain distinct even when in the same repository. Setup and the page identify the
repository, working tree, actual host paths, branch, and configured remote. A
rebuildable runtime snapshot is not the editing location.
Before consolidation, the operator checks Git status, including inspection of new
untracked files. Normal terminal Git diff commands are available for line-level
inspection when needed; no custom viewer or mandatory double review is required.
After consolidation, the operator stages the Evidence and required metadata changes,
commits, and pushes using normal Git commands. No watcher, automatic Git writes, retrying push, pull, merge,
or dedicated web execution control is required. Missing credentials, unconfigured
upstreams, and Git conflicts are handled by the operator.
Consolidation activates local changes before commit/push. If the operator omits the
Git steps or push fails, the local update remains effective while transfer to the
remote is incomplete. Pushing does not automatically update other installations.
The system relies on operator discipline to complete the sequence; it does not add
a separate editorial publication state machine or require a second reviewer.
Approved conflict repairs from the core still call the same persistence and
activation service directly. They do not require an external editor or manual
consolidation before the session can use its own approved correction. Their local
file changes are included in the operator's subsequent Git maintenance.
## Administration without concurrent core work
The owner's simplification instruction replaces the earlier Q12 requirement to
refresh open sessions after administrative edits. Core activity can be assumed
absent during administration, or its overlap can be ignored. No dedicated live
update, session notification, restart, maintenance mode, or reader coordination
is required. Completed administrative changes apply to subsequent work.
Deliberate writes from the workflow itself still exist: approved conflict repairs
and the final Memory summary use the same persistence services and handle their
outcome before proceeding. In particular, a session must be able to use its own
approved Evidence correction. This does not require updating all other sessions
or rewriting previously approved decisions, artifacts, or SQL.
## Manual corrections survive source updates
When an updated source contradicts an administrator's correction, the saved manual
Evidence remains active. Evidence management shows the conflicting content for an
explicit decision and subsequent save. The source's newer content does not
automatically override the correction. The owner accepts that the manual rule can
remain in use until that comparison is resolved.
Deleted Evidence must not silently reappear after preparation or reindexing. The
authoring implementation must preserve both manual corrections and suppression of
deleted units through source refreshes. The current protection for uncommitted Git
changes is insufficient: it does not preserve already committed manual corrections
against regeneration, and retirement currently removes the unit without recording
suppression for later preparation.
## Manual creation and explicit source refresh
Administrators can create Evidence without providing an external document. In R0,
they add Markdown using the documented example and consolidate it. The application
records a manual declaration as the managed source of the current statement.
Explicit consolidation approves that declaration; it does not
claim independent documentary verification.
A correction that changes a unit's meaning uses the same explicit manual origin.
The original document remains linked for provenance and source-change detection,
but its excerpt is not presented as support for a rule it does not contain. The
canonical contract must distinguish the current supporting source from the original
document rather than silently retaining outdated support metadata.
External sources are reacquired only when an administrator requests a source
refresh. Ordinary saves and session lookups use the acquired local content; they
do not poll sources or fetch their current versions. A remote change is therefore
detected at the next requested refresh, when the manual-precedence rule applies.
The revised storage recommendation reads the specialist's repository as an input;
ordinary local edits do not write back to it or refresh its content automatically.
## Implementation consequences
The existing [Evidence lifecycle](../evidence.md) and
[Workspace Evidence v3 contract](../contracts/workspace-evidence-v3.md) require
explicit evolution for editable local Markdown, manual consolidation, and manual
precedence. Source provenance and coherent canonical metadata remain requirements;
manual changes must not be presented as statements supported by an unrelated source
excerpt. Under the accepted local-file direction, curated files are primary
installation data preserved by Clear and backed up separately from rebuildable
indexes. Visible Markdown content must become authoritative for editing; operators
must not maintain hidden duplicate text, hashes, or manifest entries themselves.
This is a core Evidence contract change: parser, renderer, authoring, validation,
normalization, and their integration with indexing and recall must be adapted
together. E1 includes a new Curated unit format version, conversion of existing
Evidence, reindexing, and end-to-end verification that visible edits reach core
consumption. Preserve the internal typed model where possible; the change does
not require redesigning the NL-to-SQL workflow phases.
Local edits do not automatically change external drafts or another installation;
Git versioning and transfer occur through the explicit manual follow-up.
The [Evidence management project](../plans/2026-09-08-evidence-management.md)
records the implementation sequence and acceptance checks.