merge: integrate thoth authentication
This commit is contained in:
@@ -0,0 +1,210 @@
|
||||
# Internal Qdrant and Ollama Architecture Design
|
||||
|
||||
**Status:** approved on 2026-08-08
|
||||
|
||||
## Objective
|
||||
|
||||
ThothII owns its semantic infrastructure. Every supported deployment includes a private Qdrant
|
||||
service and a private Ollama embedding service. The analytical DWH remains external and read-only;
|
||||
each workspace descriptor associates that DWH with one Qdrant collection used for database schema,
|
||||
Evidence, and approved Memory records.
|
||||
|
||||
## Decisions
|
||||
|
||||
- Qdrant replaces pgvector as the only operational vector store.
|
||||
- Ollama replaces workspace-selected external embedding endpoints.
|
||||
- The default and required model is `qwen3-embedding:0.6b` with 1024-dimensional normalized dense
|
||||
embeddings and cosine distance.
|
||||
- One Qdrant collection belongs to one workspace. Schema, Evidence, and Memory points share that
|
||||
collection and are separated by indexed payload field `kind`.
|
||||
- Qdrant and Ollama are mandatory base-Compose services. They are not published on host ports and
|
||||
are reachable only from the private Compose network.
|
||||
- Existing schema-v1 and schema-v2 descriptors remain readable for migration, but they are not
|
||||
activatable. The new operational contract is workspace schema v3.
|
||||
|
||||
The model choice is based on the published Qwen model card: the 0.6B model supports more than 100
|
||||
languages, a 32K context window, Matryoshka dimensions up to 1024, and instruction-aware retrieval.
|
||||
Ollama distributes a CPU-viable quantized build and can use an exposed GPU without changing the
|
||||
application protocol.
|
||||
|
||||
References:
|
||||
|
||||
- <https://huggingface.co/Qwen/Qwen3-Embedding-0.6B>
|
||||
- <https://ollama.com/library/qwen3-embedding>
|
||||
- <https://docs.ollama.com/capabilities/embeddings>
|
||||
- <https://qdrant.tech/documentation/installation/>
|
||||
- <https://qdrant.tech/documentation/manage-data/collections/>
|
||||
|
||||
## Target topology
|
||||
|
||||
```text
|
||||
browser -> frontend -> core -> external DWH
|
||||
-> private Qdrant
|
||||
-> private Ollama embedding
|
||||
```
|
||||
|
||||
The base Compose project contains:
|
||||
|
||||
- `frontend`: static React application and same-origin API proxy.
|
||||
- `core`: Fastify, Pi, and the Python `tht` harness.
|
||||
- `qdrant`: pinned Qdrant server with persistent `qdrant-data` volume.
|
||||
- `embedding`: pinned Ollama server with persistent `embedding-models` volume.
|
||||
- `embedding-model-init`: bounded one-shot service that pulls and verifies
|
||||
`qwen3-embedding:0.6b`; `core` starts only after it succeeds.
|
||||
|
||||
`qdrant` and `embedding` use `expose`, not `ports`. The core receives installation-owned internal
|
||||
URLs:
|
||||
|
||||
```text
|
||||
THT_INTERNAL_QDRANT_URL=http://qdrant:6333
|
||||
THT_INTERNAL_EMBEDDING_URL=http://embedding:11434
|
||||
THT_INTERNAL_EMBEDDING_MODEL=qwen3-embedding:0.6b
|
||||
THT_INTERNAL_EMBEDDING_DIMENSIONS=1024
|
||||
```
|
||||
|
||||
These are deployment facts, not workspace connector bindings. The runtime rejects non-loopback or
|
||||
non-Compose-service hosts when these variables are overridden for development.
|
||||
|
||||
An optional Linux GPU override exposes an available NVIDIA/AMD device to Ollama. The base profile
|
||||
must remain CPU-safe. macOS Docker remains CPU-only because Docker Desktop cannot expose the Apple
|
||||
GPU to an Ollama container.
|
||||
|
||||
## Workspace schema v3
|
||||
|
||||
The workspace itself is the association between the external database and the internal collection:
|
||||
|
||||
```yaml
|
||||
workspace:
|
||||
schema_version: 3
|
||||
id: psd-clinical
|
||||
name: PSD Clinical
|
||||
language: it
|
||||
|
||||
dwh:
|
||||
engine: postgres
|
||||
database: postgres
|
||||
schema: datawarehouse
|
||||
supported_transports: [postgres_direct]
|
||||
|
||||
semantic_index:
|
||||
vector_store:
|
||||
engine: qdrant
|
||||
collection: psd-clinical
|
||||
dimensions: 1024
|
||||
distance: cosine
|
||||
embedding:
|
||||
provider: ollama_internal
|
||||
model: qwen3-embedding:0.6b
|
||||
dimensions: 1024
|
||||
|
||||
llm_policy:
|
||||
allowed: [zai/glm-5.2]
|
||||
```
|
||||
|
||||
Invariants:
|
||||
|
||||
- the collection name is an explicit portable identifier;
|
||||
- active workspaces cannot share a collection;
|
||||
- vector and embedding dimensions are both 1024;
|
||||
- distance is `cosine`;
|
||||
- provider and model are exactly the supported internal values;
|
||||
- no vector transport, vector credential, embedding URL, or embedding credential may appear in a
|
||||
schema-v3 descriptor or installation contract;
|
||||
- DWH connectors remain installation-local and can still use the supported external DWH transports.
|
||||
|
||||
Schema-v1/v2 pgvector descriptors are listed as `migration_required`. Migration creates a reviewed
|
||||
schema-v3 document; it does not copy vector data implicitly. Existing semantic data is rebuilt from
|
||||
the canonical schema documents, Evidence corpus, and Memory registry.
|
||||
|
||||
## Qdrant data model
|
||||
|
||||
Each point has a deterministic UUIDv5 derived from:
|
||||
|
||||
```text
|
||||
workspace_id + kind + record_key
|
||||
```
|
||||
|
||||
The vector is the 1024-dimensional Ollama result. The payload is:
|
||||
|
||||
```json
|
||||
{
|
||||
"workspace_id": "psd-clinical",
|
||||
"kind": "schema",
|
||||
"source_id": "datawarehouse.patients",
|
||||
"record_key": "schema:table:datawarehouse.patients",
|
||||
"content_hash": "sha256:...",
|
||||
"workspace_revision": "<git commit>",
|
||||
"generation": "<optional corpus generation>",
|
||||
"language": "it",
|
||||
"text": "...",
|
||||
"metadata": {}
|
||||
}
|
||||
```
|
||||
|
||||
`kind`, `source_id`, `content_hash`, `workspace_revision`, and `generation` receive keyword payload
|
||||
indexes. Queries always filter by `workspace_id` and an explicit allowed `kind` set. Upsert is
|
||||
idempotent. Evidence generation deletion is an exact filtered delete. Collection creation is also
|
||||
idempotent and fails closed if an existing collection has incompatible dimensions or distance.
|
||||
|
||||
## Harness integration
|
||||
|
||||
The existing `VectorStore` port remains the workflow boundary. A `QdrantVectorStore` adapter maps
|
||||
its operations to Qdrant REST endpoints while preserving current schema/Evidence/Memory call sites.
|
||||
The existing Ollama embedding client is narrowed to the internal `/api/embed` contract and verifies:
|
||||
|
||||
- configured model exists;
|
||||
- output count matches input count;
|
||||
- every vector has 1024 finite numeric values;
|
||||
- no remote URL or API key is accepted.
|
||||
|
||||
The JSONL Memory registry and persisted phase documents remain canonical. Qdrant remains a derived,
|
||||
rebuildable semantic index. Schema, Evidence, and Memory ingestion all use the same point builder,
|
||||
content hashing, and retry policy.
|
||||
|
||||
## Readiness and failure behavior
|
||||
|
||||
Readiness is layered:
|
||||
|
||||
1. Compose waits for Qdrant health.
|
||||
2. Compose waits for Ollama health and successful model initialization.
|
||||
3. Workspace activation validates the schema-v3 contract.
|
||||
4. Harness readiness ensures the Qdrant collection and checks its vector configuration.
|
||||
5. Harness embeds a bounded probe and verifies 1024 dimensions.
|
||||
|
||||
Failures are sanitized and fail closed:
|
||||
|
||||
- unavailable Qdrant -> `workspace_not_activatable` before session persistence;
|
||||
- unavailable or missing Ollama model -> `model_unavailable` before session persistence;
|
||||
- collection mismatch -> `semantic_index_incompatible` without recreating or deleting data;
|
||||
- embedding dimension mismatch -> no point write;
|
||||
- partial batch failure -> operation reports failure and remains safe to retry.
|
||||
|
||||
No health response, API response, or diagnostic log exposes DWH credentials or indexed text.
|
||||
|
||||
## Deployment and migration
|
||||
|
||||
The pgvector deployment path is retired:
|
||||
|
||||
- remove local-vector Compose overlays and pgvector bootstrap/migration services;
|
||||
- remove vector PostgreSQL role and password contracts;
|
||||
- remove runtime support for vector REST/SSH and external embedding URLs;
|
||||
- keep only the descriptor parser and migration code needed to recognize legacy workspaces;
|
||||
- update local/server manuals, examples, smoke tests, CI coupling scans, backup instructions, and
|
||||
release gates for four persistent stores plus Qdrant and Ollama volumes.
|
||||
|
||||
Qdrant backup/restore uses collection snapshots or the persistent volume according to the operator
|
||||
manual. Ollama model storage is a cache: it may be backed up for offline recovery but is not an
|
||||
application source of truth.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Base local and server Compose renders include healthy private `qdrant` and `embedding` services.
|
||||
- A clean CPU-only installation downloads the model, creates a workspace collection, and embeds a
|
||||
probe without external vector or embedding configuration.
|
||||
- GPU override uses the same API and persistent model volume.
|
||||
- Schema-v3 workspaces activate; schema-v1/v2 workspaces report `migration_required`.
|
||||
- Two workspaces cannot claim the same Qdrant collection.
|
||||
- Schema, Evidence, and Memory records coexist in one collection and remain filter-isolated.
|
||||
- Existing workflow behavior and persisted session contracts remain unchanged.
|
||||
- Tests reject all active pgvector deployment, external vector binding, and external embedding
|
||||
configuration paths.
|
||||
@@ -0,0 +1,773 @@
|
||||
# Internal Qdrant and Ollama Implementation Plan
|
||||
|
||||
> **Historical nomenclature:** this plan predates the native host CLI convergence. The current
|
||||
> operator command is `tht`; any older `thothctl` smoke-script or rollback wording below is retained
|
||||
> only as historical evidence.
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Make Qdrant and Ollama mandatory internal ThothII services while keeping the analytical
|
||||
DWH external and associating each workspace with one Qdrant collection for schema, Evidence, and
|
||||
Memory embeddings.
|
||||
|
||||
**Architecture:** Introduce workspace schema v3, preserve v1/v2 only as migration inputs, and keep
|
||||
the existing harness `VectorStore` port behind a new Qdrant REST adapter. Base Compose owns Qdrant,
|
||||
Ollama, their persistent volumes, and model initialization; workspace descriptors contain semantic
|
||||
identity but no vector/embedding endpoints or credentials.
|
||||
|
||||
**Tech Stack:** TypeScript/Fastify/Zod, Python 3.12/Pydantic/requests, React 18, Docker Compose,
|
||||
Qdrant REST API, Ollama `/api/embed`, Vitest, pytest.
|
||||
|
||||
---
|
||||
|
||||
## Guardrails
|
||||
|
||||
- Apply `@superpowers:test-driven-development` to every behavior change: add one focused failing
|
||||
test, observe the expected failure, implement the minimum, and rerun the focused test.
|
||||
- Do not run broad suites until the corresponding code/config changes exist; this preserves the
|
||||
requested ordering while still using TDD.
|
||||
- Preserve the external DWH connector contract and session persistence model.
|
||||
- Do not retain an operational fallback to pgvector or an external embedding endpoint.
|
||||
- Do not delete or rewrite user workspace repositories or Qdrant data. Migration is descriptor-only;
|
||||
semantic data is rebuilt explicitly.
|
||||
- Commit after each task only when focused tests are green.
|
||||
|
||||
### Task 1: Define workspace schema v3
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `backend/src/workspaces/schema.ts`
|
||||
- Modify: `backend/src/workspaces/types.ts`
|
||||
- Modify: `backend/test/workspaces-schema.test.ts`
|
||||
- Modify: `backend/test/workspaces-migrate-legacy.test.ts`
|
||||
- Create: `backend/src/workspaces/migrate-v2-qdrant.ts`
|
||||
- Create: `backend/test/workspaces-migrate-v2-qdrant.test.ts`
|
||||
|
||||
**Step 1: Write the failing schema tests**
|
||||
|
||||
Add tests proving that schema v3 accepts only this semantic shape:
|
||||
|
||||
```ts
|
||||
const semantic_index = {
|
||||
vector_store: {
|
||||
engine: "qdrant",
|
||||
collection: "psd-clinical",
|
||||
dimensions: 1024,
|
||||
distance: "cosine",
|
||||
},
|
||||
embedding: {
|
||||
provider: "ollama_internal",
|
||||
model: "qwen3-embedding:0.6b",
|
||||
dimensions: 1024,
|
||||
},
|
||||
};
|
||||
```
|
||||
|
||||
Add separate rejection cases for `pgvector`, `supported_transports`, external embedding providers,
|
||||
non-1024 dimensions, non-cosine distance, and unknown fields. Assert v1/v2 remain parseable as
|
||||
legacy descriptors but `isOperationalWorkspace()` returns false.
|
||||
|
||||
**Step 2: Run the tests and verify RED**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
npx vitest run test/workspaces-schema.test.ts test/workspaces-migrate-v2-qdrant.test.ts
|
||||
```
|
||||
|
||||
Expected: failure because schema version 3 and `migrateWorkspaceV2ToV3` do not exist.
|
||||
|
||||
**Step 3: Implement the minimum schema and migration**
|
||||
|
||||
Add `QdrantVectorStore`, `InternalEmbedding`, and `WorkspaceV3` types. Replace the operational type
|
||||
guard with schema-v3-only semantics. Implement:
|
||||
|
||||
```ts
|
||||
export function migrateWorkspaceV2ToV3(
|
||||
legacy: WorkspaceV2,
|
||||
collection: string,
|
||||
): WorkspaceV3 {
|
||||
return validateOperationalWorkspace({
|
||||
workspace: { ...legacy.workspace, schema_version: 3 },
|
||||
dwh: legacy.dwh,
|
||||
semantic_index: {
|
||||
vector_store: {
|
||||
engine: "qdrant",
|
||||
collection,
|
||||
dimensions: 1024,
|
||||
distance: "cosine",
|
||||
},
|
||||
embedding: {
|
||||
provider: "ollama_internal",
|
||||
model: "qwen3-embedding:0.6b",
|
||||
dimensions: 1024,
|
||||
},
|
||||
},
|
||||
llm_policy: legacy.llm_policy,
|
||||
...(legacy.diagnostics?.dwh_rest
|
||||
? { diagnostics: { dwh_rest: legacy.diagnostics.dwh_rest } }
|
||||
: {}),
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
Do not copy vector/embedding diagnostics or transports.
|
||||
|
||||
**Step 4: Verify GREEN**
|
||||
|
||||
Run the command from Step 2. Expected: all selected tests pass.
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add backend/src/workspaces/schema.ts backend/src/workspaces/types.ts \
|
||||
backend/src/workspaces/migrate-v2-qdrant.ts backend/test/workspaces-schema.test.ts \
|
||||
backend/test/workspaces-migrate-legacy.test.ts backend/test/workspaces-migrate-v2-qdrant.test.ts
|
||||
git commit -m "feat: define internal semantic workspace schema"
|
||||
```
|
||||
|
||||
### Task 2: Make collection ownership unique in the Git registry
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `backend/src/workspaces/registry.ts`
|
||||
- Modify: `backend/src/workspaces/migrate-legacy.ts`
|
||||
- Modify: `backend/test/workspace-registry.test.ts`
|
||||
- Modify: `backend/test/workspaces-migrate-legacy.test.ts`
|
||||
|
||||
**Step 1: Write failing registry tests**
|
||||
|
||||
Add fixtures with two schema-v3 workspaces claiming `collection: shared`. Assert snapshot activation
|
||||
fails with `workspace_invalid` and retains the previous active snapshot. Assert v1/v2 entries are
|
||||
listed as `migration_required` and cannot be acquired with `acquireSessionRevision()`.
|
||||
|
||||
**Step 2: Verify RED**
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
npx vitest run test/workspace-registry.test.ts test/workspaces-migrate-legacy.test.ts \
|
||||
-t "collection|migration_required"
|
||||
```
|
||||
|
||||
Expected: duplicate collections are currently accepted and v2 is currently operational.
|
||||
|
||||
**Step 3: Implement uniqueness and migration state**
|
||||
|
||||
During snapshot validation, build `Map<collection, workspaceId>` for operational descriptors and
|
||||
raise a sanitized `workspace_invalid` error on a duplicate. Update migration output and CLI wording
|
||||
to require an explicit target collection and schema v3.
|
||||
|
||||
**Step 4: Verify GREEN and commit**
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
npx vitest run test/workspace-registry.test.ts test/workspaces-migrate-legacy.test.ts \
|
||||
-t "collection|migration_required"
|
||||
cd ..
|
||||
git add backend/src/workspaces/registry.ts backend/src/workspaces/migrate-legacy.ts \
|
||||
backend/test/workspace-registry.test.ts backend/test/workspaces-migrate-legacy.test.ts
|
||||
git commit -m "feat: reserve one qdrant collection per workspace"
|
||||
```
|
||||
|
||||
### Task 3: Remove external semantic bindings and render internal endpoints
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `backend/src/workspaces/contracts.ts`
|
||||
- Modify: `backend/src/workspaces/bindings.ts`
|
||||
- Modify: `backend/src/workspaces/runtime-renderer.ts`
|
||||
- Modify: `backend/src/config.ts`
|
||||
- Modify: `backend/test/workspaces-contracts.test.ts`
|
||||
- Modify: `backend/test/workspaces-bindings.test.ts`
|
||||
- Modify: `backend/test/workspace-runtime-renderer.test.ts`
|
||||
- Modify: `backend/test/config.test.ts`
|
||||
|
||||
**Step 1: Write failing contract tests**
|
||||
|
||||
Assert schema-v3 installation contracts contain DWH variables only. Assert environment variables
|
||||
matching `*_VECTOR_*`, `*_EMBEDDING_BASE_URL`, or semantic API-key suffixes are ignored/rejected.
|
||||
Assert the rendered harness config always contains:
|
||||
|
||||
```yaml
|
||||
resources:
|
||||
vector:
|
||||
engine: qdrant
|
||||
base_url: http://qdrant:6333
|
||||
collection: psd-clinical
|
||||
embeddings:
|
||||
provider: ollama_internal
|
||||
base_url: http://embedding:11434
|
||||
model: qwen3-embedding:0.6b
|
||||
dimensions: 1024
|
||||
```
|
||||
|
||||
**Step 2: Verify RED**
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
npx vitest run test/workspaces-contracts.test.ts test/workspaces-bindings.test.ts \
|
||||
test/workspace-runtime-renderer.test.ts test/config.test.ts
|
||||
```
|
||||
|
||||
Expected: current contracts require external vector and embedding bindings.
|
||||
|
||||
**Step 3: Implement internal runtime configuration**
|
||||
|
||||
Add typed backend config fields with Compose defaults:
|
||||
|
||||
```ts
|
||||
internalQdrantUrl: "http://qdrant:6333"
|
||||
internalEmbeddingUrl: "http://embedding:11434"
|
||||
internalEmbeddingModel: "qwen3-embedding:0.6b"
|
||||
internalEmbeddingDimensions: 1024
|
||||
```
|
||||
|
||||
Accept only `qdrant`, `embedding`, `localhost`, or loopback hosts. Keep these values out of Git
|
||||
workspace descriptors, API payloads, and generated installation docs. Render them into the
|
||||
ephemeral backend-owned harness config after descriptor validation.
|
||||
|
||||
**Step 4: Verify GREEN and commit**
|
||||
|
||||
Run Step 2, then:
|
||||
|
||||
```bash
|
||||
git add backend/src/config.ts backend/src/workspaces/contracts.ts backend/src/workspaces/bindings.ts \
|
||||
backend/src/workspaces/runtime-renderer.ts backend/test/config.test.ts \
|
||||
backend/test/workspaces-contracts.test.ts backend/test/workspaces-bindings.test.ts \
|
||||
backend/test/workspace-runtime-renderer.test.ts
|
||||
git commit -m "feat: render private semantic service endpoints"
|
||||
```
|
||||
|
||||
### Task 4: Narrow harness embedding configuration to internal Ollama
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `harness/tht/config.py`
|
||||
- Modify: `harness/tht/config_compat.py`
|
||||
- Modify: `harness/tht/vectorstore/embeddings.py`
|
||||
- Modify: `harness/tht/cli/ollama_cmd.py`
|
||||
- Modify: `harness/tests/test_config_resources.py`
|
||||
- Create: `harness/tests/test_internal_embeddings.py`
|
||||
|
||||
**Step 1: Write failing embedding tests**
|
||||
|
||||
Use a fake `requests.Session` to prove `OllamaInternalEmbeddings.embed()` calls `/api/embed` with
|
||||
model and batch input, returns 1024-dimensional finite vectors, and rejects count/dimension/NaN
|
||||
mismatches. Add config tests rejecting external providers, API keys, and non-private base URLs.
|
||||
|
||||
**Step 2: Verify RED**
|
||||
|
||||
```bash
|
||||
cd harness
|
||||
.venv/bin/pytest tests/test_internal_embeddings.py tests/test_config_resources.py -q
|
||||
```
|
||||
|
||||
Expected: `OllamaInternalEmbeddings` and internal-only config do not exist.
|
||||
|
||||
**Step 3: Implement the client**
|
||||
|
||||
Implement one bounded `/api/embed` request per batch:
|
||||
|
||||
```python
|
||||
response = self._session.post(
|
||||
f"{self.base_url}/api/embed",
|
||||
json={"model": self.model, "input": texts},
|
||||
timeout=self.timeout,
|
||||
)
|
||||
```
|
||||
|
||||
Validate response shape before returning any vector. Keep retry behavior bounded and sanitize URLs
|
||||
and response bodies from raised errors.
|
||||
|
||||
**Step 4: Verify GREEN and commit**
|
||||
|
||||
```bash
|
||||
cd harness
|
||||
.venv/bin/pytest tests/test_internal_embeddings.py tests/test_config_resources.py -q
|
||||
cd ..
|
||||
git add harness/tht/config.py harness/tht/config_compat.py harness/tht/vectorstore/embeddings.py \
|
||||
harness/tht/cli/ollama_cmd.py harness/tests/test_config_resources.py \
|
||||
harness/tests/test_internal_embeddings.py
|
||||
git commit -m "feat: use internal ollama embeddings"
|
||||
```
|
||||
|
||||
### Task 5: Implement the Qdrant VectorStore adapter
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `harness/tht/adapters/vector/qdrant.py`
|
||||
- Modify: `harness/tht/adapters/vector/__init__.py`
|
||||
- Modify: `harness/tht/ports/vector.py`
|
||||
- Modify: `harness/tht/vectorstore/records.py`
|
||||
- Modify: `harness/tht/vectorstore/store.py`
|
||||
- Create: `harness/tests/test_qdrant_vector_store.py`
|
||||
- Modify: `harness/tests/test_vector_port_contract.py`
|
||||
|
||||
**Step 1: Write failing adapter tests**
|
||||
|
||||
Test a real adapter against a deterministic fake HTTP server. Cover:
|
||||
|
||||
- idempotent collection create with 1024/Cosine;
|
||||
- mismatch fails without delete/recreate;
|
||||
- keyword payload-index creation;
|
||||
- deterministic UUIDv5 point IDs;
|
||||
- upsert payload for `schema`, `evidence`, and `memory`;
|
||||
- query filtered by workspace and allowed kinds;
|
||||
- `existing_hashes`, exact Evidence generation list/delete, and health;
|
||||
- sanitized timeouts and malformed responses.
|
||||
|
||||
The point ID helper must satisfy:
|
||||
|
||||
```python
|
||||
def point_id(workspace_id: str, kind: str, record_key: str) -> str:
|
||||
return str(uuid5(NAMESPACE_URL, f"thothii:{workspace_id}:{kind}:{record_key}"))
|
||||
```
|
||||
|
||||
**Step 2: Verify RED**
|
||||
|
||||
```bash
|
||||
cd harness
|
||||
.venv/bin/pytest tests/test_qdrant_vector_store.py tests/test_vector_port_contract.py -q
|
||||
```
|
||||
|
||||
Expected: import failure for the Qdrant adapter.
|
||||
|
||||
**Step 3: Implement minimal REST mappings**
|
||||
|
||||
Use existing `requests` dependency and these endpoints:
|
||||
|
||||
```text
|
||||
GET /collections/{collection}
|
||||
PUT /collections/{collection}
|
||||
PUT /collections/{collection}/index
|
||||
PUT /collections/{collection}/points?wait=true
|
||||
POST /collections/{collection}/points/query
|
||||
POST /collections/{collection}/points/scroll
|
||||
POST /collections/{collection}/points/delete?wait=true
|
||||
```
|
||||
|
||||
Every operation must include the workspace filter even though the collection is workspace-owned.
|
||||
Map Qdrant scores and payloads back into existing `VectorHit` objects.
|
||||
|
||||
**Step 4: Verify GREEN and commit**
|
||||
|
||||
```bash
|
||||
cd harness
|
||||
.venv/bin/pytest tests/test_qdrant_vector_store.py tests/test_vector_port_contract.py -q
|
||||
cd ..
|
||||
git add harness/tht/adapters/vector/qdrant.py harness/tht/adapters/vector/__init__.py \
|
||||
harness/tht/ports/vector.py harness/tht/vectorstore/records.py \
|
||||
harness/tht/vectorstore/store.py harness/tests/test_qdrant_vector_store.py \
|
||||
harness/tests/test_vector_port_contract.py
|
||||
git commit -m "feat: add qdrant vector adapter"
|
||||
```
|
||||
|
||||
### Task 6: Wire schema, Evidence, and Memory through Qdrant
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `harness/tht/vectorstore/reader.py`
|
||||
- Modify: `harness/tht/cli/vector_cmd.py`
|
||||
- Modify: `harness/tht/cli/memory_cmd.py`
|
||||
- Modify: `harness/tht/corpus/pipeline.py`
|
||||
- Modify: `harness/tht/search/evidence.py`
|
||||
- Modify: `harness/tht/cli/schema_cmd.py`
|
||||
- Modify: `harness/tests/test_memory_save_one.py`
|
||||
- Modify: `harness/tests/test_search_pack.py`
|
||||
- Create: `harness/tests/test_semantic_kind_isolation.py`
|
||||
|
||||
**Step 1: Write failing integration tests**
|
||||
|
||||
Use an in-memory fake implementing the `VectorStore` port. Assert:
|
||||
|
||||
- schema records use `kind=schema`;
|
||||
- corpus records use `kind=evidence` and exact generation;
|
||||
- approved memories use `kind=memory`;
|
||||
- search pack requests only its allowed kind set;
|
||||
- all three paths share `workspace_id`, `workspace_revision`, hashing, and point-key construction;
|
||||
- retries do not duplicate points.
|
||||
|
||||
**Step 2: Verify RED**
|
||||
|
||||
```bash
|
||||
cd harness
|
||||
.venv/bin/pytest tests/test_semantic_kind_isolation.py tests/test_memory_save_one.py \
|
||||
tests/test_search_pack.py -q
|
||||
```
|
||||
|
||||
Expected: current factories select pgvector/HTTP adapters and payloads lack the v3 identity fields.
|
||||
|
||||
**Step 3: Wire the adapter**
|
||||
|
||||
Make schema-v3 `qdrant` the only operational vector factory branch. Reuse the current canonical
|
||||
record builders; add only missing identity fields. Keep the JSONL Memory registry and filesystem
|
||||
Evidence corpus as sources of truth.
|
||||
|
||||
**Step 4: Verify GREEN and commit**
|
||||
|
||||
Run Step 2, then commit the listed files with:
|
||||
|
||||
```bash
|
||||
git commit -m "feat: index semantic records in qdrant"
|
||||
```
|
||||
|
||||
### Task 7: Add mandatory Qdrant and Ollama Compose services
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `compose.yaml`
|
||||
- Create: `deploy/compose.embedding-gpu.yaml`
|
||||
- Create: `docker/embedding-model-init.sh`
|
||||
- Modify: `docker/core.Dockerfile`
|
||||
- Modify: `deploy/env/local.env.example`
|
||||
- Modify: `deploy/env/server.env.example`
|
||||
- Modify: `scripts/run-stack.sh`
|
||||
- Modify: `scripts/test-default-compose.sh`
|
||||
- Modify: `scripts/test-unified-compose.sh`
|
||||
- Create: `scripts/test-internal-semantic-compose.sh`
|
||||
|
||||
**Step 1: Write failing Compose contract tests**
|
||||
|
||||
Assert the rendered base profile has `core`, `frontend`, `qdrant`, `embedding`, and
|
||||
`embedding-model-init`; private services have no published ports; persistent volumes exist; core
|
||||
depends on Qdrant health and successful model init; no external vector/embedding binding is required.
|
||||
|
||||
Also assert all service images use version plus immutable digest. Resolve and record supported
|
||||
multi-architecture digests for Qdrant v1.18.x and Ollama v0.32.x during implementation:
|
||||
|
||||
```bash
|
||||
docker buildx imagetools inspect qdrant/qdrant:v1.18.2
|
||||
docker buildx imagetools inspect ollama/ollama:0.32.0
|
||||
```
|
||||
|
||||
**Step 2: Verify RED**
|
||||
|
||||
```bash
|
||||
./scripts/test-default-compose.sh
|
||||
./scripts/test-unified-compose.sh
|
||||
./scripts/test-internal-semantic-compose.sh
|
||||
```
|
||||
|
||||
Expected: required services and volumes are absent.
|
||||
|
||||
**Step 3: Implement the services**
|
||||
|
||||
`embedding-model-init.sh` must wait with a bounded deadline, call `ollama pull` for the exact model,
|
||||
and verify it appears in `/api/tags`. The Qdrant healthcheck uses its HTTP health endpoint. The CPU
|
||||
base has no device reservation; the GPU override adds only the supported device stanza.
|
||||
|
||||
**Step 4: Verify GREEN and commit**
|
||||
|
||||
Run Step 2, then:
|
||||
|
||||
```bash
|
||||
git add compose.yaml deploy/compose.embedding-gpu.yaml docker/embedding-model-init.sh \
|
||||
docker/core.Dockerfile deploy/env/local.env.example deploy/env/server.env.example \
|
||||
scripts/run-stack.sh scripts/test-default-compose.sh scripts/test-unified-compose.sh \
|
||||
scripts/test-internal-semantic-compose.sh
|
||||
git commit -m "feat: run qdrant and ollama inside thothii"
|
||||
```
|
||||
|
||||
### Task 8: Retire pgvector deployment and external semantic connectors
|
||||
|
||||
**Files:**
|
||||
|
||||
- Delete: `deploy/compose.local-vector.yaml`
|
||||
- Delete: `deploy/compose.preprocess-local-vector.yaml`
|
||||
- Delete: `deploy/sql/20-vector-roles.sql`
|
||||
- Delete: `deploy/vector/reconcile-roles.sh`
|
||||
- Delete: `deploy/vector/rotate-bootstrap-password.py`
|
||||
- Delete: `deploy/vector/secret-policy.sh`
|
||||
- Delete: `deploy/vector/vector-db-entrypoint.sh`
|
||||
- Delete: `scripts/local-vector-smoke.sh`
|
||||
- Delete: `scripts/test-local-vector-smoke-safety.sh`
|
||||
- Delete: `scripts/test-local-vector-smoke-live-collision.sh`
|
||||
- Delete: `scripts/test-vector-bootstrap-rotation.sh`
|
||||
- Delete: `scripts/test-vector-migration-image.sh`
|
||||
- Delete: `scripts/test-vector-secret-policy.sh`
|
||||
- Modify: `scripts/test-no-deployment-coupling.sh`
|
||||
- Modify: `scripts/test-no-deployment-coupling-scope.sh`
|
||||
- Modify: `scripts/test-compose-secret-policy.sh`
|
||||
- Modify: `.github/workflows/deployment.yml`
|
||||
|
||||
**Step 1: Write the failing coupling test**
|
||||
|
||||
Teach the coupling gate to reject active `pgvector`, `local-vector`, `THT_VECTOR_*`, workspace
|
||||
embedding URLs/API keys, and external vector transports while allowing historical specs and the
|
||||
explicit descriptor migration module.
|
||||
|
||||
**Step 2: Verify RED**
|
||||
|
||||
```bash
|
||||
./scripts/test-no-deployment-coupling-scope.sh
|
||||
./scripts/test-no-deployment-coupling.sh
|
||||
./scripts/test-compose-secret-policy.sh
|
||||
```
|
||||
|
||||
Expected: active pgvector deployment paths are reported.
|
||||
|
||||
**Step 3: Remove the retired paths and update CI**
|
||||
|
||||
Remove only repository deployment machinery. Retain harness pgvector code temporarily only if it
|
||||
is needed to read/export legacy data during migration; it must not be reachable from schema v3 or
|
||||
Compose. Remove it in a follow-up task once migration fixtures no longer import it.
|
||||
|
||||
**Step 4: Verify GREEN and commit**
|
||||
|
||||
Run Step 2 and the workflow fixture tests, then commit all deletions and modifications:
|
||||
|
||||
```bash
|
||||
git add -A deploy scripts .github/workflows/deployment.yml
|
||||
git commit -m "refactor: retire external vector deployment"
|
||||
```
|
||||
|
||||
### Task 9: Update frontend workspace editing and examples
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `frontend/src/api/workspaces.ts`
|
||||
- Modify: `frontend/src/shell/WorkspaceEditor.tsx`
|
||||
- Modify: `frontend/src/shell/WorkspaceEditor.test.tsx`
|
||||
- Modify: `frontend/src/shell/WorkspaceManager.test.tsx`
|
||||
- Modify: `frontend/src/api/workspaces.test.ts`
|
||||
- Modify: `frontend/src/workspaces/drafts.test.ts`
|
||||
- Modify: `deploy/workspaces/example.yaml`
|
||||
- Modify: `deploy/workspaces/psd.yaml.example`
|
||||
|
||||
**Step 1: Write failing UI tests**
|
||||
|
||||
Assert editor/preview show Qdrant collection and fixed internal embedding model, expose no vector
|
||||
endpoint/credential fields, and publish schema v3. Assert legacy descriptors display a migration
|
||||
banner and cannot be selected for a new session.
|
||||
|
||||
**Step 2: Verify RED**
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
npx vitest run src/shell/WorkspaceEditor.test.tsx src/shell/WorkspaceManager.test.tsx \
|
||||
src/api/workspaces.test.ts src/workspaces/drafts.test.ts
|
||||
```
|
||||
|
||||
Expected: fixtures and controls still use pgvector/external embedding.
|
||||
|
||||
**Step 3: Implement fixed semantic controls**
|
||||
|
||||
Collection remains editable and validated. Engine, provider, model, dimensions, and distance render
|
||||
as fixed architecture values. Remove external semantic diagnostics from drafts and publish payloads.
|
||||
|
||||
**Step 4: Verify GREEN and commit**
|
||||
|
||||
Run Step 2, then commit the listed files with:
|
||||
|
||||
```bash
|
||||
git commit -m "feat: edit qdrant workspace collections"
|
||||
```
|
||||
|
||||
### Task 10: Add a real internal semantic smoke
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `scripts/internal-semantic-smoke.sh`
|
||||
- Modify: `scripts/unified-deployment-smoke.sh`
|
||||
- Modify: `scripts/server-deployment-smoke.sh`
|
||||
- Modify: `scripts/task13-runtime-fixture-check.ts`
|
||||
- Modify: `scripts/test-task13-runtime-fixtures.sh`
|
||||
|
||||
**Step 1: Write failing smoke fixture assertions**
|
||||
|
||||
The fixture must require private Qdrant/Ollama services, model volume, Qdrant volume, fixed internal
|
||||
URLs, and no host ports. It must reject wrong service names, external URLs, collection reuse, and
|
||||
dimension changes.
|
||||
|
||||
**Step 2: Verify RED**
|
||||
|
||||
```bash
|
||||
./scripts/test-task13-runtime-fixtures.sh local
|
||||
./scripts/test-task13-runtime-fixtures.sh server
|
||||
```
|
||||
|
||||
Expected: current fixture expects the two-service topology.
|
||||
|
||||
**Step 3: Implement the live smoke**
|
||||
|
||||
Using disposable volumes and a fixture workspace, start the stack on CPU, wait for the model, ensure
|
||||
the collection, embed one record of each kind, query each kind with filters, restart offline, and
|
||||
prove all points and the model remain available. Cleanup must remain exact and must not prune global
|
||||
Docker resources.
|
||||
|
||||
**Step 4: Verify GREEN and commit**
|
||||
|
||||
```bash
|
||||
./scripts/test-task13-runtime-fixtures.sh local
|
||||
./scripts/test-task13-runtime-fixtures.sh server
|
||||
./scripts/internal-semantic-smoke.sh
|
||||
git add scripts/internal-semantic-smoke.sh scripts/unified-deployment-smoke.sh \
|
||||
scripts/server-deployment-smoke.sh scripts/task13-runtime-fixture-check.ts \
|
||||
scripts/test-task13-runtime-fixtures.sh
|
||||
git commit -m "test: cover internal semantic services"
|
||||
```
|
||||
|
||||
### Task 11: Update operator documentation and state
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `README.md`
|
||||
- Modify: `AGENTS.md`
|
||||
- Modify: `PROJECT_STATE.md`
|
||||
- Modify: `docs/install/local-workspace-registry.md`
|
||||
- Modify: `docs/install/server-workspace-registry.md`
|
||||
- Modify: `docs/installazione-docker-4-contesti.md`
|
||||
- Modify: `docs/workspace-diagnostic-protocol.md`
|
||||
- Modify: `docs/gestione-memory.md`
|
||||
- Modify: `deploy/secrets/README.md`
|
||||
- Modify: `scripts/verify-workspace-install-docs.sh`
|
||||
- Modify: `scripts/test-verify-workspace-install-docs.sh`
|
||||
|
||||
**Step 1: Write failing documentation contract assertions**
|
||||
|
||||
Require the four-service topology, CPU/GPU behavior, volume backup/restore, schema-v3 migration,
|
||||
Qdrant collection ownership, and removal of external vector/embedding variables from active manuals.
|
||||
|
||||
**Step 2: Verify RED**
|
||||
|
||||
```bash
|
||||
./scripts/test-verify-workspace-install-docs.sh
|
||||
./scripts/verify-workspace-install-docs.sh --fixtures-only
|
||||
```
|
||||
|
||||
Expected: manuals still describe external pgvector/embedding and a two-service mandatory stack.
|
||||
|
||||
**Step 3: Update documentation**
|
||||
|
||||
Document Qdrant as a derived but persistent index, Ollama model cache behavior, CPU-first startup,
|
||||
optional GPU override, snapshot/restore, explicit legacy migration, and the fact that only the DWH
|
||||
and LLM remain external application endpoints.
|
||||
|
||||
**Step 4: Verify GREEN and commit**
|
||||
|
||||
Run Step 2, then:
|
||||
|
||||
```bash
|
||||
git add README.md AGENTS.md PROJECT_STATE.md docs deploy/secrets/README.md \
|
||||
scripts/verify-workspace-install-docs.sh scripts/test-verify-workspace-install-docs.sh
|
||||
git commit -m "docs: document internal semantic infrastructure"
|
||||
```
|
||||
|
||||
### Task 12: Remove unreachable pgvector runtime code
|
||||
|
||||
**Files:**
|
||||
|
||||
- Delete: `harness/tht/adapters/vector/pgvector.py`
|
||||
- Delete: `harness/tht/adapters/vector/legacy_direct.py`
|
||||
- Delete: `harness/tht/adapters/vector/thoth_http.py`
|
||||
- Delete: `harness/tht/vectorstore/rest_client.py`
|
||||
- Delete: `harness/tht/vectorstore/rest_writer.py`
|
||||
- Delete: `harness/tht/migrations/vector/001_extensions.sql`
|
||||
- Delete: `harness/tht/migrations/vector/002_schema_tables.sql`
|
||||
- Delete: `harness/tht/migrations/vector/003_roles.sql`
|
||||
- Delete: `harness/tht/migrations/vector/004_evidence_generation_gc.sql`
|
||||
- Modify: `harness/pyproject.toml`
|
||||
- Modify/Delete: affected pgvector and migration tests under `harness/tests/l0/`
|
||||
|
||||
**Step 1: Prove the code is unreachable**
|
||||
|
||||
```bash
|
||||
rg -n "PgVectorStore|ThothHttpVectorStore|LegacyDirectVectorStore|migrations/vector" \
|
||||
harness backend frontend compose.yaml deploy scripts docker docs \
|
||||
--glob '!docs/plans/**' --glob '!docs/superpowers/**'
|
||||
```
|
||||
|
||||
Expected before cleanup: matches only in the files scheduled for deletion and legacy tests. If an
|
||||
operational call site remains, stop and migrate it before deleting anything.
|
||||
|
||||
**Step 2: Delete obsolete runtime and tests**
|
||||
|
||||
Retain descriptor migration tests, but remove PostgreSQL vector runtime/migration packaging tests.
|
||||
Remove `psycopg2-binary` only if the DWH/session PostgreSQL paths do not need it; otherwise keep it.
|
||||
|
||||
**Step 3: Verify focused imports and packaging**
|
||||
|
||||
```bash
|
||||
cd harness
|
||||
.venv/bin/pytest tests/test_qdrant_vector_store.py tests/test_vector_port_contract.py \
|
||||
tests/test_semantic_kind_isolation.py tests/test_vector_migration_packaging.py -q
|
||||
python -m build
|
||||
```
|
||||
|
||||
Expected: Qdrant tests pass and the wheel contains no pgvector migrations. Adjust the packaging test
|
||||
to assert Qdrant has no SQL migration payload.
|
||||
|
||||
**Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add -A harness
|
||||
git commit -m "refactor: remove pgvector runtime"
|
||||
```
|
||||
|
||||
### Task 13: Run complete verification
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify only if a genuine regression is discovered.
|
||||
|
||||
**Step 1: Deterministic layer gates**
|
||||
|
||||
```bash
|
||||
cd harness && .venv/bin/pytest -q && .venv/bin/ruff check .
|
||||
cd ../backend && npx vitest run && npx tsc --noEmit -p . && npm run build
|
||||
cd ../frontend && npx vitest run && npx tsc -b && npm run build
|
||||
cd .. && git diff --check
|
||||
```
|
||||
|
||||
Expected: all gates pass. Existing unrelated Ruff debt must be reported separately if it remains;
|
||||
new/modified files must be Ruff-clean.
|
||||
|
||||
**Step 2: Deployment contracts**
|
||||
|
||||
```bash
|
||||
./scripts/test-default-compose.sh
|
||||
./scripts/test-unified-compose.sh
|
||||
./scripts/test-internal-semantic-compose.sh
|
||||
./scripts/test-no-deployment-coupling.sh
|
||||
./scripts/test-compose-secret-policy.sh
|
||||
./scripts/verify-workspace-install-docs.sh --fixtures-only
|
||||
```
|
||||
|
||||
Expected: all pass without external vector/embedding settings.
|
||||
|
||||
**Step 3: Docker smokes**
|
||||
|
||||
```bash
|
||||
./scripts/internal-semantic-smoke.sh
|
||||
./scripts/workspace-registry-smoke.sh
|
||||
./scripts/unified-deployment-smoke.sh
|
||||
./scripts/thothctl-update-smoke.sh
|
||||
./scripts/server-deployment-smoke.sh
|
||||
```
|
||||
|
||||
Expected: CPU semantic smoke passes, persistence survives offline restart, and every script proves
|
||||
exact cleanup. Investigate the previously observed `thothctl` rollback failure independently if it
|
||||
recurs; do not weaken the new semantic gate to hide it.
|
||||
|
||||
**Step 4: Final audit**
|
||||
|
||||
```bash
|
||||
rg -n "pgvector|local-vector|THT_VECTOR_|EMBEDDING_BASE_URL|openai_compatible|ollama_compatible" \
|
||||
. --glob '!docs/plans/**' --glob '!docs/superpowers/**' --glob '!**/node_modules/**' \
|
||||
--glob '!**/.venv/**' --glob '!**/.git/**'
|
||||
git status --short
|
||||
```
|
||||
|
||||
Expected: no active operational references; only explicit legacy descriptor migration fixtures may
|
||||
remain. Worktree contains only intentional changes.
|
||||
|
||||
**Step 5: Commit verification metadata**
|
||||
|
||||
Update `PROJECT_STATE.md` with exact counts, image digests, smoke durations, CPU hardware, and any
|
||||
manual GPU/Windows gates. Commit only verified claims:
|
||||
|
||||
```bash
|
||||
git add PROJECT_STATE.md
|
||||
git commit -m "docs: record qdrant ollama verification"
|
||||
```
|
||||
@@ -0,0 +1,447 @@
|
||||
# Piano di implementazione: workspace descriptor esclusivamente schema v3
|
||||
|
||||
> **Per gli agenti esecutori:** SUB-SKILL OBBLIGATORIA: usare `superpowers:subagent-driven-development` (raccomandata) oppure `superpowers:executing-plans`, procedendo task per task con TDD e review tra i task.
|
||||
|
||||
**Obiettivo:** rimuovere dal prodotto ogni capacità di leggere, migrare, rendere operativo o presentare workspace descriptor schema v1/v2. Il solo descriptor accettato diventa schema v3. Restano intatti i formati versionati non correlati e gli state file del registry già prodotti da versioni recenti con revisioni v3.
|
||||
|
||||
**Architettura:** parser, registry, renderer, diagnostica, route e frontend convergono su un solo tipo `WorkspaceV3`. Il campo pubblico `WorkspaceRevision.state` scompare. Un decoder privato normalizza in memoria gli state file già scritti con `state: "operational"`, elimina quel campo prima di qualsiasi uso/API e rifiuta ogni combinazione non-v3 o incoerente. I build backend diventano clean-first, così la cancellazione dei migratori sorgente implica anche la loro assenza da `dist` e dall'immagine core.
|
||||
|
||||
**Tech stack:** TypeScript 5, Zod 4, Fastify 5, React 18, Vitest, Node.js 22, Bash/PowerShell, Git e Docker Compose.
|
||||
|
||||
**Stato:** piano revisionato dopo review indipendente. La sua approvazione non autorizza l'implementazione; attendere un esplicito ordine separato.
|
||||
|
||||
---
|
||||
|
||||
## Decisioni confermate
|
||||
|
||||
1. Nessun workspace v1/v2 reale deve essere preservato o migrato.
|
||||
2. Eliminare `migrate-legacy.ts`, `migrate-v2-qdrant.ts` e le relative interfacce CLI.
|
||||
3. Eliminare il campo `state` dal tipo/API `WorkspaceRevision` e da tutti i nuovi state/manifest del registry.
|
||||
4. Descriptor v1/v2 presenti in Git o negli snapshot vengono rifiutati, senza conversione automatica.
|
||||
5. Non toccare i documenti storici sotto `docs/superpowers/` e i vecchi piani; possono descrivere decisioni passate.
|
||||
6. Non iniziare P2 finché P1 non dispone di nuova evidenza automatica e di una nuova decisione manuale esplicita.
|
||||
|
||||
## Confini da non oltrepassare
|
||||
|
||||
Questa rimozione riguarda soltanto il **workspace descriptor**. Non eliminare o rinominare:
|
||||
|
||||
- `schemaVersion`/`schema_version` di bundle ZIP, report, job, ledger, manifest di sessione o artifact di fase;
|
||||
- `RevisionLeaseRecord.state` (`creating`/`persisted`), maintenance state, process state o UI state non collegati a `WorkspaceRevision`;
|
||||
- `migration_required` usato nei futuri piani P3–P6 per ownership DWH, punti semantici revisionless o altre migrazioni non-descriptor;
|
||||
- `allowLegacy` del frontend sessioni, che significa “sessione senza revisione workspace” e non descriptor v1/v2;
|
||||
- documenti storici o report conservati.
|
||||
|
||||
L'unica compatibilità legacy mantenuta nel codice è il decoder privato degli state file già scritti con il campo revisionale `state: "operational"`. Non costituisce supporto a descriptor v1/v2.
|
||||
|
||||
## Contratto v3-only
|
||||
|
||||
- `WorkspaceDescriptor`, `CanonicalWorkspace` e `WorkspaceV3` rappresentano la stessa forma v3; mantenere gli alias soltanto quando migliorano la semantica dei confini.
|
||||
- `parseWorkspaceYaml` e `validateWorkspaceDescriptor` accettano esclusivamente `workspace.schema_version === 3`.
|
||||
- v1/v2 generano l'errore pubblico già sanitizzato `workspace_invalid`; non usare più il messaggio o lo stato `migration_required` per i descriptor.
|
||||
- Un'attivazione Git contenente anche un solo descriptor non-v3 fallisce interamente e conserva il precedente active state.
|
||||
- Le revisioni restituite dalle API contengono esattamente `id`, `commit`, `blob`, `snapshotPath`, senza `state`.
|
||||
- Nuovi `active.json` e `snapshot.json` non contengono `state` nelle revisioni.
|
||||
|
||||
## Compatibilità degli state file esistenti
|
||||
|
||||
Definire due decoder stretti e distinti:
|
||||
|
||||
```ts
|
||||
interface StoredWorkspaceRevision {
|
||||
id: string;
|
||||
commit: string;
|
||||
blob: string;
|
||||
snapshotPath: string;
|
||||
state?: "operational"; // solo input compatibile; mai restituito
|
||||
}
|
||||
|
||||
interface WorkspaceRevision {
|
||||
id: string;
|
||||
commit: string;
|
||||
blob: string;
|
||||
snapshotPath: string;
|
||||
}
|
||||
```
|
||||
|
||||
Regole:
|
||||
|
||||
1. `active.json` accetta soltanto `{head,revisions}`; `snapshot.json` soltanto `{head,revisions,files}`.
|
||||
2. Ogni revision object accetta soltanto i quattro campi correnti più l'opzionale vecchio `state: "operational"`.
|
||||
3. `state: "migration_required"`, qualsiasi altro valore o campo sconosciuto è rifiutato.
|
||||
4. Il decoder ricostruisce un nuovo oggetto `WorkspaceRevision`; non restituisce mai l'oggetto JSON originale.
|
||||
5. Active state e snapshot manifest vengono confrontati dopo la normalizzazione.
|
||||
6. L'integrità continua a validare path, commit, blob, digest, descriptor v3 e Evidence context.
|
||||
7. La lettura non modifica snapshot storici. La successiva attivazione riscrive `active.json` nel formato corrente; tutti i nuovi snapshot sono state-free.
|
||||
8. Un vecchio file già privo di `state` è naturalmente il formato corrente, ma il relativo descriptor deve comunque essere v3.
|
||||
|
||||
## Mappa completa dei file
|
||||
|
||||
### Backend produttivo
|
||||
|
||||
- `backend/src/workspaces/schema.ts`
|
||||
- `backend/src/workspaces/types.ts`
|
||||
- `backend/src/workspaces/runtime-renderer.ts`
|
||||
- `backend/src/workspaces/contracts.ts`
|
||||
- `backend/src/workspaces/diagnostics.ts`
|
||||
- `backend/src/workspaces/bindings.ts`
|
||||
- `backend/src/workspaces/registry.ts`
|
||||
- `backend/src/routes/workspaces.ts`
|
||||
- `backend/src/routes/sessions.ts`
|
||||
- `backend/src/routes/sql.ts`
|
||||
- Eliminare `backend/src/workspaces/migrate-legacy.ts`
|
||||
- Eliminare `backend/src/workspaces/migrate-v2-qdrant.ts`
|
||||
|
||||
### Build e tooling P1
|
||||
|
||||
- `backend/package.json`
|
||||
- Creare `backend/scripts/clean-dist.mjs`
|
||||
- Creare un test Node per il clean build
|
||||
- `backend/scripts/p1-manual-acceptance.mjs`
|
||||
- `backend/scripts/p1-manual-acceptance.test.mjs`
|
||||
- `backend/scripts/p1-render-snapshot.test.mjs`
|
||||
|
||||
### Frontend
|
||||
|
||||
- `frontend/src/api/workspaces.ts`
|
||||
- `frontend/src/api/sessions.ts`
|
||||
- `frontend/src/shell/SteerInput.tsx`
|
||||
- `frontend/src/shell/WorkspaceManager.tsx`
|
||||
- Test/fixture in `api`, `SteerInput`, `WorkspaceManager`, `NewSessionDialog`, `WorkspacePublishDialog` e `drafts`.
|
||||
|
||||
### Deploy, fixture e verificatori
|
||||
|
||||
- `scripts/workspace-registry-smoke.sh`
|
||||
- Creare `scripts/fixtures/workspace-registry-smoke.yaml`
|
||||
- `scripts/test-no-deployment-coupling-scope.sh`
|
||||
- `scripts/test-windows-clone-contract.ps1`
|
||||
- `scripts/verify-workspace-install-docs.sh`
|
||||
- `scripts/test-verify-workspace-install-docs.sh`
|
||||
|
||||
### Documentazione corrente
|
||||
|
||||
- `README.md`
|
||||
- sezione corrente di `PROJECT_STATE.md`, prima di `## Historical snapshots`
|
||||
- `docs/workspace-diagnostic-protocol.md`
|
||||
- `docs/install/local-workspace-registry.md`
|
||||
- `docs/install/server-workspace-registry.md`
|
||||
|
||||
---
|
||||
|
||||
### Task 0: Congelare scope e baseline prima delle modifiche
|
||||
|
||||
**File:** nessuna modifica produttiva.
|
||||
|
||||
- [ ] Registrare `BASE_SHA=$(git rev-parse HEAD)` e verificare che gli altri piani non vengano inclusi nei commit di implementazione.
|
||||
- [ ] Salvare l'inventario iniziale dei simboli descriptor-legacy:
|
||||
|
||||
```bash
|
||||
git grep -nE 'WorkspaceV1|WorkspaceV2|LegacyWorkspace|migration_required|migrate-legacy|migrateWorkspaceV1ToV2|migrateWorkspaceV2ToV3' -- \
|
||||
backend/src backend/test backend/scripts frontend/src scripts README.md PROJECT_STATE.md docs/install docs/workspace-diagnostic-protocol.md
|
||||
```
|
||||
|
||||
- [ ] Classificare ogni risultato come descriptor legacy, compatibility decoder previsto, contratto diverso o documento storico.
|
||||
- [ ] Verificare nei registry/installazioni disponibili che i descriptor attivi siano v3; questa è una precondizione di deploy, non un migratore.
|
||||
- [ ] Non procedere se il worktree contiene modifiche applicative non attribuibili a questo piano.
|
||||
|
||||
### Task 1: Scrivere i test RED del contratto v3-only
|
||||
|
||||
**File:**
|
||||
- `backend/test/workspaces-schema.test.ts`
|
||||
- `backend/test/workspace-registry.test.ts`
|
||||
- `backend/test/routes-workspaces.test.ts`
|
||||
|
||||
- [ ] Aggiungere test che `parseWorkspaceYaml`, `validateWorkspaceDescriptor` e le route validate/publish rifiutino esplicitamente v1 e v2.
|
||||
- [ ] Aggiungere test registry per:
|
||||
- bootstrap pulito con solo v1/v2: fallimento, nessun `active.json` pubblicato;
|
||||
- repository misto v3+v2: attivazione atomica rifiutata;
|
||||
- pull che introduce v1/v2: precedente active state ancora leggibile;
|
||||
- retained snapshot contenente descriptor non-v3: rifiuto fail-closed;
|
||||
- risposta API state-free.
|
||||
- [ ] Eseguire:
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
npx vitest run test/workspaces-schema.test.ts test/workspace-registry.test.ts test/routes-workspaces.test.ts
|
||||
```
|
||||
|
||||
Atteso: RED per i nuovi requisiti, non errori di fixture casuali.
|
||||
|
||||
### Task 2: Rendere lo schema backend esclusivamente v3
|
||||
|
||||
**File:**
|
||||
- `backend/src/workspaces/schema.ts`
|
||||
- `backend/src/workspaces/types.ts`
|
||||
- test del Task 1
|
||||
|
||||
- [ ] Eliminare `WorkspaceV1`, `WorkspaceV2`, `LegacyWorkspace`, relativi Zod schema e `migrateWorkspaceV1ToV2`.
|
||||
- [ ] Rendere `WorkspaceDescriptorSchema = WorkspaceV3Schema`.
|
||||
- [ ] Eliminare `validateCanonicalWorkspace`, aggiornando **tutti** i chiamanti in `routes/workspaces.ts`, incluso il chiamante attualmente oltre quelli elencati nel vecchio piano.
|
||||
- [ ] Eliminare `isCanonicalWorkspace`/`isOperationalWorkspace` dopo aver sostituito i rami condizionali con validazione v3 diretta.
|
||||
- [ ] Conservare test negativi v1/v2; non cancellare le sole prove che impediscono una regressione futura.
|
||||
- [ ] Eseguire test focalizzati e typecheck.
|
||||
- [ ] Commit: `refactor: make workspace descriptors schema v3 only`.
|
||||
|
||||
### Task 3: Normalizzare in sicurezza active state e snapshot manifest
|
||||
|
||||
**File:**
|
||||
- `backend/src/workspaces/registry.ts`
|
||||
- `backend/test/workspace-registry.test.ts`
|
||||
|
||||
- [ ] Scrivere RED per state/manifest con:
|
||||
- campo assente;
|
||||
- vecchio `state: "operational"`;
|
||||
- `state: "migration_required"`;
|
||||
- valore sconosciuto;
|
||||
- campo extra;
|
||||
- active state e manifest con formati misti;
|
||||
- snapshot attivo, storico e fallback offline.
|
||||
- [ ] Rimuovere `state` da `WorkspaceRevision` e da tutti i nuovi writer.
|
||||
- [ ] Sostituire cast e vecchie migrazioni con decoder stretti che restituiscono oggetti normalizzati state-free.
|
||||
- [ ] Rimuovere `LegacyWorkspaceRevision`, `LegacyActiveState`, `LegacySnapshotManifest`, `deriveStateFromLegacyRevisions`, `migrateLegacyActiveState`, `migrateLegacySnapshotManifest`, `sameLegacyRevisions` e le condizioni operative basate su `state`.
|
||||
- [ ] Mantenere tutti i controlli di integrità e far validare ogni YAML come v3.
|
||||
- [ ] Provare che list/read/API non riemettono il vecchio campo anche immediatamente dopo un restart, prima di una nuova attivazione.
|
||||
- [ ] Commit: `refactor: remove workspace revision state`.
|
||||
|
||||
### Task 4: Eliminare i rami v1/v2 da renderer, contracts, bindings e diagnostica
|
||||
|
||||
**File:**
|
||||
- `backend/src/workspaces/runtime-renderer.ts`
|
||||
- `backend/src/workspaces/contracts.ts`
|
||||
- `backend/src/workspaces/diagnostics.ts`
|
||||
- `backend/src/workspaces/bindings.ts`
|
||||
- relativi test
|
||||
|
||||
- [ ] Scrivere/aggiornare test RED che accettano v3 e rifiutano input non-v3 al confine, senza renderer/diagnoser legacy.
|
||||
- [ ] Eliminare il renderer v2/pgvector e i rami v1.
|
||||
- [ ] Eliminare variabili contract e diagnostica solamente v2.
|
||||
- [ ] Semplificare bindings dopo la validazione v3, senza indebolire validazione secrets/trasporti.
|
||||
- [ ] Eseguire i test focalizzati:
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
npx vitest run \
|
||||
test/workspace-runtime-renderer.test.ts \
|
||||
test/workspaces-contracts.test.ts \
|
||||
test/workspaces-diagnostics.test.ts \
|
||||
test/workspaces-bindings.test.ts \
|
||||
test/workspace-runtime-handoff.test.ts
|
||||
```
|
||||
|
||||
- [ ] Commit: `refactor: remove legacy workspace runtime branches`.
|
||||
|
||||
### Task 5: Rimuovere migratori senza perdere test di deployment non correlati
|
||||
|
||||
**File:**
|
||||
- Eliminare i due migratori e i test esclusivamente di migrazione.
|
||||
- Creare/spostare in un test dedicato le prove deployment presenti in `workspaces-migrate-legacy.test.ts:81-114`.
|
||||
|
||||
- [ ] Prima di eliminare `workspaces-migrate-legacy.test.ts`, spostare in un file con nome coerente:
|
||||
- volume registry durevole e mount Git read-only;
|
||||
- contratto Dockerfile;
|
||||
- fallback offline smoke;
|
||||
- self-test di cleanup dell'immagine per-run.
|
||||
- [ ] Eliminare `migrate-legacy.ts`, `migrate-v2-qdrant.ts` e i test di trasformazione.
|
||||
- [ ] Conservare un fixture v2 soltanto nei test negativi di rifiuto.
|
||||
- [ ] Eseguire i nuovi test deployment e il typecheck.
|
||||
- [ ] Commit: `refactor: remove workspace migration utilities`.
|
||||
|
||||
### Task 6: Aggiornare tutte le route backend e il tooling P1
|
||||
|
||||
**File:**
|
||||
- `backend/src/routes/workspaces.ts`
|
||||
- `backend/src/routes/sessions.ts`
|
||||
- `backend/src/routes/sql.ts`
|
||||
- test route inclusi `routes-sql-meta.test.ts`
|
||||
- `backend/scripts/p1-manual-acceptance.mjs`
|
||||
- test manual/render P1
|
||||
|
||||
- [ ] Rimuovere filtri/gate `revision.state` da tutte le route. La garanzia deriva dal registry v3-only.
|
||||
- [ ] Aggiornare mock/fixture `WorkspaceRevision` in tutti i test backend.
|
||||
- [ ] Aggiornare il validatore del manifest P1 manuale affinché richieda esattamente la revisione state-free.
|
||||
- [ ] Aggiornare i fixture `p1-manual-acceptance.test.mjs` e `p1-render-snapshot.test.mjs`.
|
||||
- [ ] Aggiungere un test JS specifico che rifiuti manifest con revisioni malformate senza reintrodurre `migration_required`.
|
||||
- [ ] Eseguire:
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
npx vitest run test/routes-workspaces.test.ts test/routes-sessions.test.ts test/routes-sql-meta.test.ts
|
||||
cd ..
|
||||
node --test --test-concurrency=1 \
|
||||
backend/scripts/p1-manual-acceptance.test.mjs \
|
||||
backend/scripts/p1-render-snapshot.test.mjs
|
||||
```
|
||||
|
||||
- [ ] Commit: `refactor: remove workspace revision state consumers`.
|
||||
|
||||
### Task 7: Rendere il build backend clean-first
|
||||
|
||||
**File:**
|
||||
- `backend/package.json`
|
||||
- Creare `backend/scripts/clean-dist.mjs`
|
||||
- Creare test Node del clean build
|
||||
|
||||
- [ ] Scrivere RED: creare un file sentinella in `backend/dist/workspaces/`, eseguire il clean/build e verificare che non sopravviva.
|
||||
- [ ] Implementare la pulizia con API Node multipiattaforma, non con `rm -rf` nella npm script.
|
||||
- [ ] Fare eseguire il clean prima di `tsc` da `npm run build`.
|
||||
- [ ] Verificare dopo il build:
|
||||
|
||||
```bash
|
||||
test ! -e backend/dist/workspaces/migrate-legacy.js
|
||||
test ! -e backend/dist/workspaces/migrate-v2-qdrant.js
|
||||
```
|
||||
|
||||
- [ ] Costruire l'immagine core in un contesto pulito e verificare che i due moduli non esistano nell'immagine.
|
||||
- [ ] Verificare che i manifest di integrità P1 continuino a legare l'intero nuovo `dist`.
|
||||
- [ ] Commit: `build: remove stale backend distribution files`.
|
||||
|
||||
### Task 8: Aggiornare frontend e contratto API state-free
|
||||
|
||||
**File:**
|
||||
- `frontend/src/api/workspaces.ts`
|
||||
- `frontend/src/api/sessions.ts`
|
||||
- `frontend/src/shell/SteerInput.tsx`
|
||||
- `frontend/src/shell/WorkspaceManager.tsx`
|
||||
- test/fixture frontend correlati
|
||||
|
||||
- [ ] Scrivere/aggiornare test per revisioni senza `state` e risposta non-v3 rifiutata al confine workspace.
|
||||
- [ ] Eliminare `state` dal tipo e dal parser revisionale.
|
||||
- [ ] Rimuovere gate/banner/filtro `migration_required` e anche la visualizzazione `record.revision.state`.
|
||||
- [ ] Mantenere `allowLegacy` per sessioni senza revisione.
|
||||
- [ ] Aggiornare fixture in:
|
||||
- `api/workspaces.test.ts`, `api/sessions.test.ts`;
|
||||
- `SteerInput.test.tsx`, `WorkspaceManager.test.tsx`;
|
||||
- `NewSessionDialog.test.tsx`, `WorkspacePublishDialog.test.tsx`;
|
||||
- `drafts.test.ts`, mantenendo il test negativo di schema non-3.
|
||||
- [ ] Documentare che core e frontend devono essere aggiornati insieme; il parser nuovo non usa più `state`.
|
||||
- [ ] Eseguire typecheck e suite frontend.
|
||||
- [ ] Commit: `refactor: remove legacy workspace UI state`.
|
||||
|
||||
### Task 9: Sostituire fixture e smoke con descriptor v3 completi
|
||||
|
||||
**File:**
|
||||
- `scripts/workspace-registry-smoke.sh`
|
||||
- Creare `scripts/fixtures/workspace-registry-smoke.yaml`
|
||||
- `scripts/test-no-deployment-coupling-scope.sh`
|
||||
- `scripts/test-windows-clone-contract.ps1`
|
||||
- test deployment spostati nel Task 5
|
||||
|
||||
- [ ] Creare un descriptor v3 completo `id: local`, collection `local`, embedding interno 1024/cosine, LLM policy e diagnostica DWH; omettere Evidence per non richiedere un tree Git nello smoke registry.
|
||||
- [ ] Validare il fixture con il parser produttivo in un test backend.
|
||||
- [ ] Copiare il fixture nello seed repository e rimuovere sia l'invocazione del migratore sia il build backend ormai inutile allo smoke.
|
||||
- [ ] Nel test Windows non cambiare soltanto il numero di versione: fornire il contratto v3 completo mantenendo lo scopo path-with-spaces/clone.
|
||||
- [ ] Aggiornare il fixture dello scope coupling senza indebolire l'assenza-gate.
|
||||
- [ ] Eseguire test shell focalizzati e, con Docker disponibile, lo smoke reale senza retry.
|
||||
- [ ] Commit: `test: replace legacy workspace deployment fixtures`.
|
||||
|
||||
### Task 10: Aggiornare documentazione corrente e relativi verifier
|
||||
|
||||
**File:**
|
||||
- documenti/verifier indicati nella mappa
|
||||
|
||||
- [ ] Aggiornare README e soltanto la sezione corrente di `PROJECT_STATE.md`; non riscrivere gli snapshot storici.
|
||||
- [ ] Eliminare procedure di migrazione v1/v2 dai manuali local/server e dal protocollo diagnostico.
|
||||
- [ ] Modificare `verify-workspace-install-docs.sh` perché richieda “schema v3 only” e l'assenza di `migration_required` nella documentazione corrente.
|
||||
- [ ] Aggiornare i fixture negativi del test del verifier.
|
||||
- [ ] Non cambiare gli usi di `migration_required` nei piani P3–P6 relativi a ownership/artifact diversi.
|
||||
- [ ] Eseguire:
|
||||
|
||||
```bash
|
||||
bash scripts/test-verify-workspace-install-docs.sh
|
||||
bash scripts/verify-workspace-install-docs.sh --fixtures-only
|
||||
```
|
||||
|
||||
- [ ] Commit: `docs: make schema v3 the only workspace contract`.
|
||||
|
||||
### Task 11: Eseguire absence gate e suite complete
|
||||
|
||||
- [ ] Eseguire backend clean build, typecheck e test:
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
npm run build
|
||||
npx tsc --noEmit -p .
|
||||
npx vitest run
|
||||
```
|
||||
|
||||
- [ ] Eseguire frontend:
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
npx tsc -b
|
||||
npx vitest run
|
||||
npm run build
|
||||
```
|
||||
|
||||
- [ ] Eseguire script/verifier interessati, incluso lo smoke Docker obbligatorio se l'ambiente dispone di Docker. Non lasciarlo “opzionale” in una consegna che modifica lo smoke.
|
||||
- [ ] Eseguire `git diff --check`.
|
||||
- [ ] Eseguire l'absence gate ristretto:
|
||||
|
||||
```bash
|
||||
git grep -nE 'WorkspaceV1|WorkspaceV2|LegacyWorkspace|migrateWorkspaceV1ToV2|migrateWorkspaceV2ToV3' -- \
|
||||
backend/src frontend/src scripts && exit 1 || true
|
||||
|
||||
git grep -nE 'migration_required|migrate-legacy|migrate-v2-qdrant' -- \
|
||||
backend/src backend/scripts frontend/src scripts README.md docs/install docs/workspace-diagnostic-protocol.md && exit 1 || true
|
||||
|
||||
test ! -e backend/dist/workspaces/migrate-legacy.js
|
||||
test ! -e backend/dist/workspaces/migrate-v2-qdrant.js
|
||||
```
|
||||
|
||||
Nota: trasformare questi esempi in uno script con allowlist esplicita; non affidarsi a `&& exit 1 || true`, che può mascherare errori di esecuzione. Lo script deve distinguere “nessun match” da errore Git/I/O.
|
||||
|
||||
- [ ] Ispezionare il diff per assicurarsi che nessun formato non-descriptor sia stato modificato.
|
||||
|
||||
### Task 12: Rigenerare l'evidenza automatica P1
|
||||
|
||||
- [ ] Partire dal commit sorgente finale pulito.
|
||||
- [ ] Eseguire una sola integrazione completa, senza retry automatico:
|
||||
|
||||
```bash
|
||||
./scripts/p1-acceptance.sh integration --keep
|
||||
```
|
||||
|
||||
- [ ] Verificare report JSON/Markdown, hash dichiarati, manifest sorgente/dist, secret scan, ownership cleanup e porte chiuse.
|
||||
- [ ] Aggiornare `PROJECT_STATE.md` con il nuovo commit/tree/report e con stati distinti:
|
||||
|
||||
```text
|
||||
automated integration: PASS
|
||||
manual acceptance: PENDING
|
||||
```
|
||||
|
||||
- [ ] Committare soltanto lo stato tracciato, mai `.artifacts`.
|
||||
- [ ] Non riusare l'evidenza precedente legata a `c733896`.
|
||||
|
||||
### Task 13: Riaprire e chiudere il gate manuale P1
|
||||
|
||||
- [ ] Preparare un ambiente manuale nuovo:
|
||||
|
||||
```bash
|
||||
./scripts/p1-manual-acceptance.sh prepare
|
||||
./scripts/p1-manual-acceptance.sh serve
|
||||
```
|
||||
|
||||
- [ ] Il reviewer segue integralmente il nuovo `GUIDE.md`, verificando anche che revisioni/API/manifest siano state-free e che v1/v2 siano rifiutati senza mutazione.
|
||||
- [ ] Arrestare il server e verificare porte/processi:
|
||||
|
||||
```bash
|
||||
./scripts/p1-manual-acceptance.sh stop
|
||||
```
|
||||
|
||||
- [ ] Solo il reviewer crea `VERDICT.md` e decide PASS/FAIL.
|
||||
- [ ] Se PASS, aggiornare `PROJECT_STATE.md` e committare `docs: record schema-v3-only P1 acceptance`.
|
||||
- [ ] Pulire il lab soltanto dopo conferma del reviewer.
|
||||
- [ ] **STOP:** non iniziare P2 finché il reviewer non approva esplicitamente il nuovo P1.
|
||||
|
||||
---
|
||||
|
||||
## Criteri finali di accettazione
|
||||
|
||||
1. Nessun descriptor v1/v2 viene parsato, pubblicato, attivato, renderizzato, diagnosticato o mostrato.
|
||||
2. I vecchi state file di revisioni v3 con `state: "operational"` continuano a caricarsi, ma API e nuovi file sono state-free.
|
||||
3. Descriptor non-v3 o state incoerenti falliscono senza sostituire il precedente active state.
|
||||
4. Nessun migratore sopravvive in sorgenti, `dist`, immagine core, script o documentazione corrente.
|
||||
5. I formati versionati non collegati ai workspace descriptor sono invariati.
|
||||
6. Backend, frontend, verifier, smoke e build interessati sono verdi.
|
||||
7. Una nuova integrazione P1 è PASS al commit finale.
|
||||
8. La nuova acceptance manuale P1 è decisa esplicitamente dal reviewer.
|
||||
9. P2 resta non iniziato fino a ulteriore autorizzazione.
|
||||
@@ -0,0 +1,189 @@
|
||||
# Read-only Workspace Repository and Runtime Secrets Design
|
||||
|
||||
**Date:** 2026-08-14
|
||||
**Status:** Approved
|
||||
|
||||
## Purpose
|
||||
|
||||
ThothII consumes workspaces from one administrator-configured Git repository. Workspace authors
|
||||
prepare and publish source outside ThothII. The application fetches, validates, and activates
|
||||
repository revisions, but never edits, commits, pushes, imports, or exports workspace source.
|
||||
|
||||
Runtime credentials are intentionally absent from Git. After a workspace has been read, ThothII
|
||||
derives the required credentials from its connector and authentication choices and lets an
|
||||
authorized user complete them in the web application. The values are encrypted and persisted by
|
||||
the backend; the browser retains neither workspace content nor secrets.
|
||||
|
||||
## Ownership boundaries
|
||||
|
||||
### Workspace source
|
||||
|
||||
The workspace source is an ordinary directory maintained outside the ThothII runtime. It contains
|
||||
the catalog, each `workspace.yaml`, curated evidence, annotations, and other repository-owned
|
||||
content. Authors validate it using source-side tooling and publish it through their normal Git
|
||||
workflow to GitHub, GitLab, Gitea, or another standards-compatible server.
|
||||
|
||||
### ThothII installation
|
||||
|
||||
The installation descriptor selects the Git remote, branch, and one read-only authentication
|
||||
transport. SSH uses a read-only deploy key plus pinned known hosts. HTTPS uses a read-only deploy
|
||||
token and may provide a private CA. Secret values remain outside versioned configuration.
|
||||
|
||||
The installer performs a sanitized `git ls-remote` preflight. Credentials embedded in a remote URL
|
||||
are rejected. The API exposes only a normalized repository identity: host, repository path, branch,
|
||||
transport, active commit, and synchronization state.
|
||||
|
||||
### ThothII runtime
|
||||
|
||||
The local Git checkout, candidate validation area, immutable snapshots, and active state are
|
||||
application-owned. They are read-only from the workspace-management API. A pull fetches a candidate
|
||||
revision, validates the complete repository, and atomically activates it only if valid. A failed
|
||||
candidate never replaces the last valid active revision.
|
||||
|
||||
ThothII never generates or reconciles files back into the checkout and never invokes Git commit or
|
||||
push. Generated operational artifacts live under application data, not in the source repository.
|
||||
|
||||
## Repository synchronization states
|
||||
|
||||
A repository refresh has these states:
|
||||
|
||||
- `syncing`: fetching and validating a candidate revision;
|
||||
- `active`: the candidate passed validation and became the active immutable revision;
|
||||
- `invalid_candidate`: Git succeeded but repository validation failed; the previous revision stays active;
|
||||
- `unavailable`: Git or authentication failed; the previous revision stays active;
|
||||
- `empty`: no valid revision has ever been activated.
|
||||
|
||||
Validation is atomic at repository-commit level. A malformed catalog, descriptor, evidence tree, or
|
||||
cross-file reference rejects the complete candidate revision.
|
||||
|
||||
## Runtime secret model
|
||||
|
||||
### Requirement discovery
|
||||
|
||||
The workspace descriptor contains connector type, authentication method, and non-secret logical
|
||||
configuration. It never contains secret values or host filesystem paths. Connector adapters define
|
||||
the secret fields required by each supported authentication method. For example:
|
||||
|
||||
- PostgreSQL `username_password` requires `username` and `password`;
|
||||
- REST `bearer` requires `api_key`;
|
||||
- SSH tunnel authentication requires the connector password and SSH private key;
|
||||
- Evidence HTTP signed URLs and static S3 credentials contribute their own secret requirements.
|
||||
|
||||
Requirements have stable identifiers scoped by workspace and connector. Labels, descriptions,
|
||||
input kinds, and required/optional status come from trusted application code rather than repository
|
||||
HTML or executable metadata.
|
||||
|
||||
### Persistent encrypted store
|
||||
|
||||
The backend owns a `WorkspaceSecretStore` abstraction. The first implementation is a local encrypted
|
||||
vault in application-managed persistent storage. Each secret is encrypted with authenticated
|
||||
encryption and bound to its installation, workspace, connector, and field identifier as associated
|
||||
data. Plaintext values never appear in Git, API responses, logs, error messages, diagnostics, or
|
||||
browser storage.
|
||||
|
||||
The installation bootstraps one vault key independently from workspace content. Deployment tooling
|
||||
owns its platform-specific provisioning; the workspace schema and GUI never contain filesystem
|
||||
paths. The storage interface allows a future Vault, cloud secret manager, or OS keychain provider
|
||||
without changing workspace descriptors or API consumers.
|
||||
|
||||
When an existing file-oriented harness connector needs a credential, the backend materializes it as
|
||||
a restrictive temporary file in an application-owned runtime directory. Its lifetime is tied to the
|
||||
diagnostic or runtime lease and it is removed on release. Persistent storage contains ciphertext
|
||||
only.
|
||||
|
||||
### Secret API
|
||||
|
||||
For a selected workspace the API returns requirement metadata and status only:
|
||||
|
||||
```json
|
||||
{
|
||||
"workspaceId": "psd-clinical",
|
||||
"state": "configuration_required",
|
||||
"requirements": [
|
||||
{
|
||||
"id": "dwh.password",
|
||||
"connector": "dwh",
|
||||
"label": "Database password",
|
||||
"input": "password",
|
||||
"required": true,
|
||||
"configured": false
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
A write request contains values only for the selected requirement identifiers. The response returns
|
||||
status, never values. A delete operation forgets a configured value. Authorization is deliberately
|
||||
deferred; the current authenticated application user may manage runtime workspace secrets.
|
||||
|
||||
Workspace readiness is derived as follows:
|
||||
|
||||
- `invalid`: repository structure or descriptor is invalid;
|
||||
- `configuration_required`: structurally valid but required runtime values are missing;
|
||||
- `ready`: required values exist but connectivity has not yet passed or is stale;
|
||||
- `verified`: the most recent connector diagnostic passed for the active revision and current secret generation.
|
||||
|
||||
Changing or deleting a secret invalidates the previous diagnostic result.
|
||||
|
||||
## Browser behavior
|
||||
|
||||
Workspace management is a two-level read-only interface occupying at least 60 percent of viewport
|
||||
width and height.
|
||||
|
||||
Level 1 explains the source/runtime separation and displays:
|
||||
|
||||
- normalized repository host and path;
|
||||
- configured branch and read-only transport;
|
||||
- active revision and last synchronization result;
|
||||
- `Update workspace repository`, which fetches, validates, and conditionally activates a revision;
|
||||
- the workspace list, with selection required for workspace-specific actions.
|
||||
|
||||
There is no Import bundle, Export bundle, Create, Edit, Delete, Publish, or conflict-resolution
|
||||
operation. There are no browser-persisted workspace drafts or preferences.
|
||||
|
||||
Level 2 for the selected workspace explains and displays:
|
||||
|
||||
- immutable source identity and validation result;
|
||||
- required runtime configuration grouped by connector;
|
||||
- secret-entry controls whose values are write-only;
|
||||
- `Save secrets`, `Forget` per configured value, and `Test workspace connection`;
|
||||
- clear consequences for each button and a reminder that source changes must be committed and pushed
|
||||
by an author outside ThothII before repository update.
|
||||
|
||||
The browser keeps form values only in component memory and clears them after submission or dialog
|
||||
close. It never receives saved secret values.
|
||||
|
||||
## Compatibility and migration
|
||||
|
||||
Existing Git author settings, publish endpoints, bundle endpoints, generated-document
|
||||
reconciliation, bootstrap catalog slots, and browser draft storage are removed. Existing environment
|
||||
bindings may be read during a bounded migration period only to seed non-secret connector values;
|
||||
secret file paths are not part of the new public workspace contract.
|
||||
|
||||
Session manifests continue to pin an immutable validated workspace revision. An already running
|
||||
session keeps its acquired runtime lease; new or resumed work resolves the current encrypted secret
|
||||
generation and fails closed when required credentials are unavailable.
|
||||
|
||||
## Failure handling and security
|
||||
|
||||
- Repository and vault errors use stable sanitized codes and never echo remotes with user info,
|
||||
credential paths, secret identifiers that are not safe to disclose, or secret values.
|
||||
- Vault writes are atomic and authenticated; corrupted ciphertext fails closed.
|
||||
- Secret comparison uses no read API. Updating a secret is always a blind replacement.
|
||||
- The backend applies request-size and field-count limits and rejects unknown requirement IDs.
|
||||
- Temporary plaintext files use restrictive permissions, trusted directories, no-follow opens, and
|
||||
deterministic cleanup.
|
||||
- Git credentials are installation-only, read-only, and never sent to the frontend.
|
||||
|
||||
## Verification
|
||||
|
||||
Backend tests cover repository read-only behavior, atomic candidate activation, remote sanitization,
|
||||
vault encryption and corruption, requirement discovery, blind secret writes/deletes, materialization
|
||||
cleanup, readiness transitions, and absence of publish/bundle routes.
|
||||
|
||||
Frontend tests cover the two-level explanation, viewport dimensions, repository identity, selection
|
||||
gating, dynamic secret forms, write-only behavior, status changes, and absence of local-storage,
|
||||
import, export, editing, and publishing controls.
|
||||
|
||||
Deployment and CLI tests cover required remote/branch configuration, one read-only Git transport,
|
||||
sanitized remote preflight, vault-key provisioning, and removal of Git author/write configuration.
|
||||
@@ -0,0 +1,401 @@
|
||||
# Read-only Workspace Runtime Secrets Implementation Plan
|
||||
|
||||
> **Historical nomenclature:** this plan predates the native host CLI convergence. References to
|
||||
> `thothctl` and `tools/thothctl` describe the implementation snapshot from which this plan was
|
||||
> written; current operator commands and paths use native `tht` and `tools/tht`.
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Make workspace consumption strictly read-only while adding installation-scoped Git identity and persistent GUI-managed runtime secrets.
|
||||
|
||||
**Architecture:** Git remains the source of truth and is fetched into an application-owned checkout; complete candidate commits are validated before atomic activation and the backend has no Git write path. Runtime connector credentials are discovered from trusted connector contracts, stored as authenticated ciphertext by a backend vault, and materialized only for the lifetime of diagnostics or runtime leases. The browser exposes repository/readiness status and write-only secret forms without workspace persistence.
|
||||
|
||||
**Tech Stack:** Fastify, TypeScript, Node.js crypto/filesystem, React 18, TanStack Query, Vitest, Go `thothctl`, Docker Compose.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Freeze the Git repository boundary to read-only
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/src/workspaces/types.ts`
|
||||
- Modify: `backend/src/workspaces/git-repository.ts`
|
||||
- Modify: `backend/src/workspaces/registry.ts`
|
||||
- Modify: `backend/test/workspaces-git-repository.test.ts`
|
||||
- Modify: `backend/test/workspace-registry.test.ts`
|
||||
- Modify: `backend/test/workspace-registry-deployment.test.ts`
|
||||
|
||||
**Step 1: Write failing tests**
|
||||
|
||||
Add tests proving that pull never configures a Git author, writes generated files, commits, or pushes; that a malformed candidate leaves the prior active snapshot intact; and that a missing catalog descriptor rejects the whole candidate instead of producing a bootstrap slot.
|
||||
|
||||
**Step 2: Run the focused tests**
|
||||
|
||||
Run: `cd backend && npx vitest run test/workspaces-git-repository.test.ts test/workspace-registry.test.ts test/workspace-registry-deployment.test.ts`
|
||||
|
||||
Expected: FAIL on write/publish behavior and missing-descriptor semantics.
|
||||
|
||||
**Step 3: Implement the read-only boundary**
|
||||
|
||||
Remove `gitAuthorName`, `gitAuthorEmail`, mutation helpers, generated-document reconciliation, publish/conflict types, and bootstrap-slot activation. `pull()` must fetch, validate the complete commit in a candidate snapshot, and replace active state only after validation succeeds.
|
||||
|
||||
**Step 4: Run focused tests**
|
||||
|
||||
Run the command from Step 2.
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add backend/src/workspaces backend/test/workspaces-git-repository.test.ts backend/test/workspace-registry.test.ts backend/test/workspace-registry-deployment.test.ts
|
||||
git commit -m "refactor: make workspace repository strictly read only"
|
||||
```
|
||||
|
||||
### Task 2: Remove publishing and bundle HTTP contracts
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/src/routes/workspaces.ts`
|
||||
- Modify: `backend/test/routes-workspaces.test.ts`
|
||||
- Modify: `backend/test/workspaces-runtime-v3-boundaries.test.ts`
|
||||
- Modify: `backend/src/config.ts`
|
||||
- Modify: `backend/test/workspaces-config.test.ts`
|
||||
|
||||
**Step 1: Write failing route tests**
|
||||
|
||||
Assert `POST /workspaces/publish`, `GET /workspaces/:id/export`, and `POST /workspaces/import` return 404 and that the backend no longer registers multipart or ZIP handling. Assert configuration no longer accepts Git author or bundle-limit settings as workspace-registry fields.
|
||||
|
||||
**Step 2: Run tests and observe failure**
|
||||
|
||||
Run: `cd backend && npx vitest run test/routes-workspaces.test.ts test/workspaces-config.test.ts test/workspaces-runtime-v3-boundaries.test.ts`
|
||||
|
||||
Expected: FAIL because mutation and bundle routes still exist.
|
||||
|
||||
**Step 3: Remove the mutation surface**
|
||||
|
||||
Delete publish/import/export schemas and helpers, remove `multipart`, `yauzl`, and `yazl` usage from the route, and simplify safe workspace errors to read/validate/sync errors.
|
||||
|
||||
**Step 4: Run tests**
|
||||
|
||||
Run the command from Step 2 plus `cd backend && npx tsc --noEmit -p .`.
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add backend/src backend/test package.json package-lock.json
|
||||
git commit -m "refactor: remove workspace publishing and bundles"
|
||||
```
|
||||
|
||||
### Task 3: Expose a sanitized installation repository identity
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/src/workspaces/git-repository.ts`
|
||||
- Modify: `backend/src/routes/workspaces.ts`
|
||||
- Modify: `backend/test/workspaces-git-repository.test.ts`
|
||||
- Modify: `backend/test/routes-workspaces.test.ts`
|
||||
- Modify: `tools/thothctl/internal/config/installation.go`
|
||||
- Modify: `tools/thothctl/internal/config/installation_test.go`
|
||||
- Modify: `deploy/psd/thothii-installation.yaml.example`
|
||||
- Modify: `docs/install/examples/thothii-installation.local.yaml`
|
||||
- Modify: `docs/install/examples/thothii-installation.server.yaml`
|
||||
|
||||
**Step 1: Write failing parser and status tests**
|
||||
|
||||
Cover HTTPS, SSH URL, and SCP-style remotes; reject embedded user-info for HTTPS; return only `host`, `repository`, `branch`, and `transport`; never return a token, key path, or raw credential-bearing URL. Add installation-descriptor tests for a required `workspaceRepository` block and exactly one read-only transport.
|
||||
|
||||
**Step 2: Run focused tests**
|
||||
|
||||
Run: `cd backend && npx vitest run test/workspaces-git-repository.test.ts test/routes-workspaces.test.ts && cd ../tools/thothctl && go test ./internal/config`
|
||||
|
||||
Expected: FAIL because repository identity and typed installation configuration do not exist.
|
||||
|
||||
**Step 3: Implement safe normalization and installation validation**
|
||||
|
||||
Add the normalized identity to registry status. Extend `thothii-installation.yaml` with remote, branch, and SSH/HTTPS access metadata, validate it against the selected Compose override and environment without reading or returning secret values, and retain the existing environment rendering boundary.
|
||||
|
||||
**Step 4: Run focused tests**
|
||||
|
||||
Run the command from Step 2.
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add backend tools/thothctl deploy docs/install/examples
|
||||
git commit -m "feat: declare workspace repository in installation config"
|
||||
```
|
||||
|
||||
### Task 4: Add the persistent encrypted workspace secret store
|
||||
|
||||
**Files:**
|
||||
- Create: `backend/src/workspaces/secret-store.ts`
|
||||
- Create: `backend/test/workspace-secret-store.test.ts`
|
||||
- Modify: `backend/src/config.ts`
|
||||
- Modify: `backend/src/app.ts`
|
||||
- Modify: `compose.yaml`
|
||||
- Modify: `deploy/compose.local.yaml`
|
||||
- Modify: `deploy/compose.server.yaml`
|
||||
|
||||
**Step 1: Write failing vault tests**
|
||||
|
||||
Test first-start initialization, atomic blind replacement, deletion, enumeration by configured ID only, AES-256-GCM ciphertext with installation/workspace/field associated data, corruption failure, restrictive files/directories, size limits, and absence of plaintext in persistent bytes.
|
||||
|
||||
**Step 2: Run the vault test**
|
||||
|
||||
Run: `cd backend && npx vitest run test/workspace-secret-store.test.ts`
|
||||
|
||||
Expected: FAIL because `WorkspaceSecretStore` does not exist.
|
||||
|
||||
**Step 3: Implement the vault**
|
||||
|
||||
Create an injectable `WorkspaceSecretStore` backed by an application-managed data root. Persist a versioned encrypted document atomically, generate or load the installation vault key in the private control area, expose only `has`, `put`, `delete`, and scoped materialization operations, and never add a plaintext read API.
|
||||
|
||||
**Step 4: Run tests and typecheck**
|
||||
|
||||
Run: `cd backend && npx vitest run test/workspace-secret-store.test.ts && npx tsc --noEmit -p .`
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add backend compose.yaml deploy
|
||||
git commit -m "feat: persist encrypted workspace runtime secrets"
|
||||
```
|
||||
|
||||
### Task 5: Derive connector requirements and integrate temporary materialization
|
||||
|
||||
**Files:**
|
||||
- Create: `backend/src/workspaces/secret-requirements.ts`
|
||||
- Create: `backend/test/workspace-secret-requirements.test.ts`
|
||||
- Modify: `backend/src/workspaces/bindings.ts`
|
||||
- Modify: `backend/src/workspaces/runtime-config-lease.ts`
|
||||
- Modify: `backend/src/tht/tht-runner.ts`
|
||||
- Modify: `backend/src/app.ts`
|
||||
- Modify: `backend/test/workspace-runtime-config-lease.test.ts`
|
||||
- Modify: `backend/test/workspace-runtime-handoff.test.ts`
|
||||
- Modify: `backend/test/workspaces-bindings.test.ts`
|
||||
|
||||
**Step 1: Write failing requirement and lifecycle tests**
|
||||
|
||||
Cover PostgreSQL password, REST bearer API key, unauthenticated REST, SSH private key/password, signed HTTP Evidence, and static S3 credentials. Assert temporary files are restrictive, live for exactly one diagnostic/runtime lease, disappear on release and error, and are never persisted in the encrypted vault document.
|
||||
|
||||
**Step 2: Run focused tests**
|
||||
|
||||
Run: `cd backend && npx vitest run test/workspace-secret-requirements.test.ts test/workspaces-bindings.test.ts test/workspace-runtime-config-lease.test.ts test/workspace-runtime-handoff.test.ts`
|
||||
|
||||
Expected: FAIL because requirements still come from installation secret-file paths.
|
||||
|
||||
**Step 3: Implement dynamic requirement resolution**
|
||||
|
||||
Use the selected DWH transport and Evidence authentication contract to map trusted installation-contract suffixes to stable GUI requirement IDs. Overlay materialized temporary file paths only while resolving existing file-oriented connectors, and attach cleanup to every runtime lease.
|
||||
|
||||
**Step 4: Run tests and typecheck**
|
||||
|
||||
Run the command from Step 2 plus `cd backend && npx tsc --noEmit -p .`.
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add backend/src backend/test
|
||||
git commit -m "feat: resolve workspace secrets from connector requirements"
|
||||
```
|
||||
|
||||
### Task 6: Add write-only workspace secret and readiness APIs
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/src/routes/workspaces.ts`
|
||||
- Modify: `backend/src/app.ts`
|
||||
- Modify: `backend/src/workspaces/types.ts`
|
||||
- Modify: `backend/test/routes-workspaces.test.ts`
|
||||
|
||||
**Step 1: Write failing API tests**
|
||||
|
||||
Test `GET /workspaces/:id/runtime-configuration`, blind `PUT /workspaces/:id/secrets`, and `DELETE /workspaces/:id/secrets/:requirementId`. Assert strict bodies, limits, unknown-ID rejection, status-only responses, diagnostic invalidation, and `configuration_required`/`ready` state transitions.
|
||||
|
||||
**Step 2: Run tests**
|
||||
|
||||
Run: `cd backend && npx vitest run test/routes-workspaces.test.ts`
|
||||
|
||||
Expected: FAIL because the routes do not exist.
|
||||
|
||||
**Step 3: Implement the routes and readiness projection**
|
||||
|
||||
Inject the secret store into workspace routes and runtime support. Compute per-workspace readiness from active descriptor, current requirement set, configured IDs, and diagnostic generation. Materialize values only inside the diagnostic request and always clean up.
|
||||
|
||||
**Step 4: Run backend gates**
|
||||
|
||||
Run: `cd backend && npx vitest run && npx tsc --noEmit -p . && npm run build`.
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add backend
|
||||
git commit -m "feat: manage runtime workspace secrets through the API"
|
||||
```
|
||||
|
||||
### Task 7: Replace workspace management with the two-level read-only UI
|
||||
|
||||
**Files:**
|
||||
- Modify: `frontend/src/api/workspaces.ts`
|
||||
- Modify: `frontend/src/api/workspaces.test.ts`
|
||||
- Modify: `frontend/src/shell/WorkspaceManager.tsx`
|
||||
- Modify: `frontend/src/shell/WorkspaceManager.test.tsx`
|
||||
- Delete: `frontend/src/shell/WorkspacePublishDialog.tsx`
|
||||
- Delete: corresponding publish-dialog tests
|
||||
- Modify/Delete: `frontend/src/shell/WorkspaceEditor.tsx` and bootstrap-only tests as references permit
|
||||
- Modify: `frontend/src/workspaces/drafts.ts`
|
||||
- Modify: `frontend/src/workspaces/drafts.test.ts`
|
||||
|
||||
**Step 1: Write failing UI/API tests**
|
||||
|
||||
Assert the dialog uses at least 60% viewport width and height, shows general repository concepts and exact button consequences at level 1, gates workspace-specific controls on selection, renders requirement explanations and write-only fields at level 2, and has no create/edit/publish/import/export/bundle controls.
|
||||
|
||||
**Step 2: Run focused tests**
|
||||
|
||||
Run: `cd frontend && npx vitest run src/api/workspaces.test.ts src/shell/WorkspaceManager.test.tsx src/workspaces/drafts.test.ts`
|
||||
|
||||
Expected: FAIL on the old draft/publish interface.
|
||||
|
||||
**Step 3: Implement the read-only interface**
|
||||
|
||||
Replace bootstrap editor state with repository status, selection, validation/readiness details, dynamic secret fields, blind save/forget actions, and connection test. Remove workspace draft persistence and clear secret field component state after submit/close.
|
||||
|
||||
**Step 4: Run focused tests and typecheck**
|
||||
|
||||
Run the command from Step 2 plus `cd frontend && npx tsc -b`.
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add frontend
|
||||
git commit -m "feat: add read-only workspace and secret management UI"
|
||||
```
|
||||
|
||||
### Task 8: Remove browser-persisted workspace preferences
|
||||
|
||||
**Files:**
|
||||
- Modify: `frontend/src/workspaces/preferences.ts`
|
||||
- Modify: `frontend/src/workspaces/preferences.test.ts`
|
||||
- Modify: `frontend/src/api/sessions.ts`
|
||||
- Modify: `frontend/src/api/sessions.test.ts`
|
||||
- Modify: `frontend/src/shell/SteerInput.tsx`
|
||||
- Modify: `frontend/src/shell/SteerInput.test.tsx`
|
||||
|
||||
**Step 1: Write failing persistence-boundary tests**
|
||||
|
||||
Assert workspace/model/thinking choices are kept only in current application memory or saved through the existing backend settings API, and that no workspace code calls `localStorage`.
|
||||
|
||||
**Step 2: Run focused tests**
|
||||
|
||||
Run: `cd frontend && npx vitest run src/workspaces/preferences.test.ts src/api/sessions.test.ts src/shell/SteerInput.test.tsx`
|
||||
|
||||
Expected: FAIL because preferences still use browser storage.
|
||||
|
||||
**Step 3: Implement ephemeral preferences**
|
||||
|
||||
Replace the storage adapter with an in-memory external store seeded from backend settings. Preserve concurrent workspace-policy gates and session request determinism without persisting selections in the browser.
|
||||
|
||||
**Step 4: Run frontend gates**
|
||||
|
||||
Run: `cd frontend && npx vitest run && npx tsc -b && npm run build`.
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add frontend
|
||||
git commit -m "refactor: stop persisting workspace state in the browser"
|
||||
```
|
||||
|
||||
### Task 9: Update deployment contracts and documentation
|
||||
|
||||
**Files:**
|
||||
- Modify: `compose.yaml`
|
||||
- Modify: `deploy/compose.git-ssh.yaml`
|
||||
- Modify: `deploy/compose.git-https.yaml`
|
||||
- Modify: `deploy/workspace-registry.env.example`
|
||||
- Modify: `deploy/psd/operator.env.example`
|
||||
- Modify: `docs/install/local-workspace-registry.md`
|
||||
- Modify: `docs/install/server-workspace-registry.md`
|
||||
- Modify: `docs/guida-utente.md`
|
||||
- Modify: `scripts/verify-workspace-install-docs.sh`
|
||||
- Modify: `scripts/workspace-registry-smoke.sh`
|
||||
|
||||
**Step 1: Update executable contract tests first**
|
||||
|
||||
Require read-only Git wording and configuration, repository identity visibility, vault persistence,
|
||||
and absence of author/push/bundle/browser-secret instructions.
|
||||
|
||||
**Step 2: Run contract tests and observe failure**
|
||||
|
||||
Run: `bash scripts/verify-workspace-install-docs.sh`
|
||||
|
||||
Expected: FAIL against the old manuals and examples.
|
||||
|
||||
**Step 3: Update deployment and manuals**
|
||||
|
||||
Remove Git author settings and write-oriented documentation. Document installation Git bootstrap,
|
||||
GUI runtime-secret completion, platform-neutral application storage, rotation/forget flows, and
|
||||
candidate validation semantics.
|
||||
|
||||
**Step 4: Run contract and Go gates**
|
||||
|
||||
Run: `bash scripts/verify-workspace-install-docs.sh && cd tools/thothctl && go test ./...`
|
||||
|
||||
Expected: PASS.
|
||||
|
||||
**Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add compose.yaml deploy docs scripts tools/thothctl
|
||||
git commit -m "docs: describe read-only workspace runtime configuration"
|
||||
```
|
||||
|
||||
### Task 10: Full verification and deployed-container refresh
|
||||
|
||||
**Files:**
|
||||
- Modify only files needed to fix failures found by verification.
|
||||
|
||||
**Step 1: Run static and unit gates**
|
||||
|
||||
```bash
|
||||
cd backend && npx vitest run && npx tsc --noEmit -p . && npm run build
|
||||
cd ../frontend && npx vitest run && npx tsc -b && npm run build
|
||||
cd ../harness && .venv/bin/pytest -q
|
||||
cd ../tools/thothctl && go test ./...
|
||||
```
|
||||
|
||||
Expected: all gates PASS.
|
||||
|
||||
**Step 2: Run deployment contract gates**
|
||||
|
||||
Run: `bash scripts/verify-workspace-install-docs.sh` and the focused workspace registry smoke appropriate to the configured installation.
|
||||
|
||||
Expected: PASS without Git writes or secret disclosure.
|
||||
|
||||
**Step 3: Inspect the final diff and secret scan**
|
||||
|
||||
Run: `git diff --check`, inspect `git status --short`, and search active code/config for removed publish, bundle, Git author, and workspace-localStorage contracts.
|
||||
|
||||
Expected: no whitespace errors, no accidental secrets, and only intended changes.
|
||||
|
||||
**Step 4: Rebuild and restart affected services**
|
||||
|
||||
Use the installation-aware `thothctl` lifecycle for the configured installation to rebuild/restart `core` and `frontend`, then verify health and repository status. Do not restart if no valid local installation descriptor is available; report that external gate explicitly.
|
||||
|
||||
**Step 5: Commit verification fixes**
|
||||
|
||||
```bash
|
||||
git add <only-files-changed-for-verification>
|
||||
git commit -m "test: verify read-only workspace secret flow"
|
||||
```
|
||||
@@ -0,0 +1,408 @@
|
||||
# ThothII Authentication Acceptance and PSD Deployment Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to execute this plan task-by-task.
|
||||
|
||||
**Goal:** Validate local and OIDC authentication on macOS, deploy the exact feat/thoth-auth candidate to the Aritmolab/PSD server before merging it into main, and complete end-to-end acceptance with remote Authentik.
|
||||
|
||||
**Architecture:** Test the candidate first as a standalone local installation. Then install the same immutable Git revision on the existing PSD installation with the installation-aware tht lifecycle, leaving main untouched. Authentik provides OIDC login and a mandatory direct groups claim; ThothII maps exact external groups to roles and validates mapped groups through the Authentik catalog API.
|
||||
|
||||
**Tech Stack:** macOS, Docker Desktop, Docker Compose, native host `tht` plus Python workflow `tht`, local Argon2id authentication, generic OIDC Authorization Code + PKCE, Authentik, PSD workspace registry, reverse proxy/TLS.
|
||||
|
||||
---
|
||||
|
||||
## Scope and release rules
|
||||
|
||||
Do not merge feat/thoth-auth into main until every mandatory gate in Task 9 is PASS and the PSD owner accepts the evidence.
|
||||
|
||||
Capture one candidate revision and reuse it everywhere:
|
||||
|
||||
```bash
|
||||
export CANDIDATE_SHA="$(git rev-parse HEAD)"
|
||||
git fetch origin feat/thoth-auth
|
||||
test "$CANDIDATE_SHA" = "$(git rev-parse origin/feat/thoth-auth)"
|
||||
git show -s --format='%H%n%P%n%s' "$CANDIDATE_SHA"
|
||||
git status --short --untracked-files=all
|
||||
```
|
||||
|
||||
Never deploy a moving branch name without checking its resolved SHA. Never put passwords, OIDC client secrets, Authentik API tokens, cookies, authorization headers, raw ID tokens, or password hashes in Git, shell history, screenshots, logs, or evidence.
|
||||
|
||||
Use protected operator values for <PUBLIC_URL>, <OIDC_ISSUER>, <AUTHENTIK_BASE_URL>, <OIDC_CLIENT_ID>, <THT_BIN>, <INSTALLATION>, <WORKSPACE_ID>, and <OLD_SHA>.
|
||||
|
||||
The Authentik contract is mandatory: a direct non-empty JSON array claim named groups; exact groups TOT Users and TOT Admin; mappings TOT Users -> user and TOT Admin -> admin; and a separate group-view-only API service account exposed only as THT_AUTHENTIK_API_TOKEN. Extra upstream groups are valid and silently ignored.
|
||||
|
||||
## Task 0: Freeze the candidate and collect approvals
|
||||
|
||||
**Files:** None; record results in the acceptance report in Task 9.
|
||||
|
||||
Run from the candidate worktree:
|
||||
|
||||
```bash
|
||||
git diff --check
|
||||
go test ./... -count=1
|
||||
go test -race ./...
|
||||
go vet ./...
|
||||
go build ./...
|
||||
```
|
||||
|
||||
Expected: all commands pass, the candidate is pushed, and existing evidence is bound to the same SHA. A historical result from another revision is not evidence for this run.
|
||||
|
||||
Before touching PSD, obtain the maintenance window, server access, public URL, Authentik provider details, protected secret locations, test identities for ordinary/admin/unmapped users, and permission to test PSD DWH/Evidence connections.
|
||||
|
||||
## Task 1: Prepare and start the local macOS installation
|
||||
|
||||
**Files:**
|
||||
|
||||
- Read: docs/install/local.md
|
||||
- Read: docs/install/authentication-local.md
|
||||
- Read: docs/testing/authentication-manual-acceptance.md
|
||||
- Use: an untracked local installation descriptor and protected secret/password files
|
||||
|
||||
**Step 1: Verify prerequisites**
|
||||
|
||||
```bash
|
||||
docker version
|
||||
docker compose version
|
||||
bash scripts/verify-line-endings.sh
|
||||
```
|
||||
|
||||
Expected: Docker Desktop and Compose are available and line-ending validation passes.
|
||||
|
||||
**Step 2: Build and configure**
|
||||
|
||||
```bash
|
||||
bash scripts/build-local.sh
|
||||
bash scripts/build-tht.sh
|
||||
tht setup --profile local
|
||||
```
|
||||
|
||||
For an existing installation, do not overwrite data; run tht --installation <local-installation.yaml> update --check-only instead of setup.
|
||||
|
||||
**Step 3: Start and inspect**
|
||||
|
||||
```bash
|
||||
tht --installation <local-installation.yaml> start --build
|
||||
tht --installation <local-installation.yaml> status
|
||||
tht --installation <local-installation.yaml> doctor --json
|
||||
curl --fail http://127.0.0.1:8080/health
|
||||
curl --fail http://127.0.0.1:8787/health
|
||||
```
|
||||
|
||||
Expected: core, frontend, qdrant, embedding, and the completed model initializer are healthy; doctor includes authentication after configuration and before services.
|
||||
|
||||
## Task 2: Configure and test local login
|
||||
|
||||
**Files:**
|
||||
|
||||
- Read: docs/install/authentication-local.md
|
||||
- Modify only protected installation state through tht auth configure and tht auth user
|
||||
|
||||
**Step 1: Bootstrap the administrator**
|
||||
|
||||
```bash
|
||||
tht --installation <local-installation.yaml> auth configure \
|
||||
--mode local --public-url http://127.0.0.1:8080 \
|
||||
--admin-user <local-admin> --admin-display-name <display-name> \
|
||||
--password-file <protected-password-file>
|
||||
```
|
||||
|
||||
Remove the temporary password file immediately. Expected: non-secret auth.yaml is created and the user store contains Argon2id hashes, never plaintext passwords.
|
||||
|
||||
**Step 2: Add and inspect a normal user**
|
||||
|
||||
```bash
|
||||
tht --installation <local-installation.yaml> auth user add <local-user> --role user --display-name <display-name> --password-file <protected-password-file>
|
||||
tht --installation <local-installation.yaml> auth status --json
|
||||
tht --installation <local-installation.yaml> auth check --json
|
||||
```
|
||||
|
||||
Expected: pristine redacted JSON and no credential, hash, or session secret in output.
|
||||
|
||||
**Step 3: Test browser authorization**
|
||||
|
||||
At http://127.0.0.1:8080, in a private browser profile:
|
||||
|
||||
1. Verify unauthenticated access reaches login and protected routes are denied.
|
||||
2. Log in as the normal user and verify application/session routes work.
|
||||
3. Verify Pi Management and other admin-only operations return HTTP 403 or are not exposed.
|
||||
4. Log out and verify the session is invalidated.
|
||||
5. Log in as the administrator and verify admin-only routes work.
|
||||
|
||||
Expected: ordinary users authenticate without receiving admin permissions; administrators receive the configured admin permission set.
|
||||
|
||||
**Step 4: Test account failure paths**
|
||||
|
||||
Use auth user disable, enable, set-password, and logout-all user --yes on the test user. Test a wrong password and refresh the old browser session after logout-all.
|
||||
|
||||
Expected: generic safe failures, disabled login rejection, re-enabled login success, and forced reauthentication. The last enabled administrator cannot be disabled or demoted.
|
||||
|
||||
## Task 3: Test remembered sessions and local recovery
|
||||
|
||||
**Files:**
|
||||
|
||||
- Read: docs/architecture/authentication.md, Browser sessions
|
||||
- Read: docs/install/authentication-local.md, Session behavior and recovery
|
||||
|
||||
**Step 1: Test browser restart**
|
||||
|
||||
Log in as the normal user with Remember me, close the browser completely, reopen it, and revisit the application.
|
||||
|
||||
Expected: the session survives within the 7-day idle / 30-day absolute limits. Do not record the cookie.
|
||||
|
||||
**Step 2: Test ThothII restart**
|
||||
|
||||
```bash
|
||||
tht --installation <local-installation.yaml> stop
|
||||
tht --installation <local-installation.yaml> start
|
||||
```
|
||||
|
||||
Expected: the remembered session remains valid after backend restart.
|
||||
|
||||
**Step 3: Test invalidation**
|
||||
|
||||
Change the test user password or role, and separately run auth user logout-all user --yes. Refresh after each operation.
|
||||
|
||||
Expected: affected sessions are rejected and reauthentication is required; configuration revision changes invalidate all sessions.
|
||||
|
||||
Go/no-go: do not proceed to PSD if local login, role separation, logout, or remembered-session behavior fails.
|
||||
|
||||
## Task 4: Snapshot the current PSD installation
|
||||
|
||||
**Files:**
|
||||
|
||||
- Read: docs/install/server.md
|
||||
- Read: docs/install/server-workspace-registry.md
|
||||
- Use: protected server operator and backup locations
|
||||
|
||||
**Step 1: Capture live state**
|
||||
|
||||
```bash
|
||||
THT_BIN=<THT_BIN>
|
||||
INSTALLATION=<INSTALLATION>
|
||||
"$THT_BIN" --installation "$INSTALLATION" status
|
||||
"$THT_BIN" --installation "$INSTALLATION" doctor
|
||||
"$THT_BIN" --installation "$INSTALLATION" pi status
|
||||
"$THT_BIN" --installation "$INSTALLATION" pi doctor
|
||||
git -C /srv/thothii/source/ThothII status --short --untracked-files=all
|
||||
git -C /srv/thothii/source/ThothII rev-parse HEAD
|
||||
```
|
||||
|
||||
Save the live SHA as <OLD_SHA> and capture image identities, workspace registry status, and maintenance/recovery state. Stop if the checkout is dirty or recovery is pending.
|
||||
|
||||
**Step 2: Drain and back up**
|
||||
|
||||
Announce maintenance, close the reverse proxy or show its maintenance page, drain active work, and stop through tht. Create the protected, checksummed backup specified in docs/install/server.md, including runtime trees and PSD PostgreSQL/session data where applicable. Back up credentials separately. Never run docker compose down --volumes.
|
||||
|
||||
**Step 3: Check preconditions**
|
||||
|
||||
```bash
|
||||
git -C /srv/thothii/source/ThothII config --local core.autocrlf false
|
||||
bash /srv/thothii/source/ThothII/scripts/verify-line-endings.sh
|
||||
"$THT_BIN" --installation "$INSTALLATION" update --check-only
|
||||
```
|
||||
|
||||
Expected: descriptor, protected secrets, Pi-state mount, workspace repository binding, and Compose render remain valid before source changes.
|
||||
|
||||
## Task 5: Deploy the feature revision to PSD without merging main
|
||||
|
||||
**Files:**
|
||||
|
||||
- Server source checkout: /srv/thothii/source/ThothII
|
||||
- Server operator binary: protected THT_BIN path
|
||||
- Server installation descriptor and secret files: unchanged paths unless a reviewed auth update is required
|
||||
|
||||
**Step 1: Select the exact candidate**
|
||||
|
||||
```bash
|
||||
git -C /srv/thothii/source/ThothII fetch origin feat/thoth-auth
|
||||
git -C /srv/thothii/source/ThothII switch --detach <CANDIDATE_SHA>
|
||||
test "$(git -C /srv/thothii/source/ThothII rev-parse HEAD)" = "<CANDIDATE_SHA>"
|
||||
git -C /srv/thothii/source/ThothII status --short --untracked-files=all
|
||||
```
|
||||
|
||||
Do not merge or rebase main. The running installation is intentionally based on the detached feature revision until acceptance completes.
|
||||
|
||||
**Step 2: Build candidate artifacts**
|
||||
|
||||
```bash
|
||||
cd /srv/thothii/source/ThothII
|
||||
bash scripts/build-local.sh
|
||||
THT_THT_OUTPUT_DIRECTORY=/srv/thothii/operator/build-output bash scripts/build-tht.sh
|
||||
```
|
||||
|
||||
Install the architecture-appropriate candidate tht only after its build succeeds. Keep the old operator binary recoverable.
|
||||
|
||||
**Step 3: Start and verify the candidate**
|
||||
|
||||
```bash
|
||||
"$THT_BIN" --installation "$INSTALLATION" update --check-only
|
||||
"$THT_BIN" --installation "$INSTALLATION" start --build
|
||||
"$THT_BIN" --installation "$INSTALLATION" status
|
||||
"$THT_BIN" --installation "$INSTALLATION" doctor --json
|
||||
curl --fail http://127.0.0.1:8080/health
|
||||
"$THT_BIN" --installation "$INSTALLATION" pi doctor
|
||||
"$THT_BIN" --installation "$INSTALLATION" pi test
|
||||
```
|
||||
|
||||
Expected: candidate frontend/core and internal services are healthy, no data volume was replaced, and the candidate SHA is recorded. Liveness alone is not release approval.
|
||||
|
||||
## Task 6: Configure and validate remote Authentik
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify protected server authentication state through tht auth configure
|
||||
- Modify protected secret entries THT_OIDC_CLIENT_SECRET and THT_AUTHENTIK_API_TOKEN
|
||||
- Read: docs/install/authentik.md and docs/install/authentication-oidc.md
|
||||
|
||||
**Step 1: Verify Authentik**
|
||||
|
||||
Verify the OAuth2/OIDC callback exactly <PUBLIC_URL>/api/auth/oidc/callback, scopes openid/profile/email, direct groups array mapping, exact groups TOT Users and TOT Admin, and a separate group-view-only catalog service account. Inspect a disposable identity without copying its token.
|
||||
|
||||
**Step 2: Configure the ThothII mapping**
|
||||
|
||||
```bash
|
||||
tht --installation "$INSTALLATION" auth configure \
|
||||
--mode oidc --public-url <PUBLIC_URL> \
|
||||
--issuer <OIDC_ISSUER> --client-id <OIDC_CLIENT_ID> \
|
||||
--authentik-base-url <AUTHENTIK_BASE_URL> \
|
||||
--user-group 'TOT Users' --admin-group 'TOT Admin'
|
||||
```
|
||||
|
||||
Secrets are read from protected files, never command-line arguments. Confirm the non-secret mapping is:
|
||||
|
||||
```yaml
|
||||
authorization:
|
||||
groupRoles:
|
||||
TOT Users: [user]
|
||||
TOT Admin: [admin]
|
||||
```
|
||||
|
||||
**Step 3: Run static and live checks**
|
||||
|
||||
```bash
|
||||
tht --installation "$INSTALLATION" auth status --json
|
||||
tht --installation "$INSTALLATION" auth check --json
|
||||
tht --installation "$INSTALLATION" auth check --interactive
|
||||
tht --installation "$INSTALLATION" doctor --json
|
||||
```
|
||||
|
||||
Expected: configuration, discovery, issuer, JWKS, client-secret access, catalog access, and exact existence of every mapped group pass. Doctor lists authentication after configuration and before services. No output contains credentials or bearer tokens.
|
||||
|
||||
A missing mapped group must fail with redacted oidc_mapped_group_missing. Unmapped groups produce neither error nor warning. Missing, indirect, malformed, or overage-style groups claims fail closed.
|
||||
|
||||
**Step 4: Reload if required**
|
||||
|
||||
If configuration requires process reload:
|
||||
|
||||
```bash
|
||||
"$THT_BIN" --installation "$INSTALLATION" pi restart --yes --drain
|
||||
```
|
||||
|
||||
Repeat authentication, doctor, and health checks. Do not substitute raw Compose commands.
|
||||
|
||||
## Task 7: Test PSD browser login and authorization
|
||||
|
||||
**Files:**
|
||||
|
||||
- Read: docs/testing/authentication-manual-acceptance.md
|
||||
- Evidence: redacted report from Task 9
|
||||
|
||||
**Step 1: Ordinary user**
|
||||
|
||||
In a private profile, authenticate with an identity in TOT Users but not TOT Admin. Verify callback success, application/session routes, denial of Pi Management/admin operations, opaque HttpOnly ThothII cookie, no bearer token in Web Storage, and logout invalidation.
|
||||
|
||||
**Step 2: Administrator**
|
||||
|
||||
Authenticate with TOT Admin. Verify Pi Management and allowed workspace-management operations. Access must derive from the exact mapped group, not a client-supplied header or browser-local flag.
|
||||
|
||||
**Step 3: Unmapped and malformed groups**
|
||||
|
||||
Authenticate with a valid token containing no mapped group. Expected: login may complete, but protected operations return 403 with no warning. Use a disposable provider mapping that omits or corrupts groups; expected: generic HTTP 401 oidc_callback_failed, with no internal claim details exposed.
|
||||
|
||||
**Step 4: Provider outage/group drift**
|
||||
|
||||
During a controlled window, make discovery/JWKS unavailable or rename a mapped group, run the CLI check, and restore it immediately. Expected: redacted fail-closed diagnostics followed by a successful check after restoration. Do not leave production broken.
|
||||
|
||||
## Task 8: Test PSD workspace validation and real connections
|
||||
|
||||
**Files:**
|
||||
|
||||
- Read: docs/install/server-workspace-registry.md
|
||||
- Use: authenticated PSD browser sessions
|
||||
|
||||
**Step 1: Validate the workspace and authentication from the host CLI**
|
||||
|
||||
```bash
|
||||
"$THT_BIN" --installation "$INSTALLATION" \
|
||||
workspace inspect --workspace "$WORKSPACE_ID" --json
|
||||
"$THT_BIN" --installation "$INSTALLATION" auth check --json
|
||||
```
|
||||
|
||||
Expected: the workspace registry is ready, authentication readiness passes, and output is redacted while identifying the active workspace revision.
|
||||
|
||||
**Step 2: Verify the application boundary**
|
||||
|
||||
Open the configured public URL, authenticate with the approved identity, and verify that the application reaches the selected workspace without unexpected `401`/`403` responses. Keep DWH/Evidence connection tests read-only and use only the existing approved smoke question.
|
||||
|
||||
**Step 3: Test ordinary-user authorization**
|
||||
|
||||
Log in as TOT Users. Confirm inspection follows ordinary permissions while validation, secret mutation, and connection tests remain unavailable unless explicitly granted.
|
||||
|
||||
**Step 4: Run a harmless end-to-end smoke**
|
||||
|
||||
As an authorized PSD user, create or resume one harmless known-good session:
|
||||
|
||||
```text
|
||||
browser login -> same-origin API -> workspace readiness -> Pi/core -> result -> logout
|
||||
```
|
||||
|
||||
Do not run mutating production queries. Preserve only a session ID and redacted outcome if approved.
|
||||
|
||||
## Task 9: Close acceptance, rollback if needed, and decide merge readiness
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: docs/testing/evidence/2026-08-18-thothii-authentication-psd-acceptance.md or the approved external evidence location
|
||||
- Read: docs/install/server.md and docs/contracts/tht-pi.md
|
||||
|
||||
**Step 1: Mandatory gates**
|
||||
|
||||
| Gate | Required evidence |
|
||||
|---|---|
|
||||
| Candidate identity | Local and server SHA exactly match pushed feat/thoth-auth |
|
||||
| Local startup | macOS Compose, doctor, health, and Pi smoke pass |
|
||||
| Local auth | Bootstrap, ordinary/admin roles, logout, bad password, disable/enable, logout-all pass |
|
||||
| Local session | Remembered session survives browser and ThothII restart; revisions invalidate it |
|
||||
| Server safety | Old SHA/image/status captured; backup checksummed; maintenance/drain completed |
|
||||
| Candidate deploy | Server candidate status/doctor/health pass |
|
||||
| Authentik | Direct groups claim, issuer/JWKS, secrets, catalog, and mapped groups pass |
|
||||
| OIDC authorization | ordinary, admin, unmapped, malformed, logout, and outage cases pass |
|
||||
| Workspace integration | `tht workspace inspect` and `tht auth check` pass; the authenticated application reaches the selected workspace |
|
||||
| PSD smoke | One harmless known-good session completes |
|
||||
| Hygiene | No secrets, tokens, cookies, hashes, or raw claims in evidence |
|
||||
|
||||
**Step 2: Write the redacted report**
|
||||
|
||||
Include candidate SHA, old SHA, timestamps, commands, browser cases, redacted diagnostic/HTTP codes, Authentik issuer/client/group names, workspace ID/revision, backup/checksum location, rollback decision, and unrelated CI failures. Never include secret values, raw tokens, cookies, or hashes.
|
||||
|
||||
**Step 3: Roll back a failed candidate**
|
||||
|
||||
1. Keep the proxy closed and preserve .tht/<installation-id>/ recovery state.
|
||||
2. Do not use tht pi rollback as the whole-application rollback; it addresses only Pi lifecycle images.
|
||||
3. Stop with tht.
|
||||
4. Return the source checkout to <OLD_SHA>, rebuild old application/operator artifacts, and start through the same descriptor.
|
||||
5. Run update --check-only, status, doctor, health, Pi smoke, workspace diagnostics, and one harmless session.
|
||||
6. For ambiguous recovery, leave maintenance active and follow pi maintenance status / pi maintenance recover --yes. Never delete volumes, selectors, or recovery files to force progress.
|
||||
|
||||
Expected: the previous application serves again with prior data and workspace state intact. Record the failure and do not merge.
|
||||
|
||||
**Step 4: Reopen traffic**
|
||||
|
||||
After every gate passes, restore the reverse proxy, repeat one unauthenticated redirect and one authorized public login, and confirm only the proxy is externally reachable.
|
||||
|
||||
**Step 5: Merge decision**
|
||||
|
||||
Merge only after PSD owner acceptance, exact-SHA evidence, no unresolved auth/workspace/provider/deployment gate, and an accepted rollback path. If the merge creates a new commit, repeat Tasks 0, 5, 6, and 7 against the merge SHA.
|
||||
|
||||
## Handoff checklist
|
||||
|
||||
Deliver the redacted report, local result/SHA, PSD candidate SHA/images, Authentik provider and group mapping confirmation, workspace validation/connection results, backup/rollback status, and an explicit READY TO MERGE or NOT READY TO MERGE decision.
|
||||
@@ -0,0 +1,92 @@
|
||||
# Tht Documentation Convergence Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Align current documentation and documentation smoke checks with the converged native host CLI `tht`, while preserving historical references only where they describe past decisions or evidence.
|
||||
|
||||
**Architecture:** Treat `tools/tht/cmd/tht/main.go` as the canonical host CLI surface for installation, authentication, diagnostics, lifecycle, and workspace operations. Keep the Python `harness/.venv/bin/tht` distinction explicit for the workflow runtime, and update current operator/test instructions to invoke the native `tht` with `--installation`.
|
||||
|
||||
**Tech Stack:** Markdown documentation, shell smoke tests, Go CLI command surface, repository search-based verification.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Classify current and historical legacy CLI references
|
||||
|
||||
**Files:**
|
||||
- Inspect: `README.md`, `PROJECT_STATE.md`, `AGENTS.md`, `docs/**`, `scripts/**`
|
||||
- Reference: `tools/tht/cmd/tht/main.go`
|
||||
|
||||
**Step 1:** Build a complete occurrence inventory with a case-insensitive search for the former host CLI name and classify every match.
|
||||
|
||||
**Step 2:** Classify each occurrence as current operator documentation, documentation smoke expectation, executable/script contract, or historical design/evidence.
|
||||
|
||||
**Step 3:** Record the classification in the implementation notes before editing.
|
||||
|
||||
### Task 2: Update canonical operator and installation documentation
|
||||
|
||||
**Files:**
|
||||
- Modify: `README.md`
|
||||
- Modify: `AGENTS.md`
|
||||
- Modify: `PROJECT_STATE.md`
|
||||
- Modify: `docs/guida-utente.md`
|
||||
- Modify: `docs/contracts/workspace-preprocessing-cli.md`
|
||||
- Rename/update: `docs/contracts/tht-pi.md` as the current `tht` Pi contract
|
||||
- Modify: relevant installation and architecture pages that expose operator commands
|
||||
|
||||
**Step 1:** Replace current host/operator invocations with `tht --installation ...`.
|
||||
|
||||
**Step 2:** Document the distinction between the native host CLI `tht` and the Python harness CLI invoked by the backend/runtime.
|
||||
|
||||
**Step 3:** Update command examples for `start`, `status`, `doctor`, `auth`, `workspace`, and `pi`.
|
||||
|
||||
**Step 4:** Add a short historical note only where a document must explain the former name.
|
||||
|
||||
### Task 3: Rewrite authentication acceptance and manual test instructions
|
||||
|
||||
**Files:**
|
||||
- Modify: `docs/testing/authentication-manual-acceptance.md`
|
||||
- Modify: `docs/plans/2026-08-18-thothii-authentication-acceptance-and-psd-deployment.md`
|
||||
- Modify: `docs/install/authentication-local.md`
|
||||
- Modify: `docs/install/authentication-oidc.md`
|
||||
- Modify: `docs/install/authentik.md`
|
||||
|
||||
**Step 1:** Make `tht auth status`, `tht auth check`, `tht auth check --interactive`, and `tht doctor --json` the canonical terminal preflight.
|
||||
|
||||
**Step 2:** Use `tht status`, `tht start`, and `tht workspace inspect --workspace psd-clinical --json` for PSD deployment checks.
|
||||
|
||||
**Step 3:** Clarify that the P8 L2 gate is authentication-to-application integration through the first reviewer gate.
|
||||
|
||||
**Step 4:** Retain the prior functional test suite as a baseline and add only the authentication boundary smoke required for this acceptance.
|
||||
|
||||
### Task 4: Align documentation smoke tests
|
||||
|
||||
**Files:**
|
||||
- Modify: `scripts/auth-docs-smoke.sh`
|
||||
- Modify: `scripts/test-auth-docs-smoke.sh`
|
||||
- Inspect/update: any current smoke script whose user-facing command examples still require the legacy CLI name
|
||||
|
||||
**Step 1:** Replace forbidden/current command assertions with `tht` equivalents.
|
||||
|
||||
**Step 2:** Preserve negative checks for obsolete authentication CLI wording.
|
||||
|
||||
**Step 3:** Run the positive and negative documentation fixtures.
|
||||
|
||||
### Task 5: Preserve or annotate historical material
|
||||
|
||||
**Files:**
|
||||
- Inspect the historical discovery specification for context, without treating it as current operator documentation.
|
||||
- Inspect: dated reports and archived acceptance scripts
|
||||
|
||||
**Step 1:** Do not rewrite historical titles, commit evidence, or old implementation names solely to erase history.
|
||||
|
||||
**Step 2:** Add a concise “historical nomenclature” note where an archived document could otherwise be mistaken for current instructions.
|
||||
|
||||
### Task 6: Verify the convergence
|
||||
|
||||
**Step 1:** Run `scripts/auth-docs-smoke.sh` and `scripts/test-auth-docs-smoke.sh`.
|
||||
|
||||
**Step 2:** Search active documentation for remaining legacy CLI references.
|
||||
|
||||
**Step 3:** Confirm every remaining match is either an explicit historical note, an ignored runtime directory name, or a non-document executable compatibility artifact.
|
||||
|
||||
**Step 4:** Run `git diff --check` and report the exact files changed plus any intentionally retained historical references.
|
||||
Reference in New Issue
Block a user