diff --git a/docs/superpowers/plans/2026-07-11-adapter-foundations.md b/docs/superpowers/plans/2026-07-11-adapter-foundations.md new file mode 100644 index 00000000..f5974c6f --- /dev/null +++ b/docs/superpowers/plans/2026-07-11-adapter-foundations.md @@ -0,0 +1,305 @@ +# Adapter Foundations Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Route all DWH and vector operations through stable typed contracts while preserving current direct/REST behavior. + +**Architecture:** Define small Python protocols and capability records, implement adapters around existing modules, and centralize construction in one factory. Introduce a discriminated workspace schema with a compatibility translator for current YAML. + +**Tech Stack:** Python 3.11+, `typing.Protocol`, Pydantic 2, SQLAlchemy, psycopg2, requests, pytest, Typer. + +## Global Constraints + +- Do not change workflow phases or CLI output formats. +- PostgreSQL direct and current Thoth/PostgREST are the only supported DWH transports in this plan. +- Preserve reader/writer vector credential separation. +- All migrations must accept existing `database.transport`, `rest`, `vector_db`, `vector_rest`, and `vector_write_rest` fields. +- JSON stdout stays pristine; diagnostics go to stderr unless part of the JSON result. + +--- + +### Task 1: Define DWH ports and capabilities + +**Files:** +- Create: `harness/tht/ports/__init__.py` +- Create: `harness/tht/ports/dwh.py` +- Test: `harness/tests/test_dwh_port_contract.py` + +**Interfaces:** +- Produces: `DwhCapabilities`, `DwhAdapter`, `DwhHealth`, and `UnsupportedCapability`. +- Consumes: existing catalog models from `tht.db.introspect` and execution result types from `tht.db.execute`. + +- [ ] **Step 1: Write the failing protocol-shape test** + +```python +def test_fake_adapter_satisfies_runtime_protocol(): + adapter = FakeDwhAdapter() + assert isinstance(adapter, DwhAdapter) + assert adapter.capabilities.explain is True + assert adapter.health().ok is True +``` + +- [ ] **Step 2: Run the focused test and confirm failure** + +Run: `cd harness && .venv/bin/pytest tests/test_dwh_port_contract.py -q` +Expected: FAIL because `tht.ports.dwh` does not exist. + +- [ ] **Step 3: Add the minimal public contract** + +```python +@dataclass(frozen=True) +class DwhCapabilities: + introspection: bool = True + explain: bool = True + sampling: bool = True + distinct_values: bool = True + +@runtime_checkable +class DwhAdapter(Protocol): + @property + def capabilities(self) -> DwhCapabilities: ... + def health(self) -> DwhHealth: ... + def introspect(self) -> DatabaseCatalog: ... + def run_query(self, sql: str, *, limit: int | None = None) -> QueryResult: ... + def explain(self, sql: str) -> PlanSummary: ... + def sample_column(self, table: str, column: str, *, limit: int) -> list[object]: ... + def distinct_values(self, table: str, column: str) -> list[object]: ... +``` + +- [ ] **Step 4: Run contract test and type-oriented import smoke test** + +Run: `cd harness && .venv/bin/pytest tests/test_dwh_port_contract.py -q` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add harness/tht/ports harness/tests/test_dwh_port_contract.py +git commit -m "refactor(dwh): define adapter contract" +``` + +### Task 2: Wrap direct and REST DWH implementations + +**Files:** +- Create: `harness/tht/adapters/dwh/__init__.py` +- Create: `harness/tht/adapters/dwh/postgres.py` +- Create: `harness/tht/adapters/dwh/thoth_rest.py` +- Test: `harness/tests/test_dwh_adapters.py` +- Modify: `harness/tht/db/execute.py` +- Modify: `harness/tht/rest/execute.py` + +**Interfaces:** +- Consumes: `DwhAdapter` from Task 1; existing `DatabaseConfig`, `RestConfig`, catalog, sampling, execute, and explain functions. +- Produces: `PostgresDwhAdapter(config)` and `ThothRestDwhAdapter(database, rest)`. + +- [ ] **Step 1: Add parametrized contract tests for both wrappers** + +```python +@pytest.mark.parametrize("factory", [postgres_factory, rest_factory]) +def test_adapter_rejects_write_sql(factory): + with pytest.raises(ReadOnlyViolation): + factory().run_query("delete from fact_sales") +``` + +- [ ] **Step 2: Verify failure** + +Run: `cd harness && .venv/bin/pytest tests/test_dwh_adapters.py -q` +Expected: FAIL because the adapter classes are absent. + +- [ ] **Step 3: Implement thin wrappers, without duplicating transport logic** + +```python +class PostgresDwhAdapter: + capabilities = DwhCapabilities() + def __init__(self, config: DatabaseConfig): self._config = config + def run_query(self, sql: str, *, limit: int | None = None) -> QueryResult: + return run_query(self._config, sql, limit=limit) +``` + +Implement the analogous REST wrapper by delegating to `tht.rest.*`; translate transport-specific errors only at the adapter boundary. + +- [ ] **Step 4: Run adapter, read-only, sampling, and REST tests** + +Run: `cd harness && .venv/bin/pytest tests/test_dwh_adapters.py tests/test_readonly_guard.py tests/test_rest_client.py tests/l0/test_db_sampling.py -q` +Expected: PASS; L0 may deselect when Docker is unavailable. + +- [ ] **Step 5: Commit** + +```bash +git add harness/tht/adapters harness/tht/db/execute.py harness/tht/rest/execute.py harness/tests/test_dwh_adapters.py +git commit -m "refactor(dwh): adapt direct and REST transports" +``` + +### Task 3: Define vector port and wrappers + +**Files:** +- Create: `harness/tht/ports/vector.py` +- Create: `harness/tht/adapters/vector/__init__.py` +- Create: `harness/tht/adapters/vector/thoth_http.py` +- Create: `harness/tht/adapters/vector/legacy_direct.py` +- Test: `harness/tests/test_vector_port_contract.py` +- Modify: `harness/tht/vectorstore/reader.py` + +**Interfaces:** +- Produces: `VectorStore`, `VectorCapabilities`, `VectorHealth`, `VectorRecord`, `VectorHit`, `ThothHttpVectorStore`. +- Preserves: current `VectorRestClient`, `DirectSearcher`, and `RestSearcher` behavior behind wrappers. + +- [ ] **Step 1: Write read/write capability and dual-credential tests** + +```python +def test_http_store_reports_reader_without_writer(): + store = ThothHttpVectorStore(reader=reader, writer=None) + assert store.capabilities.search is True + assert store.capabilities.upsert is False + with pytest.raises(VectorWriteUnavailable): + store.upsert("memory", []) +``` + +- [ ] **Step 2: Verify failure** + +Run: `cd harness && .venv/bin/pytest tests/test_vector_port_contract.py -q` +Expected: FAIL because the vector port is absent. + +- [ ] **Step 3: Implement the vector contract and wrappers** + +```python +@runtime_checkable +class VectorStore(Protocol): + @property + def capabilities(self) -> VectorCapabilities: ... + def health(self) -> VectorHealth: ... + def search(self, collections: list[str], embedding: list[float], *, limit: int, + kinds: list[str] | None = None) -> list[VectorHit]: ... + def existing_hashes(self, collection: str, kinds: list[str]) -> dict[str, str]: ... + def upsert(self, collection: str, records: list[VectorRecord]) -> int: ... +``` + +- [ ] **Step 4: Run vector regression tests** + +Run: `cd harness && .venv/bin/pytest tests/test_vector_port_contract.py tests/test_vector_dual_key.py tests/test_search_similar_kinds.py tests/test_memory_save_one.py tests/test_solved_question.py -q` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add harness/tht/ports/vector.py harness/tht/adapters/vector harness/tht/vectorstore/reader.py harness/tests/test_vector_port_contract.py +git commit -m "refactor(vector): define store contract" +``` + +### Task 4: Introduce discriminated resource configuration with legacy translation + +**Files:** +- Create: `harness/tht/config_compat.py` +- Modify: `harness/tht/config.py` +- Modify: `harness/workspaces/tht.example.yaml` +- Test: `harness/tests/test_config_resources.py` +- Test: `harness/tests/test_config_legacy_compat.py` + +**Interfaces:** +- Produces: `DwhResourceConfig`, `VectorResourceConfig`, `WorkspaceRoots` and `translate_legacy_config(raw)`. +- Consumes: existing YAML environment expansion and `ConfigError` behavior. + +- [ ] **Step 1: Write new-schema and legacy-equivalence tests** + +```python +def test_legacy_rest_workspace_equals_new_resource_schema(tmp_path): + old = load_config(write_old_workspace(tmp_path)) + new = load_config(write_new_workspace(tmp_path)) + assert old.dwh.model_dump() == new.dwh.model_dump() +``` + +- [ ] **Step 2: Verify failure** + +Run: `cd harness && .venv/bin/pytest tests/test_config_resources.py tests/test_config_legacy_compat.py -q` +Expected: FAIL because `dwh` and discriminated vector resources are absent. + +- [ ] **Step 3: Add discriminated models and an isolated translator** + +```python +class PostgresDwhConfig(BaseModel): + type: Literal["postgres_direct"] + connection: DatabaseConfig + +class ThothRestDwhConfig(BaseModel): + type: Literal["thoth_rest"] + database: DatabaseIdentityConfig + endpoint: RestConfig + +DwhResourceConfig = Annotated[ + PostgresDwhConfig | ThothRestDwhConfig, + Field(discriminator="type"), +] +``` + +Translate legacy keys before Pydantic validation and emit one deprecation warning to stderr, never stdout. + +- [ ] **Step 4: Run all config and workspace tests** + +Run: `cd harness && .venv/bin/pytest tests/test_config_resources.py tests/test_config_legacy_compat.py tests/test_workspace.py tests/test_vector_dual_key.py -q` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add harness/tht/config.py harness/tht/config_compat.py harness/workspaces/tht.example.yaml harness/tests/test_config_resources.py harness/tests/test_config_legacy_compat.py +git commit -m "feat(config): add typed resource schema" +``` + +### Task 5: Centralize construction and migrate command call sites + +**Files:** +- Create: `harness/tht/adapters/factory.py` +- Modify: `harness/tht/cli/db_cmd.py` +- Modify: `harness/tht/cli/schema_cmd.py` +- Modify: `harness/tht/cli/sql_cmd.py` +- Modify: `harness/tht/cli/lsh_cmd.py` +- Modify: `harness/tht/cli/search_cmd.py` +- Modify: `harness/tht/cli/vector_cmd.py` +- Modify: `harness/tht/cli/memory_cmd.py` +- Test: `harness/tests/test_adapter_factory.py` + +**Interfaces:** +- Produces: `build_dwh(cfg: Config) -> DwhAdapter` and `build_vector_store(cfg: Config, *, require_write: bool = False) -> VectorStore`. +- Consumes: resource configs from Task 4 and wrappers from Tasks 2-3. + +- [ ] **Step 1: Write exact factory selection and missing-writer tests** + +```python +def test_factory_selects_http_vector_and_requires_writer(config): + assert isinstance(build_vector_store(config), ThothHttpVectorStore) + with pytest.raises(ConfigError, match="writer"): + build_vector_store(config, require_write=True) +``` + +- [ ] **Step 2: Verify failure** + +Run: `cd harness && .venv/bin/pytest tests/test_adapter_factory.py -q` +Expected: FAIL because the factory does not exist. + +- [ ] **Step 3: Implement factory and replace per-command branching** + +```python +def build_dwh(cfg: Config) -> DwhAdapter: + match cfg.dwh.type: + case "postgres_direct": return PostgresDwhAdapter(cfg.dwh.connection) + case "thoth_rest": return ThothRestDwhAdapter(cfg.dwh.database, cfg.dwh.endpoint) + case other: raise ConfigError(f"Adapter DWH non supportato: {other}") +``` + +Delete transport checks from migrated commands; keep CLI wording and exit codes stable. + +- [ ] **Step 4: Run focused command suites, full harness suite, and ruff** + +Run: `cd harness && .venv/bin/pytest tests/test_adapter_factory.py tests/test_schema_introspect_guard.py tests/test_sql_preview_json.py tests/test_search_pack.py tests/integration/test_gate_cli_signatures.py -q` +Expected: PASS. +Run: `cd harness && .venv/bin/pytest -q` +Expected: all non-L2 tests PASS. +Run: `cd harness && .venv/bin/ruff check tht tests/test_adapter_factory.py tests/test_dwh_port_contract.py tests/test_dwh_adapters.py tests/test_vector_port_contract.py tests/test_config_resources.py tests/test_config_legacy_compat.py` +Expected: no errors. + +- [ ] **Step 5: Commit** + +```bash +git add harness/tht harness/tests harness/workspaces/tht.example.yaml +git commit -m "refactor(core): route integrations through adapter factory" +``` diff --git a/docs/superpowers/plans/2026-07-11-container-packaging-portable-storage.md b/docs/superpowers/plans/2026-07-11-container-packaging-portable-storage.md new file mode 100644 index 00000000..45bf4937 --- /dev/null +++ b/docs/superpowers/plans/2026-07-11-container-packaging-portable-storage.md @@ -0,0 +1,295 @@ +# Container Packaging and Portable Storage Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Run ThothII from exactly two application images with runtime configuration and host-independent persistent roots. + +**Architecture:** Build a multi-runtime core image for Fastify, Pi, and `tht`, plus a static frontend image. Resolve workspace paths beneath mounted logical roots and provide a Compose base for external dependencies. + +**Tech Stack:** Docker BuildKit, Docker Compose v2, Node 22, Python 3.11, Fastify, React/Vite, nginx or Caddy. + +## Global Constraints + +- Plan 1 is complete and adapter factories are the only integration construction path. +- Do not bake secrets or customer workspace content into images. +- Containers run as non-root and write only beneath mounted data roots. +- `pi`, `tht`, CA certificates, and native dependencies must work on every advertised architecture. +- Frontend backend URL is runtime-configurable. + +--- + +### Task 1: Add logical workspace roots and migration diagnostics + +**Files:** +- Create: `harness/tht/paths.py` +- Create: `harness/tht/cli/doctor_cmd.py` +- Modify: `harness/tht/config.py` +- Modify: `harness/tht/cli/__init__.py` +- Test: `harness/tests/test_portable_paths.py` +- Test: `harness/tests/test_doctor_cli.py` + +**Interfaces:** +- Produces: `resolve_workspace_paths(config_path, cfg, data_root) -> ResolvedPaths` and `tht doctor --json`. + +- [ ] **Step 1: Test relative, absolute-legacy, and escape rejection** + +```python +def test_relative_paths_resolve_under_workspace_root(tmp_path): + resolved = resolve_workspace_paths(cfg_path, cfg, tmp_path / "data") + assert resolved.sessions == tmp_path / "data/workspaces/demo/sessions" + +def test_path_escape_is_rejected(tmp_path): + with pytest.raises(ConfigError, match="outside workspace root"): + resolve_workspace_paths(cfg_path, config_with_sessions("../../private"), tmp_path) +``` + +- [ ] **Step 2: Verify failure** + +Run: `cd harness && .venv/bin/pytest tests/test_portable_paths.py tests/test_doctor_cli.py -q` +Expected: FAIL. + +- [ ] **Step 3: Implement resolution and JSON diagnostics** + +```python +@dataclass(frozen=True) +class ResolvedPaths: + workspace: Path + sessions: Path + artifacts: Path + indexes: Path + corpus: Path +``` + +`doctor --json` returns component statuses without printing warnings to stdout. + +- [ ] **Step 4: Run tests** + +Run: `cd harness && .venv/bin/pytest tests/test_portable_paths.py tests/test_doctor_cli.py tests/test_workspace.py -q` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add harness/tht/paths.py harness/tht/cli/doctor_cmd.py harness/tht/config.py harness/tht/cli/__init__.py harness/tests/test_portable_paths.py harness/tests/test_doctor_cli.py +git commit -m "feat(storage): resolve portable workspace roots" +``` + +### Task 2: Make backend process paths and listening address container-safe + +**Files:** +- Modify: `backend/src/config.ts` +- Modify: `backend/src/server.ts` +- Modify: `backend/src/pi/pi-process-manager.ts` +- Modify: `backend/src/tht/tht-runner.ts` +- Test: `backend/test/config.test.ts` +- Test: `backend/test/pi-process-manager.test.ts` + +**Interfaces:** +- Produces env contract: `HOST`, `PORT`, `THT_HARNESS_DIR`, `THT_BIN`, `PI_BIN`, `SETTINGS_FILE`, `THT_DATA_ROOT`. + +- [ ] **Step 1: Add tests for explicit binaries and `0.0.0.0` listening** + +```typescript +expect(loadConfig({ HOST: "0.0.0.0", THT_BIN: "/opt/venv/bin/tht" })).toMatchObject({ + host: "0.0.0.0", thtBin: "/opt/venv/bin/tht" +}); +``` + +- [ ] **Step 2: Verify failure** + +Run: `cd backend && npx vitest run test/config.test.ts test/pi-process-manager.test.ts` +Expected: FAIL for missing host/data-root behavior. + +- [ ] **Step 3: Remove the `.venv/bin` PATH assumption** + +```typescript +const env = { ...process.env, THT_DATA_ROOT: cfg.dataRoot }; +const child = spawn(cfg.piBin, ["--mode", "rpc"], { cwd: cfg.harnessDir, env }); +``` + +- [ ] **Step 4: Run backend tests and typecheck** + +Run: `cd backend && npx vitest run && npx tsc --noEmit -p .` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add backend/src backend/test +git commit -m "feat(backend): support container runtime paths" +``` + +### Task 3: Build the core image + +**Files:** +- Create: `docker/core.Dockerfile` +- Create: `docker/core-entrypoint.sh` +- Create: `.dockerignore` +- Create: `docker/smoke/core-smoke.sh` + +**Interfaces:** +- Produces image entrypoints: `server`, `doctor`, `preprocess`, and arbitrary `tht ...`. + +- [ ] **Step 1: Add a smoke script that asserts binaries and health** + +```sh +test "$(id -u)" != "0" +node --version +python --version +tht --help >/dev/null +pi --version >/dev/null +curl --fail http://127.0.0.1:8787/health +``` + +- [ ] **Step 2: Build and observe the initial failure** + +Run: `docker build -f docker/core.Dockerfile -t thothii-core:test .` +Expected: FAIL because the Dockerfile is not present before implementation. + +- [ ] **Step 3: Implement a multi-stage core build** + +```dockerfile +FROM node:22-bookworm AS backend-build +WORKDIR /src/backend +COPY backend/package*.json ./ +RUN npm ci +COPY backend/ ./ +RUN npm run build + +FROM python:3.11-slim-bookworm AS runtime +RUN useradd --create-home --uid 10001 thoth +WORKDIR /app +COPY harness/ /app/harness/ +RUN python -m venv /opt/venv && /opt/venv/bin/pip install --no-cache-dir /app/harness +COPY --from=backend-build /src/backend/dist /app/backend/dist +COPY --from=backend-build /src/backend/node_modules /app/backend/node_modules +ENV PATH="/opt/venv/bin:$PATH" HOST=0.0.0.0 PORT=8787 +USER thoth +ENTRYPOINT ["/app/docker/core-entrypoint.sh"] +CMD ["server"] +``` + +Add Pi installation only from its pinned, redistributable source after the Phase 0 license/runtime gate; fail the build if `pi --version` is unavailable. + +- [ ] **Step 4: Build and run smoke test** + +Run: `docker build -f docker/core.Dockerfile -t thothii-core:test .` +Expected: success. +Run: `docker run --rm thothii-core:test doctor` +Expected: process starts and reports missing external configuration without traceback. + +- [ ] **Step 5: Commit** + +```bash +git add docker/core.Dockerfile docker/core-entrypoint.sh docker/smoke/core-smoke.sh .dockerignore +git commit -m "build(docker): add core application image" +``` + +### Task 4: Add runtime frontend configuration and image + +**Files:** +- Create: `frontend/public/config.js` +- Create: `frontend/src/api/runtime-config.ts` +- Modify: `frontend/src/api/client.ts` +- Modify: `frontend/index.html` +- Create: `docker/frontend.Dockerfile` +- Create: `docker/frontend-entrypoint.sh` +- Create: `docker/nginx.conf.template` +- Test: `frontend/src/api/runtime-config.test.ts` + +**Interfaces:** +- Produces browser contract: `window.__THOTHII_CONFIG__.backendBaseUrl`. + +- [ ] **Step 1: Write fallback and injected-config tests** + +```typescript +expect(resolveBackendUrl({ backendBaseUrl: "/api" })).toBe("/api"); +expect(resolveBackendUrl(undefined)).toBe(import.meta.env.VITE_BACKEND_URL ?? ""); +``` + +- [ ] **Step 2: Verify failure** + +Run: `cd frontend && npx vitest run src/api/runtime-config.test.ts` +Expected: FAIL. + +- [ ] **Step 3: Implement runtime config generation and reverse proxy** + +```sh +sed "s|__BACKEND_BASE_URL__|${BACKEND_BASE_URL:-/api}|g" \ + /usr/share/nginx/html/config.template.js > /usr/share/nginx/html/config.js +exec nginx -g 'daemon off;' +``` + +- [ ] **Step 4: Test and build** + +Run: `cd frontend && npx vitest run && npx tsc -b && npm run build` +Expected: PASS. +Run: `docker build -f docker/frontend.Dockerfile -t thothii-frontend:test .` +Expected: success. + +- [ ] **Step 5: Commit** + +```bash +git add frontend docker/frontend.Dockerfile docker/frontend-entrypoint.sh docker/nginx.conf.template +git commit -m "build(docker): add runtime-configured frontend image" +``` + +### Task 5: Compose external profile and end-to-end smoke gate + +**Files:** +- Create: `compose.yaml` +- Create: `deploy/env.example` +- Create: `deploy/workspaces/example.yaml` +- Create: `scripts/docker-smoke.sh` +- Modify: `README.md` +- Test: `backend/test/health.test.ts` + +**Interfaces:** +- Produces services `core` and `frontend`; persistent volume/mount contract beneath `/data`. + +- [ ] **Step 1: Add a smoke script for Compose config and HTTP health** + +```sh +docker compose config --quiet +docker compose up --build --wait core frontend +curl --fail http://localhost:8080/health +docker compose down +``` + +- [ ] **Step 2: Verify Compose is initially absent** + +Run: `docker compose config --quiet` +Expected: FAIL before `compose.yaml` is implemented. + +- [ ] **Step 3: Define the base deployment** + +```yaml +services: + core: + build: { context: ., dockerfile: docker/core.Dockerfile } + environment: + THT_DATA_ROOT: /data + SETTINGS_FILE: /data/settings/settings.json + volumes: ["./deploy/workspaces:/data/workspaces:ro", "thoth_data:/data"] + frontend: + build: { context: ., dockerfile: docker/frontend.Dockerfile } + environment: { BACKEND_BASE_URL: /api } + ports: ["8080:8080"] +volumes: { thoth_data: {} } +``` + +- [ ] **Step 4: Run Compose smoke and full layer gates** + +Run: `./scripts/docker-smoke.sh` +Expected: both health checks PASS. +Run: `cd harness && .venv/bin/pytest -q` +Run: `cd backend && npx vitest run && npx tsc --noEmit -p .` +Run: `cd frontend && npx vitest run && npx tsc -b` +Expected: all PASS. + +- [ ] **Step 5: Commit** + +```bash +git add compose.yaml deploy scripts/docker-smoke.sh README.md +git commit -m "feat(deploy): add portable external-service stack" +``` diff --git a/docs/superpowers/plans/2026-07-11-evidence-preprocessing.md b/docs/superpowers/plans/2026-07-11-evidence-preprocessing.md new file mode 100644 index 00000000..1cf522d6 --- /dev/null +++ b/docs/superpowers/plans/2026-07-11-evidence-preprocessing.md @@ -0,0 +1,373 @@ +# Evidence Sources and Preprocessing Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Materialize heterogeneous Evidence sources into a versioned canonical corpus and index it through resumable, idempotent jobs. + +**Architecture:** Source adapters only discover and acquire. Pure normalization/chunking stages create immutable version artifacts; vector indexing writes a staging generation; publish atomically switches the active manifest. DWH preprocessing uses the same job envelope but a separate pipeline. + +**Tech Stack:** Python 3.11+, Pydantic 2, requests, optional `fsspec`/S3 client, existing embeddings/vector ports, Typer, pytest. + +## Global Constraints + +- Plans 1-3 are complete. +- Runtime search reads only the active canonical corpus and vector generation. +- Filesystem and HTTP sources are MVP; S3-compatible follows on the same port. +- Source credentials never enter corpus metadata or logs. +- Failed runs never replace the last valid published generation. +- Document and DWH preprocessing are separate jobs with shared lock/report infrastructure. + +--- + +### Task 1: Define Evidence source port and canonical records + +**Files:** +- Create: `harness/tht/ports/evidence.py` +- Create: `harness/tht/corpus/models.py` +- Test: `harness/tests/test_evidence_port_contract.py` +- Test: `harness/tests/test_corpus_models.py` + +**Interfaces:** +- Produces: `EvidenceSource`, `SourceObject`, `AcquiredDocument`, `CanonicalDocument`, `CanonicalChunk`, `CorpusManifest`. + +- [ ] **Step 1: Write serialization and secret-exclusion tests** + +```python +def test_manifest_contains_provenance_without_credentials(): + manifest = CorpusManifest(documents=[document(source_uri="https://host/a.md")]) + payload = manifest.model_dump_json() + assert "https://host/a.md" in payload + assert "api_key" not in payload +``` + +- [ ] **Step 2: Verify failure** + +Run: `cd harness && .venv/bin/pytest tests/test_evidence_port_contract.py tests/test_corpus_models.py -q` +Expected: FAIL. + +- [ ] **Step 3: Define immutable records and source protocol** + +```python +@runtime_checkable +class EvidenceSource(Protocol): + def discover(self) -> Iterable[SourceObject]: ... + def acquire(self, item: SourceObject) -> AcquiredDocument: ... + +class SourceObject(BaseModel, frozen=True): + source_id: str + uri: str + fingerprint: str + modified_at: datetime | None = None + metadata: dict[str, JsonValue] = {} +``` + +- [ ] **Step 4: Run model and protocol tests** + +Run: `cd harness && .venv/bin/pytest tests/test_evidence_port_contract.py tests/test_corpus_models.py -q` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add harness/tht/ports/evidence.py harness/tht/corpus/models.py harness/tests/test_evidence_port_contract.py harness/tests/test_corpus_models.py +git commit -m "feat(evidence): define source and corpus contracts" +``` + +### Task 2: Implement filesystem and HTTP source adapters + +**Files:** +- Create: `harness/tht/adapters/evidence/__init__.py` +- Create: `harness/tht/adapters/evidence/filesystem.py` +- Create: `harness/tht/adapters/evidence/http.py` +- Modify: `harness/tht/config.py` +- Modify: `harness/tht/adapters/factory.py` +- Test: `harness/tests/test_filesystem_evidence_source.py` +- Test: `harness/tests/test_http_evidence_source.py` + +**Interfaces:** +- Produces: `FilesystemEvidenceSource`, `HttpManifestEvidenceSource`, `build_evidence_sources(cfg)`. + +- [ ] **Step 1: Test deterministic discovery and conditional HTTP acquisition** + +```python +def test_filesystem_discovery_is_stable(source): + assert [x.uri for x in source.discover()] == sorted(x.uri for x in source.discover()) + +def test_http_uses_etag_for_fingerprint(http_source): + assert next(http_source.discover()).fingerprint == 'etag:"abc"' +``` + +- [ ] **Step 2: Verify failure** + +Run: `cd harness && .venv/bin/pytest tests/test_filesystem_evidence_source.py tests/test_http_evidence_source.py -q` +Expected: FAIL. + +- [ ] **Step 3: Implement adapters with bounded reads** + +```python +class FilesystemEvidenceSource: + def discover(self): + for path in sorted(self.root.rglob("*.md")): + yield SourceObject(source_id=stable_id(path), uri=path.as_uri(), + fingerprint=sha256_file(path)) +``` + +HTTP uses a declared manifest of URLs, connect/read timeouts, maximum bytes, ETag/Last-Modified where available, and content hashing as fallback. + +- [ ] **Step 4: Run adapter tests** + +Run: `cd harness && .venv/bin/pytest tests/test_filesystem_evidence_source.py tests/test_http_evidence_source.py tests/test_config_resources.py -q` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add harness/tht/adapters/evidence harness/tht/config.py harness/tht/adapters/factory.py harness/tests/test_filesystem_evidence_source.py harness/tests/test_http_evidence_source.py +git commit -m "feat(evidence): add filesystem and HTTP sources" +``` + +### Task 3: Build deterministic normalization and chunking + +**Files:** +- Create: `harness/tht/corpus/normalize.py` +- Create: `harness/tht/corpus/chunk.py` +- Test: `harness/tests/test_corpus_normalize.py` +- Test: `harness/tests/test_corpus_chunk.py` + +**Interfaces:** +- Produces: `normalize(acquired, pipeline_version) -> CanonicalDocument` and `chunk(document, policy) -> list[CanonicalChunk]`. + +- [ ] **Step 1: Pin UTF-8, frontmatter, line-ending, and stable chunk-id behavior** + +```python +def test_chunk_ids_are_stable_for_same_content(): + first = chunk(document("A\n\nB"), policy(max_chars=8)) + second = chunk(document("A\r\n\r\nB"), policy(max_chars=8)) + assert [x.chunk_id for x in first] == [x.chunk_id for x in second] +``` + +- [ ] **Step 2: Verify failure** + +Run: `cd harness && .venv/bin/pytest tests/test_corpus_normalize.py tests/test_corpus_chunk.py -q` +Expected: FAIL. + +- [ ] **Step 3: Implement pure deterministic transforms** + +```python +chunk_id = sha256(f"{document.content_hash}:{ordinal}:{policy.version}".encode()).hexdigest() +``` + +Reject undecodable or oversized content with a typed permanent error; never silently truncate source documents. + +- [ ] **Step 4: Run normalization/chunk tests** + +Run: `cd harness && .venv/bin/pytest tests/test_corpus_normalize.py tests/test_corpus_chunk.py -q` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add harness/tht/corpus/normalize.py harness/tht/corpus/chunk.py harness/tests/test_corpus_normalize.py harness/tests/test_corpus_chunk.py +git commit -m "feat(corpus): add deterministic normalization and chunking" +``` + +### Task 4: Add shared job envelope, locking, and reports + +**Files:** +- Create: `harness/tht/jobs/models.py` +- Create: `harness/tht/jobs/runner.py` +- Create: `harness/tht/jobs/locking.py` +- Test: `harness/tests/test_job_runner.py` +- Test: `harness/tests/test_job_locking.py` + +**Interfaces:** +- Produces: `JobSpec`, `JobRun`, `JobReport`, `WorkspaceJobLock`, `run_job(spec, stages)`. + +- [ ] **Step 1: Test lock exclusion, resume, and JSON report schema** + +```python +def test_failed_stage_is_resumable(tmp_path): + first = run_job(spec, [ok_stage, failing_stage]) + second = run_job(spec.with_resume(first.run_id), [ok_stage, recovered_stage]) + assert second.resumed_from == first.run_id + assert second.status == "succeeded" +``` + +- [ ] **Step 2: Verify failure** + +Run: `cd harness && .venv/bin/pytest tests/test_job_runner.py tests/test_job_locking.py -q` +Expected: FAIL. + +- [ ] **Step 3: Implement atomic report writes and workspace-scoped locks** + +```python +tmp = report_path.with_suffix(".tmp") +tmp.write_text(report.model_dump_json(indent=2)) +tmp.replace(report_path) +``` + +Persist checkpoints after each stage; ensure a crashed process leaves the active corpus untouched. + +- [ ] **Step 4: Run job infrastructure tests** + +Run: `cd harness && .venv/bin/pytest tests/test_job_runner.py tests/test_job_locking.py -q` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add harness/tht/jobs harness/tests/test_job_runner.py harness/tests/test_job_locking.py +git commit -m "feat(jobs): add resumable preprocessing envelope" +``` + +### Task 5: Implement incremental document pipeline and atomic publish + +**Files:** +- Create: `harness/tht/corpus/pipeline.py` +- Create: `harness/tht/corpus/store.py` +- Create: `harness/tht/cli/preprocess_cmd.py` +- Modify: `harness/tht/cli/__init__.py` +- Create: `harness/tht/search/evidence.py` +- Test: `harness/tests/test_corpus_pipeline.py` +- Test: `harness/tests/test_corpus_publish.py` +- Test: `harness/tests/test_preprocess_cli.py` + +**Interfaces:** +- Produces: `tht preprocess evidence [--dry-run] [--resume RUN_ID] [--json]`; active pointer `corpus//ACTIVE`. +- Consumes: Evidence sources, canonical transforms, embedder, and `VectorStore`. + +- [ ] **Step 1: Test incremental skip and failed-run isolation** + +```python +def test_failed_generation_does_not_replace_active(corpus_store, pipeline): + old = corpus_store.publish(valid_generation()) + with pytest.raises(StageError): pipeline.run(source_with_failure()) + assert corpus_store.active_generation() == old +``` + +- [ ] **Step 2: Verify failure** + +Run: `cd harness && .venv/bin/pytest tests/test_corpus_pipeline.py tests/test_corpus_publish.py tests/test_preprocess_cli.py -q` +Expected: FAIL. + +- [ ] **Step 3: Implement staged generations and compare fingerprints** + +```python +changed = { + item.source_id for item in discovered + if previous.fingerprints.get(item.source_id) != item.fingerprint +} +``` + +Index changed chunks, mark removed documents, validate counts and embedding dimensions, then atomically replace `ACTIVE`. Make runtime Evidence lookup resolve files through the active manifest instead of `rglob` on the source directory. + +- [ ] **Step 4: Run pipeline and existing search/session tests** + +Run: `cd harness && .venv/bin/pytest tests/test_corpus_pipeline.py tests/test_corpus_publish.py tests/test_preprocess_cli.py tests/test_search_pack.py tests/test_session_documents.py -q` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add harness/tht/corpus harness/tht/cli/preprocess_cmd.py harness/tht/cli/__init__.py harness/tht/search/evidence.py harness/tests/test_corpus_pipeline.py harness/tests/test_corpus_publish.py harness/tests/test_preprocess_cli.py +git commit -m "feat(preprocess): publish incremental Evidence corpus" +``` + +### Task 6: Move DWH introspection and LSH into separate jobs + +**Files:** +- Create: `harness/tht/jobs/dwh_pipeline.py` +- Modify: `harness/tht/cli/schema_cmd.py` +- Modify: `harness/tht/cli/lsh_cmd.py` +- Modify: `harness/tht/cli/preprocess_cmd.py` +- Test: `harness/tests/test_dwh_preprocess_job.py` +- Test: `harness/tests/test_lsh_job_resume.py` + +**Interfaces:** +- Produces: `tht preprocess dwh --steps introspect,lsh [--resume RUN_ID] [--json]`. + +- [ ] **Step 1: Test independent document/DWH locks and LSH resume** + +```python +def test_dwh_and_evidence_jobs_have_distinct_lock_names(): + assert job_lock_name("demo", "dwh") != job_lock_name("demo", "evidence") +``` + +- [ ] **Step 2: Verify failure** + +Run: `cd harness && .venv/bin/pytest tests/test_dwh_preprocess_job.py tests/test_lsh_job_resume.py -q` +Expected: FAIL. + +- [ ] **Step 3: Wrap existing commands as stages without changing their core algorithms** + +```python +stages = { + "introspect": lambda ctx: refresh_catalog(ctx.dwh, ctx.paths.artifacts), + "lsh": lambda ctx: build_lsh(ctx.dwh, ctx.paths.indexes, ctx.checkpoint), +} +``` + +- [ ] **Step 4: Run DWH/LSH regression and full harness suite** + +Run: `cd harness && .venv/bin/pytest tests/test_dwh_preprocess_job.py tests/test_lsh_job_resume.py tests/test_schema_introspect_guard.py tests/l0/test_db_sampling.py -q` +Run: `cd harness && .venv/bin/pytest -q` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add harness/tht/jobs/dwh_pipeline.py harness/tht/cli/schema_cmd.py harness/tht/cli/lsh_cmd.py harness/tht/cli/preprocess_cmd.py harness/tests/test_dwh_preprocess_job.py harness/tests/test_lsh_job_resume.py +git commit -m "feat(preprocess): add resumable DWH jobs" +``` + +### Task 7: Add Compose job profiles, S3 follow-up adapter, and operational gates + +**Files:** +- Modify: `compose.yaml` +- Create: `harness/tht/adapters/evidence/s3.py` +- Modify: `harness/pyproject.toml` +- Test: `harness/tests/test_s3_evidence_source.py` +- Create: `scripts/preprocess-smoke.sh` +- Modify: `README.md` + +**Interfaces:** +- Produces Compose profile `preprocess`; optional `s3` source type using endpoint URL, bucket, prefix, and secret references. + +- [ ] **Step 1: Add S3 contract tests against a fake endpoint and a Compose job smoke** + +```python +def test_s3_uri_and_fingerprint(source): + item = next(source.discover()) + assert item.uri == "s3://evidence/clinical/a.md" + assert item.fingerprint.startswith("etag:") +``` + +- [ ] **Step 2: Verify failure** + +Run: `cd harness && .venv/bin/pytest tests/test_s3_evidence_source.py -q` +Expected: FAIL. + +- [ ] **Step 3: Implement S3 on the established port and Compose one-shot services** + +```yaml +preprocess-evidence: + image: thothii-core:${THOTHII_TAG:-latest} + profiles: ["preprocess"] + command: ["preprocess", "evidence", "--json"] + volumes: ["thoth_data:/data"] +``` + +- [ ] **Step 4: Run all operational gates** + +Run: `./scripts/preprocess-smoke.sh` +Expected: second run reports all documents unchanged; a modified file creates and publishes one new generation. +Run: `cd harness && .venv/bin/pytest -q` +Run: `cd harness && .venv/bin/ruff check tht tests/test_*evidence* tests/test_corpus* tests/test_*job*` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add compose.yaml harness/tht/adapters/evidence/s3.py harness/pyproject.toml harness/tests/test_s3_evidence_source.py scripts/preprocess-smoke.sh README.md +git commit -m "feat(preprocess): add deployment jobs and S3 source" +``` diff --git a/docs/superpowers/plans/2026-07-11-local-pgvector-profile.md b/docs/superpowers/plans/2026-07-11-local-pgvector-profile.md new file mode 100644 index 00000000..d269d369 --- /dev/null +++ b/docs/superpowers/plans/2026-07-11-local-pgvector-profile.md @@ -0,0 +1,211 @@ +# Optional Local pgvector Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Let server and desktop deployments run an optional persistent pgvector service with behavior equivalent to the HTTP vector adapter. + +**Architecture:** Implement the final direct `VectorStore`, version vector schema migrations, add a standard pgvector image to Compose, and provide operational backup/restore commands. + +**Tech Stack:** PostgreSQL 16, pgvector, psycopg2/SQLAlchemy, Alembic or ordered SQL migrations, Docker Compose, pytest/testcontainers. + +## Global Constraints + +- Plans 1 and 2 are complete. +- pgvector is an infrastructure image, not a ThothII-owned application image. +- Reader and writer roles are distinct even for local deployments. +- Existing remote HTTP vector behavior remains supported. +- Persistent data must survive application image replacement. + +--- + +### Task 1: Implement direct pgvector store behind `VectorStore` + +**Files:** +- Create: `harness/tht/adapters/vector/pgvector.py` +- Test: `harness/tests/l0/test_pgvector_store.py` +- Modify: `harness/tht/adapters/factory.py` +- Modify: `harness/tht/config.py` + +**Interfaces:** +- Produces: `PgVectorStore(read_config, write_config=None)` implementing the Plan 1 port. + +- [ ] **Step 1: Add L0 contract tests for search, kind filters, hashes, and upsert** + +```python +def test_pgvector_round_trip(store): + assert store.upsert("memory", [record("a", [1.0, 0.0])]) == 1 + hits = store.search(["memory"], [1.0, 0.0], limit=5, kinds=["memory"]) + assert hits[0].metadata["content_hash"] == "a" +``` + +- [ ] **Step 2: Verify failure** + +Run: `cd harness && .venv/bin/pytest tests/l0/test_pgvector_store.py -q` +Expected: FAIL because `PgVectorStore` is absent. + +- [ ] **Step 3: Implement parameterized SQL with allowlisted collection names** + +```python +def _collection(name: str) -> sql.Identifier: + if name not in ALLOWED_COLLECTIONS: + raise VectorStoreError(f"Collection not allowed: {name}") + return sql.Identifier("vectors", name) +``` + +Do not interpolate untrusted identifiers; reuse current metadata shapes and cosine distance ordering. + +- [ ] **Step 4: Run direct and HTTP parity tests** + +Run: `cd harness && .venv/bin/pytest tests/l0/test_pgvector_store.py tests/test_vector_port_contract.py tests/test_search_similar_kinds.py -q` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add harness/tht/adapters/vector/pgvector.py harness/tht/adapters/factory.py harness/tht/config.py harness/tests/l0/test_pgvector_store.py +git commit -m "feat(vector): add direct pgvector adapter" +``` + +### Task 2: Version schema and roles + +**Files:** +- Create: `harness/migrations/vector/001_extensions.sql` +- Create: `harness/migrations/vector/002_schema_tables.sql` +- Create: `harness/migrations/vector/003_roles.sql` +- Create: `harness/tht/cli/vector_migrate_cmd.py` +- Test: `harness/tests/l0/test_vector_migrations.py` + +**Interfaces:** +- Produces: `tht vector migrate`, `tht vector migrate --status --json`. + +- [ ] **Step 1: Test clean install and idempotent rerun** + +```python +def test_migrations_are_idempotent(database_url): + migrate(database_url) + migrate(database_url) + assert migration_status(database_url).pending == [] +``` + +- [ ] **Step 2: Verify failure** + +Run: `cd harness && .venv/bin/pytest tests/l0/test_vector_migrations.py -q` +Expected: FAIL. + +- [ ] **Step 3: Add ordered migrations and least-privilege roles** + +```sql +CREATE SCHEMA IF NOT EXISTS vectors; +CREATE TABLE IF NOT EXISTS vectors.schema (..., embedding vector(768) NOT NULL); +CREATE TABLE IF NOT EXISTS vectors.memory (..., embedding vector(768) NOT NULL); +REVOKE ALL ON SCHEMA vectors FROM PUBLIC; +``` + +Create reader grants for SELECT/search and writer grants for controlled insert/update; credentials are injected at deployment, not stored in SQL files. + +- [ ] **Step 4: Run migration and existing vector tests** + +Run: `cd harness && .venv/bin/pytest tests/l0/test_vector_migrations.py tests/l0/test_pgvector_store.py -q` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add harness/migrations/vector harness/tht/cli/vector_migrate_cmd.py harness/tests/l0/test_vector_migrations.py +git commit -m "feat(vector): version pgvector schema" +``` + +### Task 3: Add `local-vector` Compose profile + +**Files:** +- Modify: `compose.yaml` +- Create: `deploy/vector/init/00-bootstrap.sh` +- Modify: `deploy/env.example` +- Create: `scripts/local-vector-smoke.sh` + +**Interfaces:** +- Produces service `vector-db`, volume `vector_data`, health-gated core dependency in the profile. + +- [ ] **Step 1: Add failing profile smoke command** + +Run: `docker compose --profile local-vector config --services` +Expected: output does not yet contain `vector-db`. + +- [ ] **Step 2: Define the standard infrastructure service** + +```yaml +vector-db: + image: pgvector/pgvector:pg16 + profiles: ["local-vector"] + volumes: ["vector_data:/var/lib/postgresql/data"] + healthcheck: + test: ["CMD-SHELL", "pg_isready -U $$POSTGRES_USER -d $$POSTGRES_DB"] + interval: 5s + timeout: 3s + retries: 20 +``` + +- [ ] **Step 3: Wire the local resource config and migration job** + +Add a one-shot `vector-migrate` service using `thothii-core`; it must complete successfully before preprocessing writes. + +- [ ] **Step 4: Run local vector smoke** + +Run: `./scripts/local-vector-smoke.sh` +Expected: migration succeeds, one record is indexed, and search returns it after restarting `core`. + +- [ ] **Step 5: Commit** + +```bash +git add compose.yaml deploy/vector deploy/env.example scripts/local-vector-smoke.sh +git commit -m "feat(deploy): add optional local pgvector profile" +``` + +### Task 4: Add backup, restore, and parity gates + +**Files:** +- Create: `scripts/vector-backup.sh` +- Create: `scripts/vector-restore.sh` +- Create: `harness/tests/l0/test_vector_adapter_parity.py` +- Modify: `README.md` + +**Interfaces:** +- Produces versioned custom-format dumps and explicit restore into an empty target. + +- [ ] **Step 1: Add adapter parity scenarios** + +```python +@pytest.mark.parametrize("store_fixture", ["direct_store", "http_store"]) +def test_kind_filtered_search_parity(request, store_fixture): + store = request.getfixturevalue(store_fixture) + assert normalize(store.search(...)) == EXPECTED_HITS +``` + +- [ ] **Step 2: Verify parity test exposes any semantic differences** + +Run: `cd harness && .venv/bin/pytest tests/l0/test_vector_adapter_parity.py -q` +Expected: FAIL until result ordering/error mapping is aligned. + +- [ ] **Step 3: Implement backup/restore safety checks** + +```sh +pg_dump --format=custom --schema=vectors --file="$OUTPUT" "$DATABASE_URL" +pg_restore --exit-on-error --clean --if-exists --dbname="$TARGET_DATABASE_URL" "$INPUT" +``` + +Require explicit target and refuse restore when it equals the active source URL. + +- [ ] **Step 4: Run parity, backup/restore, and full harness gates** + +Run: `cd harness && .venv/bin/pytest tests/l0/test_vector_adapter_parity.py tests/l0/test_pgvector_store.py -q` +Run: `./scripts/local-vector-smoke.sh --backup-restore` +Run: `cd harness && .venv/bin/pytest -q` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add scripts/vector-backup.sh scripts/vector-restore.sh harness/tests/l0/test_vector_adapter_parity.py README.md +git commit -m "docs(vector): add local backup restore and parity gate" +``` + diff --git a/docs/superpowers/plans/2026-07-11-portable-deployment-program.md b/docs/superpowers/plans/2026-07-11-portable-deployment-program.md new file mode 100644 index 00000000..27805b0c --- /dev/null +++ b/docs/superpowers/plans/2026-07-11-portable-deployment-program.md @@ -0,0 +1,47 @@ +# Portable Deployment Program + +> **For agentic workers:** execute the linked plans in order. Each plan ends with a compatibility gate and can be released independently. + +**Goal:** Deliver the approved portable ThothII architecture through four independently reviewable implementation plans. + +**Architecture:** Preserve the current frontend → backend → Pi → tht workflow while moving DWH, vector, and Evidence access behind typed adapters. Build two application images and compose optional pgvector and preprocessing services through deployment profiles. + +**Tech Stack:** Python 3.11+, Pydantic 2, SQLAlchemy/PostgreSQL, Fastify/TypeScript, React/Vite, Docker BuildKit, Docker Compose, PostgreSQL+pgvector. + +## Global Constraints + +- Keep `harness/workflow.yaml` and phase semantics unchanged. +- Keep session documents and `review_decisions.jsonl` as the persistence truth. +- Preserve pristine JSON stdout for every `tht --json` command. +- Keep DWH access read-only by credentials and client-side guards. +- Keep vector reader and writer credentials separate. +- Produce exactly two ThothII-owned application images; infrastructure images are optional dependencies. +- Preserve current workspace behavior through an explicit migration window. +- UI strings remain English; workspace content retains its configured language. +- Run harness pytest, gate JS tests, backend vitest+tsc, and frontend vitest+tsc before release. + +--- + +## Ordered plans + +1. [Adapter Foundations](2026-07-11-adapter-foundations.md) + Establishes protocols, capability reporting, factories, and backwards-compatible configuration. + +2. [Container Packaging and Portable Storage](2026-07-11-container-packaging-portable-storage.md) + Builds `thothii-core` and `thothii-frontend`, runtime configuration, logical roots, Compose base, and diagnostics. + +3. [Optional Local pgvector](2026-07-11-local-pgvector-profile.md) + Adds the direct vector adapter, schema migration tooling, persistent service profile, backup/restore, and parity tests. + +4. [Evidence Sources and Preprocessing](2026-07-11-evidence-preprocessing.md) + Adds source adapters, canonical corpus, incremental manifests, atomic publish, and separate document/DWH jobs. + +## Program gates + +- [ ] After Plan 1, current server and workstation workspaces behave identically through the new factories. +- [ ] After Plan 2, the current external-service installation runs from the two images. +- [ ] After Plan 3, profiles B and C run with an optional local pgvector volume. +- [ ] After Plan 4, runtime retrieval no longer requires live access to original Evidence sources. +- [ ] Complete one L2 session for each deployed DWH transport and vector transport pairing in scope. +- [ ] Update `PROJECT_STATE.md` only after each plan's verification evidence is available. +