Merge origin/codex/portable-deployment into feat/docker-local-deploy
Unisce gli internals di Codex (secret-bundle, provider-credentials, auth upstream, security hardening, CI multiarch) mantenendo le fix portal-specific: - backend: configPath da THT_CONFIG (fix sessioni) + dataRoot di Codex; authMode 'upstream' - Docker/compose: TENUTO il mio (verificato live: omics_network+alias, env_file, pi npm-g) perche' il compose/Dockerfile/entrypoint di Codex sono accoppiati al suo modello secret-bundle (tht doctor inesistente, secret-policy.sh). Adottabile in futuro. - config.test.ts: preso Codex (superset) Verificato: tsc clean, 132/132 vitest.
This commit is contained in:
User
@@ -26,7 +26,7 @@
|
||||
- Test: `harness/tests/test_dwh_port_contract.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `DwhCapabilities`, `DwhAdapter`, `DwhHealth`, and `UnsupportedCapability`.
|
||||
- Produces: `DwhCapabilities`, `DwhAdapter`, `DwhHealth`, `DistinctValues`, and `UnsupportedCapability`.
|
||||
- Consumes: existing catalog models from `tht.db.introspect` and execution result types from `tht.db.execute`.
|
||||
|
||||
- [ ] **Step 1: Write the failing protocol-shape test**
|
||||
@@ -54,16 +54,21 @@ class DwhCapabilities:
|
||||
sampling: bool = True
|
||||
distinct_values: bool = True
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DistinctValues:
|
||||
values: list[object]
|
||||
truncated: bool
|
||||
|
||||
@runtime_checkable
|
||||
class DwhAdapter(Protocol):
|
||||
@property
|
||||
def capabilities(self) -> DwhCapabilities: ...
|
||||
def health(self) -> DwhHealth: ...
|
||||
def introspect(self) -> DatabaseCatalog: ...
|
||||
def run_query(self, sql: str, *, limit: int | None = None) -> QueryResult: ...
|
||||
def introspect(self) -> PhysicalSchema: ...
|
||||
def run_query(self, sql: str, *, limit: int) -> ExecResult: ...
|
||||
def explain(self, sql: str) -> PlanSummary: ...
|
||||
def sample_column(self, table: str, column: str, *, limit: int) -> list[object]: ...
|
||||
def distinct_values(self, table: str, column: str) -> list[object]: ...
|
||||
def distinct_values(self, table: str, column: str) -> DistinctValues: ...
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run contract test and type-oriented import smoke test**
|
||||
@@ -85,8 +90,15 @@ git commit -m "refactor(dwh): define adapter contract"
|
||||
- Create: `harness/tht/adapters/dwh/postgres.py`
|
||||
- Create: `harness/tht/adapters/dwh/thoth_rest.py`
|
||||
- Test: `harness/tests/test_dwh_adapters.py`
|
||||
- Test: `harness/tests/test_dwh_port_contract.py`
|
||||
- Test: `harness/tests/l0/test_db_sampling.py`
|
||||
- Modify: `harness/tht/ports/__init__.py`
|
||||
- Modify: `harness/tht/ports/dwh.py`
|
||||
- Modify: `harness/tht/execute/__init__.py`
|
||||
- Modify: `harness/tht/db/execute.py`
|
||||
- Modify: `harness/tht/db/sampling.py`
|
||||
- Modify: `harness/tht/rest/execute.py`
|
||||
- Modify: `docs/superpowers/plans/2026-07-11-adapter-foundations.md`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `DwhAdapter` from Task 1; existing `DatabaseConfig`, `RestConfig`, catalog, sampling, execute, and explain functions.
|
||||
@@ -97,8 +109,8 @@ git commit -m "refactor(dwh): define adapter contract"
|
||||
```python
|
||||
@pytest.mark.parametrize("factory", [postgres_factory, rest_factory])
|
||||
def test_adapter_rejects_write_sql(factory):
|
||||
with pytest.raises(ReadOnlyViolation):
|
||||
factory().run_query("delete from fact_sales")
|
||||
with pytest.raises(ExecutionError):
|
||||
factory().run_query("delete from fact_sales", limit=10)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Verify failure**
|
||||
@@ -111,12 +123,18 @@ Expected: FAIL because the adapter classes are absent.
|
||||
```python
|
||||
class PostgresDwhAdapter:
|
||||
capabilities = DwhCapabilities()
|
||||
def __init__(self, config: DatabaseConfig): self._config = config
|
||||
def run_query(self, sql: str, *, limit: int | None = None) -> QueryResult:
|
||||
return run_query(self._config, sql, limit=limit)
|
||||
def __init__(self, config: DatabaseConfig):
|
||||
self._config = config
|
||||
self._engine = make_engine(config)
|
||||
def run_query(self, sql: str, *, limit: int) -> ExecResult:
|
||||
return run_query(self._engine, sql, limit=limit)
|
||||
```
|
||||
|
||||
Implement the analogous REST wrapper by delegating to `tht.rest.*`; translate transport-specific errors only at the adapter boundary.
|
||||
Implement the analogous REST wrapper by delegating to `tht.rest.*`; translate transport-specific
|
||||
errors only at the adapter boundary. Both wrappers delegate frequency-ranked, distinct sampling to
|
||||
the paired implementations in `tht.db.sampling`. Query and sampling limits must be runtime-positive
|
||||
integers (booleans and floats are rejected), and `distinct_values` reports any cap through
|
||||
`DistinctValues.truncated`.
|
||||
|
||||
- [ ] **Step 4: Run adapter, read-only, sampling, and REST tests**
|
||||
|
||||
@@ -126,7 +144,11 @@ Expected: PASS; L0 may deselect when Docker is unavailable.
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add harness/tht/adapters harness/tht/db/execute.py harness/tht/rest/execute.py harness/tests/test_dwh_adapters.py
|
||||
git add docs/superpowers/plans/2026-07-11-adapter-foundations.md \
|
||||
harness/tht/ports harness/tht/adapters/dwh harness/tht/execute/__init__.py \
|
||||
harness/tht/db/execute.py harness/tht/db/sampling.py harness/tht/rest/execute.py \
|
||||
harness/tests/test_dwh_port_contract.py harness/tests/test_dwh_adapters.py \
|
||||
harness/tests/l0/test_db_sampling.py
|
||||
git commit -m "refactor(dwh): adapt direct and REST transports"
|
||||
```
|
||||
|
||||
@@ -141,7 +163,8 @@ git commit -m "refactor(dwh): adapt direct and REST transports"
|
||||
- Modify: `harness/tht/vectorstore/reader.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `VectorStore`, `VectorCapabilities`, `VectorHealth`, `VectorRecord`, `VectorHit`, `ThothHttpVectorStore`.
|
||||
- Produces: `VectorStore`, `VectorCapabilities`, `VectorHealth`, `VectorRecord`,
|
||||
`VectorWriteRecord`, `VectorHit`, `ThothHttpVectorStore`.
|
||||
- Preserves: current `VectorRestClient`, `DirectSearcher`, and `RestSearcher` behavior behind wrappers.
|
||||
|
||||
- [ ] **Step 1: Write read/write capability and dual-credential tests**
|
||||
@@ -171,9 +194,13 @@ class VectorStore(Protocol):
|
||||
def search(self, collections: list[str], embedding: list[float], *, limit: int,
|
||||
kinds: list[str] | None = None) -> list[VectorHit]: ...
|
||||
def existing_hashes(self, collection: str, kinds: list[str]) -> dict[str, str]: ...
|
||||
def upsert(self, collection: str, records: list[VectorRecord]) -> int: ...
|
||||
def upsert(self, collection: str, records: list[VectorWriteRecord]) -> int: ...
|
||||
```
|
||||
|
||||
`VectorWriteRecord` is the transport-neutral write envelope: it contains the canonical
|
||||
`VectorRecord`, a precomputed embedding, and a content hash. Adapters must preserve
|
||||
`VectorRecord.metadata` unchanged, including semantic keys named `embedding` or `content_hash`.
|
||||
|
||||
- [ ] **Step 4: Run vector regression tests**
|
||||
|
||||
Run: `cd harness && .venv/bin/pytest tests/test_vector_port_contract.py tests/test_vector_dual_key.py tests/test_search_similar_kinds.py tests/test_memory_save_one.py tests/test_solved_question.py -q`
|
||||
@@ -262,6 +289,18 @@ git commit -m "feat(config): add typed resource schema"
|
||||
- Produces: `build_dwh(cfg: Config) -> DwhAdapter` and `build_vector_store(cfg: Config, *, require_write: bool = False) -> VectorStore`.
|
||||
- Consumes: resource configs from Task 4 and wrappers from Tasks 2-3.
|
||||
|
||||
Correction: `DwhAdapter.distinct_values(table, column, *, limit)` requires an explicit
|
||||
positive limit, and direct DWH construction injects `cfg.execution.statement_timeout_ms`.
|
||||
Targeted vector writes consume the factory-returned `VectorStore` and pass
|
||||
`VectorWriteRecord` objects to `upsert`.
|
||||
|
||||
Transitional exception: `build_vector_loader` remains solely for bulk collection sync
|
||||
(`vector init`/rebuild/index flows). It may still construct the legacy table-scoped writer
|
||||
directly until `docs/superpowers/plans/2026-07-11-local-pgvector-profile.md` migrates the
|
||||
local pgvector/vector schema and bulk-sync path. Interactive and targeted writes
|
||||
(`memory save-one` and solved-question indexing) are not covered by this exception and must
|
||||
continue through `build_vector_store(..., require_write=True)` and the public vector port.
|
||||
|
||||
- [ ] **Step 1: Write exact factory selection and missing-writer tests**
|
||||
|
||||
```python
|
||||
|
||||
@@ -236,6 +236,13 @@ git commit -m "build(docker): add runtime-configured frontend image"
|
||||
|
||||
### Task 5: Compose external profile and end-to-end smoke gate
|
||||
|
||||
> **Final-review security amendment (2026-07-12):** the frontend port binds to `127.0.0.1` by
|
||||
> default. Public deployment uses an authenticated upstream proxy with `AUTH_MODE=upstream`;
|
||||
> `THOTH_PUBLIC_EXPOSURE=true` plus `AUTH_MODE=none` is invalid. Local env files are development
|
||||
> only; production uses read-only Compose secrets. Image gates pin exact tags and multi-platform
|
||||
> digests and verify both linux/amd64 and linux/arm64 using the shared container verification
|
||||
> script.
|
||||
|
||||
**Files:**
|
||||
- Create: `compose.yaml`
|
||||
- Create: `deploy/env.example`
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
# Local and Server Docker Deployment Implementation Plan
|
||||
|
||||
> **For Codex:** execute this plan in the current isolated worktree; keep runtime credentials out of Git.
|
||||
|
||||
**Goal:** Configure and verify a Docker Desktop deployment using GLM 5.2 and the existing PSD workspace, while retaining a portable server deployment contract.
|
||||
|
||||
**Architecture:** The base Compose file builds two applications and consumes only generic environment values and a Docker secret bundle. A tracked GLM Pi registry is mounted read-only in the core container. A Git-ignored local override supplies Mac-specific PSD workspace and CA mounts; server operators supply equivalent server runtime values separately.
|
||||
|
||||
**Tech Stack:** Docker Compose v2, Node 22, Python 3.12, Pi RPC, Fastify, nginx.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Add the non-secret GLM Pi registry
|
||||
|
||||
**Files:**
|
||||
- Create: `deploy/pi/models.json`
|
||||
- Modify: `docker/core.Dockerfile`
|
||||
- Modify: `compose.yaml`
|
||||
- Test: Compose configuration and Pi model discovery
|
||||
|
||||
1. Define the `zai/glm-5.2` OpenAI-compatible model registry without a credential.
|
||||
2. Create the Pi user configuration directory in the core image and mount the registry read-only.
|
||||
3. Verify that `get_available_models` returns `zai/glm-5.2` when the bundle supplies the model key.
|
||||
|
||||
### Task 2: Add generic PSD-compatible runtime templates
|
||||
|
||||
**Files:**
|
||||
- Create: `deploy/workspaces/psd.yaml.example`
|
||||
- Create: `deploy/compose.psd-local.yaml.example`
|
||||
- Modify: `deploy/env.example`
|
||||
- Modify: `README.md`
|
||||
|
||||
1. Define a relative `/data/workspaces/psd` workspace configuration with external REST DWH/vector adapters.
|
||||
2. Document required non-secret environment values and the local/server boundary.
|
||||
3. Keep host paths and credential values out of all tracked files.
|
||||
|
||||
### Task 3: Materialize local runtime configuration securely
|
||||
|
||||
**Files (ignored):**
|
||||
- Create: `.env`
|
||||
- Create: `deploy/secrets/thothii.secrets`
|
||||
- Create: `deploy/compose.psd-local.yaml`
|
||||
- Create: `deploy/workspaces/psd.yaml`
|
||||
|
||||
1. Transfer only required values from the existing local configuration without writing them to logs.
|
||||
2. Set `PI_PROVIDER=zai`, `PI_MODEL=glm-5.2`, and the Docker Desktop host gateway for Ollama.
|
||||
3. Bind-mount the PSD workspace and private CA read-only where appropriate; sessions remain writable.
|
||||
4. Enforce restricted modes on the secret bundle.
|
||||
|
||||
### Task 4: Build and verify the Docker deployment
|
||||
|
||||
**Commands:**
|
||||
- `docker compose config --quiet`
|
||||
- `docker compose build`
|
||||
- `docker compose up -d`
|
||||
- health/API/model/session smoke checks
|
||||
|
||||
1. Validate rendered Compose configuration without exposing secrets.
|
||||
2. Build the core and frontend images.
|
||||
3. Verify secret mount, core and frontend health, and model listing.
|
||||
4. Start a PSD session using GLM 5.2 and verify Pi emits a workflow event or gate.
|
||||
5. Capture sanitized diagnostics and stop only disposable test resources; leave the validated local stack running unless it fails.
|
||||
|
||||
### Task 5: Record the deployment result
|
||||
|
||||
**Files:**
|
||||
- Modify: `README.md` or deployment documentation
|
||||
|
||||
1. Record the exact local startup command and server-equivalent configuration steps.
|
||||
2. State verified endpoints, model, and session-start result without secret values.
|
||||
@@ -0,0 +1,156 @@
|
||||
# Simple Docker Configuration Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task with review checkpoints.
|
||||
|
||||
**Goal:** Make a fresh ThothII clone runnable with `docker compose up --build -d`, using one
|
||||
`deploy/secrets/thothii.secrets` bundle while preserving a tested legacy fallback.
|
||||
|
||||
**Architecture:** A strict Python secret-bundle loader becomes the single in-process source of
|
||||
secret values. Compose mounts the one bundle only where needed; the core converts values to
|
||||
provider/database runtime interfaces without logging or placing them in argv. The root `.env`
|
||||
is the default Compose interpolation file and selects the appropriate overlay through
|
||||
`COMPOSE_FILE`/`COMPOSE_PROFILES`; legacy `THT_*_SECRET_FILE` installations remain supported.
|
||||
|
||||
**Tech Stack:** Docker Compose v2, YAML, Python 3.12/Pydantic, Fastify/TypeScript, shell smoke
|
||||
tests, pytest, Vitest.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- The normal command must be exactly `docker compose up --build -d` from `ThothII/`.
|
||||
- The canonical secret bundle is `deploy/secrets/thothii.secrets`, key/value syntax, mode `0600`,
|
||||
ignored by Git and excluded from image build contexts.
|
||||
- Secret values must never appear in Compose config output, logs, argv, settings, health, or
|
||||
committed workspace files.
|
||||
- Existing `THT_*_SECRET_FILE` variables remain a documented compatibility path until removed by
|
||||
a later migration.
|
||||
- External, local-vector, and preprocess overlays must remain independently renderable.
|
||||
- Provider compound credentials remain fail-closed; only supported single-key providers are
|
||||
restored from the bundle.
|
||||
- Every task starts with a failing regression test and ends with focused tests, diff checks, and
|
||||
a small commit.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Add the strict secret-bundle loader and compatibility adapter
|
||||
|
||||
**Files:**
|
||||
- Create: `backend/src/config/secret-bundle.ts`
|
||||
- Modify: `backend/src/config.ts`
|
||||
- Modify: `backend/src/pi/provider-credentials.ts`
|
||||
- Modify: `backend/src/pi/pi-process-manager.ts`
|
||||
- Modify: `backend/src/pi/list-models.ts`
|
||||
- Create: `backend/test/secret-bundle.test.ts`
|
||||
- Modify: `backend/test/provider-credentials.test.ts`
|
||||
- Modify: `backend/test/pi-process-manager.test.ts`
|
||||
- Modify: `backend/test/list-models.test.ts`
|
||||
|
||||
**Interfaces:**
|
||||
- `loadSecretBundle(file: string): ReadonlyMap<string, string>` validates `NAME=VALUE` lines,
|
||||
duplicate/unknown/empty keys, `lstat`/`open(O_NOFOLLOW)`/`fstat` identity, owner and mode.
|
||||
- `secretValue(config, key)` first reads `THT_SECRETS_FILE`, then falls back to the existing
|
||||
`THT_*_SECRET_FILE` variable for compatibility.
|
||||
- The existing provider environment builder consumes a value map, so session and model-listing
|
||||
children share identical scrubbing and canonical-provider mapping.
|
||||
|
||||
- [ ] **Step 1: Write failing tests** for valid bundle parsing, comments/blank lines, duplicate
|
||||
keys, unknown keys, missing file, mode/owner failure, inode replacement, and secret redaction.
|
||||
- [ ] **Step 2: Run** `cd backend && npx vitest run test/secret-bundle.test.ts`; expected failure
|
||||
because the loader does not exist.
|
||||
- [ ] **Step 3: Implement** the loader with bounded line lengths, strict key allowlist, no shell
|
||||
evaluation, sanitized errors, and legacy adapter lookup.
|
||||
- [ ] **Step 4: Add tests** proving session spawn and model listing use the same bundle values and
|
||||
do not inherit bundle path or unselected provider credentials.
|
||||
- [ ] **Step 5: Run** `cd backend && npm run build && npx tsc --noEmit -p . && npx vitest run`;
|
||||
expected all backend tests pass.
|
||||
- [ ] **Step 6: Commit** `git commit -m "feat(config): load one validated secret bundle"`.
|
||||
|
||||
### Task 2: Make the root Compose command the default
|
||||
|
||||
**Files:**
|
||||
- Create: `.env.example`
|
||||
- Modify: `.gitignore`
|
||||
- Modify: `compose.yaml`
|
||||
- Modify: `deploy/compose.production.yaml`
|
||||
- Modify: `deploy/compose.local.yaml`
|
||||
- Modify: `deploy/env.example`
|
||||
- Create: `deploy/secrets/thothii.secrets.example`
|
||||
- Create: `scripts/test-default-compose.sh`
|
||||
- Modify: `scripts/test-container-deployment.sh`
|
||||
|
||||
**Interfaces:**
|
||||
- Root `.env` is Compose's automatic interpolation file; `.env.example` contains relative
|
||||
`THT_SECRETS_FILE=deploy/secrets/thothii.secrets`, default `COMPOSE_FILE=compose.yaml`, and
|
||||
the selected overlay/profile values.
|
||||
- `compose.yaml` starts `core` and `frontend` without requiring a profile; overlays extend it.
|
||||
- Core receives one `/run/secrets/thothii.secrets` mount and `THT_SECRETS_FILE` path.
|
||||
|
||||
- [ ] **Step 1: Write failing static tests** that run `docker compose config --quiet` from a
|
||||
temporary clone with `.env` and assert the default services are `core` and `frontend`, one
|
||||
bundle is declared, and no legacy secret file is required.
|
||||
- [ ] **Step 2: Run** `./scripts/test-default-compose.sh`; expected failure because root defaults
|
||||
still require profiles/separate secret files.
|
||||
- [ ] **Step 3: Implement** `.env.example`, `.gitignore`, Compose defaults and one secret mount.
|
||||
Preserve `deploy/compose.production.yaml` as an optional authenticated production override.
|
||||
- [ ] **Step 4: Run** `docker compose --env-file .env.example config --quiet` and the existing
|
||||
deployment/security scripts; expected no secret values in rendered YAML.
|
||||
- [ ] **Step 5: Commit** `git commit -m "build(compose): make root startup the default"`.
|
||||
|
||||
### Task 3: Convert local-vector and preprocess services to the bundle
|
||||
|
||||
**Files:**
|
||||
- Modify: `deploy/compose.local-vector.yaml`
|
||||
- Modify: `deploy/compose.preprocess-local-vector.yaml`
|
||||
- Modify: `deploy/compose.preprocess.yaml`
|
||||
- Modify: `deploy/workspaces/local-vector.yaml`
|
||||
- Modify: `deploy/workspaces/preprocess-evidence.yaml`
|
||||
- Modify: `deploy/workspaces/preprocess-dwh.yaml`
|
||||
- Modify: `scripts/local-vector-smoke.sh`
|
||||
- Modify: `scripts/preprocess-smoke.sh`
|
||||
- Modify: `scripts/test-preprocess-compose-config.sh`
|
||||
- Modify: `scripts/test-vector-backup-restore-safety.sh`
|
||||
|
||||
**Interfaces:**
|
||||
- Every local-vector/preprocess service reads the same mounted bundle path and selects only the
|
||||
named value through the shared loader/helper.
|
||||
- No service declares four file-backed Compose secrets after this task.
|
||||
|
||||
- [ ] **Step 1: Add failing tests** asserting one bundle mount, no `vector_*_password` secret
|
||||
declarations, and valid local-vector workspace resolution.
|
||||
- [ ] **Step 2: Run** focused Compose config and smoke tests; expected failure with current
|
||||
separate-secret declarations.
|
||||
- [ ] **Step 3: Implement** bundle mounts and helper invocations for bootstrap/migrator/reader/
|
||||
writer operations, keeping passwords out of URLs and shell logs.
|
||||
- [ ] **Step 4: Run** `./scripts/test-preprocess-compose-config.sh`, local-vector smoke and
|
||||
preprocess smoke with a clean generated project; expected all pass.
|
||||
- [ ] **Step 5: Commit** `git commit -m "feat(compose): use one secret bundle for local services"`.
|
||||
|
||||
### Task 4: Finish documentation and end-to-end default verification
|
||||
|
||||
**Files:**
|
||||
- Modify: `README.md`
|
||||
- Modify: `docs/installazione-docker-4-contesti.md`
|
||||
- Modify: `docs/index.md`
|
||||
- Modify: `deploy/secrets/README.md`
|
||||
- Modify: `scripts/docker-smoke.sh`
|
||||
- Modify: `scripts/test-default-compose.sh`
|
||||
|
||||
**Interfaces:**
|
||||
- Installation docs show only `cp .env.example .env`, create/fill one bundle, then
|
||||
`docker compose up --build -d`.
|
||||
- Advanced overlays are shown as optional `.env` presets, not mandatory command-line flags.
|
||||
|
||||
- [ ] **Step 1: Add failing documentation/smoke assertions** for the exact command and default
|
||||
files.
|
||||
- [ ] **Step 2: Implement** concise context-specific instructions and migration notes for old
|
||||
separate secret files.
|
||||
- [ ] **Step 3: Run** all shell syntax/config gates, backend/frontend builds/tests, full harness,
|
||||
default Docker smoke, local-vector smoke, preprocess smoke and `git diff --check`.
|
||||
- [ ] **Step 4: Commit** `git commit -m "docs: document one-command Docker installation"`.
|
||||
|
||||
### Task 5: Whole-plan review and handoff
|
||||
|
||||
- [ ] Review `a6b195b..HEAD` against this plan and confirm no secret leakage, profile regression,
|
||||
or legacy fallback bypass.
|
||||
- [ ] Run the complete verification matrix and report exact counts, skipped L2 tests, and any
|
||||
unavailable Docker/registry prerequisites.
|
||||
- [ ] Keep the branch/worktree intact for the user's integration choice.
|
||||
Reference in New Issue
Block a user