Merge origin/codex/portable-deployment into feat/docker-local-deploy
Unisce gli internals di Codex (secret-bundle, provider-credentials, auth upstream, security hardening, CI multiarch) mantenendo le fix portal-specific: - backend: configPath da THT_CONFIG (fix sessioni) + dataRoot di Codex; authMode 'upstream' - Docker/compose: TENUTO il mio (verificato live: omics_network+alias, env_file, pi npm-g) perche' il compose/Dockerfile/entrypoint di Codex sono accoppiati al suo modello secret-bundle (tht doctor inesistente, secret-policy.sh). Adottabile in futuro. - config.test.ts: preso Codex (superset) Verificato: tsc clean, 132/132 vitest.
This commit is contained in:
User
@@ -26,7 +26,7 @@
|
||||
- Test: `harness/tests/test_dwh_port_contract.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `DwhCapabilities`, `DwhAdapter`, `DwhHealth`, and `UnsupportedCapability`.
|
||||
- Produces: `DwhCapabilities`, `DwhAdapter`, `DwhHealth`, `DistinctValues`, and `UnsupportedCapability`.
|
||||
- Consumes: existing catalog models from `tht.db.introspect` and execution result types from `tht.db.execute`.
|
||||
|
||||
- [ ] **Step 1: Write the failing protocol-shape test**
|
||||
@@ -54,16 +54,21 @@ class DwhCapabilities:
|
||||
sampling: bool = True
|
||||
distinct_values: bool = True
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DistinctValues:
|
||||
values: list[object]
|
||||
truncated: bool
|
||||
|
||||
@runtime_checkable
|
||||
class DwhAdapter(Protocol):
|
||||
@property
|
||||
def capabilities(self) -> DwhCapabilities: ...
|
||||
def health(self) -> DwhHealth: ...
|
||||
def introspect(self) -> DatabaseCatalog: ...
|
||||
def run_query(self, sql: str, *, limit: int | None = None) -> QueryResult: ...
|
||||
def introspect(self) -> PhysicalSchema: ...
|
||||
def run_query(self, sql: str, *, limit: int) -> ExecResult: ...
|
||||
def explain(self, sql: str) -> PlanSummary: ...
|
||||
def sample_column(self, table: str, column: str, *, limit: int) -> list[object]: ...
|
||||
def distinct_values(self, table: str, column: str) -> list[object]: ...
|
||||
def distinct_values(self, table: str, column: str) -> DistinctValues: ...
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run contract test and type-oriented import smoke test**
|
||||
@@ -85,8 +90,15 @@ git commit -m "refactor(dwh): define adapter contract"
|
||||
- Create: `harness/tht/adapters/dwh/postgres.py`
|
||||
- Create: `harness/tht/adapters/dwh/thoth_rest.py`
|
||||
- Test: `harness/tests/test_dwh_adapters.py`
|
||||
- Test: `harness/tests/test_dwh_port_contract.py`
|
||||
- Test: `harness/tests/l0/test_db_sampling.py`
|
||||
- Modify: `harness/tht/ports/__init__.py`
|
||||
- Modify: `harness/tht/ports/dwh.py`
|
||||
- Modify: `harness/tht/execute/__init__.py`
|
||||
- Modify: `harness/tht/db/execute.py`
|
||||
- Modify: `harness/tht/db/sampling.py`
|
||||
- Modify: `harness/tht/rest/execute.py`
|
||||
- Modify: `docs/superpowers/plans/2026-07-11-adapter-foundations.md`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `DwhAdapter` from Task 1; existing `DatabaseConfig`, `RestConfig`, catalog, sampling, execute, and explain functions.
|
||||
@@ -97,8 +109,8 @@ git commit -m "refactor(dwh): define adapter contract"
|
||||
```python
|
||||
@pytest.mark.parametrize("factory", [postgres_factory, rest_factory])
|
||||
def test_adapter_rejects_write_sql(factory):
|
||||
with pytest.raises(ReadOnlyViolation):
|
||||
factory().run_query("delete from fact_sales")
|
||||
with pytest.raises(ExecutionError):
|
||||
factory().run_query("delete from fact_sales", limit=10)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Verify failure**
|
||||
@@ -111,12 +123,18 @@ Expected: FAIL because the adapter classes are absent.
|
||||
```python
|
||||
class PostgresDwhAdapter:
|
||||
capabilities = DwhCapabilities()
|
||||
def __init__(self, config: DatabaseConfig): self._config = config
|
||||
def run_query(self, sql: str, *, limit: int | None = None) -> QueryResult:
|
||||
return run_query(self._config, sql, limit=limit)
|
||||
def __init__(self, config: DatabaseConfig):
|
||||
self._config = config
|
||||
self._engine = make_engine(config)
|
||||
def run_query(self, sql: str, *, limit: int) -> ExecResult:
|
||||
return run_query(self._engine, sql, limit=limit)
|
||||
```
|
||||
|
||||
Implement the analogous REST wrapper by delegating to `tht.rest.*`; translate transport-specific errors only at the adapter boundary.
|
||||
Implement the analogous REST wrapper by delegating to `tht.rest.*`; translate transport-specific
|
||||
errors only at the adapter boundary. Both wrappers delegate frequency-ranked, distinct sampling to
|
||||
the paired implementations in `tht.db.sampling`. Query and sampling limits must be runtime-positive
|
||||
integers (booleans and floats are rejected), and `distinct_values` reports any cap through
|
||||
`DistinctValues.truncated`.
|
||||
|
||||
- [ ] **Step 4: Run adapter, read-only, sampling, and REST tests**
|
||||
|
||||
@@ -126,7 +144,11 @@ Expected: PASS; L0 may deselect when Docker is unavailable.
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add harness/tht/adapters harness/tht/db/execute.py harness/tht/rest/execute.py harness/tests/test_dwh_adapters.py
|
||||
git add docs/superpowers/plans/2026-07-11-adapter-foundations.md \
|
||||
harness/tht/ports harness/tht/adapters/dwh harness/tht/execute/__init__.py \
|
||||
harness/tht/db/execute.py harness/tht/db/sampling.py harness/tht/rest/execute.py \
|
||||
harness/tests/test_dwh_port_contract.py harness/tests/test_dwh_adapters.py \
|
||||
harness/tests/l0/test_db_sampling.py
|
||||
git commit -m "refactor(dwh): adapt direct and REST transports"
|
||||
```
|
||||
|
||||
@@ -141,7 +163,8 @@ git commit -m "refactor(dwh): adapt direct and REST transports"
|
||||
- Modify: `harness/tht/vectorstore/reader.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `VectorStore`, `VectorCapabilities`, `VectorHealth`, `VectorRecord`, `VectorHit`, `ThothHttpVectorStore`.
|
||||
- Produces: `VectorStore`, `VectorCapabilities`, `VectorHealth`, `VectorRecord`,
|
||||
`VectorWriteRecord`, `VectorHit`, `ThothHttpVectorStore`.
|
||||
- Preserves: current `VectorRestClient`, `DirectSearcher`, and `RestSearcher` behavior behind wrappers.
|
||||
|
||||
- [ ] **Step 1: Write read/write capability and dual-credential tests**
|
||||
@@ -171,9 +194,13 @@ class VectorStore(Protocol):
|
||||
def search(self, collections: list[str], embedding: list[float], *, limit: int,
|
||||
kinds: list[str] | None = None) -> list[VectorHit]: ...
|
||||
def existing_hashes(self, collection: str, kinds: list[str]) -> dict[str, str]: ...
|
||||
def upsert(self, collection: str, records: list[VectorRecord]) -> int: ...
|
||||
def upsert(self, collection: str, records: list[VectorWriteRecord]) -> int: ...
|
||||
```
|
||||
|
||||
`VectorWriteRecord` is the transport-neutral write envelope: it contains the canonical
|
||||
`VectorRecord`, a precomputed embedding, and a content hash. Adapters must preserve
|
||||
`VectorRecord.metadata` unchanged, including semantic keys named `embedding` or `content_hash`.
|
||||
|
||||
- [ ] **Step 4: Run vector regression tests**
|
||||
|
||||
Run: `cd harness && .venv/bin/pytest tests/test_vector_port_contract.py tests/test_vector_dual_key.py tests/test_search_similar_kinds.py tests/test_memory_save_one.py tests/test_solved_question.py -q`
|
||||
@@ -262,6 +289,18 @@ git commit -m "feat(config): add typed resource schema"
|
||||
- Produces: `build_dwh(cfg: Config) -> DwhAdapter` and `build_vector_store(cfg: Config, *, require_write: bool = False) -> VectorStore`.
|
||||
- Consumes: resource configs from Task 4 and wrappers from Tasks 2-3.
|
||||
|
||||
Correction: `DwhAdapter.distinct_values(table, column, *, limit)` requires an explicit
|
||||
positive limit, and direct DWH construction injects `cfg.execution.statement_timeout_ms`.
|
||||
Targeted vector writes consume the factory-returned `VectorStore` and pass
|
||||
`VectorWriteRecord` objects to `upsert`.
|
||||
|
||||
Transitional exception: `build_vector_loader` remains solely for bulk collection sync
|
||||
(`vector init`/rebuild/index flows). It may still construct the legacy table-scoped writer
|
||||
directly until `docs/superpowers/plans/2026-07-11-local-pgvector-profile.md` migrates the
|
||||
local pgvector/vector schema and bulk-sync path. Interactive and targeted writes
|
||||
(`memory save-one` and solved-question indexing) are not covered by this exception and must
|
||||
continue through `build_vector_store(..., require_write=True)` and the public vector port.
|
||||
|
||||
- [ ] **Step 1: Write exact factory selection and missing-writer tests**
|
||||
|
||||
```python
|
||||
|
||||
@@ -236,6 +236,13 @@ git commit -m "build(docker): add runtime-configured frontend image"
|
||||
|
||||
### Task 5: Compose external profile and end-to-end smoke gate
|
||||
|
||||
> **Final-review security amendment (2026-07-12):** the frontend port binds to `127.0.0.1` by
|
||||
> default. Public deployment uses an authenticated upstream proxy with `AUTH_MODE=upstream`;
|
||||
> `THOTH_PUBLIC_EXPOSURE=true` plus `AUTH_MODE=none` is invalid. Local env files are development
|
||||
> only; production uses read-only Compose secrets. Image gates pin exact tags and multi-platform
|
||||
> digests and verify both linux/amd64 and linux/arm64 using the shared container verification
|
||||
> script.
|
||||
|
||||
**Files:**
|
||||
- Create: `compose.yaml`
|
||||
- Create: `deploy/env.example`
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
# Local and Server Docker Deployment Implementation Plan
|
||||
|
||||
> **For Codex:** execute this plan in the current isolated worktree; keep runtime credentials out of Git.
|
||||
|
||||
**Goal:** Configure and verify a Docker Desktop deployment using GLM 5.2 and the existing PSD workspace, while retaining a portable server deployment contract.
|
||||
|
||||
**Architecture:** The base Compose file builds two applications and consumes only generic environment values and a Docker secret bundle. A tracked GLM Pi registry is mounted read-only in the core container. A Git-ignored local override supplies Mac-specific PSD workspace and CA mounts; server operators supply equivalent server runtime values separately.
|
||||
|
||||
**Tech Stack:** Docker Compose v2, Node 22, Python 3.12, Pi RPC, Fastify, nginx.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Add the non-secret GLM Pi registry
|
||||
|
||||
**Files:**
|
||||
- Create: `deploy/pi/models.json`
|
||||
- Modify: `docker/core.Dockerfile`
|
||||
- Modify: `compose.yaml`
|
||||
- Test: Compose configuration and Pi model discovery
|
||||
|
||||
1. Define the `zai/glm-5.2` OpenAI-compatible model registry without a credential.
|
||||
2. Create the Pi user configuration directory in the core image and mount the registry read-only.
|
||||
3. Verify that `get_available_models` returns `zai/glm-5.2` when the bundle supplies the model key.
|
||||
|
||||
### Task 2: Add generic PSD-compatible runtime templates
|
||||
|
||||
**Files:**
|
||||
- Create: `deploy/workspaces/psd.yaml.example`
|
||||
- Create: `deploy/compose.psd-local.yaml.example`
|
||||
- Modify: `deploy/env.example`
|
||||
- Modify: `README.md`
|
||||
|
||||
1. Define a relative `/data/workspaces/psd` workspace configuration with external REST DWH/vector adapters.
|
||||
2. Document required non-secret environment values and the local/server boundary.
|
||||
3. Keep host paths and credential values out of all tracked files.
|
||||
|
||||
### Task 3: Materialize local runtime configuration securely
|
||||
|
||||
**Files (ignored):**
|
||||
- Create: `.env`
|
||||
- Create: `deploy/secrets/thothii.secrets`
|
||||
- Create: `deploy/compose.psd-local.yaml`
|
||||
- Create: `deploy/workspaces/psd.yaml`
|
||||
|
||||
1. Transfer only required values from the existing local configuration without writing them to logs.
|
||||
2. Set `PI_PROVIDER=zai`, `PI_MODEL=glm-5.2`, and the Docker Desktop host gateway for Ollama.
|
||||
3. Bind-mount the PSD workspace and private CA read-only where appropriate; sessions remain writable.
|
||||
4. Enforce restricted modes on the secret bundle.
|
||||
|
||||
### Task 4: Build and verify the Docker deployment
|
||||
|
||||
**Commands:**
|
||||
- `docker compose config --quiet`
|
||||
- `docker compose build`
|
||||
- `docker compose up -d`
|
||||
- health/API/model/session smoke checks
|
||||
|
||||
1. Validate rendered Compose configuration without exposing secrets.
|
||||
2. Build the core and frontend images.
|
||||
3. Verify secret mount, core and frontend health, and model listing.
|
||||
4. Start a PSD session using GLM 5.2 and verify Pi emits a workflow event or gate.
|
||||
5. Capture sanitized diagnostics and stop only disposable test resources; leave the validated local stack running unless it fails.
|
||||
|
||||
### Task 5: Record the deployment result
|
||||
|
||||
**Files:**
|
||||
- Modify: `README.md` or deployment documentation
|
||||
|
||||
1. Record the exact local startup command and server-equivalent configuration steps.
|
||||
2. State verified endpoints, model, and session-start result without secret values.
|
||||
@@ -0,0 +1,156 @@
|
||||
# Simple Docker Configuration Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task with review checkpoints.
|
||||
|
||||
**Goal:** Make a fresh ThothII clone runnable with `docker compose up --build -d`, using one
|
||||
`deploy/secrets/thothii.secrets` bundle while preserving a tested legacy fallback.
|
||||
|
||||
**Architecture:** A strict Python secret-bundle loader becomes the single in-process source of
|
||||
secret values. Compose mounts the one bundle only where needed; the core converts values to
|
||||
provider/database runtime interfaces without logging or placing them in argv. The root `.env`
|
||||
is the default Compose interpolation file and selects the appropriate overlay through
|
||||
`COMPOSE_FILE`/`COMPOSE_PROFILES`; legacy `THT_*_SECRET_FILE` installations remain supported.
|
||||
|
||||
**Tech Stack:** Docker Compose v2, YAML, Python 3.12/Pydantic, Fastify/TypeScript, shell smoke
|
||||
tests, pytest, Vitest.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- The normal command must be exactly `docker compose up --build -d` from `ThothII/`.
|
||||
- The canonical secret bundle is `deploy/secrets/thothii.secrets`, key/value syntax, mode `0600`,
|
||||
ignored by Git and excluded from image build contexts.
|
||||
- Secret values must never appear in Compose config output, logs, argv, settings, health, or
|
||||
committed workspace files.
|
||||
- Existing `THT_*_SECRET_FILE` variables remain a documented compatibility path until removed by
|
||||
a later migration.
|
||||
- External, local-vector, and preprocess overlays must remain independently renderable.
|
||||
- Provider compound credentials remain fail-closed; only supported single-key providers are
|
||||
restored from the bundle.
|
||||
- Every task starts with a failing regression test and ends with focused tests, diff checks, and
|
||||
a small commit.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Add the strict secret-bundle loader and compatibility adapter
|
||||
|
||||
**Files:**
|
||||
- Create: `backend/src/config/secret-bundle.ts`
|
||||
- Modify: `backend/src/config.ts`
|
||||
- Modify: `backend/src/pi/provider-credentials.ts`
|
||||
- Modify: `backend/src/pi/pi-process-manager.ts`
|
||||
- Modify: `backend/src/pi/list-models.ts`
|
||||
- Create: `backend/test/secret-bundle.test.ts`
|
||||
- Modify: `backend/test/provider-credentials.test.ts`
|
||||
- Modify: `backend/test/pi-process-manager.test.ts`
|
||||
- Modify: `backend/test/list-models.test.ts`
|
||||
|
||||
**Interfaces:**
|
||||
- `loadSecretBundle(file: string): ReadonlyMap<string, string>` validates `NAME=VALUE` lines,
|
||||
duplicate/unknown/empty keys, `lstat`/`open(O_NOFOLLOW)`/`fstat` identity, owner and mode.
|
||||
- `secretValue(config, key)` first reads `THT_SECRETS_FILE`, then falls back to the existing
|
||||
`THT_*_SECRET_FILE` variable for compatibility.
|
||||
- The existing provider environment builder consumes a value map, so session and model-listing
|
||||
children share identical scrubbing and canonical-provider mapping.
|
||||
|
||||
- [ ] **Step 1: Write failing tests** for valid bundle parsing, comments/blank lines, duplicate
|
||||
keys, unknown keys, missing file, mode/owner failure, inode replacement, and secret redaction.
|
||||
- [ ] **Step 2: Run** `cd backend && npx vitest run test/secret-bundle.test.ts`; expected failure
|
||||
because the loader does not exist.
|
||||
- [ ] **Step 3: Implement** the loader with bounded line lengths, strict key allowlist, no shell
|
||||
evaluation, sanitized errors, and legacy adapter lookup.
|
||||
- [ ] **Step 4: Add tests** proving session spawn and model listing use the same bundle values and
|
||||
do not inherit bundle path or unselected provider credentials.
|
||||
- [ ] **Step 5: Run** `cd backend && npm run build && npx tsc --noEmit -p . && npx vitest run`;
|
||||
expected all backend tests pass.
|
||||
- [ ] **Step 6: Commit** `git commit -m "feat(config): load one validated secret bundle"`.
|
||||
|
||||
### Task 2: Make the root Compose command the default
|
||||
|
||||
**Files:**
|
||||
- Create: `.env.example`
|
||||
- Modify: `.gitignore`
|
||||
- Modify: `compose.yaml`
|
||||
- Modify: `deploy/compose.production.yaml`
|
||||
- Modify: `deploy/compose.local.yaml`
|
||||
- Modify: `deploy/env.example`
|
||||
- Create: `deploy/secrets/thothii.secrets.example`
|
||||
- Create: `scripts/test-default-compose.sh`
|
||||
- Modify: `scripts/test-container-deployment.sh`
|
||||
|
||||
**Interfaces:**
|
||||
- Root `.env` is Compose's automatic interpolation file; `.env.example` contains relative
|
||||
`THT_SECRETS_FILE=deploy/secrets/thothii.secrets`, default `COMPOSE_FILE=compose.yaml`, and
|
||||
the selected overlay/profile values.
|
||||
- `compose.yaml` starts `core` and `frontend` without requiring a profile; overlays extend it.
|
||||
- Core receives one `/run/secrets/thothii.secrets` mount and `THT_SECRETS_FILE` path.
|
||||
|
||||
- [ ] **Step 1: Write failing static tests** that run `docker compose config --quiet` from a
|
||||
temporary clone with `.env` and assert the default services are `core` and `frontend`, one
|
||||
bundle is declared, and no legacy secret file is required.
|
||||
- [ ] **Step 2: Run** `./scripts/test-default-compose.sh`; expected failure because root defaults
|
||||
still require profiles/separate secret files.
|
||||
- [ ] **Step 3: Implement** `.env.example`, `.gitignore`, Compose defaults and one secret mount.
|
||||
Preserve `deploy/compose.production.yaml` as an optional authenticated production override.
|
||||
- [ ] **Step 4: Run** `docker compose --env-file .env.example config --quiet` and the existing
|
||||
deployment/security scripts; expected no secret values in rendered YAML.
|
||||
- [ ] **Step 5: Commit** `git commit -m "build(compose): make root startup the default"`.
|
||||
|
||||
### Task 3: Convert local-vector and preprocess services to the bundle
|
||||
|
||||
**Files:**
|
||||
- Modify: `deploy/compose.local-vector.yaml`
|
||||
- Modify: `deploy/compose.preprocess-local-vector.yaml`
|
||||
- Modify: `deploy/compose.preprocess.yaml`
|
||||
- Modify: `deploy/workspaces/local-vector.yaml`
|
||||
- Modify: `deploy/workspaces/preprocess-evidence.yaml`
|
||||
- Modify: `deploy/workspaces/preprocess-dwh.yaml`
|
||||
- Modify: `scripts/local-vector-smoke.sh`
|
||||
- Modify: `scripts/preprocess-smoke.sh`
|
||||
- Modify: `scripts/test-preprocess-compose-config.sh`
|
||||
- Modify: `scripts/test-vector-backup-restore-safety.sh`
|
||||
|
||||
**Interfaces:**
|
||||
- Every local-vector/preprocess service reads the same mounted bundle path and selects only the
|
||||
named value through the shared loader/helper.
|
||||
- No service declares four file-backed Compose secrets after this task.
|
||||
|
||||
- [ ] **Step 1: Add failing tests** asserting one bundle mount, no `vector_*_password` secret
|
||||
declarations, and valid local-vector workspace resolution.
|
||||
- [ ] **Step 2: Run** focused Compose config and smoke tests; expected failure with current
|
||||
separate-secret declarations.
|
||||
- [ ] **Step 3: Implement** bundle mounts and helper invocations for bootstrap/migrator/reader/
|
||||
writer operations, keeping passwords out of URLs and shell logs.
|
||||
- [ ] **Step 4: Run** `./scripts/test-preprocess-compose-config.sh`, local-vector smoke and
|
||||
preprocess smoke with a clean generated project; expected all pass.
|
||||
- [ ] **Step 5: Commit** `git commit -m "feat(compose): use one secret bundle for local services"`.
|
||||
|
||||
### Task 4: Finish documentation and end-to-end default verification
|
||||
|
||||
**Files:**
|
||||
- Modify: `README.md`
|
||||
- Modify: `docs/installazione-docker-4-contesti.md`
|
||||
- Modify: `docs/index.md`
|
||||
- Modify: `deploy/secrets/README.md`
|
||||
- Modify: `scripts/docker-smoke.sh`
|
||||
- Modify: `scripts/test-default-compose.sh`
|
||||
|
||||
**Interfaces:**
|
||||
- Installation docs show only `cp .env.example .env`, create/fill one bundle, then
|
||||
`docker compose up --build -d`.
|
||||
- Advanced overlays are shown as optional `.env` presets, not mandatory command-line flags.
|
||||
|
||||
- [ ] **Step 1: Add failing documentation/smoke assertions** for the exact command and default
|
||||
files.
|
||||
- [ ] **Step 2: Implement** concise context-specific instructions and migration notes for old
|
||||
separate secret files.
|
||||
- [ ] **Step 3: Run** all shell syntax/config gates, backend/frontend builds/tests, full harness,
|
||||
default Docker smoke, local-vector smoke, preprocess smoke and `git diff --check`.
|
||||
- [ ] **Step 4: Commit** `git commit -m "docs: document one-command Docker installation"`.
|
||||
|
||||
### Task 5: Whole-plan review and handoff
|
||||
|
||||
- [ ] Review `a6b195b..HEAD` against this plan and confirm no secret leakage, profile regression,
|
||||
or legacy fallback bypass.
|
||||
- [ ] Run the complete verification matrix and report exact counts, skipped L2 tests, and any
|
||||
unavailable Docker/registry prerequisites.
|
||||
- [ ] Keep the branch/worktree intact for the user's integration choice.
|
||||
@@ -36,6 +36,7 @@ Non sono presenti Dockerfile o file Compose. Il processo di sviluppo assume Pi e
|
||||
6. **Configurazione dichiarativa e validata.** I workspace contengono riferimenti logici e configurazioni non segrete; i segreti sono in environment variables o secret store.
|
||||
7. **Capability esplicite.** Un adapter dichiara ciò che supporta. Le funzioni mancanti producono degradazione o blocco comprensibile, non emulazioni implicite.
|
||||
8. **Read-only by construction sul DWH.** Credenziali, API e guard client-side mantengono la separazione dall'autorità di scrittura.
|
||||
9. **Esposizione sicura per default.** La porta applicativa pubblicata è vincolata a loopback. Un'esposizione pubblica richiede un reverse proxy autenticante esterno e `AUTH_MODE=upstream`; la combinazione pubblico + `none` viene rifiutata all'avvio. OIDC interno non fa parte di questa fase.
|
||||
|
||||
## 4. Packaging e runtime
|
||||
|
||||
@@ -143,6 +144,11 @@ La configurazione si divide in:
|
||||
- **workspace:** lingua, adapter, namespace, collezioni e policy di preprocessing;
|
||||
- **segreti:** password, token, certificati e chiavi reader/writer.
|
||||
|
||||
Nel profilo locale i segreti possono provenire da un env-file non versionato. In produzione sono
|
||||
file read-only sotto `/run/secrets`, leggibili dall'UID 10001. Il reverse proxy autenticante è un
|
||||
confine fidato: rimuove header identità forniti dal client e inserisce
|
||||
`X-Authenticated-User` soltanto dopo autenticazione.
|
||||
|
||||
Deve esistere un comando di diagnostica che produca sia output umano sia JSON pristino, rispettando il contratto CLI corrente.
|
||||
|
||||
## 7. Pipeline di preprocessing
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
# Local and Server Docker Deployment Design
|
||||
|
||||
## Goal
|
||||
|
||||
Run ThothII locally in Docker against the existing PSD services, while keeping the
|
||||
same tracked Docker package deployable on a server hosting the DWH and pgvector.
|
||||
|
||||
## Decisions
|
||||
|
||||
- `compose.yaml` remains portable and contains no customer paths, credentials, or
|
||||
private certificate material.
|
||||
- The GLM registry is a tracked, non-secret Pi configuration. The provider key is
|
||||
supplied only through the existing Docker secret bundle.
|
||||
- A local-only Compose override binds the existing PSD workspace and CA material
|
||||
from the developer machine. It is ignored by Git and exists solely to validate
|
||||
Docker Desktop against the real workflow.
|
||||
- A server profile consumes its own workspace mount, secret bundle, and service
|
||||
endpoints. It never relies on macOS paths or a developer's `~/.pi` directory.
|
||||
- The verification scope is a live session start through Pi using `zai/glm-5.2`;
|
||||
it stops at the first human-review gate and does not finalize a datamart.
|
||||
|
||||
## Configuration Boundaries
|
||||
|
||||
Tracked files define images, Compose service contracts, templates, validation, and
|
||||
documentation. Ignored runtime files hold the selected endpoint values, secret
|
||||
bundle, certificate mount source, and local workspace path. The server receives
|
||||
only the tracked package; its operator materializes equivalent runtime files with
|
||||
server-specific values.
|
||||
|
||||
## Validation
|
||||
|
||||
The local run must validate Compose syntax, image builds, secret mount readability,
|
||||
core/frontend health endpoints, model discovery, session creation, and receipt of a
|
||||
Pi workflow event. Failure diagnostics must omit secret values.
|
||||
@@ -0,0 +1,98 @@
|
||||
# Design: configurazione Docker semplificata
|
||||
|
||||
## Obiettivo
|
||||
|
||||
Ridurre l'installazione a un file di configurazione `.env` interno al clone e a un solo file
|
||||
contenente tutti i secret, mantenendo il comando operativo standard:
|
||||
|
||||
```sh
|
||||
docker compose up --build -d
|
||||
```
|
||||
|
||||
Il comportamento di default deve essere determinato dai file presenti nella directory radice
|
||||
`ThothII/`, senza obbligare l'operatore a ricordare `-f`, `--env-file` o profili Compose.
|
||||
|
||||
## Struttura installativa
|
||||
|
||||
```text
|
||||
ThothII/
|
||||
├── .env # configurazione non segreta e default Compose
|
||||
├── .env.example # template versionato
|
||||
├── compose.yaml # file Compose principale, usabile senza -f
|
||||
├── deploy/
|
||||
│ ├── secrets/thothii.secrets # unico file secret, escluso da Git
|
||||
│ └── workspaces/ # workspace YAML versionati
|
||||
└── data/ # dati persistenti solo se bind mount esplicito
|
||||
```
|
||||
|
||||
`.env` contiene host, endpoint, profilo scelto, `COMPOSE_FILE`, `COMPOSE_PROFILES` e il percorso
|
||||
del bundle secret. Non contiene valori secret. Il file viene creato copiando `.env.example` e
|
||||
rimane nella directory `ThothII/`.
|
||||
|
||||
Il bundle `deploy/secrets/thothii.secrets` usa righe `NOME=VALORE`, con nomi documentati e
|
||||
validazione rigorosa. Non sono ammesse espansioni shell, comandi, URL con credenziali o righe
|
||||
duplicate. Il file deve essere `0600` sull'host e viene montato read-only nei soli servizi che
|
||||
ne hanno bisogno.
|
||||
|
||||
## Default Compose
|
||||
|
||||
`compose.yaml` diventa il file principale per il profilo applicativo esterno: `core` e
|
||||
`frontend` non sono nascosti dietro un profilo obbligatorio. Il `.env` seleziona eventuali
|
||||
overlay tramite la variabile Compose standard `COMPOSE_FILE` e il profilo tramite
|
||||
`COMPOSE_PROFILES`.
|
||||
|
||||
Esempi:
|
||||
|
||||
- server con DWH/vector esterni: `COMPOSE_FILE=compose.yaml:deploy/compose.production.yaml`;
|
||||
- Mac/Windows con pgvector locale: `COMPOSE_FILE=compose.yaml:deploy/compose.local-vector.yaml` e
|
||||
`COMPOSE_PROFILES=local-vector`;
|
||||
- preprocessing locale: aggiunta dell'overlay preprocess nel valore `COMPOSE_FILE`.
|
||||
|
||||
Quando `.env` è configurato, il comando non cambia tra i contesti:
|
||||
|
||||
```sh
|
||||
docker compose up --build -d
|
||||
```
|
||||
|
||||
I comandi con `-f` e `--env-file` restano documentati solo come override diagnostico, non come
|
||||
percorso normale di installazione.
|
||||
|
||||
## Bundle secret e runtime
|
||||
|
||||
Il core riceve `THT_SECRETS_FILE=/run/secrets/thothii.secrets`. Un loader comune:
|
||||
|
||||
1. apre il bundle con `O_NOFOLLOW`, verifica owner, permessi e inode;
|
||||
2. rifiuta chiavi sconosciute, duplicate, vuote o provider composti non supportati;
|
||||
3. espone i singoli valori solo in memoria al componente che ne ha bisogno;
|
||||
4. non stampa il bundle, non lo inserisce in `settings.json`, argv, health o log.
|
||||
|
||||
Per pgvector locale, il servizio di inizializzazione e le migrazioni usano lo stesso loader; non
|
||||
si creano più file `bootstrap`, `reader`, `writer` e `migrator`. Le password non vengono passate
|
||||
come argomenti URL. I workspace ricevono riferimenti logici al secret bundle, mai valori.
|
||||
|
||||
La compatibilità temporanea con le variabili `THT_*_SECRET_FILE` viene mantenuta come fallback
|
||||
esplicito per installazioni già esistenti, ma il template e la documentazione nuovi usano solo
|
||||
`THT_SECRETS_FILE`.
|
||||
|
||||
## Compatibilità e sicurezza
|
||||
|
||||
- `docker compose config --quiet` deve funzionare dalla radice senza opzioni aggiuntive;
|
||||
- il default non deve avviare pgvector locale se il `.env` seleziona servizi esterni;
|
||||
- i profili local-vector e preprocess devono aggiungere solo i servizi necessari;
|
||||
- il bundle secret deve essere escluso da `.gitignore` e dai build context Docker;
|
||||
- errori di secret mancanti o non validi devono terminare prima dell'avvio applicativo, con messaggi
|
||||
sanitizzati;
|
||||
- i test devono coprire sia il percorso standard `docker compose up --build -d` sia gli override
|
||||
legacy con file secret separati.
|
||||
|
||||
## Verifica prevista
|
||||
|
||||
La verifica finale comprende:
|
||||
|
||||
1. rendering Compose del default e dei quattro preset `.env.example`;
|
||||
2. test unitari del parser bundle e della compatibilità legacy;
|
||||
3. build delle immagini core/frontend;
|
||||
4. smoke health/SSE/persistenza;
|
||||
5. smoke local-vector con un solo bundle e migrazioni;
|
||||
6. smoke preprocess con il default selezionato dal `.env`;
|
||||
7. controllo che nessun secret compaia in `docker compose config`, log, argv o immagini.
|
||||
Reference in New Issue
Block a user