feat: unify installation model catalog

This commit is contained in:
Codex
2026-09-02 18:45:33 +02:00
parent ae053961a3
commit 7b7927bfe5
169 changed files with 3696 additions and 4572 deletions
+7 -6
View File
@@ -83,7 +83,8 @@ frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness →
`SseHub` fans them out over SSE to the browser. The separate PostgreSQL catalog stores database `SseHub` fans them out over SSE to the browser. The separate PostgreSQL catalog stores database
metadata and sequential AI description-generation runs. Description generation samples the DWH metadata and sequential AI description-generation runs. Description generation samples the DWH
through read-only connectors and calls a short-lived Python LiteLLM helper; it does not use Pi or through read-only connectors and calls a short-lived Python LiteLLM helper; it does not use Pi or
expose a public CLI command. App settings still live in `backend/data/settings.json`. expose a public CLI command. Sessions, metadata generation, and embedding resolve models from the
generated Installation Model Catalog; `thothii-installation.yaml` is its only authored source.
- **Human-in-the-loop gate contract.** The model proposes; a human reviewer decides at gates - **Human-in-the-loop gate contract.** The model proposes; a human reviewer decides at gates
via widgets (`reviewer_select` = single pick — a chosen option carrying a `decision` payload via widgets (`reviewer_select` = single pick — a chosen option carrying a `decision` payload
@@ -99,11 +100,11 @@ frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness →
- **`--json` output must be pristine** (only valid JSON on stdout) — used as a machine contract. - **`--json` output must be pristine** (only valid JSON on stdout) — used as a machine contract.
- **UI strings are English; document *content* stays the workspace language** (Italian for - **UI strings are English; document *content* stays the workspace language** (Italian for
`psd`) because it's the real data. Only chrome/labels are English. `psd`) because it's the real data. Only chrome/labels are English.
- **Workspaces** (`harness/workspaces/*.yaml`) set the DB target and **absolute** - **Workspace schema v4** defines database and Evidence concerns only. Embedding/model facts come
`paths.sessions/artifacts/indexes` — for `psd` these point at a *separate, uncommitted* repo from the installation catalog. The legacy `harness/workspaces/*.yaml` runtime snapshots still use
(`tht-workspace-psd/`). Secrets live ONLY in `harness/.env` (gitignored). absolute session/artifact/index paths; secrets stay in `harness/.env` (gitignored).
- **Settings are global** (`backend/data/settings.json`: workspace/provider/model/thinking); - **Settings are global** (`backend/data/settings.json`: workspace/thinking). Provider/model choices
the New-session form is question-only. are ephemeral canonical catalog selections pinned into the session manifest.
- **Resume**: a resumable session re-enters at its last incomplete phase. The backend refuses - **Resume**: a resumable session re-enters at its last incomplete phase. The backend refuses
resume with 409 when `finalized` or `archived`, and `PiProcessManager.spawnFor` must send resume with 409 when `finalized` or `archived`, and `PiProcessManager.spawnFor` must send
`/riprendi-sessione <id>` (resume mode) vs `/nuova-domanda` (new) — sending the wrong prompt `/riprendi-sessione <id>` (resume mode) vs `/nuova-domanda` (new) — sending the wrong prompt
+22 -3
View File
@@ -1,6 +1,6 @@
# ThothII — Project State # ThothII — Project State
Last updated: 2026-08-31. Last updated: 2026-09-02.
This file is the short operational snapshot. Stable commands and the architecture mental model This file is the short operational snapshot. Stable commands and the architecture mental model
live in `AGENTS.md`; current design and runtime contracts live under `docs/architecture/`, live in `AGENTS.md`; current design and runtime contracts live under `docs/architecture/`,
@@ -59,10 +59,28 @@ tht --installation /absolute/path/thothii-installation.yaml workspace preprocess
These commands use the profile-gated `workspace-maintenance` service. The former standalone These commands use the profile-gated `workspace-maintenance` service. The former standalone
preprocessing Compose fixtures are retired. preprocessing Compose fixtures are retired.
Workspace descriptors use schema v3. For PSD, workspace content and runtime roots point to the Workspace descriptors use schema v4 and contain only database, Evidence, diagnostics, and binding
concerns; model, provider, embedding, and vector-store configuration is installation-owned. For
PSD, workspace content and runtime roots point to the
separate uncommitted repository `/Users/mp/projects/tht-workspace-psd`. Secrets remain outside separate uncommitted repository `/Users/mp/projects/tht-workspace-psd`. Secrets remain outside
Git and are supplied only through installation-local protected files. Git and are supplied only through installation-local protected files.
## Installation Model Catalog
`thothii-installation.yaml` schema version 2 is the only operator-authored source for session,
metadata-generation, and embedding models. The host `tht` lifecycle validates `modelCatalog` and
regenerates the backend catalog, Pi `models.json`/`settings.json`, and Compose override under the
installation-local `generated/` directory. Those projections are replaceable runtime adapters:
they are not edited, backed up, or treated as configuration.
Session and metadata defaults use canonical `provider/model` IDs. Provider authentication declares
one explicit mode (`secret_env`, `pi_auth`, or `none`); `secret_env` names a protected bundle key.
The backend settings store now owns only the selected workspace and thinking level. Existing v1
installations use the explicit catalog migration command; schema-v3 workspace descriptors are
converted deterministically in their curator-owned repository before commit. Strict runtime loading
does not silently infer or merge legacy sources. ADR 0013 and
`docs/plans/2026-09-02-installation-model-catalog.md` record the decision and implementation.
## Database management ## Database management
The database, table, and authoritative physical-schema catalog slices are implemented. Database The database, table, and authoritative physical-schema catalog slices are implemented. Database
@@ -164,7 +182,8 @@ available only when no local start, worker, or helper is live. Runs remain inspe
live SSE log with ordered polling fallback; there is no automatic resume or user-facing generation live SSE log with ordered polling fallback; there is no automatic resume or user-facing generation
CLI. ADRs 0009–0010 record the runtime and source-sampling decisions. CLI. ADRs 0009–0010 record the runtime and source-sampling decisions.
Metadata-generation setup accepts the protected `DEEPSEEK_API_KEY` and `ZAI_API_KEY` references. The Installation Model Catalog accepts the protected `DEEPSEEK_API_KEY` and `ZAI_API_KEY`
references for metadata-generation providers.
It also accepts a model with no secret reference only when its OpenAI-compatible endpoint is It also accepts a model with no secret reference only when its OpenAI-compatible endpoint is
explicit; this covers the VPN-only AritmoLab Qwen 3.6 server without creating a fake operator explicit; this covers the VPN-only AritmoLab Qwen 3.6 server without creating a fake operator
credential. The Python client supplies only its fixed non-secret compatibility placeholder. credential. The Python client supplies only its fixed non-secret compatibility placeholder.
+24 -34
View File
@@ -106,15 +106,18 @@ a remote user's partial list. The isolated deployment exercise is
`./scripts/verify-workspace-install-docs.sh --profile local` or `--profile server`. `./scripts/verify-workspace-install-docs.sh --profile local` or `--profile server`.
<!-- workspace-descriptor-contract:start --> <!-- workspace-descriptor-contract:start -->
Schema v3 is the only accepted workspace descriptor. Schema v1 and v2 workspace descriptors are Schema v4 is the only accepted workspace descriptor. Schema v1, v2, and v3 workspace descriptors
rejected before activation. Candidate snapshot validation therefore makes activation or a pull fail are rejected before activation. Candidate snapshot validation therefore makes activation or a pull
atomically while the prior valid snapshot remains active. There is no in-product migrator or fail atomically while the prior valid snapshot remains active. One workspace owns one Qdrant collection;
automatic conversion. A repository must already contain reviewed v3 descriptors. One workspace
owns one Qdrant collection;
schema, Evidence, and Memory records share that collection and stay separated by indexed payload schema, Evidence, and Memory records share that collection and stay separated by indexed payload
`kind`. `kind`.
<!-- workspace-descriptor-contract:end --> <!-- workspace-descriptor-contract:end -->
<!-- non-workspace-migration:start -->
Convert a v3 descriptor before publication by setting `workspace.schema_version` to `4` and
removing `llm_policy` and `semantic_index`; no database or Evidence field changes.
<!-- non-workspace-migration:end -->
For NL→SQL runtime sessions, connector `ssh_tunnel` bindings remain diagnostic-only: their bounded For NL→SQL runtime sessions, connector `ssh_tunnel` bindings remain diagnostic-only: their bounded
probe cleans up the loopback forward and returns `workspace_not_activatable`; session creation is probe cleans up the loopback forward and returns `workspace_not_activatable`; session creation is
rejected before persistence. Database management is a separate boundary and supports a strict rejected before persistence. Database management is a separate boundary and supports a strict
@@ -176,10 +179,10 @@ secret files, upstream-auth checks, and a fail-closed `503` assertion for its de
unavailable disposable session endpoint. No real provider, database credential, or repository unavailable disposable session endpoint. No real provider, database credential, or repository
secret is required. secret is required.
For a clean server bind, `scripts/prepare-server-pi-state.sh` creates the hidden regular For a clean server bind, `scripts/prepare-server-pi-state.sh` creates the hidden regular Pi agent
`agent/auth.json`, `agent/models.json`, and `agent/settings.json` mount targets atomically before mount targets atomically before Compose. The auth target receives the protected credential bind;
Compose. The server smoke starts from an empty Pi-state root and applies this same preflight; the the model and settings targets receive generated read-only projections. The server smoke starts
real protected/tracked sources remain separate read-only mounts. Deterministic fixture tests render from an empty Pi-state root and applies this same preflight. Deterministic fixture tests render
both profiles, verify that bindings stay on `core`, check mount readability, and run the production both profiles, verify that bindings stay on `core`, check mount readability, and run the production
workspace resolver. Wrong-service, wrong-value, and broken-secret-mount mutations must fail. workspace resolver. Wrong-service, wrong-value, and broken-secret-mount mutations must fail.
@@ -189,7 +192,7 @@ an independent 32-minute outer timeout and does not retry a failed command.
Current release status (2026-08-05): clean-root render/setup and the production runtime-binding Current release status (2026-08-05): clean-root render/setup and the production runtime-binding
resolver contracts are green. The server fixture supplies all four private trusted claims, resolver contracts are green. The server fixture supplies all four private trusted claims,
including exact non-admin value `0`, and a focused test proves nginx normalization produces the including exact non-admin value `0`, and a focused test proves nginx normalization produces the
accepted non-admin backend principal. Canonical schema-v3 registry descriptors now pass through accepted non-admin backend principal. Canonical schema-v4 registry descriptors now pass through
one backend-owned, secret-safe runtime handoff for inventory and session execution; canonical one backend-owned, secret-safe runtime handoff for inventory and session execution; canonical
identity and durable session/artifact/index roots are retained. The fresh update-only smoke passed identity and durable session/artifact/index roots are retained. The fresh update-only smoke passed
bad-candidate mutation, automatic `rolled_back` compensation, exact prior-image restoration, bad-candidate mutation, automatic `rolled_back` compensation, exact prior-image restoration,
@@ -295,14 +298,12 @@ Copy `deploy/secrets/thothii.secrets.example` to a protected host file, include
keys, and set its absolute path as `THT_SECRETS_FILE` in the operator env. Keep Pi's native keys, and set its absolute path as `THT_SECRETS_FILE` in the operator env. Keep Pi's native
provider auth in the separate protected file named by `PI_AUTH_FILE`. provider auth in the separate protected file named by `PI_AUTH_FILE`.
Description Generation is configured independently in the protected installation descriptor under Interactive sessions, Description Generation, and embedding share the protected installation
`metadataGeneration`. Set `THT_INSTALLATION_CONFIG_SOURCE` to that exact host file; Compose mounts descriptor's `modelCatalog`. Set `THT_INSTALLATION_CONFIG_SOURCE` to that exact host file; `tht`
it read-only into `core` and supplies the fixed runtime `THT_INSTALLATION_CONFIG_FILE` path. Each validates it and generates the runtime catalog, Pi adapters, and Compose override before startup.
keyed model stores only an audited `apiKeyEnv` reference. The referenced value stays in the secret Each authenticated provider stores only an audited `apiKeyEnv` reference; the referenced value stays
bundle; a model may omit `apiKeyEnv` only when it declares an explicit endpoint that accepts in the secret bundle. A provider may use `authentication.mode: none` only with an explicit keyless
unauthenticated requests. The browser receives only model IDs, labels, and the configured default. endpoint. The browser receives only eligible model IDs, labels, and the catalog default.
Configuration changes take effect after restart and do not use Pi settings or workspace
`llm_policy`.
Before enabling Description Generation, approve the selected model provider for bounded source-data Before enabling Description Generation, approve the selected model provider for bounded source-data
disclosure. Every catalog column has a **Sensitive** flag that defaults to `false`. Administrators can disclosure. Every catalog column has a **Sensitive** flag that defaults to `false`. Administrators can
@@ -333,22 +334,11 @@ the host/secret-manager materialization and add a reviewed Compose override that
does not create that mount. The frontend remains on loopback; the authenticated host proxy is the does not create that mount. The frontend remains on loopback; the authenticated host proxy is the
only public listener. only public listener.
Set the selected model provider in application settings (or `PI_PROVIDER`). For each Pi spawn the For each Pi spawn, the backend resolves the selected canonical provider/model in the runtime catalog,
backend validates and reads `THT_MODEL_API_KEY` from the bundle, then exposes its value only as the provider's reads exactly that provider's declared `apiKeyEnv` value from the bundle, and exposes only that key
recognized child variable (for example `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GEMINI_API_KEY`, or to the child. Ambient provider credentials and secret-bundle paths are scrubbed. Providers needing a
`ZAI_API_KEY`). Neither the generic file path nor deprecated `PI_PROVIDER_API_KEY` is inherited by compound credential bundle remain unsupported until the catalog gains an explicit generic contract
Pi. Local providers such as Ollama require no model key. for them.
`THT_MODEL_API_KEY` supports Pi providers whose authentication is exactly one key:
`ant-ling`, `anthropic`, `cerebras`, `deepseek`, `fireworks`, `github-copilot`, `google`
(including the `gemini` alias), `google-vertex` when using its API-key mode, `groq`,
`huggingface`, `kimi-coding`, `minimax`, `minimax-cn`, `mistral`, `moonshotai`,
`moonshotai-cn`, `nvidia`, `openai`, `opencode`, `opencode-go`, `openrouter`, `together`,
`vercel-ai-gateway`, `xai`, the four `xiaomi*` providers, `zai`, and `zai-coding-cn`.
Compound providers are deliberately unsupported: `amazon-bedrock`, `azure-openai-responses`,
`cloudflare-workers-ai`, and `cloudflare-ai-gateway` require multiple credential/configuration
values. Selecting one fails before Pi starts; ambient AWS, Azure, and Cloudflare credentials are
still scrubbed. Supporting them requires a future dedicated provider-specific configuration.
## User-owned session server cutover ## User-owned session server cutover
+2 -1
View File
@@ -9,7 +9,8 @@
"catalog:migrate": "node dist/catalog/migrate.js", "catalog:migrate": "node dist/catalog/migrate.js",
"test": "vitest run", "test": "vitest run",
"start": "node dist/server.js", "start": "node dist/server.js",
"test:schema-v3-verifier": "python3 -I -B scripts/test_revision_state_policy.py && node --test scripts/verify-workspace-descriptor-files.test.mjs scripts/revision-state-policy.test.mjs" "test:schema-v4-verifier": "python3 -I -B scripts/test_revision_state_policy.py && node --test scripts/verify-workspace-descriptor-files.test.mjs scripts/revision-state-policy.test.mjs",
"test:schema-v3-verifier": "npm run test:schema-v4-verifier"
}, },
"dependencies": { "dependencies": {
"@fastify/cookie": "11.1.2", "@fastify/cookie": "11.1.2",
+1 -6
View File
@@ -1195,13 +1195,8 @@ export async function executeChecks({ checks, failAt, recorder } = {}) {
function baseWorkspace(id, evidenceSource) { function baseWorkspace(id, evidenceSource) {
return { return {
workspace: { schema_version: 3, id, name: `P1 ${id}`, language: "en" }, workspace: { schema_version: 4, id, name: `P1 ${id}`, language: "en" },
dwh: { engine: "postgres", database: "postgres", schema: "public", supported_transports: ["postgres_direct"] }, dwh: { engine: "postgres", database: "postgres", schema: "public", supported_transports: ["postgres_direct"] },
semantic_index: {
vector_store: { engine: "qdrant", collection: id, dimensions: 1024, distance: "cosine" },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
llm_policy: { allowed: ["zai/glm-5.2"] },
evidence: { source: evidenceSource, policy: { max_chunk_chars: 4000, retain_published_generations: 3 } }, evidence: { source: evidenceSource, policy: { max_chunk_chars: 4000, retain_published_generations: 3 } },
}; };
} }
+1 -1
View File
@@ -109,7 +109,7 @@ async function validateDistFiles(repo,files){const dist=join(repo,"backend","dis
export async function readManualOwnership({repositoryRoot=defaultRepositoryRoot}={}){const repo=realpathSync(repositoryRoot),root=fixedManualRoot(repo);noSymlinkExisting(repo,root);let rootEntry,ownershipEntry;try{rootEntry=await lstat(root);ownershipEntry=await lstat(join(root,"ownership.json"));}catch{throw new Error("manual ownership is missing");}if(!rootEntry.isDirectory()||rootEntry.isSymbolicLink()||await realpath(root)!==root||!ownershipEntry.isFile()||ownershipEntry.isSymbolicLink())throw new Error("manual ownership is unsafe");let value;try{value=JSON.parse(await readFile(join(root,"ownership.json"),"utf8"));}catch{throw new Error("manual ownership is malformed");}const baseValid=value.schemaVersion===1&&value.kind==="p1-manual-acceptance"&&HEX64.test(value.nonce??"")&&value.repositoryRoot===repo&&value.root===root&&value.status==="PENDING"&&["PREPARING","READY"].includes(value.stage)&&value.listener?.host===HOST&&value.listener?.port===PORT&&value.listener?.state==="stopped"&&typeof value.createdAt==="string"&&validEntrypoint(value.entrypoint,repo)&&validDistManifest(value.distManifest,root)&&JSON.stringify(value.resources)===JSON.stringify([root,{kind:"fastify",host:HOST,port:PORT}]);const readyLog=value.backendLog?.path===join(root,"logs/backend.log")&&Number.isSafeInteger(value.backendLog?.dev)&&Number.isSafeInteger(value.backendLog?.ino);if(!baseValid||(value.stage==="READY"?!readyLog:value.backendLog!==null))throw new Error("manual ownership identity mismatch");return value;} export async function readManualOwnership({repositoryRoot=defaultRepositoryRoot}={}){const repo=realpathSync(repositoryRoot),root=fixedManualRoot(repo);noSymlinkExisting(repo,root);let rootEntry,ownershipEntry;try{rootEntry=await lstat(root);ownershipEntry=await lstat(join(root,"ownership.json"));}catch{throw new Error("manual ownership is missing");}if(!rootEntry.isDirectory()||rootEntry.isSymbolicLink()||await realpath(root)!==root||!ownershipEntry.isFile()||ownershipEntry.isSymbolicLink())throw new Error("manual ownership is unsafe");let value;try{value=JSON.parse(await readFile(join(root,"ownership.json"),"utf8"));}catch{throw new Error("manual ownership is malformed");}const baseValid=value.schemaVersion===1&&value.kind==="p1-manual-acceptance"&&HEX64.test(value.nonce??"")&&value.repositoryRoot===repo&&value.root===root&&value.status==="PENDING"&&["PREPARING","READY"].includes(value.stage)&&value.listener?.host===HOST&&value.listener?.port===PORT&&value.listener?.state==="stopped"&&typeof value.createdAt==="string"&&validEntrypoint(value.entrypoint,repo)&&validDistManifest(value.distManifest,root)&&JSON.stringify(value.resources)===JSON.stringify([root,{kind:"fastify",host:HOST,port:PORT}]);const readyLog=value.backendLog?.path===join(root,"logs/backend.log")&&Number.isSafeInteger(value.backendLog?.dev)&&Number.isSafeInteger(value.backendLog?.ino);if(!baseValid||(value.stage==="READY"?!readyLog:value.backendLog!==null))throw new Error("manual ownership identity mismatch");return value;}
async function run(executable,argv,options={}){return await exec(executable,argv,{...options,maxBuffer:2*1024*1024,encoding:"utf8"});} async function run(executable,argv,options={}){return await exec(executable,argv,{...options,maxBuffer:2*1024*1024,encoding:"utf8"});}
function descriptor(id,source){return{workspace:{schema_version:3,id,name:`P1 ${id}`,language:"en"},dwh:{engine:"postgres",database:"postgres",schema:"public",supported_transports:["postgres_direct"]},semantic_index:{vector_store:{engine:"qdrant",collection:id,dimensions:1024,distance:"cosine"},embedding:{provider:"ollama_internal",model:"qwen3-embedding:0.6b",dimensions:1024}},llm_policy:{allowed:["zai/glm-5.2"]},evidence:{source,policy:{max_chunk_chars:4000,retain_published_generations:3}}};} function descriptor(id,source){return{workspace:{schema_version:4,id,name:`P1 ${id}`,language:"en"},dwh:{engine:"postgres",database:"postgres",schema:"public",supported_transports:["postgres_direct"]},evidence:{source,policy:{max_chunk_chars:4000,retain_published_generations:3}}};}
function descriptors(){return[descriptor("p1-filesystem",{type:"filesystem",uri:"workspace-content/p1-filesystem/evidence",patterns:["**/*.md"],max_bytes:10485760}),descriptor("p1-http",{type:"http",uris:["https://evidence.example.test/guide.md"],authentication:"signed_urls_file",connect_timeout_ms:1250,read_timeout_ms:30001,max_bytes:12345,max_redirects:2,allow_private_hosts:false,max_cache_bytes:67890}),descriptor("p1-s3",{type:"s3",uri:"s3://p1-evidence/published/",endpoint_url:"https://s3.example.test/",region:"eu-west-1",credentials:"static_files",trusted_endpoint:true,allow_private_endpoint:false,allow_insecure_endpoint:false,max_bytes:12345,max_objects:33,max_pages:4,page_size:5})];} function descriptors(){return[descriptor("p1-filesystem",{type:"filesystem",uri:"workspace-content/p1-filesystem/evidence",patterns:["**/*.md"],max_bytes:10485760}),descriptor("p1-http",{type:"http",uris:["https://evidence.example.test/guide.md"],authentication:"signed_urls_file",connect_timeout_ms:1250,read_timeout_ms:30001,max_bytes:12345,max_redirects:2,allow_private_hosts:false,max_cache_bytes:67890}),descriptor("p1-s3",{type:"s3",uri:"s3://p1-evidence/published/",endpoint_url:"https://s3.example.test/",region:"eu-west-1",credentials:"static_files",trusted_endpoint:true,allow_private_endpoint:false,allow_insecure_endpoint:false,max_bytes:12345,max_objects:33,max_pages:4,page_size:5})];}
function quote(value){return `'${String(value).replaceAll("'",`'"'"'`)}'`;} function quote(value){return `'${String(value).replaceAll("'",`'"'"'`)}'`;}
async function checkPrerequisites(repo){for(const path of ["scripts/p1-acceptance.sh","scripts/test-p1-acceptance.sh","backend/scripts/p1-acceptance.mjs","backend/dist/server.js"]){try{await access(join(repo,path));}catch{throw new Error(`Task 8 prerequisite is missing: ${path}`);}}for(const command of ["node","npm","git","curl","unzip","zipinfo","lsof","python3"]){try{await run(command,[command==="unzip"||command==="lsof"?"-v":command==="zipinfo"?"-h":"--version"]);}catch{throw new Error(`missing prerequisite: ${command}`);}}const tht=join(repo,"harness",".venv","bin","tht");try{await access(tht,constants.X_OK);}catch{throw new Error("missing prerequisite: harness/.venv/bin/tht");}} async function checkPrerequisites(repo){for(const path of ["scripts/p1-acceptance.sh","scripts/test-p1-acceptance.sh","backend/scripts/p1-acceptance.mjs","backend/dist/server.js"]){try{await access(join(repo,path));}catch{throw new Error(`Task 8 prerequisite is missing: ${path}`);}}for(const command of ["node","npm","git","curl","unzip","zipinfo","lsof","python3"]){try{await run(command,[command==="unzip"||command==="lsof"?"-v":command==="zipinfo"?"-h":"--version"]);}catch{throw new Error(`missing prerequisite: ${command}`);}}const tht=join(repo,"harness",".venv","bin","tht");try{await access(tht,constants.X_OK);}catch{throw new Error("missing prerequisite: harness/.venv/bin/tht");}}
@@ -403,7 +403,7 @@ test("generated render command validates saved responses and owned snapshot befo
}); });
const renderSnapshotYaml=`workspace: const renderSnapshotYaml=`workspace:
schema_version: 3 schema_version: 4
id: p1-filesystem id: p1-filesystem
name: P1 filesystem name: P1 filesystem
language: en language: en
@@ -412,11 +412,6 @@ dwh:
database: postgres database: postgres
schema: public schema: public
supported_transports: [postgres_direct] supported_transports: [postgres_direct]
semantic_index:
vector_store: {engine: qdrant, collection: p1-filesystem, dimensions: 1024, distance: cosine}
embedding: {provider: ollama_internal, model: qwen3-embedding:0.6b, dimensions: 1024}
llm_policy:
allowed: [zai/glm-5.2]
evidence: evidence:
source: {type: filesystem, uri: workspace-content/p1-filesystem/evidence, patterns: ["**/*.md"], max_bytes: 10485760} source: {type: filesystem, uri: workspace-content/p1-filesystem/evidence, patterns: ["**/*.md"], max_bytes: 10485760}
policy: {max_chunk_chars: 4000, retain_published_generations: 3} policy: {max_chunk_chars: 4000, retain_published_generations: 3}
+2 -7
View File
@@ -16,7 +16,7 @@ async function fixture() {
await writeFile(join(root,"installation/base.yaml"),"{}\n"); await writeFile(join(root,"installation/base.yaml"),"{}\n");
const secret=join(root,"fixture-secrets/dwh-password"); await writeFile(secret,"not-inspected",{mode:0o600}); const secret=join(root,"fixture-secrets/dwh-password"); await writeFile(secret,"not-inspected",{mode:0o600});
await writeFile(snapshot,`workspace: await writeFile(snapshot,`workspace:
schema_version: 3 schema_version: 4
id: p1-filesystem id: p1-filesystem
name: P1 filesystem name: P1 filesystem
language: en language: en
@@ -25,11 +25,6 @@ dwh:
database: postgres database: postgres
schema: public schema: public
supported_transports: [postgres_direct] supported_transports: [postgres_direct]
semantic_index:
vector_store: {engine: qdrant, collection: p1-filesystem, dimensions: 1024, distance: cosine}
embedding: {provider: ollama_internal, model: qwen3-embedding:0.6b, dimensions: 1024}
llm_policy:
allowed: [zai/glm-5.2]
evidence: evidence:
source: {type: filesystem, uri: workspace-content/p1-filesystem/evidence, patterns: ["**/*.md"], max_bytes: 10485760} source: {type: filesystem, uri: workspace-content/p1-filesystem/evidence, patterns: ["**/*.md"], max_bytes: 10485760}
policy: {max_chunk_chars: 4000, retain_published_generations: 3} policy: {max_chunk_chars: 4000, retain_published_generations: 3}
@@ -61,7 +56,7 @@ test("renderer refuses snapshot manifest head, digest, and expected-digest tampe
test("renderer refuses a missing or malformed snapshot manifest",async()=>{ const f=await fixture(); const output=join(f.root,"rendered/nomanifest.yaml"); await rm(f.manifestPath); await assert.rejects(call(f,{outputPath:output}),/snapshot manifest.*(missing|unbounded|unsafe)/); await writeFile(f.manifestPath,"{not json"); await assert.rejects(call(f,{outputPath:output}),/snapshot manifest.*malformed/); await assert.rejects(lstat(output)); assert.deepEqual(await runtimeLeases(f),[]); }); test("renderer refuses a missing or malformed snapshot manifest",async()=>{ const f=await fixture(); const output=join(f.root,"rendered/nomanifest.yaml"); await rm(f.manifestPath); await assert.rejects(call(f,{outputPath:output}),/snapshot manifest.*(missing|unbounded|unsafe)/); await writeFile(f.manifestPath,"{not json"); await assert.rejects(call(f,{outputPath:output}),/snapshot manifest.*malformed/); await assert.rejects(lstat(output)); assert.deepEqual(await runtimeLeases(f),[]); });
test("renderer rejects a regular snapshot replacement against its manifest",async()=>{ const f=await fixture(); const output=join(f.root,"rendered/replaced.yaml"); await assert.rejects(call(f,{outputPath:output,beforePublish:async()=>{await writeFile(f.snapshot,"workspace:\n schema_version: 3\n id: p1-filesystem\n name: replaced\n")}}),/snapshot content changed/); await assert.rejects(lstat(output)); }); test("renderer rejects a regular snapshot replacement against its manifest",async()=>{ const f=await fixture(); const output=join(f.root,"rendered/replaced.yaml"); await assert.rejects(call(f,{outputPath:output,beforePublish:async()=>{await writeFile(f.snapshot,"workspace:\n schema_version: 4\n id: p1-filesystem\n name: replaced\n")}}),/snapshot content changed/); await assert.rejects(lstat(output)); });
test("renderer anchors publication when rendered parent is concurrently swapped", async()=>{ test("renderer anchors publication when rendered parent is concurrently swapped", async()=>{
const f=await fixture(),output=join(f.root,"rendered/raced.yaml"),moved=join(f.root,"rendered-moved"),outside=join(f.repo,"outside-rendered"); await mkdir(outside); const f=await fixture(),output=join(f.root,"rendered/raced.yaml"),moved=join(f.root,"rendered-moved"),outside=join(f.repo,"outside-rendered"); await mkdir(outside);
+1 -6
View File
@@ -328,13 +328,8 @@ async function tht(ctx, argv, options = {}) {
function namespace(id) { return id.toUpperCase().replaceAll("-", "_"); } function namespace(id) { return id.toUpperCase().replaceAll("-", "_"); }
function baseWorkspace(id, evidenceSource) { function baseWorkspace(id, evidenceSource) {
return { return {
workspace: { schema_version: 3, id, name: `P1.1 ${id}`, description: `Catalog entry for ${id}`, language: "en" }, workspace: { schema_version: 4, id, name: `P1.1 ${id}`, description: `Catalog entry for ${id}`, language: "en" },
dwh: { engine: "postgres", database: "postgres", schema: "public", supported_transports: ["postgres_direct"] }, dwh: { engine: "postgres", database: "postgres", schema: "public", supported_transports: ["postgres_direct"] },
semantic_index: {
vector_store: { engine: "qdrant", collection: id, dimensions: 1024, distance: "cosine" },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
llm_policy: { allowed: ["zai/glm-5.2"] },
evidence: { source: evidenceSource, policy: { max_chunk_chars: 4000, retain_published_generations: 3 } }, evidence: { source: evidenceSource, policy: { max_chunk_chars: 4000, retain_published_generations: 3 } },
}; };
} }
+1 -6
View File
@@ -81,13 +81,8 @@ async function git(executable, argv, options = {}) {
function namespace(id) { return id.toUpperCase().replaceAll("-", "_"); } function namespace(id) { return id.toUpperCase().replaceAll("-", "_"); }
function baseWorkspace(id, evidenceSource) { function baseWorkspace(id, evidenceSource) {
return { return {
workspace: { schema_version: 3, id, name: `P1.1 ${id}`, description: `Catalog entry for ${id}`, language: "en" }, workspace: { schema_version: 4, id, name: `P1.1 ${id}`, description: `Catalog entry for ${id}`, language: "en" },
dwh: { engine: "postgres", database: "postgres", schema: "public", supported_transports: ["postgres_direct"] }, dwh: { engine: "postgres", database: "postgres", schema: "public", supported_transports: ["postgres_direct"] },
semantic_index: {
vector_store: { engine: "qdrant", collection: id, dimensions: 1024, distance: "cosine" },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
llm_policy: { allowed: ["zai/glm-5.2"] },
evidence: { source: evidenceSource, policy: { max_chunk_chars: 4000, retain_published_generations: 3 } }, evidence: { source: evidenceSource, policy: { max_chunk_chars: 4000, retain_published_generations: 3 } },
}; };
} }
+1 -6
View File
@@ -375,16 +375,11 @@ function installationProjectName(installationPath) {
function baseWorkspace(id, { dwhBaseUrl, evidenceSource }) { function baseWorkspace(id, { dwhBaseUrl, evidenceSource }) {
return { return {
workspace: { schema_version: 3, id, name: `P2 ${id}`, language: "en" }, workspace: { schema_version: 4, id, name: `P2 ${id}`, language: "en" },
dwh: { engine: "postgres", database: "warehouse", schema: "dw", supported_transports: ["rest_api"] }, dwh: { engine: "postgres", database: "warehouse", schema: "dw", supported_transports: ["rest_api"] },
semantic_index: {
vector_store: { engine: "qdrant", collection: id, dimensions: 1024, distance: "cosine" },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
diagnostics: { diagnostics: {
dwh_rest: { method: "POST", path: "/rpc/ping", auth: "x-api-key", response: { database: "database", schema: "schema" } }, dwh_rest: { method: "POST", path: "/rpc/ping", auth: "x-api-key", response: { database: "database", schema: "schema" } },
}, },
llm_policy: { allowed: ["zai/glm-5.2"] },
...(evidenceSource ? { evidence: { source: evidenceSource, policy: { max_chunk_chars: 4000, retain_published_generations: 3 } } } : {}), ...(evidenceSource ? { evidence: { source: evidenceSource, policy: { max_chunk_chars: 4000, retain_published_generations: 3 } } } : {}),
}; };
} }
+1 -6
View File
@@ -376,16 +376,11 @@ function installationProjectName(installationPath) {
function baseWorkspace(id, { dwhBaseUrl, evidenceSource }) { function baseWorkspace(id, { dwhBaseUrl, evidenceSource }) {
return { return {
workspace: { schema_version: 3, id, name: `P2 ${id}`, language: "en" }, workspace: { schema_version: 4, id, name: `P2 ${id}`, language: "en" },
dwh: { engine: "postgres", database: "warehouse", schema: "dw", supported_transports: ["rest_api"] }, dwh: { engine: "postgres", database: "warehouse", schema: "dw", supported_transports: ["rest_api"] },
semantic_index: {
vector_store: { engine: "qdrant", collection: id, dimensions: 1024, distance: "cosine" },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
diagnostics: { diagnostics: {
dwh_rest: { method: "POST", path: "/rpc/ping", auth: "x-api-key", response: { database: "database", schema: "schema" } }, dwh_rest: { method: "POST", path: "/rpc/ping", auth: "x-api-key", response: { database: "database", schema: "schema" } },
}, },
llm_policy: { allowed: ["zai/glm-5.2"] },
...(evidenceSource ? { evidence: { source: evidenceSource, policy: { max_chunk_chars: 4000, retain_published_generations: 3 } } } : {}), ...(evidenceSource ? { evidence: { source: evidenceSource, policy: { max_chunk_chars: 4000, retain_published_generations: 3 } } } : {}),
}; };
} }
+1 -6
View File
@@ -379,16 +379,11 @@ function installationProjectName(installationPath) {
function baseWorkspace(id, { dwhBaseUrl, evidenceSource }) { function baseWorkspace(id, { dwhBaseUrl, evidenceSource }) {
return { return {
workspace: { schema_version: 3, id, name: `P2 ${id}`, language: "en" }, workspace: { schema_version: 4, id, name: `P2 ${id}`, language: "en" },
dwh: { engine: "postgres", database: "warehouse", schema: "dw", supported_transports: ["rest_api"] }, dwh: { engine: "postgres", database: "warehouse", schema: "dw", supported_transports: ["rest_api"] },
semantic_index: {
vector_store: { engine: "qdrant", collection: id, dimensions: 1024, distance: "cosine" },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
diagnostics: { diagnostics: {
dwh_rest: { method: "POST", path: "/rpc/ping", auth: "x-api-key", response: { database: "database", schema: "schema" } }, dwh_rest: { method: "POST", path: "/rpc/ping", auth: "x-api-key", response: { database: "database", schema: "schema" } },
}, },
llm_policy: { allowed: ["zai/glm-5.2"] },
...(evidenceSource ? { evidence: { source: evidenceSource, policy: { max_chunk_chars: 4000, retain_published_generations: 3 } } } : {}), ...(evidenceSource ? { evidence: { source: evidenceSource, policy: { max_chunk_chars: 4000, retain_published_generations: 3 } } } : {}),
}; };
} }
+1 -6
View File
@@ -374,16 +374,11 @@ function installationProjectName(installationPath) {
function baseWorkspace(id, { dwhBaseUrl, evidenceSource }) { function baseWorkspace(id, { dwhBaseUrl, evidenceSource }) {
return { return {
workspace: { schema_version: 3, id, name: `P2 ${id}`, language: "en" }, workspace: { schema_version: 4, id, name: `P2 ${id}`, language: "en" },
dwh: { engine: "postgres", database: "warehouse", schema: "dw", supported_transports: ["rest_api"] }, dwh: { engine: "postgres", database: "warehouse", schema: "dw", supported_transports: ["rest_api"] },
semantic_index: {
vector_store: { engine: "qdrant", collection: id, dimensions: 1024, distance: "cosine" },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
diagnostics: { diagnostics: {
dwh_rest: { method: "POST", path: "/rpc/ping", auth: "x-api-key", response: { database: "database", schema: "schema" } }, dwh_rest: { method: "POST", path: "/rpc/ping", auth: "x-api-key", response: { database: "database", schema: "schema" } },
}, },
llm_policy: { allowed: ["zai/glm-5.2"] },
...(evidenceSource ? { evidence: { source: evidenceSource, policy: { max_chunk_chars: 4000, retain_published_generations: 3 } } } : {}), ...(evidenceSource ? { evidence: { source: evidenceSource, policy: { max_chunk_chars: 4000, retain_published_generations: 3 } } } : {}),
}; };
} }
+1 -6
View File
@@ -374,16 +374,11 @@ function installationProjectName(installationPath) {
function baseWorkspace(id, { dwhBaseUrl, evidenceSource }) { function baseWorkspace(id, { dwhBaseUrl, evidenceSource }) {
return { return {
workspace: { schema_version: 3, id, name: `P2 ${id}`, language: "en" }, workspace: { schema_version: 4, id, name: `P2 ${id}`, language: "en" },
dwh: { engine: "postgres", database: "warehouse", schema: "dw", supported_transports: ["rest_api"] }, dwh: { engine: "postgres", database: "warehouse", schema: "dw", supported_transports: ["rest_api"] },
semantic_index: {
vector_store: { engine: "qdrant", collection: id, dimensions: 1024, distance: "cosine" },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
diagnostics: { diagnostics: {
dwh_rest: { method: "POST", path: "/rpc/ping", auth: "x-api-key", response: { database: "database", schema: "schema" } }, dwh_rest: { method: "POST", path: "/rpc/ping", auth: "x-api-key", response: { database: "database", schema: "schema" } },
}, },
llm_policy: { allowed: ["zai/glm-5.2"] },
...(evidenceSource ? { evidence: { source: evidenceSource, policy: { max_chunk_chars: 4000, retain_published_generations: 3 } } } : {}), ...(evidenceSource ? { evidence: { source: evidenceSource, policy: { max_chunk_chars: 4000, retain_published_generations: 3 } } } : {}),
}; };
} }
@@ -36,11 +36,10 @@ const reviewedExpandableBlocks = new Map([
{ sha256: "b903e5dae953ae1372f1a5276f12a92ed3dd632b897f3afe5e00c646d90a1b42", rationale: "Same reviewed block in the repository-required CRLF checkout representation." }, { sha256: "b903e5dae953ae1372f1a5276f12a92ed3dd632b897f3afe5e00c646d90a1b42", rationale: "Same reviewed block in the repository-required CRLF checkout representation." },
]], ]],
["scripts/unified-deployment-smoke.sh", [ ["scripts/unified-deployment-smoke.sh", [
{ sha256: "1d60bf140165a8fabfa0c3729e776136904717e67becf3e0ab68c70d8e37847e", rationale: "Generates reviewed Task 13 runtime configuration." }, { sha256: "b6c0826151b2c8b955399d1abf5b691cc8fe6b6454b17da000dde7ba3bc55d2d", rationale: "Generates the reviewed local Task 13 Compose override with normalized catalog mounts." },
{ sha256: "36d3d8a2362dbdc4fad90948d6c227586d749f56b9a4bc5b6b5a91bcbec6407b", rationale: "Generates the reviewed local Task 13 Compose override." }, { sha256: "24f69d12b8554aa2bebba455be99fde3e60743eef5a40fa2ef5b29397a477c03", rationale: "Generates the reviewed local Task 13 installation descriptor with its model catalog." },
{ sha256: "c556f7d910d0788e219b042957e6b307cb9925b43920c680535d0d3a6dcbdb25", rationale: "Generates the reviewed local Task 13 installation descriptor." },
{ sha256: "526006fa6d48a8080b3834723630c64de5005a67243e944ebf1da15212b4d654", rationale: "Generates the reviewed server Task 13 Compose override." }, { sha256: "526006fa6d48a8080b3834723630c64de5005a67243e944ebf1da15212b4d654", rationale: "Generates the reviewed server Task 13 Compose override." },
{ sha256: "c57ae2205c21ead0c2015a353aaabb948fa4ddd9b78a2cdcdb71f48cf2db742d", rationale: "Generates the reviewed projected-auth server Task 13 installation descriptor." }, { sha256: "406ccead1967f642225c946fc4a23fe5b019c9764cc5153e1125876ade16ec90", rationale: "Generates the reviewed projected-auth server Task 13 installation descriptor with its model catalog." },
]], ]],
["scripts/vector-backup.sh", [ ["scripts/vector-backup.sh", [
{ sha256: "571899db49dfdcec8107fbe1e0a86a61e7581979d3c4c248c20546843e275bcf", rationale: "Generates the reviewed backup manifest inside the helper command." }, { sha256: "571899db49dfdcec8107fbe1e0a86a61e7581979d3c4c248c20546843e275bcf", rationale: "Generates the reviewed backup manifest inside the helper command." },
@@ -103,6 +102,7 @@ function isPolicyImplementationException(label, category) {
]); ]);
if (implementations.has(label)) return true; if (implementations.has(label)) return true;
if (category === "migration-marker" && new Set([ if (category === "migration-marker" && new Set([
"backend/src/workspaces/schema.ts",
"scripts/workspace_descriptor_doc_contract.py", "scripts/workspace_descriptor_doc_contract.py",
"scripts/test_workspace_descriptor_doc_contract.py", "scripts/test_workspace_descriptor_doc_contract.py",
"backend/scripts/clean-dist.test.mjs", "backend/scripts/clean-dist.test.mjs",
@@ -173,7 +173,7 @@ function validateWorkspaceSource(source, label, { requireWorkspace, expandable =
try { try {
parseWorkspaceYaml(source); parseWorkspaceYaml(source);
} catch (error) { } catch (error) {
throw new Error(`${label}: workspace descriptor is not valid schema v3: ${error instanceof Error ? error.message : String(error)}`); throw new Error(`${label}: workspace descriptor is not valid schema v4: ${error instanceof Error ? error.message : String(error)}`);
} }
return true; return true;
} }
@@ -33,20 +33,20 @@ function bashN(root, path) {
function replaceWorkspaceKeys(source, workspaceKey, schemaLine) { function replaceWorkspaceKeys(source, workspaceKey, schemaLine) {
return source return source
.replace(/^workspace:$/m, workspaceKey) .replace(/^workspace:$/m, workspaceKey)
.replace(/^ schema_version: 3$/m, schemaLine); .replace(/^ schema_version: 4$/m, schemaLine);
} }
test("production parser accepts semantic v3 with quoted Unicode/tagged keys and spacing", async (t) => { test("production parser accepts semantic v4 with quoted Unicode/tagged keys and spacing", async (t) => {
const root = await fixture(t); const root = await fixture(t);
const unicode = replaceWorkspaceKeys( const unicode = replaceWorkspaceKeys(
canonicalDescriptor, canonicalDescriptor,
'"\\u0077orkspace" :', '"\\u0077orkspace" :',
' "\\u0073chema_version" : 3', ' "\\u0073chema_version" : 4',
); );
const tagged = replaceWorkspaceKeys( const tagged = replaceWorkspaceKeys(
canonicalDescriptor, canonicalDescriptor,
"!!str workspace :", "!!str workspace :",
" !!str schema_version : 3", " !!str schema_version : 4",
); );
await put(root, "deploy/workspaces/unicode.yaml", unicode); await put(root, "deploy/workspaces/unicode.yaml", unicode);
await put(root, "deploy/workspaces/tagged.yaml", tagged); await put(root, "deploy/workspaces/tagged.yaml", tagged);
@@ -59,14 +59,15 @@ test("production parser accepts semantic v3 with quoted Unicode/tagged keys and
}); });
}); });
test("production parser rejects fancy keys with every non-v3 or ambiguous value", async (t) => { test("production parser rejects fancy keys with every non-v4 or ambiguous value", async (t) => {
const invalid = [ const invalid = [
["unicode-v2", '"\\u0077orkspace" :', ' "\\u0073chema_version" : 2'], ["unicode-v2", '"\\u0077orkspace" :', ' "\\u0073chema_version" : 2'],
["tagged-leading-zero", "!!str workspace :", " !!str schema_version : 02"], ["unicode-v3", '"\\u0077orkspace" :', ' "\\u0073chema_version" : 3'],
["hexadecimal", "workspace :", " schema_version : 0x2"], ["tagged-leading-zero", "!!str workspace :", " !!str schema_version : 03"],
["multiline", "workspace :", " schema_version : >\n 3"], ["hexadecimal", "workspace :", " schema_version : 0x3"],
["duplicate", "workspace :", " schema_version : 3\n schema_version: 3"], ["multiline", "workspace :", " schema_version : >\n 4"],
["inline", "workspace: { schema_version: 3 }", " schema_version: 3"], ["duplicate", "workspace :", " schema_version : 4\n schema_version: 4"],
["inline", "workspace: { schema_version: 4 }", " schema_version: 4"],
]; ];
for (const [name, workspaceKey, schemaLine] of invalid) { for (const [name, workspaceKey, schemaLine] of invalid) {
await t.test(name, async () => { await t.test(name, async () => {
@@ -120,7 +121,7 @@ test("PowerShell embedded workspace mappings are rejected while bundle-only stri
const root = await fixture(t); const root = await fixture(t);
const source = [ const source = [
"$workspace = @'", "$workspace = @'",
canonicalDescriptor.replace(" schema_version: 3", " schema_version: 0x2").trimEnd(), canonicalDescriptor.replace(" schema_version: 4", " schema_version: 0x2").trimEnd(),
"'@", "'@",
'$bundle = @"', '$bundle = @"',
"bundle:", "bundle:",
@@ -137,7 +138,7 @@ test("PowerShell embedded workspace mappings are rejected while bundle-only stri
test("workspace descriptor family entries require a top-level workspace", async (t) => { test("workspace descriptor family entries require a top-level workspace", async (t) => {
const root = await fixture(t); const root = await fixture(t);
await put(root, "scripts/fixtures/workspace-registry-future.yaml", "bundle:\n schema_version: 3\n"); await put(root, "scripts/fixtures/workspace-registry-future.yaml", "bundle:\n schema_version: 4\n");
await assert.rejects( await assert.rejects(
verifyEntries({ verifyEntries({
root, root,
@@ -179,7 +180,7 @@ test("script scalar workspace remains a bundle even with descriptor-like sibling
test("standalone descriptor files require workspace to be a mapping", async (t) => { test("standalone descriptor files require workspace to be a mapping", async (t) => {
const root = await fixture(t); const root = await fixture(t);
const path = "scripts/fixtures/workspace-registry-scalar.yaml"; const path = "scripts/fixtures/workspace-registry-scalar.yaml";
await put(root, path, "workspace: analytics\nschema_version: 3\n"); await put(root, path, "workspace: analytics\nschema_version: 4\n");
await assert.rejects( await assert.rejects(
verifyEntries({ root, entries: [entry("workspace_descriptor", path)] }), verifyEntries({ root, entries: [entry("workspace_descriptor", path)] }),
/workspace.*mapping/i, /workspace.*mapping/i,
@@ -193,11 +194,11 @@ test("Bash extractor supports hyphen, digit, escaped delimiters, and tab strippi
name: "hyphen-v2", name: "hyphen-v2",
opener: "cat <<'WORKSPACE-YAML'", opener: "cat <<'WORKSPACE-YAML'",
delimiter: "WORKSPACE-YAML", delimiter: "WORKSPACE-YAML",
descriptor: canonicalDescriptor.replace(" schema_version: 3", " schema_version: 2"), descriptor: canonicalDescriptor.replace(" schema_version: 4", " schema_version: 2"),
rejected: true, rejected: true,
}, },
{ {
name: "digit-v3", name: "digit-v4",
opener: "cat <<2YAML", opener: "cat <<2YAML",
delimiter: "2YAML", delimiter: "2YAML",
descriptor: canonicalDescriptor, descriptor: canonicalDescriptor,
@@ -207,11 +208,11 @@ test("Bash extractor supports hyphen, digit, escaped delimiters, and tab strippi
name: "escaped-v2", name: "escaped-v2",
opener: "cat <<WORKSPACE\\-YAML", opener: "cat <<WORKSPACE\\-YAML",
delimiter: "WORKSPACE-YAML", delimiter: "WORKSPACE-YAML",
descriptor: canonicalDescriptor.replace(" schema_version: 3", " schema_version: 2"), descriptor: canonicalDescriptor.replace(" schema_version: 4", " schema_version: 2"),
rejected: true, rejected: true,
}, },
{ {
name: "tab-strip-v3", name: "tab-strip-v4",
opener: "cat <<-'TAB-YAML'", opener: "cat <<-'TAB-YAML'",
delimiter: "\tTAB-YAML", delimiter: "\tTAB-YAML",
descriptor: canonicalDescriptor.split("\n").map((line) => `\t${line}`).join("\n"), descriptor: canonicalDescriptor.split("\n").map((line) => `\t${line}`).join("\n"),
@@ -272,7 +273,7 @@ test("non-stripping heredoc close requires an exact physical delimiter line", as
"#!/usr/bin/env bash", "#!/usr/bin/env bash",
"cat <<'---'", "cat <<'---'",
"--- ", "--- ",
canonicalDescriptor.replace(" schema_version: 3", " schema_version: 2").trimEnd(), canonicalDescriptor.replace(" schema_version: 4", " schema_version: 2").trimEnd(),
"---", "---",
"", "",
].join("\n"); ].join("\n");
@@ -313,7 +314,7 @@ test("double-quoted non-special backslash is preserved in the delimiter", async
"#!/usr/bin/env bash", "#!/usr/bin/env bash",
'cat <<"\\---"', 'cat <<"\\---"',
"---", "---",
canonicalDescriptor.replace(" schema_version: 3", " schema_version: 2").trimEnd(), canonicalDescriptor.replace(" schema_version: 4", " schema_version: 2").trimEnd(),
"\\---", "\\---",
"", "",
].join("\n"); ].join("\n");
@@ -355,7 +356,7 @@ test("split heredoc operator continuation cannot bypass v2 validation", async (t
"#!/usr/bin/env bash", "#!/usr/bin/env bash",
"cat <\\", "cat <\\",
"<'YAML'", "<'YAML'",
canonicalDescriptor.replace(" schema_version: 3", " schema_version: 2").trimEnd(), canonicalDescriptor.replace(" schema_version: 4", " schema_version: 2").trimEnd(),
"YAML", "YAML",
"", "",
].join("\n"); ].join("\n");
@@ -424,7 +425,7 @@ test("PowerShell comment backslash cannot hide a following v2 here-string", asyn
const source = [ const source = [
"# harmless PowerShell comment \\", "# harmless PowerShell comment \\",
"$workspace = @'", "$workspace = @'",
canonicalDescriptor.replace(" schema_version: 3", " schema_version: 2").trimEnd(), canonicalDescriptor.replace(" schema_version: 4", " schema_version: 2").trimEnd(),
"'@", "'@",
"", "",
].join("\n"); ].join("\n");
@@ -435,7 +436,7 @@ test("PowerShell comment backslash cannot hide a following v2 here-string", asyn
); );
}); });
test("PowerShell dialect accepts normal v3 and non-workspace bundle here-strings", async (t) => { test("PowerShell dialect accepts normal v4 and non-workspace bundle here-strings", async (t) => {
const root = await fixture(t); const root = await fixture(t);
const path = "scripts/powershell-valid-smoke.ps1"; const path = "scripts/powershell-valid-smoke.ps1";
const source = [ const source = [
@@ -494,10 +495,10 @@ test("PowerShell cast and concatenation openers cannot hide embedded descriptors
test("expandable YAML interpolation that can hide a workspace descriptor fails closed", async (t) => { test("expandable YAML interpolation that can hide a workspace descriptor fails closed", async (t) => {
const root = await fixture(t); const root = await fixture(t);
const cases = [ const cases = [
["braced-key", "${key}:\n schema_version: 3"], ["braced-key", "${key}:\n schema_version: 4"],
["plain-key", "$key:\n schema_version: 3"], ["plain-key", "$key:\n schema_version: 4"],
["quoted-key", '"$key" :\n schema_version: 3'], ["quoted-key", '"$key" :\n schema_version: 4'],
["subexpression-key", "$($key):\n schema_version: 3"], ["subexpression-key", "$($key):\n schema_version: 4"],
["version", "workspace:\n schema_version: $version"], ["version", "workspace:\n schema_version: $version"],
]; ];
for (const [name, body] of cases) { for (const [name, body] of cases) {
@@ -564,7 +565,7 @@ test("unmarked expandable Bash YAML cannot generate descriptor keys or values at
"key=workspace", "key=workspace",
"cat <<YAML", "cat <<YAML",
generatedKey, generatedKey,
" schema_version: 3", " schema_version: 4",
"YAML", "YAML",
"", "",
].join("\n"); ].join("\n");
@@ -600,14 +601,14 @@ test("an in-band marker cannot authorize expandable content", async (t) => {
for (const [path, source] of [ for (const [path, source] of [
["scripts/fake-marker.sh", [ ["scripts/fake-marker.sh", [
"#!/usr/bin/env bash", "#!/usr/bin/env bash",
"# schema-v3-only: expandable-nonworkspace", "# schema-v4-only: expandable-nonworkspace",
"cat <<YAML", "cat <<YAML",
"${DESCRIPTOR}", "${DESCRIPTOR}",
"YAML", "YAML",
"", "",
].join("\n")], ].join("\n")],
["scripts/fake-marker.ps1", [ ["scripts/fake-marker.ps1", [
"# schema-v3-only: expandable-nonworkspace", "# schema-v4-only: expandable-nonworkspace",
'$yaml = @"', '$yaml = @"',
"$descriptor", "$descriptor",
'"@', '"@',
+19 -7
View File
@@ -22,7 +22,8 @@ import { isUsableAuthenticationSecret } from "./auth/secret-policy.js";
import { secretValue } from "./config/secret-bundle.js"; import { secretValue } from "./config/secret-bundle.js";
import { sessionRoutes } from "./routes/sessions.js"; import { sessionRoutes } from "./routes/sessions.js";
import { sqlRoutes } from "./routes/sql.js"; import { sqlRoutes } from "./routes/sql.js";
import { metaRoutes, type ListModelsFn } from "./routes/meta.js"; import { metaRoutes } from "./routes/meta.js";
import type { ListModelsFn } from "./pi/list-models.js";
import { settingsRoutes, effectiveSettings } from "./routes/settings.js"; import { settingsRoutes, effectiveSettings } from "./routes/settings.js";
import { createPiModelLister } from "./pi/list-models.js"; import { createPiModelLister } from "./pi/list-models.js";
import { createPiManagement, type PiManagementService } from "./pi/management.js"; import { createPiManagement, type PiManagementService } from "./pi/management.js";
@@ -66,6 +67,7 @@ import { catalogDescriptionGenerationRoutes } from "./routes/catalog-description
import { CatalogLogicalRelationshipService } from "./catalog/logical-relationship-service.js"; import { CatalogLogicalRelationshipService } from "./catalog/logical-relationship-service.js";
import { catalogLogicalRelationshipRoutes } from "./routes/catalog-logical-relationships.js"; import { catalogLogicalRelationshipRoutes } from "./routes/catalog-logical-relationships.js";
import { EffectiveRelationshipSnapshotProvider } from "./catalog/effective-relationship-snapshot.js"; import { EffectiveRelationshipSnapshotProvider } from "./catalog/effective-relationship-snapshot.js";
import { loadRuntimeModelCatalog, type RuntimeModelCatalog } from "./models/runtime-model-catalog.js";
export interface BuildAppDeps { export interface BuildAppDeps {
thtRunner?: ThtRunner; thtRunner?: ThtRunner;
@@ -88,6 +90,7 @@ export interface BuildAppDeps {
catalogSyncWorker?: CatalogSyncWorker; catalogSyncWorker?: CatalogSyncWorker;
catalogOperationCoordinator?: CatalogOperationCoordinator; catalogOperationCoordinator?: CatalogOperationCoordinator;
metadataGenerationModels?: MetadataGenerationModels; metadataGenerationModels?: MetadataGenerationModels;
runtimeModelCatalog?: RuntimeModelCatalog;
modelCompleter?: ModelCompleter; modelCompleter?: ModelCompleter;
descriptionSourceSampler?: DescriptionSourceSampler; descriptionSourceSampler?: DescriptionSourceSampler;
workspaceRuntimeSupport?: (workspace: WorkspaceDescriptor) => boolean; workspaceRuntimeSupport?: (workspace: WorkspaceDescriptor) => boolean;
@@ -155,17 +158,22 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
semanticRuntime: { semanticRuntime: {
internalQdrantUrl: config.internalQdrantUrl, internalQdrantUrl: config.internalQdrantUrl,
internalEmbeddingUrl: config.internalEmbeddingUrl, internalEmbeddingUrl: config.internalEmbeddingUrl,
internalEmbeddingId: config.internalEmbeddingId,
internalEmbeddingModel: config.internalEmbeddingModel, internalEmbeddingModel: config.internalEmbeddingModel,
internalEmbeddingDimensions: config.internalEmbeddingDimensions, internalEmbeddingDimensions: config.internalEmbeddingDimensions,
}, },
}); });
const mgr = deps?.mgr ?? new PiProcessManager(config, deps?.spawnFn ? { spawnFn: deps.spawnFn } : undefined);
const hub = deps?.hub ?? new SseHub(); const hub = deps?.hub ?? new SseHub();
const workspaceRegistry = deps?.workspaceRegistry ?? new WorkspaceRegistry(config.workspaceRegistry); const workspaceRegistry = deps?.workspaceRegistry ?? new WorkspaceRegistry(config.workspaceRegistry);
const catalogRepository = deps?.catalogRepository ?? createCatalogRepository(config.catalogDatabase); const catalogRepository = deps?.catalogRepository ?? createCatalogRepository(config.catalogDatabase);
const catalogOperationCoordinator = deps?.catalogOperationCoordinator ?? new CatalogOperationCoordinator(); const catalogOperationCoordinator = deps?.catalogOperationCoordinator ?? new CatalogOperationCoordinator();
const runtimeModelCatalog = deps?.runtimeModelCatalog ?? loadRuntimeModelCatalog(config.modelCatalogFile);
const mgr = deps?.mgr ?? new PiProcessManager(config, {
...(deps?.spawnFn ? { spawnFn: deps.spawnFn } : {}),
modelCatalog: runtimeModelCatalog,
});
const metadataGenerationModels = deps?.metadataGenerationModels ?? loadMetadataGenerationModels({ const metadataGenerationModels = deps?.metadataGenerationModels ?? loadMetadataGenerationModels({
installationFile: config.installationConfigFile, catalogFile: config.modelCatalogFile,
secretsFile: config.secretsFile, secretsFile: config.secretsFile,
}); });
const modelCompleter = deps?.modelCompleter ?? new PythonModelCompleter({ const modelCompleter = deps?.modelCompleter ?? new PythonModelCompleter({
@@ -237,6 +245,7 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
?? createProductionWorkspaceDiagnoser(config.workspaceDiagnosticTimeoutMs, undefined, { ?? createProductionWorkspaceDiagnoser(config.workspaceDiagnosticTimeoutMs, undefined, {
internalQdrantUrl: config.internalQdrantUrl, internalQdrantUrl: config.internalQdrantUrl,
internalEmbeddingUrl: config.internalEmbeddingUrl, internalEmbeddingUrl: config.internalEmbeddingUrl,
internalEmbeddingId: config.internalEmbeddingId,
internalEmbeddingModel: config.internalEmbeddingModel, internalEmbeddingModel: config.internalEmbeddingModel,
internalEmbeddingDimensions: config.internalEmbeddingDimensions, internalEmbeddingDimensions: config.internalEmbeddingDimensions,
}); });
@@ -271,7 +280,7 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
const getSettings = async (principal: PrincipalContext): Promise<Settings> => { const getSettings = async (principal: PrincipalContext): Promise<Settings> => {
if (deps?.getSettings) return await deps.getSettings(principal); if (deps?.getSettings) return await deps.getSettings(principal);
const stored = loadSettings(config); const stored = loadSettings(config);
const effective = effectiveSettings(config, stored); const effective = effectiveSettings(config, stored, runtimeModelCatalog);
// In the registry system the legacy `harness/workspaces/*.yaml` default is obsolete: when no // In the registry system the legacy `harness/workspaces/*.yaml` default is obsolete: when no
// installation workspace is pinned, default to the first active registry workspace. // installation workspace is pinned, default to the first active registry workspace.
if (!stored.workspace) { if (!stored.workspace) {
@@ -284,7 +293,9 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
} }
return effective; return effective;
}; };
const piManagement = deps?.piManagement ?? createPiManagement(config, { listModels }); const piManagement = deps?.piManagement ?? createPiManagement(config, {
modelCatalog: runtimeModelCatalog,
});
const maintenanceBarrier = deps?.maintenanceBarrier ?? new MaintenanceBarrier(config.maintenanceFile); const maintenanceBarrier = deps?.maintenanceBarrier ?? new MaintenanceBarrier(config.maintenanceFile);
const localRegistryResolver = deps?.localUserRegistry === undefined const localRegistryResolver = deps?.localUserRegistry === undefined
@@ -408,6 +419,7 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
dwhPrecheck: config.dwhPrecheck, dwhPrecheck: config.dwhPrecheck,
legacyWorkspaceMode: config.legacyWorkspaceMode, legacyWorkspaceMode: config.legacyWorkspaceMode,
workspaceRuntimeSupport, workspaceRuntimeSupport,
modelCatalog: runtimeModelCatalog,
maintenanceBarrier, maintenanceBarrier,
effectiveRelationships, effectiveRelationships,
}); });
@@ -439,7 +451,7 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
return maintenanceBarrier.status(); return maintenanceBarrier.status();
}); });
sqlRoutes(app, { tht: tht as ThtRunner, getSettings, workspaceRegistry }); sqlRoutes(app, { tht: tht as ThtRunner, getSettings, workspaceRegistry });
metaRoutes(app, { harnessDir: config.harnessDir, listModels }); metaRoutes(app, { harnessDir: config.harnessDir, modelCatalog: runtimeModelCatalog });
workspaceRoutes(app, { workspaceRoutes(app, {
registry: workspaceRegistry, registry: workspaceRegistry,
config: config.workspaceRegistry, config: config.workspaceRegistry,
@@ -472,7 +484,7 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
worker: descriptionGenerationWorker, worker: descriptionGenerationWorker,
sensitiveDataSuggestionRunner, sensitiveDataSuggestionRunner,
}); });
settingsRoutes(app, { cfg: config, listModels, getSettings }); settingsRoutes(app, { cfg: config, getSettings });
piManagementRoutes(app, { service: piManagement }); piManagementRoutes(app, { service: piManagement });
return app; return app;
+41 -162
View File
@@ -1,63 +1,5 @@
import { import { loadSecretBundle } from "../config/secret-bundle.js";
closeSync, constants, fstatSync, lstatSync, openSync, readFileSync, import { loadRuntimeModelCatalog } from "../models/runtime-model-catalog.js";
type Stats,
} from "node:fs";
import { parseAllDocuments } from "yaml";
import { z } from "zod";
import {
loadSecretBundle,
METADATA_GENERATION_SECRET_KEYS,
} from "../config/secret-bundle.js";
const MAX_INSTALLATION_BYTES = 1024 * 1024;
const RUNTIME_INSTALLATION_FILE = "/run/thothii-installation/thothii-installation.yaml";
const modelId = z.string().regex(/^[a-z][a-z0-9._-]{0,63}$/);
const apiKeyEnvironment = z.enum(METADATA_GENERATION_SECRET_KEYS);
const endpointSchema = z.object({
baseUrl: z.string().min(1).max(2048).refine((value) => {
try {
const url = new URL(value);
return (url.protocol === "http:" || url.protocol === "https:")
&& url.username === "" && url.password === "" && url.search === "" && url.hash === "";
} catch {
return false;
}
}),
apiVersion: z.string().regex(/^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$/).optional(),
}).strict();
const configuredModelSchema = z.object({
id: modelId,
label: z.string().min(1).max(128).refine((value) => value.trim() === value && !/\p{Cc}/u.test(value)),
litellm: z.object({
provider: z.string().regex(/^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$/),
model: z.string().regex(/^[A-Za-z0-9][A-Za-z0-9._:/-]{0,255}$/),
disableThinking: z.literal(true).optional(),
endpoint: endpointSchema.optional(),
}).strict(),
apiKeyEnv: apiKeyEnvironment.optional(),
}).strict().superRefine((value, context) => {
if (value.apiKeyEnv === undefined && value.litellm.endpoint === undefined) {
context.addIssue({
code: z.ZodIssueCode.custom,
path: ["apiKeyEnv"],
message: "keyless models require an explicit endpoint",
});
}
if (value.litellm.disableThinking === true && value.litellm.endpoint === undefined) {
context.addIssue({
code: z.ZodIssueCode.custom,
path: ["litellm", "disableThinking"],
message: "thinking may be disabled only for an explicit endpoint",
});
}
});
const metadataGenerationSchema = z.object({
default: modelId.optional(),
models: z.array(configuredModelSchema).max(64).default([]),
}).strict();
const installationSchema = z.object({
metadataGeneration: metadataGenerationSchema.optional(),
}).passthrough();
export interface MetadataGenerationModelChoice { export interface MetadataGenerationModelChoice {
id: string; id: string;
@@ -86,7 +28,6 @@ export class MetadataGenerationModelUnavailableError extends Error {
} }
} }
/** The complete interface callers need: safe discovery plus fail-closed runtime resolution. */
export interface MetadataGenerationModels { export interface MetadataGenerationModels {
catalog(): MetadataGenerationModelCatalog; catalog(): MetadataGenerationModelCatalog;
resolve(selection: string): ResolvedMetadataGenerationModel; resolve(selection: string): ResolvedMetadataGenerationModel;
@@ -96,140 +37,78 @@ class RestartLoadedMetadataGenerationModels implements MetadataGenerationModels
readonly #models: ReadonlyMap<string, ResolvedMetadataGenerationModel>; readonly #models: ReadonlyMap<string, ResolvedMetadataGenerationModel>;
readonly #catalog: MetadataGenerationModelCatalog; readonly #catalog: MetadataGenerationModelCatalog;
constructor( constructor(models: ReadonlyMap<string, ResolvedMetadataGenerationModel>, defaultModel: string | null) {
models: ReadonlyMap<string, ResolvedMetadataGenerationModel> = new Map(),
defaultModel: string | null = null,
choices: MetadataGenerationModelChoice[] = [],
) {
this.#models = models; this.#models = models;
this.#catalog = { this.#catalog = {
models: choices.map((choice) => ({ ...choice })), models: [...models.values()].map(({ id }) => ({ id, label: id })),
default: defaultModel, default: defaultModel,
}; };
} }
catalog(): MetadataGenerationModelCatalog { catalog(): MetadataGenerationModelCatalog {
return { return { models: this.#catalog.models.map((choice) => ({ ...choice })), default: this.#catalog.default };
models: this.#catalog.models.map((choice) => ({ ...choice })),
default: this.#catalog.default,
};
} }
resolve(selection: string): ResolvedMetadataGenerationModel { resolve(selection: string): ResolvedMetadataGenerationModel {
const model = typeof selection === "string" ? this.#models.get(selection) : undefined; const model = this.#models.get(selection);
if (!model) throw new MetadataGenerationModelUnavailableError(); if (!model) throw new MetadataGenerationModelUnavailableError();
return model; return model;
} }
} }
function invalid(message = "metadata-generation configuration is invalid"): Error { function invalid(message = "metadata-generation runtime catalog is invalid"): Error {
return new Error(message); return new Error(message);
} }
function protectedInstallationStat(file: string, info: Stats): boolean {
const mode = info.mode & 0o777;
if (!info.isFile() || info.isSymbolicLink() || info.nlink !== 1
|| info.size < 1 || info.size > MAX_INSTALLATION_BYTES) return false;
if (file === RUNTIME_INSTALLATION_FILE && info.uid === 0 && mode === 0o444) return true;
return info.uid === (process.getuid?.() ?? info.uid) && (mode === 0o400 || mode === 0o600);
}
function readProtectedInstallation(file: string): string {
let descriptor: number | undefined;
try {
const before = lstatSync(file);
if (!protectedInstallationStat(file, before)) throw new Error("unavailable");
descriptor = openSync(file, constants.O_RDONLY | constants.O_NOFOLLOW);
const opened = fstatSync(descriptor);
if (!protectedInstallationStat(file, opened)
|| before.dev !== opened.dev || before.ino !== opened.ino) throw new Error("unavailable");
const source = readFileSync(descriptor, "utf8");
const after = fstatSync(descriptor);
const current = lstatSync(file);
if (!protectedInstallationStat(file, after) || !protectedInstallationStat(file, current)
|| opened.dev !== after.dev || opened.ino !== after.ino
|| opened.dev !== current.dev || opened.ino !== current.ino) throw new Error("unavailable");
return source;
} finally {
if (descriptor !== undefined) try { closeSync(descriptor); } catch { /* sanitized below */ }
}
}
function readInstallation(file: string): unknown {
try {
const documents = parseAllDocuments(readProtectedInstallation(file), { uniqueKeys: true });
if (documents.length !== 1) throw invalid("metadata-generation installation must contain one YAML document");
const document = documents[0];
if (document.errors.length > 0 || document.warnings.length > 0) {
throw invalid("metadata-generation installation contains invalid YAML");
}
return document.toJSON();
} catch (error) {
if (error instanceof Error && error.message.startsWith("metadata-generation")) throw error;
throw invalid("metadata-generation installation is unavailable");
}
}
export function loadMetadataGenerationModels(options: { export function loadMetadataGenerationModels(options: {
installationFile?: string; catalogFile?: string;
secretsFile?: string; secretsFile?: string;
}): MetadataGenerationModels { }): MetadataGenerationModels {
if (!options.installationFile) return new RestartLoadedMetadataGenerationModels(); const catalog = loadRuntimeModelCatalog(options.catalogFile);
const installation = installationSchema.safeParse(readInstallation(options.installationFile)); const configured = catalog.metadataModels();
if (!installation.success) throw invalid(); if (configured.length === 0) return new RestartLoadedMetadataGenerationModels(new Map(), null);
const configured = installation.data.metadataGeneration;
if (!configured || configured.models.length === 0) {
if (configured?.default !== undefined) throw invalid("metadata-generation default does not identify a configured model");
return new RestartLoadedMetadataGenerationModels();
}
if (!configured.default) throw invalid("metadata-generation default is required when models are configured");
const seen = new Set<string>(); const requiresSecrets = configured.some((model) => model.authentication.mode === "secret_env");
for (const model of configured.models) {
if (seen.has(model.id)) throw invalid(`metadata-generation model id "${model.id}" is duplicated`);
seen.add(model.id);
}
if (!seen.has(configured.default)) {
throw invalid(`metadata-generation default "${configured.default}" is not configured`);
}
const requiresSecrets = configured.models.some((model) => model.apiKeyEnv !== undefined);
let secrets: ReadonlyMap<string, string> = new Map(); let secrets: ReadonlyMap<string, string> = new Map();
if (requiresSecrets) { if (requiresSecrets) {
if (!options.secretsFile) throw invalid("metadata-generation keyed models require THT_SECRETS_FILE"); if (!options.secretsFile) throw invalid("metadata-generation keyed models require THT_SECRETS_FILE");
try { try { secrets = loadSecretBundle(options.secretsFile); }
secrets = loadSecretBundle(options.secretsFile); catch { throw invalid("metadata-generation secrets are unavailable"); }
} catch {
throw invalid("metadata-generation secrets are unavailable");
}
} }
const models = new Map<string, ResolvedMetadataGenerationModel>(); const models = new Map<string, ResolvedMetadataGenerationModel>();
for (const configuredModel of configured.models) { const labels = new Map<string, string>();
for (const configuredModel of configured) {
const adapter = configuredModel.metadataAdapter;
if (!adapter || configuredModel.authentication.mode === "pi_auth") throw invalid();
const apiKeyEnv = configuredModel.authentication.apiKeyEnv;
let apiKey: string | undefined; let apiKey: string | undefined;
if (configuredModel.apiKeyEnv !== undefined) { if (configuredModel.authentication.mode === "secret_env") {
apiKey = secrets.get(configuredModel.apiKeyEnv); if (!apiKeyEnv) throw invalid();
if (!apiKey) { apiKey = secrets.get(apiKeyEnv);
throw invalid(`metadata-generation model "${configuredModel.id}" secret "${configuredModel.apiKeyEnv}" is missing`); if (!apiKey) throw invalid(`metadata-generation model "${configuredModel.id}" secret "${apiKeyEnv}" is missing`);
}
if (apiKey.length > 16 * 1024 || /\s/u.test(apiKey)) { if (apiKey.length > 16 * 1024 || /\s/u.test(apiKey)) {
throw invalid(`metadata-generation model "${configuredModel.id}" secret "${configuredModel.apiKeyEnv}" is unusable`); throw invalid(`metadata-generation model "${configuredModel.id}" secret "${apiKeyEnv}" is unusable`);
} }
} }
labels.set(configuredModel.id, configuredModel.label);
models.set(configuredModel.id, Object.freeze({ models.set(configuredModel.id, Object.freeze({
id: configuredModel.id, id: configuredModel.id,
provider: configuredModel.litellm.provider, provider: adapter.litellmProvider,
model: configuredModel.litellm.model, model: configuredModel.upstreamModel,
...(configuredModel.litellm.disableThinking === true ? { disableThinking: true as const } : {}), ...(configuredModel.metadataGeneration?.disableThinking === true
...(configuredModel.litellm.endpoint === undefined ? { disableThinking: true as const } : {}),
? {} ...(configuredModel.endpoint ? { endpoint: Object.freeze({ ...configuredModel.endpoint }) } : {}),
: { endpoint: Object.freeze({ ...configuredModel.litellm.endpoint }) }), ...(apiKeyEnv ? { apiKeyEnv, apiKey } : {}),
...(configuredModel.apiKeyEnv === undefined
? {}
: { apiKeyEnv: configuredModel.apiKeyEnv, apiKey }),
})); }));
} }
return new RestartLoadedMetadataGenerationModels( const result = new RestartLoadedMetadataGenerationModels(models, catalog.defaultMetadataGeneration);
models, const safe = result.catalog();
configured.default, return {
configured.models.map(({ id, label }) => ({ id, label })), catalog: () => ({
); default: safe.default,
models: safe.models.map((choice) => ({ ...choice, label: labels.get(choice.id) ?? choice.id })),
}),
resolve: (selection) => result.resolve(selection),
};
} }
+2
View File
@@ -12,6 +12,7 @@ import * as sensitiveDataFlagMigration from "./migrations/006_sensitive_data_fla
import * as sensitiveDataSuggestionRunsMigration from "./migrations/007_sensitive_data_suggestion_runs.js"; import * as sensitiveDataSuggestionRunsMigration from "./migrations/007_sensitive_data_suggestion_runs.js";
import * as catalogLogicalRelationshipsMigration from "./migrations/008_catalog_logical_relationships.js"; import * as catalogLogicalRelationshipsMigration from "./migrations/008_catalog_logical_relationships.js";
import * as aiTokenUsageMigration from "./migrations/009_ai_token_usage.js"; import * as aiTokenUsageMigration from "./migrations/009_ai_token_usage.js";
import * as canonicalModelIdsMigration from "./migrations/010_canonical_model_ids.js";
const connectionString = process.env.THT_CATALOG_MIGRATOR_DATABASE_URL; const connectionString = process.env.THT_CATALOG_MIGRATOR_DATABASE_URL;
const host = process.env.THT_CATALOG_DB_HOST; const host = process.env.THT_CATALOG_DB_HOST;
@@ -46,6 +47,7 @@ const provider: MigrationProvider = {
"007_sensitive_data_suggestion_runs": sensitiveDataSuggestionRunsMigration, "007_sensitive_data_suggestion_runs": sensitiveDataSuggestionRunsMigration,
"008_catalog_logical_relationships": catalogLogicalRelationshipsMigration, "008_catalog_logical_relationships": catalogLogicalRelationshipsMigration,
"009_ai_token_usage": aiTokenUsageMigration, "009_ai_token_usage": aiTokenUsageMigration,
"010_canonical_model_ids": canonicalModelIdsMigration,
}; };
}, },
}; };
@@ -0,0 +1,27 @@
import { sql, type Kysely } from "kysely";
import type { CatalogDatabase } from "../repository.js";
const canonicalModelPattern = "^[a-z][a-z0-9._-]{0,63}/[A-Za-z0-9][A-Za-z0-9._:-]{0,255}$";
const legacyModelPattern = "^[a-z][a-z0-9._-]{0,63}$";
export async function up(db: Kysely<CatalogDatabase>): Promise<void> {
await sql.raw(`alter table description_generation_runs
drop constraint description_generation_runs_model_id_check,
add constraint description_generation_runs_model_id_check
check (model_id ~ '${canonicalModelPattern}')`).execute(db);
await sql.raw(`alter table sensitive_data_suggestion_runs
drop constraint sensitive_data_suggestion_runs_model_id_check,
add constraint sensitive_data_suggestion_runs_model_id_check
check (model_id ~ '${canonicalModelPattern}')`).execute(db);
}
export async function down(db: Kysely<CatalogDatabase>): Promise<void> {
await sql.raw(`alter table sensitive_data_suggestion_runs
drop constraint sensitive_data_suggestion_runs_model_id_check,
add constraint sensitive_data_suggestion_runs_model_id_check
check (model_id ~ '${legacyModelPattern}')`).execute(db);
await sql.raw(`alter table description_generation_runs
drop constraint description_generation_runs_model_id_check,
add constraint description_generation_runs_model_id_check
check (model_id ~ '${legacyModelPattern}')`).execute(db);
}
+32 -2
View File
@@ -32,6 +32,7 @@ export interface AppConfig {
piManagementTimeoutMs: number; piManagementTimeoutMs: number;
secretsFile?: string; secretsFile?: string;
installationConfigFile?: string; installationConfigFile?: string;
modelCatalogFile?: string;
piAuthFile?: string; piAuthFile?: string;
secretFiles: Readonly<Record<string, string | undefined>>; secretFiles: Readonly<Record<string, string | undefined>>;
modelApiKeyFile?: string; modelApiKeyFile?: string;
@@ -50,6 +51,7 @@ export interface AppConfig {
workspaceSecretRuntimeRoot: string; workspaceSecretRuntimeRoot: string;
internalQdrantUrl: string; internalQdrantUrl: string;
internalEmbeddingUrl: string; internalEmbeddingUrl: string;
internalEmbeddingId: string;
internalEmbeddingModel: string; internalEmbeddingModel: string;
internalEmbeddingDimensions: number; internalEmbeddingDimensions: number;
} }
@@ -189,6 +191,21 @@ function positiveDimension(value: string | undefined, fallback: number): number
return parsed; return parsed;
} }
function internalEmbeddingIdentity(
identityValue: string | undefined,
modelValue: string | undefined,
): { id: string; model: string } {
const id = identityValue ?? "ollama/qwen3-embedding:0.6b";
if (!/^ollama\/[A-Za-z0-9][A-Za-z0-9._:-]{0,255}$/.test(id)) {
throw new Error("internal embedding identity configuration is invalid");
}
const model = id.slice(id.indexOf("/") + 1);
if (modelValue !== undefined && modelValue !== model) {
throw new Error("internal embedding model does not match its canonical identity");
}
return { id, model };
}
function catalogDatabase(env: Record<string, string | undefined>): CatalogConnectionConfig | undefined { function catalogDatabase(env: Record<string, string | undefined>): CatalogConnectionConfig | undefined {
const value = env.THT_CATALOG_DATABASE_URL; const value = env.THT_CATALOG_DATABASE_URL;
if (value !== undefined) { if (value !== undefined) {
@@ -330,6 +347,13 @@ export function loadConfig(
|| installationConfigFile.includes("\0") || installationConfigFile.includes("\0")
|| !path.isAbsolute(installationConfigFile) || !path.isAbsolute(installationConfigFile)
)) throw new Error("installation configuration is invalid"); )) throw new Error("installation configuration is invalid");
const modelCatalogFile = env.THT_MODEL_CATALOG_FILE;
if (modelCatalogFile !== undefined && (
modelCatalogFile.trim() !== modelCatalogFile
|| modelCatalogFile.length === 0
|| modelCatalogFile.includes("\0")
|| !path.isAbsolute(modelCatalogFile)
)) throw new Error("runtime model catalog configuration is invalid");
const piAuthFile = env.THT_PI_AUTH_FILE; const piAuthFile = env.THT_PI_AUTH_FILE;
if (piAuthFile !== undefined && ( if (piAuthFile !== undefined && (
piAuthFile.trim() !== piAuthFile || piAuthFile.length === 0 || piAuthFile.includes("\0") piAuthFile.trim() !== piAuthFile || piAuthFile.length === 0 || piAuthFile.includes("\0")
@@ -391,6 +415,10 @@ export function loadConfig(
"internal embedding URL", "internal embedding URL",
["embedding", "localhost"], ["embedding", "localhost"],
); );
const internalEmbedding = internalEmbeddingIdentity(
env.THT_INTERNAL_EMBEDDING_ID,
env.THT_INTERNAL_EMBEDDING_MODEL,
);
return { return {
host: env.HOST ?? "127.0.0.1", host: env.HOST ?? "127.0.0.1",
port: Number(env.PORT ?? 8787), port: Number(env.PORT ?? 8787),
@@ -404,7 +432,7 @@ export function loadConfig(
publicExposure, publicExposure,
sessionStorage, sessionStorage,
catalogDatabase: catalogDatabase(env), catalogDatabase: catalogDatabase(env),
defaults: { provider: env.PI_PROVIDER, model: env.PI_MODEL, thinking: env.PI_THINKING }, defaults: { thinking: env.PI_THINKING },
maxPiProcesses: Number(env.MAX_PI_PROCESSES ?? 4), maxPiProcesses: Number(env.MAX_PI_PROCESSES ?? 4),
settingsFile, settingsFile,
maintenanceFile: env.THT_MAINTENANCE_FILE ?? path.join(path.dirname(settingsFile), "maintenance.json"), maintenanceFile: env.THT_MAINTENANCE_FILE ?? path.join(path.dirname(settingsFile), "maintenance.json"),
@@ -413,6 +441,7 @@ export function loadConfig(
piManagementTimeoutMs: piManagementTimeout(env.PI_MANAGEMENT_TIMEOUT_MS), piManagementTimeoutMs: piManagementTimeout(env.PI_MANAGEMENT_TIMEOUT_MS),
secretsFile, secretsFile,
installationConfigFile, installationConfigFile,
modelCatalogFile,
piAuthFile, piAuthFile,
secretFiles, secretFiles,
modelApiKeyFile, modelApiKeyFile,
@@ -425,7 +454,8 @@ export function loadConfig(
workspaceSecretRuntimeRoot, workspaceSecretRuntimeRoot,
internalQdrantUrl, internalQdrantUrl,
internalEmbeddingUrl, internalEmbeddingUrl,
internalEmbeddingModel: env.THT_INTERNAL_EMBEDDING_MODEL ?? "qwen3-embedding:0.6b", internalEmbeddingId: internalEmbedding.id,
internalEmbeddingModel: internalEmbedding.model,
internalEmbeddingDimensions: positiveDimension(env.THT_INTERNAL_EMBEDDING_DIMENSIONS, 1024), internalEmbeddingDimensions: positiveDimension(env.THT_INTERNAL_EMBEDDING_DIMENSIONS, 1024),
}; };
} }
+1 -1
View File
@@ -8,7 +8,7 @@ import {
isUsableAuthenticationSecret, isUsableAuthenticationSecret,
} from "../auth/secret-policy.js"; } from "../auth/secret-policy.js";
/** Credential names that metadata-generation model entries may reference. */ /** Credential names that Installation Model Catalog providers may reference. */
export const METADATA_GENERATION_SECRET_KEYS = Object.freeze([ export const METADATA_GENERATION_SECRET_KEYS = Object.freeze([
"THT_METADATA_API_KEY", "ANTHROPIC_API_KEY", "AZURE_API_KEY", "GEMINI_API_KEY", "THT_METADATA_API_KEY", "ANTHROPIC_API_KEY", "AZURE_API_KEY", "GEMINI_API_KEY",
"DEEPSEEK_API_KEY", "OPENAI_API_KEY", "OPENROUTER_API_KEY", "ZAI_API_KEY", "DEEPSEEK_API_KEY", "OPENAI_API_KEY", "OPENROUTER_API_KEY", "ZAI_API_KEY",
+161
View File
@@ -0,0 +1,161 @@
import {
closeSync, constants, fstatSync, lstatSync, openSync, readFileSync, type Stats,
} from "node:fs";
import { z } from "zod";
const MAX_CATALOG_BYTES = 1024 * 1024;
const RUNTIME_CATALOG_FILE = "/run/thothii-model-catalog/catalog.json";
const canonicalId = z.string().regex(/^[a-z][a-z0-9._-]{0,63}\/[A-Za-z0-9][A-Za-z0-9._:-]{0,255}$/);
const secretBundleKey = /^[A-Z][A-Z0-9_]{0,63}$/;
const endpointSchema = z.object({
baseUrl: z.string().url(),
apiVersion: z.string().optional(),
}).strict();
const authenticationSchema = z.object({
mode: z.enum(["secret_env", "pi_auth", "none"]),
apiKeyEnv: z.string().optional(),
}).strict();
const runtimeModelSchema = z.object({
id: canonicalId,
provider: z.string().min(1),
model: z.string().min(1),
label: z.string().min(1),
upstreamModel: z.string().min(1),
endpoint: endpointSchema.optional(),
authentication: authenticationSchema,
sessionAdapter: z.object({ mode: z.enum(["pi_builtin", "openai_compatible"]) }).strict().optional(),
metadataAdapter: z.object({ litellmProvider: z.string().min(1) }).strict().optional(),
session: z.object({
reasoning: z.boolean(),
input: z.array(z.string()).optional(),
cost: z.object({
input: z.number(), output: z.number(), cacheRead: z.number(), cacheWrite: z.number(),
}).strict().optional(),
contextWindow: z.number().int().positive().optional(),
maxTokens: z.number().int().positive().optional(),
compatibility: z.object({
supportsDeveloperRole: z.boolean(),
supportsReasoningEffort: z.boolean(),
supportsStore: z.boolean(),
maxTokensField: z.string().optional(),
}).strict().optional(),
}).strict().optional(),
metadataGeneration: z.object({ disableThinking: z.boolean() }).strict().optional(),
}).strict();
const catalogSchema = z.object({
schemaVersion: z.literal(1),
defaultSession: canonicalId,
defaultMetadataGeneration: canonicalId.optional(),
embedding: z.object({ id: canonicalId, dimensions: z.number().int().positive() }).strict(),
models: z.array(runtimeModelSchema).max(64),
}).strict();
export type RuntimeModel = z.infer<typeof runtimeModelSchema>;
export interface RuntimeModelCatalog {
readonly defaultSession: string | null;
readonly defaultMetadataGeneration: string | null;
readonly embedding: Readonly<{ id: string; dimensions: number }> | null;
sessionModels(): readonly RuntimeModel[];
metadataModels(): readonly RuntimeModel[];
hasSession(id: string): boolean;
}
class RestartLoadedRuntimeModelCatalog implements RuntimeModelCatalog {
readonly defaultSession: string | null;
readonly defaultMetadataGeneration: string | null;
readonly embedding: Readonly<{ id: string; dimensions: number }> | null;
readonly #sessions: readonly RuntimeModel[];
readonly #metadata: readonly RuntimeModel[];
readonly #sessionIds: ReadonlySet<string>;
constructor(catalog?: z.infer<typeof catalogSchema>) {
this.defaultSession = catalog?.defaultSession ?? null;
this.defaultMetadataGeneration = catalog?.defaultMetadataGeneration ?? null;
this.embedding = catalog ? Object.freeze({ ...catalog.embedding }) : null;
this.#sessions = Object.freeze((catalog?.models ?? []).filter((model) => model.session !== undefined));
this.#metadata = Object.freeze((catalog?.models ?? []).filter((model) => model.metadataGeneration !== undefined));
this.#sessionIds = new Set(this.#sessions.map((model) => model.id));
}
sessionModels(): readonly RuntimeModel[] { return this.#sessions.map((model) => ({ ...model })); }
metadataModels(): readonly RuntimeModel[] { return this.#metadata.map((model) => ({ ...model })); }
hasSession(id: string): boolean { return this.#sessionIds.has(id); }
}
function protectedCatalogStat(file: string, info: Stats): boolean {
const mode = info.mode & 0o777;
if (!info.isFile() || info.isSymbolicLink() || info.nlink !== 1
|| info.size < 1 || info.size > MAX_CATALOG_BYTES) return false;
if (file === RUNTIME_CATALOG_FILE && info.uid === 0 && (mode === 0o444 || mode === 0o644)) return true;
return info.uid === (process.getuid?.() ?? info.uid) && (mode === 0o400 || mode === 0o600 || mode === 0o644);
}
function readProtectedCatalog(file: string): unknown {
let descriptor: number | undefined;
try {
const before = lstatSync(file);
if (!protectedCatalogStat(file, before)) throw new Error("runtime model catalog is unavailable");
descriptor = openSync(file, constants.O_RDONLY | constants.O_NOFOLLOW);
const opened = fstatSync(descriptor);
if (!protectedCatalogStat(file, opened)
|| before.dev !== opened.dev || before.ino !== opened.ino) throw new Error("runtime model catalog is unavailable");
const source = readFileSync(descriptor, "utf8");
const after = fstatSync(descriptor);
const current = lstatSync(file);
if (!protectedCatalogStat(file, after) || !protectedCatalogStat(file, current)
|| opened.dev !== after.dev || opened.ino !== after.ino
|| opened.dev !== current.dev || opened.ino !== current.ino) throw new Error("runtime model catalog is unavailable");
return JSON.parse(source);
} catch {
throw new Error("runtime model catalog is unavailable");
} finally {
if (descriptor !== undefined) try { closeSync(descriptor); } catch { /* sanitized above */ }
}
}
export function loadRuntimeModelCatalog(file?: string): RuntimeModelCatalog {
if (!file) return new RestartLoadedRuntimeModelCatalog();
const parsed = catalogSchema.safeParse(readProtectedCatalog(file));
if (!parsed.success) throw new Error("runtime model catalog is invalid");
if (parsed.data.models.some((model) => !validRuntimeModel(model))) {
throw new Error("runtime model catalog is invalid");
}
const ids = new Set(parsed.data.models.map((model) => model.id));
if (ids.size !== parsed.data.models.length) throw new Error("runtime model catalog contains duplicate models");
const sessions = parsed.data.models.filter((model) => model.session !== undefined).map((model) => model.id);
const metadata = parsed.data.models.filter((model) => model.metadataGeneration !== undefined).map((model) => model.id);
if (!sessions.includes(parsed.data.defaultSession)) throw new Error("runtime model catalog session default is invalid");
if ((metadata.length > 0) !== (parsed.data.defaultMetadataGeneration !== undefined)
|| (parsed.data.defaultMetadataGeneration !== undefined
&& !metadata.includes(parsed.data.defaultMetadataGeneration))) {
throw new Error("runtime model catalog metadata default is invalid");
}
return new RestartLoadedRuntimeModelCatalog(parsed.data);
}
function validRuntimeModel(model: RuntimeModel): boolean {
if ((model.session !== undefined) !== (model.sessionAdapter !== undefined)) return false;
if ((model.metadataGeneration !== undefined) !== (model.metadataAdapter !== undefined)) return false;
switch (model.authentication.mode) {
case "secret_env":
return model.authentication.apiKeyEnv !== undefined
&& secretBundleKey.test(model.authentication.apiKeyEnv);
case "pi_auth":
return model.authentication.apiKeyEnv === undefined
&& model.metadataGeneration === undefined
&& model.sessionAdapter?.mode === "pi_builtin";
case "none":
return model.authentication.apiKeyEnv === undefined && model.endpoint !== undefined;
}
}
export function splitCanonicalModelId(id: string): { provider: string; model: string } {
const slash = id.indexOf("/");
if (slash <= 0 || slash === id.length - 1) throw new Error("model identity is invalid");
return { provider: id.slice(0, slash), model: id.slice(slash + 1) };
}
+8 -6
View File
@@ -8,8 +8,8 @@ import { join } from "node:path";
import { loadConfig, type AppConfig } from "./config.js"; import { loadConfig, type AppConfig } from "./config.js";
import { rolesToPermissions } from "./auth/config.js"; import { rolesToPermissions } from "./auth/config.js";
import type { PrincipalContext } from "./auth/principal.js"; import type { PrincipalContext } from "./auth/principal.js";
import { createPiModelLister } from "./pi/list-models.js";
import { createPiManagement } from "./pi/management.js"; import { createPiManagement } from "./pi/management.js";
import { loadRuntimeModelCatalog } from "./models/runtime-model-catalog.js";
import { effectiveSettings } from "./routes/settings.js"; import { effectiveSettings } from "./routes/settings.js";
import { MaintenanceBarrier } from "./runtime/maintenance-gate.js"; import { MaintenanceBarrier } from "./runtime/maintenance-gate.js";
import { loadSettings } from "./settings/settings-store.js"; import { loadSettings } from "./settings/settings-store.js";
@@ -19,7 +19,7 @@ import { WorkspaceSecretStore } from "./workspaces/secret-store.js";
type OperatorAction = "maintenance-activate" | "maintenance-deactivate" | "maintenance-status" type OperatorAction = "maintenance-activate" | "maintenance-deactivate" | "maintenance-status"
| "session-inventory" | "workflow-doctor" | "workspace-integrity" | "session-inventory" | "workflow-doctor" | "workspace-integrity"
| "pi-options" | "pi-test" | "effective-settings"; | "pi-test" | "effective-settings";
const lifecyclePrincipal: PrincipalContext = { const lifecyclePrincipal: PrincipalContext = {
issuer: "tht-operator-command", issuer: "tht-operator-command",
@@ -110,9 +110,11 @@ export async function runOperatorAction(
if (action === "session-inventory") return await sessionInventory(config); if (action === "session-inventory") return await sessionInventory(config);
if (action === "workflow-doctor") return await workflowDiagnostics(config); if (action === "workflow-doctor") return await workflowDiagnostics(config);
if (action === "workspace-integrity") return await workspaceIntegrity(config); if (action === "workspace-integrity") return await workspaceIntegrity(config);
if (action === "effective-settings") return effectiveSettings(config, loadSettings(config)); const modelCatalog = loadRuntimeModelCatalog(config.modelCatalogFile);
const service = createPiManagement(config, { listModels: createPiModelLister(config) }); if (action === "effective-settings") {
if (action === "pi-options") return await service.options(); return effectiveSettings(config, loadSettings(config), modelCatalog);
}
const service = createPiManagement(config, { modelCatalog });
if (action === "pi-test") return await service.test(); if (action === "pi-test") return await service.test();
throw new Error("unsupported operator action"); throw new Error("unsupported operator action");
} }
@@ -121,7 +123,7 @@ async function main(): Promise<void> {
const action = process.argv[2] as OperatorAction | undefined; const action = process.argv[2] as OperatorAction | undefined;
if (!action || ![ if (!action || ![
"maintenance-activate", "maintenance-deactivate", "maintenance-status", "session-inventory", "maintenance-activate", "maintenance-deactivate", "maintenance-status", "session-inventory",
"workflow-doctor", "workspace-integrity", "pi-options", "pi-test", "effective-settings", "workflow-doctor", "workspace-integrity", "pi-test", "effective-settings",
].includes(action)) throw new Error("invalid operator action"); ].includes(action)) throw new Error("invalid operator action");
const result = await runOperatorAction(action, loadConfig(process.env)); const result = await runOperatorAction(action, loadConfig(process.env));
process.stdout.write(`${JSON.stringify(result)}\n`); process.stdout.write(`${JSON.stringify(result)}\n`);
+3 -1
View File
@@ -18,6 +18,8 @@ export interface PiModel {
reasoning: boolean; reasoning: boolean;
} }
export type ListModelsFn = () => Promise<PiModel[]>;
interface Opts { interface Opts {
spawnFn?: ( spawnFn?: (
command: string, command: string,
@@ -36,7 +38,7 @@ interface Opts {
* configured) via an ephemeral `pi --mode rpc` process. Result is cached for * configured) via an ephemeral `pi --mode rpc` process. Result is cached for
* `ttlMs`. The returned function rejects on timeout/error; callers degrade. * `ttlMs`. The returned function rejects on timeout/error; callers degrade.
*/ */
export function createPiModelLister(cfg: AppConfig, opts: Opts = {}): () => Promise<PiModel[]> { export function createPiModelLister(cfg: AppConfig, opts: Opts = {}): ListModelsFn {
const ttlMs = opts.ttlMs ?? 60_000; const ttlMs = opts.ttlMs ?? 60_000;
const now = opts.nowMs ?? (() => Date.now()); const now = opts.nowMs ?? (() => Date.now());
const spawnFn = opts.spawnFn ?? nodeSpawn; const spawnFn = opts.spawnFn ?? nodeSpawn;
+27 -92
View File
@@ -2,12 +2,11 @@ import { execFile as nodeExecFile } from "node:child_process";
import { promisify } from "node:util"; import { promisify } from "node:util";
import type { AppConfig } from "../config.js"; import type { AppConfig } from "../config.js";
import { secretValue } from "../config/secret-bundle.js"; import { secretValue } from "../config/secret-bundle.js";
import { loadSettings, type Settings } from "../settings/settings-store.js";
import { import {
loadSettings, splitCanonicalModelId,
saveSettings, type RuntimeModelCatalog,
type Settings, } from "../models/runtime-model-catalog.js";
} from "../settings/settings-store.js";
import type { PiModel } from "./list-models.js";
import { import {
configuredPiProviderApiKey, configuredPiProviderApiKey,
PI_MANAGED_CONFIG_ERROR_MESSAGE, PI_MANAGED_CONFIG_ERROR_MESSAGE,
@@ -45,13 +44,6 @@ export interface PiStatus {
message?: string; message?: string;
} }
export interface PiOptions {
providers: string[];
models: Array<{ provider: string; id: string }>;
reasoning: PiReasoning[];
checkedAt: string;
}
export interface PiTestResult { export interface PiTestResult {
ready: boolean; ready: boolean;
checkedAt: string; checkedAt: string;
@@ -76,15 +68,13 @@ export type PiExecFile = (
export interface PiManagementService { export interface PiManagementService {
status(): Promise<PiStatus>; status(): Promise<PiStatus>;
options(): Promise<PiOptions>;
configure(value: PiInstallationConfig): Promise<PiInstallationConfig & { updatedAt: string }>;
test(): Promise<PiTestResult>; test(): Promise<PiTestResult>;
logs(): Promise<PiLogs>; logs(): Promise<PiLogs>;
} }
export class PiManagementError extends Error { export class PiManagementError extends Error {
constructor( constructor(
public readonly code: "pi_management_invalid_config" | "pi_management_unavailable" | "pi_management_write_failed", public readonly code: "pi_management_unavailable",
message: string, message: string,
) { ) {
super(message); super(message);
@@ -93,10 +83,9 @@ export class PiManagementError extends Error {
interface PiManagementDeps { interface PiManagementDeps {
execute?: PiExecFile; execute?: PiExecFile;
listModels: () => Promise<PiModel[]>; modelCatalog: RuntimeModelCatalog;
smokeProvider?: PiProviderSmoke; smokeProvider?: PiProviderSmoke;
readSettings?: () => Settings; readSettings?: () => Settings;
saveSettings?: (settings: Settings) => Settings;
readLogs?: () => string | Promise<string>; readLogs?: () => string | Promise<string>;
credentialStatus?: (provider: string | undefined) => PiCredentialStatus; credentialStatus?: (provider: string | undefined) => PiCredentialStatus;
now?: () => Date; now?: () => Date;
@@ -111,54 +100,36 @@ export function createPiManagement(config: AppConfig, deps: PiManagementDeps): P
}; };
const execute = deps.execute ?? defaultExecFile; const execute = deps.execute ?? defaultExecFile;
const readSettings = deps.readSettings ?? (() => loadSettings(config)); const readSettings = deps.readSettings ?? (() => loadSettings(config));
const persistSettings = deps.saveSettings ?? ((settings) => saveSettings(config, settings));
const readLogs = deps.readLogs ?? (() => diagnostics.join("\n")); const readLogs = deps.readLogs ?? (() => diagnostics.join("\n"));
const smokeProvider = deps.smokeProvider ?? createPiProviderSmoke(config); const smokeProvider = deps.smokeProvider ?? createPiProviderSmoke(config, {
modelCatalog: deps.modelCatalog,
});
const credentialStatus = deps.credentialStatus ?? ((provider: string | undefined) => { const credentialStatus = deps.credentialStatus ?? ((provider: string | undefined) => {
try { try {
const model = deps.modelCatalog.defaultSession
? deps.modelCatalog.sessionModels().find((entry) => entry.id === deps.modelCatalog.defaultSession)
: undefined;
const credentialName = model?.authentication.mode === "secret_env"
? model.authentication.apiKeyEnv
: undefined;
const configuredApiKey = configuredPiProviderApiKey(
readConfiguredPiAgentFile("models.json", true),
provider,
) ?? (credentialName ? `$${credentialName}` : undefined);
return piProviderCredentialStatus({ return piProviderCredentialStatus({
provider, provider,
authProviders: loadPiAuthProviders(), authProviders: loadPiAuthProviders(),
resolveCredentialValue: () => secretValue(config, "THT_MODEL_API_KEY"), resolveCredentialValue: () => credentialName
? secretValue(config, credentialName)
: config.modelCatalogFile ? undefined : secretValue(config, "THT_MODEL_API_KEY"),
credentialFile: config.modelApiKeyFile, credentialFile: config.modelApiKeyFile,
configuredApiKey: configuredPiProviderApiKey( configuredApiKey,
readConfiguredPiAgentFile("models.json", true),
provider,
),
}); });
} catch { } catch {
return "missing"; return "missing";
} }
}); });
const closedOptions = async (): Promise<Omit<PiOptions, "checkedAt">> => {
let listed: PiModel[];
try {
listed = await deps.listModels();
} catch (error) {
if (isPiManagedConfigError(error)) {
throw new PiManagementError("pi_management_unavailable", PI_MANAGED_CONFIG_ERROR_MESSAGE);
}
throw new PiManagementError("pi_management_unavailable", "Pi model choices are unavailable");
}
const models: Array<{ provider: string; id: string }> = [];
const providers: string[] = [];
const seenModels = new Set<string>();
const seenProviders = new Set<string>();
for (const model of listed) {
if (!isChoice(model?.provider) || !isChoice(model?.id)) continue;
const key = `${model.provider}\u0000${model.id}`;
if (seenModels.has(key)) continue;
seenModels.add(key);
models.push({ provider: model.provider, id: model.id });
if (!seenProviders.has(model.provider)) {
seenProviders.add(model.provider);
providers.push(model.provider);
}
}
return { providers, models, reasoning: [...REASONING_CHOICES] };
};
const version = async (timeoutMs = config.piManagementTimeoutMs): Promise<string> => { const version = async (timeoutMs = config.piManagementTimeoutMs): Promise<string> => {
let output: { stdout: string; stderr: string }; let output: { stdout: string; stderr: string };
try { try {
@@ -180,12 +151,12 @@ export function createPiManagement(config: AppConfig, deps: PiManagementDeps): P
const installationConfig = (): PiInstallationConfig => { const installationConfig = (): PiInstallationConfig => {
const settings = readSettings(); const settings = readSettings();
const provider = config.defaults.provider ?? settings.provider;
const model = config.defaults.model ?? settings.model;
const reasoning = config.defaults.thinking ?? settings.thinking; const reasoning = config.defaults.thinking ?? settings.thinking;
const selected = deps.modelCatalog.defaultSession
? splitCanonicalModelId(deps.modelCatalog.defaultSession)
: undefined;
return { return {
...(isChoice(provider) ? { provider } : {}), ...(selected ? selected : {}),
...(isChoice(model) ? { model } : {}),
...(isReasoning(reasoning) ? { reasoning } : {}), ...(isReasoning(reasoning) ? { reasoning } : {}),
}; };
}; };
@@ -206,28 +177,6 @@ export function createPiManagement(config: AppConfig, deps: PiManagementDeps): P
} }
}, },
async options(): Promise<PiOptions> {
const choices = await closedOptions();
return { ...choices, checkedAt: now().toISOString() };
},
async configure(value: PiInstallationConfig): Promise<PiInstallationConfig & { updatedAt: string }> {
if (!isInstallationConfig(value)) {
throw new PiManagementError("pi_management_invalid_config", "Pi installation configuration is invalid");
}
const choices = await closedOptions();
if (!choices.models.some((model) => model.provider === value.provider && model.id === value.model)) {
throw new PiManagementError("pi_management_invalid_config", "Pi provider and model must be selected from available choices");
}
try {
persistSettings({ ...readSettings(), provider: value.provider, model: value.model, thinking: value.reasoning });
} catch {
throw new PiManagementError("pi_management_write_failed", "Pi installation configuration could not be saved");
}
addDiagnostic("Pi installation defaults updated");
return { ...value, updatedAt: now().toISOString() };
},
async test(): Promise<PiTestResult> { async test(): Promise<PiTestResult> {
const checkedAt = now().toISOString(); const checkedAt = now().toISOString();
const deadline = Date.now() + config.piManagementTimeoutMs; const deadline = Date.now() + config.piManagementTimeoutMs;
@@ -293,24 +242,10 @@ async function defaultExecFile(command: string, args: string[], options: PiExecF
return { stdout: String(result.stdout), stderr: String(result.stderr) }; return { stdout: String(result.stdout), stderr: String(result.stderr) };
} }
function isChoice(value: unknown): value is string {
return typeof value === "string" && value.length > 0 && value.length <= 128 && value.trim() === value
&& /^[A-Za-z0-9][A-Za-z0-9._/-]*$/u.test(value);
}
function isReasoning(value: unknown): value is PiReasoning { function isReasoning(value: unknown): value is PiReasoning {
return typeof value === "string" && (REASONING_CHOICES as readonly string[]).includes(value); return typeof value === "string" && (REASONING_CHOICES as readonly string[]).includes(value);
} }
function isInstallationConfig(value: unknown): value is Required<PiInstallationConfig> {
if (!value || typeof value !== "object" || Array.isArray(value)) return false;
const candidate = value as Record<string, unknown>;
if (Object.keys(candidate).length !== 3 || Object.keys(candidate).some((key) => !["provider", "model", "reasoning"].includes(key))) {
return false;
}
return isChoice(candidate.provider) && isChoice(candidate.model) && isReasoning(candidate.reasoning);
}
function isTimeout(error: unknown): boolean { function isTimeout(error: unknown): boolean {
return Boolean( return Boolean(
error && typeof error === "object" && ( error && typeof error === "object" && (
+36 -11
View File
@@ -11,6 +11,10 @@ import {
configuredPiProviderApiKey, configuredPiProviderApiKey,
createPiRuntimeAgentSnapshot, createPiRuntimeAgentSnapshot,
} from "./managed-config.js"; } from "./managed-config.js";
import {
loadRuntimeModelCatalog,
type RuntimeModelCatalog,
} from "../models/runtime-model-catalog.js";
export interface SessionRuntime { export interface SessionRuntime {
rpc: RpcClient; rpc: RpcClient;
@@ -42,23 +46,32 @@ export class PiProcessManager {
private runtimes = new Map<string, SessionRuntime>(); private runtimes = new Map<string, SessionRuntime>();
private agentSnapshotCleanups = new WeakMap<ChildProcessWithoutNullStreams, () => void>(); private agentSnapshotCleanups = new WeakMap<ChildProcessWithoutNullStreams, () => void>();
private spawnFn: ( private spawnFn: (
sessionId: string, author: string, provider: string | undefined, principal?: PrincipalContext, sessionId: string, author: string, provider: string | undefined, model: string | undefined,
runtimeConfigPath?: string, principal?: PrincipalContext, runtimeConfigPath?: string,
) => ChildProcessWithoutNullStreams; ) => ChildProcessWithoutNullStreams;
private loadAuthProviders: (agentDir: string) => ReadonlySet<string>; private loadAuthProviders: (agentDir: string) => ReadonlySet<string>;
private modelCatalog: RuntimeModelCatalog;
private modelCatalogConfigured: boolean;
constructor( constructor(
private cfg: AppConfig, private cfg: AppConfig,
opts?: { spawnFn?: SpawnFn; authProviders?: (agentDir: string) => ReadonlySet<string> }, opts?: {
spawnFn?: SpawnFn;
authProviders?: (agentDir: string) => ReadonlySet<string>;
modelCatalog?: RuntimeModelCatalog;
},
) { ) {
this.modelCatalog = opts?.modelCatalog ?? loadRuntimeModelCatalog(cfg.modelCatalogFile);
this.modelCatalogConfigured = cfg.modelCatalogFile !== undefined
|| this.modelCatalog.defaultSession !== null;
this.loadAuthProviders = opts?.authProviders this.loadAuthProviders = opts?.authProviders
?? ((agentDir) => loadPiAuthProviders({ agentDir })); ?? ((agentDir) => loadPiAuthProviders({ agentDir }));
if (opts?.spawnFn) { if (opts?.spawnFn) {
this.spawnFn = (sessionId, author, provider, principal, runtimeConfigPath) => this.spawnFn = (sessionId, author, provider, model, principal, runtimeConfigPath) =>
this.spawnPi(opts.spawnFn!, sessionId, author, provider, principal, runtimeConfigPath); this.spawnPi(opts.spawnFn!, sessionId, author, provider, model, principal, runtimeConfigPath);
} else { } else {
this.spawnFn = (sessionId, author, provider, principal, runtimeConfigPath) => this.spawnFn = (sessionId, author, provider, model, principal, runtimeConfigPath) =>
this.spawnPi(nodeSpawn, sessionId, author, provider, principal, runtimeConfigPath); this.spawnPi(nodeSpawn, sessionId, author, provider, model, principal, runtimeConfigPath);
} }
} }
@@ -71,7 +84,7 @@ export class PiProcessManager {
private spawnPi( private spawnPi(
spawnFn: SpawnFn, sessionId: string, author: string, provider: string | undefined, spawnFn: SpawnFn, sessionId: string, author: string, provider: string | undefined,
principal?: PrincipalContext, runtimeConfigPath?: string, model: string | undefined, principal?: PrincipalContext, runtimeConfigPath?: string,
): ChildProcessWithoutNullStreams { ): ChildProcessWithoutNullStreams {
// This is the final shared boundary for createFor(), spawnFor(), and resume(). Validate // This is the final shared boundary for createFor(), spawnFor(), and resume(). Validate
// before auth-provider inspection, then make Pi consume the exact copied bytes rather than // before auth-provider inspection, then make Pi consume the exact copied bytes rather than
@@ -79,12 +92,23 @@ export class PiProcessManager {
const agent = createPiRuntimeAgentSnapshot(); const agent = createPiRuntimeAgentSnapshot();
let child: ChildProcessWithoutNullStreams | undefined; let child: ChildProcessWithoutNullStreams | undefined;
try { try {
const catalogModel = provider && model
? this.modelCatalog.sessionModels()
.find((entry) => entry.provider === provider && entry.model === model)
: undefined;
const credentialName = catalogModel?.authentication.mode === "secret_env"
? catalogModel.authentication.apiKeyEnv
: undefined;
const projectedApiKey = configuredPiProviderApiKey(agent.models, provider)
?? (credentialName ? `$${credentialName}` : undefined);
const env = buildPiChildEnv({ const env = buildPiChildEnv({
provider, provider,
authProviders: this.loadAuthProviders(agent.agentDir), authProviders: this.loadAuthProviders(agent.agentDir),
credentialValue: secretValue(this.cfg, "THT_MODEL_API_KEY"), credentialValue: credentialName
? secretValue(this.cfg, credentialName)
: this.modelCatalogConfigured ? undefined : secretValue(this.cfg, "THT_MODEL_API_KEY"),
credentialFile: this.cfg.modelApiKeyFile, credentialFile: this.cfg.modelApiKeyFile,
configuredApiKey: configuredPiProviderApiKey(agent.models, provider), configuredApiKey: projectedApiKey,
additions: { THT_SESSION: sessionId, THT_AUTHOR: author }, additions: { THT_SESSION: sessionId, THT_AUTHOR: author },
}); });
env.PI_CODING_AGENT_DIR = agent.agentDir; env.PI_CODING_AGENT_DIR = agent.agentDir;
@@ -162,9 +186,10 @@ export class PiProcessManager {
} }
const author = o.author ?? "dev@local"; const author = o.author ?? "dev@local";
const provider = canonicalPiProvider(o.provider ?? this.cfg.defaults.provider); const provider = canonicalPiProvider(o.provider ?? this.cfg.defaults.provider);
const model = o.model ?? this.cfg.defaults.model;
let child: ChildProcessWithoutNullStreams; let child: ChildProcessWithoutNullStreams;
try { try {
child = this.spawnFn(sessionId, author, provider, o.principal, o.runtimeConfig?.path); child = this.spawnFn(sessionId, author, provider, model, o.principal, o.runtimeConfig?.path);
} catch (error) { } catch (error) {
o.runtimeConfig?.release(); o.runtimeConfig?.release();
throw error; throw error;
+19 -2
View File
@@ -17,6 +17,10 @@ import {
readConfiguredPiAgentFile, readConfiguredPiAgentFile,
validateDeclarativePiConfig, validateDeclarativePiConfig,
} from "./managed-config.js"; } from "./managed-config.js";
import {
loadRuntimeModelCatalog,
type RuntimeModelCatalog,
} from "../models/runtime-model-catalog.js";
const SMOKE_PROMPT = "Provider health check. Reply with exactly OK."; const SMOKE_PROMPT = "Provider health check. Reply with exactly OK.";
const SMOKE_ARGS = [ const SMOKE_ARGS = [
@@ -49,6 +53,7 @@ interface ProviderSmokeOptions {
authProviders?: () => ReadonlySet<string>; authProviders?: () => ReadonlySet<string>;
readAuthStore?: () => string; readAuthStore?: () => string;
readModelsStore?: () => string | undefined; readModelsStore?: () => string | undefined;
modelCatalog?: RuntimeModelCatalog;
} }
export function createPiProviderSmoke( export function createPiProviderSmoke(
@@ -69,12 +74,24 @@ export function createPiProviderSmoke(
const configuredModels = options.readModelsStore const configuredModels = options.readModelsStore
? options.readModelsStore() ? options.readModelsStore()
: readConfiguredPiAgentFile("models.json", true); : readConfiguredPiAgentFile("models.json", true);
const catalog = options.modelCatalog ?? loadRuntimeModelCatalog(config.modelCatalogFile);
const catalogConfigured = config.modelCatalogFile !== undefined
|| catalog.defaultSession !== null;
const catalogModel = catalog.sessionModels()
.find((entry) => entry.provider === canonicalProvider && entry.model === model);
const credentialName = catalogModel?.authentication.mode === "secret_env"
? catalogModel.authentication.apiKeyEnv
: undefined;
const projectedApiKey = configuredPiProviderApiKey(configuredModels, canonicalProvider)
?? (credentialName ? `$${credentialName}` : undefined);
const env = buildPiChildEnv({ const env = buildPiChildEnv({
provider: canonicalProvider, provider: canonicalProvider,
authProviders: configuredAuthProviders, authProviders: configuredAuthProviders,
credentialValue: secretValue(config, "THT_MODEL_API_KEY"), credentialValue: credentialName
? secretValue(config, credentialName)
: catalogConfigured ? undefined : secretValue(config, "THT_MODEL_API_KEY"),
credentialFile: config.modelApiKeyFile, credentialFile: config.modelApiKeyFile,
configuredApiKey: configuredPiProviderApiKey(configuredModels, canonicalProvider), configuredApiKey: projectedApiKey,
}); });
clearPrincipalEnvironment(env); clearPrincipalEnvironment(env);
delete env.THT_DATA_ROOT; delete env.THT_DATA_ROOT;
@@ -32,7 +32,7 @@ import {
} from "../catalog/types.js"; } from "../catalog/types.js";
const idSchema = z.uuid(); const idSchema = z.uuid();
const modelIdSchema = z.string().regex(/^[a-z][a-z0-9._-]{0,63}$/); const modelIdSchema = z.string().regex(/^[a-z][a-z0-9._-]{0,63}\/[A-Za-z0-9][A-Za-z0-9._:-]{0,255}$/);
const selectedTargetIdsSchema = z.array(idSchema).min(1); const selectedTargetIdsSchema = z.array(idSchema).min(1);
const suggestionSchema = z.discriminatedUnion("scope", [ const suggestionSchema = z.discriminatedUnion("scope", [
z.object({ modelId: modelIdSchema, scope: z.literal("all") }).strict(), z.object({ modelId: modelIdSchema, scope: z.literal("all") }).strict(),
+10 -15
View File
@@ -1,10 +1,8 @@
import { readdirSync } from "node:fs"; import { readdirSync } from "node:fs";
import { join } from "node:path"; import { join } from "node:path";
import type { FastifyInstance } from "fastify"; import type { FastifyInstance } from "fastify";
import type { PiModel } from "../pi/list-models.js";
import { isPrincipalContext, requirePermission } from "../auth/authorization.js"; import { isPrincipalContext, requirePermission } from "../auth/authorization.js";
import type { RuntimeModelCatalog } from "../models/runtime-model-catalog.js";
export type ListModelsFn = () => Promise<PiModel[]>;
/** /**
* List YAML workspace configs found in <harnessDir>/workspaces/*.yaml. * List YAML workspace configs found in <harnessDir>/workspaces/*.yaml.
@@ -25,20 +23,17 @@ export function listWorkspaces(harnessDir: string): { name: string; file: string
export function metaRoutes( export function metaRoutes(
app: FastifyInstance, app: FastifyInstance,
deps: { harnessDir: string; listModels?: ListModelsFn }, deps: { harnessDir: string; modelCatalog: RuntimeModelCatalog },
): void { ): void {
app.get("/models", async (request, reply) => { app.get("/models", async (request, reply) => {
if (!isPrincipalContext(requirePermission(request, reply, "session.use"))) return reply; if (!isPrincipalContext(requirePermission(request, reply, "session.use"))) return reply;
const fn = deps.listModels ?? (async () => []); return {
try { models: deps.modelCatalog.sessionModels().map((entry) => ({
return { models: await fn() }; provider: entry.provider,
} catch (error) { id: entry.model,
app.log.warn({ name: entry.label,
component: "pi-model-list", reasoning: entry.session?.reasoning ?? false,
errorType: error instanceof Error ? error.name : typeof error, })),
}, "Pi model listing failed"); };
// Graceful fallback: Pi may not be running; don't crash the server.
return { models: [] as PiModel[] };
}
}); });
} }
+1 -9
View File
@@ -11,13 +11,6 @@ export function piManagementRoutes(
deps: { service: PiManagementService }, deps: { service: PiManagementService },
): void { ): void {
app.get("/pi-management/status", async (request, reply) => run(request, reply, deps, () => deps.service.status())); app.get("/pi-management/status", async (request, reply) => run(request, reply, deps, () => deps.service.status()));
app.get("/pi-management/options", async (request, reply) => run(request, reply, deps, () => deps.service.options()));
app.put("/pi-management/config", async (request, reply) => run(
request,
reply,
deps,
() => deps.service.configure((request.body ?? {}) as Record<string, unknown>),
));
app.post("/pi-management/test", async (request, reply) => run(request, reply, deps, () => deps.service.test())); app.post("/pi-management/test", async (request, reply) => run(request, reply, deps, () => deps.service.test()));
app.get("/pi-management/logs", async (request, reply) => run(request, reply, deps, () => deps.service.logs())); app.get("/pi-management/logs", async (request, reply) => run(request, reply, deps, () => deps.service.logs()));
} }
@@ -38,8 +31,7 @@ async function run<T>(
return await action(); return await action();
} catch (error) { } catch (error) {
if (error instanceof PiManagementError) { if (error instanceof PiManagementError) {
const statusCode = error.code === "pi_management_invalid_config" ? 400 : 503; return reply.code(503).send({ code: error.code, error: error.message });
return reply.code(statusCode).send({ code: error.code, error: error.message });
} }
return reply.code(503).send({ code: "pi_management_unavailable", error: "Pi management is unavailable" }); return reply.code(503).send({ code: "pi_management_unavailable", error: "Pi management is unavailable" });
} }
+18 -9
View File
@@ -6,12 +6,13 @@ import type { Settings } from "../settings/settings-store.js";
import { getPrincipal } from "../auth/auth.js"; import { getPrincipal } from "../auth/auth.js";
import type { PrincipalContext } from "../auth/principal.js"; import type { PrincipalContext } from "../auth/principal.js";
import type { ReadinessManager } from "../runtime/readiness-manager.js"; import type { ReadinessManager } from "../runtime/readiness-manager.js";
import type { ListModelsFn } from "./meta.js"; import type { ListModelsFn } from "../pi/list-models.js";
import type { WorkspaceRegistry } from "../workspaces/registry.js"; import type { WorkspaceRegistry } from "../workspaces/registry.js";
import { validateOperationalWorkspace, type WorkspaceDescriptor } from "../workspaces/schema.js"; import { validateOperationalWorkspace, type WorkspaceDescriptor } from "../workspaces/schema.js";
import type { MaintenanceBarrier } from "../runtime/maintenance-gate.js"; import type { MaintenanceBarrier } from "../runtime/maintenance-gate.js";
import { hasPermission, isPrincipalContext, requirePermission } from "../auth/authorization.js"; import { hasPermission, isPrincipalContext, requirePermission } from "../auth/authorization.js";
import type { EffectiveRelationshipSnapshotProvider } from "../catalog/effective-relationship-snapshot.js"; import type { EffectiveRelationshipSnapshotProvider } from "../catalog/effective-relationship-snapshot.js";
import { splitCanonicalModelId, type RuntimeModelCatalog } from "../models/runtime-model-catalog.js";
const BOOTSTRAP_FAILURE_MESSAGE = const BOOTSTRAP_FAILURE_MESSAGE =
"Session startup failed. Check configuration and connectivity, then Resume the session."; "Session startup failed. Check configuration and connectivity, then Resume the session.";
@@ -43,6 +44,7 @@ export function sessionRoutes(
maintenanceBarrier: MaintenanceBarrier; maintenanceBarrier: MaintenanceBarrier;
/** Optional only for narrow route-test stubs and installations without a Catalog database. */ /** Optional only for narrow route-test stubs and installations without a Catalog database. */
effectiveRelationships?: EffectiveRelationshipSnapshotProvider; effectiveRelationships?: EffectiveRelationshipSnapshotProvider;
modelCatalog: RuntimeModelCatalog;
}, },
) { ) {
const lifecycleTails = new Map<string, Promise<void>>(); const lifecycleTails = new Map<string, Promise<void>>();
@@ -358,7 +360,6 @@ export function sessionRoutes(
let workspaceId: string | undefined; let workspaceId: string | undefined;
let workspaceRevision: string | undefined; let workspaceRevision: string | undefined;
let workspaceDescriptor: WorkspaceDescriptor | undefined; let workspaceDescriptor: WorkspaceDescriptor | undefined;
let allowedModels: readonly string[] | undefined;
if (requestedWorkspaceId) { if (requestedWorkspaceId) {
try { try {
const registry = d.workspaceRegistry as Partial<WorkspaceRegistry>; const registry = d.workspaceRegistry as Partial<WorkspaceRegistry>;
@@ -378,7 +379,6 @@ export function sessionRoutes(
workspaceId = resolved.revision.id; workspaceId = resolved.revision.id;
workspaceRevision = resolved.revision.commit; workspaceRevision = resolved.revision.commit;
workspaceDescriptor = resolved.workspace; workspaceDescriptor = resolved.workspace;
allowedModels = resolved.workspace.llm_policy.allowed;
} catch { } catch {
return reply.code(409).send({ return reply.code(409).send({
error: WORKSPACE_REVISION_UNAVAILABLE_MESSAGE, error: WORKSPACE_REVISION_UNAVAILABLE_MESSAGE,
@@ -386,12 +386,17 @@ export function sessionRoutes(
}); });
} }
} }
const provider = b.provider ?? s.provider; const requestedCanonical = b.provider && b.model ? `${b.provider}/${b.model}` : undefined;
const model = b.model ?? s.model; let selectedCanonical = requestedCanonical ?? d.modelCatalog.defaultSession;
let modelWarning: string | undefined;
if (selectedCanonical && d.modelCatalog.defaultSession && !d.modelCatalog.hasSession(selectedCanonical)) {
selectedCanonical = d.modelCatalog.defaultSession;
modelWarning = `Configured model ${requestedCanonical ?? "selection"} is unavailable; using ${selectedCanonical}.`;
}
const selected = selectedCanonical ? splitCanonicalModelId(selectedCanonical) : undefined;
const provider = selected?.provider ?? b.provider;
const model = selected?.model ?? b.model;
const thinking = b.thinking ?? s.thinking; const thinking = b.thinking ?? s.thinking;
if (allowedModels && provider && model && !allowedModels.includes(`${provider}/${model}`)) {
return reply.code(400).send({ error: "Selected model is not allowed by this workspace." });
}
// A persisted session is resumable without keeping Pi alive. New work replaces every // A persisted session is resumable without keeping Pi alive. New work replaces every
// runtime owned by this principal, while runtimes belonging to other users remain intact. // runtime owned by this principal, while runtimes belonging to other users remain intact.
// Optional chaining preserves the deliberately narrow manager stubs used by route tests. // Optional chaining preserves the deliberately narrow manager stubs used by route tests.
@@ -490,7 +495,7 @@ export function sessionRoutes(
), ),
() => d.mgr.start(id, rt, runtimeOptions), () => d.mgr.start(id, rt, runtimeOptions),
); );
return { id }; return { id, ...(modelWarning ? { warning: modelWarning } : {}) };
} finally { } finally {
if (revisionLease && !manifestPersisted) { if (revisionLease && !manifestPersisted) {
await revisionLease.abort().catch((error: unknown) => { await revisionLease.abort().catch((error: unknown) => {
@@ -598,6 +603,10 @@ export function sessionRoutes(
provider?: string; model?: string; thinking?: string; provider?: string; model?: string; thinking?: string;
workspace_id?: string; workspace_revision?: string; workspace_id?: string; workspace_revision?: string;
}; };
const savedCanonical = saved.provider && saved.model ? `${saved.provider}/${saved.model}` : "";
if (d.modelCatalog.defaultSession && (!savedCanonical || !d.modelCatalog.hasSession(savedCanonical))) {
return reply.code(503).send({ error: MODEL_UNAVAILABLE_MESSAGE, code: "model_unavailable" });
}
let workspaceConfigPath: string; let workspaceConfigPath: string;
let workspaceDescriptor: WorkspaceDescriptor | undefined; let workspaceDescriptor: WorkspaceDescriptor | undefined;
try { try {
+16 -22
View File
@@ -1,17 +1,27 @@
import type { FastifyInstance } from "fastify"; import type { FastifyInstance } from "fastify";
import type { AppConfig } from "../config.js"; import type { AppConfig } from "../config.js";
import type { Settings } from "../settings/settings-store.js"; import type { Settings } from "../settings/settings-store.js";
import { listWorkspaces, type ListModelsFn } from "./meta.js"; import { listWorkspaces } from "./meta.js";
import type { PrincipalContext } from "../auth/principal.js"; import type { PrincipalContext } from "../auth/principal.js";
import { isPrincipalContext, requirePermission } from "../auth/authorization.js"; import { isPrincipalContext, requirePermission } from "../auth/authorization.js";
import {
splitCanonicalModelId,
type RuntimeModelCatalog,
} from "../models/runtime-model-catalog.js";
/** Merge stored settings over env/first-workspace defaults. */ /** Merge only workspace and runtime-thinking preferences; model defaults belong to modelCatalog. */
export function effectiveSettings(cfg: AppConfig, stored: Settings): Settings { export function effectiveSettings(
cfg: AppConfig,
stored: Settings,
modelCatalog?: RuntimeModelCatalog,
): Settings {
const workspaces = listWorkspaces(cfg.harnessDir); const workspaces = listWorkspaces(cfg.harnessDir);
const selected = modelCatalog?.defaultSession
? splitCanonicalModelId(modelCatalog.defaultSession)
: undefined;
return { return {
workspace: stored.workspace ?? workspaces[0]?.name, workspace: stored.workspace ?? workspaces[0]?.name,
provider: cfg.defaults.provider ?? stored.provider, ...(selected ?? {}),
model: cfg.defaults.model ?? stored.model,
thinking: cfg.defaults.thinking ?? stored.thinking, thinking: cfg.defaults.thinking ?? stored.thinking,
}; };
} }
@@ -19,7 +29,7 @@ export function effectiveSettings(cfg: AppConfig, stored: Settings): Settings {
export function settingsRoutes( export function settingsRoutes(
app: FastifyInstance, app: FastifyInstance,
deps: { deps: {
cfg: AppConfig; listModels: ListModelsFn; cfg: AppConfig;
getSettings: (principal: PrincipalContext) => Promise<Settings>; getSettings: (principal: PrincipalContext) => Promise<Settings>;
}, },
): void { ): void {
@@ -37,22 +47,6 @@ export function settingsRoutes(
const principal = requirePermission(req, reply, "settings.manage"); const principal = requirePermission(req, reply, "settings.manage");
if (!isPrincipalContext(principal)) return principal; if (!isPrincipalContext(principal)) return principal;
const b = (req.body ?? {}) as Settings; const b = (req.body ?? {}) as Settings;
if (b.model) {
let available: { provider: string; id: string }[] = [];
try {
available = await deps.listModels();
} catch {
available = [];
}
// Only validate when Pi gave us a non-empty list; otherwise allow (degraded).
if (available.length > 0 && !available.some(
(candidate) => candidate.provider === b.provider && candidate.id === b.model,
)) {
return reply.code(400).send({
error: `Unknown model: ${b.provider ?? "unknown"}/${b.model}`,
});
}
}
try { try {
// Retain this endpoint as a validating compatibility surface for older clients, but do // Retain this endpoint as a validating compatibility surface for older clients, but do
// not write anonymous users' choices to shared server storage. // not write anonymous users' choices to shared server storage.
+8 -1
View File
@@ -13,6 +13,7 @@ import type { AppConfig } from "../config.js";
export interface Settings { export interface Settings {
workspace?: string; workspace?: string;
/** Legacy input fields are ignored when loading/evaluating installation settings. */
provider?: string; provider?: string;
model?: string; model?: string;
thinking?: string; thinking?: string;
@@ -37,7 +38,13 @@ export function loadSettings(cfg: AppConfig): Settings {
try { try {
const raw = readFileSync(cfg.settingsFile, "utf8"); const raw = readFileSync(cfg.settingsFile, "utf8");
const parsed = JSON.parse(raw); const parsed = JSON.parse(raw);
if (parsed && typeof parsed === "object") return parsed as Settings; if (parsed && typeof parsed === "object") {
const value = parsed as Record<string, unknown>;
return {
...(typeof value.workspace === "string" ? { workspace: value.workspace } : {}),
...(typeof value.thinking === "string" ? { thinking: value.thinking } : {}),
};
}
return {}; return {};
} catch { } catch {
return {}; return {};
+5 -1
View File
@@ -598,7 +598,11 @@ export class ThtRunner {
} catch { } catch {
return { ok: false, code: "workspace_not_activatable" }; return { ok: false, code: "workspace_not_activatable" };
} }
const collection = descriptor.semantic_index.vector_store; const collection = {
collection: descriptor.workspace.id,
dimensions: this.cfg.semanticRuntime.internalEmbeddingDimensions,
distance: "cosine" as const,
};
const controller = new AbortController(); const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), Math.max(1, timeoutSec) * 1000); const timer = setTimeout(() => controller.abort(), Math.max(1, timeoutSec) * 1000);
try { try {
+1
View File
@@ -238,6 +238,7 @@ function createProductionService(): WorkspacePreprocessingService {
}); });
return new WorkspacePreprocessingService({ return new WorkspacePreprocessingService({
dataRoot: config.dataRoot ?? "/data", dataRoot: config.dataRoot ?? "/data",
embeddingDimensions: config.internalEmbeddingDimensions,
httpPrivateHostAllowlist: (process.env.THT_EVIDENCE_PRIVATE_HOST_ALLOWLIST ?? "") httpPrivateHostAllowlist: (process.env.THT_EVIDENCE_PRIVATE_HOST_ALLOWLIST ?? "")
.split(",").map((value) => value.trim()).filter((value) => value.length > 0), .split(",").map((value) => value.trim()).filter((value) => value.length > 0),
acquireActiveRuntime: async (workspaceId) => { acquireActiveRuntime: async (workspaceId) => {
+4 -4
View File
@@ -59,12 +59,12 @@ function safeSecretFilePath(path: string, secretRoots: readonly string[]): strin
function requireSupportedDescriptor(workspace: unknown): void { function requireSupportedDescriptor(workspace: unknown): void {
if (typeof workspace !== "object" || workspace === null) { if (typeof workspace !== "object" || workspace === null) {
throw new Error("Workspace bindings support only workspace schema version 3"); throw new Error("Workspace bindings support only workspace schema version 4");
} }
const metadata = Reflect.get(workspace, "workspace"); const metadata = Reflect.get(workspace, "workspace");
if (typeof metadata !== "object" || metadata === null if (typeof metadata !== "object" || metadata === null
|| Reflect.get(metadata, "schema_version") !== 3) { || Reflect.get(metadata, "schema_version") !== 4) {
throw new Error("Workspace bindings support only workspace schema version 3"); throw new Error("Workspace bindings support only workspace schema version 4");
} }
} }
@@ -155,7 +155,7 @@ export function resolveEvidenceBinding(
return { values, missing }; return { values, missing };
} }
/** Resolve the complete schema-v3 runtime binding set. */ /** Resolve the complete schema-v4 runtime binding set. */
export function resolveRuntimeBindings( export function resolveRuntimeBindings(
workspace: WorkspaceDescriptor, workspace: WorkspaceDescriptor,
env: NodeJS.ProcessEnv, env: NodeJS.ProcessEnv,
+3 -3
View File
@@ -137,12 +137,12 @@ function evidenceVariables(
function requireSupportedDescriptor(workspace: unknown): void { function requireSupportedDescriptor(workspace: unknown): void {
if (typeof workspace !== "object" || workspace === null) { if (typeof workspace !== "object" || workspace === null) {
throw new Error("Installation contract supports only workspace schema version 3"); throw new Error("Installation contract supports only workspace schema version 4");
} }
const metadata = Reflect.get(workspace, "workspace"); const metadata = Reflect.get(workspace, "workspace");
if (typeof metadata !== "object" || metadata === null if (typeof metadata !== "object" || metadata === null
|| Reflect.get(metadata, "schema_version") !== 3) { || Reflect.get(metadata, "schema_version") !== 4) {
throw new Error("Installation contract supports only workspace schema version 3"); throw new Error("Installation contract supports only workspace schema version 4");
} }
} }
+11 -9
View File
@@ -406,12 +406,12 @@ function numericBinding(binding: Record<string, string>, name: string): number |
function requireSupportedDescriptor(workspace: unknown): void { function requireSupportedDescriptor(workspace: unknown): void {
if (typeof workspace !== "object" || workspace === null) { if (typeof workspace !== "object" || workspace === null) {
throw new Error("Workspace diagnoser supports only workspace schema version 3"); throw new Error("Workspace diagnoser supports only workspace schema version 4");
} }
const metadata = Reflect.get(workspace, "workspace"); const metadata = Reflect.get(workspace, "workspace");
if (typeof metadata !== "object" || metadata === null if (typeof metadata !== "object" || metadata === null
|| Reflect.get(metadata, "schema_version") !== 3) { || Reflect.get(metadata, "schema_version") !== 4) {
throw new Error("Workspace diagnoser supports only workspace schema version 3"); throw new Error("Workspace diagnoser supports only workspace schema version 4");
} }
} }
@@ -506,11 +506,15 @@ async function diagnoseValidatedWorkspace(
try { try {
const vector = await withTimeout(timeoutMs, (signal) => adapters.inspectQdrant({ const vector = await withTimeout(timeoutMs, (signal) => adapters.inspectQdrant({
baseUrl: semanticRuntime.internalQdrantUrl, baseUrl: semanticRuntime.internalQdrantUrl,
collection: descriptor.semantic_index.vector_store.collection, collection: descriptor.workspace.id,
timeoutMs, timeoutMs,
signal, signal,
})); }));
const expected = descriptor.semantic_index.vector_store; const expected = {
collection: descriptor.workspace.id,
dimensions: semanticRuntime.internalEmbeddingDimensions,
distance: "cosine",
};
if (vector.collection !== expected.collection if (vector.collection !== expected.collection
|| vector.dimensions !== expected.dimensions || vector.dimensions !== expected.dimensions
|| vector.distance !== expected.distance) { || vector.distance !== expected.distance) {
@@ -529,10 +533,8 @@ async function diagnoseValidatedWorkspace(
timeoutMs, timeoutMs,
signal, signal,
})); }));
if (semanticRuntime.internalEmbeddingModel !== descriptor.semantic_index.embedding.model if (!embedding.available
|| semanticRuntime.internalEmbeddingDimensions !== descriptor.semantic_index.embedding.dimensions || embedding.dimensions !== semanticRuntime.internalEmbeddingDimensions) {
|| !embedding.available
|| embedding.dimensions !== descriptor.semantic_index.embedding.dimensions) {
diagnostics.push(diagnosticError("semantic_index_incompatible")); diagnostics.push(diagnosticError("semantic_index_incompatible"));
activatable = false; activatable = false;
} }
+6 -3
View File
@@ -2,7 +2,7 @@ import { createHash } from "node:crypto";
import { normalize } from "node:path"; import { normalize } from "node:path";
export interface CanonicalEffectiveConfig { export interface CanonicalEffectiveConfig {
schemaVersion: 1; schemaVersion: 2;
dwh: CanonicalDwhConfig; dwh: CanonicalDwhConfig;
vector: CanonicalVectorConfig; vector: CanonicalVectorConfig;
embedding: CanonicalEmbeddingConfig; embedding: CanonicalEmbeddingConfig;
@@ -27,6 +27,7 @@ export interface CanonicalVectorConfig {
} }
export interface CanonicalEmbeddingConfig { export interface CanonicalEmbeddingConfig {
id: string;
model: string; model: string;
dimensions: number; dimensions: number;
} }
@@ -137,8 +138,10 @@ function buildEmbeddingConfig(rendered: Record<string, unknown>): CanonicalEmbed
if (!embeddings) { if (!embeddings) {
throw new TypeError("effective config is missing embedding resources"); throw new TypeError("effective config is missing embedding resources");
} }
const model = requireString(embeddings, "model");
return { return {
model: requireString(embeddings, "model"), id: `ollama/${model}`,
model,
dimensions: requireNumber(embeddings, "dimensions"), dimensions: requireNumber(embeddings, "dimensions"),
}; };
} }
@@ -166,7 +169,7 @@ export function buildCanonicalEffectiveConfig(renderedConfig: unknown): Canonica
throw new TypeError("effective config requires a rendered configuration object"); throw new TypeError("effective config requires a rendered configuration object");
} }
return { return {
schemaVersion: 1, schemaVersion: 2,
dwh: buildDwhConfig(rendered), dwh: buildDwhConfig(rendered),
vector: buildVectorConfig(rendered), vector: buildVectorConfig(rendered),
embedding: buildEmbeddingConfig(rendered), embedding: buildEmbeddingConfig(rendered),
@@ -77,6 +77,7 @@ export interface WorkspacePreprocessingServiceDeps {
{ ok: true } | { ok: false; code: "workspace_not_activatable" | "semantic_index_incompatible" } { ok: true } | { ok: false; code: "workspace_not_activatable" | "semantic_index_incompatible" }
>; >;
httpPrivateHostAllowlist?: readonly string[]; httpPrivateHostAllowlist?: readonly string[];
embeddingDimensions?: number;
} }
interface RunScope { interface RunScope {
@@ -118,7 +119,7 @@ export class WorkspacePreprocessingService {
async vectorInspect(options: { workspaceId: string }): Promise<WorkspaceOperationResult> { async vectorInspect(options: { workspaceId: string }): Promise<WorkspaceOperationResult> {
const runtime = await this.deps.acquireActiveRuntime(options.workspaceId); const runtime = await this.deps.acquireActiveRuntime(options.workspaceId);
const collection = runtime.workspace.semantic_index.vector_store.collection; const collection = runtime.workspace.workspace.id;
const res = await fetch(`${runtime.configLease.semanticQdrantUrl}/collections/${encodeURIComponent(collection)}`, { method: "GET" }); const res = await fetch(`${runtime.configLease.semanticQdrantUrl}/collections/${encodeURIComponent(collection)}`, { method: "GET" });
if (!res.ok) return baseResult(runtime, "vector inspect", "failed", "semantic_index_incompatible", { warnings: ["collection unavailable"] }); if (!res.ok) return baseResult(runtime, "vector inspect", "failed", "semantic_index_incompatible", { warnings: ["collection unavailable"] });
const body = await res.json() as any; const body = await res.json() as any;
@@ -132,7 +133,7 @@ export class WorkspacePreprocessingService {
async vectorRebuild(options: { workspaceId: string; collection?: string; confirm?: string; destroy?: boolean }): Promise<WorkspaceOperationResult> { async vectorRebuild(options: { workspaceId: string; collection?: string; confirm?: string; destroy?: boolean }): Promise<WorkspaceOperationResult> {
const runtime = await this.deps.acquireActiveRuntime(options.workspaceId); const runtime = await this.deps.acquireActiveRuntime(options.workspaceId);
const collection = runtime.workspace.semantic_index.vector_store.collection; const collection = runtime.workspace.workspace.id;
if (options.collection !== collection || options.confirm !== collection || options.destroy !== true) { if (options.collection !== collection || options.confirm !== collection || options.destroy !== true) {
return baseResult(runtime, "vector rebuild", "failed", "semantic_index_incompatible", { warnings: ["rebuild requires exact confirmation and --destroy"] }); return baseResult(runtime, "vector rebuild", "failed", "semantic_index_incompatible", { warnings: ["rebuild requires exact confirmation and --destroy"] });
} }
@@ -143,8 +144,8 @@ export class WorkspacePreprocessingService {
const recreated = await reconcileCollection({ const recreated = await reconcileCollection({
baseUrl: runtime.configLease.semanticQdrantUrl, baseUrl: runtime.configLease.semanticQdrantUrl,
collection, collection,
dimensions: runtime.workspace.semantic_index.vector_store.dimensions, dimensions: this.deps.embeddingDimensions ?? 1024,
distance: runtime.workspace.semantic_index.vector_store.distance, distance: "cosine",
mode: "self_heal", mode: "self_heal",
}); });
if (!recreated.ok) return baseResult(runtime, "vector rebuild", "failed", "semantic_index_incompatible", { warnings: ["collection recreate failed"] }); if (!recreated.ok) return baseResult(runtime, "vector rebuild", "failed", "semantic_index_incompatible", { warnings: ["collection recreate failed"] });
@@ -400,6 +401,8 @@ export class WorkspacePreprocessingService {
catalogBlob: runtime.catalogBlob, catalogBlob: runtime.catalogBlob,
configDigest: runtime.configLease.configDigest, configDigest: runtime.configLease.configDigest,
bindingDigest: runtime.configLease.bindingDigest, bindingDigest: runtime.configLease.bindingDigest,
embeddingId: runtime.configLease.effectiveConfig.embedding.id,
embeddingDimensions: runtime.configLease.effectiveConfig.embedding.dimensions,
}); });
return { runtime, state, job }; return { runtime, state, job };
} }
+13 -3
View File
@@ -44,11 +44,13 @@ export interface BeginPreprocessingJobOptions {
catalogBlob: string; catalogBlob: string;
configDigest: string; configDigest: string;
bindingDigest: string; bindingDigest: string;
embeddingId: string;
embeddingDimensions: number;
runId?: string; runId?: string;
} }
export interface PreprocessingJobState { export interface PreprocessingJobState {
schemaVersion: 1; schemaVersion: 2;
runId: string; runId: string;
operation: string; operation: string;
workspaceId: string; workspaceId: string;
@@ -57,6 +59,8 @@ export interface PreprocessingJobState {
catalogBlob: string; catalogBlob: string;
configDigest: string; configDigest: string;
bindingDigest: string; bindingDigest: string;
embeddingId: string;
embeddingDimensions: number;
completedStages: string[]; completedStages: string[];
childRuns: Record<string, string>; childRuns: Record<string, string>;
status: "active" | "succeeded" | "blocked" | "failed"; status: "active" | "succeeded" | "blocked" | "failed";
@@ -146,7 +150,7 @@ function decodeJob(value: unknown): PreprocessingJobState {
} }
const record = value as Record<string, unknown>; const record = value as Record<string, unknown>;
if ( if (
record.schemaVersion !== 1 record.schemaVersion !== 2
|| typeof record.runId !== "string" || typeof record.runId !== "string"
|| typeof record.operation !== "string" || typeof record.operation !== "string"
|| typeof record.workspaceId !== "string" || typeof record.workspaceId !== "string"
@@ -155,6 +159,8 @@ function decodeJob(value: unknown): PreprocessingJobState {
|| typeof record.catalogBlob !== "string" || typeof record.catalogBlob !== "string"
|| typeof record.configDigest !== "string" || typeof record.configDigest !== "string"
|| typeof record.bindingDigest !== "string" || typeof record.bindingDigest !== "string"
|| typeof record.embeddingId !== "string"
|| typeof record.embeddingDimensions !== "number"
|| !Array.isArray(record.completedStages) || !Array.isArray(record.completedStages)
|| typeof record.childRuns !== "object" || record.childRuns === null || Array.isArray(record.childRuns) || typeof record.childRuns !== "object" || record.childRuns === null || Array.isArray(record.childRuns)
|| !["active", "succeeded", "blocked", "failed"].includes(String(record.status)) || !["active", "succeeded", "blocked", "failed"].includes(String(record.status))
@@ -275,6 +281,8 @@ export class PreprocessingStateStore {
|| existing.catalogBlob !== options.catalogBlob || existing.catalogBlob !== options.catalogBlob
|| existing.configDigest !== options.configDigest || existing.configDigest !== options.configDigest
|| existing.bindingDigest !== options.bindingDigest || existing.bindingDigest !== options.bindingDigest
|| existing.embeddingId !== options.embeddingId
|| existing.embeddingDimensions !== options.embeddingDimensions
) { ) {
throw new PreprocessingStateError( throw new PreprocessingStateError(
"preprocessing_resume_mismatch", "preprocessing_resume_mismatch",
@@ -295,7 +303,7 @@ export class PreprocessingStateStore {
} }
} }
const job: PreprocessingJobState = { const job: PreprocessingJobState = {
schemaVersion: 1, schemaVersion: 2,
runId, runId,
operation: options.operation, operation: options.operation,
workspaceId: this.options.workspaceId, workspaceId: this.options.workspaceId,
@@ -304,6 +312,8 @@ export class PreprocessingStateStore {
catalogBlob: options.catalogBlob, catalogBlob: options.catalogBlob,
configDigest: options.configDigest, configDigest: options.configDigest,
bindingDigest: options.bindingDigest, bindingDigest: options.bindingDigest,
embeddingId: options.embeddingId,
embeddingDimensions: options.embeddingDimensions,
completedStages: [], completedStages: [],
childRuns: {}, childRuns: {},
status: "active", status: "active",
+1 -1
View File
@@ -560,7 +560,7 @@ export class WorkspaceRegistry {
}); });
} }
} }
const collection = workspace.semantic_index.vector_store.collection; const collection = workspace.workspace.id;
const owner = collectionOwners.get(collection); const owner = collectionOwners.get(collection);
if (owner !== undefined) { if (owner !== undefined) {
throw new Error(`duplicate qdrant collection ownership: ${collection} (${owner}, ${id})`); throw new Error(`duplicate qdrant collection ownership: ${collection} (${owner}, ${id})`);
+20 -6
View File
@@ -34,6 +34,7 @@ export interface RuntimeInstallationOverlay {
export interface SemanticRuntimeConfig { export interface SemanticRuntimeConfig {
internalQdrantUrl: string; internalQdrantUrl: string;
internalEmbeddingUrl: string; internalEmbeddingUrl: string;
internalEmbeddingId?: string;
internalEmbeddingModel: string; internalEmbeddingModel: string;
internalEmbeddingDimensions: number; internalEmbeddingDimensions: number;
} }
@@ -41,6 +42,7 @@ export interface SemanticRuntimeConfig {
export const DEFAULT_SEMANTIC_RUNTIME: SemanticRuntimeConfig = { export const DEFAULT_SEMANTIC_RUNTIME: SemanticRuntimeConfig = {
internalQdrantUrl: "http://qdrant:6333", internalQdrantUrl: "http://qdrant:6333",
internalEmbeddingUrl: "http://embedding:11434", internalEmbeddingUrl: "http://embedding:11434",
internalEmbeddingId: "ollama/qwen3-embedding:0.6b",
internalEmbeddingModel: "qwen3-embedding:0.6b", internalEmbeddingModel: "qwen3-embedding:0.6b",
internalEmbeddingDimensions: 1024, internalEmbeddingDimensions: 1024,
}; };
@@ -202,16 +204,16 @@ function placeholderConnection(identity: { database: string; schema: string }):
function requireSupportedDescriptor(workspace: unknown): void { function requireSupportedDescriptor(workspace: unknown): void {
if (typeof workspace !== "object" || workspace === null) { if (typeof workspace !== "object" || workspace === null) {
throw new Error("Runtime renderer supports only workspace schema version 3"); throw new Error("Runtime renderer supports only workspace schema version 4");
} }
const metadata = Reflect.get(workspace, "workspace"); const metadata = Reflect.get(workspace, "workspace");
if (typeof metadata !== "object" || metadata === null if (typeof metadata !== "object" || metadata === null
|| Reflect.get(metadata, "schema_version") !== 3) { || Reflect.get(metadata, "schema_version") !== 4) {
throw new Error("Runtime renderer supports only workspace schema version 3"); throw new Error("Runtime renderer supports only workspace schema version 4");
} }
} }
/** Render the schema-v3 compatibility fields consumed by the current Python harness. */ /** Render the schema-v4 compatibility fields consumed by the current Python harness. */
export function renderRuntimeConfig( export function renderRuntimeConfig(
workspace: WorkspaceDescriptor, workspace: WorkspaceDescriptor,
bindings: RuntimeBindings, bindings: RuntimeBindings,
@@ -263,12 +265,24 @@ export function renderRuntimeConfig(
...(installation.profile === undefined ? {} : { profile: installation.profile }), ...(installation.profile === undefined ? {} : { profile: installation.profile }),
language: descriptor.workspace.language, language: descriptor.workspace.language,
database, database,
semantic_index: descriptor.semantic_index, semantic_index: {
vector_store: {
engine: "qdrant",
collection: descriptor.workspace.id,
dimensions: semanticRuntime.internalEmbeddingDimensions,
distance: "cosine",
},
embedding: {
provider: "ollama_internal",
model: semanticRuntime.internalEmbeddingModel,
dimensions: semanticRuntime.internalEmbeddingDimensions,
},
},
resources: { resources: {
vector: { vector: {
engine: "qdrant", engine: "qdrant",
base_url: semanticRuntime.internalQdrantUrl, base_url: semanticRuntime.internalQdrantUrl,
collection: descriptor.semantic_index.vector_store.collection, collection: descriptor.workspace.id,
}, },
embeddings: { embeddings: {
provider: "ollama_internal", provider: "ollama_internal",
+29 -67
View File
@@ -35,7 +35,7 @@ export interface CanonicalDiagnostics {
} }
interface WorkspaceMetadata { interface WorkspaceMetadata {
schema_version: 3; schema_version: 4;
id: string; id: string;
name: string; name: string;
description?: string; description?: string;
@@ -51,31 +51,12 @@ interface WorkspaceDwh {
supported_transports: DwhTransport[]; supported_transports: DwhTransport[];
} }
interface WorkspaceBase<TVectorStore> { interface WorkspaceBase {
workspace: WorkspaceMetadata; workspace: WorkspaceMetadata;
dwh: WorkspaceDwh; dwh: WorkspaceDwh;
semantic_index: {
vector_store: TVectorStore;
embedding: {
provider: "ollama_internal";
model: "qwen3-embedding:0.6b";
dimensions: 1024;
};
};
llm_policy: {
default?: `${string}/${string}`;
allowed: `${string}/${string}`[];
};
diagnostics?: Pick<CanonicalDiagnostics, "dwh_rest">; diagnostics?: Pick<CanonicalDiagnostics, "dwh_rest">;
} }
interface QdrantVectorStore {
engine: "qdrant";
collection: string;
dimensions: 1024;
distance: "cosine";
}
export interface EvidencePolicy { export interface EvidencePolicy {
max_chunk_chars: number; max_chunk_chars: number;
retain_published_generations: number; retain_published_generations: number;
@@ -120,12 +101,12 @@ export interface WorkspaceEvidence {
policy: EvidencePolicy; policy: EvidencePolicy;
} }
export interface WorkspaceV3 extends WorkspaceBase<QdrantVectorStore> { export interface WorkspaceV4 extends WorkspaceBase {
evidence?: WorkspaceEvidence; evidence?: WorkspaceEvidence;
} }
export type CanonicalWorkspace = WorkspaceV3; export type CanonicalWorkspace = WorkspaceV4;
export type WorkspaceDescriptor = WorkspaceV3; export type WorkspaceDescriptor = WorkspaceV4;
const workspaceId = z.string().regex(/^[a-z][a-z0-9-]{2,62}$/, { const workspaceId = z.string().regex(/^[a-z][a-z0-9-]{2,62}$/, {
message: "workspace id must match ^[a-z][a-z0-9-]{2,62}$", message: "workspace id must match ^[a-z][a-z0-9-]{2,62}$",
@@ -135,9 +116,6 @@ const identifier = z.string().regex(/^[A-Za-z_][A-Za-z0-9_]*$/, {
}); });
const port = z.number().int().min(1).max(65_535); const port = z.number().int().min(1).max(65_535);
const timeoutMs = z.number().int().positive(); const timeoutMs = z.number().int().positive();
const modelReference = z.string().regex(/^[^/\s]+\/[^/\s]+$/, {
message: "model must use provider/model syntax",
});
function isOriginRelativeDiagnosticPath(value: string): boolean { function isOriginRelativeDiagnosticPath(value: string): boolean {
return /^\/(?!\/)[^\\\u0000-\u001F\u007F?#]*$/.test(value) && !/%5c/i.test(value); return /^\/(?!\/)[^\\\u0000-\u001F\u007F?#]*$/.test(value) && !/%5c/i.test(value);
@@ -166,21 +144,6 @@ const dwhSchema = z.object({
timeout_ms: timeoutMs.optional(), timeout_ms: timeoutMs.optional(),
supported_transports: z.array(z.enum(DWH_TRANSPORTS)).min(1), supported_transports: z.array(z.enum(DWH_TRANSPORTS)).min(1),
}).strict(); }).strict();
const internalEmbeddingSchema = z.object({
provider: z.literal("ollama_internal"),
model: z.literal("qwen3-embedding:0.6b"),
dimensions: z.literal(1024),
}).strict();
const qdrantVectorStoreSchema = z.object({
engine: z.literal("qdrant"),
collection: workspaceId,
dimensions: z.literal(1024),
distance: z.literal("cosine"),
}).strict();
const llmPolicySchema = z.object({
default: modelReference.optional(),
allowed: z.array(modelReference).min(1),
}).strict();
const positiveSafeInteger = z.number().int().safe().positive(); const positiveSafeInteger = z.number().int().safe().positive();
const nonnegativeSafeInteger = z.number().int().safe().nonnegative(); const nonnegativeSafeInteger = z.number().int().safe().nonnegative();
@@ -366,7 +329,6 @@ function unique<T>(values: readonly T[], context: z.RefinementCtx, path: Propert
function workspaceInvariants(workspace: any, context: z.RefinementCtx): void { function workspaceInvariants(workspace: any, context: z.RefinementCtx): void {
unique(workspace.dwh.supported_transports, context, ["dwh", "supported_transports"]); unique(workspace.dwh.supported_transports, context, ["dwh", "supported_transports"]);
unique(workspace.llm_policy.allowed, context, ["llm_policy", "allowed"]);
if (workspace.evidence?.source.type === "filesystem") { if (workspace.evidence?.source.type === "filesystem") {
const expected = `${workspace.workspace.id}/evidence`; const expected = `${workspace.workspace.id}/evidence`;
@@ -379,20 +341,6 @@ function workspaceInvariants(workspace: any, context: z.RefinementCtx): void {
} }
} }
if (workspace.semantic_index.vector_store.dimensions !== workspace.semantic_index.embedding.dimensions) {
context.addIssue({
code: "custom",
path: ["semantic_index", "embedding", "dimensions"],
message: "embedding dimensions must match vector store dimensions",
});
}
if (workspace.llm_policy.default && !workspace.llm_policy.allowed.includes(workspace.llm_policy.default)) {
context.addIssue({
code: "custom",
path: ["llm_policy", "default"],
message: "LLM default must be included in the allowlist",
});
}
if (workspace.diagnostics?.dwh_rest && !workspace.dwh.supported_transports.includes("rest_api")) { if (workspace.diagnostics?.dwh_rest && !workspace.dwh.supported_transports.includes("rest_api")) {
context.addIssue({ context.addIssue({
code: "custom", code: "custom",
@@ -402,23 +350,18 @@ function workspaceInvariants(workspace: any, context: z.RefinementCtx): void {
} }
} }
const WorkspaceV3Schema = z.object({ const WorkspaceV4Schema = z.object({
dwh: dwhSchema, dwh: dwhSchema,
llm_policy: llmPolicySchema,
evidence: workspaceEvidenceSchema.optional(), evidence: workspaceEvidenceSchema.optional(),
diagnostics: z.object({ diagnostics: z.object({
dwh_rest: dwhRestDiagnostic.optional(), dwh_rest: dwhRestDiagnostic.optional(),
}).strict().optional(), }).strict().optional(),
workspace: z.object({ workspace: z.object({
schema_version: z.literal(3), id: workspaceId, name: z.string().trim().min(1), schema_version: z.literal(4), id: workspaceId, name: z.string().trim().min(1),
description: z.string().trim().min(1).optional(), language: z.enum(["en", "it"]), description: z.string().trim().min(1).optional(), language: z.enum(["en", "it"]),
}).strict(), }).strict(),
semantic_index: z.object({
vector_store: qdrantVectorStoreSchema,
embedding: internalEmbeddingSchema,
}).strict(),
}).strict().superRefine(workspaceInvariants); }).strict().superRefine(workspaceInvariants);
const WorkspaceDescriptorSchema = WorkspaceV3Schema; const WorkspaceDescriptorSchema = WorkspaceV4Schema;
export function parseWorkspaceYaml(source: string): WorkspaceDescriptor { export function parseWorkspaceYaml(source: string): WorkspaceDescriptor {
const documents = parseAllDocuments(source, { uniqueKeys: true }); const documents = parseAllDocuments(source, { uniqueKeys: true });
@@ -431,6 +374,25 @@ export function parseWorkspaceYaml(source: string): WorkspaceDescriptor {
return validateWorkspaceDescriptor(document.toJSON()); return validateWorkspaceDescriptor(document.toJSON());
} }
/** Deterministically removes the two installation-owned v3 blocks without altering workspace data. */
export function migrateWorkspaceV3Yaml(source: string): string {
const documents = parseAllDocuments(source, { uniqueKeys: true });
if (documents.length !== 1) throw new Error("Workspace YAML must contain exactly one document");
const document = documents[0];
if (document.errors.length > 0 || document.warnings.length > 0) {
throw new Error("Invalid workspace YAML");
}
const value = document.toJSON() as Record<string, unknown>;
const metadata = value.workspace as Record<string, unknown> | undefined;
if (!metadata || metadata.schema_version !== 3) {
throw new Error("Workspace migration requires schema version 3");
}
metadata.schema_version = 4;
delete value.semantic_index;
delete value.llm_policy;
return serializeWorkspaceYaml(validateWorkspaceDescriptor(value));
}
export function validateWorkspaceDescriptor(workspace: unknown): WorkspaceDescriptor { export function validateWorkspaceDescriptor(workspace: unknown): WorkspaceDescriptor {
return WorkspaceDescriptorSchema.parse(workspace) as WorkspaceDescriptor; return WorkspaceDescriptorSchema.parse(workspace) as WorkspaceDescriptor;
} }
@@ -439,11 +401,11 @@ export function isCanonicalWorkspace(workspace: unknown): workspace is Canonical
return WorkspaceDescriptorSchema.safeParse(workspace).success; return WorkspaceDescriptorSchema.safeParse(workspace).success;
} }
export function isOperationalWorkspace(workspace: unknown): workspace is WorkspaceV3 { export function isOperationalWorkspace(workspace: unknown): workspace is WorkspaceV4 {
return WorkspaceDescriptorSchema.safeParse(workspace).success; return WorkspaceDescriptorSchema.safeParse(workspace).success;
} }
export function validateOperationalWorkspace(workspace: unknown): WorkspaceV3 { export function validateOperationalWorkspace(workspace: unknown): WorkspaceV4 {
return validateWorkspaceDescriptor(workspace); return validateWorkspaceDescriptor(workspace);
} }
+1 -1
View File
@@ -19,4 +19,4 @@ export type WorkspaceErrorCode =
| "workspace_stale" | "git_unavailable" | "git_auth_failed" | "git_non_fast_forward" | "workspace_stale" | "git_unavailable" | "git_auth_failed" | "git_non_fast_forward"
| "connector_unavailable" | "semantic_index_incompatible"; | "connector_unavailable" | "semantic_index_incompatible";
export type { WorkspaceV3 } from "./schema.js"; export type { WorkspaceV4 } from "./schema.js";
+3 -16
View File
@@ -5,14 +5,12 @@ import { createLocalAuthFixture } from "./auth-test-fixtures.js";
function fakeService(): PiManagementService { function fakeService(): PiManagementService {
return { return {
status: vi.fn(async () => ({ ready: true })), status: vi.fn(async () => ({ ready: true })),
options: vi.fn(async () => ({ providers: [], models: [], reasoning: [], checkedAt: "2026-08-17T00:00:00.000Z" })),
configure: vi.fn(async (value) => ({ ...value, updatedAt: "2026-08-17T00:00:00.000Z" })),
test: vi.fn(async () => ({ ready: true, checkedAt: "2026-08-17T00:00:00.000Z" })), test: vi.fn(async () => ({ ready: true, checkedAt: "2026-08-17T00:00:00.000Z" })),
logs: vi.fn(async () => ({ lines: [] })), logs: vi.fn(async () => ({ lines: [] })),
}; };
} }
test("a local HTTPS cookie session authorizes Pi writes through an untrusted internal HTTP hop", async () => { test("a local HTTPS cookie session authorizes the Pi smoke check through an untrusted internal HTTP hop", async () => {
const service = fakeService(); const service = fakeService();
const fixture = await createLocalAuthFixture( const fixture = await createLocalAuthFixture(
{ piManagement: service }, { piManagement: service },
@@ -25,31 +23,21 @@ test("a local HTTPS cookie session authorizes Pi writes through an untrusted int
expect(fixture.publicUrl).toBe("HTTPS://thothii.example.test"); expect(fixture.publicUrl).toBe("HTTPS://thothii.example.test");
const proxyHeaders = fixture.sessionHeaders({ host: "127.0.0.1:8080" }); const proxyHeaders = fixture.sessionHeaders({ host: "127.0.0.1:8080" });
const configured = await fixture.app.inject({
method: "PUT",
url: "/pi-management/config",
headers: proxyHeaders,
payload: { provider: "zai", model: "glm-5.2", reasoning: "high" },
});
const smoke = await fixture.app.inject({ const smoke = await fixture.app.inject({
method: "POST", method: "POST",
url: "/pi-management/test", url: "/pi-management/test",
headers: proxyHeaders, headers: proxyHeaders,
}); });
expect(configured.statusCode).toBe(200);
expect(smoke.statusCode).toBe(200); expect(smoke.statusCode).toBe(200);
expect(service.configure).toHaveBeenCalledTimes(1);
expect(service.test).toHaveBeenCalledTimes(1); expect(service.test).toHaveBeenCalledTimes(1);
fixture.resetDownstreamHits(); fixture.resetDownstreamHits();
vi.mocked(service.configure).mockClear();
vi.mocked(service.test).mockClear(); vi.mocked(service.test).mockClear();
const wrongOrigin = await fixture.app.inject({ const wrongOrigin = await fixture.app.inject({
method: "PUT", method: "POST",
url: "/pi-management/config", url: "/pi-management/test",
headers: fixture.sessionHeaders({ host: "127.0.0.1:8080", origin: "https://evil.example" }), headers: fixture.sessionHeaders({ host: "127.0.0.1:8080", origin: "https://evil.example" }),
payload: { provider: "zai", model: "glm-5.2", reasoning: "high" },
}); });
const wrongCsrf = await fixture.app.inject({ const wrongCsrf = await fixture.app.inject({
method: "POST", method: "POST",
@@ -62,7 +50,6 @@ test("a local HTTPS cookie session authorizes Pi writes through an untrusted int
expect(response.json()).toEqual({ code: "csrf_failed", error: "Request origin validation failed" }); expect(response.json()).toEqual({ code: "csrf_failed", error: "Request origin validation failed" });
} }
expect(fixture.downstreamHits()).toBe(0); expect(fixture.downstreamHits()).toBe(0);
expect(service.configure).not.toHaveBeenCalled();
expect(service.test).not.toHaveBeenCalled(); expect(service.test).not.toHaveBeenCalled();
} finally { } finally {
await fixture.close(); await fixture.close();
@@ -19,16 +19,11 @@ afterEach(() => {
}); });
const workspace: WorkspaceDescriptor = { const workspace: WorkspaceDescriptor = {
workspace: { schema_version: 3, id: "psd-clinical", name: "Policlinico San Donato", language: "it" }, workspace: { schema_version: 4, id: "psd-clinical", name: "Policlinico San Donato", language: "it" },
dwh: { dwh: {
engine: "postgres", database: "warehouse", schema: "datawarehouse", port: 5432, engine: "postgres", database: "warehouse", schema: "datawarehouse", port: 5432,
supported_transports: ["postgres_direct", "rest_api"], supported_transports: ["postgres_direct", "rest_api"],
}, },
semantic_index: {
vector_store: { engine: "qdrant", collection: "psd", dimensions: 1024, distance: "cosine" },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
llm_policy: { allowed: ["zai/glm-5.2"] },
diagnostics: { dwh_rest: { method: "GET", path: "/health", auth: "bearer", response: { database: "database", schema: "schema" } } }, diagnostics: { dwh_rest: { method: "GET", path: "/health", auth: "bearer", response: { database: "database", schema: "schema" } } },
}; };
const revision: WorkspaceRevision = { id: "psd-clinical", commit: "a".repeat(40), blob: "b".repeat(40), snapshotPath: "/tmp/psd.yaml" }; const revision: WorkspaceRevision = { id: "psd-clinical", commit: "a".repeat(40), blob: "b".repeat(40), snapshotPath: "/tmp/psd.yaml" };
@@ -24,7 +24,7 @@ import type { WorkspaceDescriptor } from "../src/workspaces/schema.js";
const workspace: WorkspaceDescriptor = { const workspace: WorkspaceDescriptor = {
workspace: { workspace: {
schema_version: 3, schema_version: 4,
id: "psd-clinical", id: "psd-clinical",
name: "Policlinico San Donato", name: "Policlinico San Donato",
language: "it", language: "it",
@@ -36,11 +36,6 @@ const workspace: WorkspaceDescriptor = {
port: 5432, port: 5432,
supported_transports: ["postgres_direct"], supported_transports: ["postgres_direct"],
}, },
semantic_index: {
vector_store: { engine: "qdrant", collection: "psd", dimensions: 1024, distance: "cosine" },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
llm_policy: { allowed: ["zai/glm-5.2"] },
}; };
const revision: WorkspaceRevision = { const revision: WorkspaceRevision = {
id: "psd-clinical", id: "psd-clinical",
@@ -49,7 +44,7 @@ const revision: WorkspaceRevision = {
snapshotPath: "/tmp/psd.yaml", snapshotPath: "/tmp/psd.yaml",
}; };
const configuredModel: ResolvedMetadataGenerationModel = { const configuredModel: ResolvedMetadataGenerationModel = {
id: "openai-mini", id: "openai/gpt-4.1-mini",
provider: "openai", provider: "openai",
model: "gpt-4.1-mini", model: "gpt-4.1-mini",
apiKeyEnv: "OPENAI_API_KEY", apiKeyEnv: "OPENAI_API_KEY",
@@ -705,7 +700,7 @@ test("generates one selected Catalog Column from a single JSON code fence", asyn
expect(completionRequest.messages[0]?.content).toContain('{"results":['); expect(completionRequest.messages[0]?.content).toContain('{"results":[');
expect(completionRequest.messages[1]?.content).toContain(`"targetId":"${column.id}"`); expect(completionRequest.messages[1]?.content).toContain(`"targetId":"${column.id}"`);
expect(completionRequest.messages[1]?.content).not.toMatch(/source rows|samples|example values/i); expect(completionRequest.messages[1]?.content).not.toMatch(/source rows|samples|example values/i);
expect(start.body).not.toMatch(/test-provider-secret|gpt-4\.1|openai\/gpt|Catalog metadata/); expect(start.body).not.toMatch(/test-provider-secret|Catalog metadata/);
resolveCompletion(`\`\`\`json\n${JSON.stringify({ resolveCompletion(`\`\`\`json\n${JSON.stringify({
results: [{ results: [{
@@ -2909,7 +2904,7 @@ test("validates selected targets and resolves every requested target before laun
const unknownModel = await app.inject({ const unknownModel = await app.inject({
method: "POST", method: "POST",
url: `/catalog/databases/${database.id}/description-generation-runs`, url: `/catalog/databases/${database.id}/description-generation-runs`,
payload: { modelId: "unknown-model", scope: "selected_columns", targetIds: [column.id] }, payload: { modelId: "openai/unknown-model", scope: "selected_columns", targetIds: [column.id] },
}); });
expect(unknownModel.statusCode).toBe(409); expect(unknownModel.statusCode).toBe(409);
expect(unknownModel.json().code).toBe("metadata_generation_model_unavailable"); expect(unknownModel.json().code).toBe("metadata_generation_model_unavailable");
@@ -14,6 +14,7 @@ import { up as upDescriptionGeneration } from "../src/catalog/migrations/005_des
import { up as upSensitiveDataFlag } from "../src/catalog/migrations/006_sensitive_data_flag.js"; import { up as upSensitiveDataFlag } from "../src/catalog/migrations/006_sensitive_data_flag.js";
import { up as upSensitiveSuggestionRuns } from "../src/catalog/migrations/007_sensitive_data_suggestion_runs.js"; import { up as upSensitiveSuggestionRuns } from "../src/catalog/migrations/007_sensitive_data_suggestion_runs.js";
import { up as upAiTokenUsage } from "../src/catalog/migrations/009_ai_token_usage.js"; import { up as upAiTokenUsage } from "../src/catalog/migrations/009_ai_token_usage.js";
import { up as upCanonicalModelIds } from "../src/catalog/migrations/010_canonical_model_ids.js";
import { KyselyCatalogRepository, type CatalogDatabase } from "../src/catalog/repository.js"; import { KyselyCatalogRepository, type CatalogDatabase } from "../src/catalog/repository.js";
import { loadConfig } from "../src/config.js"; import { loadConfig } from "../src/config.js";
import type { WorkspaceRegistry } from "../src/workspaces/registry.js"; import type { WorkspaceRegistry } from "../src/workspaces/registry.js";
@@ -50,6 +51,7 @@ test.skipIf(!dockerAvailable)("Fastify persists Description Generation success a
await upDescriptionGeneration(db); await upDescriptionGeneration(db);
await upSensitiveSuggestionRuns(db); await upSensitiveSuggestionRuns(db);
await upAiTokenUsage(db); await upAiTokenUsage(db);
await upCanonicalModelIds(db);
const repository = new KyselyCatalogRepository(db); const repository = new KyselyCatalogRepository(db);
const database = await repository.create({ const database = await repository.create({
workspaceId: "psd-clinical", workspaceId: "psd-clinical",
@@ -158,9 +160,9 @@ test.skipIf(!dockerAvailable)("Fastify persists Description Generation success a
}), }),
}; };
const models: MetadataGenerationModels = { const models: MetadataGenerationModels = {
catalog: () => ({ models: [{ id: "openai-mini", label: "OpenAI Mini" }], default: "openai-mini" }), catalog: () => ({ models: [{ id: "openai/gpt-4.1-mini", label: "OpenAI Mini" }], default: "openai/gpt-4.1-mini" }),
resolve: () => ({ resolve: () => ({
id: "openai-mini", id: "openai/gpt-4.1-mini",
provider: "openai", provider: "openai",
model: "gpt-4.1-mini", model: "gpt-4.1-mini",
apiKeyEnv: "OPENAI_API_KEY", apiKeyEnv: "OPENAI_API_KEY",
@@ -205,12 +207,12 @@ test.skipIf(!dockerAvailable)("Fastify persists Description Generation success a
method: "POST", method: "POST",
url: `/catalog/databases/${database.id}/description-generation-runs`, url: `/catalog/databases/${database.id}/description-generation-runs`,
payload: { payload: {
modelId: "openai-mini", modelId: "openai/gpt-4.1-mini",
scope: "selected_columns", scope: "selected_columns",
targetIds: [status.id, birthDate.id], targetIds: [status.id, birthDate.id],
}, },
}); });
expect(successfulStart.statusCode).toBe(202); expect(successfulStart.statusCode, successfulStart.body).toBe(202);
expect(await terminalRun(app, successfulStart.json().id)).toMatchObject({ expect(await terminalRun(app, successfulStart.json().id)).toMatchObject({
status: "completed", status: "completed",
total: 2, total: 2,
@@ -233,7 +235,7 @@ test.skipIf(!dockerAvailable)("Fastify persists Description Generation success a
const tableStart = await app.inject({ const tableStart = await app.inject({
method: "POST", method: "POST",
url: `/catalog/databases/${database.id}/description-generation-runs`, url: `/catalog/databases/${database.id}/description-generation-runs`,
payload: { modelId: "openai-mini", scope: "selected_tables", targetIds: [table.id] }, payload: { modelId: "openai/gpt-4.1-mini", scope: "selected_tables", targetIds: [table.id] },
}); });
expect(tableStart.statusCode).toBe(202); expect(tableStart.statusCode).toBe(202);
expect(await terminalRun(app, tableStart.json().id)).toMatchObject({ expect(await terminalRun(app, tableStart.json().id)).toMatchObject({
@@ -253,7 +255,7 @@ test.skipIf(!dockerAvailable)("Fastify persists Description Generation success a
const failedStart = await app.inject({ const failedStart = await app.inject({
method: "POST", method: "POST",
url: `/catalog/databases/${database.id}/description-generation-runs`, url: `/catalog/databases/${database.id}/description-generation-runs`,
payload: { modelId: "openai-mini", scope: "selected_columns", targetIds: [status.id] }, payload: { modelId: "openai/gpt-4.1-mini", scope: "selected_columns", targetIds: [status.id] },
}); });
expect(failedStart.statusCode).toBe(202); expect(failedStart.statusCode).toBe(202);
const failedRun = await terminalRun(app, failedStart.json().id); const failedRun = await terminalRun(app, failedStart.json().id);
@@ -283,7 +285,7 @@ test.skipIf(!dockerAvailable)("Fastify persists Description Generation success a
const allStart = await app.inject({ const allStart = await app.inject({
method: "POST", method: "POST",
url: `/catalog/databases/${database.id}/description-generation-runs`, url: `/catalog/databases/${database.id}/description-generation-runs`,
payload: { modelId: "openai-mini", scope: "all" }, payload: { modelId: "openai/gpt-4.1-mini", scope: "all" },
}); });
expect(allStart.statusCode).toBe(202); expect(allStart.statusCode).toBe(202);
const allRun = await terminalRun(app, allStart.json().id); const allRun = await terminalRun(app, allStart.json().id);
@@ -327,7 +329,7 @@ test.skipIf(!dockerAvailable)("Fastify persists Description Generation success a
const missingStart = await app.inject({ const missingStart = await app.inject({
method: "POST", method: "POST",
url: `/catalog/databases/${database.id}/description-generation-runs`, url: `/catalog/databases/${database.id}/description-generation-runs`,
payload: { modelId: "openai-mini", scope: "missing" }, payload: { modelId: "openai/gpt-4.1-mini", scope: "missing" },
}); });
expect(missingStart.statusCode).toBe(202); expect(missingStart.statusCode).toBe(202);
expect(await terminalRun(app, missingStart.json().id)).toMatchObject({ expect(await terminalRun(app, missingStart.json().id)).toMatchObject({
@@ -14,6 +14,7 @@ import { up as upSensitiveDataFlag } from "../src/catalog/migrations/006_sensiti
import { up as upSensitiveSuggestionRuns } from "../src/catalog/migrations/007_sensitive_data_suggestion_runs.js"; import { up as upSensitiveSuggestionRuns } from "../src/catalog/migrations/007_sensitive_data_suggestion_runs.js";
import { up as upLogicalRelationships } from "../src/catalog/migrations/008_catalog_logical_relationships.js"; import { up as upLogicalRelationships } from "../src/catalog/migrations/008_catalog_logical_relationships.js";
import { up as upAiTokenUsage } from "../src/catalog/migrations/009_ai_token_usage.js"; import { up as upAiTokenUsage } from "../src/catalog/migrations/009_ai_token_usage.js";
import { up as upCanonicalModelIds } from "../src/catalog/migrations/010_canonical_model_ids.js";
const dockerAvailable = spawnSync("docker", ["info"], { stdio: "ignore" }).status === 0; const dockerAvailable = spawnSync("docker", ["info"], { stdio: "ignore" }).status === 0;
@@ -32,6 +33,7 @@ test.skipIf(!dockerAvailable)("PostgreSQL migration enforces one database per wo
await upDescriptionGeneration(db); await upDescriptionGeneration(db);
await upSensitiveSuggestionRuns(db); await upSensitiveSuggestionRuns(db);
await upAiTokenUsage(db); await upAiTokenUsage(db);
await upCanonicalModelIds(db);
await sql`CREATE ROLE thothii_catalog_runtime`.execute(db); await sql`CREATE ROLE thothii_catalog_runtime`.execute(db);
await upRuntimeSequencePrivileges(db); await upRuntimeSequencePrivileges(db);
const sequencePrivilege = await sql<{ allowed: boolean }>` const sequencePrivilege = await sql<{ allowed: boolean }>`
@@ -391,6 +393,7 @@ test.skipIf(!dockerAvailable)("PostgreSQL repository persists description and se
await upDescriptionGeneration(db); await upDescriptionGeneration(db);
await upSensitiveSuggestionRuns(db); await upSensitiveSuggestionRuns(db);
await upAiTokenUsage(db); await upAiTokenUsage(db);
await upCanonicalModelIds(db);
const repository = new KyselyCatalogRepository(db); const repository = new KyselyCatalogRepository(db);
const firstDatabase = await repository.create({ const firstDatabase = await repository.create({
workspaceId: "generation-one", workspaceId: "generation-one",
@@ -428,14 +431,14 @@ test.skipIf(!dockerAvailable)("PostgreSQL repository persists description and se
const run = await repository.createDescriptionGenerationRun( const run = await repository.createDescriptionGenerationRun(
firstDatabase.id, firstDatabase.id,
"selected_columns", "selected_columns",
"openai-mini", "openai/gpt-4.1-mini",
"it", "it",
1, 1,
); );
expect(run).toMatchObject({ expect(run).toMatchObject({
databaseId: firstDatabase.id, databaseId: firstDatabase.id,
scope: "selected_columns", scope: "selected_columns",
modelId: "openai-mini", modelId: "openai/gpt-4.1-mini",
language: "it", language: "it",
status: "queued", status: "queued",
total: 1, total: 1,
@@ -450,7 +453,7 @@ test.skipIf(!dockerAvailable)("PostgreSQL repository persists description and se
await expect(repository.createDescriptionGenerationRun( await expect(repository.createDescriptionGenerationRun(
secondDatabase.id, secondDatabase.id,
"selected_columns", "selected_columns",
"openai-mini", "openai/gpt-4.1-mini",
"en", "en",
1, 1,
)).rejects.toThrow("A description generation run is already active"); )).rejects.toThrow("A description generation run is already active");
@@ -461,7 +464,7 @@ test.skipIf(!dockerAvailable)("PostgreSQL repository persists description and se
await expect(repository.createDescriptionGenerationRun( await expect(repository.createDescriptionGenerationRun(
secondDatabase.id, secondDatabase.id,
"selected_columns", "selected_columns",
"openai-mini", "openai/gpt-4.1-mini",
"en", "en",
1, 1,
)).rejects.toThrow("A description generation run is already active"); )).rejects.toThrow("A description generation run is already active");
@@ -505,7 +508,7 @@ test.skipIf(!dockerAvailable)("PostgreSQL repository persists description and se
const next = await repository.createDescriptionGenerationRun( const next = await repository.createDescriptionGenerationRun(
secondDatabase.id, secondDatabase.id,
"missing", "missing",
"openai-mini", "openai/gpt-4.1-mini",
"en", "en",
1, 1,
); );
@@ -533,7 +536,7 @@ test.skipIf(!dockerAvailable)("PostgreSQL repository persists description and se
const allRun = await repository.createDescriptionGenerationRun( const allRun = await repository.createDescriptionGenerationRun(
firstDatabase.id, firstDatabase.id,
"all", "all",
"openai-mini", "openai/gpt-4.1-mini",
"it", "it",
2, 2,
); );
@@ -562,7 +565,7 @@ test.skipIf(!dockerAvailable)("PostgreSQL repository persists description and se
const suggestionRun = await repository.createSensitiveDataSuggestionRun( const suggestionRun = await repository.createSensitiveDataSuggestionRun(
firstDatabase.id, firstDatabase.id,
"selected_columns", "selected_columns",
"openai-mini", "openai/gpt-4.1-mini",
); );
expect(suggestionRun).toMatchObject({ expect(suggestionRun).toMatchObject({
databaseId: firstDatabase.id, databaseId: firstDatabase.id,
@@ -604,7 +607,7 @@ test.skipIf(!dockerAvailable)("PostgreSQL repository persists description and se
const interruptedSuggestionRun = await repository.createSensitiveDataSuggestionRun( const interruptedSuggestionRun = await repository.createSensitiveDataSuggestionRun(
secondDatabase.id, secondDatabase.id,
"all", "all",
"openai-mini", "openai/gpt-4.1-mini",
); );
expect(await repository.interruptActiveSensitiveDataSuggestionRuns( expect(await repository.interruptActiveSensitiveDataSuggestionRuns(
"Sensitive-field suggestion generation was interrupted by backend restart.", "Sensitive-field suggestion generation was interrupted by backend restart.",
+1 -6
View File
@@ -16,13 +16,8 @@ const roots: string[] = [];
afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); }); afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); });
const workspace: WorkspaceDescriptor = { const workspace: WorkspaceDescriptor = {
workspace: { schema_version: 3, id: "psd-clinical", name: "Policlinico San Donato", language: "it" }, workspace: { schema_version: 4, id: "psd-clinical", name: "Policlinico San Donato", language: "it" },
dwh: { engine: "postgres", database: "warehouse", schema: "datawarehouse", port: 5432, supported_transports: ["postgres_direct"] }, dwh: { engine: "postgres", database: "warehouse", schema: "datawarehouse", port: 5432, supported_transports: ["postgres_direct"] },
semantic_index: {
vector_store: { engine: "qdrant", collection: "psd", dimensions: 1024, distance: "cosine" },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
llm_policy: { allowed: ["zai/glm-5.2"] },
}; };
const revision: WorkspaceRevision = { id: "psd-clinical", commit: "a".repeat(40), blob: "b".repeat(40), snapshotPath: "/tmp/psd.yaml" }; const revision: WorkspaceRevision = { id: "psd-clinical", commit: "a".repeat(40), blob: "b".repeat(40), snapshotPath: "/tmp/psd.yaml" };
+1 -6
View File
@@ -13,16 +13,11 @@ const roots: string[] = [];
afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); }); afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); });
const workspace: WorkspaceDescriptor = { const workspace: WorkspaceDescriptor = {
workspace: { schema_version: 3, id: "psd-clinical", name: "Policlinico San Donato", language: "it" }, workspace: { schema_version: 4, id: "psd-clinical", name: "Policlinico San Donato", language: "it" },
dwh: { dwh: {
engine: "postgres", database: "warehouse", schema: "datawarehouse", port: 5432, engine: "postgres", database: "warehouse", schema: "datawarehouse", port: 5432,
supported_transports: ["postgres_direct", "rest_api"], supported_transports: ["postgres_direct", "rest_api"],
}, },
semantic_index: {
vector_store: { engine: "qdrant", collection: "psd", dimensions: 1024, distance: "cosine" },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
llm_policy: { allowed: ["zai/glm-5.2"] },
diagnostics: { dwh_rest: { method: "GET", path: "/health", auth: "bearer", response: { database: "database", schema: "schema" } } }, diagnostics: { dwh_rest: { method: "GET", path: "/health", auth: "bearer", response: { database: "database", schema: "schema" } } },
}; };
const revision: WorkspaceRevision = { const revision: WorkspaceRevision = {
+17
View File
@@ -71,6 +71,7 @@ test("loadConfig keeps local development defaults", () => {
workspaceSecretRuntimeRoot: "/tmp/thothii-workspace-secrets", workspaceSecretRuntimeRoot: "/tmp/thothii-workspace-secrets",
internalQdrantUrl: "http://qdrant:6333", internalQdrantUrl: "http://qdrant:6333",
internalEmbeddingUrl: "http://embedding:11434", internalEmbeddingUrl: "http://embedding:11434",
internalEmbeddingId: "ollama/qwen3-embedding:0.6b",
internalEmbeddingModel: "qwen3-embedding:0.6b", internalEmbeddingModel: "qwen3-embedding:0.6b",
internalEmbeddingDimensions: 1024, internalEmbeddingDimensions: 1024,
authMode: "none", authMode: "none",
@@ -198,6 +199,22 @@ test("loadConfig accepts only the allowed internal semantic runtime hosts", () =
.toThrow(/internal.*embedding|invalid/i); .toThrow(/internal.*embedding|invalid/i);
}); });
test("loadConfig derives the embedding runtime model from its canonical catalog identity", () => {
expect(loadConfig({
THT_INTERNAL_EMBEDDING_ID: "ollama/nomic-embed-text",
THT_INTERNAL_EMBEDDING_MODEL: "nomic-embed-text",
})).toMatchObject({
internalEmbeddingId: "ollama/nomic-embed-text",
internalEmbeddingModel: "nomic-embed-text",
});
expect(() => loadConfig({
THT_INTERNAL_EMBEDDING_ID: "ollama/nomic-embed-text",
THT_INTERNAL_EMBEDDING_MODEL: "different-model",
})).toThrow("does not match its canonical identity");
expect(() => loadConfig({ THT_INTERNAL_EMBEDDING_ID: "not-canonical" }))
.toThrow("embedding identity configuration is invalid");
});
test("loadConfig enables the legacy workspace request only through explicit local mode", () => { test("loadConfig enables the legacy workspace request only through explicit local mode", () => {
expect(loadConfig({ THT_LEGACY_WORKSPACE_MODE: "local" }).legacyWorkspaceMode).toBe(true); expect(loadConfig({ THT_LEGACY_WORKSPACE_MODE: "local" }).legacyWorkspaceMode).toBe(true);
+6 -2
View File
@@ -11,6 +11,7 @@ import {
const semanticRuntime = { const semanticRuntime = {
internalQdrantUrl: "http://qdrant:6333", internalQdrantUrl: "http://qdrant:6333",
internalEmbeddingUrl: "http://embedding:11434", internalEmbeddingUrl: "http://embedding:11434",
internalEmbeddingId: "ollama/qwen3-embedding:0.6b",
internalEmbeddingModel: "qwen3-embedding:0.6b", internalEmbeddingModel: "qwen3-embedding:0.6b",
internalEmbeddingDimensions: 1024, internalEmbeddingDimensions: 1024,
}; };
@@ -102,11 +103,11 @@ function restRendered(): Record<string, unknown> {
} }
const directCanonical = const directCanonical =
`{"schemaVersion":1,"dwh":{` + `{"schemaVersion":2,"dwh":{` +
`"engine":"postgres","database":"postgres","schema":"datawarehouse",` + `"engine":"postgres","database":"postgres","schema":"datawarehouse",` +
`"transport":"postgres_direct","host":"dwh.internal","port":5432,"user":"thoth_reader"},` + `"transport":"postgres_direct","host":"dwh.internal","port":5432,"user":"thoth_reader"},` +
`"vector":{"collection":"psd-clinical","dimensions":1024,"distance":"cosine"},` + `"vector":{"collection":"psd-clinical","dimensions":1024,"distance":"cosine"},` +
`"embedding":{"model":"qwen3-embedding:0.6b","dimensions":1024},` + `"embedding":{"id":"ollama/qwen3-embedding:0.6b","model":"qwen3-embedding:0.6b","dimensions":1024},` +
`"roots":{"artifacts":"/data/sessions/psd-clinical/artifacts",` + `"roots":{"artifacts":"/data/sessions/psd-clinical/artifacts",` +
`"indexes":"/data/sessions/psd-clinical/indexes"}}`; `"indexes":"/data/sessions/psd-clinical/indexes"}}`;
@@ -192,6 +193,9 @@ test("DWH-affecting changes alter the effective config identity", () => {
const changedCollection = { ...base, resources: { ...base.resources, vector: { ...(base.resources as Record<string, any>).vector, collection: "other" } } }; const changedCollection = { ...base, resources: { ...base.resources, vector: { ...(base.resources as Record<string, any>).vector, collection: "other" } } };
expect(effectiveConfigIdentity("psd-clinical", changedCollection)).not.toBe(identityBefore); expect(effectiveConfigIdentity("psd-clinical", changedCollection)).not.toBe(identityBefore);
const changedEmbeddingIdentity = { ...base, resources: { ...base.resources, embeddings: { ...(base.resources as Record<string, any>).embeddings, model: "other-embedding" } } };
expect(effectiveConfigIdentity("psd-clinical", changedEmbeddingIdentity)).not.toBe(identityBefore);
const changedTransport = restRendered(); const changedTransport = restRendered();
expect(effectiveConfigIdentity("psd-clinical", changedTransport)).not.toBe(identityBefore); expect(effectiveConfigIdentity("psd-clinical", changedTransport)).not.toBe(identityBefore);
}); });
+85 -247
View File
@@ -1,15 +1,12 @@
import { chmodSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; import { chmodSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os"; import { tmpdir } from "node:os";
import { join } from "node:path"; import { join } from "node:path";
import { afterEach, expect, test, vi } from "vitest"; import { afterEach, expect, test } from "vitest";
import { buildApp } from "../src/app.js";
import { MemoryCatalogRepository } from "../src/catalog/memory-repository.js";
import { import {
loadMetadataGenerationModels, loadMetadataGenerationModels,
MetadataGenerationModelUnavailableError, MetadataGenerationModelUnavailableError,
} from "../src/catalog/metadata-generation-models.js"; } from "../src/catalog/metadata-generation-models.js";
import { loadConfig } from "../src/config.js"; import { loadRuntimeModelCatalog, splitCanonicalModelId } from "../src/models/runtime-model-catalog.js";
import type { WorkspaceRegistry } from "../src/workspaces/registry.js";
const roots: string[] = []; const roots: string[] = [];
@@ -17,260 +14,101 @@ afterEach(() => {
for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true });
}); });
function metadataConfiguration( function runtimeCatalog(overrides: Record<string, unknown> = {}, secrets = "OPENAI_API_KEY=raw-provider-secret\n") {
metadataGeneration: string, const root = mkdtempSync(join(tmpdir(), "thothii-runtime-models-"));
secrets = "OPENAI_API_KEY=raw-provider-secret\n",
) {
const root = mkdtempSync(join(tmpdir(), "thothii-metadata-models-"));
roots.push(root); roots.push(root);
const installationFile = join(root, "thothii-installation.yaml"); const catalogFile = join(root, "catalog.json");
const secretsFile = join(root, "thothii.secrets"); const secretsFile = join(root, "thothii.secrets");
writeFileSync(installationFile, metadataGeneration, { mode: 0o600 }); const catalog = {
writeFileSync(secretsFile, secrets, { mode: 0o600 }); schemaVersion: 1,
chmodSync(installationFile, 0o600); defaultSession: "zai/glm-5.3",
chmodSync(secretsFile, 0o600); defaultMetadataGeneration: "zai/glm-5.3",
return { installationFile, secretsFile }; embedding: { id: "ollama/qwen3-embedding:0.6b", dimensions: 1024 },
}
function appFor(installationFile: string, secretsFile: string) {
const config = loadConfig({
NODE_ENV: "test",
THT_HARNESS_DIR: "/missing",
THT_INSTALLATION_CONFIG_FILE: installationFile,
THT_SECRETS_FILE: secretsFile,
PI_PROVIDER: "unrelated-pi-provider",
PI_MODEL: "unrelated-pi-model",
});
return buildApp(config, {
thtRunner: {} as never,
workspaceRegistry: { list: vi.fn(async () => []) } as unknown as WorkspaceRegistry,
workspaceDiagnoser: vi.fn(),
catalogRepository: new MemoryCatalogRepository(),
});
}
test("exposes only safe metadata-generation choices and their configured default", async () => {
const { installationFile, secretsFile } = metadataConfiguration(`metadataGeneration:
default: openai-mini
models:
- id: openai-mini
label: OpenAI Mini
litellm:
provider: openai
model: gpt-4.1-mini
endpoint:
baseUrl: https://api.openai.example/v1
apiVersion: "2026-08-01"
apiKeyEnv: OPENAI_API_KEY
`);
const app = appFor(installationFile, secretsFile);
const response = await app.inject({ method: "GET", url: "/catalog/metadata-generation/models" });
expect(response.statusCode).toBe(200);
expect(response.json()).toEqual({
models: [{ id: "openai-mini", label: "OpenAI Mini" }],
default: "openai-mini",
});
expect(response.body).not.toMatch(/openai\/gpt|gpt-4\.1|api\.openai|OPENAI_API_KEY|raw-provider-secret/);
await app.close();
});
test("rejects an unprotected installation descriptor", () => {
const { installationFile, secretsFile } = metadataConfiguration(`metadataGeneration:
default: openai-mini
models:
- id: openai-mini
label: OpenAI Mini
litellm: {provider: openai, model: gpt-4.1-mini}
apiKeyEnv: OPENAI_API_KEY
`);
chmodSync(installationFile, 0o644);
expect(() => loadMetadataGenerationModels({ installationFile, secretsFile }))
.toThrow("metadata-generation installation is unavailable");
});
test("returns an empty safe catalog when no metadata-generation model is configured", async () => {
const { installationFile, secretsFile } = metadataConfiguration("profile: local\n");
const app = appFor(installationFile, secretsFile);
const response = await app.inject({ method: "GET", url: "/catalog/metadata-generation/models" });
expect(response.statusCode).toBe(200);
expect(response.json()).toEqual({ models: [], default: null });
await app.close();
});
test("resolves only a configured selection for the later generation boundary", () => {
const { installationFile, secretsFile } = metadataConfiguration(`metadataGeneration:
default: openai-mini
models:
- id: openai-mini
label: OpenAI Mini
litellm:
provider: openai
model: gpt-4.1-mini
endpoint: {baseUrl: https://api.openai.example/v1, apiVersion: "2026-08-01"}
apiKeyEnv: OPENAI_API_KEY
`);
const models = loadMetadataGenerationModels({ installationFile, secretsFile });
expect(models.resolve("openai-mini")).toEqual({
id: "openai-mini",
provider: "openai",
model: "gpt-4.1-mini",
endpoint: { baseUrl: "https://api.openai.example/v1", apiVersion: "2026-08-01" },
apiKeyEnv: "OPENAI_API_KEY",
apiKey: "raw-provider-secret",
});
expect(() => models.resolve("unknown-model")).toThrow(MetadataGenerationModelUnavailableError);
});
test("loads DeepSeek models, GLM, and an explicit keyless Qwen endpoint from installation setup", () => {
const { installationFile, secretsFile } = metadataConfiguration(`metadataGeneration:
default: glm-53
models:
- id: deepseek-v4-pro
label: DeepSeek V4 Pro
litellm: {provider: deepseek, model: deepseek-v4-pro}
apiKeyEnv: DEEPSEEK_API_KEY
- id: deepseek-v4-flash
label: DeepSeek V4 Flash
litellm: {provider: deepseek, model: deepseek-v4-flash}
apiKeyEnv: DEEPSEEK_API_KEY
- id: glm-53
label: GLM 5.3
litellm:
provider: openai
model: glm-5.3
endpoint: {baseUrl: https://api.z.ai/api/coding/paas/v4}
apiKeyEnv: ZAI_API_KEY
- id: qwen-36
label: Qwen 3.6
litellm:
provider: openai
model: qwen3.6-35b-a3b
disableThinking: true
endpoint: {baseUrl: https://models.internal.example/v1}
`, "DEEPSEEK_API_KEY=deepseek-secret\nZAI_API_KEY=zai-secret\n");
const models = loadMetadataGenerationModels({ installationFile, secretsFile });
expect(models.catalog()).toEqual({
models: [ models: [
{ id: "deepseek-v4-pro", label: "DeepSeek V4 Pro" }, {
{ id: "deepseek-v4-flash", label: "DeepSeek V4 Flash" }, id: "zai/glm-5.3", provider: "zai", model: "glm-5.3", label: "GLM 5.3",
{ id: "glm-53", label: "GLM 5.3" }, upstreamModel: "glm-5.3", endpoint: { baseUrl: "https://api.z.ai/v1" },
{ id: "qwen-36", label: "Qwen 3.6" }, authentication: { mode: "secret_env", apiKeyEnv: "OPENAI_API_KEY" },
sessionAdapter: { mode: "openai_compatible" },
metadataAdapter: { litellmProvider: "openai" },
session: { reasoning: true, contextWindow: 200000, maxTokens: 131072 },
metadataGeneration: { disableThinking: false },
},
{
id: "deepseek/deepseek-v4-pro", provider: "deepseek", model: "deepseek-v4-pro",
label: "DeepSeek V4 Pro", upstreamModel: "deepseek-v4-pro",
authentication: { mode: "pi_auth" }, sessionAdapter: { mode: "pi_builtin" },
session: { reasoning: false },
},
], ],
default: "glm-53", ...overrides,
};
writeFileSync(catalogFile, JSON.stringify(catalog), { mode: 0o600 });
writeFileSync(secretsFile, secrets, { mode: 0o600 });
chmodSync(catalogFile, 0o600);
chmodSync(secretsFile, 0o600);
return { catalogFile, secretsFile };
}
test("loads session default and safe metadata choices from the normalized runtime catalog", () => {
const { catalogFile, secretsFile } = runtimeCatalog();
const runtime = loadRuntimeModelCatalog(catalogFile);
const metadata = loadMetadataGenerationModels({ catalogFile, secretsFile });
expect(runtime.defaultSession).toBe("zai/glm-5.3");
expect(runtime.hasSession("deepseek/deepseek-v4-pro")).toBe(true);
expect(metadata.catalog()).toEqual({
models: [{ id: "zai/glm-5.3", label: "GLM 5.3" }],
default: "zai/glm-5.3",
}); });
expect(models.resolve("deepseek-v4-pro")).toMatchObject({ expect(metadata.resolve("zai/glm-5.3")).toEqual({
apiKeyEnv: "DEEPSEEK_API_KEY", id: "zai/glm-5.3", provider: "openai", model: "glm-5.3",
apiKey: "deepseek-secret", endpoint: { baseUrl: "https://api.z.ai/v1" },
}); apiKeyEnv: "OPENAI_API_KEY", apiKey: "raw-provider-secret",
expect(models.resolve("qwen-36")).toEqual({
id: "qwen-36",
provider: "openai",
model: "qwen3.6-35b-a3b",
disableThinking: true,
endpoint: { baseUrl: "https://models.internal.example/v1" },
}); });
expect(() => metadata.resolve("zai/missing")).toThrow(MetadataGenerationModelUnavailableError);
}); });
test("loads an explicit keyless endpoint without a secret bundle", () => { test("returns empty catalogs when no runtime projection is configured", () => {
const { installationFile } = metadataConfiguration(`metadataGeneration: expect(loadRuntimeModelCatalog().defaultSession).toBeNull();
default: qwen-36 expect(loadMetadataGenerationModels({}).catalog()).toEqual({ models: [], default: null });
models: });
- id: qwen-36
label: Qwen 3.6
litellm:
provider: openai
model: qwen3.6-35b-a3b
disableThinking: true
endpoint: {baseUrl: https://models.internal.example/v1}
`);
expect(loadMetadataGenerationModels({ installationFile }).resolve("qwen-36")).toEqual({ test("rejects a drifted default and an unprotected projection", () => {
id: "qwen-36", const drifted = runtimeCatalog({ defaultSession: "zai/missing" });
provider: "openai", expect(() => loadRuntimeModelCatalog(drifted.catalogFile)).toThrow("session default is invalid");
model: "qwen3.6-35b-a3b",
disableThinking: true, const unprotected = runtimeCatalog();
endpoint: { baseUrl: "https://models.internal.example/v1" }, chmodSync(unprotected.catalogFile, 0o666);
expect(() => loadRuntimeModelCatalog(unprotected.catalogFile)).toThrow("runtime model catalog is unavailable");
});
test("rejects authentication semantics that cannot come from the installation catalog", () => {
const invalid = runtimeCatalog({
defaultMetadataGeneration: undefined,
models: [{
id: "zai/glm-5.3",
provider: "zai",
model: "glm-5.3",
label: "GLM 5.3",
upstreamModel: "glm-5.3",
authentication: { mode: "secret_env" },
sessionAdapter: { mode: "pi_builtin" },
session: { reasoning: true },
}],
}); });
expect(() => loadRuntimeModelCatalog(invalid.catalogFile)).toThrow("runtime model catalog is invalid");
}); });
test.each([ test("fails closed for missing or unusable provider secrets", () => {
["invalid YAML", "metadataGeneration: [\n", "OPENAI_API_KEY=secret\n", /invalid YAML/], const missing = runtimeCatalog({}, "THT_DWH_API_KEY=other\n");
["duplicate ids", `metadataGeneration: expect(() => loadMetadataGenerationModels(missing)).toThrow('secret "OPENAI_API_KEY" is missing');
default: openai-mini
models: const unusable = runtimeCatalog({}, "OPENAI_API_KEY=contains whitespace\n");
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY} expect(() => loadMetadataGenerationModels(unusable)).toThrow('secret "OPENAI_API_KEY" is unusable');
- {id: openai-mini, label: Two, litellm: {provider: openai, model: gpt-4.1}, apiKeyEnv: OPENAI_API_KEY}
`, "OPENAI_API_KEY=secret\n", /model id "openai-mini" is duplicated/],
["missing default", `metadataGeneration:
models:
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY}
`, "OPENAI_API_KEY=secret\n", /default is required/],
["unknown default", `metadataGeneration:
default: absent
models:
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY}
`, "OPENAI_API_KEY=secret\n", /default "absent" is not configured/],
["malformed settings", `metadataGeneration:
default: openai-mini
models:
- {id: openai-mini, label: One, litellm: {provider: "open ai", model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY}
`, "OPENAI_API_KEY=secret\n", /configuration is invalid/],
["malformed endpoint", `metadataGeneration:
default: openai-mini
models:
- id: openai-mini
label: One
litellm: {provider: openai, model: gpt-4.1-mini, endpoint: {baseUrl: not-a-url}}
apiKeyEnv: OPENAI_API_KEY
`, "OPENAI_API_KEY=secret\n", /configuration is invalid/],
["keyless hosted model without endpoint", `metadataGeneration:
default: openai-mini
models:
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}}
`, "", /configuration is invalid/],
["disable thinking without endpoint", `metadataGeneration:
default: openai-mini
models:
- id: openai-mini
label: One
litellm: {provider: openai, model: gpt-4.1-mini, disableThinking: true}
apiKeyEnv: OPENAI_API_KEY
`, "OPENAI_API_KEY=secret\n", /configuration is invalid/],
["unallowed secret reference", `metadataGeneration:
default: openai-mini
models:
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: THT_DWH_API_KEY}
`, "THT_DWH_API_KEY=secret\n", /configuration is invalid/],
["missing referenced secret", `metadataGeneration:
default: openai-mini
models:
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY}
`, "THT_DWH_API_KEY=secret\n", /secret "OPENAI_API_KEY" is missing/],
["unusable referenced secret", `metadataGeneration:
default: openai-mini
models:
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY}
`, "OPENAI_API_KEY=secret with whitespace\n", /secret "OPENAI_API_KEY" is unusable/],
] as const)("rejects %s metadata-generation configuration", (_name, yaml, secrets, expected) => {
const { installationFile, secretsFile } = metadataConfiguration(yaml, secrets);
expect(() => loadMetadataGenerationModels({ installationFile, secretsFile })).toThrow(expected);
}); });
test("rejects a missing secret-bundle declaration for configured models", () => { test("splits canonical session identities without provider aliases", () => {
const { installationFile } = metadataConfiguration(`metadataGeneration: expect(splitCanonicalModelId("zai/glm-5.3")).toEqual({ provider: "zai", model: "glm-5.3" });
default: openai-mini expect(() => splitCanonicalModelId("glm-5.3")).toThrow("model identity is invalid");
models:
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY}
`);
expect(() => loadMetadataGenerationModels({ installationFile }))
.toThrow("metadata-generation keyed models require THT_SECRETS_FILE");
}); });
+18 -108
View File
@@ -1,13 +1,13 @@
import { mkdtempSync, readFileSync, readdirSync, rmSync } from "node:fs"; import { mkdtempSync } from "node:fs";
import { tmpdir } from "node:os"; import { tmpdir } from "node:os";
import { join } from "node:path"; import { join } from "node:path";
import { expect, test, vi } from "vitest"; import { expect, test, vi } from "vitest";
import { loadConfig } from "../src/config.js"; import { loadConfig } from "../src/config.js";
import { import {
PiManagementError,
createPiManagement, createPiManagement,
type PiExecFile, type PiExecFile,
} from "../src/pi/management.js"; } from "../src/pi/management.js";
import type { RuntimeModelCatalog } from "../src/models/runtime-model-catalog.js";
function configFor(settingsFile = join(mkdtempSync(join(tmpdir(), "tht-pi-management-")), "settings.json")) { function configFor(settingsFile = join(mkdtempSync(join(tmpdir(), "tht-pi-management-")), "settings.json")) {
return loadConfig({ return loadConfig({
@@ -18,10 +18,14 @@ function configFor(settingsFile = join(mkdtempSync(join(tmpdir(), "tht-pi-manage
}); });
} }
const supportedModels = [ const modelCatalog: RuntimeModelCatalog = {
{ provider: "zai", id: "glm-5.2", name: "GLM 5.2", reasoning: true }, defaultSession: "zai/glm-5.2",
{ provider: "deepseek", id: "deepseek-v4", name: "DeepSeek V4", reasoning: true }, defaultMetadataGeneration: null,
]; embedding: { id: "ollama/qwen3-embedding:0.6b", dimensions: 1024 },
sessionModels: () => [],
metadataModels: () => [],
hasSession: (id) => id === "zai/glm-5.2",
};
function successfulExec(calls: Array<{ command: string; args: string[]; timeout: number }>): PiExecFile { function successfulExec(calls: Array<{ command: string; args: string[]; timeout: number }>): PiExecFile {
return async (command, args, options) => { return async (command, args, options) => {
@@ -36,7 +40,7 @@ test("status parses only a Pi version from a fixed execFile argument array", asy
const calls: Array<{ command: string; args: string[]; timeout: number }> = []; const calls: Array<{ command: string; args: string[]; timeout: number }> = [];
const service = createPiManagement(configFor(), { const service = createPiManagement(configFor(), {
execute: successfulExec(calls), execute: successfulExec(calls),
listModels: async () => supportedModels, modelCatalog,
readSettings: () => ({ provider: "zai", model: "glm-5.2", thinking: "medium" }), readSettings: () => ({ provider: "zai", model: "glm-5.2", thinking: "medium" }),
credentialStatus: () => "missing", credentialStatus: () => "missing",
now: () => new Date("2026-08-05T10:00:00.000Z"), now: () => new Date("2026-08-05T10:00:00.000Z"),
@@ -63,7 +67,7 @@ test.each(["present", "missing"] as const)(
const checkedProviders: Array<string | undefined> = []; const checkedProviders: Array<string | undefined> = [];
const service = createPiManagement(configFor(), { const service = createPiManagement(configFor(), {
execute: successfulExec([]), execute: successfulExec([]),
listModels: async () => supportedModels, modelCatalog,
readSettings: () => ({ provider: "zai", model: "glm-5.2", thinking: "medium" }), readSettings: () => ({ provider: "zai", model: "glm-5.2", thinking: "medium" }),
credentialStatus: (provider) => { credentialStatus: (provider) => {
checkedProviders.push(provider); checkedProviders.push(provider);
@@ -85,100 +89,6 @@ test.each(["present", "missing"] as const)(
}, },
); );
// Catches an options response that leaks provider metadata or lets callers choose model IDs that
// Pi did not explicitly enable for this installation.
test("options expose only closed provider, model, and reasoning choices", async () => {
const service = createPiManagement(configFor(), {
execute: successfulExec([]),
listModels: async () => supportedModels,
now: () => new Date("2026-08-05T10:00:00.000Z"),
});
await expect(service.options()).resolves.toEqual({
providers: ["zai", "deepseek"],
models: [
{ provider: "zai", id: "glm-5.2" },
{ provider: "deepseek", id: "deepseek-v4" },
],
reasoning: ["low", "medium", "high"],
checkedAt: "2026-08-05T10:00:00.000Z",
});
});
// Catches raw managed models.json validation details being collapsed into an ambiguous model-list
// failure or escaping through the Pi Management options API.
test("options report invalid managed model configuration with a stable sanitized error", async () => {
const service = createPiManagement(configFor(), {
execute: successfulExec([]),
listModels: async () => {
throw Object.assign(
new Error("!sensitive-command /private/models.json raw-secret"),
{ code: "PI_MANAGED_CONFIG_INVALID" },
);
},
});
let caught: unknown;
try {
await service.options();
} catch (error) {
caught = error;
}
expect(caught).toMatchObject<PiManagementError>({
code: "pi_management_unavailable",
message: "Pi provider/model configuration is invalid",
});
expect(String(caught)).not.toMatch(/sensitive|private|models\.json|secret/i);
});
// Catches configuration writes that accept whitespace, unknown choices, or extra free-form fields
// before reaching the durable installation settings file.
test("config rejects invalid free-form values before writing settings", async () => {
const directory = mkdtempSync(join(tmpdir(), "tht-pi-management-invalid-"));
try {
let writes = 0;
const service = createPiManagement(configFor(join(directory, "settings.json")), {
execute: successfulExec([]),
listModels: async () => supportedModels,
readSettings: () => ({}),
saveSettings: () => { writes += 1; return {}; },
});
await expect(service.configure({
provider: "zai ", model: "glm-5.2", reasoning: "medium", unexpected: "value",
} as any)).rejects.toMatchObject<PiManagementError>({ code: "pi_management_invalid_config" });
expect(writes).toBe(0);
} finally {
rmSync(directory, { recursive: true, force: true });
}
});
// Catches a non-atomic implementation that can leave partial settings or temporary files after a
// normal installation-default update.
test("config validates closed choices and atomically persists non-secret defaults", async () => {
const directory = mkdtempSync(join(tmpdir(), "tht-pi-management-write-"));
const settingsFile = join(directory, "settings.json");
try {
const service = createPiManagement(configFor(settingsFile), {
execute: successfulExec([]),
listModels: async () => supportedModels,
now: () => new Date("2026-08-05T10:00:00.000Z"),
});
await expect(service.configure({
provider: "zai", model: "glm-5.2", reasoning: "high",
})).resolves.toEqual({
provider: "zai", model: "glm-5.2", reasoning: "high", updatedAt: "2026-08-05T10:00:00.000Z",
});
expect(JSON.parse(readFileSync(settingsFile, "utf8"))).toEqual({
provider: "zai", model: "glm-5.2", thinking: "high",
});
expect(readdirSync(directory)).toEqual(["settings.json"]);
} finally {
rmSync(directory, { recursive: true, force: true });
}
});
// Catches a hung Pi smoke check that leaves an operator waiting indefinitely or returns raw child // Catches a hung Pi smoke check that leaves an operator waiting indefinitely or returns raw child
// diagnostics containing provider credentials. // diagnostics containing provider credentials.
test("smoke uses the configured timeout and reports a sanitized timeout", async () => { test("smoke uses the configured timeout and reports a sanitized timeout", async () => {
@@ -188,7 +98,7 @@ test("smoke uses the configured timeout and reports a sanitized timeout", async
calls.push({ command, args, timeout: options.timeout }); calls.push({ command, args, timeout: options.timeout });
throw Object.assign(new Error("provider token=raw-provider-token"), { code: "ETIMEDOUT" }); throw Object.assign(new Error("provider token=raw-provider-token"), { code: "ETIMEDOUT" });
}, },
listModels: async () => supportedModels, modelCatalog,
readSettings: () => ({ provider: "zai", model: "glm-5.2", thinking: "medium" }), readSettings: () => ({ provider: "zai", model: "glm-5.2", thinking: "medium" }),
now: () => new Date("2026-08-05T10:00:00.000Z"), now: () => new Date("2026-08-05T10:00:00.000Z"),
}); });
@@ -210,7 +120,7 @@ test("smoke exercises the configured provider and model", async () => {
const providerChecks: unknown[] = []; const providerChecks: unknown[] = [];
const service = createPiManagement(configFor(), { const service = createPiManagement(configFor(), {
execute: successfulExec([]), execute: successfulExec([]),
listModels: async () => supportedModels, modelCatalog,
readSettings: () => ({ provider: "zai", model: "glm-5.2", thinking: "medium" }), readSettings: () => ({ provider: "zai", model: "glm-5.2", thinking: "medium" }),
smokeProvider: async (request) => { providerChecks.push(request); }, smokeProvider: async (request) => { providerChecks.push(request); },
now: () => new Date("2026-08-05T10:00:00.000Z"), now: () => new Date("2026-08-05T10:00:00.000Z"),
@@ -230,7 +140,7 @@ test("smoke exercises the configured provider and model", async () => {
test("smoke fails closed and sanitizes configured-provider authentication errors", async () => { test("smoke fails closed and sanitizes configured-provider authentication errors", async () => {
const service = createPiManagement(configFor(), { const service = createPiManagement(configFor(), {
execute: successfulExec([]), execute: successfulExec([]),
listModels: async () => supportedModels, modelCatalog,
readSettings: () => ({ provider: "zai", model: "glm-5.2", thinking: "medium" }), readSettings: () => ({ provider: "zai", model: "glm-5.2", thinking: "medium" }),
smokeProvider: async () => { smokeProvider: async () => {
throw new Error('401 {"token":"raw-expired-token","output":"raw-provider-output"}'); throw new Error('401 {"token":"raw-expired-token","output":"raw-provider-output"}');
@@ -252,7 +162,7 @@ test("smoke fails closed and sanitizes configured-provider authentication errors
test("smoke reports invalid managed provider configuration with a stable sanitized error", async () => { test("smoke reports invalid managed provider configuration with a stable sanitized error", async () => {
const service = createPiManagement(configFor(), { const service = createPiManagement(configFor(), {
execute: successfulExec([]), execute: successfulExec([]),
listModels: async () => supportedModels, modelCatalog,
readSettings: () => ({ provider: "zai", model: "glm-5.2", thinking: "medium" }), readSettings: () => ({ provider: "zai", model: "glm-5.2", thinking: "medium" }),
smokeProvider: async () => { smokeProvider: async () => {
throw Object.assign( throw Object.assign(
@@ -284,7 +194,7 @@ test("smoke applies one deadline across version and a hung provider turn", async
() => resolve({ stdout: "pi 0.80.3\n", stderr: "" }), () => resolve({ stdout: "pi 0.80.3\n", stderr: "" }),
500, 500,
)), )),
listModels: async () => supportedModels, modelCatalog,
readSettings: () => ({ provider: "zai", model: "glm-5.2", thinking: "medium" }), readSettings: () => ({ provider: "zai", model: "glm-5.2", thinking: "medium" }),
smokeProvider: async ({ timeoutMs }) => { smokeProvider: async ({ timeoutMs }) => {
providerTimeouts.push(timeoutMs); providerTimeouts.push(timeoutMs);
@@ -320,7 +230,7 @@ test("logs keep only the latest 200 redacted lines", async () => {
source[201] = "THT_MODEL_API_KEY=raw-env-secret"; source[201] = "THT_MODEL_API_KEY=raw-env-secret";
const service = createPiManagement(configFor(), { const service = createPiManagement(configFor(), {
execute: successfulExec([]), execute: successfulExec([]),
listModels: async () => supportedModels, modelCatalog,
readLogs: () => source.join("\n"), readLogs: () => source.join("\n"),
now: () => new Date("2026-08-05T10:00:00.000Z"), now: () => new Date("2026-08-05T10:00:00.000Z"),
}); });
+48
View File
@@ -8,6 +8,7 @@ import {
} from "node:fs"; } from "node:fs";
import { tmpdir } from "node:os"; import { tmpdir } from "node:os";
import { PiProcessManager } from "../src/pi/pi-process-manager.js"; import { PiProcessManager } from "../src/pi/pi-process-manager.js";
import type { RuntimeModelCatalog, RuntimeModel } from "../src/models/runtime-model-catalog.js";
import { loadConfig } from "../src/config.js"; import { loadConfig } from "../src/config.js";
import { import {
PI_MANAGED_CONFIG_ERROR_MESSAGE, PI_MANAGED_CONFIG_ERROR_MESSAGE,
@@ -641,6 +642,53 @@ test("session Pi spawn reads the single secret bundle and scrubs its path", asyn
} }
}); });
test("session Pi spawn resolves the selected catalog credential from the secret bundle", () => {
const root = mkdtempSync(path.join(tmpdir(), "thothii-catalog-credential-"));
const agentDir = path.join(root, "agent");
mkdirSync(agentDir, { mode: 0o700 });
writeFileSync(path.join(agentDir, "auth.json"), "{}\n", { mode: 0o600 });
writeFileSync(path.join(agentDir, "models.json"), '{"providers":{}}\n', { mode: 0o600 });
const secret = path.join(root, "thothii.secrets");
writeFileSync(secret, "ZAI_API_KEY=catalog-secret\nTHT_MODEL_API_KEY=legacy-secret\n", { mode: 0o600 });
const model: RuntimeModel = {
id: "openai/test-model",
provider: "openai",
model: "test-model",
label: "Test model",
upstreamModel: "test-model",
authentication: { mode: "secret_env", apiKeyEnv: "ZAI_API_KEY" },
sessionAdapter: { mode: "pi_builtin" },
session: { reasoning: false },
};
const modelCatalog: RuntimeModelCatalog = {
defaultSession: model.id,
defaultMetadataGeneration: null,
embedding: null,
sessionModels: () => [model],
metadataModels: () => [],
hasSession: (id) => id === model.id,
};
const calls: any[][] = [];
const child = recordingChild();
child.stderr.resume = () => {};
vi.stubEnv("PI_CODING_AGENT_DIR", agentDir);
const mgr = new PiProcessManager(loadConfig({ THT_SECRETS_FILE: secret }), {
modelCatalog,
authProviders: () => new Set(),
spawnFn: (...args: any[]) => { calls.push(args); return child as any; },
});
try {
mgr.createFor("catalog-credential", { provider: "openai", model: "test-model" });
expect(calls[0][2].env.ZAI_API_KEY).toBe("catalog-secret");
expect(calls[0][2].env).not.toHaveProperty("OPENAI_API_KEY");
expect(calls[0][2].env).not.toHaveProperty("THT_MODEL_API_KEY");
} finally {
mgr.teardown("catalog-credential");
vi.unstubAllEnvs();
rmSync(root, { recursive: true, force: true });
}
});
test.each([["OpenAI", "openai"], ["gemini", "google"]])( test.each([["OpenAI", "openai"], ["gemini", "google"]])(
"set_model uses canonical packaged provider ID for %s", async (provider, canonical) => { "set_model uses canonical packaged provider ID for %s", async (provider, canonical) => {
const secret = path.resolve(__dirname, `.canonical-key-${process.pid}-${provider}`); const secret = path.resolve(__dirname, `.canonical-key-${process.pid}-${provider}`);
+48 -2
View File
@@ -1,9 +1,11 @@
import { EventEmitter } from "node:events"; import { EventEmitter } from "node:events";
import { existsSync, readFileSync, readdirSync } from "node:fs"; import { existsSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from "node:fs";
import { dirname } from "node:path"; import { tmpdir } from "node:os";
import { dirname, join } from "node:path";
import { afterEach, expect, test, vi } from "vitest"; import { afterEach, expect, test, vi } from "vitest";
import { loadConfig } from "../src/config.js"; import { loadConfig } from "../src/config.js";
import { createPiProviderSmoke } from "../src/pi/provider-smoke.js"; import { createPiProviderSmoke } from "../src/pi/provider-smoke.js";
import type { RuntimeModel, RuntimeModelCatalog } from "../src/models/runtime-model-catalog.js";
afterEach(() => vi.unstubAllEnvs()); afterEach(() => vi.unstubAllEnvs());
@@ -44,6 +46,50 @@ function successfulProviderChild() {
const MANAGED_CONFIG_ERROR = "Pi provider/model configuration is invalid"; const MANAGED_CONFIG_ERROR = "Pi provider/model configuration is invalid";
test("provider smoke resolves the selected catalog credential from the secret bundle", async () => {
const root = mkdtempSync(join(tmpdir(), "thothii-smoke-catalog-credential-"));
const secret = join(root, "thothii.secrets");
writeFileSync(secret, "ZAI_API_KEY=catalog-secret\nTHT_MODEL_API_KEY=legacy-secret\n", { mode: 0o600 });
const model: RuntimeModel = {
id: "openai/test-model",
provider: "openai",
model: "test-model",
label: "Test model",
upstreamModel: "test-model",
authentication: { mode: "secret_env", apiKeyEnv: "ZAI_API_KEY" },
sessionAdapter: { mode: "pi_builtin" },
session: { reasoning: false },
};
const modelCatalog: RuntimeModelCatalog = {
defaultSession: model.id,
defaultMetadataGeneration: null,
embedding: null,
sessionModels: () => [model],
metadataModels: () => [],
hasSession: (id) => id === model.id,
};
let spawnEnv: NodeJS.ProcessEnv | undefined;
const smoke = createPiProviderSmoke(loadConfig({ THT_SECRETS_FILE: secret }), {
modelCatalog,
authProviders: () => new Set(),
readModelsStore: () => undefined,
spawnFn: (_command, _args, options) => {
spawnEnv = options.env;
return successfulProviderChild();
},
});
try {
await expect(smoke({
provider: "openai", model: "test-model", reasoning: "medium", timeoutMs: 750,
})).resolves.toBeUndefined();
expect(spawnEnv?.ZAI_API_KEY).toBe("catalog-secret");
expect(spawnEnv).not.toHaveProperty("OPENAI_API_KEY");
expect(spawnEnv).not.toHaveProperty("THT_MODEL_API_KEY");
} finally {
rmSync(root, { recursive: true, force: true });
}
});
// Catches an isolated smoke agent that copies auth.json but drops the selected custom // Catches an isolated smoke agent that copies auth.json but drops the selected custom
// provider/model from models.json, causing set_model to fail before the real request. // provider/model from models.json, causing set_model to fail before the real request.
test("provider smoke reaches the selected custom provider from an isolated models.json", async () => { test("provider smoke reaches the selected custom provider from an isolated models.json", async () => {
+1 -8
View File
@@ -2,18 +2,11 @@ import { expect, test } from "vitest";
import { ReadinessManager } from "../src/runtime/readiness-manager.js"; import { ReadinessManager } from "../src/runtime/readiness-manager.js";
const workspace = { const workspace = {
workspace: { schema_version: 3, id: "psd", name: "PSD", language: "it" }, workspace: { schema_version: 4, id: "psd", name: "PSD", language: "it" },
dwh: { dwh: {
engine: "postgres", database: "warehouse", schema: "public", engine: "postgres", database: "warehouse", schema: "public",
supported_transports: ["postgres_direct"], supported_transports: ["postgres_direct"],
}, },
semantic_index: {
vector_store: { engine: "qdrant", collection: "psd", dimensions: 1024, distance: "cosine" },
embedding: {
provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024,
},
},
llm_policy: { allowed: ["zai/glm-5.2"] },
} as const; } as const;
function deferred<T>() { function deferred<T>() {
+1 -13
View File
@@ -15,7 +15,7 @@ afterEach(() => {
}); });
const validYaml = `workspace: const validYaml = `workspace:
schema_version: 3 schema_version: 4
id: psd-clinical id: psd-clinical
name: Policlinico San Donato name: Policlinico San Donato
language: it language: it
@@ -24,18 +24,6 @@ dwh:
database: postgres database: postgres
schema: datawarehouse schema: datawarehouse
supported_transports: [postgres_direct] supported_transports: [postgres_direct]
semantic_index:
vector_store:
engine: qdrant
collection: psd-clinical
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
llm_policy:
allowed: [zai/glm-5.2]
`; `;
async function git(cwd: string, args: string[]): Promise<string> { async function git(cwd: string, args: string[]): Promise<string> {
+1 -13
View File
@@ -20,7 +20,7 @@ async function git(cwd: string, args: string[]): Promise<string> {
} }
const descriptor = `workspace: const descriptor = `workspace:
schema_version: 3 schema_version: 4
id: research id: research
name: Research name: Research
language: en language: en
@@ -29,18 +29,6 @@ dwh:
database: analytics database: analytics
schema: mart schema: mart
supported_transports: [postgres_direct] supported_transports: [postgres_direct]
semantic_index:
vector_store:
engine: qdrant
collection: research
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
llm_policy:
allowed: [zai/glm-5.2]
evidence: evidence:
source: source:
type: filesystem type: filesystem
+9 -27
View File
@@ -14,11 +14,6 @@ function fakeService(): PiManagementService {
config: { provider: "zai", model: "glm-5.2", reasoning: "medium" }, config: { provider: "zai", model: "glm-5.2", reasoning: "medium" },
checkedAt: "2026-08-05T10:00:00.000Z", checkedAt: "2026-08-05T10:00:00.000Z",
})), })),
options: vi.fn(async () => ({
providers: ["zai"], models: [{ provider: "zai", id: "glm-5.2" }],
reasoning: ["low", "medium", "high"], checkedAt: "2026-08-05T10:00:00.000Z",
})),
configure: vi.fn(async (value) => ({ ...value, updatedAt: "2026-08-05T10:00:00.000Z" })),
test: vi.fn(async () => ({ ready: true, checkedAt: "2026-08-05T10:00:00.000Z" })), test: vi.fn(async () => ({ ready: true, checkedAt: "2026-08-05T10:00:00.000Z" })),
logs: vi.fn(async () => ({ lines: ["Pi smoke check succeeded"], checkedAt: "2026-08-05T10:00:00.000Z" })), logs: vi.fn(async () => ({ lines: ["Pi smoke check succeeded"], checkedAt: "2026-08-05T10:00:00.000Z" })),
}; };
@@ -106,24 +101,17 @@ test("loopback-only AUTH_MODE=none may read the sanitized Pi status", async () =
}); });
// A local implicit administrator has pi.manage, but a browser origin still cannot borrow that // A local implicit administrator has pi.manage, but a browser origin still cannot borrow that
// authority to mutate local configuration or trigger provider work. // authority to trigger provider work.
test("loopback-only management rejects cross-origin writes for its local administrator", async () => { test("loopback-only management rejects cross-origin writes for its local administrator", async () => {
const service = fakeService(); const service = fakeService();
const app = appWith(service); const app = appWith(service);
try { try {
const configured = await app.inject({
method: "PUT", url: "/pi-management/config",
headers: { host: "127.0.0.1:8080", origin: "https://evil.example" },
payload: { provider: "zai", model: "glm-5.2", reasoning: "high" },
});
const smoke = await app.inject({ const smoke = await app.inject({
method: "POST", url: "/pi-management/test", method: "POST", url: "/pi-management/test",
headers: { host: "127.0.0.1:8080", origin: "https://evil.example" }, headers: { host: "127.0.0.1:8080", origin: "https://evil.example" },
}); });
expect(configured.statusCode).toBe(403);
expect(smoke.statusCode).toBe(403); expect(smoke.statusCode).toBe(403);
expect(service.configure).not.toHaveBeenCalled();
expect(service.test).not.toHaveBeenCalled(); expect(service.test).not.toHaveBeenCalled();
} finally { } finally {
await app.close(); await app.close();
@@ -132,14 +120,13 @@ test("loopback-only management rejects cross-origin writes for its local adminis
// Catches an origin guard that also blocks the same-origin Docker frontend or non-browser local // Catches an origin guard that also blocks the same-origin Docker frontend or non-browser local
// lifecycle clients that do not send Origin. // lifecycle clients that do not send Origin.
test("loopback-only management preserves same-origin frontend and origin-less local writes", async () => { test("loopback-only management preserves same-origin and origin-less smoke checks", async () => {
const service = fakeService(); const service = fakeService();
const app = appWith(service); const app = appWith(service);
try { try {
const sameOrigin = await app.inject({ const sameOrigin = await app.inject({
method: "PUT", url: "/pi-management/config", method: "POST", url: "/pi-management/test",
headers: { host: "127.0.0.1:8080", origin: "http://127.0.0.1:8080" }, headers: { host: "127.0.0.1:8080", origin: "http://127.0.0.1:8080" },
payload: { provider: "zai", model: "glm-5.2", reasoning: "high" },
}); });
const lifecycleClient = await app.inject({ method: "POST", url: "/pi-management/test" }); const lifecycleClient = await app.inject({ method: "POST", url: "/pi-management/test" });
@@ -150,25 +137,20 @@ test("loopback-only management preserves same-origin frontend and origin-less lo
} }
}); });
// Catches route wiring that bypasses closed service validation or gives the browser a Docker/image // Catches a regression that reintroduces a browser-writable provider/model source.
// lifecycle endpoint rather than only installation-default configuration and diagnostics. test("trusted admins receive only status, smoke, and log endpoints", async () => {
test("trusted admins receive only configuration, smoke, options, and log endpoints", async () => {
const service = fakeService(); const service = fakeService();
const app = appWith(service, exposedServerEnv); const app = appWith(service, exposedServerEnv);
try { try {
const options = await app.inject({ method: "GET", url: "/pi-management/options", headers: adminHeaders }); const status = await app.inject({ method: "GET", url: "/pi-management/status", headers: adminHeaders });
const configured = await app.inject({
method: "PUT", url: "/pi-management/config", headers: adminHeaders,
payload: { provider: "zai", model: "glm-5.2", reasoning: "high" },
});
const smoke = await app.inject({ method: "POST", url: "/pi-management/test", headers: adminHeaders }); const smoke = await app.inject({ method: "POST", url: "/pi-management/test", headers: adminHeaders });
const logs = await app.inject({ method: "GET", url: "/pi-management/logs", headers: adminHeaders }); const logs = await app.inject({ method: "GET", url: "/pi-management/logs", headers: adminHeaders });
expect(options.statusCode).toBe(200); expect(status.statusCode).toBe(200);
expect(configured.statusCode).toBe(200);
expect(configured.json()).toMatchObject({ provider: "zai", model: "glm-5.2", reasoning: "high" });
expect(smoke.statusCode).toBe(200); expect(smoke.statusCode).toBe(200);
expect(logs.statusCode).toBe(200); expect(logs.statusCode).toBe(200);
expect((await app.inject({ method: "GET", url: "/pi-management/options", headers: adminHeaders })).statusCode).toBe(404);
expect((await app.inject({ method: "PUT", url: "/pi-management/config", headers: adminHeaders })).statusCode).toBe(404);
expect(app.printRoutes()).not.toContain("update"); expect(app.printRoutes()).not.toContain("update");
expect(app.printRoutes()).not.toContain("rollback"); expect(app.printRoutes()).not.toContain("rollback");
} finally { } finally {
+45 -44
View File
@@ -18,25 +18,25 @@ const SCRIPT = path.resolve("../harness/tests/fake_pi/scripts/f1_disambiguation.
function operationalWorkspace(id = "default") { function operationalWorkspace(id = "default") {
return { return {
workspace: { schema_version: 3, id, name: id, language: "en" }, workspace: { schema_version: 4, id, name: id, language: "en" },
dwh: { dwh: {
engine: "postgres", database: "warehouse", schema: "public", engine: "postgres", database: "warehouse", schema: "public",
supported_transports: ["postgres_direct"], supported_transports: ["postgres_direct"],
}, },
semantic_index: {
vector_store: {
engine: "qdrant", collection: id, dimensions: 1024, distance: "cosine",
},
embedding: {
provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024,
},
},
llm_policy: {
allowed: ["zai/glm-5.2", "deepseek/deepseek-v4-pro", "local-qwen/qwen3.6-35b-a3b"],
},
} as const; } as const;
} }
function sessionCatalog(defaultSession = "zai/glm-5.2", available = [defaultSession]) {
return {
defaultSession,
defaultMetadataGeneration: null,
embedding: { id: "ollama/qwen3-embedding:0.6b", dimensions: 1024 },
sessionModels: () => [],
metadataModels: () => [],
hasSession: (id: string) => available.includes(id),
} as any;
}
const defaultWorkspaceRegistry = { const defaultWorkspaceRegistry = {
list: vi.fn(async () => [{ list: vi.fn(async () => [{
id: "default", commit: "e".repeat(40), blob: "f".repeat(40), id: "default", commit: "e".repeat(40), blob: "f".repeat(40),
@@ -495,8 +495,8 @@ test("creates a session from the active immutable workspace revision", async ()
workspaceRegistry: { workspaceRegistry: {
read: vi.fn(async () => ({ read: vi.fn(async () => ({
workspace: { workspace: {
workspace: { schema_version: 2, id: "psd-clinical", name: "PSD", language: "it" }, workspace: { schema_version: 4, id: "psd-clinical", name: "PSD", language: "it" },
dwh: {}, semantic_index: {}, llm_policy: { allowed: ["zai/glm-5.2"] }, dwh: {},
}, },
revision: { revision: {
id: "psd-clinical", commit: "a".repeat(40), blob: "b".repeat(40), id: "psd-clinical", commit: "a".repeat(40), blob: "b".repeat(40),
@@ -589,22 +589,11 @@ test("rejects an SSH-only workspace before persisting or starting a session", as
workspaceRegistry: { workspaceRegistry: {
acquireSessionRevision: vi.fn(async () => ({ acquireSessionRevision: vi.fn(async () => ({
workspace: { workspace: {
workspace: { schema_version: 2, id: "ssh-workspace", name: "SSH", language: "en" }, workspace: { schema_version: 4, id: "ssh-workspace", name: "SSH", language: "en" },
dwh: { dwh: {
engine: "postgres", database: "postgres", schema: "public", engine: "postgres", database: "postgres", schema: "public",
supported_transports: ["ssh_tunnel"], supported_transports: ["ssh_tunnel"],
}, },
semantic_index: {
vector_store: {
engine: "pgvector", database: "postgres", schema: "vectors",
collection: "documents", dimensions: 768, distance: "cosine",
supported_transports: ["ssh_tunnel"],
},
embedding: {
provider: "ollama_compatible", model: "nomic-embed-text", dimensions: 768,
},
},
llm_policy: { allowed: ["zai/glm-5.2"] },
}, },
revision: { revision: {
id: "ssh-workspace", commit: "a".repeat(40), blob: "b".repeat(40), id: "ssh-workspace", commit: "a".repeat(40), blob: "b".repeat(40),
@@ -632,7 +621,7 @@ test("hands a revision lease to retention only after the session manifest is dur
const markPersisted = vi.fn(async () => {}); const markPersisted = vi.fn(async () => {});
const abort = vi.fn(async () => {}); const abort = vi.fn(async () => {});
const acquireSessionRevision = vi.fn(async () => ({ const acquireSessionRevision = vi.fn(async () => ({
workspace: { llm_policy: { allowed: ["zai/glm-5.2"] } }, workspace: operationalWorkspace("leased"),
revision: { revision: {
id: "leased", commit: "a".repeat(40), blob: "b".repeat(40), id: "leased", commit: "a".repeat(40), blob: "b".repeat(40),
snapshotPath: `/data/workspace-registry/snapshots/${"a".repeat(40)}/leased.yaml`, snapshotPath: `/data/workspace-registry/snapshots/${"a".repeat(40)}/leased.yaml`,
@@ -670,7 +659,7 @@ test("creates a session from the configured default workspace revision when work
const sessionNew = vi.fn(async () => ({ id: "default-pinned" })); const sessionNew = vi.fn(async () => ({ id: "default-pinned" }));
const registry = { const registry = {
read: vi.fn(async (id: string) => ({ read: vi.fn(async (id: string) => ({
workspace: { llm_policy: { allowed: ["zai/glm-5.2"] } }, workspace: operationalWorkspace(id),
revision: { revision: {
id, commit: "c".repeat(40), blob: "d".repeat(40), id, commit: "c".repeat(40), blob: "d".repeat(40),
snapshotPath: `/data/workspace-registry/snapshots/${"c".repeat(40)}/${id}.yaml`, snapshotPath: `/data/workspace-registry/snapshots/${"c".repeat(40)}/${id}.yaml`,
@@ -758,7 +747,7 @@ test("session lifecycle locates a B session when installation default is A", asy
listModels: async () => [{ provider: "zai", id: "glm-5.2", name: "GLM 5.2", reasoning: true }], listModels: async () => [{ provider: "zai", id: "glm-5.2", name: "GLM 5.2", reasoning: true }],
workspaceRegistry: { workspaceRegistry: {
read: async (id: string) => ({ read: async (id: string) => ({
workspace: { llm_policy: { allowed: ["zai/glm-5.2"] } }, workspace: operationalWorkspace(id),
revision: { id, commit: "b".repeat(40), blob: "d".repeat(40), snapshotPath: bPath }, revision: { id, commit: "b".repeat(40), blob: "d".repeat(40), snapshotPath: bPath },
}), }),
list: async () => [ list: async () => [
@@ -793,7 +782,7 @@ test("session lifecycle locates a B session when installation default is A", asy
expect(runtimeOptions).toEqual(["/runtime/1.yaml", "/runtime/2.yaml"]); expect(runtimeOptions).toEqual(["/runtime/1.yaml", "/runtime/2.yaml"]);
}); });
test("POST /sessions usa i settings (workspace/provider/model/thinking) e crea+avvia", async () => { test("POST /sessions uses the catalog default with workspace/thinking settings and starts", async () => {
const modelKey = path.join(os.tmpdir(), `thoth-model-key-${process.pid}`); const modelKey = path.join(os.tmpdir(), `thoth-model-key-${process.pid}`);
writeFileSync(modelKey, "test-model-key", { mode: 0o600 }); writeFileSync(modelKey, "test-model-key", { mode: 0o600 });
chmodSync(modelKey, 0o600); chmodSync(modelKey, 0o600);
@@ -808,7 +797,8 @@ test("POST /sessions usa i settings (workspace/provider/model/thinking) e crea+a
sessionNew: async (o: any) => { sessionNewArg = o; return { id: "s1" }; }, sessionNew: async (o: any) => { sessionNewArg = o; return { id: "s1" }; },
sessionList: async () => [{ id: "s1" }], sessionList: async () => [{ id: "s1" }],
} as any, } as any,
getSettings: () => ({ workspace: "w", provider: "zai", model: "glm-5.2", thinking: "high" }), getSettings: () => ({ workspace: "w", thinking: "high" }),
runtimeModelCatalog: sessionCatalog(),
listModels: async () => [ listModels: async () => [
{ provider: "zai", id: "glm-5.2", name: "GLM 5.2", reasoning: true }, { provider: "zai", id: "glm-5.2", name: "GLM 5.2", reasoning: true },
], ],
@@ -2565,32 +2555,43 @@ test("POST /sessions proceeds when ollamaEnsure succeeds", async () => {
expect(ensureWs).toContain(`/snapshots/${"e".repeat(40)}/psd.yaml`); expect(ensureWs).toContain(`/snapshots/${"e".repeat(40)}/psd.yaml`);
}); });
test("POST /sessions rejects an unavailable saved model before persisting a session", async () => { test("POST /sessions falls back from a stale requested model to the catalog default", async () => {
let created = 0; let created = 0;
let persisted: any;
const runtime = { bridge: { onClientEvent: () => {} } };
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), { const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
thtRunner: { thtRunner: {
sessionNew: async () => { created += 1; return { id: "must-not-exist" }; }, sessionNew: async (options: any) => { created += 1; persisted = options; return { id: "fallback" }; },
searchPack: async () => {},
} as any, } as any,
readiness: { ensure: async () => ({ ok: true }) } as any, readiness: { ensure: async () => ({ ok: true }) } as any,
getSettings: () => ({ getSettings: () => ({ workspace: "psd", thinking: "medium" }) as any,
workspace: "psd", runtimeModelCatalog: sessionCatalog(),
provider: "deepseek",
model: "deepseek-v4-pro",
thinking: "medium",
}) as any,
listModels: async () => [ listModels: async () => [
{ provider: "zai", id: "glm-5.2", name: "GLM 5.2", reasoning: true }, { provider: "zai", id: "glm-5.2", name: "GLM 5.2", reasoning: true },
], ],
mgr: {
teardownForPrincipal: () => [],
createFor: () => runtime,
get: () => runtime,
configure: async () => {},
start: () => {},
} as any,
}); });
const res = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } }); const res = await app.inject({
method: "POST",
url: "/sessions",
payload: { question: "q", provider: "deepseek", model: "deepseek-v4-pro" },
});
expect(res.statusCode).toBe(503); expect(res.statusCode).toBe(200);
expect(res.json()).toEqual({ expect(res.json()).toEqual({
error: "Selected model is unavailable. Check Pi authentication and model settings, then try again.", id: "fallback",
code: "model_unavailable", warning: "Configured model deepseek/deepseek-v4-pro is unavailable; using zai/glm-5.2.",
}); });
expect(created).toBe(0); expect(persisted).toMatchObject({ provider: "zai", model: "glm-5.2" });
expect(created).toBe(1);
}); });
test("POST /sessions marks a persisted session failed when runtime construction throws", async () => { test("POST /sessions marks a persisted session failed when runtime construction throws", async () => {
+16 -16
View File
@@ -21,7 +21,7 @@ function appWithTmpSettings(extraEnv: Record<string, string> = {}, deps = {}) {
return { app, dir }; return { app, dir };
} }
test("GET /settings returns effective defaults (env provider/model/thinking, first workspace)", async () => { test("GET /settings returns thinking and the first workspace without legacy model defaults", async () => {
const { app, dir } = appWithTmpSettings({ PI_PROVIDER: "zai", PI_MODEL: "glm-5.2", PI_THINKING: "medium" }, { const { app, dir } = appWithTmpSettings({ PI_PROVIDER: "zai", PI_MODEL: "glm-5.2", PI_THINKING: "medium" }, {
listModels: async () => [{ provider: "zai", id: "glm-5.2", name: "GLM 5.2", reasoning: true }], listModels: async () => [{ provider: "zai", id: "glm-5.2", name: "GLM 5.2", reasoning: true }],
}); });
@@ -29,8 +29,8 @@ test("GET /settings returns effective defaults (env provider/model/thinking, fir
const res = await app.inject({ method: "GET", url: "/settings" }); const res = await app.inject({ method: "GET", url: "/settings" });
expect(res.statusCode).toBe(200); expect(res.statusCode).toBe(200);
const body = res.json(); const body = res.json();
expect(body.provider).toBe("zai"); expect(body).not.toHaveProperty("provider");
expect(body.model).toBe("glm-5.2"); expect(body).not.toHaveProperty("model");
expect(body.thinking).toBe("medium"); expect(body.thinking).toBe("medium");
expect(typeof body.workspace).toBe("string"); // first workspace from ../harness/workspaces expect(typeof body.workspace).toBe("string"); // first workspace from ../harness/workspaces
} finally { } finally {
@@ -62,7 +62,9 @@ test("PUT /settings does not persist personal workspace or LLM choices", async (
}); });
expect(put.statusCode).toBe(200); expect(put.statusCode).toBe(200);
const got = await app.inject({ method: "GET", url: "/settings" }); const got = await app.inject({ method: "GET", url: "/settings" });
expect(got.json()).toMatchObject({ provider: "zai", model: "glm-5.2", thinking: "medium" }); expect(got.json()).toMatchObject({ thinking: "medium" });
expect(got.json()).not.toHaveProperty("provider");
expect(got.json()).not.toHaveProperty("model");
expect(got.json()).not.toMatchObject({ workspace: "psd", thinking: "high" }); expect(got.json()).not.toMatchObject({ workspace: "psd", thinking: "high" });
} finally { } finally {
rmSync(dir, { recursive: true, force: true }); rmSync(dir, { recursive: true, force: true });
@@ -114,7 +116,7 @@ test("settings no longer read or write principal-specific preferences", async ()
expect(preferences.size).toBe(0); expect(preferences.size).toBe(0);
}); });
test("GET /settings retains complete legacy installation defaults without seeding a private profile", async () => { test("GET /settings drops legacy installation model fields without seeding a private profile", async () => {
let preferences: Record<string, unknown> = {}; let preferences: Record<string, unknown> = {};
const writes: Record<string, unknown>[] = []; const writes: Record<string, unknown>[] = [];
const runner = { const runner = {
@@ -135,9 +137,7 @@ test("GET /settings retains complete legacy installation defaults without seedin
const first = await app.inject({ method: "GET", url: "/settings" }); const first = await app.inject({ method: "GET", url: "/settings" });
const second = await app.inject({ method: "GET", url: "/settings" }); const second = await app.inject({ method: "GET", url: "/settings" });
const expected = { const expected = { workspace: "local", thinking: "low" };
workspace: "local", provider: "local-qwen", model: "qwen3.6-35b-a3b", thinking: "low",
};
expect(first.statusCode).toBe(200); expect(first.statusCode).toBe(200);
expect(first.json()).toEqual(expected); expect(first.json()).toEqual(expected);
expect(second.json()).toEqual(expected); expect(second.json()).toEqual(expected);
@@ -167,15 +167,13 @@ test("GET /settings ignores stale private preferences in favor of installation d
const response = await app.inject({ method: "GET", url: "/settings" }); const response = await app.inject({ method: "GET", url: "/settings" });
expect(response.statusCode).toBe(200); expect(response.statusCode).toBe(200);
expect(response.json()).toEqual({ expect(response.json()).toEqual({ workspace: "local", thinking: "low" });
workspace: "local", provider: "local-qwen", model: "qwen3.6-35b-a3b", thinking: "low",
});
} finally { } finally {
rmSync(dir, { recursive: true, force: true }); rmSync(dir, { recursive: true, force: true });
} }
}); });
test("PUT /settings rejects an unknown model when a model list is available", async () => { test("PUT /settings ignores a legacy unknown model because the catalog owns model validity", async () => {
const { app, dir } = appWithTmpSettings({}, { const { app, dir } = appWithTmpSettings({}, {
listModels: async () => [{ provider: "zai", id: "glm-5.2", name: "GLM 5.2", reasoning: true }], listModels: async () => [{ provider: "zai", id: "glm-5.2", name: "GLM 5.2", reasoning: true }],
}); });
@@ -184,14 +182,14 @@ test("PUT /settings rejects an unknown model when a model list is available", as
method: "PUT", url: "/settings", method: "PUT", url: "/settings",
payload: { workspace: "psd", provider: "zai", model: "does-not-exist", thinking: "low" }, payload: { workspace: "psd", provider: "zai", model: "does-not-exist", thinking: "low" },
}); });
expect(put.statusCode).toBe(400); expect(put.statusCode).toBe(200);
expect(put.json()).toMatchObject({ error: expect.stringMatching(/model/i) }); expect(put.json()).not.toHaveProperty("model");
} finally { } finally {
rmSync(dir, { recursive: true, force: true }); rmSync(dir, { recursive: true, force: true });
} }
}); });
test("PUT /settings validates provider and model as one composite identifier", async () => { test("PUT /settings ignores legacy provider/model pairs", async () => {
const { app, dir } = appWithTmpSettings({}, { const { app, dir } = appWithTmpSettings({}, {
listModels: async () => [ listModels: async () => [
{ provider: "provider-a", id: "shared-id", name: "A", reasoning: false }, { provider: "provider-a", id: "shared-id", name: "A", reasoning: false },
@@ -204,7 +202,7 @@ test("PUT /settings validates provider and model as one composite identifier", a
workspace: "psd", provider: "provider-b", model: "shared-id", thinking: "low", workspace: "psd", provider: "provider-b", model: "shared-id", thinking: "low",
}, },
}); });
expect(wrongProvider.statusCode).toBe(400); expect(wrongProvider.statusCode).toBe(200);
const exactPair = await app.inject({ const exactPair = await app.inject({
method: "PUT", url: "/settings", method: "PUT", url: "/settings",
@@ -213,6 +211,8 @@ test("PUT /settings validates provider and model as one composite identifier", a
}, },
}); });
expect(exactPair.statusCode).toBe(200); expect(exactPair.statusCode).toBe(200);
expect(exactPair.json()).not.toHaveProperty("provider");
expect(exactPair.json()).not.toHaveProperty("model");
} finally { } finally {
rmSync(dir, { recursive: true, force: true }); rmSync(dir, { recursive: true, force: true });
} }
+25 -33
View File
@@ -1,4 +1,4 @@
import { test, expect, vi } from "vitest"; import { test, expect } from "vitest";
import Fastify from "fastify"; import Fastify from "fastify";
import { buildApp } from "../src/app.js"; import { buildApp } from "../src/app.js";
import { loadConfig } from "../src/config.js"; import { loadConfig } from "../src/config.js";
@@ -156,12 +156,23 @@ test("registry-backed SQL preview resolves and uses the session's pinned runtime
// Workspace registry route coverage lives in routes-workspaces.test.ts. `/workspaces` no longer // Workspace registry route coverage lives in routes-workspaces.test.ts. `/workspaces` no longer
// reads legacy harness files: the Git registry is the single shared source of truth. // reads legacy harness files: the Git registry is the single shared source of truth.
test("GET /models returns {models:[...]} from injected listModels stub", async () => { test("GET /models returns session choices from the installation model catalog", async () => {
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), { const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
thtRunner: {} as any, thtRunner: {} as any,
listModels: async () => [ runtimeModelCatalog: {
{ provider: "zai", id: "glm-5.2", name: "GLM 5.2", reasoning: true }, defaultSession: "zai/glm-5.2",
], defaultMetadataGeneration: null,
embedding: { id: "ollama/qwen3-embedding:0.6b", dimensions: 1024 },
sessionModels: () => [{
id: "zai/glm-5.2", provider: "zai", model: "glm-5.2", label: "GLM 5.2",
upstreamModel: "glm-5.2", authentication: { mode: "pi_auth" },
sessionAdapter: { mode: "pi_builtin" }, session: { reasoning: true },
}],
metadataModels: () => [],
hasSession: (id: string) => id === "zai/glm-5.2",
},
// Runtime introspection is a health gate for starting a session, not a second catalog.
listModels: async () => { throw new Error("Pi is unavailable"); },
}); });
const res = await app.inject({ method: "GET", url: "/models" }); const res = await app.inject({ method: "GET", url: "/models" });
@@ -172,36 +183,17 @@ test("GET /models returns {models:[...]} from injected listModels stub", async (
}); });
}); });
test("GET /models returns {models:[]} when listModels throws (graceful fallback)", async () => { test("GET /models returns an empty list when the catalog has no session models", async () => {
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), { const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
thtRunner: {} as any, thtRunner: {} as any,
listModels: async () => { throw new Error("Pi not running"); }, runtimeModelCatalog: {
}); defaultSession: null,
defaultMetadataGeneration: null,
const res = await app.inject({ method: "GET", url: "/models" }); embedding: null,
sessionModels: () => [],
expect(res.statusCode).toBe(200); metadataModels: () => [],
expect(res.json()).toEqual({ models: [] }); hasSession: () => false,
}); },
test("GET /models logs a sanitized warning when listing fails", async () => {
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
thtRunner: {} as any,
listModels: async () => { throw new Error("credential-value-must-not-appear"); },
});
const warn = vi.spyOn(app.log, "warn");
const res = await app.inject({ method: "GET", url: "/models" });
expect(res.json()).toEqual({ models: [] });
expect(JSON.stringify(warn.mock.calls)).not.toContain("credential-value-must-not-appear");
expect(warn).toHaveBeenCalled();
});
test("GET /models with empty listModels stub returns empty array", async () => {
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
thtRunner: {} as any,
listModels: async () => [],
}); });
const res = await app.inject({ method: "GET", url: "/models" }); const res = await app.inject({ method: "GET", url: "/models" });
+3 -17
View File
@@ -14,7 +14,7 @@ import type { AuthDiagnoser, AuthDiagnostics } from "../src/auth/diagnostics.js"
const workspace: CanonicalWorkspace = { const workspace: CanonicalWorkspace = {
workspace: { workspace: {
schema_version: 3, schema_version: 4,
id: "psd-clinical", id: "psd-clinical",
name: "Policlinico San Donato", name: "Policlinico San Donato",
description: "Clinical analytics workspace", description: "Clinical analytics workspace",
@@ -26,20 +26,6 @@ const workspace: CanonicalWorkspace = {
schema: "datawarehouse", schema: "datawarehouse",
supported_transports: ["postgres_direct"], supported_transports: ["postgres_direct"],
}, },
semantic_index: {
vector_store: {
engine: "qdrant",
collection: "psd-clinical",
dimensions: 1024,
distance: "cosine",
},
embedding: {
provider: "ollama_internal",
model: "qwen3-embedding:0.6b",
dimensions: 1024,
},
},
llm_policy: { allowed: ["zai/glm-5.2"] },
}; };
const revision: WorkspaceRevision = { const revision: WorkspaceRevision = {
@@ -194,7 +180,7 @@ test("lists workspace summaries and reads a validated immutable workspace", asyn
expect(read.json()).toEqual({ workspace, revision }); expect(read.json()).toEqual({ workspace, revision });
}); });
test("validates a schema v3 workspace without mutating the repository", async () => { test("validates a schema v4 workspace without mutating the repository", async () => {
const app = appFor(registryFake()); const app = appFor(registryFake());
const response = await app.inject({ const response = await app.inject({
@@ -284,7 +270,7 @@ test.each([1, 2])("rejects schema v%s at the validation boundary with a sanitize
expect(response.body).not.toMatch(/migration_required|schema version/i); expect(response.body).not.toMatch(/migration_required|schema version/i);
}); });
test("runs diagnostics for a schema v3 workspace", async () => { test("runs diagnostics for a schema v4 workspace", async () => {
const diagnose = vi.fn(async () => ({ activatable: true, diagnostics: [] })); const diagnose = vi.fn(async () => ({ activatable: true, diagnostics: [] }));
const app = appFor(registryFake(), diagnose); const app = appFor(registryFake(), diagnose);
+6 -6
View File
@@ -27,9 +27,9 @@ test("saveSettings writes the file and loadSettings reads it back", () => {
const dir = mkdtempSync(join(tmpdir(), "tht-set-")); const dir = mkdtempSync(join(tmpdir(), "tht-set-"));
try { try {
const cfg = cfgWith(join(dir, "nested", "settings.json")); const cfg = cfgWith(join(dir, "nested", "settings.json"));
const saved = saveSettings(cfg, { workspace: "psd", provider: "zai", model: "glm-5.2", thinking: "medium" }); const saved = saveSettings(cfg, { workspace: "psd", thinking: "medium" });
expect(saved.model).toBe("glm-5.2"); expect(saved.thinking).toBe("medium");
expect(loadSettings(cfg)).toEqual({ workspace: "psd", provider: "zai", model: "glm-5.2", thinking: "medium" }); expect(loadSettings(cfg)).toEqual({ workspace: "psd", thinking: "medium" });
} finally { } finally {
rmSync(dir, { recursive: true, force: true }); rmSync(dir, { recursive: true, force: true });
} }
@@ -55,11 +55,11 @@ test("saveSettings restores the previous file when post-rename directory durabil
const dir = mkdtempSync(join(tmpdir(), "tht-set-transaction-")); const dir = mkdtempSync(join(tmpdir(), "tht-set-transaction-"));
try { try {
const cfg = cfgWith(join(dir, "settings.json")); const cfg = cfgWith(join(dir, "settings.json"));
saveSettings(cfg, { provider: "old", model: "old-model", thinking: "low" }); saveSettings(cfg, { thinking: "low" });
let syncs = 0; let syncs = 0;
expect(() => saveSettings( expect(() => saveSettings(
cfg, cfg,
{ provider: "new", model: "new-model", thinking: "high" }, { thinking: "high" },
{ {
syncDirectory(directory: string) { syncDirectory(directory: string) {
syncs += 1; syncs += 1;
@@ -69,7 +69,7 @@ test("saveSettings restores the previous file when post-rename directory durabil
}, },
}, },
)).toThrow(/directory fsync failure/); )).toThrow(/directory fsync failure/);
expect(loadSettings(cfg)).toEqual({ provider: "old", model: "old-model", thinking: "low" }); expect(loadSettings(cfg)).toEqual({ thinking: "low" });
expect(syncs).toBeGreaterThanOrEqual(2); expect(syncs).toBeGreaterThanOrEqual(2);
} finally { } finally {
rmSync(dir, { recursive: true, force: true }); rmSync(dir, { recursive: true, force: true });
+1 -8
View File
@@ -8,18 +8,11 @@ const keywordIndexes = [
]; ];
const workspace: CanonicalWorkspace = { const workspace: CanonicalWorkspace = {
workspace: { schema_version: 3, id: "psd", name: "PSD", language: "it" }, workspace: { schema_version: 4, id: "psd", name: "PSD", language: "it" },
dwh: { dwh: {
engine: "postgres", database: "warehouse", schema: "public", engine: "postgres", database: "warehouse", schema: "public",
supported_transports: ["postgres_direct"], supported_transports: ["postgres_direct"],
}, },
semantic_index: {
vector_store: { engine: "qdrant", collection: "psd", dimensions: 1024, distance: "cosine" },
embedding: {
provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024,
},
},
llm_policy: { allowed: ["zai/glm-5.2"] },
}; };
function runner(request: (...args: any[]) => Promise<any>) { function runner(request: (...args: any[]) => Promise<any>) {
@@ -19,12 +19,13 @@ afterEach(() => {
const semanticRuntime = { const semanticRuntime = {
internalQdrantUrl: "http://qdrant:6333", internalQdrantUrl: "http://qdrant:6333",
internalEmbeddingUrl: "http://embedding:11434", internalEmbeddingUrl: "http://embedding:11434",
internalEmbeddingId: "ollama/qwen3-embedding:0.6b",
internalEmbeddingModel: "qwen3-embedding:0.6b", internalEmbeddingModel: "qwen3-embedding:0.6b",
internalEmbeddingDimensions: 1024, internalEmbeddingDimensions: 1024,
}; };
const baseWorkspace = parseWorkspaceYaml(`workspace: const baseWorkspace = parseWorkspaceYaml(`workspace:
schema_version: 3 schema_version: 4
id: psd-clinical id: psd-clinical
name: Runtime Lease name: Runtime Lease
language: en language: en
@@ -33,21 +34,9 @@ dwh:
database: analytics database: analytics
schema: mart schema: mart
supported_transports: [postgres_direct] supported_transports: [postgres_direct]
semantic_index:
vector_store:
engine: qdrant
collection: psd-clinical
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
llm_policy:
allowed: [zai/glm-5.2]
`); `);
const filesystemWorkspace = parseWorkspaceYaml(`${baseWorkspace ? '' : ''}workspace: const filesystemWorkspace = parseWorkspaceYaml(`${baseWorkspace ? '' : ''}workspace:
schema_version: 3 schema_version: 4
id: fs-workspace id: fs-workspace
name: Filesystem name: Filesystem
language: en language: en
@@ -56,25 +45,13 @@ dwh:
database: analytics database: analytics
schema: mart schema: mart
supported_transports: [postgres_direct] supported_transports: [postgres_direct]
semantic_index:
vector_store:
engine: qdrant
collection: fs-workspace
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
llm_policy:
allowed: [zai/glm-5.2]
evidence: evidence:
source: source:
type: filesystem type: filesystem
uri: fs-workspace/evidence uri: fs-workspace/evidence
`); `);
const privateHttpWorkspace = parseWorkspaceYaml(`workspace: const privateHttpWorkspace = parseWorkspaceYaml(`workspace:
schema_version: 3 schema_version: 4
id: http-workspace id: http-workspace
name: Http name: Http
language: en language: en
@@ -83,18 +60,6 @@ dwh:
database: analytics database: analytics
schema: mart schema: mart
supported_transports: [postgres_direct] supported_transports: [postgres_direct]
semantic_index:
vector_store:
engine: qdrant
collection: http-workspace
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
llm_policy:
allowed: [zai/glm-5.2]
evidence: evidence:
source: source:
type: http type: http
@@ -125,7 +90,7 @@ function runtime(workspace = baseWorkspace, workspaceId = workspace.workspace.id
bindingDigest: "sha256:bindings", bindingDigest: "sha256:bindings",
semanticQdrantUrl: "http://qdrant:6333", semanticQdrantUrl: "http://qdrant:6333",
effectiveConfig: { effectiveConfig: {
schemaVersion: 1, schemaVersion: 2,
dwh: { dwh: {
engine: "postgres", engine: "postgres",
database: "analytics", database: "analytics",
@@ -136,7 +101,7 @@ function runtime(workspace = baseWorkspace, workspaceId = workspace.workspace.id
user: "reader", user: "reader",
}, },
vector: { collection: workspaceId, dimensions: 1024, distance: "cosine" }, vector: { collection: workspaceId, dimensions: 1024, distance: "cosine" },
embedding: { model: "qwen3-embedding:0.6b", dimensions: 1024 }, embedding: { id: "ollama/qwen3-embedding:0.6b", model: "qwen3-embedding:0.6b", dimensions: 1024 },
roots: { artifacts: "/data/artifacts", indexes: "/data/indexes" }, roots: { artifacts: "/data/artifacts", indexes: "/data/indexes" },
}, },
effectiveConfigIdentity: "workspace://psd-clinical@v1:" + "d".repeat(64), effectiveConfigIdentity: "workspace://psd-clinical@v1:" + "d".repeat(64),
@@ -28,13 +28,15 @@ test("job state creates durable 0600 JSON and enforces same-revision resume", as
catalogBlob: "c".repeat(40), catalogBlob: "c".repeat(40),
configDigest: "sha256:config", configDigest: "sha256:config",
bindingDigest: "sha256:bindings", bindingDigest: "sha256:bindings",
embeddingId: "ollama/qwen3-embedding:0.6b",
embeddingDimensions: 1024,
}); });
const path = store.jobPath(job.runId); const path = store.jobPath(job.runId);
expect(existsSync(path)).toBe(true); expect(existsSync(path)).toBe(true);
expect(statSync(path).mode & 0o777).toBe(0o600); expect(statSync(path).mode & 0o777).toBe(0o600);
expect(JSON.parse(readFileSync(path, "utf8"))).toMatchObject({ expect(JSON.parse(readFileSync(path, "utf8"))).toMatchObject({
schemaVersion: 1, schemaVersion: 2,
operation: "preprocess dwh", operation: "preprocess dwh",
workspaceId: "psd-clinical", workspaceId: "psd-clinical",
workspaceRevision: "a".repeat(40), workspaceRevision: "a".repeat(40),
@@ -42,6 +44,8 @@ test("job state creates durable 0600 JSON and enforces same-revision resume", as
catalogBlob: "c".repeat(40), catalogBlob: "c".repeat(40),
configDigest: "sha256:config", configDigest: "sha256:config",
bindingDigest: "sha256:bindings", bindingDigest: "sha256:bindings",
embeddingId: "ollama/qwen3-embedding:0.6b",
embeddingDimensions: 1024,
}); });
await expect(store.beginJob({ await expect(store.beginJob({
@@ -52,6 +56,20 @@ test("job state creates durable 0600 JSON and enforces same-revision resume", as
catalogBlob: "c".repeat(40), catalogBlob: "c".repeat(40),
configDigest: "sha256:config", configDigest: "sha256:config",
bindingDigest: "sha256:bindings", bindingDigest: "sha256:bindings",
embeddingId: "ollama/qwen3-embedding:0.6b",
embeddingDimensions: 1024,
})).rejects.toMatchObject({ code: "preprocessing_resume_mismatch" });
await expect(store.beginJob({
operation: "preprocess dwh",
runId: job.runId,
workspaceRevision: "a".repeat(40),
descriptorBlob: "b".repeat(40),
catalogBlob: "c".repeat(40),
configDigest: "sha256:config",
bindingDigest: "sha256:bindings",
embeddingId: "ollama/replacement-embedding",
embeddingDimensions: 1024,
})).rejects.toMatchObject({ code: "preprocessing_resume_mismatch" }); })).rejects.toMatchObject({ code: "preprocessing_resume_mismatch" });
const resumed = await store.beginJob({ const resumed = await store.beginJob({
@@ -62,6 +80,8 @@ test("job state creates durable 0600 JSON and enforces same-revision resume", as
catalogBlob: "c".repeat(40), catalogBlob: "c".repeat(40),
configDigest: "sha256:config", configDigest: "sha256:config",
bindingDigest: "sha256:bindings", bindingDigest: "sha256:bindings",
embeddingId: "ollama/qwen3-embedding:0.6b",
embeddingDimensions: 1024,
}); });
expect(resumed.runId).toBe(job.runId); expect(resumed.runId).toBe(job.runId);
}); });
@@ -75,10 +75,6 @@ function workspaceVariant(
return { return {
...workspace, ...workspace,
workspace: { ...workspace.workspace, ...changes, id }, workspace: { ...workspace.workspace, ...changes, id },
semantic_index: {
...workspace.semantic_index,
vector_store: { ...workspace.semantic_index.vector_store, collection: id },
},
...(workspace.evidence?.source.type === "filesystem" ...(workspace.evidence?.source.type === "filesystem"
? { ? {
evidence: { evidence: {
@@ -182,7 +178,7 @@ test("shared deployment fixtures remain valid standalone descriptors with canoni
expect(smoke).toMatchObject({ expect(smoke).toMatchObject({
workspace: { workspace: {
schema_version: 3, schema_version: 4,
id: "local", id: "local",
name: "Local", name: "Local",
description: "Isolated workspace registry smoke fixture.", description: "Isolated workspace registry smoke fixture.",
@@ -193,14 +189,14 @@ test("shared deployment fixtures remain valid standalone descriptors with canoni
}, },
}); });
expect(task13).toMatchObject({ expect(task13).toMatchObject({
workspace: { schema_version: 3, id: "task13-smoke", name: "Task 13 Smoke" }, workspace: { schema_version: 4, id: "task13-smoke", name: "Task 13 Smoke" },
evidence: { evidence: {
source: { type: "filesystem", uri: "task13-smoke/evidence", patterns: ["**/*.md"] }, source: { type: "filesystem", uri: "task13-smoke/evidence", patterns: ["**/*.md"] },
policy: { max_chunk_chars: 4000, retain_published_generations: 3 }, policy: { max_chunk_chars: 4000, retain_published_generations: 3 },
}, },
}); });
expect(windows).toMatchObject({ expect(windows).toMatchObject({
workspace: { schema_version: 3, id: "task13-windows", name: "Task 13 Windows" }, workspace: { schema_version: 4, id: "task13-windows", name: "Task 13 Windows" },
evidence: { evidence: {
source: { type: "filesystem", uri: "task13-windows/evidence", patterns: ["**/*.md"] }, source: { type: "filesystem", uri: "task13-windows/evidence", patterns: ["**/*.md"] },
policy: { max_chunk_chars: 4000, retain_published_generations: 3 }, policy: { max_chunk_chars: 4000, retain_published_generations: 3 },
@@ -282,7 +278,7 @@ test("registry rejects orphan descriptors, metadata mismatches, and the retired
}); });
}); });
test("Windows clone contract copies the shared complete schema v3 descriptor into the nested registry layout", () => { test("Windows clone contract copies the shared complete schema v4 descriptor into the nested registry layout", () => {
const descriptor = parseWorkspaceYaml(readFixture("workspace-registry-windows.yaml")); const descriptor = parseWorkspaceYaml(readFixture("workspace-registry-windows.yaml"));
const windows = readFileSync( const windows = readFileSync(
new URL("../../scripts/test-windows-clone-contract.ps1", import.meta.url), new URL("../../scripts/test-windows-clone-contract.ps1", import.meta.url),
@@ -299,7 +295,7 @@ test("Windows clone contract copies the shared complete schema v3 descriptor int
expect(windows).not.toContain('schema_version: 3'); expect(windows).not.toContain('schema_version: 3');
expect(descriptor).toMatchObject({ expect(descriptor).toMatchObject({
workspace: { workspace: {
schema_version: 3, schema_version: 4,
id: "task13-windows", id: "task13-windows",
name: "Task 13 Windows", name: "Task 13 Windows",
language: "en", language: "en",
+6 -57
View File
@@ -15,7 +15,7 @@ import {
import type { WorkspaceRegistryConfig } from "../src/workspaces/types.js"; import type { WorkspaceRegistryConfig } from "../src/workspaces/types.js";
const validYaml = `workspace: const validYaml = `workspace:
schema_version: 3 schema_version: 4
id: psd-clinical id: psd-clinical
name: Policlinico San Donato name: Policlinico San Donato
language: it language: it
@@ -24,18 +24,6 @@ dwh:
database: postgres database: postgres
schema: datawarehouse schema: datawarehouse
supported_transports: [postgres_direct] supported_transports: [postgres_direct]
semantic_index:
vector_store:
engine: qdrant
collection: psd-clinical
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
llm_policy:
allowed: [zai/glm-5.2]
`; `;
function withFilesystemEvidence(source: string, id = "psd-clinical"): string { function withFilesystemEvidence(source: string, id = "psd-clinical"): string {
@@ -146,7 +134,7 @@ function legacyV1Yaml(source = validYaml): string {
.replace(" model: qwen3-embedding:0.6b", " model: nomic-embed-text-v2-moe") .replace(" model: qwen3-embedding:0.6b", " model: nomic-embed-text-v2-moe")
.replace(" dimensions: 1024", " dimensions: 768") .replace(" dimensions: 1024", " dimensions: 768")
.replace("distance: cosine\n", "distance: cosine\n supported_transports: [pgvector_direct]\n") .replace("distance: cosine\n", "distance: cosine\n supported_transports: [pgvector_direct]\n")
.replace("schema_version: 3", "schema_version: 1"); .replace("schema_version: 4", "schema_version: 1");
} }
function legacyV2Yaml(source = validYaml): string { function legacyV2Yaml(source = validYaml): string {
@@ -158,7 +146,7 @@ function legacyV2Yaml(source = validYaml): string {
.replace(" model: qwen3-embedding:0.6b", " model: nomic-embed-text-v2-moe") .replace(" model: qwen3-embedding:0.6b", " model: nomic-embed-text-v2-moe")
.replace(" dimensions: 1024", " dimensions: 768") .replace(" dimensions: 1024", " dimensions: 768")
.replace("distance: cosine\n", "distance: cosine\n supported_transports: [pgvector_direct]\n") .replace("distance: cosine\n", "distance: cosine\n supported_transports: [pgvector_direct]\n")
.replace("schema_version: 3", "schema_version: 2"); .replace("schema_version: 4", "schema_version: 2");
} }
const runFile = promisify(execFile); const runFile = promisify(execFile);
@@ -197,7 +185,7 @@ async function fixture(workspaceSource = validYaml): Promise<{
await git(source, ["init", "--initial-branch=main"]); await git(source, ["init", "--initial-branch=main"]);
await git(source, ["config", "user.name", "Workspace Registry Test"]); await git(source, ["config", "user.name", "Workspace Registry Test"]);
await git(source, ["config", "user.email", "workspace-registry@example.invalid"]); await git(source, ["config", "user.email", "workspace-registry@example.invalid"]);
const workspace = workspaceSource.includes("schema_version: 3") const workspace = workspaceSource.includes("schema_version: 4")
? parseWorkspaceYaml(workspaceSource) : undefined; ? parseWorkspaceYaml(workspaceSource) : undefined;
mkdirSync(join(source, "psd-clinical"), { recursive: true }); mkdirSync(join(source, "psd-clinical"), { recursive: true });
writeFileSync(join(source, "thoth-workspaces.yaml"), catalogYaml([ writeFileSync(join(source, "thoth-workspaces.yaml"), catalogYaml([
@@ -256,7 +244,7 @@ async function multiWorkspaceFixture(workspaces: Record<string, string>): Promis
await git(source, ["config", "user.email", "workspace-registry@example.invalid"]); await git(source, ["config", "user.email", "workspace-registry@example.invalid"]);
const entries = []; const entries = [];
for (const [id, workspaceSource] of Object.entries(workspaces)) { for (const [id, workspaceSource] of Object.entries(workspaces)) {
const workspace = workspaceSource.includes("schema_version: 3") ? parseWorkspaceYaml(workspaceSource) : undefined; const workspace = workspaceSource.includes("schema_version: 4") ? parseWorkspaceYaml(workspaceSource) : undefined;
entries.push({ id, name: workspace.workspace.name, ...(workspace.workspace.description ? { description: workspace.workspace.description } : {}) }); entries.push({ id, name: workspace.workspace.name, ...(workspace.workspace.description ? { description: workspace.workspace.description } : {}) });
mkdirSync(join(source, id), { recursive: true }); mkdirSync(join(source, id), { recursive: true });
writeFileSync(join(source, id, "workspace.yaml"), workspaceSource); writeFileSync(join(source, id, "workspace.yaml"), workspaceSource);
@@ -788,7 +776,7 @@ test("normalizes historical operational state during offline fallback after rest
expect(read.revision).not.toHaveProperty("state"); expect(read.revision).not.toHaveProperty("state");
}); });
test("fails closed when a retained snapshot descriptor is not schema v3", async () => { test("fails closed when a retained snapshot descriptor is not schema v4", async () => {
const remote = await fixture(); const remote = await fixture();
const root = join(remote.root, "registry"); const root = join(remote.root, "registry");
await new WorkspaceRegistry(config(root, remote.remote)).bootstrap(); await new WorkspaceRegistry(config(root, remote.remote)).bootstrap();
@@ -820,45 +808,6 @@ test("keeps the last valid snapshot when a pulled commit has invalid YAML", asyn
}); });
}); });
test("rejects duplicate schema v3 collection ownership and keeps the previous active snapshot", async () => {
const v3Yaml = validYaml;
const remote = await multiWorkspaceFixture({
"psd-clinical": v3Yaml,
"research-clinical": v3Yaml
.replace("id: psd-clinical", "id: research-clinical")
.replace("name: Policlinico San Donato", "name: Research Clinical")
.replace("collection: psd-clinical", "collection: research-clinical"),
});
const registry = new WorkspaceRegistry(config(join(remote.root, "registry"), remote.remote));
await registry.bootstrap();
writeFileSync(
join(remote.source, "research-clinical", "workspace.yaml"),
v3Yaml
.replace("id: psd-clinical", "id: research-clinical")
.replace("name: Policlinico San Donato", "name: Research Clinical")
.replace("collection: psd-clinical", "collection: shared"),
);
writeFileSync(
join(remote.source, "psd-clinical", "workspace.yaml"),
v3Yaml.replace("collection: psd-clinical", "collection: shared"),
);
await git(remote.source, ["add", "-A"]);
await git(remote.source, ["commit", "-m", "Duplicate collection ownership"]);
await git(remote.source, ["push", "origin", "main"]);
await expect(registry.pull()).rejects.toMatchObject({
code: "workspace_invalid",
message: "Workspace repository content is invalid",
});
await expect(registry.read("psd-clinical")).resolves.toMatchObject({
revision: { commit: remote.initialCommit },
});
await expect(registry.read("research-clinical")).resolves.toMatchObject({
revision: { commit: remote.initialCommit },
});
});
test("retains a historical snapshot while a resumable manifest still references its revision", async () => { test("retains a historical snapshot while a resumable manifest still references its revision", async () => {
const remote = await fixture(); const remote = await fixture();
const root = join(remote.root, "registry"); const root = join(remote.root, "registry");
@@ -66,7 +66,7 @@ workspaces: [{id: psd-clinical, name: Runtime Lease}]
`); `);
mkdirSync(join(source, "psd-clinical", "evidence"), { recursive: true }); mkdirSync(join(source, "psd-clinical", "evidence"), { recursive: true });
writeFileSync(join(source, "psd-clinical", "workspace.yaml"), `workspace: writeFileSync(join(source, "psd-clinical", "workspace.yaml"), `workspace:
schema_version: 3 schema_version: 4
id: psd-clinical id: psd-clinical
name: Runtime Lease name: Runtime Lease
language: en language: en
@@ -75,18 +75,6 @@ dwh:
database: analytics database: analytics
schema: mart schema: mart
supported_transports: [postgres_direct] supported_transports: [postgres_direct]
semantic_index:
vector_store:
engine: qdrant
collection: psd-clinical
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
llm_policy:
allowed: [zai/glm-5.2]
evidence: evidence:
source: source:
type: filesystem type: filesystem
+2 -14
View File
@@ -21,7 +21,7 @@ const thtBin = join(harnessDir, ".venv", "bin", "tht");
const roots: string[] = []; const roots: string[] = [];
const canonicalWorkspace = `workspace: const canonicalWorkspace = `workspace:
schema_version: 3 schema_version: 4
id: psd-clinical id: psd-clinical
name: Runtime handoff name: Runtime handoff
language: en language: en
@@ -30,18 +30,6 @@ dwh:
database: analytics database: analytics
schema: mart schema: mart
supported_transports: [postgres_direct] supported_transports: [postgres_direct]
semantic_index:
vector_store:
engine: qdrant
collection: psd-clinical
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
llm_policy:
allowed: [zai/glm-5.2]
`; `;
const filesystemWorkspace = `${canonicalWorkspace}evidence: const filesystemWorkspace = `${canonicalWorkspace}evidence:
@@ -145,7 +133,7 @@ function runnerFor(f: Awaited<ReturnType<typeof fixture>>): ThtRunner {
} as any); } as any);
} }
test("real schema-v3 registry revision loads through ThtRunner and the harness contract", async () => { test("real schema-v4 registry revision loads through ThtRunner and the harness contract", async () => {
const f = await fixture(); const f = await fixture();
const runner = runnerFor(f); const runner = runnerFor(f);
@@ -12,8 +12,8 @@ import {
import { supportsSessionRuntime } from "../src/workspaces/bindings.js"; import { supportsSessionRuntime } from "../src/workspaces/bindings.js";
import { parseWorkspaceYaml } from "../src/workspaces/schema.js"; import { parseWorkspaceYaml } from "../src/workspaces/schema.js";
const workspaceV3 = parseWorkspaceYaml(`workspace: const workspaceV4 = parseWorkspaceYaml(`workspace:
schema_version: 3 schema_version: 4
id: psd-clinical id: psd-clinical
name: Policlinico San Donato name: Policlinico San Donato
language: it language: it
@@ -22,19 +22,6 @@ dwh:
database: postgres database: postgres
schema: datawarehouse schema: datawarehouse
supported_transports: [postgres_direct, rest_api, ssh_tunnel] supported_transports: [postgres_direct, rest_api, ssh_tunnel]
semantic_index:
vector_store:
engine: qdrant
collection: psd-clinical
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
llm_policy:
default: zai/glm-5.2
allowed: [zai/glm-5.2]
`); `);
const paths: RuntimePaths = { const paths: RuntimePaths = {
sessions: "/data/workspaces/psd-clinical/sessions", sessions: "/data/workspaces/psd-clinical/sessions",
@@ -45,6 +32,7 @@ const paths: RuntimePaths = {
const semanticRuntime: SemanticRuntimeConfig = { const semanticRuntime: SemanticRuntimeConfig = {
internalQdrantUrl: "http://qdrant:6333", internalQdrantUrl: "http://qdrant:6333",
internalEmbeddingUrl: "http://embedding:11434", internalEmbeddingUrl: "http://embedding:11434",
internalEmbeddingId: "ollama/qwen3-embedding:0.6b",
internalEmbeddingModel: "qwen3-embedding:0.6b", internalEmbeddingModel: "qwen3-embedding:0.6b",
internalEmbeddingDimensions: 1024, internalEmbeddingDimensions: 1024,
}; };
@@ -63,8 +51,8 @@ const directBindings: RuntimeBindings = {
evidence: { missing: [], values: {} }, evidence: { missing: [], values: {} },
}; };
test("renders only the schema-v3 internal Qdrant and Ollama runtime shape", () => { test("derives the internal Qdrant and Ollama runtime shape from workspace v4 plus installation config", () => {
const rendered = parse(renderRuntimeConfig(workspaceV3, directBindings, paths, { const rendered = parse(renderRuntimeConfig(workspaceV4, directBindings, paths, {
workspaceId: "psd-clinical", workspaceRevision: "a".repeat(40), workspaceId: "psd-clinical", workspaceRevision: "a".repeat(40),
}, {}, semanticRuntime)); }, {}, semanticRuntime));
@@ -95,8 +83,8 @@ test("renders only the schema-v3 internal Qdrant and Ollama runtime shape", () =
expect(rendered).not.toHaveProperty("vector_rest"); expect(rendered).not.toHaveProperty("vector_rest");
}); });
test("renders schema-v3 DWH REST without exposing secret contents", () => { test("renders workspace-v4 DWH REST without exposing secret contents", () => {
const rendered = parse(renderRuntimeConfig(workspaceV3, { const rendered = parse(renderRuntimeConfig(workspaceV4, {
dwh: { dwh: {
transport: "rest_api", missing: [], values: { transport: "rest_api", missing: [], values: {
THT_WS_PSD_CLINICAL_DWH_BASE_URL: "https://dwh.example.test", THT_WS_PSD_CLINICAL_DWH_BASE_URL: "https://dwh.example.test",
@@ -152,7 +140,7 @@ function evidenceWorkspace(
} }
const canonicalEvidenceWorkspace = `workspace: const canonicalEvidenceWorkspace = `workspace:
schema_version: 3 schema_version: 4
id: psd-clinical id: psd-clinical
name: Runtime Evidence name: Runtime Evidence
language: en language: en
@@ -161,18 +149,6 @@ dwh:
database: analytics database: analytics
schema: mart schema: mart
supported_transports: [postgres_direct] supported_transports: [postgres_direct]
semantic_index:
vector_store:
engine: qdrant
collection: psd-clinical
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
llm_policy:
allowed: [zai/glm-5.2]
`; `;
const evidenceRevision = "1".repeat(40); const evidenceRevision = "1".repeat(40);
@@ -374,7 +350,7 @@ test("renders static S3 Evidence with endpoint policy, limits, and file paths bu
test("omits Evidence configuration and policy when the descriptor has no Evidence", () => { test("omits Evidence configuration and policy when the descriptor has no Evidence", () => {
const rendered = parse(renderRuntimeConfig( const rendered = parse(renderRuntimeConfig(
workspaceV3, workspaceV4,
directBindings, directBindings,
paths, paths,
evidenceContext, evidenceContext,
@@ -14,7 +14,7 @@ const roots: string[] = [];
function workspace(extra = "") { function workspace(extra = "") {
return parseWorkspaceYaml(`workspace: return parseWorkspaceYaml(`workspace:
schema_version: 3 schema_version: 4
id: psd-clinical id: psd-clinical
name: Policlinico San Donato name: Policlinico San Donato
language: en language: en
@@ -23,10 +23,6 @@ dwh:
database: postgres database: postgres
schema: datawarehouse schema: datawarehouse
supported_transports: [postgres_direct, rest_api, ssh_tunnel] supported_transports: [postgres_direct, rest_api, ssh_tunnel]
semantic_index:
vector_store: { engine: qdrant, collection: psd-clinical, dimensions: 1024, distance: cosine }
embedding: { provider: ollama_internal, model: qwen3-embedding:0.6b, dimensions: 1024 }
llm_policy: { allowed: [zai/glm-5.2] }
${extra}`); ${extra}`);
} }
@@ -0,0 +1,36 @@
import { expect, test } from "vitest";
import { migrateWorkspaceV3Yaml, parseWorkspaceYaml } from "../src/workspaces/schema.js";
const legacy = `workspace:
schema_version: 3
id: abc
name: Example
language: en
dwh:
engine: postgres
database: warehouse
schema: public
supported_transports: [postgres_direct]
semantic_index:
vector_store: {engine: qdrant, collection: abc, dimensions: 1024, distance: cosine}
embedding: {provider: ollama_internal, model: qwen3-embedding:0.6b, dimensions: 1024}
llm_policy:
default: zai/glm-5.3
allowed: [zai/glm-5.3]
`;
test("strict workspace v4 rejects model-bearing v3 descriptors", () => {
expect(() => parseWorkspaceYaml(legacy)).toThrow();
});
test("v3 to v4 migration removes only model/vector policy and bumps the version", () => {
const migrated = migrateWorkspaceV3Yaml(legacy);
const workspace = parseWorkspaceYaml(migrated);
expect(workspace.workspace).toMatchObject({ schema_version: 4, id: "abc" });
expect(migrated).not.toMatch(/semantic_index|llm_policy/);
expect(workspace.dwh).toEqual({
engine: "postgres", database: "warehouse", schema: "public",
supported_transports: ["postgres_direct"],
});
});
+12 -28
View File
@@ -9,8 +9,8 @@ import {
} from "../src/workspaces/bindings.js"; } from "../src/workspaces/bindings.js";
import { parseWorkspaceYaml } from "../src/workspaces/schema.js"; import { parseWorkspaceYaml } from "../src/workspaces/schema.js";
const workspaceV3 = parseWorkspaceYaml(`workspace: const workspaceV4 = parseWorkspaceYaml(`workspace:
schema_version: 3 schema_version: 4
id: psd-clinical id: psd-clinical
name: Policlinico San Donato name: Policlinico San Donato
language: it language: it
@@ -19,10 +19,6 @@ dwh:
database: postgres database: postgres
schema: datawarehouse schema: datawarehouse
supported_transports: [postgres_direct, rest_api, ssh_tunnel] supported_transports: [postgres_direct, rest_api, ssh_tunnel]
semantic_index:
vector_store: { engine: qdrant, collection: psd-clinical, dimensions: 1024, distance: cosine }
embedding: { provider: ollama_internal, model: qwen3-embedding:0.6b, dimensions: 1024 }
llm_policy: { allowed: [zai/glm-5.2] }
`); `);
const temporaryRoots: string[] = []; const temporaryRoots: string[] = [];
@@ -40,9 +36,9 @@ function secretPath(name: string): { root: string; path: string } {
return { root: secrets, path }; return { root: secrets, path };
} }
test("resolves schema-v3 direct DWH bindings from the stable namespace", () => { test("resolves workspace-v4 direct DWH bindings from the stable namespace", () => {
const password = secretPath("dwh-password"); const password = secretPath("dwh-password");
const result = resolveBinding(workspaceV3, "DWH", { const result = resolveBinding(workspaceV4, "DWH", {
THT_WS_PSD_CLINICAL_DWH_TRANSPORT: "postgres_direct", THT_WS_PSD_CLINICAL_DWH_TRANSPORT: "postgres_direct",
THT_WS_PSD_CLINICAL_DWH_HOST: "dwh.internal", THT_WS_PSD_CLINICAL_DWH_HOST: "dwh.internal",
THT_WS_PSD_CLINICAL_DWH_PORT: "5432", THT_WS_PSD_CLINICAL_DWH_PORT: "5432",
@@ -61,14 +57,14 @@ test("resolves schema-v3 direct DWH bindings from the stable namespace", () => {
}); });
}); });
test("requires schema-v3 REST credentials unless the DWH diagnostic declares auth none", () => { test("requires workspace-v4 REST credentials unless the DWH diagnostic declares auth none", () => {
expect(resolveBinding(workspaceV3, "DWH", { expect(resolveBinding(workspaceV4, "DWH", {
THT_WS_PSD_CLINICAL_DWH_TRANSPORT: "rest_api", THT_WS_PSD_CLINICAL_DWH_TRANSPORT: "rest_api",
THT_WS_PSD_CLINICAL_DWH_BASE_URL: "https://dwh.example.test", THT_WS_PSD_CLINICAL_DWH_BASE_URL: "https://dwh.example.test",
}, []).missing).toContain("THT_WS_PSD_CLINICAL_DWH_API_KEY_FILE"); }, []).missing).toContain("THT_WS_PSD_CLINICAL_DWH_API_KEY_FILE");
const noAuth = parseWorkspaceYaml(`workspace: const noAuth = parseWorkspaceYaml(`workspace:
schema_version: 3 schema_version: 4
id: psd-clinical id: psd-clinical
name: No auth name: No auth
language: en language: en
@@ -77,16 +73,12 @@ dwh:
database: postgres database: postgres
schema: public schema: public
supported_transports: [rest_api] supported_transports: [rest_api]
semantic_index:
vector_store: { engine: qdrant, collection: psd-clinical, dimensions: 1024, distance: cosine }
embedding: { provider: ollama_internal, model: qwen3-embedding:0.6b, dimensions: 1024 }
diagnostics: diagnostics:
dwh_rest: dwh_rest:
method: GET method: GET
path: /health path: /health
auth: none auth: none
response: { database: database, schema: schema } response: { database: database, schema: schema }
llm_policy: { allowed: [zai/glm-5.2] }
`); `);
expect(resolveBinding(noAuth, "DWH", { expect(resolveBinding(noAuth, "DWH", {
THT_WS_PSD_CLINICAL_DWH_TRANSPORT: "rest_api", THT_WS_PSD_CLINICAL_DWH_TRANSPORT: "rest_api",
@@ -98,7 +90,7 @@ test("rejects unsupported transports and secret paths outside configured roots",
const outside = secretPath("outside-password"); const outside = secretPath("outside-password");
const allowed = secretPath("allowed-password"); const allowed = secretPath("allowed-password");
const directOnly = parseWorkspaceYaml(`workspace: const directOnly = parseWorkspaceYaml(`workspace:
schema_version: 3 schema_version: 4
id: psd-clinical id: psd-clinical
name: Direct only name: Direct only
language: en language: en
@@ -107,15 +99,11 @@ dwh:
database: postgres database: postgres
schema: public schema: public
supported_transports: [postgres_direct] supported_transports: [postgres_direct]
semantic_index:
vector_store: { engine: qdrant, collection: psd-clinical, dimensions: 1024, distance: cosine }
embedding: { provider: ollama_internal, model: qwen3-embedding:0.6b, dimensions: 1024 }
llm_policy: { allowed: [zai/glm-5.2] }
`); `);
expect(resolveBinding(directOnly, "DWH", { expect(resolveBinding(directOnly, "DWH", {
THT_WS_PSD_CLINICAL_DWH_TRANSPORT: "rest_api", THT_WS_PSD_CLINICAL_DWH_TRANSPORT: "rest_api",
}, []).missing).toContain("THT_WS_PSD_CLINICAL_DWH_TRANSPORT"); }, []).missing).toContain("THT_WS_PSD_CLINICAL_DWH_TRANSPORT");
const result = resolveBinding(workspaceV3, "DWH", { const result = resolveBinding(workspaceV4, "DWH", {
THT_WS_PSD_CLINICAL_DWH_TRANSPORT: "postgres_direct", THT_WS_PSD_CLINICAL_DWH_TRANSPORT: "postgres_direct",
THT_WS_PSD_CLINICAL_DWH_HOST: "dwh.internal", THT_WS_PSD_CLINICAL_DWH_HOST: "dwh.internal",
THT_WS_PSD_CLINICAL_DWH_PORT: "5432", THT_WS_PSD_CLINICAL_DWH_PORT: "5432",
@@ -128,7 +116,7 @@ llm_policy: { allowed: [zai/glm-5.2] }
test("runtime bindings contain only DWH and Evidence roles", () => { test("runtime bindings contain only DWH and Evidence roles", () => {
const password = secretPath("dwh-password"); const password = secretPath("dwh-password");
const bindings = resolveRuntimeBindings(workspaceV3, { const bindings = resolveRuntimeBindings(workspaceV4, {
THT_WS_PSD_CLINICAL_DWH_TRANSPORT: "postgres_direct", THT_WS_PSD_CLINICAL_DWH_TRANSPORT: "postgres_direct",
THT_WS_PSD_CLINICAL_DWH_HOST: "dwh.internal", THT_WS_PSD_CLINICAL_DWH_HOST: "dwh.internal",
THT_WS_PSD_CLINICAL_DWH_PORT: "5432", THT_WS_PSD_CLINICAL_DWH_PORT: "5432",
@@ -143,7 +131,7 @@ test("runtime bindings contain only DWH and Evidence roles", () => {
function withEvidence(source: Record<string, unknown>) { function withEvidence(source: Record<string, unknown>) {
return parseWorkspaceYaml(`workspace: return parseWorkspaceYaml(`workspace:
schema_version: 3 schema_version: 4
id: psd-clinical id: psd-clinical
name: Policlinico San Donato name: Policlinico San Donato
language: it language: it
@@ -152,10 +140,6 @@ dwh:
database: postgres database: postgres
schema: datawarehouse schema: datawarehouse
supported_transports: [postgres_direct] supported_transports: [postgres_direct]
semantic_index:
vector_store: { engine: qdrant, collection: psd-clinical, dimensions: 1024, distance: cosine }
embedding: { provider: ollama_internal, model: qwen3-embedding:0.6b, dimensions: 1024 }
llm_policy: { allowed: [zai/glm-5.2] }
evidence: evidence:
source: ${JSON.stringify(source)} source: ${JSON.stringify(source)}
`); `);
@@ -260,7 +244,7 @@ test("rejects relative, missing, directory, unreadable, and escaping symlink Evi
}); });
test("includes Evidence binding completeness in session runtime support without changing v3 compatibility", () => { test("includes Evidence binding completeness in session runtime support without changing v3 compatibility", () => {
const unsigned = resolveRuntimeBindings(workspaceV3, {}, ["/run/secrets"]); const unsigned = resolveRuntimeBindings(workspaceV4, {}, ["/run/secrets"]);
expect(unsigned.evidence).toEqual({ values: {}, missing: [] }); expect(unsigned.evidence).toEqual({ values: {}, missing: [] });
expect(supportsSessionRuntime(unsigned)).toBe(true); expect(supportsSessionRuntime(unsigned)).toBe(true);
+1 -5
View File
@@ -7,7 +7,7 @@ import {
} from "../src/workspaces/catalog.js"; } from "../src/workspaces/catalog.js";
const descriptor = parseWorkspaceYaml(`workspace: const descriptor = parseWorkspaceYaml(`workspace:
schema_version: 3 schema_version: 4
id: psd id: psd
name: Policlinico San Donato name: Policlinico San Donato
description: Clinical warehouse description: Clinical warehouse
@@ -17,10 +17,6 @@ dwh:
database: postgres database: postgres
schema: datawarehouse schema: datawarehouse
supported_transports: [rest_api] supported_transports: [rest_api]
semantic_index:
vector_store: { engine: qdrant, collection: psd, dimensions: 1024, distance: cosine }
embedding: { provider: ollama_internal, model: qwen3-embedding:0.6b, dimensions: 1024 }
llm_policy: { allowed: [zai/glm-5.2] }
`); `);
test("parses the strict ordered root catalog", () => { test("parses the strict ordered root catalog", () => {
+11 -20
View File
@@ -5,8 +5,8 @@ import { join } from "node:path";
import { buildInstallationContract, renderWorkspaceDocs } from "../src/workspaces/contracts.js"; import { buildInstallationContract, renderWorkspaceDocs } from "../src/workspaces/contracts.js";
import { type CanonicalWorkspace, parseWorkspaceYaml } from "../src/workspaces/schema.js"; import { type CanonicalWorkspace, parseWorkspaceYaml } from "../src/workspaces/schema.js";
const workspaceV3 = parseWorkspaceYaml(`workspace: const workspaceV4 = parseWorkspaceYaml(`workspace:
schema_version: 3 schema_version: 4
id: psd-clinical id: psd-clinical
name: Policlinico San Donato name: Policlinico San Donato
language: it language: it
@@ -15,17 +15,12 @@ dwh:
database: postgres database: postgres
schema: datawarehouse schema: datawarehouse
supported_transports: [postgres_direct, rest_api, ssh_tunnel] supported_transports: [postgres_direct, rest_api, ssh_tunnel]
semantic_index:
vector_store: { engine: qdrant, collection: psd-clinical, dimensions: 1024, distance: cosine }
embedding: { provider: ollama_internal, model: qwen3-embedding:0.6b, dimensions: 1024 }
llm_policy:
allowed: [zai/glm-5.2]
`); `);
test("schema-v3 installation contracts expose only DWH bindings and no semantic variables", () => { test("workspace-v4 installation contracts expose only DWH bindings and no semantic variables", () => {
const contract = buildInstallationContract(workspaceV3); const contract = buildInstallationContract(workspaceV4);
const names = contract.variables.map((variable) => variable.name); const names = contract.variables.map((variable) => variable.name);
const docs = renderWorkspaceDocs(workspaceV3); const docs = renderWorkspaceDocs(workspaceV4);
expect(contract.workspaceId).toBe("psd-clinical"); expect(contract.workspaceId).toBe("psd-clinical");
expect(contract.namespace).toBe("PSD_CLINICAL"); expect(contract.namespace).toBe("PSD_CLINICAL");
@@ -54,22 +49,22 @@ test.each([
test("validates public contract and documentation inputs at runtime", () => { test("validates public contract and documentation inputs at runtime", () => {
const unsafeWorkspace = { const unsafeWorkspace = {
...workspaceV3, ...workspaceV4,
workspace: { ...workspaceV3.workspace, id: "psd\nclinical" }, workspace: { ...workspaceV4.workspace, id: "psd\nclinical" },
} as CanonicalWorkspace; } as CanonicalWorkspace;
expect(() => buildInstallationContract(unsafeWorkspace)).toThrow(/id/i); expect(() => buildInstallationContract(unsafeWorkspace)).toThrow(/id/i);
expect(() => renderWorkspaceDocs(unsafeWorkspace)).toThrow(/id/i); expect(() => renderWorkspaceDocs(unsafeWorkspace)).toThrow(/id/i);
}); });
test("v3 installation contract omits external vector and embedding bindings", () => { test("v4 installation contract omits external vector and embedding bindings", () => {
const contract = buildInstallationContract(workspaceV3); const contract = buildInstallationContract(workspaceV4);
const names = contract.variables.map((variable) => variable.name); const names = contract.variables.map((variable) => variable.name);
expect(names).toContain("THT_WS_PSD_CLINICAL_DWH_TRANSPORT"); expect(names).toContain("THT_WS_PSD_CLINICAL_DWH_TRANSPORT");
expect(names.some((name) => name.includes("_VECTOR_"))).toBe(false); expect(names.some((name) => name.includes("_VECTOR_"))).toBe(false);
expect(names.some((name) => name.includes("_EMBEDDING_"))).toBe(false); expect(names.some((name) => name.includes("_EMBEDDING_"))).toBe(false);
expect(renderWorkspaceDocs(workspaceV3).markdown).not.toContain("Embedding service"); expect(renderWorkspaceDocs(workspaceV4).markdown).not.toContain("Embedding service");
}); });
@@ -114,7 +109,7 @@ test.each([
function renderWorkspaceWithoutEvidence(): string { function renderWorkspaceWithoutEvidence(): string {
return `workspace: return `workspace:
schema_version: 3 schema_version: 4
id: psd-clinical id: psd-clinical
name: Policlinico San Donato name: Policlinico San Donato
language: it language: it
@@ -123,10 +118,6 @@ dwh:
database: postgres database: postgres
schema: datawarehouse schema: datawarehouse
supported_transports: [postgres_direct] supported_transports: [postgres_direct]
semantic_index:
vector_store: { engine: qdrant, collection: psd-clinical, dimensions: 1024, distance: cosine }
embedding: { provider: ollama_internal, model: qwen3-embedding:0.6b, dimensions: 1024 }
llm_policy: { allowed: [zai/glm-5.2] }
`; `;
} }
+6 -26
View File
@@ -12,7 +12,7 @@ import type { RuntimeBindings } from "../src/workspaces/runtime-renderer.js";
import { parseWorkspaceYaml, resolveDiagnosticUrl } from "../src/workspaces/schema.js"; import { parseWorkspaceYaml, resolveDiagnosticUrl } from "../src/workspaces/schema.js";
const workspace = parseWorkspaceYaml(`workspace: const workspace = parseWorkspaceYaml(`workspace:
schema_version: 3 schema_version: 4
id: psd-clinical id: psd-clinical
name: Policlinico San Donato name: Policlinico San Donato
language: it language: it
@@ -22,18 +22,6 @@ dwh:
schema: datawarehouse schema: datawarehouse
timeout_ms: 8000 timeout_ms: 8000
supported_transports: [postgres_direct, rest_api, ssh_tunnel] supported_transports: [postgres_direct, rest_api, ssh_tunnel]
semantic_index:
vector_store:
engine: qdrant
collection: psd-clinical
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
llm_policy:
allowed: [zai/glm-5.2]
`); `);
const bindings: RuntimeBindings = { const bindings: RuntimeBindings = {
@@ -78,7 +66,7 @@ afterEach(() => {
vi.restoreAllMocks(); vi.restoreAllMocks();
}); });
test("diagnoses schema-v3 DWH, internal Qdrant, and internal Ollama without semantic bindings", async () => { test("diagnoses workspace-v4 DWH plus installation-derived Qdrant and Ollama", async () => {
const adapters = successfulAdapters(); const adapters = successfulAdapters();
const result = await diagnose(adapters)(workspace, bindings, { writeProbe: false }); const result = await diagnose(adapters)(workspace, bindings, { writeProbe: false });
@@ -139,7 +127,7 @@ test("reports only sanitized DWH and Evidence binding names before network diagn
expect(adapters.inspectQdrant).not.toHaveBeenCalled(); expect(adapters.inspectQdrant).not.toHaveBeenCalled();
}); });
test("keeps schema-v3 DWH SSH diagnostic-only and runtime-inactive", async () => { test("keeps workspace-v4 DWH SSH diagnostic-only and runtime-inactive", async () => {
const adapters = successfulAdapters(); const adapters = successfulAdapters();
const result = await diagnose(adapters)(workspace, { const result = await diagnose(adapters)(workspace, {
...bindings, ...bindings,
@@ -152,9 +140,9 @@ test("keeps schema-v3 DWH SSH diagnostic-only and runtime-inactive", async () =>
expect(adapters.probeConnector).not.toHaveBeenCalled(); expect(adapters.probeConnector).not.toHaveBeenCalled();
}); });
test("uses the schema-v3 declared DWH REST diagnostic and auth policy", async () => { test("uses the workspace-v4 declared DWH REST diagnostic and auth policy", async () => {
const restWorkspace = parseWorkspaceYaml(`workspace: const restWorkspace = parseWorkspaceYaml(`workspace:
schema_version: 3 schema_version: 4
id: psd-clinical id: psd-clinical
name: REST workspace name: REST workspace
language: en language: en
@@ -163,16 +151,12 @@ dwh:
database: warehouse database: warehouse
schema: datawarehouse schema: datawarehouse
supported_transports: [rest_api] supported_transports: [rest_api]
semantic_index:
vector_store: { engine: qdrant, collection: psd-clinical, dimensions: 1024, distance: cosine }
embedding: { provider: ollama_internal, model: qwen3-embedding:0.6b, dimensions: 1024 }
diagnostics: diagnostics:
dwh_rest: dwh_rest:
method: POST method: POST
path: /rpc/ping path: /rpc/ping
auth: bearer auth: bearer
response: { database: database, schema: schema } response: { database: database, schema: schema }
llm_policy: { allowed: [zai/glm-5.2] }
`); `);
const adapters = successfulAdapters(); const adapters = successfulAdapters();
const result = await diagnose(adapters)(restWorkspace, { const result = await diagnose(adapters)(restWorkspace, {
@@ -423,7 +407,7 @@ test("uses a REST secret only as a header and redacts it from failed diagnostics
const canary = "CANARY-REST-AUTH-SECRET"; const canary = "CANARY-REST-AUTH-SECRET";
await writeFile(credentialFile, canary); await writeFile(credentialFile, canary);
const restDescriptor = parseWorkspaceYaml(`workspace: const restDescriptor = parseWorkspaceYaml(`workspace:
schema_version: 3 schema_version: 4
id: psd-clinical id: psd-clinical
name: REST auth name: REST auth
language: en language: en
@@ -432,16 +416,12 @@ dwh:
database: warehouse database: warehouse
schema: datawarehouse schema: datawarehouse
supported_transports: [rest_api] supported_transports: [rest_api]
semantic_index:
vector_store: { engine: qdrant, collection: psd-clinical, dimensions: 1024, distance: cosine }
embedding: { provider: ollama_internal, model: qwen3-embedding:0.6b, dimensions: 1024 }
diagnostics: diagnostics:
dwh_rest: dwh_rest:
method: GET method: GET
path: /health path: /health
auth: bearer auth: bearer
response: { database: database, schema: schema } response: { database: database, schema: schema }
llm_policy: { allowed: [zai/glm-5.2] }
`); `);
const fetchMock = vi.fn() const fetchMock = vi.fn()
.mockResolvedValueOnce(new Response("upstream CANARY-REST-AUTH-SECRET", { status: 503 })) .mockResolvedValueOnce(new Response("upstream CANARY-REST-AUTH-SECRET", { status: 503 }))
@@ -53,7 +53,7 @@ async function makeRepo(id: string, annotations: string | Buffer | "dir" | "syml
writeFileSync(join(source, "thoth-workspaces.yaml"), writeFileSync(join(source, "thoth-workspaces.yaml"),
`schema_version: 1\nworkspaces: [{id: ${id}, name: Workspace}]\n`); `schema_version: 1\nworkspaces: [{id: ${id}, name: Workspace}]\n`);
mkdirSync(join(source, id, "schema"), { recursive: true }); mkdirSync(join(source, id, "schema"), { recursive: true });
writeFileSync(join(source, id, "workspace.yaml"), `workspace:\n schema_version: 3\n id: ${id}\n`); writeFileSync(join(source, id, "workspace.yaml"), `workspace:\n schema_version: 4\n id: ${id}\n`);
const annotationsPath = join(source, id, "schema", "annotations.yaml"); const annotationsPath = join(source, id, "schema", "annotations.yaml");
if (annotations === "dir") { if (annotations === "dir") {
mkdirSync(annotationsPath, { recursive: true }); mkdirSync(annotationsPath, { recursive: true });
+1 -1
View File
@@ -47,7 +47,7 @@ async function fixture(layout: EvidenceLayout): Promise<{ root: string; remote:
await git(source, ["config", "user.email", "evidence@example.invalid"]); await git(source, ["config", "user.email", "evidence@example.invalid"]);
writeFileSync(join(source, "thoth-workspaces.yaml"), "schema_version: 1\nworkspaces: [{id: research, name: Research}]\n"); writeFileSync(join(source, "thoth-workspaces.yaml"), "schema_version: 1\nworkspaces: [{id: research, name: Research}]\n");
mkdirSync(join(source, "research"), { recursive: true }); mkdirSync(join(source, "research"), { recursive: true });
writeFileSync(join(source, "research", "workspace.yaml"), "workspace:\n schema_version: 3\n id: research\n"); writeFileSync(join(source, "research", "workspace.yaml"), "workspace:\n schema_version: 4\n id: research\n");
const evidence = join(source, "research", "evidence"); const evidence = join(source, "research", "evidence");
if (layout === "tree") { if (layout === "tree") {
mkdirSync(join(evidence, "nested"), { recursive: true }); mkdirSync(join(evidence, "nested"), { recursive: true });
+1 -16
View File
@@ -12,7 +12,7 @@ import {
import type { WorkspaceRegistryConfig } from "../src/workspaces/types.js"; import type { WorkspaceRegistryConfig } from "../src/workspaces/types.js";
const validYaml = `workspace: const validYaml = `workspace:
schema_version: 2 schema_version: 4
id: psd-clinical id: psd-clinical
name: Policlinico San Donato name: Policlinico San Donato
language: it language: it
@@ -21,21 +21,6 @@ dwh:
database: postgres database: postgres
schema: datawarehouse schema: datawarehouse
supported_transports: [postgres_direct] supported_transports: [postgres_direct]
semantic_index:
vector_store:
engine: pgvector
database: postgres
schema: vectors
collection: clinical_documents
dimensions: 768
distance: cosine
supported_transports: [pgvector_direct]
embedding:
provider: ollama_compatible
model: nomic-embed-text-v2-moe
dimensions: 768
llm_policy:
allowed: [zai/glm-5.2]
`; `;
const runFile = promisify(execFile); const runFile = promisify(execFile);
@@ -13,20 +13,11 @@ import {
import { renderRuntimeConfig, type RuntimeBindings } from "../src/workspaces/runtime-renderer.js"; import { renderRuntimeConfig, type RuntimeBindings } from "../src/workspaces/runtime-renderer.js";
const unsupportedWorkspace = { const unsupportedWorkspace = {
workspace: { schema_version: 2, id: "legacy-workspace", name: "Legacy", language: "en" }, workspace: { schema_version: 3, id: "legacy-workspace", name: "Legacy", language: "en" },
dwh: { dwh: {
engine: "postgres", database: "warehouse", schema: "public", engine: "postgres", database: "warehouse", schema: "public",
supported_transports: ["postgres_direct"], supported_transports: ["postgres_direct"],
}, },
semantic_index: {
vector_store: {
engine: "pgvector", database: "warehouse", schema: "vectors",
collection: "documents", dimensions: 768, distance: "cosine",
supported_transports: ["pgvector_direct"],
},
embedding: { provider: "ollama_compatible", model: "legacy", dimensions: 768 },
},
llm_policy: { allowed: ["zai/glm-5.2"] },
}; };
const bindings: RuntimeBindings = { const bindings: RuntimeBindings = {
@@ -40,31 +31,31 @@ const adapters: DiagnosticAdapters = {
probeEmbedding: vi.fn(), probeEmbedding: vi.fn(),
}; };
test("renderer rejects callers that bypass the schema-v3 type contract", () => { test("renderer rejects callers that bypass the schema-v4 type contract", () => {
expect(() => renderRuntimeConfig(unsupportedWorkspace as never, bindings, { expect(() => renderRuntimeConfig(unsupportedWorkspace as never, bindings, {
sessions: "/data/sessions", artifacts: "/data/artifacts", indexes: "/data/indexes", sessions: "/data/sessions", artifacts: "/data/artifacts", indexes: "/data/indexes",
})).toThrow("Runtime renderer supports only workspace schema version 3"); })).toThrow("Runtime renderer supports only workspace schema version 4");
}); });
test("installation contract rejects callers that bypass the schema-v3 type contract", () => { test("installation contract rejects callers that bypass the schema-v4 type contract", () => {
expect(() => buildInstallationContract(unsupportedWorkspace as never)) expect(() => buildInstallationContract(unsupportedWorkspace as never))
.toThrow("Installation contract supports only workspace schema version 3"); .toThrow("Installation contract supports only workspace schema version 4");
}); });
test("binding entry points reject callers that bypass the schema-v3 type contract", () => { test("binding entry points reject callers that bypass the schema-v4 type contract", () => {
expect(() => resolveBinding(unsupportedWorkspace as never, "DWH", {}, [])) expect(() => resolveBinding(unsupportedWorkspace as never, "DWH", {}, []))
.toThrow("Workspace bindings support only workspace schema version 3"); .toThrow("Workspace bindings support only workspace schema version 4");
expect(() => resolveEvidenceBinding(unsupportedWorkspace as never, {}, [])) expect(() => resolveEvidenceBinding(unsupportedWorkspace as never, {}, []))
.toThrow("Workspace bindings support only workspace schema version 3"); .toThrow("Workspace bindings support only workspace schema version 4");
expect(() => resolveRuntimeBindings(unsupportedWorkspace as never, {}, [])) expect(() => resolveRuntimeBindings(unsupportedWorkspace as never, {}, []))
.toThrow("Workspace bindings support only workspace schema version 3"); .toThrow("Workspace bindings support only workspace schema version 4");
}); });
test("diagnoser factories reject callers that bypass the schema-v3 type contract", async () => { test("diagnoser factories reject callers that bypass the schema-v4 type contract", async () => {
await expect(createWorkspaceDiagnoser(adapters)(unsupportedWorkspace as never, bindings, { await expect(createWorkspaceDiagnoser(adapters)(unsupportedWorkspace as never, bindings, {
writeProbe: false, writeProbe: false,
})).rejects.toThrow("Workspace diagnoser supports only workspace schema version 3"); })).rejects.toThrow("Workspace diagnoser supports only workspace schema version 4");
await expect(createProductionWorkspaceDiagnoser(5_000, adapters)( await expect(createProductionWorkspaceDiagnoser(5_000, adapters)(
unsupportedWorkspace as never, bindings, { writeProbe: false }, unsupportedWorkspace as never, bindings, { writeProbe: false },
)).rejects.toThrow("Workspace diagnoser supports only workspace schema version 3"); )).rejects.toThrow("Workspace diagnoser supports only workspace schema version 4");
}); });
+17 -103
View File
@@ -10,7 +10,7 @@ import {
} from "../src/workspaces/schema.js"; } from "../src/workspaces/schema.js";
export const validYaml = `workspace: export const validYaml = `workspace:
schema_version: 3 schema_version: 4
id: psd-clinical id: psd-clinical
name: Policlinico San Donato name: Policlinico San Donato
description: Clinical data warehouse workspace description: Clinical data warehouse workspace
@@ -25,33 +25,8 @@ dwh:
- postgres_direct - postgres_direct
- rest_api - rest_api
- ssh_tunnel - ssh_tunnel
semantic_index:
vector_store:
engine: qdrant
collection: psd-clinical
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
llm_policy:
default: zai/glm-5.2
allowed:
- zai/glm-5.2
- openai/gpt-5
`; `;
test("rejects a workspace whose embedding dimensions differ from its collection", () => {
expect(() => parseWorkspaceYaml(validYaml.replace("dimensions: 1024", "dimensions: 1536")))
.toThrow(/dimensions/i);
});
test("rejects an LLM default outside its allowlist", () => {
expect(() => parseWorkspaceYaml(validYaml.replace("- zai/glm-5.2", "- openai/gpt-5")))
.toThrow(/allowlist/i);
});
test("rejects unknown keys and invalid immutable IDs", () => { test("rejects unknown keys and invalid immutable IDs", () => {
expect(() => parseWorkspaceYaml(validYaml.replace(" language: it", " language: it\n label: PSD"))) expect(() => parseWorkspaceYaml(validYaml.replace(" language: it", " language: it\n label: PSD")))
.toThrow(/unrecognized key/i); .toThrow(/unrecognized key/i);
@@ -74,94 +49,34 @@ test("accepts optional connection ports and timeouts but rejects unsafe values",
.toThrow(/port/i); .toThrow(/port/i);
expect(() => parseWorkspaceYaml(validYaml.replace("timeout_ms: 5000", "timeout_ms: 0"))) expect(() => parseWorkspaceYaml(validYaml.replace("timeout_ms: 5000", "timeout_ms: 0")))
.toThrow(/timeout/i); .toThrow(/timeout/i);
expect(() => parseWorkspaceYaml(validYaml.replace("dimensions: 1024", "dimensions: 2048")))
.toThrow(/1024|dimensions/i);
}); });
test("accepts only the schema v3 internal qdrant semantic shape", () => { test("accepts a model-free schema v4 workspace", () => {
expect(parseWorkspaceYaml(validYaml)).toMatchObject({ expect(parseWorkspaceYaml(validYaml)).toMatchObject({
workspace: { schema_version: 3, id: "psd-clinical" }, workspace: { schema_version: 4, id: "psd-clinical" },
semantic_index: { dwh: { database: "postgres", schema: "datawarehouse" },
vector_store: {
engine: "qdrant",
collection: "psd-clinical",
dimensions: 1024,
distance: "cosine",
},
embedding: {
provider: "ollama_internal",
model: "qwen3-embedding:0.6b",
dimensions: 1024,
},
},
}); });
}); });
test("committed example descriptors parse as exact schema v3 workspaces", () => { test("committed example descriptors parse as exact schema v4 workspaces", () => {
const example = readFileSync(resolve(process.cwd(), "../deploy/workspaces/example.yaml"), "utf8"); const example = readFileSync(resolve(process.cwd(), "../deploy/workspaces/example.yaml"), "utf8");
const psdExample = readFileSync(resolve(process.cwd(), "../deploy/workspaces/psd.yaml.example"), "utf8"); const psdExample = readFileSync(resolve(process.cwd(), "../deploy/workspaces/psd.yaml.example"), "utf8");
expect(() => parseWorkspaceYaml(example)).not.toThrow(); expect(() => parseWorkspaceYaml(example)).not.toThrow();
expect(() => parseWorkspaceYaml(psdExample)).not.toThrow(); expect(() => parseWorkspaceYaml(psdExample)).not.toThrow();
expect(parseWorkspaceYaml(example)).toMatchObject({ expect(parseWorkspaceYaml(example)).toMatchObject({
workspace: { schema_version: 3 }, workspace: { schema_version: 4 },
semantic_index: {
vector_store: { engine: "qdrant", distance: "cosine", dimensions: 1024 },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
}); });
expect(parseWorkspaceYaml(psdExample)).toMatchObject({ expect(parseWorkspaceYaml(psdExample)).toMatchObject({
workspace: { schema_version: 3 }, workspace: { schema_version: 4 },
semantic_index: {
vector_store: { engine: "qdrant", distance: "cosine", dimensions: 1024 },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
}); });
}); });
test("rejects pgvector semantic stores in schema v3", () => { test("rejects installation-owned model and vector fields", () => {
expect(() => parseWorkspaceYaml(validYaml.replace("engine: qdrant", "engine: pgvector"))) expect(() => parseWorkspaceYaml(`${validYaml}llm_policy:\n allowed: [zai/glm-5.3]\n`))
.toThrow(/qdrant|pgvector/i); .toThrow(/unrecognized key|llm_policy/i);
}); expect(() => parseWorkspaceYaml(`${validYaml}semantic_index: {}\n`))
.toThrow(/unrecognized key|semantic_index/i);
test("rejects supported_transports inside schema v3 semantic identity", () => {
const withTransport = validYaml.replace(
" distance: cosine\n",
" distance: cosine\n supported_transports:\n - rest_api\n",
);
expect(() => parseWorkspaceYaml(withTransport)).toThrow(/unrecognized key|supported_transports/i);
});
test("rejects external embedding providers in schema v3", () => {
expect(() => parseWorkspaceYaml(validYaml.replace("provider: ollama_internal", "provider: openai_compatible")))
.toThrow(/ollama_internal|provider/i);
});
test("rejects non-cosine distance in schema v3", () => {
expect(() => parseWorkspaceYaml(validYaml.replace("distance: cosine", "distance: l2")))
.toThrow(/cosine|distance/i);
});
test("rejects unknown fields in schema v3 semantic identity", () => {
const withUnknownField = validYaml.replace(
" collection: psd-clinical\n",
" collection: psd-clinical\n namespace: psd\n",
);
expect(() => parseWorkspaceYaml(withUnknownField)).toThrow(/unrecognized key/i);
});
test("rejects legacy semantic connector fields and diagnostics in schema v3", () => {
expect(() => parseWorkspaceYaml(validYaml.replace(
" collection: psd-clinical\n",
" collection: psd-clinical\n database: postgres\n",
))).toThrow(/unrecognized key|database/i);
expect(() => parseWorkspaceYaml(validYaml.replace(
"llm_policy:\n",
"diagnostics:\n vector_rest:\n metadata:\n method: GET\n path: /metadata\n auth: bearer\n response:\n collection: collection\n dimensions: dimensions\n distance: distance\nllm_policy:\n",
))).toThrow(/unrecognized key|vector_rest/i);
}); });
test.each([ test.each([
@@ -222,8 +137,8 @@ llm_policy:
- zai/glm-5.2 - zai/glm-5.2
`], `],
])("rejects schema %s descriptors at parser and object-validator boundaries", (_version, yaml) => { ])("rejects schema %s descriptors at parser and object-validator boundaries", (_version, yaml) => {
expect(() => parseWorkspaceYaml(yaml)).toThrow(/schema_version|invalid literal|3/i); expect(() => parseWorkspaceYaml(yaml)).toThrow(/schema_version|invalid literal|4/i);
expect(() => validateWorkspaceDescriptor(parse(yaml))).toThrow(/schema_version|invalid literal|3/i); expect(() => validateWorkspaceDescriptor(parse(yaml))).toThrow(/schema_version|invalid literal|4/i);
}); });
test("does not expose the redundant canonical validator or v1 migration", () => { test("does not expose the redundant canonical validator or v1 migration", () => {
@@ -253,9 +168,8 @@ test("rejects REST diagnostic declarations without their matching connector tran
response: response:
database: database database: database
schema: schema schema: schema
llm_policy:
`; `;
const declared = validYaml.replace("llm_policy:\n", diagnostics); const declared = `${validYaml}${diagnostics}`;
expect(() => parseWorkspaceYaml(declared.replace(" - rest_api\n", ""))).toThrow(/dwh_rest/i); expect(() => parseWorkspaceYaml(declared.replace(" - rest_api\n", ""))).toThrow(/dwh_rest/i);
}); });
@@ -462,7 +376,7 @@ test.each(["curated/**/*.yaml", "curated/**"])(
}, },
); );
test("keeps evidence optional on schema v3", () => { test("keeps evidence optional on schema v4", () => {
expect(validateWorkspaceDescriptor(validWorkspaceObject())).not.toHaveProperty("evidence"); expect(validateWorkspaceDescriptor(validWorkspaceObject())).not.toHaveProperty("evidence");
}); });
@@ -471,7 +385,7 @@ test("serializes defaulted evidence canonically and parses it without loss", ()
type: "filesystem", type: "filesystem",
uri: "psd-clinical/evidence", uri: "psd-clinical/evidence",
})); }));
if (canonical.workspace.schema_version !== 3) throw new Error("expected schema v3"); if (canonical.workspace.schema_version !== 4) throw new Error("expected schema v4");
expect(parseWorkspaceYaml(serializeWorkspaceYaml(canonical))).toEqual(canonical); expect(parseWorkspaceYaml(serializeWorkspaceYaml(canonical))).toEqual(canonical);
}); });
@@ -46,7 +46,7 @@ async function fixture(): Promise<{ root: string; remote: string; commit: string
await git(source, ["config", "user.email", "evidence-materializer@example.invalid"]); await git(source, ["config", "user.email", "evidence-materializer@example.invalid"]);
writeFileSync(join(source, "thoth-workspaces.yaml"), "schema_version: 1\nworkspaces: [{id: research, name: Research}]\n"); writeFileSync(join(source, "thoth-workspaces.yaml"), "schema_version: 1\nworkspaces: [{id: research, name: Research}]\n");
mkdirSync(join(source, "research", "evidence", "nested"), { recursive: true }); mkdirSync(join(source, "research", "evidence", "nested"), { recursive: true });
writeFileSync(join(source, "research", "workspace.yaml"), "workspace:\n schema_version: 3\n id: research\n"); writeFileSync(join(source, "research", "workspace.yaml"), "workspace:\n schema_version: 4\n id: research\n");
writeFileSync(join(source, "research", "evidence", "guide.md"), "# guide\n"); writeFileSync(join(source, "research", "evidence", "guide.md"), "# guide\n");
writeFileSync(join(source, "research", "evidence", "nested", "deep.md"), "# deep\n"); writeFileSync(join(source, "research", "evidence", "nested", "deep.md"), "# deep\n");
await git(source, ["add", "-A"]); await git(source, ["add", "-A"]);
@@ -102,7 +102,7 @@ test("refuses symlink-containing trees and bound violations without publishing",
await git(source, ["config", "user.email", "e@e.invalid"]); await git(source, ["config", "user.email", "e@e.invalid"]);
writeFileSync(join(source, "thoth-workspaces.yaml"), "schema_version: 1\nworkspaces: [{id: research, name: Research}]\n"); writeFileSync(join(source, "thoth-workspaces.yaml"), "schema_version: 1\nworkspaces: [{id: research, name: Research}]\n");
mkdirSync(join(source, "research", "evidence"), { recursive: true }); mkdirSync(join(source, "research", "evidence"), { recursive: true });
writeFileSync(join(source, "research", "workspace.yaml"), "workspace:\n schema_version: 3\n id: research\n"); writeFileSync(join(source, "research", "workspace.yaml"), "workspace:\n schema_version: 4\n id: research\n");
writeFileSync(join(source, "research", "outside.md"), "# outside\n"); writeFileSync(join(source, "research", "outside.md"), "# outside\n");
symlinkSync("../outside.md", join(source, "research", "evidence", "link.md")); symlinkSync("../outside.md", join(source, "research", "evidence", "link.md"));
await git(source, ["add", "-A"]); await git(source, ["add", "-A"]);
-7
View File
@@ -36,15 +36,11 @@ services:
THT_LLM_URL: ${THT_LLM_URL:-} THT_LLM_URL: ${THT_LLM_URL:-}
THT_INTERNAL_QDRANT_URL: http://qdrant:6333 THT_INTERNAL_QDRANT_URL: http://qdrant:6333
THT_INTERNAL_EMBEDDING_URL: http://embedding:11434 THT_INTERNAL_EMBEDDING_URL: http://embedding:11434
THT_INTERNAL_EMBEDDING_MODEL: qwen3-embedding:0.6b
THT_INTERNAL_EMBEDDING_DIMENSIONS: "1024"
MAX_PI_PROCESSES: ${MAX_PI_PROCESSES:-4} MAX_PI_PROCESSES: ${MAX_PI_PROCESSES:-4}
volumes: volumes:
- settings:/data/settings - settings:/data/settings
- pi-state:/home/thoth/.pi - pi-state:/home/thoth/.pi
- ${PI_AUTH_FILE:?set PI_AUTH_FILE}:/home/thoth/.pi/agent/auth.json:ro - ${PI_AUTH_FILE:?set PI_AUTH_FILE}:/home/thoth/.pi/agent/auth.json:ro
- ./deploy/pi/models.json:/home/thoth/.pi/agent/models.json:ro
- ./deploy/pi/settings.json:/home/thoth/.pi/agent/settings.json:ro
- workspace-registry:/data/workspace-registry - workspace-registry:/data/workspace-registry
- workspace-secrets:/data/workspace-secrets - workspace-secrets:/data/workspace-secrets
- sessions:/data/sessions - sessions:/data/sessions
@@ -140,8 +136,6 @@ services:
THT_LLM_URL: ${THT_LLM_URL:-} THT_LLM_URL: ${THT_LLM_URL:-}
THT_INTERNAL_QDRANT_URL: http://qdrant:6333 THT_INTERNAL_QDRANT_URL: http://qdrant:6333
THT_INTERNAL_EMBEDDING_URL: http://embedding:11434 THT_INTERNAL_EMBEDDING_URL: http://embedding:11434
THT_INTERNAL_EMBEDDING_MODEL: qwen3-embedding:0.6b
THT_INTERNAL_EMBEDDING_DIMENSIONS: "1024"
HOME: /tmp/thoth HOME: /tmp/thoth
AWS_ACCESS_KEY_ID: "" AWS_ACCESS_KEY_ID: ""
AWS_SECRET_ACCESS_KEY: "" AWS_SECRET_ACCESS_KEY: ""
@@ -243,7 +237,6 @@ services:
entrypoint: ["/usr/bin/bash", "/opt/thoth/embedding-model-init.sh"] entrypoint: ["/usr/bin/bash", "/opt/thoth/embedding-model-init.sh"]
environment: environment:
OLLAMA_BASE_URL: http://embedding:11434 OLLAMA_BASE_URL: http://embedding:11434
OLLAMA_MODEL: qwen3-embedding:0.6b
OLLAMA_WAIT_TIMEOUT_SEC: "180" OLLAMA_WAIT_TIMEOUT_SEC: "180"
volumes: volumes:
- embedding-models:/root/.ollama - embedding-models:/root/.ollama
-8
View File
@@ -21,14 +21,6 @@ services:
source: ${PI_AUTH_FILE:?set PI_AUTH_FILE} source: ${PI_AUTH_FILE:?set PI_AUTH_FILE}
target: /home/thoth/.pi/agent/auth.json target: /home/thoth/.pi/agent/auth.json
read_only: true read_only: true
- type: bind
source: ./deploy/pi/models.json
target: /home/thoth/.pi/agent/models.json
read_only: true
- type: bind
source: ./deploy/pi/settings.json
target: /home/thoth/.pi/agent/settings.json
read_only: true
- type: bind - type: bind
source: ${THT_WORKSPACE_REGISTRY_ROOT:?set THT_WORKSPACE_REGISTRY_ROOT} source: ${THT_WORKSPACE_REGISTRY_ROOT:?set THT_WORKSPACE_REGISTRY_ROOT}
target: /data/workspace-registry target: /data/workspace-registry
-46
View File
@@ -1,46 +0,0 @@
{
"providers": {
"zai": {
"baseUrl": "https://api.z.ai/api/coding/paas/v4",
"api": "openai-completions",
"apiKey": "$ZAI_API_KEY",
"models": [
{
"id": "glm-5.3",
"name": "GLM-5.3",
"reasoning": true,
"contextWindow": 200000,
"maxTokens": 131072
}
]
},
"local-qwen": {
"name": "Local Qwen",
"baseUrl": "https://ml-aritmolab.policlinicosandonato.it/v1",
"api": "openai-completions",
"apiKey": "local",
"models": [
{
"id": "qwen3.6-35b-a3b",
"name": "Qwen3.6 35B A3B",
"reasoning": false,
"input": ["text"],
"cost": {
"input": 0,
"output": 0,
"cacheRead": 0,
"cacheWrite": 0
},
"contextWindow": 131072,
"maxTokens": 16384,
"compat": {
"supportsDeveloperRole": false,
"supportsReasoningEffort": false,
"supportsStore": false,
"maxTokensField": "max_tokens"
}
}
]
}
}
}
-9
View File
@@ -1,9 +0,0 @@
{
"defaultProjectTrust": "always",
"enabledModels": [
"zai/glm-5.3",
"deepseek/deepseek-v4-flash",
"deepseek/deepseek-v4-pro",
"local-qwen/qwen3.6-35b-a3b"
]
}
+1 -3
View File
@@ -14,9 +14,7 @@ THT_AUTH_CONFIG_ROOT=<abs>/deploy/psd/auth
# DWH and Evidence credentials are entered later in Workspace management and stored encrypted # DWH and Evidence credentials are entered later in Workspace management and stored encrypted
# by the backend. They do not depend on host filesystem paths. # by the backend. They do not depend on host filesystem paths.
# Pi (LLM) # Pi runtime preference; provider/model defaults live only in installation modelCatalog.
PI_PROVIDER=zai
PI_MODEL=glm-5.3
PI_THINKING=medium PI_THINKING=medium
# App defaults # App defaults
+64 -35
View File
@@ -1,6 +1,6 @@
# Esempio soltanto: `tht setup` genera deploy/<installation-id>/thothii-installation.yaml. # Example only: `tht setup` generates deploy/<installation-id>/thothii-installation.yaml.
# Sostituisci i path assoluti se usi questo riferimento per una configurazione avanzata. # Replace every absolute path before using this as an advanced reference.
# Seleziona UN solo override Git (https o ssh). schemaVersion: 2
profile: local profile: local
projectDirectory: "<abs>/projects/ThothII" projectDirectory: "<abs>/projects/ThothII"
envFile: "<abs>/projects/ThothII/deploy/psd/operator.env" envFile: "<abs>/projects/ThothII/deploy/psd/operator.env"
@@ -8,38 +8,67 @@ workspaceRepository:
remote: git@github.com:mptyl/tht-workspace-psd.git remote: git@github.com:mptyl/tht-workspace-psd.git
branch: main branch: main
access: ssh access: ssh
metadataGeneration: modelCatalog:
default: glm-53 defaults:
models: session: zai/glm-5.3
- id: deepseek-v4-pro metadataGeneration: zai/glm-5.3
label: DeepSeek V4 Pro embedding:
litellm: id: ollama/qwen3-embedding:0.6b
provider: deepseek dimensions: 1024
model: deepseek-v4-pro providers:
apiKeyEnv: DEEPSEEK_API_KEY deepseek:
- id: deepseek-v4-flash authentication:
label: DeepSeek V4 Flash mode: pi_auth
litellm: session:
provider: deepseek mode: pi_builtin
model: deepseek-v4-flash models:
apiKeyEnv: DEEPSEEK_API_KEY deepseek-v4-pro:
- id: glm-53 session: {}
label: GLM 5.3 deepseek-v4-flash:
litellm: session: {}
provider: openai zai:
model: glm-5.3 endpoint:
endpoint: baseUrl: https://api.z.ai/api/coding/paas/v4
baseUrl: https://api.z.ai/api/coding/paas/v4 authentication:
apiKeyEnv: ZAI_API_KEY mode: secret_env
- id: qwen-36 apiKeyEnv: ZAI_API_KEY
label: AritmoLab Qwen 3.6 35B A3B session:
litellm: mode: openai_compatible
provider: openai metadataGeneration:
model: qwen3.6-35b-a3b litellmProvider: openai
disableThinking: true models:
endpoint: glm-5.3:
baseUrl: https://ml-aritmolab.policlinicosandonato.it/v1 label: GLM 5.3
# Qwen omette apiKeyEnv: l'endpoint VPN non autentica le richieste. session:
reasoning: true
contextWindow: 200000
maxTokens: 131072
metadataGeneration: {}
local-qwen:
endpoint:
baseUrl: https://ml-aritmolab.policlinicosandonato.it/v1
authentication:
mode: none
session:
mode: openai_compatible
metadataGeneration:
litellmProvider: openai
models:
qwen3.6-35b-a3b:
label: AritmoLab Qwen 3.6 35B A3B
session:
reasoning: false
input: [text]
cost: {input: 0, output: 0, cacheRead: 0, cacheWrite: 0}
contextWindow: 131072
maxTokens: 16384
compatibility:
supportsDeveloperRole: false
supportsReasoningEffort: false
supportsStore: false
maxTokensField: max_tokens
metadataGeneration:
disableThinking: true
authentication: authentication:
configDirectory: "<abs>/projects/ThothII/deploy/psd/auth" configDirectory: "<abs>/projects/ThothII/deploy/psd/auth"
overrides: overrides:
+8 -7
View File
@@ -10,8 +10,9 @@ chmod 600 deploy/secrets/thothii.secrets
The file uses strict `KEY=VALUE` lines (comments and blank lines are allowed). The supported The file uses strict `KEY=VALUE` lines (comments and blank lines are allowed). The supported
installation keys are `THT_MODEL_API_KEY`, `THT_DWH_API_KEY`, `THT_CA`, `THT_SSL_CA`, installation keys are `THT_MODEL_API_KEY`, `THT_DWH_API_KEY`, `THT_CA`, `THT_SSL_CA`,
`THT_OIDC_CLIENT_SECRET`, and `THT_AUTHENTIK_API_TOKEN`. Metadata-generation models may reference `THT_OIDC_CLIENT_SECRET`, and `THT_AUTHENTIK_API_TOKEN`. Installation Model Catalog providers may
exactly one of `THT_METADATA_API_KEY`, `ANTHROPIC_API_KEY`, `AZURE_API_KEY`, `GEMINI_API_KEY`, reference exactly one of `THT_MODEL_API_KEY`, `THT_METADATA_API_KEY`, `ANTHROPIC_API_KEY`,
`AZURE_API_KEY`, `GEMINI_API_KEY`,
`DEEPSEEK_API_KEY`, `OPENAI_API_KEY`, `OPENROUTER_API_KEY`, or `ZAI_API_KEY` through their `DEEPSEEK_API_KEY`, `OPENAI_API_KEY`, `OPENROUTER_API_KEY`, or `ZAI_API_KEY` through their
descriptor `apiKeyEnv`. An entry for an explicitly configured endpoint that accepts unauthenticated descriptor `apiKeyEnv`. An entry for an explicitly configured endpoint that accepts unauthenticated
requests may omit `apiKeyEnv`; hosted/default endpoints must always reference a key. requests may omit `apiKeyEnv`; hosted/default endpoints must always reference a key.
@@ -23,10 +24,10 @@ installation. Other configured values must be non-empty and contain no
whitespace. Do not put secrets in the root `.env`, installation YAML, workspace YAML, URLs, logs, whitespace. Do not put secrets in the root `.env`, installation YAML, workspace YAML, URLs, logs,
or rendered Compose output. or rendered Compose output.
`THT_MODEL_API_KEY` remains the generic Pi child credential. Metadata generation is a separate Session and metadata-generation runtimes read only the provider key named by
backend-owned runtime and reads only the key named by its own `metadataGeneration.models[].apiKeyEnv` `modelCatalog.providers.<provider>.authentication.apiKeyEnv`. They share the declaration and
(when present); credential reference, not their execution lifecycle. Pi-owned authentication remains available only
it does not read Pi settings, `PI_AUTH_FILE`, or workspace `llm_policy`. to session-only built-in providers through `authentication.mode: pi_auth`.
Do not add vector or embedding endpoint credentials to the bundle. Active operator manuals use Do not add vector or embedding endpoint credentials to the bundle. Active operator manuals use
internal Qdrant and Ollama services, so vector/embedding runtime endpoint secrets are not part of internal Qdrant and Ollama services, so vector/embedding runtime endpoint secrets are not part of
@@ -56,7 +57,7 @@ and only then deleting the old files. The old variables remain a compatibility p
upgrades, but the documented and tested default is an absolute `THT_SECRETS_FILE` path to the upgrades, but the documented and tested default is an absolute `THT_SECRETS_FILE` path to the
protected bundle. protected bundle.
Hosted Pi providers must use a single model key through `THT_MODEL_API_KEY`. Compound providers Hosted providers must use one explicitly named catalog key. Compound providers
(Bedrock, Azure OpenAI Responses, Cloudflare Workers AI/Gateway) fail closed until a (Bedrock, Azure OpenAI Responses, Cloudflare Workers AI/Gateway) fail closed until a
provider-specific credential adapter is implemented. provider-specific credential adapter is implemented.
+5 -6
View File
@@ -2,14 +2,13 @@
# Values are read as literal strings (no shell expansion or command substitution). # Values are read as literal strings (no shell expansion or command substitution).
# Leave unused keys out of the file. # Leave unused keys out of the file.
# Hosted model provider (single-key providers only). # Hosted catalog provider (single-key providers only). The selected name must match
# THT_MODEL_API_KEY=replace-me # modelCatalog.providers.<provider>.authentication.apiKeyEnv.
# Allowed names include THT_MODEL_API_KEY, THT_METADATA_API_KEY, ANTHROPIC_API_KEY,
# Description Generation only. The selected name must match apiKeyEnv in metadataGeneration. # AZURE_API_KEY, GEMINI_API_KEY,
# Allowed names: THT_METADATA_API_KEY, ANTHROPIC_API_KEY, AZURE_API_KEY, GEMINI_API_KEY,
# DEEPSEEK_API_KEY, OPENAI_API_KEY, OPENROUTER_API_KEY, or ZAI_API_KEY. # DEEPSEEK_API_KEY, OPENAI_API_KEY, OPENROUTER_API_KEY, or ZAI_API_KEY.
# OPENAI_API_KEY=replace-me # OPENAI_API_KEY=replace-me
# A model with an explicit unauthenticated endpoint omits apiKeyEnv and needs no bundle entry. # A provider with an explicit keyless endpoint uses authentication.mode: none.
# External DWH adapter. # External DWH adapter.
# THT_DWH_API_KEY=replace-me # THT_DWH_API_KEY=replace-me
+2 -18
View File
@@ -1,8 +1,8 @@
workspace: workspace:
schema_version: 3 schema_version: 4
id: example id: example
name: Example workspace name: Example workspace
description: Generic example WorkspaceV3 descriptor. description: Generic example WorkspaceV4 descriptor.
language: en language: en
dwh: dwh:
@@ -13,17 +13,6 @@ dwh:
- postgres_direct - postgres_direct
- rest_api - rest_api
semantic_index:
vector_store:
engine: qdrant
collection: example
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
evidence: evidence:
source: source:
type: filesystem type: filesystem
@@ -35,11 +24,6 @@ evidence:
max_chunk_chars: 4000 max_chunk_chars: 4000
retain_published_generations: 3 retain_published_generations: 3
llm_policy:
default: zai/glm-5.2
allowed:
- zai/glm-5.2
diagnostics: diagnostics:
dwh_rest: dwh_rest:
method: GET method: GET
+2 -18
View File
@@ -1,8 +1,8 @@
workspace: workspace:
schema_version: 3 schema_version: 4
id: example-workspace id: example-workspace
name: Example Workspace name: Example Workspace
description: Example WorkspaceV3 descriptor. description: Example WorkspaceV4 descriptor.
language: en language: en
dwh: dwh:
@@ -13,17 +13,6 @@ dwh:
- postgres_direct - postgres_direct
- rest_api - rest_api
semantic_index:
vector_store:
engine: qdrant
collection: example-workspace
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
evidence: evidence:
source: source:
type: filesystem type: filesystem
@@ -35,11 +24,6 @@ evidence:
max_chunk_chars: 4000 max_chunk_chars: 4000
retain_published_generations: 3 retain_published_generations: 3
llm_policy:
default: zai/glm-5.2
allowed:
- zai/glm-5.2
diagnostics: diagnostics:
dwh_rest: dwh_rest:
method: GET method: GET
+2 -8
View File
@@ -1,6 +1,5 @@
# ThothII standalone development/smoke stack. # ThothII standalone development/smoke base. Model settings are intentionally absent;
# Run with the canonical local env file: # use `tht start --build`, which adds the generated modelCatalog projection.
# docker compose --env-file deploy/env/local.env -f docker-compose.dev.yml up -d --build
# frontend: http://localhost:8090 backend: http://localhost:8787 # frontend: http://localhost:8090 backend: http://localhost:8787
name: thothii-dev name: thothii-dev
@@ -36,8 +35,6 @@ services:
THT_LLM_URL: ${THT_LLM_URL:-} THT_LLM_URL: ${THT_LLM_URL:-}
THT_INTERNAL_QDRANT_URL: http://qdrant:6333 THT_INTERNAL_QDRANT_URL: http://qdrant:6333
THT_INTERNAL_EMBEDDING_URL: http://embedding:11434 THT_INTERNAL_EMBEDDING_URL: http://embedding:11434
THT_INTERNAL_EMBEDDING_MODEL: qwen3-embedding:0.6b
THT_INTERNAL_EMBEDDING_DIMENSIONS: "1024"
MAX_PI_PROCESSES: ${MAX_PI_PROCESSES:-4} MAX_PI_PROCESSES: ${MAX_PI_PROCESSES:-4}
extra_hosts: extra_hosts:
- "host.docker.internal:host-gateway" - "host.docker.internal:host-gateway"
@@ -45,8 +42,6 @@ services:
- dev-data:/data - dev-data:/data
- dev-pi-state:/home/thoth/.pi - dev-pi-state:/home/thoth/.pi
- ${PI_AUTH_FILE:?set PI_AUTH_FILE}:/home/thoth/.pi/agent/auth.json:ro - ${PI_AUTH_FILE:?set PI_AUTH_FILE}:/home/thoth/.pi/agent/auth.json:ro
- ./deploy/pi/models.json:/home/thoth/.pi/agent/models.json:ro
- ./deploy/pi/settings.json:/home/thoth/.pi/agent/settings.json:ro
- workspace-registry:/data/workspace-registry - workspace-registry:/data/workspace-registry
- workspace-secrets:/data/workspace-secrets - workspace-secrets:/data/workspace-secrets
- ${THT_DEV_EVIDENCE_HOST_PATH:-./evidence}:/data/evidence:ro - ${THT_DEV_EVIDENCE_HOST_PATH:-./evidence}:/data/evidence:ro
@@ -129,7 +124,6 @@ services:
entrypoint: ["/usr/bin/bash", "/opt/thoth/embedding-model-init.sh"] entrypoint: ["/usr/bin/bash", "/opt/thoth/embedding-model-init.sh"]
environment: environment:
OLLAMA_BASE_URL: http://embedding:11434 OLLAMA_BASE_URL: http://embedding:11434
OLLAMA_MODEL: qwen3-embedding:0.6b
OLLAMA_WAIT_TIMEOUT_SEC: "180" OLLAMA_WAIT_TIMEOUT_SEC: "180"
volumes: volumes:
- embedding-models:/root/.ollama - embedding-models:/root/.ollama

Some files were not shown because too many files have changed in this diff Show More