feat: unify installation model catalog

This commit is contained in:
Codex
2026-09-02 18:45:33 +02:00
parent ae053961a3
commit 7b7927bfe5
169 changed files with 3696 additions and 4572 deletions
+7 -6
View File
@@ -83,7 +83,8 @@ frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness →
`SseHub` fans them out over SSE to the browser. The separate PostgreSQL catalog stores database
metadata and sequential AI description-generation runs. Description generation samples the DWH
through read-only connectors and calls a short-lived Python LiteLLM helper; it does not use Pi or
expose a public CLI command. App settings still live in `backend/data/settings.json`.
expose a public CLI command. Sessions, metadata generation, and embedding resolve models from the
generated Installation Model Catalog; `thothii-installation.yaml` is its only authored source.
- **Human-in-the-loop gate contract.** The model proposes; a human reviewer decides at gates
via widgets (`reviewer_select` = single pick — a chosen option carrying a `decision` payload
@@ -99,11 +100,11 @@ frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness →
- **`--json` output must be pristine** (only valid JSON on stdout) — used as a machine contract.
- **UI strings are English; document *content* stays the workspace language** (Italian for
`psd`) because it's the real data. Only chrome/labels are English.
- **Workspaces** (`harness/workspaces/*.yaml`) set the DB target and **absolute**
`paths.sessions/artifacts/indexes` — for `psd` these point at a *separate, uncommitted* repo
(`tht-workspace-psd/`). Secrets live ONLY in `harness/.env` (gitignored).
- **Settings are global** (`backend/data/settings.json`: workspace/provider/model/thinking);
the New-session form is question-only.
- **Workspace schema v4** defines database and Evidence concerns only. Embedding/model facts come
from the installation catalog. The legacy `harness/workspaces/*.yaml` runtime snapshots still use
absolute session/artifact/index paths; secrets stay in `harness/.env` (gitignored).
- **Settings are global** (`backend/data/settings.json`: workspace/thinking). Provider/model choices
are ephemeral canonical catalog selections pinned into the session manifest.
- **Resume**: a resumable session re-enters at its last incomplete phase. The backend refuses
resume with 409 when `finalized` or `archived`, and `PiProcessManager.spawnFor` must send
`/riprendi-sessione <id>` (resume mode) vs `/nuova-domanda` (new) — sending the wrong prompt
+22 -3
View File
@@ -1,6 +1,6 @@
# ThothII — Project State
Last updated: 2026-08-31.
Last updated: 2026-09-02.
This file is the short operational snapshot. Stable commands and the architecture mental model
live in `AGENTS.md`; current design and runtime contracts live under `docs/architecture/`,
@@ -59,10 +59,28 @@ tht --installation /absolute/path/thothii-installation.yaml workspace preprocess
These commands use the profile-gated `workspace-maintenance` service. The former standalone
preprocessing Compose fixtures are retired.
Workspace descriptors use schema v3. For PSD, workspace content and runtime roots point to the
Workspace descriptors use schema v4 and contain only database, Evidence, diagnostics, and binding
concerns; model, provider, embedding, and vector-store configuration is installation-owned. For
PSD, workspace content and runtime roots point to the
separate uncommitted repository `/Users/mp/projects/tht-workspace-psd`. Secrets remain outside
Git and are supplied only through installation-local protected files.
## Installation Model Catalog
`thothii-installation.yaml` schema version 2 is the only operator-authored source for session,
metadata-generation, and embedding models. The host `tht` lifecycle validates `modelCatalog` and
regenerates the backend catalog, Pi `models.json`/`settings.json`, and Compose override under the
installation-local `generated/` directory. Those projections are replaceable runtime adapters:
they are not edited, backed up, or treated as configuration.
Session and metadata defaults use canonical `provider/model` IDs. Provider authentication declares
one explicit mode (`secret_env`, `pi_auth`, or `none`); `secret_env` names a protected bundle key.
The backend settings store now owns only the selected workspace and thinking level. Existing v1
installations use the explicit catalog migration command; schema-v3 workspace descriptors are
converted deterministically in their curator-owned repository before commit. Strict runtime loading
does not silently infer or merge legacy sources. ADR 0013 and
`docs/plans/2026-09-02-installation-model-catalog.md` record the decision and implementation.
## Database management
The database, table, and authoritative physical-schema catalog slices are implemented. Database
@@ -164,7 +182,8 @@ available only when no local start, worker, or helper is live. Runs remain inspe
live SSE log with ordered polling fallback; there is no automatic resume or user-facing generation
CLI. ADRs 0009–0010 record the runtime and source-sampling decisions.
Metadata-generation setup accepts the protected `DEEPSEEK_API_KEY` and `ZAI_API_KEY` references.
The Installation Model Catalog accepts the protected `DEEPSEEK_API_KEY` and `ZAI_API_KEY`
references for metadata-generation providers.
It also accepts a model with no secret reference only when its OpenAI-compatible endpoint is
explicit; this covers the VPN-only AritmoLab Qwen 3.6 server without creating a fake operator
credential. The Python client supplies only its fixed non-secret compatibility placeholder.
+24 -34
View File
@@ -106,15 +106,18 @@ a remote user's partial list. The isolated deployment exercise is
`./scripts/verify-workspace-install-docs.sh --profile local` or `--profile server`.
<!-- workspace-descriptor-contract:start -->
Schema v3 is the only accepted workspace descriptor. Schema v1 and v2 workspace descriptors are
rejected before activation. Candidate snapshot validation therefore makes activation or a pull fail
atomically while the prior valid snapshot remains active. There is no in-product migrator or
automatic conversion. A repository must already contain reviewed v3 descriptors. One workspace
owns one Qdrant collection;
Schema v4 is the only accepted workspace descriptor. Schema v1, v2, and v3 workspace descriptors
are rejected before activation. Candidate snapshot validation therefore makes activation or a pull
fail atomically while the prior valid snapshot remains active. One workspace owns one Qdrant collection;
schema, Evidence, and Memory records share that collection and stay separated by indexed payload
`kind`.
<!-- workspace-descriptor-contract:end -->
<!-- non-workspace-migration:start -->
Convert a v3 descriptor before publication by setting `workspace.schema_version` to `4` and
removing `llm_policy` and `semantic_index`; no database or Evidence field changes.
<!-- non-workspace-migration:end -->
For NL→SQL runtime sessions, connector `ssh_tunnel` bindings remain diagnostic-only: their bounded
probe cleans up the loopback forward and returns `workspace_not_activatable`; session creation is
rejected before persistence. Database management is a separate boundary and supports a strict
@@ -176,10 +179,10 @@ secret files, upstream-auth checks, and a fail-closed `503` assertion for its de
unavailable disposable session endpoint. No real provider, database credential, or repository
secret is required.
For a clean server bind, `scripts/prepare-server-pi-state.sh` creates the hidden regular
`agent/auth.json`, `agent/models.json`, and `agent/settings.json` mount targets atomically before
Compose. The server smoke starts from an empty Pi-state root and applies this same preflight; the
real protected/tracked sources remain separate read-only mounts. Deterministic fixture tests render
For a clean server bind, `scripts/prepare-server-pi-state.sh` creates the hidden regular Pi agent
mount targets atomically before Compose. The auth target receives the protected credential bind;
the model and settings targets receive generated read-only projections. The server smoke starts
from an empty Pi-state root and applies this same preflight. Deterministic fixture tests render
both profiles, verify that bindings stay on `core`, check mount readability, and run the production
workspace resolver. Wrong-service, wrong-value, and broken-secret-mount mutations must fail.
@@ -189,7 +192,7 @@ an independent 32-minute outer timeout and does not retry a failed command.
Current release status (2026-08-05): clean-root render/setup and the production runtime-binding
resolver contracts are green. The server fixture supplies all four private trusted claims,
including exact non-admin value `0`, and a focused test proves nginx normalization produces the
accepted non-admin backend principal. Canonical schema-v3 registry descriptors now pass through
accepted non-admin backend principal. Canonical schema-v4 registry descriptors now pass through
one backend-owned, secret-safe runtime handoff for inventory and session execution; canonical
identity and durable session/artifact/index roots are retained. The fresh update-only smoke passed
bad-candidate mutation, automatic `rolled_back` compensation, exact prior-image restoration,
@@ -295,14 +298,12 @@ Copy `deploy/secrets/thothii.secrets.example` to a protected host file, include
keys, and set its absolute path as `THT_SECRETS_FILE` in the operator env. Keep Pi's native
provider auth in the separate protected file named by `PI_AUTH_FILE`.
Description Generation is configured independently in the protected installation descriptor under
`metadataGeneration`. Set `THT_INSTALLATION_CONFIG_SOURCE` to that exact host file; Compose mounts
it read-only into `core` and supplies the fixed runtime `THT_INSTALLATION_CONFIG_FILE` path. Each
keyed model stores only an audited `apiKeyEnv` reference. The referenced value stays in the secret
bundle; a model may omit `apiKeyEnv` only when it declares an explicit endpoint that accepts
unauthenticated requests. The browser receives only model IDs, labels, and the configured default.
Configuration changes take effect after restart and do not use Pi settings or workspace
`llm_policy`.
Interactive sessions, Description Generation, and embedding share the protected installation
descriptor's `modelCatalog`. Set `THT_INSTALLATION_CONFIG_SOURCE` to that exact host file; `tht`
validates it and generates the runtime catalog, Pi adapters, and Compose override before startup.
Each authenticated provider stores only an audited `apiKeyEnv` reference; the referenced value stays
in the secret bundle. A provider may use `authentication.mode: none` only with an explicit keyless
endpoint. The browser receives only eligible model IDs, labels, and the catalog default.
Before enabling Description Generation, approve the selected model provider for bounded source-data
disclosure. Every catalog column has a **Sensitive** flag that defaults to `false`. Administrators can
@@ -333,22 +334,11 @@ the host/secret-manager materialization and add a reviewed Compose override that
does not create that mount. The frontend remains on loopback; the authenticated host proxy is the
only public listener.
Set the selected model provider in application settings (or `PI_PROVIDER`). For each Pi spawn the
backend validates and reads `THT_MODEL_API_KEY` from the bundle, then exposes its value only as the provider's
recognized child variable (for example `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GEMINI_API_KEY`, or
`ZAI_API_KEY`). Neither the generic file path nor deprecated `PI_PROVIDER_API_KEY` is inherited by
Pi. Local providers such as Ollama require no model key.
`THT_MODEL_API_KEY` supports Pi providers whose authentication is exactly one key:
`ant-ling`, `anthropic`, `cerebras`, `deepseek`, `fireworks`, `github-copilot`, `google`
(including the `gemini` alias), `google-vertex` when using its API-key mode, `groq`,
`huggingface`, `kimi-coding`, `minimax`, `minimax-cn`, `mistral`, `moonshotai`,
`moonshotai-cn`, `nvidia`, `openai`, `opencode`, `opencode-go`, `openrouter`, `together`,
`vercel-ai-gateway`, `xai`, the four `xiaomi*` providers, `zai`, and `zai-coding-cn`.
Compound providers are deliberately unsupported: `amazon-bedrock`, `azure-openai-responses`,
`cloudflare-workers-ai`, and `cloudflare-ai-gateway` require multiple credential/configuration
values. Selecting one fails before Pi starts; ambient AWS, Azure, and Cloudflare credentials are
still scrubbed. Supporting them requires a future dedicated provider-specific configuration.
For each Pi spawn, the backend resolves the selected canonical provider/model in the runtime catalog,
reads exactly that provider's declared `apiKeyEnv` value from the bundle, and exposes only that key
to the child. Ambient provider credentials and secret-bundle paths are scrubbed. Providers needing a
compound credential bundle remain unsupported until the catalog gains an explicit generic contract
for them.
## User-owned session server cutover
+2 -1
View File
@@ -9,7 +9,8 @@
"catalog:migrate": "node dist/catalog/migrate.js",
"test": "vitest run",
"start": "node dist/server.js",
"test:schema-v3-verifier": "python3 -I -B scripts/test_revision_state_policy.py && node --test scripts/verify-workspace-descriptor-files.test.mjs scripts/revision-state-policy.test.mjs"
"test:schema-v4-verifier": "python3 -I -B scripts/test_revision_state_policy.py && node --test scripts/verify-workspace-descriptor-files.test.mjs scripts/revision-state-policy.test.mjs",
"test:schema-v3-verifier": "npm run test:schema-v4-verifier"
},
"dependencies": {
"@fastify/cookie": "11.1.2",
+1 -6
View File
@@ -1195,13 +1195,8 @@ export async function executeChecks({ checks, failAt, recorder } = {}) {
function baseWorkspace(id, evidenceSource) {
return {
workspace: { schema_version: 3, id, name: `P1 ${id}`, language: "en" },
workspace: { schema_version: 4, id, name: `P1 ${id}`, language: "en" },
dwh: { engine: "postgres", database: "postgres", schema: "public", supported_transports: ["postgres_direct"] },
semantic_index: {
vector_store: { engine: "qdrant", collection: id, dimensions: 1024, distance: "cosine" },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
llm_policy: { allowed: ["zai/glm-5.2"] },
evidence: { source: evidenceSource, policy: { max_chunk_chars: 4000, retain_published_generations: 3 } },
};
}
+1 -1
View File
@@ -109,7 +109,7 @@ async function validateDistFiles(repo,files){const dist=join(repo,"backend","dis
export async function readManualOwnership({repositoryRoot=defaultRepositoryRoot}={}){const repo=realpathSync(repositoryRoot),root=fixedManualRoot(repo);noSymlinkExisting(repo,root);let rootEntry,ownershipEntry;try{rootEntry=await lstat(root);ownershipEntry=await lstat(join(root,"ownership.json"));}catch{throw new Error("manual ownership is missing");}if(!rootEntry.isDirectory()||rootEntry.isSymbolicLink()||await realpath(root)!==root||!ownershipEntry.isFile()||ownershipEntry.isSymbolicLink())throw new Error("manual ownership is unsafe");let value;try{value=JSON.parse(await readFile(join(root,"ownership.json"),"utf8"));}catch{throw new Error("manual ownership is malformed");}const baseValid=value.schemaVersion===1&&value.kind==="p1-manual-acceptance"&&HEX64.test(value.nonce??"")&&value.repositoryRoot===repo&&value.root===root&&value.status==="PENDING"&&["PREPARING","READY"].includes(value.stage)&&value.listener?.host===HOST&&value.listener?.port===PORT&&value.listener?.state==="stopped"&&typeof value.createdAt==="string"&&validEntrypoint(value.entrypoint,repo)&&validDistManifest(value.distManifest,root)&&JSON.stringify(value.resources)===JSON.stringify([root,{kind:"fastify",host:HOST,port:PORT}]);const readyLog=value.backendLog?.path===join(root,"logs/backend.log")&&Number.isSafeInteger(value.backendLog?.dev)&&Number.isSafeInteger(value.backendLog?.ino);if(!baseValid||(value.stage==="READY"?!readyLog:value.backendLog!==null))throw new Error("manual ownership identity mismatch");return value;}
async function run(executable,argv,options={}){return await exec(executable,argv,{...options,maxBuffer:2*1024*1024,encoding:"utf8"});}
function descriptor(id,source){return{workspace:{schema_version:3,id,name:`P1 ${id}`,language:"en"},dwh:{engine:"postgres",database:"postgres",schema:"public",supported_transports:["postgres_direct"]},semantic_index:{vector_store:{engine:"qdrant",collection:id,dimensions:1024,distance:"cosine"},embedding:{provider:"ollama_internal",model:"qwen3-embedding:0.6b",dimensions:1024}},llm_policy:{allowed:["zai/glm-5.2"]},evidence:{source,policy:{max_chunk_chars:4000,retain_published_generations:3}}};}
function descriptor(id,source){return{workspace:{schema_version:4,id,name:`P1 ${id}`,language:"en"},dwh:{engine:"postgres",database:"postgres",schema:"public",supported_transports:["postgres_direct"]},evidence:{source,policy:{max_chunk_chars:4000,retain_published_generations:3}}};}
function descriptors(){return[descriptor("p1-filesystem",{type:"filesystem",uri:"workspace-content/p1-filesystem/evidence",patterns:["**/*.md"],max_bytes:10485760}),descriptor("p1-http",{type:"http",uris:["https://evidence.example.test/guide.md"],authentication:"signed_urls_file",connect_timeout_ms:1250,read_timeout_ms:30001,max_bytes:12345,max_redirects:2,allow_private_hosts:false,max_cache_bytes:67890}),descriptor("p1-s3",{type:"s3",uri:"s3://p1-evidence/published/",endpoint_url:"https://s3.example.test/",region:"eu-west-1",credentials:"static_files",trusted_endpoint:true,allow_private_endpoint:false,allow_insecure_endpoint:false,max_bytes:12345,max_objects:33,max_pages:4,page_size:5})];}
function quote(value){return `'${String(value).replaceAll("'",`'"'"'`)}'`;}
async function checkPrerequisites(repo){for(const path of ["scripts/p1-acceptance.sh","scripts/test-p1-acceptance.sh","backend/scripts/p1-acceptance.mjs","backend/dist/server.js"]){try{await access(join(repo,path));}catch{throw new Error(`Task 8 prerequisite is missing: ${path}`);}}for(const command of ["node","npm","git","curl","unzip","zipinfo","lsof","python3"]){try{await run(command,[command==="unzip"||command==="lsof"?"-v":command==="zipinfo"?"-h":"--version"]);}catch{throw new Error(`missing prerequisite: ${command}`);}}const tht=join(repo,"harness",".venv","bin","tht");try{await access(tht,constants.X_OK);}catch{throw new Error("missing prerequisite: harness/.venv/bin/tht");}}
@@ -403,7 +403,7 @@ test("generated render command validates saved responses and owned snapshot befo
});
const renderSnapshotYaml=`workspace:
schema_version: 3
schema_version: 4
id: p1-filesystem
name: P1 filesystem
language: en
@@ -412,11 +412,6 @@ dwh:
database: postgres
schema: public
supported_transports: [postgres_direct]
semantic_index:
vector_store: {engine: qdrant, collection: p1-filesystem, dimensions: 1024, distance: cosine}
embedding: {provider: ollama_internal, model: qwen3-embedding:0.6b, dimensions: 1024}
llm_policy:
allowed: [zai/glm-5.2]
evidence:
source: {type: filesystem, uri: workspace-content/p1-filesystem/evidence, patterns: ["**/*.md"], max_bytes: 10485760}
policy: {max_chunk_chars: 4000, retain_published_generations: 3}
+2 -7
View File
@@ -16,7 +16,7 @@ async function fixture() {
await writeFile(join(root,"installation/base.yaml"),"{}\n");
const secret=join(root,"fixture-secrets/dwh-password"); await writeFile(secret,"not-inspected",{mode:0o600});
await writeFile(snapshot,`workspace:
schema_version: 3
schema_version: 4
id: p1-filesystem
name: P1 filesystem
language: en
@@ -25,11 +25,6 @@ dwh:
database: postgres
schema: public
supported_transports: [postgres_direct]
semantic_index:
vector_store: {engine: qdrant, collection: p1-filesystem, dimensions: 1024, distance: cosine}
embedding: {provider: ollama_internal, model: qwen3-embedding:0.6b, dimensions: 1024}
llm_policy:
allowed: [zai/glm-5.2]
evidence:
source: {type: filesystem, uri: workspace-content/p1-filesystem/evidence, patterns: ["**/*.md"], max_bytes: 10485760}
policy: {max_chunk_chars: 4000, retain_published_generations: 3}
@@ -61,7 +56,7 @@ test("renderer refuses snapshot manifest head, digest, and expected-digest tampe
test("renderer refuses a missing or malformed snapshot manifest",async()=>{ const f=await fixture(); const output=join(f.root,"rendered/nomanifest.yaml"); await rm(f.manifestPath); await assert.rejects(call(f,{outputPath:output}),/snapshot manifest.*(missing|unbounded|unsafe)/); await writeFile(f.manifestPath,"{not json"); await assert.rejects(call(f,{outputPath:output}),/snapshot manifest.*malformed/); await assert.rejects(lstat(output)); assert.deepEqual(await runtimeLeases(f),[]); });
test("renderer rejects a regular snapshot replacement against its manifest",async()=>{ const f=await fixture(); const output=join(f.root,"rendered/replaced.yaml"); await assert.rejects(call(f,{outputPath:output,beforePublish:async()=>{await writeFile(f.snapshot,"workspace:\n schema_version: 3\n id: p1-filesystem\n name: replaced\n")}}),/snapshot content changed/); await assert.rejects(lstat(output)); });
test("renderer rejects a regular snapshot replacement against its manifest",async()=>{ const f=await fixture(); const output=join(f.root,"rendered/replaced.yaml"); await assert.rejects(call(f,{outputPath:output,beforePublish:async()=>{await writeFile(f.snapshot,"workspace:\n schema_version: 4\n id: p1-filesystem\n name: replaced\n")}}),/snapshot content changed/); await assert.rejects(lstat(output)); });
test("renderer anchors publication when rendered parent is concurrently swapped", async()=>{
const f=await fixture(),output=join(f.root,"rendered/raced.yaml"),moved=join(f.root,"rendered-moved"),outside=join(f.repo,"outside-rendered"); await mkdir(outside);
+1 -6
View File
@@ -328,13 +328,8 @@ async function tht(ctx, argv, options = {}) {
function namespace(id) { return id.toUpperCase().replaceAll("-", "_"); }
function baseWorkspace(id, evidenceSource) {
return {
workspace: { schema_version: 3, id, name: `P1.1 ${id}`, description: `Catalog entry for ${id}`, language: "en" },
workspace: { schema_version: 4, id, name: `P1.1 ${id}`, description: `Catalog entry for ${id}`, language: "en" },
dwh: { engine: "postgres", database: "postgres", schema: "public", supported_transports: ["postgres_direct"] },
semantic_index: {
vector_store: { engine: "qdrant", collection: id, dimensions: 1024, distance: "cosine" },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
llm_policy: { allowed: ["zai/glm-5.2"] },
evidence: { source: evidenceSource, policy: { max_chunk_chars: 4000, retain_published_generations: 3 } },
};
}
+1 -6
View File
@@ -81,13 +81,8 @@ async function git(executable, argv, options = {}) {
function namespace(id) { return id.toUpperCase().replaceAll("-", "_"); }
function baseWorkspace(id, evidenceSource) {
return {
workspace: { schema_version: 3, id, name: `P1.1 ${id}`, description: `Catalog entry for ${id}`, language: "en" },
workspace: { schema_version: 4, id, name: `P1.1 ${id}`, description: `Catalog entry for ${id}`, language: "en" },
dwh: { engine: "postgres", database: "postgres", schema: "public", supported_transports: ["postgres_direct"] },
semantic_index: {
vector_store: { engine: "qdrant", collection: id, dimensions: 1024, distance: "cosine" },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
llm_policy: { allowed: ["zai/glm-5.2"] },
evidence: { source: evidenceSource, policy: { max_chunk_chars: 4000, retain_published_generations: 3 } },
};
}
+1 -6
View File
@@ -375,16 +375,11 @@ function installationProjectName(installationPath) {
function baseWorkspace(id, { dwhBaseUrl, evidenceSource }) {
return {
workspace: { schema_version: 3, id, name: `P2 ${id}`, language: "en" },
workspace: { schema_version: 4, id, name: `P2 ${id}`, language: "en" },
dwh: { engine: "postgres", database: "warehouse", schema: "dw", supported_transports: ["rest_api"] },
semantic_index: {
vector_store: { engine: "qdrant", collection: id, dimensions: 1024, distance: "cosine" },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
diagnostics: {
dwh_rest: { method: "POST", path: "/rpc/ping", auth: "x-api-key", response: { database: "database", schema: "schema" } },
},
llm_policy: { allowed: ["zai/glm-5.2"] },
...(evidenceSource ? { evidence: { source: evidenceSource, policy: { max_chunk_chars: 4000, retain_published_generations: 3 } } } : {}),
};
}
+1 -6
View File
@@ -376,16 +376,11 @@ function installationProjectName(installationPath) {
function baseWorkspace(id, { dwhBaseUrl, evidenceSource }) {
return {
workspace: { schema_version: 3, id, name: `P2 ${id}`, language: "en" },
workspace: { schema_version: 4, id, name: `P2 ${id}`, language: "en" },
dwh: { engine: "postgres", database: "warehouse", schema: "dw", supported_transports: ["rest_api"] },
semantic_index: {
vector_store: { engine: "qdrant", collection: id, dimensions: 1024, distance: "cosine" },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
diagnostics: {
dwh_rest: { method: "POST", path: "/rpc/ping", auth: "x-api-key", response: { database: "database", schema: "schema" } },
},
llm_policy: { allowed: ["zai/glm-5.2"] },
...(evidenceSource ? { evidence: { source: evidenceSource, policy: { max_chunk_chars: 4000, retain_published_generations: 3 } } } : {}),
};
}
+1 -6
View File
@@ -379,16 +379,11 @@ function installationProjectName(installationPath) {
function baseWorkspace(id, { dwhBaseUrl, evidenceSource }) {
return {
workspace: { schema_version: 3, id, name: `P2 ${id}`, language: "en" },
workspace: { schema_version: 4, id, name: `P2 ${id}`, language: "en" },
dwh: { engine: "postgres", database: "warehouse", schema: "dw", supported_transports: ["rest_api"] },
semantic_index: {
vector_store: { engine: "qdrant", collection: id, dimensions: 1024, distance: "cosine" },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
diagnostics: {
dwh_rest: { method: "POST", path: "/rpc/ping", auth: "x-api-key", response: { database: "database", schema: "schema" } },
},
llm_policy: { allowed: ["zai/glm-5.2"] },
...(evidenceSource ? { evidence: { source: evidenceSource, policy: { max_chunk_chars: 4000, retain_published_generations: 3 } } } : {}),
};
}
+1 -6
View File
@@ -374,16 +374,11 @@ function installationProjectName(installationPath) {
function baseWorkspace(id, { dwhBaseUrl, evidenceSource }) {
return {
workspace: { schema_version: 3, id, name: `P2 ${id}`, language: "en" },
workspace: { schema_version: 4, id, name: `P2 ${id}`, language: "en" },
dwh: { engine: "postgres", database: "warehouse", schema: "dw", supported_transports: ["rest_api"] },
semantic_index: {
vector_store: { engine: "qdrant", collection: id, dimensions: 1024, distance: "cosine" },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
diagnostics: {
dwh_rest: { method: "POST", path: "/rpc/ping", auth: "x-api-key", response: { database: "database", schema: "schema" } },
},
llm_policy: { allowed: ["zai/glm-5.2"] },
...(evidenceSource ? { evidence: { source: evidenceSource, policy: { max_chunk_chars: 4000, retain_published_generations: 3 } } } : {}),
};
}
+1 -6
View File
@@ -374,16 +374,11 @@ function installationProjectName(installationPath) {
function baseWorkspace(id, { dwhBaseUrl, evidenceSource }) {
return {
workspace: { schema_version: 3, id, name: `P2 ${id}`, language: "en" },
workspace: { schema_version: 4, id, name: `P2 ${id}`, language: "en" },
dwh: { engine: "postgres", database: "warehouse", schema: "dw", supported_transports: ["rest_api"] },
semantic_index: {
vector_store: { engine: "qdrant", collection: id, dimensions: 1024, distance: "cosine" },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
diagnostics: {
dwh_rest: { method: "POST", path: "/rpc/ping", auth: "x-api-key", response: { database: "database", schema: "schema" } },
},
llm_policy: { allowed: ["zai/glm-5.2"] },
...(evidenceSource ? { evidence: { source: evidenceSource, policy: { max_chunk_chars: 4000, retain_published_generations: 3 } } } : {}),
};
}
@@ -36,11 +36,10 @@ const reviewedExpandableBlocks = new Map([
{ sha256: "b903e5dae953ae1372f1a5276f12a92ed3dd632b897f3afe5e00c646d90a1b42", rationale: "Same reviewed block in the repository-required CRLF checkout representation." },
]],
["scripts/unified-deployment-smoke.sh", [
{ sha256: "1d60bf140165a8fabfa0c3729e776136904717e67becf3e0ab68c70d8e37847e", rationale: "Generates reviewed Task 13 runtime configuration." },
{ sha256: "36d3d8a2362dbdc4fad90948d6c227586d749f56b9a4bc5b6b5a91bcbec6407b", rationale: "Generates the reviewed local Task 13 Compose override." },
{ sha256: "c556f7d910d0788e219b042957e6b307cb9925b43920c680535d0d3a6dcbdb25", rationale: "Generates the reviewed local Task 13 installation descriptor." },
{ sha256: "b6c0826151b2c8b955399d1abf5b691cc8fe6b6454b17da000dde7ba3bc55d2d", rationale: "Generates the reviewed local Task 13 Compose override with normalized catalog mounts." },
{ sha256: "24f69d12b8554aa2bebba455be99fde3e60743eef5a40fa2ef5b29397a477c03", rationale: "Generates the reviewed local Task 13 installation descriptor with its model catalog." },
{ sha256: "526006fa6d48a8080b3834723630c64de5005a67243e944ebf1da15212b4d654", rationale: "Generates the reviewed server Task 13 Compose override." },
{ sha256: "c57ae2205c21ead0c2015a353aaabb948fa4ddd9b78a2cdcdb71f48cf2db742d", rationale: "Generates the reviewed projected-auth server Task 13 installation descriptor." },
{ sha256: "406ccead1967f642225c946fc4a23fe5b019c9764cc5153e1125876ade16ec90", rationale: "Generates the reviewed projected-auth server Task 13 installation descriptor with its model catalog." },
]],
["scripts/vector-backup.sh", [
{ sha256: "571899db49dfdcec8107fbe1e0a86a61e7581979d3c4c248c20546843e275bcf", rationale: "Generates the reviewed backup manifest inside the helper command." },
@@ -103,6 +102,7 @@ function isPolicyImplementationException(label, category) {
]);
if (implementations.has(label)) return true;
if (category === "migration-marker" && new Set([
"backend/src/workspaces/schema.ts",
"scripts/workspace_descriptor_doc_contract.py",
"scripts/test_workspace_descriptor_doc_contract.py",
"backend/scripts/clean-dist.test.mjs",
@@ -173,7 +173,7 @@ function validateWorkspaceSource(source, label, { requireWorkspace, expandable =
try {
parseWorkspaceYaml(source);
} catch (error) {
throw new Error(`${label}: workspace descriptor is not valid schema v3: ${error instanceof Error ? error.message : String(error)}`);
throw new Error(`${label}: workspace descriptor is not valid schema v4: ${error instanceof Error ? error.message : String(error)}`);
}
return true;
}
@@ -33,20 +33,20 @@ function bashN(root, path) {
function replaceWorkspaceKeys(source, workspaceKey, schemaLine) {
return source
.replace(/^workspace:$/m, workspaceKey)
.replace(/^ schema_version: 3$/m, schemaLine);
.replace(/^ schema_version: 4$/m, schemaLine);
}
test("production parser accepts semantic v3 with quoted Unicode/tagged keys and spacing", async (t) => {
test("production parser accepts semantic v4 with quoted Unicode/tagged keys and spacing", async (t) => {
const root = await fixture(t);
const unicode = replaceWorkspaceKeys(
canonicalDescriptor,
'"\\u0077orkspace" :',
' "\\u0073chema_version" : 3',
' "\\u0073chema_version" : 4',
);
const tagged = replaceWorkspaceKeys(
canonicalDescriptor,
"!!str workspace :",
" !!str schema_version : 3",
" !!str schema_version : 4",
);
await put(root, "deploy/workspaces/unicode.yaml", unicode);
await put(root, "deploy/workspaces/tagged.yaml", tagged);
@@ -59,14 +59,15 @@ test("production parser accepts semantic v3 with quoted Unicode/tagged keys and
});
});
test("production parser rejects fancy keys with every non-v3 or ambiguous value", async (t) => {
test("production parser rejects fancy keys with every non-v4 or ambiguous value", async (t) => {
const invalid = [
["unicode-v2", '"\\u0077orkspace" :', ' "\\u0073chema_version" : 2'],
["tagged-leading-zero", "!!str workspace :", " !!str schema_version : 02"],
["hexadecimal", "workspace :", " schema_version : 0x2"],
["multiline", "workspace :", " schema_version : >\n 3"],
["duplicate", "workspace :", " schema_version : 3\n schema_version: 3"],
["inline", "workspace: { schema_version: 3 }", " schema_version: 3"],
["unicode-v3", '"\\u0077orkspace" :', ' "\\u0073chema_version" : 3'],
["tagged-leading-zero", "!!str workspace :", " !!str schema_version : 03"],
["hexadecimal", "workspace :", " schema_version : 0x3"],
["multiline", "workspace :", " schema_version : >\n 4"],
["duplicate", "workspace :", " schema_version : 4\n schema_version: 4"],
["inline", "workspace: { schema_version: 4 }", " schema_version: 4"],
];
for (const [name, workspaceKey, schemaLine] of invalid) {
await t.test(name, async () => {
@@ -120,7 +121,7 @@ test("PowerShell embedded workspace mappings are rejected while bundle-only stri
const root = await fixture(t);
const source = [
"$workspace = @'",
canonicalDescriptor.replace(" schema_version: 3", " schema_version: 0x2").trimEnd(),
canonicalDescriptor.replace(" schema_version: 4", " schema_version: 0x2").trimEnd(),
"'@",
'$bundle = @"',
"bundle:",
@@ -137,7 +138,7 @@ test("PowerShell embedded workspace mappings are rejected while bundle-only stri
test("workspace descriptor family entries require a top-level workspace", async (t) => {
const root = await fixture(t);
await put(root, "scripts/fixtures/workspace-registry-future.yaml", "bundle:\n schema_version: 3\n");
await put(root, "scripts/fixtures/workspace-registry-future.yaml", "bundle:\n schema_version: 4\n");
await assert.rejects(
verifyEntries({
root,
@@ -179,7 +180,7 @@ test("script scalar workspace remains a bundle even with descriptor-like sibling
test("standalone descriptor files require workspace to be a mapping", async (t) => {
const root = await fixture(t);
const path = "scripts/fixtures/workspace-registry-scalar.yaml";
await put(root, path, "workspace: analytics\nschema_version: 3\n");
await put(root, path, "workspace: analytics\nschema_version: 4\n");
await assert.rejects(
verifyEntries({ root, entries: [entry("workspace_descriptor", path)] }),
/workspace.*mapping/i,
@@ -193,11 +194,11 @@ test("Bash extractor supports hyphen, digit, escaped delimiters, and tab strippi
name: "hyphen-v2",
opener: "cat <<'WORKSPACE-YAML'",
delimiter: "WORKSPACE-YAML",
descriptor: canonicalDescriptor.replace(" schema_version: 3", " schema_version: 2"),
descriptor: canonicalDescriptor.replace(" schema_version: 4", " schema_version: 2"),
rejected: true,
},
{
name: "digit-v3",
name: "digit-v4",
opener: "cat <<2YAML",
delimiter: "2YAML",
descriptor: canonicalDescriptor,
@@ -207,11 +208,11 @@ test("Bash extractor supports hyphen, digit, escaped delimiters, and tab strippi
name: "escaped-v2",
opener: "cat <<WORKSPACE\\-YAML",
delimiter: "WORKSPACE-YAML",
descriptor: canonicalDescriptor.replace(" schema_version: 3", " schema_version: 2"),
descriptor: canonicalDescriptor.replace(" schema_version: 4", " schema_version: 2"),
rejected: true,
},
{
name: "tab-strip-v3",
name: "tab-strip-v4",
opener: "cat <<-'TAB-YAML'",
delimiter: "\tTAB-YAML",
descriptor: canonicalDescriptor.split("\n").map((line) => `\t${line}`).join("\n"),
@@ -272,7 +273,7 @@ test("non-stripping heredoc close requires an exact physical delimiter line", as
"#!/usr/bin/env bash",
"cat <<'---'",
"--- ",
canonicalDescriptor.replace(" schema_version: 3", " schema_version: 2").trimEnd(),
canonicalDescriptor.replace(" schema_version: 4", " schema_version: 2").trimEnd(),
"---",
"",
].join("\n");
@@ -313,7 +314,7 @@ test("double-quoted non-special backslash is preserved in the delimiter", async
"#!/usr/bin/env bash",
'cat <<"\\---"',
"---",
canonicalDescriptor.replace(" schema_version: 3", " schema_version: 2").trimEnd(),
canonicalDescriptor.replace(" schema_version: 4", " schema_version: 2").trimEnd(),
"\\---",
"",
].join("\n");
@@ -355,7 +356,7 @@ test("split heredoc operator continuation cannot bypass v2 validation", async (t
"#!/usr/bin/env bash",
"cat <\\",
"<'YAML'",
canonicalDescriptor.replace(" schema_version: 3", " schema_version: 2").trimEnd(),
canonicalDescriptor.replace(" schema_version: 4", " schema_version: 2").trimEnd(),
"YAML",
"",
].join("\n");
@@ -424,7 +425,7 @@ test("PowerShell comment backslash cannot hide a following v2 here-string", asyn
const source = [
"# harmless PowerShell comment \\",
"$workspace = @'",
canonicalDescriptor.replace(" schema_version: 3", " schema_version: 2").trimEnd(),
canonicalDescriptor.replace(" schema_version: 4", " schema_version: 2").trimEnd(),
"'@",
"",
].join("\n");
@@ -435,7 +436,7 @@ test("PowerShell comment backslash cannot hide a following v2 here-string", asyn
);
});
test("PowerShell dialect accepts normal v3 and non-workspace bundle here-strings", async (t) => {
test("PowerShell dialect accepts normal v4 and non-workspace bundle here-strings", async (t) => {
const root = await fixture(t);
const path = "scripts/powershell-valid-smoke.ps1";
const source = [
@@ -494,10 +495,10 @@ test("PowerShell cast and concatenation openers cannot hide embedded descriptors
test("expandable YAML interpolation that can hide a workspace descriptor fails closed", async (t) => {
const root = await fixture(t);
const cases = [
["braced-key", "${key}:\n schema_version: 3"],
["plain-key", "$key:\n schema_version: 3"],
["quoted-key", '"$key" :\n schema_version: 3'],
["subexpression-key", "$($key):\n schema_version: 3"],
["braced-key", "${key}:\n schema_version: 4"],
["plain-key", "$key:\n schema_version: 4"],
["quoted-key", '"$key" :\n schema_version: 4'],
["subexpression-key", "$($key):\n schema_version: 4"],
["version", "workspace:\n schema_version: $version"],
];
for (const [name, body] of cases) {
@@ -564,7 +565,7 @@ test("unmarked expandable Bash YAML cannot generate descriptor keys or values at
"key=workspace",
"cat <<YAML",
generatedKey,
" schema_version: 3",
" schema_version: 4",
"YAML",
"",
].join("\n");
@@ -600,14 +601,14 @@ test("an in-band marker cannot authorize expandable content", async (t) => {
for (const [path, source] of [
["scripts/fake-marker.sh", [
"#!/usr/bin/env bash",
"# schema-v3-only: expandable-nonworkspace",
"# schema-v4-only: expandable-nonworkspace",
"cat <<YAML",
"${DESCRIPTOR}",
"YAML",
"",
].join("\n")],
["scripts/fake-marker.ps1", [
"# schema-v3-only: expandable-nonworkspace",
"# schema-v4-only: expandable-nonworkspace",
'$yaml = @"',
"$descriptor",
'"@',
+19 -7
View File
@@ -22,7 +22,8 @@ import { isUsableAuthenticationSecret } from "./auth/secret-policy.js";
import { secretValue } from "./config/secret-bundle.js";
import { sessionRoutes } from "./routes/sessions.js";
import { sqlRoutes } from "./routes/sql.js";
import { metaRoutes, type ListModelsFn } from "./routes/meta.js";
import { metaRoutes } from "./routes/meta.js";
import type { ListModelsFn } from "./pi/list-models.js";
import { settingsRoutes, effectiveSettings } from "./routes/settings.js";
import { createPiModelLister } from "./pi/list-models.js";
import { createPiManagement, type PiManagementService } from "./pi/management.js";
@@ -66,6 +67,7 @@ import { catalogDescriptionGenerationRoutes } from "./routes/catalog-description
import { CatalogLogicalRelationshipService } from "./catalog/logical-relationship-service.js";
import { catalogLogicalRelationshipRoutes } from "./routes/catalog-logical-relationships.js";
import { EffectiveRelationshipSnapshotProvider } from "./catalog/effective-relationship-snapshot.js";
import { loadRuntimeModelCatalog, type RuntimeModelCatalog } from "./models/runtime-model-catalog.js";
export interface BuildAppDeps {
thtRunner?: ThtRunner;
@@ -88,6 +90,7 @@ export interface BuildAppDeps {
catalogSyncWorker?: CatalogSyncWorker;
catalogOperationCoordinator?: CatalogOperationCoordinator;
metadataGenerationModels?: MetadataGenerationModels;
runtimeModelCatalog?: RuntimeModelCatalog;
modelCompleter?: ModelCompleter;
descriptionSourceSampler?: DescriptionSourceSampler;
workspaceRuntimeSupport?: (workspace: WorkspaceDescriptor) => boolean;
@@ -155,17 +158,22 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
semanticRuntime: {
internalQdrantUrl: config.internalQdrantUrl,
internalEmbeddingUrl: config.internalEmbeddingUrl,
internalEmbeddingId: config.internalEmbeddingId,
internalEmbeddingModel: config.internalEmbeddingModel,
internalEmbeddingDimensions: config.internalEmbeddingDimensions,
},
});
const mgr = deps?.mgr ?? new PiProcessManager(config, deps?.spawnFn ? { spawnFn: deps.spawnFn } : undefined);
const hub = deps?.hub ?? new SseHub();
const workspaceRegistry = deps?.workspaceRegistry ?? new WorkspaceRegistry(config.workspaceRegistry);
const catalogRepository = deps?.catalogRepository ?? createCatalogRepository(config.catalogDatabase);
const catalogOperationCoordinator = deps?.catalogOperationCoordinator ?? new CatalogOperationCoordinator();
const runtimeModelCatalog = deps?.runtimeModelCatalog ?? loadRuntimeModelCatalog(config.modelCatalogFile);
const mgr = deps?.mgr ?? new PiProcessManager(config, {
...(deps?.spawnFn ? { spawnFn: deps.spawnFn } : {}),
modelCatalog: runtimeModelCatalog,
});
const metadataGenerationModels = deps?.metadataGenerationModels ?? loadMetadataGenerationModels({
installationFile: config.installationConfigFile,
catalogFile: config.modelCatalogFile,
secretsFile: config.secretsFile,
});
const modelCompleter = deps?.modelCompleter ?? new PythonModelCompleter({
@@ -237,6 +245,7 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
?? createProductionWorkspaceDiagnoser(config.workspaceDiagnosticTimeoutMs, undefined, {
internalQdrantUrl: config.internalQdrantUrl,
internalEmbeddingUrl: config.internalEmbeddingUrl,
internalEmbeddingId: config.internalEmbeddingId,
internalEmbeddingModel: config.internalEmbeddingModel,
internalEmbeddingDimensions: config.internalEmbeddingDimensions,
});
@@ -271,7 +280,7 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
const getSettings = async (principal: PrincipalContext): Promise<Settings> => {
if (deps?.getSettings) return await deps.getSettings(principal);
const stored = loadSettings(config);
const effective = effectiveSettings(config, stored);
const effective = effectiveSettings(config, stored, runtimeModelCatalog);
// In the registry system the legacy `harness/workspaces/*.yaml` default is obsolete: when no
// installation workspace is pinned, default to the first active registry workspace.
if (!stored.workspace) {
@@ -284,7 +293,9 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
}
return effective;
};
const piManagement = deps?.piManagement ?? createPiManagement(config, { listModels });
const piManagement = deps?.piManagement ?? createPiManagement(config, {
modelCatalog: runtimeModelCatalog,
});
const maintenanceBarrier = deps?.maintenanceBarrier ?? new MaintenanceBarrier(config.maintenanceFile);
const localRegistryResolver = deps?.localUserRegistry === undefined
@@ -408,6 +419,7 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
dwhPrecheck: config.dwhPrecheck,
legacyWorkspaceMode: config.legacyWorkspaceMode,
workspaceRuntimeSupport,
modelCatalog: runtimeModelCatalog,
maintenanceBarrier,
effectiveRelationships,
});
@@ -439,7 +451,7 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
return maintenanceBarrier.status();
});
sqlRoutes(app, { tht: tht as ThtRunner, getSettings, workspaceRegistry });
metaRoutes(app, { harnessDir: config.harnessDir, listModels });
metaRoutes(app, { harnessDir: config.harnessDir, modelCatalog: runtimeModelCatalog });
workspaceRoutes(app, {
registry: workspaceRegistry,
config: config.workspaceRegistry,
@@ -472,7 +484,7 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
worker: descriptionGenerationWorker,
sensitiveDataSuggestionRunner,
});
settingsRoutes(app, { cfg: config, listModels, getSettings });
settingsRoutes(app, { cfg: config, getSettings });
piManagementRoutes(app, { service: piManagement });
return app;
+41 -162
View File
@@ -1,63 +1,5 @@
import {
closeSync, constants, fstatSync, lstatSync, openSync, readFileSync,
type Stats,
} from "node:fs";
import { parseAllDocuments } from "yaml";
import { z } from "zod";
import {
loadSecretBundle,
METADATA_GENERATION_SECRET_KEYS,
} from "../config/secret-bundle.js";
const MAX_INSTALLATION_BYTES = 1024 * 1024;
const RUNTIME_INSTALLATION_FILE = "/run/thothii-installation/thothii-installation.yaml";
const modelId = z.string().regex(/^[a-z][a-z0-9._-]{0,63}$/);
const apiKeyEnvironment = z.enum(METADATA_GENERATION_SECRET_KEYS);
const endpointSchema = z.object({
baseUrl: z.string().min(1).max(2048).refine((value) => {
try {
const url = new URL(value);
return (url.protocol === "http:" || url.protocol === "https:")
&& url.username === "" && url.password === "" && url.search === "" && url.hash === "";
} catch {
return false;
}
}),
apiVersion: z.string().regex(/^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$/).optional(),
}).strict();
const configuredModelSchema = z.object({
id: modelId,
label: z.string().min(1).max(128).refine((value) => value.trim() === value && !/\p{Cc}/u.test(value)),
litellm: z.object({
provider: z.string().regex(/^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$/),
model: z.string().regex(/^[A-Za-z0-9][A-Za-z0-9._:/-]{0,255}$/),
disableThinking: z.literal(true).optional(),
endpoint: endpointSchema.optional(),
}).strict(),
apiKeyEnv: apiKeyEnvironment.optional(),
}).strict().superRefine((value, context) => {
if (value.apiKeyEnv === undefined && value.litellm.endpoint === undefined) {
context.addIssue({
code: z.ZodIssueCode.custom,
path: ["apiKeyEnv"],
message: "keyless models require an explicit endpoint",
});
}
if (value.litellm.disableThinking === true && value.litellm.endpoint === undefined) {
context.addIssue({
code: z.ZodIssueCode.custom,
path: ["litellm", "disableThinking"],
message: "thinking may be disabled only for an explicit endpoint",
});
}
});
const metadataGenerationSchema = z.object({
default: modelId.optional(),
models: z.array(configuredModelSchema).max(64).default([]),
}).strict();
const installationSchema = z.object({
metadataGeneration: metadataGenerationSchema.optional(),
}).passthrough();
import { loadSecretBundle } from "../config/secret-bundle.js";
import { loadRuntimeModelCatalog } from "../models/runtime-model-catalog.js";
export interface MetadataGenerationModelChoice {
id: string;
@@ -86,7 +28,6 @@ export class MetadataGenerationModelUnavailableError extends Error {
}
}
/** The complete interface callers need: safe discovery plus fail-closed runtime resolution. */
export interface MetadataGenerationModels {
catalog(): MetadataGenerationModelCatalog;
resolve(selection: string): ResolvedMetadataGenerationModel;
@@ -96,140 +37,78 @@ class RestartLoadedMetadataGenerationModels implements MetadataGenerationModels
readonly #models: ReadonlyMap<string, ResolvedMetadataGenerationModel>;
readonly #catalog: MetadataGenerationModelCatalog;
constructor(
models: ReadonlyMap<string, ResolvedMetadataGenerationModel> = new Map(),
defaultModel: string | null = null,
choices: MetadataGenerationModelChoice[] = [],
) {
constructor(models: ReadonlyMap<string, ResolvedMetadataGenerationModel>, defaultModel: string | null) {
this.#models = models;
this.#catalog = {
models: choices.map((choice) => ({ ...choice })),
models: [...models.values()].map(({ id }) => ({ id, label: id })),
default: defaultModel,
};
}
catalog(): MetadataGenerationModelCatalog {
return {
models: this.#catalog.models.map((choice) => ({ ...choice })),
default: this.#catalog.default,
};
return { models: this.#catalog.models.map((choice) => ({ ...choice })), default: this.#catalog.default };
}
resolve(selection: string): ResolvedMetadataGenerationModel {
const model = typeof selection === "string" ? this.#models.get(selection) : undefined;
const model = this.#models.get(selection);
if (!model) throw new MetadataGenerationModelUnavailableError();
return model;
}
}
function invalid(message = "metadata-generation configuration is invalid"): Error {
function invalid(message = "metadata-generation runtime catalog is invalid"): Error {
return new Error(message);
}
function protectedInstallationStat(file: string, info: Stats): boolean {
const mode = info.mode & 0o777;
if (!info.isFile() || info.isSymbolicLink() || info.nlink !== 1
|| info.size < 1 || info.size > MAX_INSTALLATION_BYTES) return false;
if (file === RUNTIME_INSTALLATION_FILE && info.uid === 0 && mode === 0o444) return true;
return info.uid === (process.getuid?.() ?? info.uid) && (mode === 0o400 || mode === 0o600);
}
function readProtectedInstallation(file: string): string {
let descriptor: number | undefined;
try {
const before = lstatSync(file);
if (!protectedInstallationStat(file, before)) throw new Error("unavailable");
descriptor = openSync(file, constants.O_RDONLY | constants.O_NOFOLLOW);
const opened = fstatSync(descriptor);
if (!protectedInstallationStat(file, opened)
|| before.dev !== opened.dev || before.ino !== opened.ino) throw new Error("unavailable");
const source = readFileSync(descriptor, "utf8");
const after = fstatSync(descriptor);
const current = lstatSync(file);
if (!protectedInstallationStat(file, after) || !protectedInstallationStat(file, current)
|| opened.dev !== after.dev || opened.ino !== after.ino
|| opened.dev !== current.dev || opened.ino !== current.ino) throw new Error("unavailable");
return source;
} finally {
if (descriptor !== undefined) try { closeSync(descriptor); } catch { /* sanitized below */ }
}
}
function readInstallation(file: string): unknown {
try {
const documents = parseAllDocuments(readProtectedInstallation(file), { uniqueKeys: true });
if (documents.length !== 1) throw invalid("metadata-generation installation must contain one YAML document");
const document = documents[0];
if (document.errors.length > 0 || document.warnings.length > 0) {
throw invalid("metadata-generation installation contains invalid YAML");
}
return document.toJSON();
} catch (error) {
if (error instanceof Error && error.message.startsWith("metadata-generation")) throw error;
throw invalid("metadata-generation installation is unavailable");
}
}
export function loadMetadataGenerationModels(options: {
installationFile?: string;
catalogFile?: string;
secretsFile?: string;
}): MetadataGenerationModels {
if (!options.installationFile) return new RestartLoadedMetadataGenerationModels();
const installation = installationSchema.safeParse(readInstallation(options.installationFile));
if (!installation.success) throw invalid();
const configured = installation.data.metadataGeneration;
if (!configured || configured.models.length === 0) {
if (configured?.default !== undefined) throw invalid("metadata-generation default does not identify a configured model");
return new RestartLoadedMetadataGenerationModels();
}
if (!configured.default) throw invalid("metadata-generation default is required when models are configured");
const catalog = loadRuntimeModelCatalog(options.catalogFile);
const configured = catalog.metadataModels();
if (configured.length === 0) return new RestartLoadedMetadataGenerationModels(new Map(), null);
const seen = new Set<string>();
for (const model of configured.models) {
if (seen.has(model.id)) throw invalid(`metadata-generation model id "${model.id}" is duplicated`);
seen.add(model.id);
}
if (!seen.has(configured.default)) {
throw invalid(`metadata-generation default "${configured.default}" is not configured`);
}
const requiresSecrets = configured.models.some((model) => model.apiKeyEnv !== undefined);
const requiresSecrets = configured.some((model) => model.authentication.mode === "secret_env");
let secrets: ReadonlyMap<string, string> = new Map();
if (requiresSecrets) {
if (!options.secretsFile) throw invalid("metadata-generation keyed models require THT_SECRETS_FILE");
try {
secrets = loadSecretBundle(options.secretsFile);
} catch {
throw invalid("metadata-generation secrets are unavailable");
}
try { secrets = loadSecretBundle(options.secretsFile); }
catch { throw invalid("metadata-generation secrets are unavailable"); }
}
const models = new Map<string, ResolvedMetadataGenerationModel>();
for (const configuredModel of configured.models) {
const labels = new Map<string, string>();
for (const configuredModel of configured) {
const adapter = configuredModel.metadataAdapter;
if (!adapter || configuredModel.authentication.mode === "pi_auth") throw invalid();
const apiKeyEnv = configuredModel.authentication.apiKeyEnv;
let apiKey: string | undefined;
if (configuredModel.apiKeyEnv !== undefined) {
apiKey = secrets.get(configuredModel.apiKeyEnv);
if (!apiKey) {
throw invalid(`metadata-generation model "${configuredModel.id}" secret "${configuredModel.apiKeyEnv}" is missing`);
}
if (configuredModel.authentication.mode === "secret_env") {
if (!apiKeyEnv) throw invalid();
apiKey = secrets.get(apiKeyEnv);
if (!apiKey) throw invalid(`metadata-generation model "${configuredModel.id}" secret "${apiKeyEnv}" is missing`);
if (apiKey.length > 16 * 1024 || /\s/u.test(apiKey)) {
throw invalid(`metadata-generation model "${configuredModel.id}" secret "${configuredModel.apiKeyEnv}" is unusable`);
throw invalid(`metadata-generation model "${configuredModel.id}" secret "${apiKeyEnv}" is unusable`);
}
}
labels.set(configuredModel.id, configuredModel.label);
models.set(configuredModel.id, Object.freeze({
id: configuredModel.id,
provider: configuredModel.litellm.provider,
model: configuredModel.litellm.model,
...(configuredModel.litellm.disableThinking === true ? { disableThinking: true as const } : {}),
...(configuredModel.litellm.endpoint === undefined
? {}
: { endpoint: Object.freeze({ ...configuredModel.litellm.endpoint }) }),
...(configuredModel.apiKeyEnv === undefined
? {}
: { apiKeyEnv: configuredModel.apiKeyEnv, apiKey }),
provider: adapter.litellmProvider,
model: configuredModel.upstreamModel,
...(configuredModel.metadataGeneration?.disableThinking === true
? { disableThinking: true as const } : {}),
...(configuredModel.endpoint ? { endpoint: Object.freeze({ ...configuredModel.endpoint }) } : {}),
...(apiKeyEnv ? { apiKeyEnv, apiKey } : {}),
}));
}
return new RestartLoadedMetadataGenerationModels(
models,
configured.default,
configured.models.map(({ id, label }) => ({ id, label })),
);
const result = new RestartLoadedMetadataGenerationModels(models, catalog.defaultMetadataGeneration);
const safe = result.catalog();
return {
catalog: () => ({
default: safe.default,
models: safe.models.map((choice) => ({ ...choice, label: labels.get(choice.id) ?? choice.id })),
}),
resolve: (selection) => result.resolve(selection),
};
}
+2
View File
@@ -12,6 +12,7 @@ import * as sensitiveDataFlagMigration from "./migrations/006_sensitive_data_fla
import * as sensitiveDataSuggestionRunsMigration from "./migrations/007_sensitive_data_suggestion_runs.js";
import * as catalogLogicalRelationshipsMigration from "./migrations/008_catalog_logical_relationships.js";
import * as aiTokenUsageMigration from "./migrations/009_ai_token_usage.js";
import * as canonicalModelIdsMigration from "./migrations/010_canonical_model_ids.js";
const connectionString = process.env.THT_CATALOG_MIGRATOR_DATABASE_URL;
const host = process.env.THT_CATALOG_DB_HOST;
@@ -46,6 +47,7 @@ const provider: MigrationProvider = {
"007_sensitive_data_suggestion_runs": sensitiveDataSuggestionRunsMigration,
"008_catalog_logical_relationships": catalogLogicalRelationshipsMigration,
"009_ai_token_usage": aiTokenUsageMigration,
"010_canonical_model_ids": canonicalModelIdsMigration,
};
},
};
@@ -0,0 +1,27 @@
import { sql, type Kysely } from "kysely";
import type { CatalogDatabase } from "../repository.js";
const canonicalModelPattern = "^[a-z][a-z0-9._-]{0,63}/[A-Za-z0-9][A-Za-z0-9._:-]{0,255}$";
const legacyModelPattern = "^[a-z][a-z0-9._-]{0,63}$";
export async function up(db: Kysely<CatalogDatabase>): Promise<void> {
await sql.raw(`alter table description_generation_runs
drop constraint description_generation_runs_model_id_check,
add constraint description_generation_runs_model_id_check
check (model_id ~ '${canonicalModelPattern}')`).execute(db);
await sql.raw(`alter table sensitive_data_suggestion_runs
drop constraint sensitive_data_suggestion_runs_model_id_check,
add constraint sensitive_data_suggestion_runs_model_id_check
check (model_id ~ '${canonicalModelPattern}')`).execute(db);
}
export async function down(db: Kysely<CatalogDatabase>): Promise<void> {
await sql.raw(`alter table sensitive_data_suggestion_runs
drop constraint sensitive_data_suggestion_runs_model_id_check,
add constraint sensitive_data_suggestion_runs_model_id_check
check (model_id ~ '${legacyModelPattern}')`).execute(db);
await sql.raw(`alter table description_generation_runs
drop constraint description_generation_runs_model_id_check,
add constraint description_generation_runs_model_id_check
check (model_id ~ '${legacyModelPattern}')`).execute(db);
}
+32 -2
View File
@@ -32,6 +32,7 @@ export interface AppConfig {
piManagementTimeoutMs: number;
secretsFile?: string;
installationConfigFile?: string;
modelCatalogFile?: string;
piAuthFile?: string;
secretFiles: Readonly<Record<string, string | undefined>>;
modelApiKeyFile?: string;
@@ -50,6 +51,7 @@ export interface AppConfig {
workspaceSecretRuntimeRoot: string;
internalQdrantUrl: string;
internalEmbeddingUrl: string;
internalEmbeddingId: string;
internalEmbeddingModel: string;
internalEmbeddingDimensions: number;
}
@@ -189,6 +191,21 @@ function positiveDimension(value: string | undefined, fallback: number): number
return parsed;
}
function internalEmbeddingIdentity(
identityValue: string | undefined,
modelValue: string | undefined,
): { id: string; model: string } {
const id = identityValue ?? "ollama/qwen3-embedding:0.6b";
if (!/^ollama\/[A-Za-z0-9][A-Za-z0-9._:-]{0,255}$/.test(id)) {
throw new Error("internal embedding identity configuration is invalid");
}
const model = id.slice(id.indexOf("/") + 1);
if (modelValue !== undefined && modelValue !== model) {
throw new Error("internal embedding model does not match its canonical identity");
}
return { id, model };
}
function catalogDatabase(env: Record<string, string | undefined>): CatalogConnectionConfig | undefined {
const value = env.THT_CATALOG_DATABASE_URL;
if (value !== undefined) {
@@ -330,6 +347,13 @@ export function loadConfig(
|| installationConfigFile.includes("\0")
|| !path.isAbsolute(installationConfigFile)
)) throw new Error("installation configuration is invalid");
const modelCatalogFile = env.THT_MODEL_CATALOG_FILE;
if (modelCatalogFile !== undefined && (
modelCatalogFile.trim() !== modelCatalogFile
|| modelCatalogFile.length === 0
|| modelCatalogFile.includes("\0")
|| !path.isAbsolute(modelCatalogFile)
)) throw new Error("runtime model catalog configuration is invalid");
const piAuthFile = env.THT_PI_AUTH_FILE;
if (piAuthFile !== undefined && (
piAuthFile.trim() !== piAuthFile || piAuthFile.length === 0 || piAuthFile.includes("\0")
@@ -391,6 +415,10 @@ export function loadConfig(
"internal embedding URL",
["embedding", "localhost"],
);
const internalEmbedding = internalEmbeddingIdentity(
env.THT_INTERNAL_EMBEDDING_ID,
env.THT_INTERNAL_EMBEDDING_MODEL,
);
return {
host: env.HOST ?? "127.0.0.1",
port: Number(env.PORT ?? 8787),
@@ -404,7 +432,7 @@ export function loadConfig(
publicExposure,
sessionStorage,
catalogDatabase: catalogDatabase(env),
defaults: { provider: env.PI_PROVIDER, model: env.PI_MODEL, thinking: env.PI_THINKING },
defaults: { thinking: env.PI_THINKING },
maxPiProcesses: Number(env.MAX_PI_PROCESSES ?? 4),
settingsFile,
maintenanceFile: env.THT_MAINTENANCE_FILE ?? path.join(path.dirname(settingsFile), "maintenance.json"),
@@ -413,6 +441,7 @@ export function loadConfig(
piManagementTimeoutMs: piManagementTimeout(env.PI_MANAGEMENT_TIMEOUT_MS),
secretsFile,
installationConfigFile,
modelCatalogFile,
piAuthFile,
secretFiles,
modelApiKeyFile,
@@ -425,7 +454,8 @@ export function loadConfig(
workspaceSecretRuntimeRoot,
internalQdrantUrl,
internalEmbeddingUrl,
internalEmbeddingModel: env.THT_INTERNAL_EMBEDDING_MODEL ?? "qwen3-embedding:0.6b",
internalEmbeddingId: internalEmbedding.id,
internalEmbeddingModel: internalEmbedding.model,
internalEmbeddingDimensions: positiveDimension(env.THT_INTERNAL_EMBEDDING_DIMENSIONS, 1024),
};
}
+1 -1
View File
@@ -8,7 +8,7 @@ import {
isUsableAuthenticationSecret,
} from "../auth/secret-policy.js";
/** Credential names that metadata-generation model entries may reference. */
/** Credential names that Installation Model Catalog providers may reference. */
export const METADATA_GENERATION_SECRET_KEYS = Object.freeze([
"THT_METADATA_API_KEY", "ANTHROPIC_API_KEY", "AZURE_API_KEY", "GEMINI_API_KEY",
"DEEPSEEK_API_KEY", "OPENAI_API_KEY", "OPENROUTER_API_KEY", "ZAI_API_KEY",
+161
View File
@@ -0,0 +1,161 @@
import {
closeSync, constants, fstatSync, lstatSync, openSync, readFileSync, type Stats,
} from "node:fs";
import { z } from "zod";
const MAX_CATALOG_BYTES = 1024 * 1024;
const RUNTIME_CATALOG_FILE = "/run/thothii-model-catalog/catalog.json";
const canonicalId = z.string().regex(/^[a-z][a-z0-9._-]{0,63}\/[A-Za-z0-9][A-Za-z0-9._:-]{0,255}$/);
const secretBundleKey = /^[A-Z][A-Z0-9_]{0,63}$/;
const endpointSchema = z.object({
baseUrl: z.string().url(),
apiVersion: z.string().optional(),
}).strict();
const authenticationSchema = z.object({
mode: z.enum(["secret_env", "pi_auth", "none"]),
apiKeyEnv: z.string().optional(),
}).strict();
const runtimeModelSchema = z.object({
id: canonicalId,
provider: z.string().min(1),
model: z.string().min(1),
label: z.string().min(1),
upstreamModel: z.string().min(1),
endpoint: endpointSchema.optional(),
authentication: authenticationSchema,
sessionAdapter: z.object({ mode: z.enum(["pi_builtin", "openai_compatible"]) }).strict().optional(),
metadataAdapter: z.object({ litellmProvider: z.string().min(1) }).strict().optional(),
session: z.object({
reasoning: z.boolean(),
input: z.array(z.string()).optional(),
cost: z.object({
input: z.number(), output: z.number(), cacheRead: z.number(), cacheWrite: z.number(),
}).strict().optional(),
contextWindow: z.number().int().positive().optional(),
maxTokens: z.number().int().positive().optional(),
compatibility: z.object({
supportsDeveloperRole: z.boolean(),
supportsReasoningEffort: z.boolean(),
supportsStore: z.boolean(),
maxTokensField: z.string().optional(),
}).strict().optional(),
}).strict().optional(),
metadataGeneration: z.object({ disableThinking: z.boolean() }).strict().optional(),
}).strict();
const catalogSchema = z.object({
schemaVersion: z.literal(1),
defaultSession: canonicalId,
defaultMetadataGeneration: canonicalId.optional(),
embedding: z.object({ id: canonicalId, dimensions: z.number().int().positive() }).strict(),
models: z.array(runtimeModelSchema).max(64),
}).strict();
export type RuntimeModel = z.infer<typeof runtimeModelSchema>;
export interface RuntimeModelCatalog {
readonly defaultSession: string | null;
readonly defaultMetadataGeneration: string | null;
readonly embedding: Readonly<{ id: string; dimensions: number }> | null;
sessionModels(): readonly RuntimeModel[];
metadataModels(): readonly RuntimeModel[];
hasSession(id: string): boolean;
}
class RestartLoadedRuntimeModelCatalog implements RuntimeModelCatalog {
readonly defaultSession: string | null;
readonly defaultMetadataGeneration: string | null;
readonly embedding: Readonly<{ id: string; dimensions: number }> | null;
readonly #sessions: readonly RuntimeModel[];
readonly #metadata: readonly RuntimeModel[];
readonly #sessionIds: ReadonlySet<string>;
constructor(catalog?: z.infer<typeof catalogSchema>) {
this.defaultSession = catalog?.defaultSession ?? null;
this.defaultMetadataGeneration = catalog?.defaultMetadataGeneration ?? null;
this.embedding = catalog ? Object.freeze({ ...catalog.embedding }) : null;
this.#sessions = Object.freeze((catalog?.models ?? []).filter((model) => model.session !== undefined));
this.#metadata = Object.freeze((catalog?.models ?? []).filter((model) => model.metadataGeneration !== undefined));
this.#sessionIds = new Set(this.#sessions.map((model) => model.id));
}
sessionModels(): readonly RuntimeModel[] { return this.#sessions.map((model) => ({ ...model })); }
metadataModels(): readonly RuntimeModel[] { return this.#metadata.map((model) => ({ ...model })); }
hasSession(id: string): boolean { return this.#sessionIds.has(id); }
}
function protectedCatalogStat(file: string, info: Stats): boolean {
const mode = info.mode & 0o777;
if (!info.isFile() || info.isSymbolicLink() || info.nlink !== 1
|| info.size < 1 || info.size > MAX_CATALOG_BYTES) return false;
if (file === RUNTIME_CATALOG_FILE && info.uid === 0 && (mode === 0o444 || mode === 0o644)) return true;
return info.uid === (process.getuid?.() ?? info.uid) && (mode === 0o400 || mode === 0o600 || mode === 0o644);
}
function readProtectedCatalog(file: string): unknown {
let descriptor: number | undefined;
try {
const before = lstatSync(file);
if (!protectedCatalogStat(file, before)) throw new Error("runtime model catalog is unavailable");
descriptor = openSync(file, constants.O_RDONLY | constants.O_NOFOLLOW);
const opened = fstatSync(descriptor);
if (!protectedCatalogStat(file, opened)
|| before.dev !== opened.dev || before.ino !== opened.ino) throw new Error("runtime model catalog is unavailable");
const source = readFileSync(descriptor, "utf8");
const after = fstatSync(descriptor);
const current = lstatSync(file);
if (!protectedCatalogStat(file, after) || !protectedCatalogStat(file, current)
|| opened.dev !== after.dev || opened.ino !== after.ino
|| opened.dev !== current.dev || opened.ino !== current.ino) throw new Error("runtime model catalog is unavailable");
return JSON.parse(source);
} catch {
throw new Error("runtime model catalog is unavailable");
} finally {
if (descriptor !== undefined) try { closeSync(descriptor); } catch { /* sanitized above */ }
}
}
export function loadRuntimeModelCatalog(file?: string): RuntimeModelCatalog {
if (!file) return new RestartLoadedRuntimeModelCatalog();
const parsed = catalogSchema.safeParse(readProtectedCatalog(file));
if (!parsed.success) throw new Error("runtime model catalog is invalid");
if (parsed.data.models.some((model) => !validRuntimeModel(model))) {
throw new Error("runtime model catalog is invalid");
}
const ids = new Set(parsed.data.models.map((model) => model.id));
if (ids.size !== parsed.data.models.length) throw new Error("runtime model catalog contains duplicate models");
const sessions = parsed.data.models.filter((model) => model.session !== undefined).map((model) => model.id);
const metadata = parsed.data.models.filter((model) => model.metadataGeneration !== undefined).map((model) => model.id);
if (!sessions.includes(parsed.data.defaultSession)) throw new Error("runtime model catalog session default is invalid");
if ((metadata.length > 0) !== (parsed.data.defaultMetadataGeneration !== undefined)
|| (parsed.data.defaultMetadataGeneration !== undefined
&& !metadata.includes(parsed.data.defaultMetadataGeneration))) {
throw new Error("runtime model catalog metadata default is invalid");
}
return new RestartLoadedRuntimeModelCatalog(parsed.data);
}
function validRuntimeModel(model: RuntimeModel): boolean {
if ((model.session !== undefined) !== (model.sessionAdapter !== undefined)) return false;
if ((model.metadataGeneration !== undefined) !== (model.metadataAdapter !== undefined)) return false;
switch (model.authentication.mode) {
case "secret_env":
return model.authentication.apiKeyEnv !== undefined
&& secretBundleKey.test(model.authentication.apiKeyEnv);
case "pi_auth":
return model.authentication.apiKeyEnv === undefined
&& model.metadataGeneration === undefined
&& model.sessionAdapter?.mode === "pi_builtin";
case "none":
return model.authentication.apiKeyEnv === undefined && model.endpoint !== undefined;
}
}
export function splitCanonicalModelId(id: string): { provider: string; model: string } {
const slash = id.indexOf("/");
if (slash <= 0 || slash === id.length - 1) throw new Error("model identity is invalid");
return { provider: id.slice(0, slash), model: id.slice(slash + 1) };
}
+8 -6
View File
@@ -8,8 +8,8 @@ import { join } from "node:path";
import { loadConfig, type AppConfig } from "./config.js";
import { rolesToPermissions } from "./auth/config.js";
import type { PrincipalContext } from "./auth/principal.js";
import { createPiModelLister } from "./pi/list-models.js";
import { createPiManagement } from "./pi/management.js";
import { loadRuntimeModelCatalog } from "./models/runtime-model-catalog.js";
import { effectiveSettings } from "./routes/settings.js";
import { MaintenanceBarrier } from "./runtime/maintenance-gate.js";
import { loadSettings } from "./settings/settings-store.js";
@@ -19,7 +19,7 @@ import { WorkspaceSecretStore } from "./workspaces/secret-store.js";
type OperatorAction = "maintenance-activate" | "maintenance-deactivate" | "maintenance-status"
| "session-inventory" | "workflow-doctor" | "workspace-integrity"
| "pi-options" | "pi-test" | "effective-settings";
| "pi-test" | "effective-settings";
const lifecyclePrincipal: PrincipalContext = {
issuer: "tht-operator-command",
@@ -110,9 +110,11 @@ export async function runOperatorAction(
if (action === "session-inventory") return await sessionInventory(config);
if (action === "workflow-doctor") return await workflowDiagnostics(config);
if (action === "workspace-integrity") return await workspaceIntegrity(config);
if (action === "effective-settings") return effectiveSettings(config, loadSettings(config));
const service = createPiManagement(config, { listModels: createPiModelLister(config) });
if (action === "pi-options") return await service.options();
const modelCatalog = loadRuntimeModelCatalog(config.modelCatalogFile);
if (action === "effective-settings") {
return effectiveSettings(config, loadSettings(config), modelCatalog);
}
const service = createPiManagement(config, { modelCatalog });
if (action === "pi-test") return await service.test();
throw new Error("unsupported operator action");
}
@@ -121,7 +123,7 @@ async function main(): Promise<void> {
const action = process.argv[2] as OperatorAction | undefined;
if (!action || ![
"maintenance-activate", "maintenance-deactivate", "maintenance-status", "session-inventory",
"workflow-doctor", "workspace-integrity", "pi-options", "pi-test", "effective-settings",
"workflow-doctor", "workspace-integrity", "pi-test", "effective-settings",
].includes(action)) throw new Error("invalid operator action");
const result = await runOperatorAction(action, loadConfig(process.env));
process.stdout.write(`${JSON.stringify(result)}\n`);
+3 -1
View File
@@ -18,6 +18,8 @@ export interface PiModel {
reasoning: boolean;
}
export type ListModelsFn = () => Promise<PiModel[]>;
interface Opts {
spawnFn?: (
command: string,
@@ -36,7 +38,7 @@ interface Opts {
* configured) via an ephemeral `pi --mode rpc` process. Result is cached for
* `ttlMs`. The returned function rejects on timeout/error; callers degrade.
*/
export function createPiModelLister(cfg: AppConfig, opts: Opts = {}): () => Promise<PiModel[]> {
export function createPiModelLister(cfg: AppConfig, opts: Opts = {}): ListModelsFn {
const ttlMs = opts.ttlMs ?? 60_000;
const now = opts.nowMs ?? (() => Date.now());
const spawnFn = opts.spawnFn ?? nodeSpawn;
+27 -92
View File
@@ -2,12 +2,11 @@ import { execFile as nodeExecFile } from "node:child_process";
import { promisify } from "node:util";
import type { AppConfig } from "../config.js";
import { secretValue } from "../config/secret-bundle.js";
import { loadSettings, type Settings } from "../settings/settings-store.js";
import {
loadSettings,
saveSettings,
type Settings,
} from "../settings/settings-store.js";
import type { PiModel } from "./list-models.js";
splitCanonicalModelId,
type RuntimeModelCatalog,
} from "../models/runtime-model-catalog.js";
import {
configuredPiProviderApiKey,
PI_MANAGED_CONFIG_ERROR_MESSAGE,
@@ -45,13 +44,6 @@ export interface PiStatus {
message?: string;
}
export interface PiOptions {
providers: string[];
models: Array<{ provider: string; id: string }>;
reasoning: PiReasoning[];
checkedAt: string;
}
export interface PiTestResult {
ready: boolean;
checkedAt: string;
@@ -76,15 +68,13 @@ export type PiExecFile = (
export interface PiManagementService {
status(): Promise<PiStatus>;
options(): Promise<PiOptions>;
configure(value: PiInstallationConfig): Promise<PiInstallationConfig & { updatedAt: string }>;
test(): Promise<PiTestResult>;
logs(): Promise<PiLogs>;
}
export class PiManagementError extends Error {
constructor(
public readonly code: "pi_management_invalid_config" | "pi_management_unavailable" | "pi_management_write_failed",
public readonly code: "pi_management_unavailable",
message: string,
) {
super(message);
@@ -93,10 +83,9 @@ export class PiManagementError extends Error {
interface PiManagementDeps {
execute?: PiExecFile;
listModels: () => Promise<PiModel[]>;
modelCatalog: RuntimeModelCatalog;
smokeProvider?: PiProviderSmoke;
readSettings?: () => Settings;
saveSettings?: (settings: Settings) => Settings;
readLogs?: () => string | Promise<string>;
credentialStatus?: (provider: string | undefined) => PiCredentialStatus;
now?: () => Date;
@@ -111,54 +100,36 @@ export function createPiManagement(config: AppConfig, deps: PiManagementDeps): P
};
const execute = deps.execute ?? defaultExecFile;
const readSettings = deps.readSettings ?? (() => loadSettings(config));
const persistSettings = deps.saveSettings ?? ((settings) => saveSettings(config, settings));
const readLogs = deps.readLogs ?? (() => diagnostics.join("\n"));
const smokeProvider = deps.smokeProvider ?? createPiProviderSmoke(config);
const smokeProvider = deps.smokeProvider ?? createPiProviderSmoke(config, {
modelCatalog: deps.modelCatalog,
});
const credentialStatus = deps.credentialStatus ?? ((provider: string | undefined) => {
try {
const model = deps.modelCatalog.defaultSession
? deps.modelCatalog.sessionModels().find((entry) => entry.id === deps.modelCatalog.defaultSession)
: undefined;
const credentialName = model?.authentication.mode === "secret_env"
? model.authentication.apiKeyEnv
: undefined;
const configuredApiKey = configuredPiProviderApiKey(
readConfiguredPiAgentFile("models.json", true),
provider,
) ?? (credentialName ? `$${credentialName}` : undefined);
return piProviderCredentialStatus({
provider,
authProviders: loadPiAuthProviders(),
resolveCredentialValue: () => secretValue(config, "THT_MODEL_API_KEY"),
resolveCredentialValue: () => credentialName
? secretValue(config, credentialName)
: config.modelCatalogFile ? undefined : secretValue(config, "THT_MODEL_API_KEY"),
credentialFile: config.modelApiKeyFile,
configuredApiKey: configuredPiProviderApiKey(
readConfiguredPiAgentFile("models.json", true),
provider,
),
configuredApiKey,
});
} catch {
return "missing";
}
});
const closedOptions = async (): Promise<Omit<PiOptions, "checkedAt">> => {
let listed: PiModel[];
try {
listed = await deps.listModels();
} catch (error) {
if (isPiManagedConfigError(error)) {
throw new PiManagementError("pi_management_unavailable", PI_MANAGED_CONFIG_ERROR_MESSAGE);
}
throw new PiManagementError("pi_management_unavailable", "Pi model choices are unavailable");
}
const models: Array<{ provider: string; id: string }> = [];
const providers: string[] = [];
const seenModels = new Set<string>();
const seenProviders = new Set<string>();
for (const model of listed) {
if (!isChoice(model?.provider) || !isChoice(model?.id)) continue;
const key = `${model.provider}\u0000${model.id}`;
if (seenModels.has(key)) continue;
seenModels.add(key);
models.push({ provider: model.provider, id: model.id });
if (!seenProviders.has(model.provider)) {
seenProviders.add(model.provider);
providers.push(model.provider);
}
}
return { providers, models, reasoning: [...REASONING_CHOICES] };
};
const version = async (timeoutMs = config.piManagementTimeoutMs): Promise<string> => {
let output: { stdout: string; stderr: string };
try {
@@ -180,12 +151,12 @@ export function createPiManagement(config: AppConfig, deps: PiManagementDeps): P
const installationConfig = (): PiInstallationConfig => {
const settings = readSettings();
const provider = config.defaults.provider ?? settings.provider;
const model = config.defaults.model ?? settings.model;
const reasoning = config.defaults.thinking ?? settings.thinking;
const selected = deps.modelCatalog.defaultSession
? splitCanonicalModelId(deps.modelCatalog.defaultSession)
: undefined;
return {
...(isChoice(provider) ? { provider } : {}),
...(isChoice(model) ? { model } : {}),
...(selected ? selected : {}),
...(isReasoning(reasoning) ? { reasoning } : {}),
};
};
@@ -206,28 +177,6 @@ export function createPiManagement(config: AppConfig, deps: PiManagementDeps): P
}
},
async options(): Promise<PiOptions> {
const choices = await closedOptions();
return { ...choices, checkedAt: now().toISOString() };
},
async configure(value: PiInstallationConfig): Promise<PiInstallationConfig & { updatedAt: string }> {
if (!isInstallationConfig(value)) {
throw new PiManagementError("pi_management_invalid_config", "Pi installation configuration is invalid");
}
const choices = await closedOptions();
if (!choices.models.some((model) => model.provider === value.provider && model.id === value.model)) {
throw new PiManagementError("pi_management_invalid_config", "Pi provider and model must be selected from available choices");
}
try {
persistSettings({ ...readSettings(), provider: value.provider, model: value.model, thinking: value.reasoning });
} catch {
throw new PiManagementError("pi_management_write_failed", "Pi installation configuration could not be saved");
}
addDiagnostic("Pi installation defaults updated");
return { ...value, updatedAt: now().toISOString() };
},
async test(): Promise<PiTestResult> {
const checkedAt = now().toISOString();
const deadline = Date.now() + config.piManagementTimeoutMs;
@@ -293,24 +242,10 @@ async function defaultExecFile(command: string, args: string[], options: PiExecF
return { stdout: String(result.stdout), stderr: String(result.stderr) };
}
function isChoice(value: unknown): value is string {
return typeof value === "string" && value.length > 0 && value.length <= 128 && value.trim() === value
&& /^[A-Za-z0-9][A-Za-z0-9._/-]*$/u.test(value);
}
function isReasoning(value: unknown): value is PiReasoning {
return typeof value === "string" && (REASONING_CHOICES as readonly string[]).includes(value);
}
function isInstallationConfig(value: unknown): value is Required<PiInstallationConfig> {
if (!value || typeof value !== "object" || Array.isArray(value)) return false;
const candidate = value as Record<string, unknown>;
if (Object.keys(candidate).length !== 3 || Object.keys(candidate).some((key) => !["provider", "model", "reasoning"].includes(key))) {
return false;
}
return isChoice(candidate.provider) && isChoice(candidate.model) && isReasoning(candidate.reasoning);
}
function isTimeout(error: unknown): boolean {
return Boolean(
error && typeof error === "object" && (
+36 -11
View File
@@ -11,6 +11,10 @@ import {
configuredPiProviderApiKey,
createPiRuntimeAgentSnapshot,
} from "./managed-config.js";
import {
loadRuntimeModelCatalog,
type RuntimeModelCatalog,
} from "../models/runtime-model-catalog.js";
export interface SessionRuntime {
rpc: RpcClient;
@@ -42,23 +46,32 @@ export class PiProcessManager {
private runtimes = new Map<string, SessionRuntime>();
private agentSnapshotCleanups = new WeakMap<ChildProcessWithoutNullStreams, () => void>();
private spawnFn: (
sessionId: string, author: string, provider: string | undefined, principal?: PrincipalContext,
runtimeConfigPath?: string,
sessionId: string, author: string, provider: string | undefined, model: string | undefined,
principal?: PrincipalContext, runtimeConfigPath?: string,
) => ChildProcessWithoutNullStreams;
private loadAuthProviders: (agentDir: string) => ReadonlySet<string>;
private modelCatalog: RuntimeModelCatalog;
private modelCatalogConfigured: boolean;
constructor(
private cfg: AppConfig,
opts?: { spawnFn?: SpawnFn; authProviders?: (agentDir: string) => ReadonlySet<string> },
opts?: {
spawnFn?: SpawnFn;
authProviders?: (agentDir: string) => ReadonlySet<string>;
modelCatalog?: RuntimeModelCatalog;
},
) {
this.modelCatalog = opts?.modelCatalog ?? loadRuntimeModelCatalog(cfg.modelCatalogFile);
this.modelCatalogConfigured = cfg.modelCatalogFile !== undefined
|| this.modelCatalog.defaultSession !== null;
this.loadAuthProviders = opts?.authProviders
?? ((agentDir) => loadPiAuthProviders({ agentDir }));
if (opts?.spawnFn) {
this.spawnFn = (sessionId, author, provider, principal, runtimeConfigPath) =>
this.spawnPi(opts.spawnFn!, sessionId, author, provider, principal, runtimeConfigPath);
this.spawnFn = (sessionId, author, provider, model, principal, runtimeConfigPath) =>
this.spawnPi(opts.spawnFn!, sessionId, author, provider, model, principal, runtimeConfigPath);
} else {
this.spawnFn = (sessionId, author, provider, principal, runtimeConfigPath) =>
this.spawnPi(nodeSpawn, sessionId, author, provider, principal, runtimeConfigPath);
this.spawnFn = (sessionId, author, provider, model, principal, runtimeConfigPath) =>
this.spawnPi(nodeSpawn, sessionId, author, provider, model, principal, runtimeConfigPath);
}
}
@@ -71,7 +84,7 @@ export class PiProcessManager {
private spawnPi(
spawnFn: SpawnFn, sessionId: string, author: string, provider: string | undefined,
principal?: PrincipalContext, runtimeConfigPath?: string,
model: string | undefined, principal?: PrincipalContext, runtimeConfigPath?: string,
): ChildProcessWithoutNullStreams {
// This is the final shared boundary for createFor(), spawnFor(), and resume(). Validate
// before auth-provider inspection, then make Pi consume the exact copied bytes rather than
@@ -79,12 +92,23 @@ export class PiProcessManager {
const agent = createPiRuntimeAgentSnapshot();
let child: ChildProcessWithoutNullStreams | undefined;
try {
const catalogModel = provider && model
? this.modelCatalog.sessionModels()
.find((entry) => entry.provider === provider && entry.model === model)
: undefined;
const credentialName = catalogModel?.authentication.mode === "secret_env"
? catalogModel.authentication.apiKeyEnv
: undefined;
const projectedApiKey = configuredPiProviderApiKey(agent.models, provider)
?? (credentialName ? `$${credentialName}` : undefined);
const env = buildPiChildEnv({
provider,
authProviders: this.loadAuthProviders(agent.agentDir),
credentialValue: secretValue(this.cfg, "THT_MODEL_API_KEY"),
credentialValue: credentialName
? secretValue(this.cfg, credentialName)
: this.modelCatalogConfigured ? undefined : secretValue(this.cfg, "THT_MODEL_API_KEY"),
credentialFile: this.cfg.modelApiKeyFile,
configuredApiKey: configuredPiProviderApiKey(agent.models, provider),
configuredApiKey: projectedApiKey,
additions: { THT_SESSION: sessionId, THT_AUTHOR: author },
});
env.PI_CODING_AGENT_DIR = agent.agentDir;
@@ -162,9 +186,10 @@ export class PiProcessManager {
}
const author = o.author ?? "dev@local";
const provider = canonicalPiProvider(o.provider ?? this.cfg.defaults.provider);
const model = o.model ?? this.cfg.defaults.model;
let child: ChildProcessWithoutNullStreams;
try {
child = this.spawnFn(sessionId, author, provider, o.principal, o.runtimeConfig?.path);
child = this.spawnFn(sessionId, author, provider, model, o.principal, o.runtimeConfig?.path);
} catch (error) {
o.runtimeConfig?.release();
throw error;
+19 -2
View File
@@ -17,6 +17,10 @@ import {
readConfiguredPiAgentFile,
validateDeclarativePiConfig,
} from "./managed-config.js";
import {
loadRuntimeModelCatalog,
type RuntimeModelCatalog,
} from "../models/runtime-model-catalog.js";
const SMOKE_PROMPT = "Provider health check. Reply with exactly OK.";
const SMOKE_ARGS = [
@@ -49,6 +53,7 @@ interface ProviderSmokeOptions {
authProviders?: () => ReadonlySet<string>;
readAuthStore?: () => string;
readModelsStore?: () => string | undefined;
modelCatalog?: RuntimeModelCatalog;
}
export function createPiProviderSmoke(
@@ -69,12 +74,24 @@ export function createPiProviderSmoke(
const configuredModels = options.readModelsStore
? options.readModelsStore()
: readConfiguredPiAgentFile("models.json", true);
const catalog = options.modelCatalog ?? loadRuntimeModelCatalog(config.modelCatalogFile);
const catalogConfigured = config.modelCatalogFile !== undefined
|| catalog.defaultSession !== null;
const catalogModel = catalog.sessionModels()
.find((entry) => entry.provider === canonicalProvider && entry.model === model);
const credentialName = catalogModel?.authentication.mode === "secret_env"
? catalogModel.authentication.apiKeyEnv
: undefined;
const projectedApiKey = configuredPiProviderApiKey(configuredModels, canonicalProvider)
?? (credentialName ? `$${credentialName}` : undefined);
const env = buildPiChildEnv({
provider: canonicalProvider,
authProviders: configuredAuthProviders,
credentialValue: secretValue(config, "THT_MODEL_API_KEY"),
credentialValue: credentialName
? secretValue(config, credentialName)
: catalogConfigured ? undefined : secretValue(config, "THT_MODEL_API_KEY"),
credentialFile: config.modelApiKeyFile,
configuredApiKey: configuredPiProviderApiKey(configuredModels, canonicalProvider),
configuredApiKey: projectedApiKey,
});
clearPrincipalEnvironment(env);
delete env.THT_DATA_ROOT;
@@ -32,7 +32,7 @@ import {
} from "../catalog/types.js";
const idSchema = z.uuid();
const modelIdSchema = z.string().regex(/^[a-z][a-z0-9._-]{0,63}$/);
const modelIdSchema = z.string().regex(/^[a-z][a-z0-9._-]{0,63}\/[A-Za-z0-9][A-Za-z0-9._:-]{0,255}$/);
const selectedTargetIdsSchema = z.array(idSchema).min(1);
const suggestionSchema = z.discriminatedUnion("scope", [
z.object({ modelId: modelIdSchema, scope: z.literal("all") }).strict(),
+10 -15
View File
@@ -1,10 +1,8 @@
import { readdirSync } from "node:fs";
import { join } from "node:path";
import type { FastifyInstance } from "fastify";
import type { PiModel } from "../pi/list-models.js";
import { isPrincipalContext, requirePermission } from "../auth/authorization.js";
export type ListModelsFn = () => Promise<PiModel[]>;
import type { RuntimeModelCatalog } from "../models/runtime-model-catalog.js";
/**
* List YAML workspace configs found in <harnessDir>/workspaces/*.yaml.
@@ -25,20 +23,17 @@ export function listWorkspaces(harnessDir: string): { name: string; file: string
export function metaRoutes(
app: FastifyInstance,
deps: { harnessDir: string; listModels?: ListModelsFn },
deps: { harnessDir: string; modelCatalog: RuntimeModelCatalog },
): void {
app.get("/models", async (request, reply) => {
if (!isPrincipalContext(requirePermission(request, reply, "session.use"))) return reply;
const fn = deps.listModels ?? (async () => []);
try {
return { models: await fn() };
} catch (error) {
app.log.warn({
component: "pi-model-list",
errorType: error instanceof Error ? error.name : typeof error,
}, "Pi model listing failed");
// Graceful fallback: Pi may not be running; don't crash the server.
return { models: [] as PiModel[] };
}
return {
models: deps.modelCatalog.sessionModels().map((entry) => ({
provider: entry.provider,
id: entry.model,
name: entry.label,
reasoning: entry.session?.reasoning ?? false,
})),
};
});
}
+1 -9
View File
@@ -11,13 +11,6 @@ export function piManagementRoutes(
deps: { service: PiManagementService },
): void {
app.get("/pi-management/status", async (request, reply) => run(request, reply, deps, () => deps.service.status()));
app.get("/pi-management/options", async (request, reply) => run(request, reply, deps, () => deps.service.options()));
app.put("/pi-management/config", async (request, reply) => run(
request,
reply,
deps,
() => deps.service.configure((request.body ?? {}) as Record<string, unknown>),
));
app.post("/pi-management/test", async (request, reply) => run(request, reply, deps, () => deps.service.test()));
app.get("/pi-management/logs", async (request, reply) => run(request, reply, deps, () => deps.service.logs()));
}
@@ -38,8 +31,7 @@ async function run<T>(
return await action();
} catch (error) {
if (error instanceof PiManagementError) {
const statusCode = error.code === "pi_management_invalid_config" ? 400 : 503;
return reply.code(statusCode).send({ code: error.code, error: error.message });
return reply.code(503).send({ code: error.code, error: error.message });
}
return reply.code(503).send({ code: "pi_management_unavailable", error: "Pi management is unavailable" });
}
+18 -9
View File
@@ -6,12 +6,13 @@ import type { Settings } from "../settings/settings-store.js";
import { getPrincipal } from "../auth/auth.js";
import type { PrincipalContext } from "../auth/principal.js";
import type { ReadinessManager } from "../runtime/readiness-manager.js";
import type { ListModelsFn } from "./meta.js";
import type { ListModelsFn } from "../pi/list-models.js";
import type { WorkspaceRegistry } from "../workspaces/registry.js";
import { validateOperationalWorkspace, type WorkspaceDescriptor } from "../workspaces/schema.js";
import type { MaintenanceBarrier } from "../runtime/maintenance-gate.js";
import { hasPermission, isPrincipalContext, requirePermission } from "../auth/authorization.js";
import type { EffectiveRelationshipSnapshotProvider } from "../catalog/effective-relationship-snapshot.js";
import { splitCanonicalModelId, type RuntimeModelCatalog } from "../models/runtime-model-catalog.js";
const BOOTSTRAP_FAILURE_MESSAGE =
"Session startup failed. Check configuration and connectivity, then Resume the session.";
@@ -43,6 +44,7 @@ export function sessionRoutes(
maintenanceBarrier: MaintenanceBarrier;
/** Optional only for narrow route-test stubs and installations without a Catalog database. */
effectiveRelationships?: EffectiveRelationshipSnapshotProvider;
modelCatalog: RuntimeModelCatalog;
},
) {
const lifecycleTails = new Map<string, Promise<void>>();
@@ -358,7 +360,6 @@ export function sessionRoutes(
let workspaceId: string | undefined;
let workspaceRevision: string | undefined;
let workspaceDescriptor: WorkspaceDescriptor | undefined;
let allowedModels: readonly string[] | undefined;
if (requestedWorkspaceId) {
try {
const registry = d.workspaceRegistry as Partial<WorkspaceRegistry>;
@@ -378,7 +379,6 @@ export function sessionRoutes(
workspaceId = resolved.revision.id;
workspaceRevision = resolved.revision.commit;
workspaceDescriptor = resolved.workspace;
allowedModels = resolved.workspace.llm_policy.allowed;
} catch {
return reply.code(409).send({
error: WORKSPACE_REVISION_UNAVAILABLE_MESSAGE,
@@ -386,12 +386,17 @@ export function sessionRoutes(
});
}
}
const provider = b.provider ?? s.provider;
const model = b.model ?? s.model;
const requestedCanonical = b.provider && b.model ? `${b.provider}/${b.model}` : undefined;
let selectedCanonical = requestedCanonical ?? d.modelCatalog.defaultSession;
let modelWarning: string | undefined;
if (selectedCanonical && d.modelCatalog.defaultSession && !d.modelCatalog.hasSession(selectedCanonical)) {
selectedCanonical = d.modelCatalog.defaultSession;
modelWarning = `Configured model ${requestedCanonical ?? "selection"} is unavailable; using ${selectedCanonical}.`;
}
const selected = selectedCanonical ? splitCanonicalModelId(selectedCanonical) : undefined;
const provider = selected?.provider ?? b.provider;
const model = selected?.model ?? b.model;
const thinking = b.thinking ?? s.thinking;
if (allowedModels && provider && model && !allowedModels.includes(`${provider}/${model}`)) {
return reply.code(400).send({ error: "Selected model is not allowed by this workspace." });
}
// A persisted session is resumable without keeping Pi alive. New work replaces every
// runtime owned by this principal, while runtimes belonging to other users remain intact.
// Optional chaining preserves the deliberately narrow manager stubs used by route tests.
@@ -490,7 +495,7 @@ export function sessionRoutes(
),
() => d.mgr.start(id, rt, runtimeOptions),
);
return { id };
return { id, ...(modelWarning ? { warning: modelWarning } : {}) };
} finally {
if (revisionLease && !manifestPersisted) {
await revisionLease.abort().catch((error: unknown) => {
@@ -598,6 +603,10 @@ export function sessionRoutes(
provider?: string; model?: string; thinking?: string;
workspace_id?: string; workspace_revision?: string;
};
const savedCanonical = saved.provider && saved.model ? `${saved.provider}/${saved.model}` : "";
if (d.modelCatalog.defaultSession && (!savedCanonical || !d.modelCatalog.hasSession(savedCanonical))) {
return reply.code(503).send({ error: MODEL_UNAVAILABLE_MESSAGE, code: "model_unavailable" });
}
let workspaceConfigPath: string;
let workspaceDescriptor: WorkspaceDescriptor | undefined;
try {
+16 -22
View File
@@ -1,17 +1,27 @@
import type { FastifyInstance } from "fastify";
import type { AppConfig } from "../config.js";
import type { Settings } from "../settings/settings-store.js";
import { listWorkspaces, type ListModelsFn } from "./meta.js";
import { listWorkspaces } from "./meta.js";
import type { PrincipalContext } from "../auth/principal.js";
import { isPrincipalContext, requirePermission } from "../auth/authorization.js";
import {
splitCanonicalModelId,
type RuntimeModelCatalog,
} from "../models/runtime-model-catalog.js";
/** Merge stored settings over env/first-workspace defaults. */
export function effectiveSettings(cfg: AppConfig, stored: Settings): Settings {
/** Merge only workspace and runtime-thinking preferences; model defaults belong to modelCatalog. */
export function effectiveSettings(
cfg: AppConfig,
stored: Settings,
modelCatalog?: RuntimeModelCatalog,
): Settings {
const workspaces = listWorkspaces(cfg.harnessDir);
const selected = modelCatalog?.defaultSession
? splitCanonicalModelId(modelCatalog.defaultSession)
: undefined;
return {
workspace: stored.workspace ?? workspaces[0]?.name,
provider: cfg.defaults.provider ?? stored.provider,
model: cfg.defaults.model ?? stored.model,
...(selected ?? {}),
thinking: cfg.defaults.thinking ?? stored.thinking,
};
}
@@ -19,7 +29,7 @@ export function effectiveSettings(cfg: AppConfig, stored: Settings): Settings {
export function settingsRoutes(
app: FastifyInstance,
deps: {
cfg: AppConfig; listModels: ListModelsFn;
cfg: AppConfig;
getSettings: (principal: PrincipalContext) => Promise<Settings>;
},
): void {
@@ -37,22 +47,6 @@ export function settingsRoutes(
const principal = requirePermission(req, reply, "settings.manage");
if (!isPrincipalContext(principal)) return principal;
const b = (req.body ?? {}) as Settings;
if (b.model) {
let available: { provider: string; id: string }[] = [];
try {
available = await deps.listModels();
} catch {
available = [];
}
// Only validate when Pi gave us a non-empty list; otherwise allow (degraded).
if (available.length > 0 && !available.some(
(candidate) => candidate.provider === b.provider && candidate.id === b.model,
)) {
return reply.code(400).send({
error: `Unknown model: ${b.provider ?? "unknown"}/${b.model}`,
});
}
}
try {
// Retain this endpoint as a validating compatibility surface for older clients, but do
// not write anonymous users' choices to shared server storage.
+8 -1
View File
@@ -13,6 +13,7 @@ import type { AppConfig } from "../config.js";
export interface Settings {
workspace?: string;
/** Legacy input fields are ignored when loading/evaluating installation settings. */
provider?: string;
model?: string;
thinking?: string;
@@ -37,7 +38,13 @@ export function loadSettings(cfg: AppConfig): Settings {
try {
const raw = readFileSync(cfg.settingsFile, "utf8");
const parsed = JSON.parse(raw);
if (parsed && typeof parsed === "object") return parsed as Settings;
if (parsed && typeof parsed === "object") {
const value = parsed as Record<string, unknown>;
return {
...(typeof value.workspace === "string" ? { workspace: value.workspace } : {}),
...(typeof value.thinking === "string" ? { thinking: value.thinking } : {}),
};
}
return {};
} catch {
return {};
+5 -1
View File
@@ -598,7 +598,11 @@ export class ThtRunner {
} catch {
return { ok: false, code: "workspace_not_activatable" };
}
const collection = descriptor.semantic_index.vector_store;
const collection = {
collection: descriptor.workspace.id,
dimensions: this.cfg.semanticRuntime.internalEmbeddingDimensions,
distance: "cosine" as const,
};
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), Math.max(1, timeoutSec) * 1000);
try {
+1
View File
@@ -238,6 +238,7 @@ function createProductionService(): WorkspacePreprocessingService {
});
return new WorkspacePreprocessingService({
dataRoot: config.dataRoot ?? "/data",
embeddingDimensions: config.internalEmbeddingDimensions,
httpPrivateHostAllowlist: (process.env.THT_EVIDENCE_PRIVATE_HOST_ALLOWLIST ?? "")
.split(",").map((value) => value.trim()).filter((value) => value.length > 0),
acquireActiveRuntime: async (workspaceId) => {
+4 -4
View File
@@ -59,12 +59,12 @@ function safeSecretFilePath(path: string, secretRoots: readonly string[]): strin
function requireSupportedDescriptor(workspace: unknown): void {
if (typeof workspace !== "object" || workspace === null) {
throw new Error("Workspace bindings support only workspace schema version 3");
throw new Error("Workspace bindings support only workspace schema version 4");
}
const metadata = Reflect.get(workspace, "workspace");
if (typeof metadata !== "object" || metadata === null
|| Reflect.get(metadata, "schema_version") !== 3) {
throw new Error("Workspace bindings support only workspace schema version 3");
|| Reflect.get(metadata, "schema_version") !== 4) {
throw new Error("Workspace bindings support only workspace schema version 4");
}
}
@@ -155,7 +155,7 @@ export function resolveEvidenceBinding(
return { values, missing };
}
/** Resolve the complete schema-v3 runtime binding set. */
/** Resolve the complete schema-v4 runtime binding set. */
export function resolveRuntimeBindings(
workspace: WorkspaceDescriptor,
env: NodeJS.ProcessEnv,
+3 -3
View File
@@ -137,12 +137,12 @@ function evidenceVariables(
function requireSupportedDescriptor(workspace: unknown): void {
if (typeof workspace !== "object" || workspace === null) {
throw new Error("Installation contract supports only workspace schema version 3");
throw new Error("Installation contract supports only workspace schema version 4");
}
const metadata = Reflect.get(workspace, "workspace");
if (typeof metadata !== "object" || metadata === null
|| Reflect.get(metadata, "schema_version") !== 3) {
throw new Error("Installation contract supports only workspace schema version 3");
|| Reflect.get(metadata, "schema_version") !== 4) {
throw new Error("Installation contract supports only workspace schema version 4");
}
}
+11 -9
View File
@@ -406,12 +406,12 @@ function numericBinding(binding: Record<string, string>, name: string): number |
function requireSupportedDescriptor(workspace: unknown): void {
if (typeof workspace !== "object" || workspace === null) {
throw new Error("Workspace diagnoser supports only workspace schema version 3");
throw new Error("Workspace diagnoser supports only workspace schema version 4");
}
const metadata = Reflect.get(workspace, "workspace");
if (typeof metadata !== "object" || metadata === null
|| Reflect.get(metadata, "schema_version") !== 3) {
throw new Error("Workspace diagnoser supports only workspace schema version 3");
|| Reflect.get(metadata, "schema_version") !== 4) {
throw new Error("Workspace diagnoser supports only workspace schema version 4");
}
}
@@ -506,11 +506,15 @@ async function diagnoseValidatedWorkspace(
try {
const vector = await withTimeout(timeoutMs, (signal) => adapters.inspectQdrant({
baseUrl: semanticRuntime.internalQdrantUrl,
collection: descriptor.semantic_index.vector_store.collection,
collection: descriptor.workspace.id,
timeoutMs,
signal,
}));
const expected = descriptor.semantic_index.vector_store;
const expected = {
collection: descriptor.workspace.id,
dimensions: semanticRuntime.internalEmbeddingDimensions,
distance: "cosine",
};
if (vector.collection !== expected.collection
|| vector.dimensions !== expected.dimensions
|| vector.distance !== expected.distance) {
@@ -529,10 +533,8 @@ async function diagnoseValidatedWorkspace(
timeoutMs,
signal,
}));
if (semanticRuntime.internalEmbeddingModel !== descriptor.semantic_index.embedding.model
|| semanticRuntime.internalEmbeddingDimensions !== descriptor.semantic_index.embedding.dimensions
|| !embedding.available
|| embedding.dimensions !== descriptor.semantic_index.embedding.dimensions) {
if (!embedding.available
|| embedding.dimensions !== semanticRuntime.internalEmbeddingDimensions) {
diagnostics.push(diagnosticError("semantic_index_incompatible"));
activatable = false;
}
+6 -3
View File
@@ -2,7 +2,7 @@ import { createHash } from "node:crypto";
import { normalize } from "node:path";
export interface CanonicalEffectiveConfig {
schemaVersion: 1;
schemaVersion: 2;
dwh: CanonicalDwhConfig;
vector: CanonicalVectorConfig;
embedding: CanonicalEmbeddingConfig;
@@ -27,6 +27,7 @@ export interface CanonicalVectorConfig {
}
export interface CanonicalEmbeddingConfig {
id: string;
model: string;
dimensions: number;
}
@@ -137,8 +138,10 @@ function buildEmbeddingConfig(rendered: Record<string, unknown>): CanonicalEmbed
if (!embeddings) {
throw new TypeError("effective config is missing embedding resources");
}
const model = requireString(embeddings, "model");
return {
model: requireString(embeddings, "model"),
id: `ollama/${model}`,
model,
dimensions: requireNumber(embeddings, "dimensions"),
};
}
@@ -166,7 +169,7 @@ export function buildCanonicalEffectiveConfig(renderedConfig: unknown): Canonica
throw new TypeError("effective config requires a rendered configuration object");
}
return {
schemaVersion: 1,
schemaVersion: 2,
dwh: buildDwhConfig(rendered),
vector: buildVectorConfig(rendered),
embedding: buildEmbeddingConfig(rendered),
@@ -77,6 +77,7 @@ export interface WorkspacePreprocessingServiceDeps {
{ ok: true } | { ok: false; code: "workspace_not_activatable" | "semantic_index_incompatible" }
>;
httpPrivateHostAllowlist?: readonly string[];
embeddingDimensions?: number;
}
interface RunScope {
@@ -118,7 +119,7 @@ export class WorkspacePreprocessingService {
async vectorInspect(options: { workspaceId: string }): Promise<WorkspaceOperationResult> {
const runtime = await this.deps.acquireActiveRuntime(options.workspaceId);
const collection = runtime.workspace.semantic_index.vector_store.collection;
const collection = runtime.workspace.workspace.id;
const res = await fetch(`${runtime.configLease.semanticQdrantUrl}/collections/${encodeURIComponent(collection)}`, { method: "GET" });
if (!res.ok) return baseResult(runtime, "vector inspect", "failed", "semantic_index_incompatible", { warnings: ["collection unavailable"] });
const body = await res.json() as any;
@@ -132,7 +133,7 @@ export class WorkspacePreprocessingService {
async vectorRebuild(options: { workspaceId: string; collection?: string; confirm?: string; destroy?: boolean }): Promise<WorkspaceOperationResult> {
const runtime = await this.deps.acquireActiveRuntime(options.workspaceId);
const collection = runtime.workspace.semantic_index.vector_store.collection;
const collection = runtime.workspace.workspace.id;
if (options.collection !== collection || options.confirm !== collection || options.destroy !== true) {
return baseResult(runtime, "vector rebuild", "failed", "semantic_index_incompatible", { warnings: ["rebuild requires exact confirmation and --destroy"] });
}
@@ -143,8 +144,8 @@ export class WorkspacePreprocessingService {
const recreated = await reconcileCollection({
baseUrl: runtime.configLease.semanticQdrantUrl,
collection,
dimensions: runtime.workspace.semantic_index.vector_store.dimensions,
distance: runtime.workspace.semantic_index.vector_store.distance,
dimensions: this.deps.embeddingDimensions ?? 1024,
distance: "cosine",
mode: "self_heal",
});
if (!recreated.ok) return baseResult(runtime, "vector rebuild", "failed", "semantic_index_incompatible", { warnings: ["collection recreate failed"] });
@@ -400,6 +401,8 @@ export class WorkspacePreprocessingService {
catalogBlob: runtime.catalogBlob,
configDigest: runtime.configLease.configDigest,
bindingDigest: runtime.configLease.bindingDigest,
embeddingId: runtime.configLease.effectiveConfig.embedding.id,
embeddingDimensions: runtime.configLease.effectiveConfig.embedding.dimensions,
});
return { runtime, state, job };
}
+13 -3
View File
@@ -44,11 +44,13 @@ export interface BeginPreprocessingJobOptions {
catalogBlob: string;
configDigest: string;
bindingDigest: string;
embeddingId: string;
embeddingDimensions: number;
runId?: string;
}
export interface PreprocessingJobState {
schemaVersion: 1;
schemaVersion: 2;
runId: string;
operation: string;
workspaceId: string;
@@ -57,6 +59,8 @@ export interface PreprocessingJobState {
catalogBlob: string;
configDigest: string;
bindingDigest: string;
embeddingId: string;
embeddingDimensions: number;
completedStages: string[];
childRuns: Record<string, string>;
status: "active" | "succeeded" | "blocked" | "failed";
@@ -146,7 +150,7 @@ function decodeJob(value: unknown): PreprocessingJobState {
}
const record = value as Record<string, unknown>;
if (
record.schemaVersion !== 1
record.schemaVersion !== 2
|| typeof record.runId !== "string"
|| typeof record.operation !== "string"
|| typeof record.workspaceId !== "string"
@@ -155,6 +159,8 @@ function decodeJob(value: unknown): PreprocessingJobState {
|| typeof record.catalogBlob !== "string"
|| typeof record.configDigest !== "string"
|| typeof record.bindingDigest !== "string"
|| typeof record.embeddingId !== "string"
|| typeof record.embeddingDimensions !== "number"
|| !Array.isArray(record.completedStages)
|| typeof record.childRuns !== "object" || record.childRuns === null || Array.isArray(record.childRuns)
|| !["active", "succeeded", "blocked", "failed"].includes(String(record.status))
@@ -275,6 +281,8 @@ export class PreprocessingStateStore {
|| existing.catalogBlob !== options.catalogBlob
|| existing.configDigest !== options.configDigest
|| existing.bindingDigest !== options.bindingDigest
|| existing.embeddingId !== options.embeddingId
|| existing.embeddingDimensions !== options.embeddingDimensions
) {
throw new PreprocessingStateError(
"preprocessing_resume_mismatch",
@@ -295,7 +303,7 @@ export class PreprocessingStateStore {
}
}
const job: PreprocessingJobState = {
schemaVersion: 1,
schemaVersion: 2,
runId,
operation: options.operation,
workspaceId: this.options.workspaceId,
@@ -304,6 +312,8 @@ export class PreprocessingStateStore {
catalogBlob: options.catalogBlob,
configDigest: options.configDigest,
bindingDigest: options.bindingDigest,
embeddingId: options.embeddingId,
embeddingDimensions: options.embeddingDimensions,
completedStages: [],
childRuns: {},
status: "active",
+1 -1
View File
@@ -560,7 +560,7 @@ export class WorkspaceRegistry {
});
}
}
const collection = workspace.semantic_index.vector_store.collection;
const collection = workspace.workspace.id;
const owner = collectionOwners.get(collection);
if (owner !== undefined) {
throw new Error(`duplicate qdrant collection ownership: ${collection} (${owner}, ${id})`);
+20 -6
View File
@@ -34,6 +34,7 @@ export interface RuntimeInstallationOverlay {
export interface SemanticRuntimeConfig {
internalQdrantUrl: string;
internalEmbeddingUrl: string;
internalEmbeddingId?: string;
internalEmbeddingModel: string;
internalEmbeddingDimensions: number;
}
@@ -41,6 +42,7 @@ export interface SemanticRuntimeConfig {
export const DEFAULT_SEMANTIC_RUNTIME: SemanticRuntimeConfig = {
internalQdrantUrl: "http://qdrant:6333",
internalEmbeddingUrl: "http://embedding:11434",
internalEmbeddingId: "ollama/qwen3-embedding:0.6b",
internalEmbeddingModel: "qwen3-embedding:0.6b",
internalEmbeddingDimensions: 1024,
};
@@ -202,16 +204,16 @@ function placeholderConnection(identity: { database: string; schema: string }):
function requireSupportedDescriptor(workspace: unknown): void {
if (typeof workspace !== "object" || workspace === null) {
throw new Error("Runtime renderer supports only workspace schema version 3");
throw new Error("Runtime renderer supports only workspace schema version 4");
}
const metadata = Reflect.get(workspace, "workspace");
if (typeof metadata !== "object" || metadata === null
|| Reflect.get(metadata, "schema_version") !== 3) {
throw new Error("Runtime renderer supports only workspace schema version 3");
|| Reflect.get(metadata, "schema_version") !== 4) {
throw new Error("Runtime renderer supports only workspace schema version 4");
}
}
/** Render the schema-v3 compatibility fields consumed by the current Python harness. */
/** Render the schema-v4 compatibility fields consumed by the current Python harness. */
export function renderRuntimeConfig(
workspace: WorkspaceDescriptor,
bindings: RuntimeBindings,
@@ -263,12 +265,24 @@ export function renderRuntimeConfig(
...(installation.profile === undefined ? {} : { profile: installation.profile }),
language: descriptor.workspace.language,
database,
semantic_index: descriptor.semantic_index,
semantic_index: {
vector_store: {
engine: "qdrant",
collection: descriptor.workspace.id,
dimensions: semanticRuntime.internalEmbeddingDimensions,
distance: "cosine",
},
embedding: {
provider: "ollama_internal",
model: semanticRuntime.internalEmbeddingModel,
dimensions: semanticRuntime.internalEmbeddingDimensions,
},
},
resources: {
vector: {
engine: "qdrant",
base_url: semanticRuntime.internalQdrantUrl,
collection: descriptor.semantic_index.vector_store.collection,
collection: descriptor.workspace.id,
},
embeddings: {
provider: "ollama_internal",
+29 -67
View File
@@ -35,7 +35,7 @@ export interface CanonicalDiagnostics {
}
interface WorkspaceMetadata {
schema_version: 3;
schema_version: 4;
id: string;
name: string;
description?: string;
@@ -51,31 +51,12 @@ interface WorkspaceDwh {
supported_transports: DwhTransport[];
}
interface WorkspaceBase<TVectorStore> {
interface WorkspaceBase {
workspace: WorkspaceMetadata;
dwh: WorkspaceDwh;
semantic_index: {
vector_store: TVectorStore;
embedding: {
provider: "ollama_internal";
model: "qwen3-embedding:0.6b";
dimensions: 1024;
};
};
llm_policy: {
default?: `${string}/${string}`;
allowed: `${string}/${string}`[];
};
diagnostics?: Pick<CanonicalDiagnostics, "dwh_rest">;
}
interface QdrantVectorStore {
engine: "qdrant";
collection: string;
dimensions: 1024;
distance: "cosine";
}
export interface EvidencePolicy {
max_chunk_chars: number;
retain_published_generations: number;
@@ -120,12 +101,12 @@ export interface WorkspaceEvidence {
policy: EvidencePolicy;
}
export interface WorkspaceV3 extends WorkspaceBase<QdrantVectorStore> {
export interface WorkspaceV4 extends WorkspaceBase {
evidence?: WorkspaceEvidence;
}
export type CanonicalWorkspace = WorkspaceV3;
export type WorkspaceDescriptor = WorkspaceV3;
export type CanonicalWorkspace = WorkspaceV4;
export type WorkspaceDescriptor = WorkspaceV4;
const workspaceId = z.string().regex(/^[a-z][a-z0-9-]{2,62}$/, {
message: "workspace id must match ^[a-z][a-z0-9-]{2,62}$",
@@ -135,9 +116,6 @@ const identifier = z.string().regex(/^[A-Za-z_][A-Za-z0-9_]*$/, {
});
const port = z.number().int().min(1).max(65_535);
const timeoutMs = z.number().int().positive();
const modelReference = z.string().regex(/^[^/\s]+\/[^/\s]+$/, {
message: "model must use provider/model syntax",
});
function isOriginRelativeDiagnosticPath(value: string): boolean {
return /^\/(?!\/)[^\\\u0000-\u001F\u007F?#]*$/.test(value) && !/%5c/i.test(value);
@@ -166,21 +144,6 @@ const dwhSchema = z.object({
timeout_ms: timeoutMs.optional(),
supported_transports: z.array(z.enum(DWH_TRANSPORTS)).min(1),
}).strict();
const internalEmbeddingSchema = z.object({
provider: z.literal("ollama_internal"),
model: z.literal("qwen3-embedding:0.6b"),
dimensions: z.literal(1024),
}).strict();
const qdrantVectorStoreSchema = z.object({
engine: z.literal("qdrant"),
collection: workspaceId,
dimensions: z.literal(1024),
distance: z.literal("cosine"),
}).strict();
const llmPolicySchema = z.object({
default: modelReference.optional(),
allowed: z.array(modelReference).min(1),
}).strict();
const positiveSafeInteger = z.number().int().safe().positive();
const nonnegativeSafeInteger = z.number().int().safe().nonnegative();
@@ -366,7 +329,6 @@ function unique<T>(values: readonly T[], context: z.RefinementCtx, path: Propert
function workspaceInvariants(workspace: any, context: z.RefinementCtx): void {
unique(workspace.dwh.supported_transports, context, ["dwh", "supported_transports"]);
unique(workspace.llm_policy.allowed, context, ["llm_policy", "allowed"]);
if (workspace.evidence?.source.type === "filesystem") {
const expected = `${workspace.workspace.id}/evidence`;
@@ -379,20 +341,6 @@ function workspaceInvariants(workspace: any, context: z.RefinementCtx): void {
}
}
if (workspace.semantic_index.vector_store.dimensions !== workspace.semantic_index.embedding.dimensions) {
context.addIssue({
code: "custom",
path: ["semantic_index", "embedding", "dimensions"],
message: "embedding dimensions must match vector store dimensions",
});
}
if (workspace.llm_policy.default && !workspace.llm_policy.allowed.includes(workspace.llm_policy.default)) {
context.addIssue({
code: "custom",
path: ["llm_policy", "default"],
message: "LLM default must be included in the allowlist",
});
}
if (workspace.diagnostics?.dwh_rest && !workspace.dwh.supported_transports.includes("rest_api")) {
context.addIssue({
code: "custom",
@@ -402,23 +350,18 @@ function workspaceInvariants(workspace: any, context: z.RefinementCtx): void {
}
}
const WorkspaceV3Schema = z.object({
const WorkspaceV4Schema = z.object({
dwh: dwhSchema,
llm_policy: llmPolicySchema,
evidence: workspaceEvidenceSchema.optional(),
diagnostics: z.object({
dwh_rest: dwhRestDiagnostic.optional(),
}).strict().optional(),
workspace: z.object({
schema_version: z.literal(3), id: workspaceId, name: z.string().trim().min(1),
schema_version: z.literal(4), id: workspaceId, name: z.string().trim().min(1),
description: z.string().trim().min(1).optional(), language: z.enum(["en", "it"]),
}).strict(),
semantic_index: z.object({
vector_store: qdrantVectorStoreSchema,
embedding: internalEmbeddingSchema,
}).strict(),
}).strict().superRefine(workspaceInvariants);
const WorkspaceDescriptorSchema = WorkspaceV3Schema;
const WorkspaceDescriptorSchema = WorkspaceV4Schema;
export function parseWorkspaceYaml(source: string): WorkspaceDescriptor {
const documents = parseAllDocuments(source, { uniqueKeys: true });
@@ -431,6 +374,25 @@ export function parseWorkspaceYaml(source: string): WorkspaceDescriptor {
return validateWorkspaceDescriptor(document.toJSON());
}
/** Deterministically removes the two installation-owned v3 blocks without altering workspace data. */
export function migrateWorkspaceV3Yaml(source: string): string {
const documents = parseAllDocuments(source, { uniqueKeys: true });
if (documents.length !== 1) throw new Error("Workspace YAML must contain exactly one document");
const document = documents[0];
if (document.errors.length > 0 || document.warnings.length > 0) {
throw new Error("Invalid workspace YAML");
}
const value = document.toJSON() as Record<string, unknown>;
const metadata = value.workspace as Record<string, unknown> | undefined;
if (!metadata || metadata.schema_version !== 3) {
throw new Error("Workspace migration requires schema version 3");
}
metadata.schema_version = 4;
delete value.semantic_index;
delete value.llm_policy;
return serializeWorkspaceYaml(validateWorkspaceDescriptor(value));
}
export function validateWorkspaceDescriptor(workspace: unknown): WorkspaceDescriptor {
return WorkspaceDescriptorSchema.parse(workspace) as WorkspaceDescriptor;
}
@@ -439,11 +401,11 @@ export function isCanonicalWorkspace(workspace: unknown): workspace is Canonical
return WorkspaceDescriptorSchema.safeParse(workspace).success;
}
export function isOperationalWorkspace(workspace: unknown): workspace is WorkspaceV3 {
export function isOperationalWorkspace(workspace: unknown): workspace is WorkspaceV4 {
return WorkspaceDescriptorSchema.safeParse(workspace).success;
}
export function validateOperationalWorkspace(workspace: unknown): WorkspaceV3 {
export function validateOperationalWorkspace(workspace: unknown): WorkspaceV4 {
return validateWorkspaceDescriptor(workspace);
}
+1 -1
View File
@@ -19,4 +19,4 @@ export type WorkspaceErrorCode =
| "workspace_stale" | "git_unavailable" | "git_auth_failed" | "git_non_fast_forward"
| "connector_unavailable" | "semantic_index_incompatible";
export type { WorkspaceV3 } from "./schema.js";
export type { WorkspaceV4 } from "./schema.js";
+3 -16
View File
@@ -5,14 +5,12 @@ import { createLocalAuthFixture } from "./auth-test-fixtures.js";
function fakeService(): PiManagementService {
return {
status: vi.fn(async () => ({ ready: true })),
options: vi.fn(async () => ({ providers: [], models: [], reasoning: [], checkedAt: "2026-08-17T00:00:00.000Z" })),
configure: vi.fn(async (value) => ({ ...value, updatedAt: "2026-08-17T00:00:00.000Z" })),
test: vi.fn(async () => ({ ready: true, checkedAt: "2026-08-17T00:00:00.000Z" })),
logs: vi.fn(async () => ({ lines: [] })),
};
}
test("a local HTTPS cookie session authorizes Pi writes through an untrusted internal HTTP hop", async () => {
test("a local HTTPS cookie session authorizes the Pi smoke check through an untrusted internal HTTP hop", async () => {
const service = fakeService();
const fixture = await createLocalAuthFixture(
{ piManagement: service },
@@ -25,31 +23,21 @@ test("a local HTTPS cookie session authorizes Pi writes through an untrusted int
expect(fixture.publicUrl).toBe("HTTPS://thothii.example.test");
const proxyHeaders = fixture.sessionHeaders({ host: "127.0.0.1:8080" });
const configured = await fixture.app.inject({
method: "PUT",
url: "/pi-management/config",
headers: proxyHeaders,
payload: { provider: "zai", model: "glm-5.2", reasoning: "high" },
});
const smoke = await fixture.app.inject({
method: "POST",
url: "/pi-management/test",
headers: proxyHeaders,
});
expect(configured.statusCode).toBe(200);
expect(smoke.statusCode).toBe(200);
expect(service.configure).toHaveBeenCalledTimes(1);
expect(service.test).toHaveBeenCalledTimes(1);
fixture.resetDownstreamHits();
vi.mocked(service.configure).mockClear();
vi.mocked(service.test).mockClear();
const wrongOrigin = await fixture.app.inject({
method: "PUT",
url: "/pi-management/config",
method: "POST",
url: "/pi-management/test",
headers: fixture.sessionHeaders({ host: "127.0.0.1:8080", origin: "https://evil.example" }),
payload: { provider: "zai", model: "glm-5.2", reasoning: "high" },
});
const wrongCsrf = await fixture.app.inject({
method: "POST",
@@ -62,7 +50,6 @@ test("a local HTTPS cookie session authorizes Pi writes through an untrusted int
expect(response.json()).toEqual({ code: "csrf_failed", error: "Request origin validation failed" });
}
expect(fixture.downstreamHits()).toBe(0);
expect(service.configure).not.toHaveBeenCalled();
expect(service.test).not.toHaveBeenCalled();
} finally {
await fixture.close();
@@ -19,16 +19,11 @@ afterEach(() => {
});
const workspace: WorkspaceDescriptor = {
workspace: { schema_version: 3, id: "psd-clinical", name: "Policlinico San Donato", language: "it" },
workspace: { schema_version: 4, id: "psd-clinical", name: "Policlinico San Donato", language: "it" },
dwh: {
engine: "postgres", database: "warehouse", schema: "datawarehouse", port: 5432,
supported_transports: ["postgres_direct", "rest_api"],
},
semantic_index: {
vector_store: { engine: "qdrant", collection: "psd", dimensions: 1024, distance: "cosine" },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
llm_policy: { allowed: ["zai/glm-5.2"] },
diagnostics: { dwh_rest: { method: "GET", path: "/health", auth: "bearer", response: { database: "database", schema: "schema" } } },
};
const revision: WorkspaceRevision = { id: "psd-clinical", commit: "a".repeat(40), blob: "b".repeat(40), snapshotPath: "/tmp/psd.yaml" };
@@ -24,7 +24,7 @@ import type { WorkspaceDescriptor } from "../src/workspaces/schema.js";
const workspace: WorkspaceDescriptor = {
workspace: {
schema_version: 3,
schema_version: 4,
id: "psd-clinical",
name: "Policlinico San Donato",
language: "it",
@@ -36,11 +36,6 @@ const workspace: WorkspaceDescriptor = {
port: 5432,
supported_transports: ["postgres_direct"],
},
semantic_index: {
vector_store: { engine: "qdrant", collection: "psd", dimensions: 1024, distance: "cosine" },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
llm_policy: { allowed: ["zai/glm-5.2"] },
};
const revision: WorkspaceRevision = {
id: "psd-clinical",
@@ -49,7 +44,7 @@ const revision: WorkspaceRevision = {
snapshotPath: "/tmp/psd.yaml",
};
const configuredModel: ResolvedMetadataGenerationModel = {
id: "openai-mini",
id: "openai/gpt-4.1-mini",
provider: "openai",
model: "gpt-4.1-mini",
apiKeyEnv: "OPENAI_API_KEY",
@@ -705,7 +700,7 @@ test("generates one selected Catalog Column from a single JSON code fence", asyn
expect(completionRequest.messages[0]?.content).toContain('{"results":[');
expect(completionRequest.messages[1]?.content).toContain(`"targetId":"${column.id}"`);
expect(completionRequest.messages[1]?.content).not.toMatch(/source rows|samples|example values/i);
expect(start.body).not.toMatch(/test-provider-secret|gpt-4\.1|openai\/gpt|Catalog metadata/);
expect(start.body).not.toMatch(/test-provider-secret|Catalog metadata/);
resolveCompletion(`\`\`\`json\n${JSON.stringify({
results: [{
@@ -2909,7 +2904,7 @@ test("validates selected targets and resolves every requested target before laun
const unknownModel = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/description-generation-runs`,
payload: { modelId: "unknown-model", scope: "selected_columns", targetIds: [column.id] },
payload: { modelId: "openai/unknown-model", scope: "selected_columns", targetIds: [column.id] },
});
expect(unknownModel.statusCode).toBe(409);
expect(unknownModel.json().code).toBe("metadata_generation_model_unavailable");
@@ -14,6 +14,7 @@ import { up as upDescriptionGeneration } from "../src/catalog/migrations/005_des
import { up as upSensitiveDataFlag } from "../src/catalog/migrations/006_sensitive_data_flag.js";
import { up as upSensitiveSuggestionRuns } from "../src/catalog/migrations/007_sensitive_data_suggestion_runs.js";
import { up as upAiTokenUsage } from "../src/catalog/migrations/009_ai_token_usage.js";
import { up as upCanonicalModelIds } from "../src/catalog/migrations/010_canonical_model_ids.js";
import { KyselyCatalogRepository, type CatalogDatabase } from "../src/catalog/repository.js";
import { loadConfig } from "../src/config.js";
import type { WorkspaceRegistry } from "../src/workspaces/registry.js";
@@ -50,6 +51,7 @@ test.skipIf(!dockerAvailable)("Fastify persists Description Generation success a
await upDescriptionGeneration(db);
await upSensitiveSuggestionRuns(db);
await upAiTokenUsage(db);
await upCanonicalModelIds(db);
const repository = new KyselyCatalogRepository(db);
const database = await repository.create({
workspaceId: "psd-clinical",
@@ -158,9 +160,9 @@ test.skipIf(!dockerAvailable)("Fastify persists Description Generation success a
}),
};
const models: MetadataGenerationModels = {
catalog: () => ({ models: [{ id: "openai-mini", label: "OpenAI Mini" }], default: "openai-mini" }),
catalog: () => ({ models: [{ id: "openai/gpt-4.1-mini", label: "OpenAI Mini" }], default: "openai/gpt-4.1-mini" }),
resolve: () => ({
id: "openai-mini",
id: "openai/gpt-4.1-mini",
provider: "openai",
model: "gpt-4.1-mini",
apiKeyEnv: "OPENAI_API_KEY",
@@ -205,12 +207,12 @@ test.skipIf(!dockerAvailable)("Fastify persists Description Generation success a
method: "POST",
url: `/catalog/databases/${database.id}/description-generation-runs`,
payload: {
modelId: "openai-mini",
modelId: "openai/gpt-4.1-mini",
scope: "selected_columns",
targetIds: [status.id, birthDate.id],
},
});
expect(successfulStart.statusCode).toBe(202);
expect(successfulStart.statusCode, successfulStart.body).toBe(202);
expect(await terminalRun(app, successfulStart.json().id)).toMatchObject({
status: "completed",
total: 2,
@@ -233,7 +235,7 @@ test.skipIf(!dockerAvailable)("Fastify persists Description Generation success a
const tableStart = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/description-generation-runs`,
payload: { modelId: "openai-mini", scope: "selected_tables", targetIds: [table.id] },
payload: { modelId: "openai/gpt-4.1-mini", scope: "selected_tables", targetIds: [table.id] },
});
expect(tableStart.statusCode).toBe(202);
expect(await terminalRun(app, tableStart.json().id)).toMatchObject({
@@ -253,7 +255,7 @@ test.skipIf(!dockerAvailable)("Fastify persists Description Generation success a
const failedStart = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/description-generation-runs`,
payload: { modelId: "openai-mini", scope: "selected_columns", targetIds: [status.id] },
payload: { modelId: "openai/gpt-4.1-mini", scope: "selected_columns", targetIds: [status.id] },
});
expect(failedStart.statusCode).toBe(202);
const failedRun = await terminalRun(app, failedStart.json().id);
@@ -283,7 +285,7 @@ test.skipIf(!dockerAvailable)("Fastify persists Description Generation success a
const allStart = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/description-generation-runs`,
payload: { modelId: "openai-mini", scope: "all" },
payload: { modelId: "openai/gpt-4.1-mini", scope: "all" },
});
expect(allStart.statusCode).toBe(202);
const allRun = await terminalRun(app, allStart.json().id);
@@ -327,7 +329,7 @@ test.skipIf(!dockerAvailable)("Fastify persists Description Generation success a
const missingStart = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/description-generation-runs`,
payload: { modelId: "openai-mini", scope: "missing" },
payload: { modelId: "openai/gpt-4.1-mini", scope: "missing" },
});
expect(missingStart.statusCode).toBe(202);
expect(await terminalRun(app, missingStart.json().id)).toMatchObject({
@@ -14,6 +14,7 @@ import { up as upSensitiveDataFlag } from "../src/catalog/migrations/006_sensiti
import { up as upSensitiveSuggestionRuns } from "../src/catalog/migrations/007_sensitive_data_suggestion_runs.js";
import { up as upLogicalRelationships } from "../src/catalog/migrations/008_catalog_logical_relationships.js";
import { up as upAiTokenUsage } from "../src/catalog/migrations/009_ai_token_usage.js";
import { up as upCanonicalModelIds } from "../src/catalog/migrations/010_canonical_model_ids.js";
const dockerAvailable = spawnSync("docker", ["info"], { stdio: "ignore" }).status === 0;
@@ -32,6 +33,7 @@ test.skipIf(!dockerAvailable)("PostgreSQL migration enforces one database per wo
await upDescriptionGeneration(db);
await upSensitiveSuggestionRuns(db);
await upAiTokenUsage(db);
await upCanonicalModelIds(db);
await sql`CREATE ROLE thothii_catalog_runtime`.execute(db);
await upRuntimeSequencePrivileges(db);
const sequencePrivilege = await sql<{ allowed: boolean }>`
@@ -391,6 +393,7 @@ test.skipIf(!dockerAvailable)("PostgreSQL repository persists description and se
await upDescriptionGeneration(db);
await upSensitiveSuggestionRuns(db);
await upAiTokenUsage(db);
await upCanonicalModelIds(db);
const repository = new KyselyCatalogRepository(db);
const firstDatabase = await repository.create({
workspaceId: "generation-one",
@@ -428,14 +431,14 @@ test.skipIf(!dockerAvailable)("PostgreSQL repository persists description and se
const run = await repository.createDescriptionGenerationRun(
firstDatabase.id,
"selected_columns",
"openai-mini",
"openai/gpt-4.1-mini",
"it",
1,
);
expect(run).toMatchObject({
databaseId: firstDatabase.id,
scope: "selected_columns",
modelId: "openai-mini",
modelId: "openai/gpt-4.1-mini",
language: "it",
status: "queued",
total: 1,
@@ -450,7 +453,7 @@ test.skipIf(!dockerAvailable)("PostgreSQL repository persists description and se
await expect(repository.createDescriptionGenerationRun(
secondDatabase.id,
"selected_columns",
"openai-mini",
"openai/gpt-4.1-mini",
"en",
1,
)).rejects.toThrow("A description generation run is already active");
@@ -461,7 +464,7 @@ test.skipIf(!dockerAvailable)("PostgreSQL repository persists description and se
await expect(repository.createDescriptionGenerationRun(
secondDatabase.id,
"selected_columns",
"openai-mini",
"openai/gpt-4.1-mini",
"en",
1,
)).rejects.toThrow("A description generation run is already active");
@@ -505,7 +508,7 @@ test.skipIf(!dockerAvailable)("PostgreSQL repository persists description and se
const next = await repository.createDescriptionGenerationRun(
secondDatabase.id,
"missing",
"openai-mini",
"openai/gpt-4.1-mini",
"en",
1,
);
@@ -533,7 +536,7 @@ test.skipIf(!dockerAvailable)("PostgreSQL repository persists description and se
const allRun = await repository.createDescriptionGenerationRun(
firstDatabase.id,
"all",
"openai-mini",
"openai/gpt-4.1-mini",
"it",
2,
);
@@ -562,7 +565,7 @@ test.skipIf(!dockerAvailable)("PostgreSQL repository persists description and se
const suggestionRun = await repository.createSensitiveDataSuggestionRun(
firstDatabase.id,
"selected_columns",
"openai-mini",
"openai/gpt-4.1-mini",
);
expect(suggestionRun).toMatchObject({
databaseId: firstDatabase.id,
@@ -604,7 +607,7 @@ test.skipIf(!dockerAvailable)("PostgreSQL repository persists description and se
const interruptedSuggestionRun = await repository.createSensitiveDataSuggestionRun(
secondDatabase.id,
"all",
"openai-mini",
"openai/gpt-4.1-mini",
);
expect(await repository.interruptActiveSensitiveDataSuggestionRuns(
"Sensitive-field suggestion generation was interrupted by backend restart.",
+1 -6
View File
@@ -16,13 +16,8 @@ const roots: string[] = [];
afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); });
const workspace: WorkspaceDescriptor = {
workspace: { schema_version: 3, id: "psd-clinical", name: "Policlinico San Donato", language: "it" },
workspace: { schema_version: 4, id: "psd-clinical", name: "Policlinico San Donato", language: "it" },
dwh: { engine: "postgres", database: "warehouse", schema: "datawarehouse", port: 5432, supported_transports: ["postgres_direct"] },
semantic_index: {
vector_store: { engine: "qdrant", collection: "psd", dimensions: 1024, distance: "cosine" },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
llm_policy: { allowed: ["zai/glm-5.2"] },
};
const revision: WorkspaceRevision = { id: "psd-clinical", commit: "a".repeat(40), blob: "b".repeat(40), snapshotPath: "/tmp/psd.yaml" };
+1 -6
View File
@@ -13,16 +13,11 @@ const roots: string[] = [];
afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); });
const workspace: WorkspaceDescriptor = {
workspace: { schema_version: 3, id: "psd-clinical", name: "Policlinico San Donato", language: "it" },
workspace: { schema_version: 4, id: "psd-clinical", name: "Policlinico San Donato", language: "it" },
dwh: {
engine: "postgres", database: "warehouse", schema: "datawarehouse", port: 5432,
supported_transports: ["postgres_direct", "rest_api"],
},
semantic_index: {
vector_store: { engine: "qdrant", collection: "psd", dimensions: 1024, distance: "cosine" },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
llm_policy: { allowed: ["zai/glm-5.2"] },
diagnostics: { dwh_rest: { method: "GET", path: "/health", auth: "bearer", response: { database: "database", schema: "schema" } } },
};
const revision: WorkspaceRevision = {
+17
View File
@@ -71,6 +71,7 @@ test("loadConfig keeps local development defaults", () => {
workspaceSecretRuntimeRoot: "/tmp/thothii-workspace-secrets",
internalQdrantUrl: "http://qdrant:6333",
internalEmbeddingUrl: "http://embedding:11434",
internalEmbeddingId: "ollama/qwen3-embedding:0.6b",
internalEmbeddingModel: "qwen3-embedding:0.6b",
internalEmbeddingDimensions: 1024,
authMode: "none",
@@ -198,6 +199,22 @@ test("loadConfig accepts only the allowed internal semantic runtime hosts", () =
.toThrow(/internal.*embedding|invalid/i);
});
test("loadConfig derives the embedding runtime model from its canonical catalog identity", () => {
expect(loadConfig({
THT_INTERNAL_EMBEDDING_ID: "ollama/nomic-embed-text",
THT_INTERNAL_EMBEDDING_MODEL: "nomic-embed-text",
})).toMatchObject({
internalEmbeddingId: "ollama/nomic-embed-text",
internalEmbeddingModel: "nomic-embed-text",
});
expect(() => loadConfig({
THT_INTERNAL_EMBEDDING_ID: "ollama/nomic-embed-text",
THT_INTERNAL_EMBEDDING_MODEL: "different-model",
})).toThrow("does not match its canonical identity");
expect(() => loadConfig({ THT_INTERNAL_EMBEDDING_ID: "not-canonical" }))
.toThrow("embedding identity configuration is invalid");
});
test("loadConfig enables the legacy workspace request only through explicit local mode", () => {
expect(loadConfig({ THT_LEGACY_WORKSPACE_MODE: "local" }).legacyWorkspaceMode).toBe(true);
+6 -2
View File
@@ -11,6 +11,7 @@ import {
const semanticRuntime = {
internalQdrantUrl: "http://qdrant:6333",
internalEmbeddingUrl: "http://embedding:11434",
internalEmbeddingId: "ollama/qwen3-embedding:0.6b",
internalEmbeddingModel: "qwen3-embedding:0.6b",
internalEmbeddingDimensions: 1024,
};
@@ -102,11 +103,11 @@ function restRendered(): Record<string, unknown> {
}
const directCanonical =
`{"schemaVersion":1,"dwh":{` +
`{"schemaVersion":2,"dwh":{` +
`"engine":"postgres","database":"postgres","schema":"datawarehouse",` +
`"transport":"postgres_direct","host":"dwh.internal","port":5432,"user":"thoth_reader"},` +
`"vector":{"collection":"psd-clinical","dimensions":1024,"distance":"cosine"},` +
`"embedding":{"model":"qwen3-embedding:0.6b","dimensions":1024},` +
`"embedding":{"id":"ollama/qwen3-embedding:0.6b","model":"qwen3-embedding:0.6b","dimensions":1024},` +
`"roots":{"artifacts":"/data/sessions/psd-clinical/artifacts",` +
`"indexes":"/data/sessions/psd-clinical/indexes"}}`;
@@ -192,6 +193,9 @@ test("DWH-affecting changes alter the effective config identity", () => {
const changedCollection = { ...base, resources: { ...base.resources, vector: { ...(base.resources as Record<string, any>).vector, collection: "other" } } };
expect(effectiveConfigIdentity("psd-clinical", changedCollection)).not.toBe(identityBefore);
const changedEmbeddingIdentity = { ...base, resources: { ...base.resources, embeddings: { ...(base.resources as Record<string, any>).embeddings, model: "other-embedding" } } };
expect(effectiveConfigIdentity("psd-clinical", changedEmbeddingIdentity)).not.toBe(identityBefore);
const changedTransport = restRendered();
expect(effectiveConfigIdentity("psd-clinical", changedTransport)).not.toBe(identityBefore);
});
+85 -247
View File
@@ -1,15 +1,12 @@
import { chmodSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { afterEach, expect, test, vi } from "vitest";
import { buildApp } from "../src/app.js";
import { MemoryCatalogRepository } from "../src/catalog/memory-repository.js";
import { afterEach, expect, test } from "vitest";
import {
loadMetadataGenerationModels,
MetadataGenerationModelUnavailableError,
} from "../src/catalog/metadata-generation-models.js";
import { loadConfig } from "../src/config.js";
import type { WorkspaceRegistry } from "../src/workspaces/registry.js";
import { loadRuntimeModelCatalog, splitCanonicalModelId } from "../src/models/runtime-model-catalog.js";
const roots: string[] = [];
@@ -17,260 +14,101 @@ afterEach(() => {
for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true });
});
function metadataConfiguration(
metadataGeneration: string,
secrets = "OPENAI_API_KEY=raw-provider-secret\n",
) {
const root = mkdtempSync(join(tmpdir(), "thothii-metadata-models-"));
function runtimeCatalog(overrides: Record<string, unknown> = {}, secrets = "OPENAI_API_KEY=raw-provider-secret\n") {
const root = mkdtempSync(join(tmpdir(), "thothii-runtime-models-"));
roots.push(root);
const installationFile = join(root, "thothii-installation.yaml");
const catalogFile = join(root, "catalog.json");
const secretsFile = join(root, "thothii.secrets");
writeFileSync(installationFile, metadataGeneration, { mode: 0o600 });
writeFileSync(secretsFile, secrets, { mode: 0o600 });
chmodSync(installationFile, 0o600);
chmodSync(secretsFile, 0o600);
return { installationFile, secretsFile };
}
function appFor(installationFile: string, secretsFile: string) {
const config = loadConfig({
NODE_ENV: "test",
THT_HARNESS_DIR: "/missing",
THT_INSTALLATION_CONFIG_FILE: installationFile,
THT_SECRETS_FILE: secretsFile,
PI_PROVIDER: "unrelated-pi-provider",
PI_MODEL: "unrelated-pi-model",
});
return buildApp(config, {
thtRunner: {} as never,
workspaceRegistry: { list: vi.fn(async () => []) } as unknown as WorkspaceRegistry,
workspaceDiagnoser: vi.fn(),
catalogRepository: new MemoryCatalogRepository(),
});
}
test("exposes only safe metadata-generation choices and their configured default", async () => {
const { installationFile, secretsFile } = metadataConfiguration(`metadataGeneration:
default: openai-mini
models:
- id: openai-mini
label: OpenAI Mini
litellm:
provider: openai
model: gpt-4.1-mini
endpoint:
baseUrl: https://api.openai.example/v1
apiVersion: "2026-08-01"
apiKeyEnv: OPENAI_API_KEY
`);
const app = appFor(installationFile, secretsFile);
const response = await app.inject({ method: "GET", url: "/catalog/metadata-generation/models" });
expect(response.statusCode).toBe(200);
expect(response.json()).toEqual({
models: [{ id: "openai-mini", label: "OpenAI Mini" }],
default: "openai-mini",
});
expect(response.body).not.toMatch(/openai\/gpt|gpt-4\.1|api\.openai|OPENAI_API_KEY|raw-provider-secret/);
await app.close();
});
test("rejects an unprotected installation descriptor", () => {
const { installationFile, secretsFile } = metadataConfiguration(`metadataGeneration:
default: openai-mini
models:
- id: openai-mini
label: OpenAI Mini
litellm: {provider: openai, model: gpt-4.1-mini}
apiKeyEnv: OPENAI_API_KEY
`);
chmodSync(installationFile, 0o644);
expect(() => loadMetadataGenerationModels({ installationFile, secretsFile }))
.toThrow("metadata-generation installation is unavailable");
});
test("returns an empty safe catalog when no metadata-generation model is configured", async () => {
const { installationFile, secretsFile } = metadataConfiguration("profile: local\n");
const app = appFor(installationFile, secretsFile);
const response = await app.inject({ method: "GET", url: "/catalog/metadata-generation/models" });
expect(response.statusCode).toBe(200);
expect(response.json()).toEqual({ models: [], default: null });
await app.close();
});
test("resolves only a configured selection for the later generation boundary", () => {
const { installationFile, secretsFile } = metadataConfiguration(`metadataGeneration:
default: openai-mini
models:
- id: openai-mini
label: OpenAI Mini
litellm:
provider: openai
model: gpt-4.1-mini
endpoint: {baseUrl: https://api.openai.example/v1, apiVersion: "2026-08-01"}
apiKeyEnv: OPENAI_API_KEY
`);
const models = loadMetadataGenerationModels({ installationFile, secretsFile });
expect(models.resolve("openai-mini")).toEqual({
id: "openai-mini",
provider: "openai",
model: "gpt-4.1-mini",
endpoint: { baseUrl: "https://api.openai.example/v1", apiVersion: "2026-08-01" },
apiKeyEnv: "OPENAI_API_KEY",
apiKey: "raw-provider-secret",
});
expect(() => models.resolve("unknown-model")).toThrow(MetadataGenerationModelUnavailableError);
});
test("loads DeepSeek models, GLM, and an explicit keyless Qwen endpoint from installation setup", () => {
const { installationFile, secretsFile } = metadataConfiguration(`metadataGeneration:
default: glm-53
models:
- id: deepseek-v4-pro
label: DeepSeek V4 Pro
litellm: {provider: deepseek, model: deepseek-v4-pro}
apiKeyEnv: DEEPSEEK_API_KEY
- id: deepseek-v4-flash
label: DeepSeek V4 Flash
litellm: {provider: deepseek, model: deepseek-v4-flash}
apiKeyEnv: DEEPSEEK_API_KEY
- id: glm-53
label: GLM 5.3
litellm:
provider: openai
model: glm-5.3
endpoint: {baseUrl: https://api.z.ai/api/coding/paas/v4}
apiKeyEnv: ZAI_API_KEY
- id: qwen-36
label: Qwen 3.6
litellm:
provider: openai
model: qwen3.6-35b-a3b
disableThinking: true
endpoint: {baseUrl: https://models.internal.example/v1}
`, "DEEPSEEK_API_KEY=deepseek-secret\nZAI_API_KEY=zai-secret\n");
const models = loadMetadataGenerationModels({ installationFile, secretsFile });
expect(models.catalog()).toEqual({
const catalog = {
schemaVersion: 1,
defaultSession: "zai/glm-5.3",
defaultMetadataGeneration: "zai/glm-5.3",
embedding: { id: "ollama/qwen3-embedding:0.6b", dimensions: 1024 },
models: [
{ id: "deepseek-v4-pro", label: "DeepSeek V4 Pro" },
{ id: "deepseek-v4-flash", label: "DeepSeek V4 Flash" },
{ id: "glm-53", label: "GLM 5.3" },
{ id: "qwen-36", label: "Qwen 3.6" },
{
id: "zai/glm-5.3", provider: "zai", model: "glm-5.3", label: "GLM 5.3",
upstreamModel: "glm-5.3", endpoint: { baseUrl: "https://api.z.ai/v1" },
authentication: { mode: "secret_env", apiKeyEnv: "OPENAI_API_KEY" },
sessionAdapter: { mode: "openai_compatible" },
metadataAdapter: { litellmProvider: "openai" },
session: { reasoning: true, contextWindow: 200000, maxTokens: 131072 },
metadataGeneration: { disableThinking: false },
},
{
id: "deepseek/deepseek-v4-pro", provider: "deepseek", model: "deepseek-v4-pro",
label: "DeepSeek V4 Pro", upstreamModel: "deepseek-v4-pro",
authentication: { mode: "pi_auth" }, sessionAdapter: { mode: "pi_builtin" },
session: { reasoning: false },
},
],
default: "glm-53",
...overrides,
};
writeFileSync(catalogFile, JSON.stringify(catalog), { mode: 0o600 });
writeFileSync(secretsFile, secrets, { mode: 0o600 });
chmodSync(catalogFile, 0o600);
chmodSync(secretsFile, 0o600);
return { catalogFile, secretsFile };
}
test("loads session default and safe metadata choices from the normalized runtime catalog", () => {
const { catalogFile, secretsFile } = runtimeCatalog();
const runtime = loadRuntimeModelCatalog(catalogFile);
const metadata = loadMetadataGenerationModels({ catalogFile, secretsFile });
expect(runtime.defaultSession).toBe("zai/glm-5.3");
expect(runtime.hasSession("deepseek/deepseek-v4-pro")).toBe(true);
expect(metadata.catalog()).toEqual({
models: [{ id: "zai/glm-5.3", label: "GLM 5.3" }],
default: "zai/glm-5.3",
});
expect(models.resolve("deepseek-v4-pro")).toMatchObject({
apiKeyEnv: "DEEPSEEK_API_KEY",
apiKey: "deepseek-secret",
});
expect(models.resolve("qwen-36")).toEqual({
id: "qwen-36",
provider: "openai",
model: "qwen3.6-35b-a3b",
disableThinking: true,
endpoint: { baseUrl: "https://models.internal.example/v1" },
expect(metadata.resolve("zai/glm-5.3")).toEqual({
id: "zai/glm-5.3", provider: "openai", model: "glm-5.3",
endpoint: { baseUrl: "https://api.z.ai/v1" },
apiKeyEnv: "OPENAI_API_KEY", apiKey: "raw-provider-secret",
});
expect(() => metadata.resolve("zai/missing")).toThrow(MetadataGenerationModelUnavailableError);
});
test("loads an explicit keyless endpoint without a secret bundle", () => {
const { installationFile } = metadataConfiguration(`metadataGeneration:
default: qwen-36
models:
- id: qwen-36
label: Qwen 3.6
litellm:
provider: openai
model: qwen3.6-35b-a3b
disableThinking: true
endpoint: {baseUrl: https://models.internal.example/v1}
`);
test("returns empty catalogs when no runtime projection is configured", () => {
expect(loadRuntimeModelCatalog().defaultSession).toBeNull();
expect(loadMetadataGenerationModels({}).catalog()).toEqual({ models: [], default: null });
});
expect(loadMetadataGenerationModels({ installationFile }).resolve("qwen-36")).toEqual({
id: "qwen-36",
provider: "openai",
model: "qwen3.6-35b-a3b",
disableThinking: true,
endpoint: { baseUrl: "https://models.internal.example/v1" },
test("rejects a drifted default and an unprotected projection", () => {
const drifted = runtimeCatalog({ defaultSession: "zai/missing" });
expect(() => loadRuntimeModelCatalog(drifted.catalogFile)).toThrow("session default is invalid");
const unprotected = runtimeCatalog();
chmodSync(unprotected.catalogFile, 0o666);
expect(() => loadRuntimeModelCatalog(unprotected.catalogFile)).toThrow("runtime model catalog is unavailable");
});
test("rejects authentication semantics that cannot come from the installation catalog", () => {
const invalid = runtimeCatalog({
defaultMetadataGeneration: undefined,
models: [{
id: "zai/glm-5.3",
provider: "zai",
model: "glm-5.3",
label: "GLM 5.3",
upstreamModel: "glm-5.3",
authentication: { mode: "secret_env" },
sessionAdapter: { mode: "pi_builtin" },
session: { reasoning: true },
}],
});
expect(() => loadRuntimeModelCatalog(invalid.catalogFile)).toThrow("runtime model catalog is invalid");
});
test.each([
["invalid YAML", "metadataGeneration: [\n", "OPENAI_API_KEY=secret\n", /invalid YAML/],
["duplicate ids", `metadataGeneration:
default: openai-mini
models:
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY}
- {id: openai-mini, label: Two, litellm: {provider: openai, model: gpt-4.1}, apiKeyEnv: OPENAI_API_KEY}
`, "OPENAI_API_KEY=secret\n", /model id "openai-mini" is duplicated/],
["missing default", `metadataGeneration:
models:
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY}
`, "OPENAI_API_KEY=secret\n", /default is required/],
["unknown default", `metadataGeneration:
default: absent
models:
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY}
`, "OPENAI_API_KEY=secret\n", /default "absent" is not configured/],
["malformed settings", `metadataGeneration:
default: openai-mini
models:
- {id: openai-mini, label: One, litellm: {provider: "open ai", model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY}
`, "OPENAI_API_KEY=secret\n", /configuration is invalid/],
["malformed endpoint", `metadataGeneration:
default: openai-mini
models:
- id: openai-mini
label: One
litellm: {provider: openai, model: gpt-4.1-mini, endpoint: {baseUrl: not-a-url}}
apiKeyEnv: OPENAI_API_KEY
`, "OPENAI_API_KEY=secret\n", /configuration is invalid/],
["keyless hosted model without endpoint", `metadataGeneration:
default: openai-mini
models:
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}}
`, "", /configuration is invalid/],
["disable thinking without endpoint", `metadataGeneration:
default: openai-mini
models:
- id: openai-mini
label: One
litellm: {provider: openai, model: gpt-4.1-mini, disableThinking: true}
apiKeyEnv: OPENAI_API_KEY
`, "OPENAI_API_KEY=secret\n", /configuration is invalid/],
["unallowed secret reference", `metadataGeneration:
default: openai-mini
models:
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: THT_DWH_API_KEY}
`, "THT_DWH_API_KEY=secret\n", /configuration is invalid/],
["missing referenced secret", `metadataGeneration:
default: openai-mini
models:
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY}
`, "THT_DWH_API_KEY=secret\n", /secret "OPENAI_API_KEY" is missing/],
["unusable referenced secret", `metadataGeneration:
default: openai-mini
models:
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY}
`, "OPENAI_API_KEY=secret with whitespace\n", /secret "OPENAI_API_KEY" is unusable/],
] as const)("rejects %s metadata-generation configuration", (_name, yaml, secrets, expected) => {
const { installationFile, secretsFile } = metadataConfiguration(yaml, secrets);
expect(() => loadMetadataGenerationModels({ installationFile, secretsFile })).toThrow(expected);
test("fails closed for missing or unusable provider secrets", () => {
const missing = runtimeCatalog({}, "THT_DWH_API_KEY=other\n");
expect(() => loadMetadataGenerationModels(missing)).toThrow('secret "OPENAI_API_KEY" is missing');
const unusable = runtimeCatalog({}, "OPENAI_API_KEY=contains whitespace\n");
expect(() => loadMetadataGenerationModels(unusable)).toThrow('secret "OPENAI_API_KEY" is unusable');
});
test("rejects a missing secret-bundle declaration for configured models", () => {
const { installationFile } = metadataConfiguration(`metadataGeneration:
default: openai-mini
models:
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY}
`);
expect(() => loadMetadataGenerationModels({ installationFile }))
.toThrow("metadata-generation keyed models require THT_SECRETS_FILE");
test("splits canonical session identities without provider aliases", () => {
expect(splitCanonicalModelId("zai/glm-5.3")).toEqual({ provider: "zai", model: "glm-5.3" });
expect(() => splitCanonicalModelId("glm-5.3")).toThrow("model identity is invalid");
});
+18 -108
View File
@@ -1,13 +1,13 @@
import { mkdtempSync, readFileSync, readdirSync, rmSync } from "node:fs";
import { mkdtempSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { expect, test, vi } from "vitest";
import { loadConfig } from "../src/config.js";
import {
PiManagementError,
createPiManagement,
type PiExecFile,
} from "../src/pi/management.js";
import type { RuntimeModelCatalog } from "../src/models/runtime-model-catalog.js";
function configFor(settingsFile = join(mkdtempSync(join(tmpdir(), "tht-pi-management-")), "settings.json")) {
return loadConfig({
@@ -18,10 +18,14 @@ function configFor(settingsFile = join(mkdtempSync(join(tmpdir(), "tht-pi-manage
});
}
const supportedModels = [
{ provider: "zai", id: "glm-5.2", name: "GLM 5.2", reasoning: true },
{ provider: "deepseek", id: "deepseek-v4", name: "DeepSeek V4", reasoning: true },
];
const modelCatalog: RuntimeModelCatalog = {
defaultSession: "zai/glm-5.2",
defaultMetadataGeneration: null,
embedding: { id: "ollama/qwen3-embedding:0.6b", dimensions: 1024 },
sessionModels: () => [],
metadataModels: () => [],
hasSession: (id) => id === "zai/glm-5.2",
};
function successfulExec(calls: Array<{ command: string; args: string[]; timeout: number }>): PiExecFile {
return async (command, args, options) => {
@@ -36,7 +40,7 @@ test("status parses only a Pi version from a fixed execFile argument array", asy
const calls: Array<{ command: string; args: string[]; timeout: number }> = [];
const service = createPiManagement(configFor(), {
execute: successfulExec(calls),
listModels: async () => supportedModels,
modelCatalog,
readSettings: () => ({ provider: "zai", model: "glm-5.2", thinking: "medium" }),
credentialStatus: () => "missing",
now: () => new Date("2026-08-05T10:00:00.000Z"),
@@ -63,7 +67,7 @@ test.each(["present", "missing"] as const)(
const checkedProviders: Array<string | undefined> = [];
const service = createPiManagement(configFor(), {
execute: successfulExec([]),
listModels: async () => supportedModels,
modelCatalog,
readSettings: () => ({ provider: "zai", model: "glm-5.2", thinking: "medium" }),
credentialStatus: (provider) => {
checkedProviders.push(provider);
@@ -85,100 +89,6 @@ test.each(["present", "missing"] as const)(
},
);
// Catches an options response that leaks provider metadata or lets callers choose model IDs that
// Pi did not explicitly enable for this installation.
test("options expose only closed provider, model, and reasoning choices", async () => {
const service = createPiManagement(configFor(), {
execute: successfulExec([]),
listModels: async () => supportedModels,
now: () => new Date("2026-08-05T10:00:00.000Z"),
});
await expect(service.options()).resolves.toEqual({
providers: ["zai", "deepseek"],
models: [
{ provider: "zai", id: "glm-5.2" },
{ provider: "deepseek", id: "deepseek-v4" },
],
reasoning: ["low", "medium", "high"],
checkedAt: "2026-08-05T10:00:00.000Z",
});
});
// Catches raw managed models.json validation details being collapsed into an ambiguous model-list
// failure or escaping through the Pi Management options API.
test("options report invalid managed model configuration with a stable sanitized error", async () => {
const service = createPiManagement(configFor(), {
execute: successfulExec([]),
listModels: async () => {
throw Object.assign(
new Error("!sensitive-command /private/models.json raw-secret"),
{ code: "PI_MANAGED_CONFIG_INVALID" },
);
},
});
let caught: unknown;
try {
await service.options();
} catch (error) {
caught = error;
}
expect(caught).toMatchObject<PiManagementError>({
code: "pi_management_unavailable",
message: "Pi provider/model configuration is invalid",
});
expect(String(caught)).not.toMatch(/sensitive|private|models\.json|secret/i);
});
// Catches configuration writes that accept whitespace, unknown choices, or extra free-form fields
// before reaching the durable installation settings file.
test("config rejects invalid free-form values before writing settings", async () => {
const directory = mkdtempSync(join(tmpdir(), "tht-pi-management-invalid-"));
try {
let writes = 0;
const service = createPiManagement(configFor(join(directory, "settings.json")), {
execute: successfulExec([]),
listModels: async () => supportedModels,
readSettings: () => ({}),
saveSettings: () => { writes += 1; return {}; },
});
await expect(service.configure({
provider: "zai ", model: "glm-5.2", reasoning: "medium", unexpected: "value",
} as any)).rejects.toMatchObject<PiManagementError>({ code: "pi_management_invalid_config" });
expect(writes).toBe(0);
} finally {
rmSync(directory, { recursive: true, force: true });
}
});
// Catches a non-atomic implementation that can leave partial settings or temporary files after a
// normal installation-default update.
test("config validates closed choices and atomically persists non-secret defaults", async () => {
const directory = mkdtempSync(join(tmpdir(), "tht-pi-management-write-"));
const settingsFile = join(directory, "settings.json");
try {
const service = createPiManagement(configFor(settingsFile), {
execute: successfulExec([]),
listModels: async () => supportedModels,
now: () => new Date("2026-08-05T10:00:00.000Z"),
});
await expect(service.configure({
provider: "zai", model: "glm-5.2", reasoning: "high",
})).resolves.toEqual({
provider: "zai", model: "glm-5.2", reasoning: "high", updatedAt: "2026-08-05T10:00:00.000Z",
});
expect(JSON.parse(readFileSync(settingsFile, "utf8"))).toEqual({
provider: "zai", model: "glm-5.2", thinking: "high",
});
expect(readdirSync(directory)).toEqual(["settings.json"]);
} finally {
rmSync(directory, { recursive: true, force: true });
}
});
// Catches a hung Pi smoke check that leaves an operator waiting indefinitely or returns raw child
// diagnostics containing provider credentials.
test("smoke uses the configured timeout and reports a sanitized timeout", async () => {
@@ -188,7 +98,7 @@ test("smoke uses the configured timeout and reports a sanitized timeout", async
calls.push({ command, args, timeout: options.timeout });
throw Object.assign(new Error("provider token=raw-provider-token"), { code: "ETIMEDOUT" });
},
listModels: async () => supportedModels,
modelCatalog,
readSettings: () => ({ provider: "zai", model: "glm-5.2", thinking: "medium" }),
now: () => new Date("2026-08-05T10:00:00.000Z"),
});
@@ -210,7 +120,7 @@ test("smoke exercises the configured provider and model", async () => {
const providerChecks: unknown[] = [];
const service = createPiManagement(configFor(), {
execute: successfulExec([]),
listModels: async () => supportedModels,
modelCatalog,
readSettings: () => ({ provider: "zai", model: "glm-5.2", thinking: "medium" }),
smokeProvider: async (request) => { providerChecks.push(request); },
now: () => new Date("2026-08-05T10:00:00.000Z"),
@@ -230,7 +140,7 @@ test("smoke exercises the configured provider and model", async () => {
test("smoke fails closed and sanitizes configured-provider authentication errors", async () => {
const service = createPiManagement(configFor(), {
execute: successfulExec([]),
listModels: async () => supportedModels,
modelCatalog,
readSettings: () => ({ provider: "zai", model: "glm-5.2", thinking: "medium" }),
smokeProvider: async () => {
throw new Error('401 {"token":"raw-expired-token","output":"raw-provider-output"}');
@@ -252,7 +162,7 @@ test("smoke fails closed and sanitizes configured-provider authentication errors
test("smoke reports invalid managed provider configuration with a stable sanitized error", async () => {
const service = createPiManagement(configFor(), {
execute: successfulExec([]),
listModels: async () => supportedModels,
modelCatalog,
readSettings: () => ({ provider: "zai", model: "glm-5.2", thinking: "medium" }),
smokeProvider: async () => {
throw Object.assign(
@@ -284,7 +194,7 @@ test("smoke applies one deadline across version and a hung provider turn", async
() => resolve({ stdout: "pi 0.80.3\n", stderr: "" }),
500,
)),
listModels: async () => supportedModels,
modelCatalog,
readSettings: () => ({ provider: "zai", model: "glm-5.2", thinking: "medium" }),
smokeProvider: async ({ timeoutMs }) => {
providerTimeouts.push(timeoutMs);
@@ -320,7 +230,7 @@ test("logs keep only the latest 200 redacted lines", async () => {
source[201] = "THT_MODEL_API_KEY=raw-env-secret";
const service = createPiManagement(configFor(), {
execute: successfulExec([]),
listModels: async () => supportedModels,
modelCatalog,
readLogs: () => source.join("\n"),
now: () => new Date("2026-08-05T10:00:00.000Z"),
});
+48
View File
@@ -8,6 +8,7 @@ import {
} from "node:fs";
import { tmpdir } from "node:os";
import { PiProcessManager } from "../src/pi/pi-process-manager.js";
import type { RuntimeModelCatalog, RuntimeModel } from "../src/models/runtime-model-catalog.js";
import { loadConfig } from "../src/config.js";
import {
PI_MANAGED_CONFIG_ERROR_MESSAGE,
@@ -641,6 +642,53 @@ test("session Pi spawn reads the single secret bundle and scrubs its path", asyn
}
});
test("session Pi spawn resolves the selected catalog credential from the secret bundle", () => {
const root = mkdtempSync(path.join(tmpdir(), "thothii-catalog-credential-"));
const agentDir = path.join(root, "agent");
mkdirSync(agentDir, { mode: 0o700 });
writeFileSync(path.join(agentDir, "auth.json"), "{}\n", { mode: 0o600 });
writeFileSync(path.join(agentDir, "models.json"), '{"providers":{}}\n', { mode: 0o600 });
const secret = path.join(root, "thothii.secrets");
writeFileSync(secret, "ZAI_API_KEY=catalog-secret\nTHT_MODEL_API_KEY=legacy-secret\n", { mode: 0o600 });
const model: RuntimeModel = {
id: "openai/test-model",
provider: "openai",
model: "test-model",
label: "Test model",
upstreamModel: "test-model",
authentication: { mode: "secret_env", apiKeyEnv: "ZAI_API_KEY" },
sessionAdapter: { mode: "pi_builtin" },
session: { reasoning: false },
};
const modelCatalog: RuntimeModelCatalog = {
defaultSession: model.id,
defaultMetadataGeneration: null,
embedding: null,
sessionModels: () => [model],
metadataModels: () => [],
hasSession: (id) => id === model.id,
};
const calls: any[][] = [];
const child = recordingChild();
child.stderr.resume = () => {};
vi.stubEnv("PI_CODING_AGENT_DIR", agentDir);
const mgr = new PiProcessManager(loadConfig({ THT_SECRETS_FILE: secret }), {
modelCatalog,
authProviders: () => new Set(),
spawnFn: (...args: any[]) => { calls.push(args); return child as any; },
});
try {
mgr.createFor("catalog-credential", { provider: "openai", model: "test-model" });
expect(calls[0][2].env.ZAI_API_KEY).toBe("catalog-secret");
expect(calls[0][2].env).not.toHaveProperty("OPENAI_API_KEY");
expect(calls[0][2].env).not.toHaveProperty("THT_MODEL_API_KEY");
} finally {
mgr.teardown("catalog-credential");
vi.unstubAllEnvs();
rmSync(root, { recursive: true, force: true });
}
});
test.each([["OpenAI", "openai"], ["gemini", "google"]])(
"set_model uses canonical packaged provider ID for %s", async (provider, canonical) => {
const secret = path.resolve(__dirname, `.canonical-key-${process.pid}-${provider}`);
+48 -2
View File
@@ -1,9 +1,11 @@
import { EventEmitter } from "node:events";
import { existsSync, readFileSync, readdirSync } from "node:fs";
import { dirname } from "node:path";
import { existsSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { dirname, join } from "node:path";
import { afterEach, expect, test, vi } from "vitest";
import { loadConfig } from "../src/config.js";
import { createPiProviderSmoke } from "../src/pi/provider-smoke.js";
import type { RuntimeModel, RuntimeModelCatalog } from "../src/models/runtime-model-catalog.js";
afterEach(() => vi.unstubAllEnvs());
@@ -44,6 +46,50 @@ function successfulProviderChild() {
const MANAGED_CONFIG_ERROR = "Pi provider/model configuration is invalid";
test("provider smoke resolves the selected catalog credential from the secret bundle", async () => {
const root = mkdtempSync(join(tmpdir(), "thothii-smoke-catalog-credential-"));
const secret = join(root, "thothii.secrets");
writeFileSync(secret, "ZAI_API_KEY=catalog-secret\nTHT_MODEL_API_KEY=legacy-secret\n", { mode: 0o600 });
const model: RuntimeModel = {
id: "openai/test-model",
provider: "openai",
model: "test-model",
label: "Test model",
upstreamModel: "test-model",
authentication: { mode: "secret_env", apiKeyEnv: "ZAI_API_KEY" },
sessionAdapter: { mode: "pi_builtin" },
session: { reasoning: false },
};
const modelCatalog: RuntimeModelCatalog = {
defaultSession: model.id,
defaultMetadataGeneration: null,
embedding: null,
sessionModels: () => [model],
metadataModels: () => [],
hasSession: (id) => id === model.id,
};
let spawnEnv: NodeJS.ProcessEnv | undefined;
const smoke = createPiProviderSmoke(loadConfig({ THT_SECRETS_FILE: secret }), {
modelCatalog,
authProviders: () => new Set(),
readModelsStore: () => undefined,
spawnFn: (_command, _args, options) => {
spawnEnv = options.env;
return successfulProviderChild();
},
});
try {
await expect(smoke({
provider: "openai", model: "test-model", reasoning: "medium", timeoutMs: 750,
})).resolves.toBeUndefined();
expect(spawnEnv?.ZAI_API_KEY).toBe("catalog-secret");
expect(spawnEnv).not.toHaveProperty("OPENAI_API_KEY");
expect(spawnEnv).not.toHaveProperty("THT_MODEL_API_KEY");
} finally {
rmSync(root, { recursive: true, force: true });
}
});
// Catches an isolated smoke agent that copies auth.json but drops the selected custom
// provider/model from models.json, causing set_model to fail before the real request.
test("provider smoke reaches the selected custom provider from an isolated models.json", async () => {
+1 -8
View File
@@ -2,18 +2,11 @@ import { expect, test } from "vitest";
import { ReadinessManager } from "../src/runtime/readiness-manager.js";
const workspace = {
workspace: { schema_version: 3, id: "psd", name: "PSD", language: "it" },
workspace: { schema_version: 4, id: "psd", name: "PSD", language: "it" },
dwh: {
engine: "postgres", database: "warehouse", schema: "public",
supported_transports: ["postgres_direct"],
},
semantic_index: {
vector_store: { engine: "qdrant", collection: "psd", dimensions: 1024, distance: "cosine" },
embedding: {
provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024,
},
},
llm_policy: { allowed: ["zai/glm-5.2"] },
} as const;
function deferred<T>() {
+1 -13
View File
@@ -15,7 +15,7 @@ afterEach(() => {
});
const validYaml = `workspace:
schema_version: 3
schema_version: 4
id: psd-clinical
name: Policlinico San Donato
language: it
@@ -24,18 +24,6 @@ dwh:
database: postgres
schema: datawarehouse
supported_transports: [postgres_direct]
semantic_index:
vector_store:
engine: qdrant
collection: psd-clinical
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
llm_policy:
allowed: [zai/glm-5.2]
`;
async function git(cwd: string, args: string[]): Promise<string> {
+1 -13
View File
@@ -20,7 +20,7 @@ async function git(cwd: string, args: string[]): Promise<string> {
}
const descriptor = `workspace:
schema_version: 3
schema_version: 4
id: research
name: Research
language: en
@@ -29,18 +29,6 @@ dwh:
database: analytics
schema: mart
supported_transports: [postgres_direct]
semantic_index:
vector_store:
engine: qdrant
collection: research
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
llm_policy:
allowed: [zai/glm-5.2]
evidence:
source:
type: filesystem
+9 -27
View File
@@ -14,11 +14,6 @@ function fakeService(): PiManagementService {
config: { provider: "zai", model: "glm-5.2", reasoning: "medium" },
checkedAt: "2026-08-05T10:00:00.000Z",
})),
options: vi.fn(async () => ({
providers: ["zai"], models: [{ provider: "zai", id: "glm-5.2" }],
reasoning: ["low", "medium", "high"], checkedAt: "2026-08-05T10:00:00.000Z",
})),
configure: vi.fn(async (value) => ({ ...value, updatedAt: "2026-08-05T10:00:00.000Z" })),
test: vi.fn(async () => ({ ready: true, checkedAt: "2026-08-05T10:00:00.000Z" })),
logs: vi.fn(async () => ({ lines: ["Pi smoke check succeeded"], checkedAt: "2026-08-05T10:00:00.000Z" })),
};
@@ -106,24 +101,17 @@ test("loopback-only AUTH_MODE=none may read the sanitized Pi status", async () =
});
// A local implicit administrator has pi.manage, but a browser origin still cannot borrow that
// authority to mutate local configuration or trigger provider work.
// authority to trigger provider work.
test("loopback-only management rejects cross-origin writes for its local administrator", async () => {
const service = fakeService();
const app = appWith(service);
try {
const configured = await app.inject({
method: "PUT", url: "/pi-management/config",
headers: { host: "127.0.0.1:8080", origin: "https://evil.example" },
payload: { provider: "zai", model: "glm-5.2", reasoning: "high" },
});
const smoke = await app.inject({
method: "POST", url: "/pi-management/test",
headers: { host: "127.0.0.1:8080", origin: "https://evil.example" },
});
expect(configured.statusCode).toBe(403);
expect(smoke.statusCode).toBe(403);
expect(service.configure).not.toHaveBeenCalled();
expect(service.test).not.toHaveBeenCalled();
} finally {
await app.close();
@@ -132,14 +120,13 @@ test("loopback-only management rejects cross-origin writes for its local adminis
// Catches an origin guard that also blocks the same-origin Docker frontend or non-browser local
// lifecycle clients that do not send Origin.
test("loopback-only management preserves same-origin frontend and origin-less local writes", async () => {
test("loopback-only management preserves same-origin and origin-less smoke checks", async () => {
const service = fakeService();
const app = appWith(service);
try {
const sameOrigin = await app.inject({
method: "PUT", url: "/pi-management/config",
method: "POST", url: "/pi-management/test",
headers: { host: "127.0.0.1:8080", origin: "http://127.0.0.1:8080" },
payload: { provider: "zai", model: "glm-5.2", reasoning: "high" },
});
const lifecycleClient = await app.inject({ method: "POST", url: "/pi-management/test" });
@@ -150,25 +137,20 @@ test("loopback-only management preserves same-origin frontend and origin-less lo
}
});
// Catches route wiring that bypasses closed service validation or gives the browser a Docker/image
// lifecycle endpoint rather than only installation-default configuration and diagnostics.
test("trusted admins receive only configuration, smoke, options, and log endpoints", async () => {
// Catches a regression that reintroduces a browser-writable provider/model source.
test("trusted admins receive only status, smoke, and log endpoints", async () => {
const service = fakeService();
const app = appWith(service, exposedServerEnv);
try {
const options = await app.inject({ method: "GET", url: "/pi-management/options", headers: adminHeaders });
const configured = await app.inject({
method: "PUT", url: "/pi-management/config", headers: adminHeaders,
payload: { provider: "zai", model: "glm-5.2", reasoning: "high" },
});
const status = await app.inject({ method: "GET", url: "/pi-management/status", headers: adminHeaders });
const smoke = await app.inject({ method: "POST", url: "/pi-management/test", headers: adminHeaders });
const logs = await app.inject({ method: "GET", url: "/pi-management/logs", headers: adminHeaders });
expect(options.statusCode).toBe(200);
expect(configured.statusCode).toBe(200);
expect(configured.json()).toMatchObject({ provider: "zai", model: "glm-5.2", reasoning: "high" });
expect(status.statusCode).toBe(200);
expect(smoke.statusCode).toBe(200);
expect(logs.statusCode).toBe(200);
expect((await app.inject({ method: "GET", url: "/pi-management/options", headers: adminHeaders })).statusCode).toBe(404);
expect((await app.inject({ method: "PUT", url: "/pi-management/config", headers: adminHeaders })).statusCode).toBe(404);
expect(app.printRoutes()).not.toContain("update");
expect(app.printRoutes()).not.toContain("rollback");
} finally {
+45 -44
View File
@@ -18,25 +18,25 @@ const SCRIPT = path.resolve("../harness/tests/fake_pi/scripts/f1_disambiguation.
function operationalWorkspace(id = "default") {
return {
workspace: { schema_version: 3, id, name: id, language: "en" },
workspace: { schema_version: 4, id, name: id, language: "en" },
dwh: {
engine: "postgres", database: "warehouse", schema: "public",
supported_transports: ["postgres_direct"],
},
semantic_index: {
vector_store: {
engine: "qdrant", collection: id, dimensions: 1024, distance: "cosine",
},
embedding: {
provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024,
},
},
llm_policy: {
allowed: ["zai/glm-5.2", "deepseek/deepseek-v4-pro", "local-qwen/qwen3.6-35b-a3b"],
},
} as const;
}
function sessionCatalog(defaultSession = "zai/glm-5.2", available = [defaultSession]) {
return {
defaultSession,
defaultMetadataGeneration: null,
embedding: { id: "ollama/qwen3-embedding:0.6b", dimensions: 1024 },
sessionModels: () => [],
metadataModels: () => [],
hasSession: (id: string) => available.includes(id),
} as any;
}
const defaultWorkspaceRegistry = {
list: vi.fn(async () => [{
id: "default", commit: "e".repeat(40), blob: "f".repeat(40),
@@ -495,8 +495,8 @@ test("creates a session from the active immutable workspace revision", async ()
workspaceRegistry: {
read: vi.fn(async () => ({
workspace: {
workspace: { schema_version: 2, id: "psd-clinical", name: "PSD", language: "it" },
dwh: {}, semantic_index: {}, llm_policy: { allowed: ["zai/glm-5.2"] },
workspace: { schema_version: 4, id: "psd-clinical", name: "PSD", language: "it" },
dwh: {},
},
revision: {
id: "psd-clinical", commit: "a".repeat(40), blob: "b".repeat(40),
@@ -589,22 +589,11 @@ test("rejects an SSH-only workspace before persisting or starting a session", as
workspaceRegistry: {
acquireSessionRevision: vi.fn(async () => ({
workspace: {
workspace: { schema_version: 2, id: "ssh-workspace", name: "SSH", language: "en" },
workspace: { schema_version: 4, id: "ssh-workspace", name: "SSH", language: "en" },
dwh: {
engine: "postgres", database: "postgres", schema: "public",
supported_transports: ["ssh_tunnel"],
},
semantic_index: {
vector_store: {
engine: "pgvector", database: "postgres", schema: "vectors",
collection: "documents", dimensions: 768, distance: "cosine",
supported_transports: ["ssh_tunnel"],
},
embedding: {
provider: "ollama_compatible", model: "nomic-embed-text", dimensions: 768,
},
},
llm_policy: { allowed: ["zai/glm-5.2"] },
},
revision: {
id: "ssh-workspace", commit: "a".repeat(40), blob: "b".repeat(40),
@@ -632,7 +621,7 @@ test("hands a revision lease to retention only after the session manifest is dur
const markPersisted = vi.fn(async () => {});
const abort = vi.fn(async () => {});
const acquireSessionRevision = vi.fn(async () => ({
workspace: { llm_policy: { allowed: ["zai/glm-5.2"] } },
workspace: operationalWorkspace("leased"),
revision: {
id: "leased", commit: "a".repeat(40), blob: "b".repeat(40),
snapshotPath: `/data/workspace-registry/snapshots/${"a".repeat(40)}/leased.yaml`,
@@ -670,7 +659,7 @@ test("creates a session from the configured default workspace revision when work
const sessionNew = vi.fn(async () => ({ id: "default-pinned" }));
const registry = {
read: vi.fn(async (id: string) => ({
workspace: { llm_policy: { allowed: ["zai/glm-5.2"] } },
workspace: operationalWorkspace(id),
revision: {
id, commit: "c".repeat(40), blob: "d".repeat(40),
snapshotPath: `/data/workspace-registry/snapshots/${"c".repeat(40)}/${id}.yaml`,
@@ -758,7 +747,7 @@ test("session lifecycle locates a B session when installation default is A", asy
listModels: async () => [{ provider: "zai", id: "glm-5.2", name: "GLM 5.2", reasoning: true }],
workspaceRegistry: {
read: async (id: string) => ({
workspace: { llm_policy: { allowed: ["zai/glm-5.2"] } },
workspace: operationalWorkspace(id),
revision: { id, commit: "b".repeat(40), blob: "d".repeat(40), snapshotPath: bPath },
}),
list: async () => [
@@ -793,7 +782,7 @@ test("session lifecycle locates a B session when installation default is A", asy
expect(runtimeOptions).toEqual(["/runtime/1.yaml", "/runtime/2.yaml"]);
});
test("POST /sessions usa i settings (workspace/provider/model/thinking) e crea+avvia", async () => {
test("POST /sessions uses the catalog default with workspace/thinking settings and starts", async () => {
const modelKey = path.join(os.tmpdir(), `thoth-model-key-${process.pid}`);
writeFileSync(modelKey, "test-model-key", { mode: 0o600 });
chmodSync(modelKey, 0o600);
@@ -808,7 +797,8 @@ test("POST /sessions usa i settings (workspace/provider/model/thinking) e crea+a
sessionNew: async (o: any) => { sessionNewArg = o; return { id: "s1" }; },
sessionList: async () => [{ id: "s1" }],
} as any,
getSettings: () => ({ workspace: "w", provider: "zai", model: "glm-5.2", thinking: "high" }),
getSettings: () => ({ workspace: "w", thinking: "high" }),
runtimeModelCatalog: sessionCatalog(),
listModels: async () => [
{ provider: "zai", id: "glm-5.2", name: "GLM 5.2", reasoning: true },
],
@@ -2565,32 +2555,43 @@ test("POST /sessions proceeds when ollamaEnsure succeeds", async () => {
expect(ensureWs).toContain(`/snapshots/${"e".repeat(40)}/psd.yaml`);
});
test("POST /sessions rejects an unavailable saved model before persisting a session", async () => {
test("POST /sessions falls back from a stale requested model to the catalog default", async () => {
let created = 0;
let persisted: any;
const runtime = { bridge: { onClientEvent: () => {} } };
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
thtRunner: {
sessionNew: async () => { created += 1; return { id: "must-not-exist" }; },
sessionNew: async (options: any) => { created += 1; persisted = options; return { id: "fallback" }; },
searchPack: async () => {},
} as any,
readiness: { ensure: async () => ({ ok: true }) } as any,
getSettings: () => ({
workspace: "psd",
provider: "deepseek",
model: "deepseek-v4-pro",
thinking: "medium",
}) as any,
getSettings: () => ({ workspace: "psd", thinking: "medium" }) as any,
runtimeModelCatalog: sessionCatalog(),
listModels: async () => [
{ provider: "zai", id: "glm-5.2", name: "GLM 5.2", reasoning: true },
],
mgr: {
teardownForPrincipal: () => [],
createFor: () => runtime,
get: () => runtime,
configure: async () => {},
start: () => {},
} as any,
});
const res = await app.inject({ method: "POST", url: "/sessions", payload: { question: "q" } });
const res = await app.inject({
method: "POST",
url: "/sessions",
payload: { question: "q", provider: "deepseek", model: "deepseek-v4-pro" },
});
expect(res.statusCode).toBe(503);
expect(res.statusCode).toBe(200);
expect(res.json()).toEqual({
error: "Selected model is unavailable. Check Pi authentication and model settings, then try again.",
code: "model_unavailable",
id: "fallback",
warning: "Configured model deepseek/deepseek-v4-pro is unavailable; using zai/glm-5.2.",
});
expect(created).toBe(0);
expect(persisted).toMatchObject({ provider: "zai", model: "glm-5.2" });
expect(created).toBe(1);
});
test("POST /sessions marks a persisted session failed when runtime construction throws", async () => {
+16 -16
View File
@@ -21,7 +21,7 @@ function appWithTmpSettings(extraEnv: Record<string, string> = {}, deps = {}) {
return { app, dir };
}
test("GET /settings returns effective defaults (env provider/model/thinking, first workspace)", async () => {
test("GET /settings returns thinking and the first workspace without legacy model defaults", async () => {
const { app, dir } = appWithTmpSettings({ PI_PROVIDER: "zai", PI_MODEL: "glm-5.2", PI_THINKING: "medium" }, {
listModels: async () => [{ provider: "zai", id: "glm-5.2", name: "GLM 5.2", reasoning: true }],
});
@@ -29,8 +29,8 @@ test("GET /settings returns effective defaults (env provider/model/thinking, fir
const res = await app.inject({ method: "GET", url: "/settings" });
expect(res.statusCode).toBe(200);
const body = res.json();
expect(body.provider).toBe("zai");
expect(body.model).toBe("glm-5.2");
expect(body).not.toHaveProperty("provider");
expect(body).not.toHaveProperty("model");
expect(body.thinking).toBe("medium");
expect(typeof body.workspace).toBe("string"); // first workspace from ../harness/workspaces
} finally {
@@ -62,7 +62,9 @@ test("PUT /settings does not persist personal workspace or LLM choices", async (
});
expect(put.statusCode).toBe(200);
const got = await app.inject({ method: "GET", url: "/settings" });
expect(got.json()).toMatchObject({ provider: "zai", model: "glm-5.2", thinking: "medium" });
expect(got.json()).toMatchObject({ thinking: "medium" });
expect(got.json()).not.toHaveProperty("provider");
expect(got.json()).not.toHaveProperty("model");
expect(got.json()).not.toMatchObject({ workspace: "psd", thinking: "high" });
} finally {
rmSync(dir, { recursive: true, force: true });
@@ -114,7 +116,7 @@ test("settings no longer read or write principal-specific preferences", async ()
expect(preferences.size).toBe(0);
});
test("GET /settings retains complete legacy installation defaults without seeding a private profile", async () => {
test("GET /settings drops legacy installation model fields without seeding a private profile", async () => {
let preferences: Record<string, unknown> = {};
const writes: Record<string, unknown>[] = [];
const runner = {
@@ -135,9 +137,7 @@ test("GET /settings retains complete legacy installation defaults without seedin
const first = await app.inject({ method: "GET", url: "/settings" });
const second = await app.inject({ method: "GET", url: "/settings" });
const expected = {
workspace: "local", provider: "local-qwen", model: "qwen3.6-35b-a3b", thinking: "low",
};
const expected = { workspace: "local", thinking: "low" };
expect(first.statusCode).toBe(200);
expect(first.json()).toEqual(expected);
expect(second.json()).toEqual(expected);
@@ -167,15 +167,13 @@ test("GET /settings ignores stale private preferences in favor of installation d
const response = await app.inject({ method: "GET", url: "/settings" });
expect(response.statusCode).toBe(200);
expect(response.json()).toEqual({
workspace: "local", provider: "local-qwen", model: "qwen3.6-35b-a3b", thinking: "low",
});
expect(response.json()).toEqual({ workspace: "local", thinking: "low" });
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
test("PUT /settings rejects an unknown model when a model list is available", async () => {
test("PUT /settings ignores a legacy unknown model because the catalog owns model validity", async () => {
const { app, dir } = appWithTmpSettings({}, {
listModels: async () => [{ provider: "zai", id: "glm-5.2", name: "GLM 5.2", reasoning: true }],
});
@@ -184,14 +182,14 @@ test("PUT /settings rejects an unknown model when a model list is available", as
method: "PUT", url: "/settings",
payload: { workspace: "psd", provider: "zai", model: "does-not-exist", thinking: "low" },
});
expect(put.statusCode).toBe(400);
expect(put.json()).toMatchObject({ error: expect.stringMatching(/model/i) });
expect(put.statusCode).toBe(200);
expect(put.json()).not.toHaveProperty("model");
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
test("PUT /settings validates provider and model as one composite identifier", async () => {
test("PUT /settings ignores legacy provider/model pairs", async () => {
const { app, dir } = appWithTmpSettings({}, {
listModels: async () => [
{ provider: "provider-a", id: "shared-id", name: "A", reasoning: false },
@@ -204,7 +202,7 @@ test("PUT /settings validates provider and model as one composite identifier", a
workspace: "psd", provider: "provider-b", model: "shared-id", thinking: "low",
},
});
expect(wrongProvider.statusCode).toBe(400);
expect(wrongProvider.statusCode).toBe(200);
const exactPair = await app.inject({
method: "PUT", url: "/settings",
@@ -213,6 +211,8 @@ test("PUT /settings validates provider and model as one composite identifier", a
},
});
expect(exactPair.statusCode).toBe(200);
expect(exactPair.json()).not.toHaveProperty("provider");
expect(exactPair.json()).not.toHaveProperty("model");
} finally {
rmSync(dir, { recursive: true, force: true });
}
+25 -33
View File
@@ -1,4 +1,4 @@
import { test, expect, vi } from "vitest";
import { test, expect } from "vitest";
import Fastify from "fastify";
import { buildApp } from "../src/app.js";
import { loadConfig } from "../src/config.js";
@@ -156,12 +156,23 @@ test("registry-backed SQL preview resolves and uses the session's pinned runtime
// Workspace registry route coverage lives in routes-workspaces.test.ts. `/workspaces` no longer
// reads legacy harness files: the Git registry is the single shared source of truth.
test("GET /models returns {models:[...]} from injected listModels stub", async () => {
test("GET /models returns session choices from the installation model catalog", async () => {
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
thtRunner: {} as any,
listModels: async () => [
{ provider: "zai", id: "glm-5.2", name: "GLM 5.2", reasoning: true },
],
runtimeModelCatalog: {
defaultSession: "zai/glm-5.2",
defaultMetadataGeneration: null,
embedding: { id: "ollama/qwen3-embedding:0.6b", dimensions: 1024 },
sessionModels: () => [{
id: "zai/glm-5.2", provider: "zai", model: "glm-5.2", label: "GLM 5.2",
upstreamModel: "glm-5.2", authentication: { mode: "pi_auth" },
sessionAdapter: { mode: "pi_builtin" }, session: { reasoning: true },
}],
metadataModels: () => [],
hasSession: (id: string) => id === "zai/glm-5.2",
},
// Runtime introspection is a health gate for starting a session, not a second catalog.
listModels: async () => { throw new Error("Pi is unavailable"); },
});
const res = await app.inject({ method: "GET", url: "/models" });
@@ -172,36 +183,17 @@ test("GET /models returns {models:[...]} from injected listModels stub", async (
});
});
test("GET /models returns {models:[]} when listModels throws (graceful fallback)", async () => {
test("GET /models returns an empty list when the catalog has no session models", async () => {
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
thtRunner: {} as any,
listModels: async () => { throw new Error("Pi not running"); },
});
const res = await app.inject({ method: "GET", url: "/models" });
expect(res.statusCode).toBe(200);
expect(res.json()).toEqual({ models: [] });
});
test("GET /models logs a sanitized warning when listing fails", async () => {
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
thtRunner: {} as any,
listModels: async () => { throw new Error("credential-value-must-not-appear"); },
});
const warn = vi.spyOn(app.log, "warn");
const res = await app.inject({ method: "GET", url: "/models" });
expect(res.json()).toEqual({ models: [] });
expect(JSON.stringify(warn.mock.calls)).not.toContain("credential-value-must-not-appear");
expect(warn).toHaveBeenCalled();
});
test("GET /models with empty listModels stub returns empty array", async () => {
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
thtRunner: {} as any,
listModels: async () => [],
runtimeModelCatalog: {
defaultSession: null,
defaultMetadataGeneration: null,
embedding: null,
sessionModels: () => [],
metadataModels: () => [],
hasSession: () => false,
},
});
const res = await app.inject({ method: "GET", url: "/models" });
+3 -17
View File
@@ -14,7 +14,7 @@ import type { AuthDiagnoser, AuthDiagnostics } from "../src/auth/diagnostics.js"
const workspace: CanonicalWorkspace = {
workspace: {
schema_version: 3,
schema_version: 4,
id: "psd-clinical",
name: "Policlinico San Donato",
description: "Clinical analytics workspace",
@@ -26,20 +26,6 @@ const workspace: CanonicalWorkspace = {
schema: "datawarehouse",
supported_transports: ["postgres_direct"],
},
semantic_index: {
vector_store: {
engine: "qdrant",
collection: "psd-clinical",
dimensions: 1024,
distance: "cosine",
},
embedding: {
provider: "ollama_internal",
model: "qwen3-embedding:0.6b",
dimensions: 1024,
},
},
llm_policy: { allowed: ["zai/glm-5.2"] },
};
const revision: WorkspaceRevision = {
@@ -194,7 +180,7 @@ test("lists workspace summaries and reads a validated immutable workspace", asyn
expect(read.json()).toEqual({ workspace, revision });
});
test("validates a schema v3 workspace without mutating the repository", async () => {
test("validates a schema v4 workspace without mutating the repository", async () => {
const app = appFor(registryFake());
const response = await app.inject({
@@ -284,7 +270,7 @@ test.each([1, 2])("rejects schema v%s at the validation boundary with a sanitize
expect(response.body).not.toMatch(/migration_required|schema version/i);
});
test("runs diagnostics for a schema v3 workspace", async () => {
test("runs diagnostics for a schema v4 workspace", async () => {
const diagnose = vi.fn(async () => ({ activatable: true, diagnostics: [] }));
const app = appFor(registryFake(), diagnose);
+6 -6
View File
@@ -27,9 +27,9 @@ test("saveSettings writes the file and loadSettings reads it back", () => {
const dir = mkdtempSync(join(tmpdir(), "tht-set-"));
try {
const cfg = cfgWith(join(dir, "nested", "settings.json"));
const saved = saveSettings(cfg, { workspace: "psd", provider: "zai", model: "glm-5.2", thinking: "medium" });
expect(saved.model).toBe("glm-5.2");
expect(loadSettings(cfg)).toEqual({ workspace: "psd", provider: "zai", model: "glm-5.2", thinking: "medium" });
const saved = saveSettings(cfg, { workspace: "psd", thinking: "medium" });
expect(saved.thinking).toBe("medium");
expect(loadSettings(cfg)).toEqual({ workspace: "psd", thinking: "medium" });
} finally {
rmSync(dir, { recursive: true, force: true });
}
@@ -55,11 +55,11 @@ test("saveSettings restores the previous file when post-rename directory durabil
const dir = mkdtempSync(join(tmpdir(), "tht-set-transaction-"));
try {
const cfg = cfgWith(join(dir, "settings.json"));
saveSettings(cfg, { provider: "old", model: "old-model", thinking: "low" });
saveSettings(cfg, { thinking: "low" });
let syncs = 0;
expect(() => saveSettings(
cfg,
{ provider: "new", model: "new-model", thinking: "high" },
{ thinking: "high" },
{
syncDirectory(directory: string) {
syncs += 1;
@@ -69,7 +69,7 @@ test("saveSettings restores the previous file when post-rename directory durabil
},
},
)).toThrow(/directory fsync failure/);
expect(loadSettings(cfg)).toEqual({ provider: "old", model: "old-model", thinking: "low" });
expect(loadSettings(cfg)).toEqual({ thinking: "low" });
expect(syncs).toBeGreaterThanOrEqual(2);
} finally {
rmSync(dir, { recursive: true, force: true });
+1 -8
View File
@@ -8,18 +8,11 @@ const keywordIndexes = [
];
const workspace: CanonicalWorkspace = {
workspace: { schema_version: 3, id: "psd", name: "PSD", language: "it" },
workspace: { schema_version: 4, id: "psd", name: "PSD", language: "it" },
dwh: {
engine: "postgres", database: "warehouse", schema: "public",
supported_transports: ["postgres_direct"],
},
semantic_index: {
vector_store: { engine: "qdrant", collection: "psd", dimensions: 1024, distance: "cosine" },
embedding: {
provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024,
},
},
llm_policy: { allowed: ["zai/glm-5.2"] },
};
function runner(request: (...args: any[]) => Promise<any>) {
@@ -19,12 +19,13 @@ afterEach(() => {
const semanticRuntime = {
internalQdrantUrl: "http://qdrant:6333",
internalEmbeddingUrl: "http://embedding:11434",
internalEmbeddingId: "ollama/qwen3-embedding:0.6b",
internalEmbeddingModel: "qwen3-embedding:0.6b",
internalEmbeddingDimensions: 1024,
};
const baseWorkspace = parseWorkspaceYaml(`workspace:
schema_version: 3
schema_version: 4
id: psd-clinical
name: Runtime Lease
language: en
@@ -33,21 +34,9 @@ dwh:
database: analytics
schema: mart
supported_transports: [postgres_direct]
semantic_index:
vector_store:
engine: qdrant
collection: psd-clinical
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
llm_policy:
allowed: [zai/glm-5.2]
`);
const filesystemWorkspace = parseWorkspaceYaml(`${baseWorkspace ? '' : ''}workspace:
schema_version: 3
schema_version: 4
id: fs-workspace
name: Filesystem
language: en
@@ -56,25 +45,13 @@ dwh:
database: analytics
schema: mart
supported_transports: [postgres_direct]
semantic_index:
vector_store:
engine: qdrant
collection: fs-workspace
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
llm_policy:
allowed: [zai/glm-5.2]
evidence:
source:
type: filesystem
uri: fs-workspace/evidence
`);
const privateHttpWorkspace = parseWorkspaceYaml(`workspace:
schema_version: 3
schema_version: 4
id: http-workspace
name: Http
language: en
@@ -83,18 +60,6 @@ dwh:
database: analytics
schema: mart
supported_transports: [postgres_direct]
semantic_index:
vector_store:
engine: qdrant
collection: http-workspace
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
llm_policy:
allowed: [zai/glm-5.2]
evidence:
source:
type: http
@@ -125,7 +90,7 @@ function runtime(workspace = baseWorkspace, workspaceId = workspace.workspace.id
bindingDigest: "sha256:bindings",
semanticQdrantUrl: "http://qdrant:6333",
effectiveConfig: {
schemaVersion: 1,
schemaVersion: 2,
dwh: {
engine: "postgres",
database: "analytics",
@@ -136,7 +101,7 @@ function runtime(workspace = baseWorkspace, workspaceId = workspace.workspace.id
user: "reader",
},
vector: { collection: workspaceId, dimensions: 1024, distance: "cosine" },
embedding: { model: "qwen3-embedding:0.6b", dimensions: 1024 },
embedding: { id: "ollama/qwen3-embedding:0.6b", model: "qwen3-embedding:0.6b", dimensions: 1024 },
roots: { artifacts: "/data/artifacts", indexes: "/data/indexes" },
},
effectiveConfigIdentity: "workspace://psd-clinical@v1:" + "d".repeat(64),
@@ -28,13 +28,15 @@ test("job state creates durable 0600 JSON and enforces same-revision resume", as
catalogBlob: "c".repeat(40),
configDigest: "sha256:config",
bindingDigest: "sha256:bindings",
embeddingId: "ollama/qwen3-embedding:0.6b",
embeddingDimensions: 1024,
});
const path = store.jobPath(job.runId);
expect(existsSync(path)).toBe(true);
expect(statSync(path).mode & 0o777).toBe(0o600);
expect(JSON.parse(readFileSync(path, "utf8"))).toMatchObject({
schemaVersion: 1,
schemaVersion: 2,
operation: "preprocess dwh",
workspaceId: "psd-clinical",
workspaceRevision: "a".repeat(40),
@@ -42,6 +44,8 @@ test("job state creates durable 0600 JSON and enforces same-revision resume", as
catalogBlob: "c".repeat(40),
configDigest: "sha256:config",
bindingDigest: "sha256:bindings",
embeddingId: "ollama/qwen3-embedding:0.6b",
embeddingDimensions: 1024,
});
await expect(store.beginJob({
@@ -52,6 +56,20 @@ test("job state creates durable 0600 JSON and enforces same-revision resume", as
catalogBlob: "c".repeat(40),
configDigest: "sha256:config",
bindingDigest: "sha256:bindings",
embeddingId: "ollama/qwen3-embedding:0.6b",
embeddingDimensions: 1024,
})).rejects.toMatchObject({ code: "preprocessing_resume_mismatch" });
await expect(store.beginJob({
operation: "preprocess dwh",
runId: job.runId,
workspaceRevision: "a".repeat(40),
descriptorBlob: "b".repeat(40),
catalogBlob: "c".repeat(40),
configDigest: "sha256:config",
bindingDigest: "sha256:bindings",
embeddingId: "ollama/replacement-embedding",
embeddingDimensions: 1024,
})).rejects.toMatchObject({ code: "preprocessing_resume_mismatch" });
const resumed = await store.beginJob({
@@ -62,6 +80,8 @@ test("job state creates durable 0600 JSON and enforces same-revision resume", as
catalogBlob: "c".repeat(40),
configDigest: "sha256:config",
bindingDigest: "sha256:bindings",
embeddingId: "ollama/qwen3-embedding:0.6b",
embeddingDimensions: 1024,
});
expect(resumed.runId).toBe(job.runId);
});
@@ -75,10 +75,6 @@ function workspaceVariant(
return {
...workspace,
workspace: { ...workspace.workspace, ...changes, id },
semantic_index: {
...workspace.semantic_index,
vector_store: { ...workspace.semantic_index.vector_store, collection: id },
},
...(workspace.evidence?.source.type === "filesystem"
? {
evidence: {
@@ -182,7 +178,7 @@ test("shared deployment fixtures remain valid standalone descriptors with canoni
expect(smoke).toMatchObject({
workspace: {
schema_version: 3,
schema_version: 4,
id: "local",
name: "Local",
description: "Isolated workspace registry smoke fixture.",
@@ -193,14 +189,14 @@ test("shared deployment fixtures remain valid standalone descriptors with canoni
},
});
expect(task13).toMatchObject({
workspace: { schema_version: 3, id: "task13-smoke", name: "Task 13 Smoke" },
workspace: { schema_version: 4, id: "task13-smoke", name: "Task 13 Smoke" },
evidence: {
source: { type: "filesystem", uri: "task13-smoke/evidence", patterns: ["**/*.md"] },
policy: { max_chunk_chars: 4000, retain_published_generations: 3 },
},
});
expect(windows).toMatchObject({
workspace: { schema_version: 3, id: "task13-windows", name: "Task 13 Windows" },
workspace: { schema_version: 4, id: "task13-windows", name: "Task 13 Windows" },
evidence: {
source: { type: "filesystem", uri: "task13-windows/evidence", patterns: ["**/*.md"] },
policy: { max_chunk_chars: 4000, retain_published_generations: 3 },
@@ -282,7 +278,7 @@ test("registry rejects orphan descriptors, metadata mismatches, and the retired
});
});
test("Windows clone contract copies the shared complete schema v3 descriptor into the nested registry layout", () => {
test("Windows clone contract copies the shared complete schema v4 descriptor into the nested registry layout", () => {
const descriptor = parseWorkspaceYaml(readFixture("workspace-registry-windows.yaml"));
const windows = readFileSync(
new URL("../../scripts/test-windows-clone-contract.ps1", import.meta.url),
@@ -299,7 +295,7 @@ test("Windows clone contract copies the shared complete schema v3 descriptor int
expect(windows).not.toContain('schema_version: 3');
expect(descriptor).toMatchObject({
workspace: {
schema_version: 3,
schema_version: 4,
id: "task13-windows",
name: "Task 13 Windows",
language: "en",
+6 -57
View File
@@ -15,7 +15,7 @@ import {
import type { WorkspaceRegistryConfig } from "../src/workspaces/types.js";
const validYaml = `workspace:
schema_version: 3
schema_version: 4
id: psd-clinical
name: Policlinico San Donato
language: it
@@ -24,18 +24,6 @@ dwh:
database: postgres
schema: datawarehouse
supported_transports: [postgres_direct]
semantic_index:
vector_store:
engine: qdrant
collection: psd-clinical
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
llm_policy:
allowed: [zai/glm-5.2]
`;
function withFilesystemEvidence(source: string, id = "psd-clinical"): string {
@@ -146,7 +134,7 @@ function legacyV1Yaml(source = validYaml): string {
.replace(" model: qwen3-embedding:0.6b", " model: nomic-embed-text-v2-moe")
.replace(" dimensions: 1024", " dimensions: 768")
.replace("distance: cosine\n", "distance: cosine\n supported_transports: [pgvector_direct]\n")
.replace("schema_version: 3", "schema_version: 1");
.replace("schema_version: 4", "schema_version: 1");
}
function legacyV2Yaml(source = validYaml): string {
@@ -158,7 +146,7 @@ function legacyV2Yaml(source = validYaml): string {
.replace(" model: qwen3-embedding:0.6b", " model: nomic-embed-text-v2-moe")
.replace(" dimensions: 1024", " dimensions: 768")
.replace("distance: cosine\n", "distance: cosine\n supported_transports: [pgvector_direct]\n")
.replace("schema_version: 3", "schema_version: 2");
.replace("schema_version: 4", "schema_version: 2");
}
const runFile = promisify(execFile);
@@ -197,7 +185,7 @@ async function fixture(workspaceSource = validYaml): Promise<{
await git(source, ["init", "--initial-branch=main"]);
await git(source, ["config", "user.name", "Workspace Registry Test"]);
await git(source, ["config", "user.email", "workspace-registry@example.invalid"]);
const workspace = workspaceSource.includes("schema_version: 3")
const workspace = workspaceSource.includes("schema_version: 4")
? parseWorkspaceYaml(workspaceSource) : undefined;
mkdirSync(join(source, "psd-clinical"), { recursive: true });
writeFileSync(join(source, "thoth-workspaces.yaml"), catalogYaml([
@@ -256,7 +244,7 @@ async function multiWorkspaceFixture(workspaces: Record<string, string>): Promis
await git(source, ["config", "user.email", "workspace-registry@example.invalid"]);
const entries = [];
for (const [id, workspaceSource] of Object.entries(workspaces)) {
const workspace = workspaceSource.includes("schema_version: 3") ? parseWorkspaceYaml(workspaceSource) : undefined;
const workspace = workspaceSource.includes("schema_version: 4") ? parseWorkspaceYaml(workspaceSource) : undefined;
entries.push({ id, name: workspace.workspace.name, ...(workspace.workspace.description ? { description: workspace.workspace.description } : {}) });
mkdirSync(join(source, id), { recursive: true });
writeFileSync(join(source, id, "workspace.yaml"), workspaceSource);
@@ -788,7 +776,7 @@ test("normalizes historical operational state during offline fallback after rest
expect(read.revision).not.toHaveProperty("state");
});
test("fails closed when a retained snapshot descriptor is not schema v3", async () => {
test("fails closed when a retained snapshot descriptor is not schema v4", async () => {
const remote = await fixture();
const root = join(remote.root, "registry");
await new WorkspaceRegistry(config(root, remote.remote)).bootstrap();
@@ -820,45 +808,6 @@ test("keeps the last valid snapshot when a pulled commit has invalid YAML", asyn
});
});
test("rejects duplicate schema v3 collection ownership and keeps the previous active snapshot", async () => {
const v3Yaml = validYaml;
const remote = await multiWorkspaceFixture({
"psd-clinical": v3Yaml,
"research-clinical": v3Yaml
.replace("id: psd-clinical", "id: research-clinical")
.replace("name: Policlinico San Donato", "name: Research Clinical")
.replace("collection: psd-clinical", "collection: research-clinical"),
});
const registry = new WorkspaceRegistry(config(join(remote.root, "registry"), remote.remote));
await registry.bootstrap();
writeFileSync(
join(remote.source, "research-clinical", "workspace.yaml"),
v3Yaml
.replace("id: psd-clinical", "id: research-clinical")
.replace("name: Policlinico San Donato", "name: Research Clinical")
.replace("collection: psd-clinical", "collection: shared"),
);
writeFileSync(
join(remote.source, "psd-clinical", "workspace.yaml"),
v3Yaml.replace("collection: psd-clinical", "collection: shared"),
);
await git(remote.source, ["add", "-A"]);
await git(remote.source, ["commit", "-m", "Duplicate collection ownership"]);
await git(remote.source, ["push", "origin", "main"]);
await expect(registry.pull()).rejects.toMatchObject({
code: "workspace_invalid",
message: "Workspace repository content is invalid",
});
await expect(registry.read("psd-clinical")).resolves.toMatchObject({
revision: { commit: remote.initialCommit },
});
await expect(registry.read("research-clinical")).resolves.toMatchObject({
revision: { commit: remote.initialCommit },
});
});
test("retains a historical snapshot while a resumable manifest still references its revision", async () => {
const remote = await fixture();
const root = join(remote.root, "registry");
@@ -66,7 +66,7 @@ workspaces: [{id: psd-clinical, name: Runtime Lease}]
`);
mkdirSync(join(source, "psd-clinical", "evidence"), { recursive: true });
writeFileSync(join(source, "psd-clinical", "workspace.yaml"), `workspace:
schema_version: 3
schema_version: 4
id: psd-clinical
name: Runtime Lease
language: en
@@ -75,18 +75,6 @@ dwh:
database: analytics
schema: mart
supported_transports: [postgres_direct]
semantic_index:
vector_store:
engine: qdrant
collection: psd-clinical
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
llm_policy:
allowed: [zai/glm-5.2]
evidence:
source:
type: filesystem
+2 -14
View File
@@ -21,7 +21,7 @@ const thtBin = join(harnessDir, ".venv", "bin", "tht");
const roots: string[] = [];
const canonicalWorkspace = `workspace:
schema_version: 3
schema_version: 4
id: psd-clinical
name: Runtime handoff
language: en
@@ -30,18 +30,6 @@ dwh:
database: analytics
schema: mart
supported_transports: [postgres_direct]
semantic_index:
vector_store:
engine: qdrant
collection: psd-clinical
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
llm_policy:
allowed: [zai/glm-5.2]
`;
const filesystemWorkspace = `${canonicalWorkspace}evidence:
@@ -145,7 +133,7 @@ function runnerFor(f: Awaited<ReturnType<typeof fixture>>): ThtRunner {
} as any);
}
test("real schema-v3 registry revision loads through ThtRunner and the harness contract", async () => {
test("real schema-v4 registry revision loads through ThtRunner and the harness contract", async () => {
const f = await fixture();
const runner = runnerFor(f);
@@ -12,8 +12,8 @@ import {
import { supportsSessionRuntime } from "../src/workspaces/bindings.js";
import { parseWorkspaceYaml } from "../src/workspaces/schema.js";
const workspaceV3 = parseWorkspaceYaml(`workspace:
schema_version: 3
const workspaceV4 = parseWorkspaceYaml(`workspace:
schema_version: 4
id: psd-clinical
name: Policlinico San Donato
language: it
@@ -22,19 +22,6 @@ dwh:
database: postgres
schema: datawarehouse
supported_transports: [postgres_direct, rest_api, ssh_tunnel]
semantic_index:
vector_store:
engine: qdrant
collection: psd-clinical
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
llm_policy:
default: zai/glm-5.2
allowed: [zai/glm-5.2]
`);
const paths: RuntimePaths = {
sessions: "/data/workspaces/psd-clinical/sessions",
@@ -45,6 +32,7 @@ const paths: RuntimePaths = {
const semanticRuntime: SemanticRuntimeConfig = {
internalQdrantUrl: "http://qdrant:6333",
internalEmbeddingUrl: "http://embedding:11434",
internalEmbeddingId: "ollama/qwen3-embedding:0.6b",
internalEmbeddingModel: "qwen3-embedding:0.6b",
internalEmbeddingDimensions: 1024,
};
@@ -63,8 +51,8 @@ const directBindings: RuntimeBindings = {
evidence: { missing: [], values: {} },
};
test("renders only the schema-v3 internal Qdrant and Ollama runtime shape", () => {
const rendered = parse(renderRuntimeConfig(workspaceV3, directBindings, paths, {
test("derives the internal Qdrant and Ollama runtime shape from workspace v4 plus installation config", () => {
const rendered = parse(renderRuntimeConfig(workspaceV4, directBindings, paths, {
workspaceId: "psd-clinical", workspaceRevision: "a".repeat(40),
}, {}, semanticRuntime));
@@ -95,8 +83,8 @@ test("renders only the schema-v3 internal Qdrant and Ollama runtime shape", () =
expect(rendered).not.toHaveProperty("vector_rest");
});
test("renders schema-v3 DWH REST without exposing secret contents", () => {
const rendered = parse(renderRuntimeConfig(workspaceV3, {
test("renders workspace-v4 DWH REST without exposing secret contents", () => {
const rendered = parse(renderRuntimeConfig(workspaceV4, {
dwh: {
transport: "rest_api", missing: [], values: {
THT_WS_PSD_CLINICAL_DWH_BASE_URL: "https://dwh.example.test",
@@ -152,7 +140,7 @@ function evidenceWorkspace(
}
const canonicalEvidenceWorkspace = `workspace:
schema_version: 3
schema_version: 4
id: psd-clinical
name: Runtime Evidence
language: en
@@ -161,18 +149,6 @@ dwh:
database: analytics
schema: mart
supported_transports: [postgres_direct]
semantic_index:
vector_store:
engine: qdrant
collection: psd-clinical
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
llm_policy:
allowed: [zai/glm-5.2]
`;
const evidenceRevision = "1".repeat(40);
@@ -374,7 +350,7 @@ test("renders static S3 Evidence with endpoint policy, limits, and file paths bu
test("omits Evidence configuration and policy when the descriptor has no Evidence", () => {
const rendered = parse(renderRuntimeConfig(
workspaceV3,
workspaceV4,
directBindings,
paths,
evidenceContext,
@@ -14,7 +14,7 @@ const roots: string[] = [];
function workspace(extra = "") {
return parseWorkspaceYaml(`workspace:
schema_version: 3
schema_version: 4
id: psd-clinical
name: Policlinico San Donato
language: en
@@ -23,10 +23,6 @@ dwh:
database: postgres
schema: datawarehouse
supported_transports: [postgres_direct, rest_api, ssh_tunnel]
semantic_index:
vector_store: { engine: qdrant, collection: psd-clinical, dimensions: 1024, distance: cosine }
embedding: { provider: ollama_internal, model: qwen3-embedding:0.6b, dimensions: 1024 }
llm_policy: { allowed: [zai/glm-5.2] }
${extra}`);
}
@@ -0,0 +1,36 @@
import { expect, test } from "vitest";
import { migrateWorkspaceV3Yaml, parseWorkspaceYaml } from "../src/workspaces/schema.js";
const legacy = `workspace:
schema_version: 3
id: abc
name: Example
language: en
dwh:
engine: postgres
database: warehouse
schema: public
supported_transports: [postgres_direct]
semantic_index:
vector_store: {engine: qdrant, collection: abc, dimensions: 1024, distance: cosine}
embedding: {provider: ollama_internal, model: qwen3-embedding:0.6b, dimensions: 1024}
llm_policy:
default: zai/glm-5.3
allowed: [zai/glm-5.3]
`;
test("strict workspace v4 rejects model-bearing v3 descriptors", () => {
expect(() => parseWorkspaceYaml(legacy)).toThrow();
});
test("v3 to v4 migration removes only model/vector policy and bumps the version", () => {
const migrated = migrateWorkspaceV3Yaml(legacy);
const workspace = parseWorkspaceYaml(migrated);
expect(workspace.workspace).toMatchObject({ schema_version: 4, id: "abc" });
expect(migrated).not.toMatch(/semantic_index|llm_policy/);
expect(workspace.dwh).toEqual({
engine: "postgres", database: "warehouse", schema: "public",
supported_transports: ["postgres_direct"],
});
});
+12 -28
View File
@@ -9,8 +9,8 @@ import {
} from "../src/workspaces/bindings.js";
import { parseWorkspaceYaml } from "../src/workspaces/schema.js";
const workspaceV3 = parseWorkspaceYaml(`workspace:
schema_version: 3
const workspaceV4 = parseWorkspaceYaml(`workspace:
schema_version: 4
id: psd-clinical
name: Policlinico San Donato
language: it
@@ -19,10 +19,6 @@ dwh:
database: postgres
schema: datawarehouse
supported_transports: [postgres_direct, rest_api, ssh_tunnel]
semantic_index:
vector_store: { engine: qdrant, collection: psd-clinical, dimensions: 1024, distance: cosine }
embedding: { provider: ollama_internal, model: qwen3-embedding:0.6b, dimensions: 1024 }
llm_policy: { allowed: [zai/glm-5.2] }
`);
const temporaryRoots: string[] = [];
@@ -40,9 +36,9 @@ function secretPath(name: string): { root: string; path: string } {
return { root: secrets, path };
}
test("resolves schema-v3 direct DWH bindings from the stable namespace", () => {
test("resolves workspace-v4 direct DWH bindings from the stable namespace", () => {
const password = secretPath("dwh-password");
const result = resolveBinding(workspaceV3, "DWH", {
const result = resolveBinding(workspaceV4, "DWH", {
THT_WS_PSD_CLINICAL_DWH_TRANSPORT: "postgres_direct",
THT_WS_PSD_CLINICAL_DWH_HOST: "dwh.internal",
THT_WS_PSD_CLINICAL_DWH_PORT: "5432",
@@ -61,14 +57,14 @@ test("resolves schema-v3 direct DWH bindings from the stable namespace", () => {
});
});
test("requires schema-v3 REST credentials unless the DWH diagnostic declares auth none", () => {
expect(resolveBinding(workspaceV3, "DWH", {
test("requires workspace-v4 REST credentials unless the DWH diagnostic declares auth none", () => {
expect(resolveBinding(workspaceV4, "DWH", {
THT_WS_PSD_CLINICAL_DWH_TRANSPORT: "rest_api",
THT_WS_PSD_CLINICAL_DWH_BASE_URL: "https://dwh.example.test",
}, []).missing).toContain("THT_WS_PSD_CLINICAL_DWH_API_KEY_FILE");
const noAuth = parseWorkspaceYaml(`workspace:
schema_version: 3
schema_version: 4
id: psd-clinical
name: No auth
language: en
@@ -77,16 +73,12 @@ dwh:
database: postgres
schema: public
supported_transports: [rest_api]
semantic_index:
vector_store: { engine: qdrant, collection: psd-clinical, dimensions: 1024, distance: cosine }
embedding: { provider: ollama_internal, model: qwen3-embedding:0.6b, dimensions: 1024 }
diagnostics:
dwh_rest:
method: GET
path: /health
auth: none
response: { database: database, schema: schema }
llm_policy: { allowed: [zai/glm-5.2] }
`);
expect(resolveBinding(noAuth, "DWH", {
THT_WS_PSD_CLINICAL_DWH_TRANSPORT: "rest_api",
@@ -98,7 +90,7 @@ test("rejects unsupported transports and secret paths outside configured roots",
const outside = secretPath("outside-password");
const allowed = secretPath("allowed-password");
const directOnly = parseWorkspaceYaml(`workspace:
schema_version: 3
schema_version: 4
id: psd-clinical
name: Direct only
language: en
@@ -107,15 +99,11 @@ dwh:
database: postgres
schema: public
supported_transports: [postgres_direct]
semantic_index:
vector_store: { engine: qdrant, collection: psd-clinical, dimensions: 1024, distance: cosine }
embedding: { provider: ollama_internal, model: qwen3-embedding:0.6b, dimensions: 1024 }
llm_policy: { allowed: [zai/glm-5.2] }
`);
expect(resolveBinding(directOnly, "DWH", {
THT_WS_PSD_CLINICAL_DWH_TRANSPORT: "rest_api",
}, []).missing).toContain("THT_WS_PSD_CLINICAL_DWH_TRANSPORT");
const result = resolveBinding(workspaceV3, "DWH", {
const result = resolveBinding(workspaceV4, "DWH", {
THT_WS_PSD_CLINICAL_DWH_TRANSPORT: "postgres_direct",
THT_WS_PSD_CLINICAL_DWH_HOST: "dwh.internal",
THT_WS_PSD_CLINICAL_DWH_PORT: "5432",
@@ -128,7 +116,7 @@ llm_policy: { allowed: [zai/glm-5.2] }
test("runtime bindings contain only DWH and Evidence roles", () => {
const password = secretPath("dwh-password");
const bindings = resolveRuntimeBindings(workspaceV3, {
const bindings = resolveRuntimeBindings(workspaceV4, {
THT_WS_PSD_CLINICAL_DWH_TRANSPORT: "postgres_direct",
THT_WS_PSD_CLINICAL_DWH_HOST: "dwh.internal",
THT_WS_PSD_CLINICAL_DWH_PORT: "5432",
@@ -143,7 +131,7 @@ test("runtime bindings contain only DWH and Evidence roles", () => {
function withEvidence(source: Record<string, unknown>) {
return parseWorkspaceYaml(`workspace:
schema_version: 3
schema_version: 4
id: psd-clinical
name: Policlinico San Donato
language: it
@@ -152,10 +140,6 @@ dwh:
database: postgres
schema: datawarehouse
supported_transports: [postgres_direct]
semantic_index:
vector_store: { engine: qdrant, collection: psd-clinical, dimensions: 1024, distance: cosine }
embedding: { provider: ollama_internal, model: qwen3-embedding:0.6b, dimensions: 1024 }
llm_policy: { allowed: [zai/glm-5.2] }
evidence:
source: ${JSON.stringify(source)}
`);
@@ -260,7 +244,7 @@ test("rejects relative, missing, directory, unreadable, and escaping symlink Evi
});
test("includes Evidence binding completeness in session runtime support without changing v3 compatibility", () => {
const unsigned = resolveRuntimeBindings(workspaceV3, {}, ["/run/secrets"]);
const unsigned = resolveRuntimeBindings(workspaceV4, {}, ["/run/secrets"]);
expect(unsigned.evidence).toEqual({ values: {}, missing: [] });
expect(supportsSessionRuntime(unsigned)).toBe(true);
+1 -5
View File
@@ -7,7 +7,7 @@ import {
} from "../src/workspaces/catalog.js";
const descriptor = parseWorkspaceYaml(`workspace:
schema_version: 3
schema_version: 4
id: psd
name: Policlinico San Donato
description: Clinical warehouse
@@ -17,10 +17,6 @@ dwh:
database: postgres
schema: datawarehouse
supported_transports: [rest_api]
semantic_index:
vector_store: { engine: qdrant, collection: psd, dimensions: 1024, distance: cosine }
embedding: { provider: ollama_internal, model: qwen3-embedding:0.6b, dimensions: 1024 }
llm_policy: { allowed: [zai/glm-5.2] }
`);
test("parses the strict ordered root catalog", () => {
+11 -20
View File
@@ -5,8 +5,8 @@ import { join } from "node:path";
import { buildInstallationContract, renderWorkspaceDocs } from "../src/workspaces/contracts.js";
import { type CanonicalWorkspace, parseWorkspaceYaml } from "../src/workspaces/schema.js";
const workspaceV3 = parseWorkspaceYaml(`workspace:
schema_version: 3
const workspaceV4 = parseWorkspaceYaml(`workspace:
schema_version: 4
id: psd-clinical
name: Policlinico San Donato
language: it
@@ -15,17 +15,12 @@ dwh:
database: postgres
schema: datawarehouse
supported_transports: [postgres_direct, rest_api, ssh_tunnel]
semantic_index:
vector_store: { engine: qdrant, collection: psd-clinical, dimensions: 1024, distance: cosine }
embedding: { provider: ollama_internal, model: qwen3-embedding:0.6b, dimensions: 1024 }
llm_policy:
allowed: [zai/glm-5.2]
`);
test("schema-v3 installation contracts expose only DWH bindings and no semantic variables", () => {
const contract = buildInstallationContract(workspaceV3);
test("workspace-v4 installation contracts expose only DWH bindings and no semantic variables", () => {
const contract = buildInstallationContract(workspaceV4);
const names = contract.variables.map((variable) => variable.name);
const docs = renderWorkspaceDocs(workspaceV3);
const docs = renderWorkspaceDocs(workspaceV4);
expect(contract.workspaceId).toBe("psd-clinical");
expect(contract.namespace).toBe("PSD_CLINICAL");
@@ -54,22 +49,22 @@ test.each([
test("validates public contract and documentation inputs at runtime", () => {
const unsafeWorkspace = {
...workspaceV3,
workspace: { ...workspaceV3.workspace, id: "psd\nclinical" },
...workspaceV4,
workspace: { ...workspaceV4.workspace, id: "psd\nclinical" },
} as CanonicalWorkspace;
expect(() => buildInstallationContract(unsafeWorkspace)).toThrow(/id/i);
expect(() => renderWorkspaceDocs(unsafeWorkspace)).toThrow(/id/i);
});
test("v3 installation contract omits external vector and embedding bindings", () => {
const contract = buildInstallationContract(workspaceV3);
test("v4 installation contract omits external vector and embedding bindings", () => {
const contract = buildInstallationContract(workspaceV4);
const names = contract.variables.map((variable) => variable.name);
expect(names).toContain("THT_WS_PSD_CLINICAL_DWH_TRANSPORT");
expect(names.some((name) => name.includes("_VECTOR_"))).toBe(false);
expect(names.some((name) => name.includes("_EMBEDDING_"))).toBe(false);
expect(renderWorkspaceDocs(workspaceV3).markdown).not.toContain("Embedding service");
expect(renderWorkspaceDocs(workspaceV4).markdown).not.toContain("Embedding service");
});
@@ -114,7 +109,7 @@ test.each([
function renderWorkspaceWithoutEvidence(): string {
return `workspace:
schema_version: 3
schema_version: 4
id: psd-clinical
name: Policlinico San Donato
language: it
@@ -123,10 +118,6 @@ dwh:
database: postgres
schema: datawarehouse
supported_transports: [postgres_direct]
semantic_index:
vector_store: { engine: qdrant, collection: psd-clinical, dimensions: 1024, distance: cosine }
embedding: { provider: ollama_internal, model: qwen3-embedding:0.6b, dimensions: 1024 }
llm_policy: { allowed: [zai/glm-5.2] }
`;
}
+6 -26
View File
@@ -12,7 +12,7 @@ import type { RuntimeBindings } from "../src/workspaces/runtime-renderer.js";
import { parseWorkspaceYaml, resolveDiagnosticUrl } from "../src/workspaces/schema.js";
const workspace = parseWorkspaceYaml(`workspace:
schema_version: 3
schema_version: 4
id: psd-clinical
name: Policlinico San Donato
language: it
@@ -22,18 +22,6 @@ dwh:
schema: datawarehouse
timeout_ms: 8000
supported_transports: [postgres_direct, rest_api, ssh_tunnel]
semantic_index:
vector_store:
engine: qdrant
collection: psd-clinical
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
llm_policy:
allowed: [zai/glm-5.2]
`);
const bindings: RuntimeBindings = {
@@ -78,7 +66,7 @@ afterEach(() => {
vi.restoreAllMocks();
});
test("diagnoses schema-v3 DWH, internal Qdrant, and internal Ollama without semantic bindings", async () => {
test("diagnoses workspace-v4 DWH plus installation-derived Qdrant and Ollama", async () => {
const adapters = successfulAdapters();
const result = await diagnose(adapters)(workspace, bindings, { writeProbe: false });
@@ -139,7 +127,7 @@ test("reports only sanitized DWH and Evidence binding names before network diagn
expect(adapters.inspectQdrant).not.toHaveBeenCalled();
});
test("keeps schema-v3 DWH SSH diagnostic-only and runtime-inactive", async () => {
test("keeps workspace-v4 DWH SSH diagnostic-only and runtime-inactive", async () => {
const adapters = successfulAdapters();
const result = await diagnose(adapters)(workspace, {
...bindings,
@@ -152,9 +140,9 @@ test("keeps schema-v3 DWH SSH diagnostic-only and runtime-inactive", async () =>
expect(adapters.probeConnector).not.toHaveBeenCalled();
});
test("uses the schema-v3 declared DWH REST diagnostic and auth policy", async () => {
test("uses the workspace-v4 declared DWH REST diagnostic and auth policy", async () => {
const restWorkspace = parseWorkspaceYaml(`workspace:
schema_version: 3
schema_version: 4
id: psd-clinical
name: REST workspace
language: en
@@ -163,16 +151,12 @@ dwh:
database: warehouse
schema: datawarehouse
supported_transports: [rest_api]
semantic_index:
vector_store: { engine: qdrant, collection: psd-clinical, dimensions: 1024, distance: cosine }
embedding: { provider: ollama_internal, model: qwen3-embedding:0.6b, dimensions: 1024 }
diagnostics:
dwh_rest:
method: POST
path: /rpc/ping
auth: bearer
response: { database: database, schema: schema }
llm_policy: { allowed: [zai/glm-5.2] }
`);
const adapters = successfulAdapters();
const result = await diagnose(adapters)(restWorkspace, {
@@ -423,7 +407,7 @@ test("uses a REST secret only as a header and redacts it from failed diagnostics
const canary = "CANARY-REST-AUTH-SECRET";
await writeFile(credentialFile, canary);
const restDescriptor = parseWorkspaceYaml(`workspace:
schema_version: 3
schema_version: 4
id: psd-clinical
name: REST auth
language: en
@@ -432,16 +416,12 @@ dwh:
database: warehouse
schema: datawarehouse
supported_transports: [rest_api]
semantic_index:
vector_store: { engine: qdrant, collection: psd-clinical, dimensions: 1024, distance: cosine }
embedding: { provider: ollama_internal, model: qwen3-embedding:0.6b, dimensions: 1024 }
diagnostics:
dwh_rest:
method: GET
path: /health
auth: bearer
response: { database: database, schema: schema }
llm_policy: { allowed: [zai/glm-5.2] }
`);
const fetchMock = vi.fn()
.mockResolvedValueOnce(new Response("upstream CANARY-REST-AUTH-SECRET", { status: 503 }))
@@ -53,7 +53,7 @@ async function makeRepo(id: string, annotations: string | Buffer | "dir" | "syml
writeFileSync(join(source, "thoth-workspaces.yaml"),
`schema_version: 1\nworkspaces: [{id: ${id}, name: Workspace}]\n`);
mkdirSync(join(source, id, "schema"), { recursive: true });
writeFileSync(join(source, id, "workspace.yaml"), `workspace:\n schema_version: 3\n id: ${id}\n`);
writeFileSync(join(source, id, "workspace.yaml"), `workspace:\n schema_version: 4\n id: ${id}\n`);
const annotationsPath = join(source, id, "schema", "annotations.yaml");
if (annotations === "dir") {
mkdirSync(annotationsPath, { recursive: true });
+1 -1
View File
@@ -47,7 +47,7 @@ async function fixture(layout: EvidenceLayout): Promise<{ root: string; remote:
await git(source, ["config", "user.email", "evidence@example.invalid"]);
writeFileSync(join(source, "thoth-workspaces.yaml"), "schema_version: 1\nworkspaces: [{id: research, name: Research}]\n");
mkdirSync(join(source, "research"), { recursive: true });
writeFileSync(join(source, "research", "workspace.yaml"), "workspace:\n schema_version: 3\n id: research\n");
writeFileSync(join(source, "research", "workspace.yaml"), "workspace:\n schema_version: 4\n id: research\n");
const evidence = join(source, "research", "evidence");
if (layout === "tree") {
mkdirSync(join(evidence, "nested"), { recursive: true });
+1 -16
View File
@@ -12,7 +12,7 @@ import {
import type { WorkspaceRegistryConfig } from "../src/workspaces/types.js";
const validYaml = `workspace:
schema_version: 2
schema_version: 4
id: psd-clinical
name: Policlinico San Donato
language: it
@@ -21,21 +21,6 @@ dwh:
database: postgres
schema: datawarehouse
supported_transports: [postgres_direct]
semantic_index:
vector_store:
engine: pgvector
database: postgres
schema: vectors
collection: clinical_documents
dimensions: 768
distance: cosine
supported_transports: [pgvector_direct]
embedding:
provider: ollama_compatible
model: nomic-embed-text-v2-moe
dimensions: 768
llm_policy:
allowed: [zai/glm-5.2]
`;
const runFile = promisify(execFile);
@@ -13,20 +13,11 @@ import {
import { renderRuntimeConfig, type RuntimeBindings } from "../src/workspaces/runtime-renderer.js";
const unsupportedWorkspace = {
workspace: { schema_version: 2, id: "legacy-workspace", name: "Legacy", language: "en" },
workspace: { schema_version: 3, id: "legacy-workspace", name: "Legacy", language: "en" },
dwh: {
engine: "postgres", database: "warehouse", schema: "public",
supported_transports: ["postgres_direct"],
},
semantic_index: {
vector_store: {
engine: "pgvector", database: "warehouse", schema: "vectors",
collection: "documents", dimensions: 768, distance: "cosine",
supported_transports: ["pgvector_direct"],
},
embedding: { provider: "ollama_compatible", model: "legacy", dimensions: 768 },
},
llm_policy: { allowed: ["zai/glm-5.2"] },
};
const bindings: RuntimeBindings = {
@@ -40,31 +31,31 @@ const adapters: DiagnosticAdapters = {
probeEmbedding: vi.fn(),
};
test("renderer rejects callers that bypass the schema-v3 type contract", () => {
test("renderer rejects callers that bypass the schema-v4 type contract", () => {
expect(() => renderRuntimeConfig(unsupportedWorkspace as never, bindings, {
sessions: "/data/sessions", artifacts: "/data/artifacts", indexes: "/data/indexes",
})).toThrow("Runtime renderer supports only workspace schema version 3");
})).toThrow("Runtime renderer supports only workspace schema version 4");
});
test("installation contract rejects callers that bypass the schema-v3 type contract", () => {
test("installation contract rejects callers that bypass the schema-v4 type contract", () => {
expect(() => buildInstallationContract(unsupportedWorkspace as never))
.toThrow("Installation contract supports only workspace schema version 3");
.toThrow("Installation contract supports only workspace schema version 4");
});
test("binding entry points reject callers that bypass the schema-v3 type contract", () => {
test("binding entry points reject callers that bypass the schema-v4 type contract", () => {
expect(() => resolveBinding(unsupportedWorkspace as never, "DWH", {}, []))
.toThrow("Workspace bindings support only workspace schema version 3");
.toThrow("Workspace bindings support only workspace schema version 4");
expect(() => resolveEvidenceBinding(unsupportedWorkspace as never, {}, []))
.toThrow("Workspace bindings support only workspace schema version 3");
.toThrow("Workspace bindings support only workspace schema version 4");
expect(() => resolveRuntimeBindings(unsupportedWorkspace as never, {}, []))
.toThrow("Workspace bindings support only workspace schema version 3");
.toThrow("Workspace bindings support only workspace schema version 4");
});
test("diagnoser factories reject callers that bypass the schema-v3 type contract", async () => {
test("diagnoser factories reject callers that bypass the schema-v4 type contract", async () => {
await expect(createWorkspaceDiagnoser(adapters)(unsupportedWorkspace as never, bindings, {
writeProbe: false,
})).rejects.toThrow("Workspace diagnoser supports only workspace schema version 3");
})).rejects.toThrow("Workspace diagnoser supports only workspace schema version 4");
await expect(createProductionWorkspaceDiagnoser(5_000, adapters)(
unsupportedWorkspace as never, bindings, { writeProbe: false },
)).rejects.toThrow("Workspace diagnoser supports only workspace schema version 3");
)).rejects.toThrow("Workspace diagnoser supports only workspace schema version 4");
});
+17 -103
View File
@@ -10,7 +10,7 @@ import {
} from "../src/workspaces/schema.js";
export const validYaml = `workspace:
schema_version: 3
schema_version: 4
id: psd-clinical
name: Policlinico San Donato
description: Clinical data warehouse workspace
@@ -25,33 +25,8 @@ dwh:
- postgres_direct
- rest_api
- ssh_tunnel
semantic_index:
vector_store:
engine: qdrant
collection: psd-clinical
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
llm_policy:
default: zai/glm-5.2
allowed:
- zai/glm-5.2
- openai/gpt-5
`;
test("rejects a workspace whose embedding dimensions differ from its collection", () => {
expect(() => parseWorkspaceYaml(validYaml.replace("dimensions: 1024", "dimensions: 1536")))
.toThrow(/dimensions/i);
});
test("rejects an LLM default outside its allowlist", () => {
expect(() => parseWorkspaceYaml(validYaml.replace("- zai/glm-5.2", "- openai/gpt-5")))
.toThrow(/allowlist/i);
});
test("rejects unknown keys and invalid immutable IDs", () => {
expect(() => parseWorkspaceYaml(validYaml.replace(" language: it", " language: it\n label: PSD")))
.toThrow(/unrecognized key/i);
@@ -74,94 +49,34 @@ test("accepts optional connection ports and timeouts but rejects unsafe values",
.toThrow(/port/i);
expect(() => parseWorkspaceYaml(validYaml.replace("timeout_ms: 5000", "timeout_ms: 0")))
.toThrow(/timeout/i);
expect(() => parseWorkspaceYaml(validYaml.replace("dimensions: 1024", "dimensions: 2048")))
.toThrow(/1024|dimensions/i);
});
test("accepts only the schema v3 internal qdrant semantic shape", () => {
test("accepts a model-free schema v4 workspace", () => {
expect(parseWorkspaceYaml(validYaml)).toMatchObject({
workspace: { schema_version: 3, id: "psd-clinical" },
semantic_index: {
vector_store: {
engine: "qdrant",
collection: "psd-clinical",
dimensions: 1024,
distance: "cosine",
},
embedding: {
provider: "ollama_internal",
model: "qwen3-embedding:0.6b",
dimensions: 1024,
},
},
workspace: { schema_version: 4, id: "psd-clinical" },
dwh: { database: "postgres", schema: "datawarehouse" },
});
});
test("committed example descriptors parse as exact schema v3 workspaces", () => {
test("committed example descriptors parse as exact schema v4 workspaces", () => {
const example = readFileSync(resolve(process.cwd(), "../deploy/workspaces/example.yaml"), "utf8");
const psdExample = readFileSync(resolve(process.cwd(), "../deploy/workspaces/psd.yaml.example"), "utf8");
expect(() => parseWorkspaceYaml(example)).not.toThrow();
expect(() => parseWorkspaceYaml(psdExample)).not.toThrow();
expect(parseWorkspaceYaml(example)).toMatchObject({
workspace: { schema_version: 3 },
semantic_index: {
vector_store: { engine: "qdrant", distance: "cosine", dimensions: 1024 },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
workspace: { schema_version: 4 },
});
expect(parseWorkspaceYaml(psdExample)).toMatchObject({
workspace: { schema_version: 3 },
semantic_index: {
vector_store: { engine: "qdrant", distance: "cosine", dimensions: 1024 },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
workspace: { schema_version: 4 },
});
});
test("rejects pgvector semantic stores in schema v3", () => {
expect(() => parseWorkspaceYaml(validYaml.replace("engine: qdrant", "engine: pgvector")))
.toThrow(/qdrant|pgvector/i);
});
test("rejects supported_transports inside schema v3 semantic identity", () => {
const withTransport = validYaml.replace(
" distance: cosine\n",
" distance: cosine\n supported_transports:\n - rest_api\n",
);
expect(() => parseWorkspaceYaml(withTransport)).toThrow(/unrecognized key|supported_transports/i);
});
test("rejects external embedding providers in schema v3", () => {
expect(() => parseWorkspaceYaml(validYaml.replace("provider: ollama_internal", "provider: openai_compatible")))
.toThrow(/ollama_internal|provider/i);
});
test("rejects non-cosine distance in schema v3", () => {
expect(() => parseWorkspaceYaml(validYaml.replace("distance: cosine", "distance: l2")))
.toThrow(/cosine|distance/i);
});
test("rejects unknown fields in schema v3 semantic identity", () => {
const withUnknownField = validYaml.replace(
" collection: psd-clinical\n",
" collection: psd-clinical\n namespace: psd\n",
);
expect(() => parseWorkspaceYaml(withUnknownField)).toThrow(/unrecognized key/i);
});
test("rejects legacy semantic connector fields and diagnostics in schema v3", () => {
expect(() => parseWorkspaceYaml(validYaml.replace(
" collection: psd-clinical\n",
" collection: psd-clinical\n database: postgres\n",
))).toThrow(/unrecognized key|database/i);
expect(() => parseWorkspaceYaml(validYaml.replace(
"llm_policy:\n",
"diagnostics:\n vector_rest:\n metadata:\n method: GET\n path: /metadata\n auth: bearer\n response:\n collection: collection\n dimensions: dimensions\n distance: distance\nllm_policy:\n",
))).toThrow(/unrecognized key|vector_rest/i);
test("rejects installation-owned model and vector fields", () => {
expect(() => parseWorkspaceYaml(`${validYaml}llm_policy:\n allowed: [zai/glm-5.3]\n`))
.toThrow(/unrecognized key|llm_policy/i);
expect(() => parseWorkspaceYaml(`${validYaml}semantic_index: {}\n`))
.toThrow(/unrecognized key|semantic_index/i);
});
test.each([
@@ -222,8 +137,8 @@ llm_policy:
- zai/glm-5.2
`],
])("rejects schema %s descriptors at parser and object-validator boundaries", (_version, yaml) => {
expect(() => parseWorkspaceYaml(yaml)).toThrow(/schema_version|invalid literal|3/i);
expect(() => validateWorkspaceDescriptor(parse(yaml))).toThrow(/schema_version|invalid literal|3/i);
expect(() => parseWorkspaceYaml(yaml)).toThrow(/schema_version|invalid literal|4/i);
expect(() => validateWorkspaceDescriptor(parse(yaml))).toThrow(/schema_version|invalid literal|4/i);
});
test("does not expose the redundant canonical validator or v1 migration", () => {
@@ -253,9 +168,8 @@ test("rejects REST diagnostic declarations without their matching connector tran
response:
database: database
schema: schema
llm_policy:
`;
const declared = validYaml.replace("llm_policy:\n", diagnostics);
const declared = `${validYaml}${diagnostics}`;
expect(() => parseWorkspaceYaml(declared.replace(" - rest_api\n", ""))).toThrow(/dwh_rest/i);
});
@@ -462,7 +376,7 @@ test.each(["curated/**/*.yaml", "curated/**"])(
},
);
test("keeps evidence optional on schema v3", () => {
test("keeps evidence optional on schema v4", () => {
expect(validateWorkspaceDescriptor(validWorkspaceObject())).not.toHaveProperty("evidence");
});
@@ -471,7 +385,7 @@ test("serializes defaulted evidence canonically and parses it without loss", ()
type: "filesystem",
uri: "psd-clinical/evidence",
}));
if (canonical.workspace.schema_version !== 3) throw new Error("expected schema v3");
if (canonical.workspace.schema_version !== 4) throw new Error("expected schema v4");
expect(parseWorkspaceYaml(serializeWorkspaceYaml(canonical))).toEqual(canonical);
});
@@ -46,7 +46,7 @@ async function fixture(): Promise<{ root: string; remote: string; commit: string
await git(source, ["config", "user.email", "evidence-materializer@example.invalid"]);
writeFileSync(join(source, "thoth-workspaces.yaml"), "schema_version: 1\nworkspaces: [{id: research, name: Research}]\n");
mkdirSync(join(source, "research", "evidence", "nested"), { recursive: true });
writeFileSync(join(source, "research", "workspace.yaml"), "workspace:\n schema_version: 3\n id: research\n");
writeFileSync(join(source, "research", "workspace.yaml"), "workspace:\n schema_version: 4\n id: research\n");
writeFileSync(join(source, "research", "evidence", "guide.md"), "# guide\n");
writeFileSync(join(source, "research", "evidence", "nested", "deep.md"), "# deep\n");
await git(source, ["add", "-A"]);
@@ -102,7 +102,7 @@ test("refuses symlink-containing trees and bound violations without publishing",
await git(source, ["config", "user.email", "e@e.invalid"]);
writeFileSync(join(source, "thoth-workspaces.yaml"), "schema_version: 1\nworkspaces: [{id: research, name: Research}]\n");
mkdirSync(join(source, "research", "evidence"), { recursive: true });
writeFileSync(join(source, "research", "workspace.yaml"), "workspace:\n schema_version: 3\n id: research\n");
writeFileSync(join(source, "research", "workspace.yaml"), "workspace:\n schema_version: 4\n id: research\n");
writeFileSync(join(source, "research", "outside.md"), "# outside\n");
symlinkSync("../outside.md", join(source, "research", "evidence", "link.md"));
await git(source, ["add", "-A"]);
-7
View File
@@ -36,15 +36,11 @@ services:
THT_LLM_URL: ${THT_LLM_URL:-}
THT_INTERNAL_QDRANT_URL: http://qdrant:6333
THT_INTERNAL_EMBEDDING_URL: http://embedding:11434
THT_INTERNAL_EMBEDDING_MODEL: qwen3-embedding:0.6b
THT_INTERNAL_EMBEDDING_DIMENSIONS: "1024"
MAX_PI_PROCESSES: ${MAX_PI_PROCESSES:-4}
volumes:
- settings:/data/settings
- pi-state:/home/thoth/.pi
- ${PI_AUTH_FILE:?set PI_AUTH_FILE}:/home/thoth/.pi/agent/auth.json:ro
- ./deploy/pi/models.json:/home/thoth/.pi/agent/models.json:ro
- ./deploy/pi/settings.json:/home/thoth/.pi/agent/settings.json:ro
- workspace-registry:/data/workspace-registry
- workspace-secrets:/data/workspace-secrets
- sessions:/data/sessions
@@ -140,8 +136,6 @@ services:
THT_LLM_URL: ${THT_LLM_URL:-}
THT_INTERNAL_QDRANT_URL: http://qdrant:6333
THT_INTERNAL_EMBEDDING_URL: http://embedding:11434
THT_INTERNAL_EMBEDDING_MODEL: qwen3-embedding:0.6b
THT_INTERNAL_EMBEDDING_DIMENSIONS: "1024"
HOME: /tmp/thoth
AWS_ACCESS_KEY_ID: ""
AWS_SECRET_ACCESS_KEY: ""
@@ -243,7 +237,6 @@ services:
entrypoint: ["/usr/bin/bash", "/opt/thoth/embedding-model-init.sh"]
environment:
OLLAMA_BASE_URL: http://embedding:11434
OLLAMA_MODEL: qwen3-embedding:0.6b
OLLAMA_WAIT_TIMEOUT_SEC: "180"
volumes:
- embedding-models:/root/.ollama
-8
View File
@@ -21,14 +21,6 @@ services:
source: ${PI_AUTH_FILE:?set PI_AUTH_FILE}
target: /home/thoth/.pi/agent/auth.json
read_only: true
- type: bind
source: ./deploy/pi/models.json
target: /home/thoth/.pi/agent/models.json
read_only: true
- type: bind
source: ./deploy/pi/settings.json
target: /home/thoth/.pi/agent/settings.json
read_only: true
- type: bind
source: ${THT_WORKSPACE_REGISTRY_ROOT:?set THT_WORKSPACE_REGISTRY_ROOT}
target: /data/workspace-registry
-46
View File
@@ -1,46 +0,0 @@
{
"providers": {
"zai": {
"baseUrl": "https://api.z.ai/api/coding/paas/v4",
"api": "openai-completions",
"apiKey": "$ZAI_API_KEY",
"models": [
{
"id": "glm-5.3",
"name": "GLM-5.3",
"reasoning": true,
"contextWindow": 200000,
"maxTokens": 131072
}
]
},
"local-qwen": {
"name": "Local Qwen",
"baseUrl": "https://ml-aritmolab.policlinicosandonato.it/v1",
"api": "openai-completions",
"apiKey": "local",
"models": [
{
"id": "qwen3.6-35b-a3b",
"name": "Qwen3.6 35B A3B",
"reasoning": false,
"input": ["text"],
"cost": {
"input": 0,
"output": 0,
"cacheRead": 0,
"cacheWrite": 0
},
"contextWindow": 131072,
"maxTokens": 16384,
"compat": {
"supportsDeveloperRole": false,
"supportsReasoningEffort": false,
"supportsStore": false,
"maxTokensField": "max_tokens"
}
}
]
}
}
}
-9
View File
@@ -1,9 +0,0 @@
{
"defaultProjectTrust": "always",
"enabledModels": [
"zai/glm-5.3",
"deepseek/deepseek-v4-flash",
"deepseek/deepseek-v4-pro",
"local-qwen/qwen3.6-35b-a3b"
]
}
+1 -3
View File
@@ -14,9 +14,7 @@ THT_AUTH_CONFIG_ROOT=<abs>/deploy/psd/auth
# DWH and Evidence credentials are entered later in Workspace management and stored encrypted
# by the backend. They do not depend on host filesystem paths.
# Pi (LLM)
PI_PROVIDER=zai
PI_MODEL=glm-5.3
# Pi runtime preference; provider/model defaults live only in installation modelCatalog.
PI_THINKING=medium
# App defaults
+64 -35
View File
@@ -1,6 +1,6 @@
# Esempio soltanto: `tht setup` genera deploy/<installation-id>/thothii-installation.yaml.
# Sostituisci i path assoluti se usi questo riferimento per una configurazione avanzata.
# Seleziona UN solo override Git (https o ssh).
# Example only: `tht setup` generates deploy/<installation-id>/thothii-installation.yaml.
# Replace every absolute path before using this as an advanced reference.
schemaVersion: 2
profile: local
projectDirectory: "<abs>/projects/ThothII"
envFile: "<abs>/projects/ThothII/deploy/psd/operator.env"
@@ -8,38 +8,67 @@ workspaceRepository:
remote: git@github.com:mptyl/tht-workspace-psd.git
branch: main
access: ssh
metadataGeneration:
default: glm-53
models:
- id: deepseek-v4-pro
label: DeepSeek V4 Pro
litellm:
provider: deepseek
model: deepseek-v4-pro
apiKeyEnv: DEEPSEEK_API_KEY
- id: deepseek-v4-flash
label: DeepSeek V4 Flash
litellm:
provider: deepseek
model: deepseek-v4-flash
apiKeyEnv: DEEPSEEK_API_KEY
- id: glm-53
label: GLM 5.3
litellm:
provider: openai
model: glm-5.3
endpoint:
baseUrl: https://api.z.ai/api/coding/paas/v4
apiKeyEnv: ZAI_API_KEY
- id: qwen-36
label: AritmoLab Qwen 3.6 35B A3B
litellm:
provider: openai
model: qwen3.6-35b-a3b
disableThinking: true
endpoint:
baseUrl: https://ml-aritmolab.policlinicosandonato.it/v1
# Qwen omette apiKeyEnv: l'endpoint VPN non autentica le richieste.
modelCatalog:
defaults:
session: zai/glm-5.3
metadataGeneration: zai/glm-5.3
embedding:
id: ollama/qwen3-embedding:0.6b
dimensions: 1024
providers:
deepseek:
authentication:
mode: pi_auth
session:
mode: pi_builtin
models:
deepseek-v4-pro:
session: {}
deepseek-v4-flash:
session: {}
zai:
endpoint:
baseUrl: https://api.z.ai/api/coding/paas/v4
authentication:
mode: secret_env
apiKeyEnv: ZAI_API_KEY
session:
mode: openai_compatible
metadataGeneration:
litellmProvider: openai
models:
glm-5.3:
label: GLM 5.3
session:
reasoning: true
contextWindow: 200000
maxTokens: 131072
metadataGeneration: {}
local-qwen:
endpoint:
baseUrl: https://ml-aritmolab.policlinicosandonato.it/v1
authentication:
mode: none
session:
mode: openai_compatible
metadataGeneration:
litellmProvider: openai
models:
qwen3.6-35b-a3b:
label: AritmoLab Qwen 3.6 35B A3B
session:
reasoning: false
input: [text]
cost: {input: 0, output: 0, cacheRead: 0, cacheWrite: 0}
contextWindow: 131072
maxTokens: 16384
compatibility:
supportsDeveloperRole: false
supportsReasoningEffort: false
supportsStore: false
maxTokensField: max_tokens
metadataGeneration:
disableThinking: true
authentication:
configDirectory: "<abs>/projects/ThothII/deploy/psd/auth"
overrides:
+8 -7
View File
@@ -10,8 +10,9 @@ chmod 600 deploy/secrets/thothii.secrets
The file uses strict `KEY=VALUE` lines (comments and blank lines are allowed). The supported
installation keys are `THT_MODEL_API_KEY`, `THT_DWH_API_KEY`, `THT_CA`, `THT_SSL_CA`,
`THT_OIDC_CLIENT_SECRET`, and `THT_AUTHENTIK_API_TOKEN`. Metadata-generation models may reference
exactly one of `THT_METADATA_API_KEY`, `ANTHROPIC_API_KEY`, `AZURE_API_KEY`, `GEMINI_API_KEY`,
`THT_OIDC_CLIENT_SECRET`, and `THT_AUTHENTIK_API_TOKEN`. Installation Model Catalog providers may
reference exactly one of `THT_MODEL_API_KEY`, `THT_METADATA_API_KEY`, `ANTHROPIC_API_KEY`,
`AZURE_API_KEY`, `GEMINI_API_KEY`,
`DEEPSEEK_API_KEY`, `OPENAI_API_KEY`, `OPENROUTER_API_KEY`, or `ZAI_API_KEY` through their
descriptor `apiKeyEnv`. An entry for an explicitly configured endpoint that accepts unauthenticated
requests may omit `apiKeyEnv`; hosted/default endpoints must always reference a key.
@@ -23,10 +24,10 @@ installation. Other configured values must be non-empty and contain no
whitespace. Do not put secrets in the root `.env`, installation YAML, workspace YAML, URLs, logs,
or rendered Compose output.
`THT_MODEL_API_KEY` remains the generic Pi child credential. Metadata generation is a separate
backend-owned runtime and reads only the key named by its own `metadataGeneration.models[].apiKeyEnv`
(when present);
it does not read Pi settings, `PI_AUTH_FILE`, or workspace `llm_policy`.
Session and metadata-generation runtimes read only the provider key named by
`modelCatalog.providers.<provider>.authentication.apiKeyEnv`. They share the declaration and
credential reference, not their execution lifecycle. Pi-owned authentication remains available only
to session-only built-in providers through `authentication.mode: pi_auth`.
Do not add vector or embedding endpoint credentials to the bundle. Active operator manuals use
internal Qdrant and Ollama services, so vector/embedding runtime endpoint secrets are not part of
@@ -56,7 +57,7 @@ and only then deleting the old files. The old variables remain a compatibility p
upgrades, but the documented and tested default is an absolute `THT_SECRETS_FILE` path to the
protected bundle.
Hosted Pi providers must use a single model key through `THT_MODEL_API_KEY`. Compound providers
Hosted providers must use one explicitly named catalog key. Compound providers
(Bedrock, Azure OpenAI Responses, Cloudflare Workers AI/Gateway) fail closed until a
provider-specific credential adapter is implemented.
+5 -6
View File
@@ -2,14 +2,13 @@
# Values are read as literal strings (no shell expansion or command substitution).
# Leave unused keys out of the file.
# Hosted model provider (single-key providers only).
# THT_MODEL_API_KEY=replace-me
# Description Generation only. The selected name must match apiKeyEnv in metadataGeneration.
# Allowed names: THT_METADATA_API_KEY, ANTHROPIC_API_KEY, AZURE_API_KEY, GEMINI_API_KEY,
# Hosted catalog provider (single-key providers only). The selected name must match
# modelCatalog.providers.<provider>.authentication.apiKeyEnv.
# Allowed names include THT_MODEL_API_KEY, THT_METADATA_API_KEY, ANTHROPIC_API_KEY,
# AZURE_API_KEY, GEMINI_API_KEY,
# DEEPSEEK_API_KEY, OPENAI_API_KEY, OPENROUTER_API_KEY, or ZAI_API_KEY.
# OPENAI_API_KEY=replace-me
# A model with an explicit unauthenticated endpoint omits apiKeyEnv and needs no bundle entry.
# A provider with an explicit keyless endpoint uses authentication.mode: none.
# External DWH adapter.
# THT_DWH_API_KEY=replace-me
+2 -18
View File
@@ -1,8 +1,8 @@
workspace:
schema_version: 3
schema_version: 4
id: example
name: Example workspace
description: Generic example WorkspaceV3 descriptor.
description: Generic example WorkspaceV4 descriptor.
language: en
dwh:
@@ -13,17 +13,6 @@ dwh:
- postgres_direct
- rest_api
semantic_index:
vector_store:
engine: qdrant
collection: example
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
evidence:
source:
type: filesystem
@@ -35,11 +24,6 @@ evidence:
max_chunk_chars: 4000
retain_published_generations: 3
llm_policy:
default: zai/glm-5.2
allowed:
- zai/glm-5.2
diagnostics:
dwh_rest:
method: GET
+2 -18
View File
@@ -1,8 +1,8 @@
workspace:
schema_version: 3
schema_version: 4
id: example-workspace
name: Example Workspace
description: Example WorkspaceV3 descriptor.
description: Example WorkspaceV4 descriptor.
language: en
dwh:
@@ -13,17 +13,6 @@ dwh:
- postgres_direct
- rest_api
semantic_index:
vector_store:
engine: qdrant
collection: example-workspace
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
evidence:
source:
type: filesystem
@@ -35,11 +24,6 @@ evidence:
max_chunk_chars: 4000
retain_published_generations: 3
llm_policy:
default: zai/glm-5.2
allowed:
- zai/glm-5.2
diagnostics:
dwh_rest:
method: GET
+2 -8
View File
@@ -1,6 +1,5 @@
# ThothII standalone development/smoke stack.
# Run with the canonical local env file:
# docker compose --env-file deploy/env/local.env -f docker-compose.dev.yml up -d --build
# ThothII standalone development/smoke base. Model settings are intentionally absent;
# use `tht start --build`, which adds the generated modelCatalog projection.
# frontend: http://localhost:8090 backend: http://localhost:8787
name: thothii-dev
@@ -36,8 +35,6 @@ services:
THT_LLM_URL: ${THT_LLM_URL:-}
THT_INTERNAL_QDRANT_URL: http://qdrant:6333
THT_INTERNAL_EMBEDDING_URL: http://embedding:11434
THT_INTERNAL_EMBEDDING_MODEL: qwen3-embedding:0.6b
THT_INTERNAL_EMBEDDING_DIMENSIONS: "1024"
MAX_PI_PROCESSES: ${MAX_PI_PROCESSES:-4}
extra_hosts:
- "host.docker.internal:host-gateway"
@@ -45,8 +42,6 @@ services:
- dev-data:/data
- dev-pi-state:/home/thoth/.pi
- ${PI_AUTH_FILE:?set PI_AUTH_FILE}:/home/thoth/.pi/agent/auth.json:ro
- ./deploy/pi/models.json:/home/thoth/.pi/agent/models.json:ro
- ./deploy/pi/settings.json:/home/thoth/.pi/agent/settings.json:ro
- workspace-registry:/data/workspace-registry
- workspace-secrets:/data/workspace-secrets
- ${THT_DEV_EVIDENCE_HOST_PATH:-./evidence}:/data/evidence:ro
@@ -129,7 +124,6 @@ services:
entrypoint: ["/usr/bin/bash", "/opt/thoth/embedding-model-init.sh"]
environment:
OLLAMA_BASE_URL: http://embedding:11434
OLLAMA_MODEL: qwen3-embedding:0.6b
OLLAMA_WAIT_TIMEOUT_SEC: "180"
volumes:
- embedding-models:/root/.ollama

Some files were not shown because too many files have changed in this diff Show More