feat: unify installation model catalog
This commit is contained in:
Codex
@@ -20,11 +20,12 @@ flowchart LR
|
||||
THT --> FS["Sessions and artifacts\nworkspace repository"]
|
||||
THT --> DWH["DWH\nread-only"]
|
||||
THT --> VDB["Qdrant / vector store"]
|
||||
BE --> CFG["settings.json\nworkspace registry"]
|
||||
BE --> CFG["settings.json\nworkspace + thinking"]
|
||||
BE --> MODELS["generated runtime catalog\nfrom installation YAML"]
|
||||
BE --> CAT["catalog-db\nPostgreSQL + Kysely"]
|
||||
BE -->|catalog Test + Sync + bounded AI sampling| DWH
|
||||
BE -->|one request per subprocess| LLMHELPER["LiteLLM helper\nPython, short-lived"]
|
||||
LLMHELPER -->|configured model| PROVIDER["AI provider"]
|
||||
LLMHELPER -->|catalog-selected model| PROVIDER["AI provider"]
|
||||
FE -.->|renders widgets| EXT
|
||||
```
|
||||
|
||||
|
||||
@@ -58,8 +58,9 @@ A session is a directory under `sessions/` (the workspace defines the path): `se
|
||||
the configured DWH connection; `ProcessModelCompleter` invokes the short-lived Python LiteLLM
|
||||
helper with the installation-selected model.
|
||||
|
||||
Application settings remain in `backend/data/settings.json`; session state remains in harness phase
|
||||
documents. PostgreSQL stores only the administrative database catalog, bindings, observed tables,
|
||||
Application settings keep only workspace and thinking preferences in `backend/data/settings.json`;
|
||||
provider/model defaults come from the generated Installation Model Catalog. Session state remains in
|
||||
harness phase documents. PostgreSQL stores only the administrative database catalog, bindings, observed tables,
|
||||
curated and generated descriptions, description-generation runs, and sanitized run events.
|
||||
Connector secrets remain write-only in the encrypted workspace secret store; model credentials
|
||||
remain in the protected installation secret bundle. Catalog SSH support is limited to connection
|
||||
@@ -97,7 +98,8 @@ operations that perform this upgrade.
|
||||
- `--json` output must be plain JSON on stdout. It is a machine-readable contract.
|
||||
- UI strings are in English. Document *content* stays in the workspace language because it is the actual data; only chrome and labels are in English.
|
||||
- Each workspace defines its DWH target and working directories. Secrets remain in protected installation files, not in the workspace repository.
|
||||
- Settings are global (`backend/data/settings.json`: workspace/provider/model/thinking); the new-session form asks only for the question.
|
||||
- Settings are global (`backend/data/settings.json`: workspace/thinking); provider/model choices are
|
||||
canonical catalog references selected per session.
|
||||
- **Resume**: a resumable session returns to its last incomplete phase. The backend rejects resume with 409 when `finalized` or `archived`; `PiProcessManager.spawnFor` must send `/riprendi-sessione <id>` for resume and `/nuova-domanda` for a new session. The wrong prompt silently turns a resume into a new question.
|
||||
|
||||
## Runtime composition
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Workspace Evidence v3 contract
|
||||
# Workspace Evidence contract
|
||||
|
||||
This is the canonical public contract for the optional `evidence` object in a schema-v3
|
||||
workspace descriptor. Evidence is optional: a valid v3 descriptor without it remains operational.
|
||||
This is the canonical public contract for the optional `evidence` object in a schema-v4
|
||||
workspace descriptor. Evidence is optional: a valid v4 descriptor without it remains operational.
|
||||
When present, `evidence` is strict: it contains `source` and a defaulted strict `policy`; every
|
||||
source variant and the policy reject unknown keys.
|
||||
|
||||
|
||||
+128
-177
@@ -1,207 +1,158 @@
|
||||
# Local Pi model configuration
|
||||
# Installation Model Catalog
|
||||
|
||||
Pi orchestrates the NL-to-SQL workflow and resolves built-in models and OpenAI-compatible providers
|
||||
declared in the local catalog. ThothII applies a stricter rule than Pi: code in
|
||||
`harness/.pi/extensions/` cannot register a provider or custom model. Endpoints, protocols,
|
||||
compatibility settings, and model identifiers belong exclusively in the local files
|
||||
`deploy/pi/models.json` and `deploy/pi/settings.json`.
|
||||
ThothII has one operator-authored model source: `modelCatalog` in
|
||||
`deploy/<installation-id>/thothii-installation.yaml`. It declares models used by interactive Pi
|
||||
sessions, metadata generation, and the internal embedding service. Workspace descriptors never
|
||||
declare providers, model allowlists, defaults, embeddings, dimensions, or vector-store settings.
|
||||
|
||||
> **ThothII operator note:** ThothII runs Pi only in Docker Compose. Paths under
|
||||
> `~/.pi/agent/` in this document describe Pi's container-side behavior. Operators edit
|
||||
> `deploy/pi/models.json` and `deploy/pi/settings.json` in the ThothII project root and use
|
||||
> the protected host credential file selected by `PI_AUTH_FILE`; they do not edit files inside
|
||||
> the running container.
|
||||
Do not edit `deploy/pi/models.json`, `deploy/pi/settings.json`, files under `generated/`, or
|
||||
provider/model environment defaults. Those former sources are retired.
|
||||
|
||||
## Credentials in the backend container
|
||||
## Minimal catalog
|
||||
|
||||
In production, configure one generic source: the `THT_SECRETS_FILE` bundle with the
|
||||
`THT_MODEL_API_KEY` entry, which is the default for the unified distribution, or
|
||||
`THT_MODEL_API_KEY_FILE` as an absolute secret-file path. Do not put the key value in `.env`.
|
||||
For each process, `PiProcessManager` rereads and validates the source, normalizes the selected
|
||||
provider, and passes only the appropriate native variable to the Pi child
|
||||
(`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GEMINI_API_KEY`, `ZAI_API_KEY`, and so on). It removes the
|
||||
generic path, keys for unselected providers, and the old `PI_PROVIDER_API_KEY` from the child
|
||||
environment. For a custom provider, the backend derives the variable name from the declarative
|
||||
`apiKey` field in `models.json`; a literal value means the catalog is self-contained. Provider-name
|
||||
exceptions are not compiled into the code. A missing or insecure secret fails before spawn with a
|
||||
sanitized error.
|
||||
```yaml
|
||||
schemaVersion: 2
|
||||
modelCatalog:
|
||||
defaults:
|
||||
session: zai/glm-5.3
|
||||
metadataGeneration: zai/glm-5.3
|
||||
|
||||
The generic source supports only single-key providers: `ant-ling`, `anthropic`,
|
||||
`cerebras`, `deepseek`, `fireworks`, `github-copilot`, `google` (including the `gemini` alias),
|
||||
`google-vertex` in API-key mode, `groq`, `huggingface`, `kimi-coding`, `minimax`, `minimax-cn`,
|
||||
`mistral`, `moonshotai`, `moonshotai-cn`, `nvidia`, `openai`, `opencode`, `opencode-go`,
|
||||
`openrouter`, `together`, `vercel-ai-gateway`, `xai`, the four `xiaomi*` providers, `zai`, and
|
||||
`zai-coding-cn`.
|
||||
embedding:
|
||||
id: ollama/qwen3-embedding:0.6b
|
||||
dimensions: 1024
|
||||
|
||||
The composite providers `amazon-bedrock`, `azure-openai-responses`, `cloudflare-workers-ai`, and
|
||||
`cloudflare-ai-gateway` cannot be represented by one file. Selection fails before spawn, including
|
||||
when models are listed. AWS, Azure, and Cloudflare environment credentials are still removed. A
|
||||
future provider-specific configuration will be needed to support these bundles unambiguously.
|
||||
|
||||
## Model sources
|
||||
|
||||
### 1. Built-in (compiled into Pi)
|
||||
|
||||
Pi ships with a list of known models (`models.generated.js` in the `@earendil-works/pi-ai` package):
|
||||
Anthropic, OpenAI, Google, and third-party providers with well-known public APIs such as DeepSeek.
|
||||
These require **no configuration** beyond credentials, provided through an environment variable or
|
||||
`pi auth`.
|
||||
|
||||
`deepseek/deepseek-v4-pro` is one of these models. It is included in Pi because `api.deepseek.com`
|
||||
is a documented public API, not an internal endpoint.
|
||||
|
||||
### 2. User-level `models.json` (`~/.pi/agent/models.json`)
|
||||
|
||||
For an **OpenAI-compatible** endpoint that is not built in but needs no special transport logic,
|
||||
declare it with its baseUrl, apiKey, and model list. This file exists **only at user level**; there is
|
||||
no project-level equivalent, and `./.pi/models.json` is not read.
|
||||
|
||||
In ThothII-managed installations, the file must be entirely declarative. ThothII recursively
|
||||
rejects any JSON value that starts with `!`, including values inside `headers`, `models`,
|
||||
`modelOverrides`, `compat`, arrays, or fields it does not yet know. Pi 0.80.3 would treat that
|
||||
prefix as a shell command when making a request, so managed model catalogs do not allow it. The
|
||||
returned error is fixed and contains no command, path, or secret.
|
||||
|
||||
For secrets, use an environment reference such as `"$ZAI_API_KEY"` or `"${ZAI_API_KEY}"`. The
|
||||
backend can populate the native variable for the selected provider by reading
|
||||
`THT_MODEL_API_KEY_FILE`, or from the `THT_SECRETS_FILE` bundle (`THT_MODEL_API_KEY`). Credentials
|
||||
can also come from the protected file mounted through `PI_AUTH_FILE`, with `apiKey` omitted from
|
||||
`models.json`. There is no direct secret-file reference syntax in `models.json`. With `THT_MODEL_*`
|
||||
sources, ThothII reads the file and turns it into the Pi child's environment variable; `PI_AUTH_FILE`
|
||||
is mounted instead as a protected Pi credential store. For a literal leading exclamation mark, Pi's
|
||||
declarative syntax is `$!`, not `!`.
|
||||
|
||||
Real example in use on this machine: GLM (provider `zai`):
|
||||
|
||||
```json
|
||||
{
|
||||
"providers": {
|
||||
"zai": {
|
||||
"baseUrl": "https://api.z.ai/api/coding/paas/v4",
|
||||
"api": "openai-completions",
|
||||
"apiKey": "$ZAI_API_KEY",
|
||||
"models": [
|
||||
{
|
||||
"id": "glm-5.2",
|
||||
"name": "GLM-5.2",
|
||||
"reasoning": true,
|
||||
"contextWindow": 200000,
|
||||
"maxTokens": 131072
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
providers:
|
||||
zai:
|
||||
endpoint:
|
||||
baseUrl: https://api.z.ai/api/coding/paas/v4
|
||||
authentication:
|
||||
mode: secret_env
|
||||
apiKeyEnv: ZAI_API_KEY
|
||||
session:
|
||||
mode: openai_compatible
|
||||
metadataGeneration:
|
||||
litellmProvider: openai
|
||||
models:
|
||||
glm-5.3:
|
||||
label: GLM 5.3
|
||||
session:
|
||||
reasoning: true
|
||||
contextWindow: 200000
|
||||
maxTokens: 131072
|
||||
metadataGeneration: {}
|
||||
```
|
||||
|
||||
Because it is user-level, GLM is visible to **every project**.
|
||||
Provider and model entries are maps. The keys form the canonical identity
|
||||
`<provider-key>/<model-key>`; `label` is only display text. A model is eligible for a use only when
|
||||
it contains that use block:
|
||||
|
||||
### Extension providers: not allowed in ThothII
|
||||
- `session` makes it selectable for interactive sessions;
|
||||
- `metadataGeneration` makes it selectable for description generation;
|
||||
- `embedding` is a single installation-level model rather than a selectable list.
|
||||
|
||||
Pi technically supports providers registered by JavaScript extensions, but ThothII does not use
|
||||
that capability. Project extensions are reserved for the workflow and gates; they must not contain
|
||||
`registerProvider(...)`. An endpoint that cannot be described by the OpenAI-compatible catalog is
|
||||
not supported by this installation until the declarative contract is extended generically.
|
||||
`defaults.session` is required. `defaults.metadataGeneration` is required exactly when at least
|
||||
one metadata-generation model exists. A session manifest pins its canonical identity, so removing a
|
||||
model never silently changes an existing session: resume fails with `model_unavailable`.
|
||||
|
||||
## Summary table (current machine state)
|
||||
## Session adapters
|
||||
|
||||
| Model | Level | Why | Visibility |
|
||||
|---|---|---|---|
|
||||
| `deepseek/deepseek-v4-pro` | Built-in Pi | Known public API, already in the build | All projects |
|
||||
| `deepseek/deepseek-v4-flash` | Built-in Pi | Known public API, already in the build | All projects |
|
||||
| `zai/glm-5.3` | `deploy/pi/models.json` | Custom OpenAI-compatible endpoint | ThothII installation |
|
||||
| `local-qwen/qwen3.6-35b-a3b` | `deploy/pi/models.json` | Locally configured OpenAI-compatible endpoint | ThothII installation |
|
||||
Use `pi_builtin` for a model whose technical definition ships with Pi:
|
||||
|
||||
## Choosing the right level for a new model
|
||||
|
||||
1. **Is the endpoint a public API already known to Pi?** Enable the exact identifier in `deploy/pi/settings.json`.
|
||||
2. **Is it OpenAI-compatible but not built in?** Declare it in `deploy/pi/models.json`, then enable it in `deploy/pi/settings.json`.
|
||||
3. **Does it require provider-specific transport code?** Do not add a provider-specific extension. The provider is unsupported until a generic declarative capability exists.
|
||||
|
||||
---
|
||||
|
||||
## User versus project scope: general summary
|
||||
|
||||
In addition to models, Pi loads other resources from two parallel trees: `~/.pi/agent/` (user) and
|
||||
`<cwd>/.pi/` (project, resolved from the directory where `pi` is launched).
|
||||
|
||||
| File/Directory | Livello utente | Livello progetto | Auto-discovery | Precedenza |
|
||||
|---|---|---|---|---|
|
||||
| `models.json` | `~/.pi/agent/models.json` | not supported | no | user only |
|
||||
| `settings.json` | `~/.pi/agent/settings.json` | `./.pi/settings.json` | no | project overrides user |
|
||||
| `extensions/` | `~/.pi/agent/extensions/` | `./.pi/extensions/` | yes (`.ts`/`.js`) | merged (project + user) |
|
||||
| `prompts/` | `~/.pi/agent/prompts/` | `./.pi/prompts/` | yes (`.md`) | merged |
|
||||
| `themes/` | `~/.pi/agent/themes/` | `./.pi/themes/` | yes (`.json`) | project preferred |
|
||||
| `skills/` | `~/.pi/agent/skills/` | `./.pi/skills/` | yes (`.md`) | merged |
|
||||
|
||||
### Real example: ThothII
|
||||
|
||||
```
|
||||
harness/.pi/
|
||||
├── extensions/
|
||||
│ ├── tht-gate.js # human-in-the-loop gate
|
||||
│ └── gate/
|
||||
│ ├── core/ # shared gate enforcement and utilities
|
||||
│ ├── disambiguation/ # F1/F3 policy
|
||||
│ └── memory/ # F2/F8 policy
|
||||
├── settings.json # optional override of user settings
|
||||
└── themes/
|
||||
└── thothii-mono.json # project theme
|
||||
```yaml
|
||||
deepseek:
|
||||
authentication:
|
||||
mode: pi_auth
|
||||
session:
|
||||
mode: pi_builtin
|
||||
models:
|
||||
deepseek-v4-pro:
|
||||
session: {}
|
||||
```
|
||||
|
||||
### Behavior relative to cwd
|
||||
Use `openai_compatible` for an explicit compatible endpoint. Each eligible session model must then
|
||||
declare the technical limits Pi needs. `upstreamModel` is optional and is used only when the
|
||||
endpoint expects a model name different from the catalog key.
|
||||
|
||||
The directory from which you launch `pi` determines which workflow extensions are found, but not
|
||||
which models ThothII makes available. The catalog is mounted in the container's Pi agent directory.
|
||||
Provider integrations remain declarative. Do not register providers from
|
||||
`harness/.pi/extensions/`; those extensions implement the workflow and human gates only.
|
||||
|
||||
## Authentication
|
||||
|
||||
Every provider chooses one explicit mode:
|
||||
|
||||
- `secret_env` names an approved key in the protected ThothII secret bundle through `apiKeyEnv`;
|
||||
- `pi_auth` uses Pi's protected authentication projection and is valid only for session-only
|
||||
`pi_builtin` providers;
|
||||
- `none` is valid only with an explicit keyless endpoint.
|
||||
|
||||
Secret values never belong in installation YAML, generated files, logs, CLI arguments, or browser
|
||||
requests. The YAML contains only an environment-variable name or an authentication mode. Pi's
|
||||
protected credential file remains selected by the installation authentication configuration.
|
||||
|
||||
## Generated runtime projections
|
||||
|
||||
Before Compose starts, `tht` validates the installation and atomically writes deterministic files
|
||||
under `deploy/<installation-id>/generated/`:
|
||||
|
||||
```text
|
||||
generated/
|
||||
├── catalog.json
|
||||
├── pi/
|
||||
│ ├── models.json
|
||||
│ └── settings.json
|
||||
└── compose.model-catalog.yaml
|
||||
```
|
||||
|
||||
The normalized catalog is consumed by the backend. The Pi files and Compose override are boundary
|
||||
adapters. They are not configuration sources and are excluded from backup. Restore regenerates
|
||||
them from the installation descriptor.
|
||||
|
||||
Run all lifecycle commands from the project root and select the descriptor explicitly when more
|
||||
than one installation exists:
|
||||
|
||||
```bash
|
||||
# From harness/: load the project gate and mounted local catalog
|
||||
cd /path/to/ThothII/harness
|
||||
pi --model local-qwen/qwen3.6-35b-a3b "..."
|
||||
INSTALLATION=/absolute/path/deploy/example/thothii-installation.yaml
|
||||
|
||||
tht --installation "$INSTALLATION" start
|
||||
tht --installation "$INSTALLATION" doctor
|
||||
```
|
||||
|
||||
The ThothII backend always launches Pi with `cwd: harnessDir` for the workflow. Model availability
|
||||
continues to depend only on `models.json`, `settings.json`, and local credentials.
|
||||
After editing `modelCatalog` or provider credentials, reload the current Pi image. Restart validates
|
||||
the YAML and regenerates projections before recreating `core`:
|
||||
|
||||
---
|
||||
|
||||
## Auto-discovery trap: `.mjs` is ignored
|
||||
|
||||
Pi's extension auto-discovery pattern is **`/\.(ts|js)$/`**; it does not include `.mjs`.
|
||||
|
||||
```
|
||||
.pi/extensions/
|
||||
├── my-extension.js ✅ auto-loaded
|
||||
├── my-extension.ts ✅ auto-loaded
|
||||
├── my-extension.mjs ❌ silently ignored (does not match the pattern)
|
||||
└── shared-module.mjs ✅ suitable for helper modules (deliberately not loaded as an extension)
|
||||
```
|
||||
|
||||
If an extension imports a shared module, use `.mjs` so Pi does not treat it as an extension on its own.
|
||||
|
||||
---
|
||||
|
||||
## Verifica
|
||||
|
||||
### List models
|
||||
```bash
|
||||
pi --list-models
|
||||
tht --installation "$INSTALLATION" pi restart --yes --drain
|
||||
tht --installation "$INSTALLATION" pi doctor
|
||||
tht --installation "$INSTALLATION" pi test
|
||||
```
|
||||
This shows built-in models and models declared in `models.json`.
|
||||
|
||||
### Check the catalog used by the application
|
||||
`tht pi update` changes the Pi version; it is not the configuration command. There is no
|
||||
`tht pi configure` and no separate apply command.
|
||||
|
||||
## Migrating a legacy installation
|
||||
|
||||
The migrator reads the former installation `metadataGeneration` block and the two former Pi JSON
|
||||
files, but never modifies them. Supply the facts that cannot be inferred safely and write a separate
|
||||
candidate:
|
||||
|
||||
```bash
|
||||
cd harness # or the project's relevant cwd
|
||||
pi --mode rpc
|
||||
# poi: {"type": "get_available_models", "id": "1"}
|
||||
tht --installation /absolute/path/legacy-installation.yaml installation migrate \
|
||||
--output /absolute/path/thothii-installation.v2.yaml \
|
||||
--session-default zai/glm-5.3 \
|
||||
--embedding-id ollama/qwen3-embedding:0.6b \
|
||||
--embedding-dimensions 1024
|
||||
```
|
||||
The RPC response must include only built-in models or models declared in the local catalog.
|
||||
|
||||
---
|
||||
Review the candidate, move the legacy source files out of the installation only after approval,
|
||||
then select the v2 descriptor. Ambiguous aliases, endpoint conflicts, or missing authentication
|
||||
facts produce field-level errors; the migrator does not guess.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Problem | Cause | Solution |
|
||||
|---|---|---|
|
||||
| `Model "X/Y" not found` | Provider/model missing from `models.json` or identifier missing from `enabledModels` | Fix the two local files and reload Pi |
|
||||
| Project settings not applied | Project `settings.json` has a syntax error, or `pi` is launched from the wrong cwd | Validate the JSON and check the cwd |
|
||||
| Symptom | Meaning | Action |
|
||||
| --- | --- | --- |
|
||||
| `migration_required` | A retired model source or installation schema is still present | Run the installation migrator and review its candidate |
|
||||
| Unknown session or metadata default | The canonical ID is missing the corresponding use block | Correct the provider/model key or add the intended use block |
|
||||
| Generated projection drift | Runtime files differ from the descriptor-derived bytes | Run `tht start` or `tht pi restart --yes --drain` |
|
||||
| `model_unavailable` on resume | The session's pinned model is no longer session-eligible | Restore that catalog entry or keep the session unavailable; do not remap it |
|
||||
| Provider smoke failure | Credentials, endpoint, or provider availability is invalid | Correct the protected credential or catalog endpoint, restart, then run `tht pi test` |
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
# Copy this file to a protected operator-controlled path named exactly thothii-installation.yaml
|
||||
# and set mode 0600 (or 0400) before using it as THT_INSTALLATION_CONFIG_SOURCE.
|
||||
# Replace every absolute placeholder. Select exactly one Git transport override.
|
||||
schemaVersion: 2
|
||||
profile: local
|
||||
projectDirectory: "/absolute/path/to/ThothII"
|
||||
envFile: "/absolute/path/to/ThothII/deploy/env/local.env"
|
||||
@@ -8,17 +9,27 @@ workspaceRepository:
|
||||
remote: git@git.example.com:organization/workspaces.git
|
||||
branch: main
|
||||
access: ssh
|
||||
metadataGeneration:
|
||||
default: openai-mini
|
||||
models:
|
||||
- id: openai-mini
|
||||
label: OpenAI Mini
|
||||
litellm:
|
||||
provider: openai
|
||||
model: gpt-4.1-mini
|
||||
apiKeyEnv: OPENAI_API_KEY
|
||||
# apiKeyEnv may be omitted only for an explicit endpoint that accepts
|
||||
# unauthenticated requests.
|
||||
modelCatalog:
|
||||
defaults:
|
||||
session: deepseek/deepseek-v4-pro
|
||||
metadataGeneration: openai/gpt-4.1-mini
|
||||
embedding:
|
||||
id: ollama/qwen3-embedding:0.6b
|
||||
dimensions: 1024
|
||||
providers:
|
||||
deepseek:
|
||||
authentication: {mode: pi_auth}
|
||||
session: {mode: pi_builtin}
|
||||
models:
|
||||
deepseek-v4-pro:
|
||||
session: {}
|
||||
openai:
|
||||
authentication: {mode: secret_env, apiKeyEnv: OPENAI_API_KEY}
|
||||
metadataGeneration: {litellmProvider: openai}
|
||||
models:
|
||||
gpt-4.1-mini:
|
||||
label: OpenAI Mini
|
||||
metadataGeneration: {}
|
||||
authentication:
|
||||
configDirectory: "/absolute/path/to/thothii-auth"
|
||||
overrides:
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
# Copy this file to a protected operator path named exactly thothii-installation.yaml
|
||||
# and set mode 0600 (or 0400) before using it as THT_INSTALLATION_CONFIG_SOURCE.
|
||||
# Replace every absolute placeholder. Select exactly one Git transport override.
|
||||
schemaVersion: 2
|
||||
profile: server
|
||||
projectDirectory: "/absolute/path/to/ThothII"
|
||||
envFile: "/absolute/path/to/thothii-server-operator/server.env"
|
||||
@@ -8,19 +9,28 @@ workspaceRepository:
|
||||
remote: git@git.example.com:organization/workspaces.git
|
||||
branch: main
|
||||
access: ssh
|
||||
metadataGeneration:
|
||||
default: openai-mini
|
||||
models:
|
||||
- id: openai-mini
|
||||
label: OpenAI Mini
|
||||
litellm:
|
||||
provider: openai
|
||||
model: gpt-4.1-mini
|
||||
endpoint:
|
||||
baseUrl: https://api.openai.example/v1
|
||||
apiKeyEnv: OPENAI_API_KEY
|
||||
# apiKeyEnv may be omitted only for an explicit endpoint that accepts
|
||||
# unauthenticated requests.
|
||||
modelCatalog:
|
||||
defaults:
|
||||
session: deepseek/deepseek-v4-pro
|
||||
metadataGeneration: openai/gpt-4.1-mini
|
||||
embedding:
|
||||
id: ollama/qwen3-embedding:0.6b
|
||||
dimensions: 1024
|
||||
providers:
|
||||
deepseek:
|
||||
authentication: {mode: pi_auth}
|
||||
session: {mode: pi_builtin}
|
||||
models:
|
||||
deepseek-v4-pro:
|
||||
session: {}
|
||||
openai:
|
||||
endpoint: {baseUrl: https://api.openai.example/v1}
|
||||
authentication: {mode: secret_env, apiKeyEnv: OPENAI_API_KEY}
|
||||
metadataGeneration: {litellmProvider: openai}
|
||||
models:
|
||||
gpt-4.1-mini:
|
||||
label: OpenAI Mini
|
||||
metadataGeneration: {}
|
||||
authentication:
|
||||
# Root-operated source of truth; it is never mounted into core.
|
||||
configDirectory: "/srv/example/thothii/auth-canonical"
|
||||
|
||||
@@ -80,8 +80,9 @@ THT_DWH_API_KEY=...
|
||||
OPENAI_API_KEY=...
|
||||
```
|
||||
|
||||
`THT_MODEL_API_KEY` remains Pi-only. A metadata-generation model instead references one audited
|
||||
bundle name from `deploy/secrets/README.md` through `metadataGeneration.models[].apiKeyEnv`.
|
||||
`THT_MODEL_API_KEY` is available only as an explicitly declared catalog bundle key. A
|
||||
metadata-generation provider references one audited bundle name from `deploy/secrets/README.md`
|
||||
through `modelCatalog.providers.<provider>.authentication.apiKeyEnv`.
|
||||
`apiKeyEnv` may be omitted only for an explicit endpoint that accepts unauthenticated requests;
|
||||
hosted/default endpoints remain keyed.
|
||||
Provider/model/endpoint settings stay in the protected installation descriptor; raw keys do not.
|
||||
|
||||
@@ -70,14 +70,63 @@ capability, malformed snapshot, or connector error applies no catalog changes. S
|
||||
|
||||
## Synchronize authoritative schema metadata
|
||||
|
||||
Start a synchronization from a database or a selected table set. The available scopes are tables,
|
||||
columns, relationships, and all. One database can have only one active catalog operation at a
|
||||
time; cleanup shares this exclusion.
|
||||
Schema synchronization reads the external database and reconciles the installation-local catalog.
|
||||
It never changes the source database. The available synchronization scopes are **tables**,
|
||||
**columns**, **relationships**, and **all**, but the UI exposes them at different levels:
|
||||
|
||||
| Location | Action | Effective scope |
|
||||
| --- | --- | --- |
|
||||
| Database view | **Synchronize tables** | All tables in the selected database |
|
||||
| Database view | **Synchronize relationships** | All physical foreign-key relationships in the selected database |
|
||||
| Database view | **Synchronize all** | Tables, columns, and physical relationships in the selected database |
|
||||
| Tables view | **Synchronize database tables** | All tables in the selected database. Selecting a table enables the action, but does not narrow its scope. |
|
||||
| Tables view | **Synchronize columns for selected tables** | Columns belonging to the selected tables |
|
||||
| Columns view | **Synchronize columns for this table** | All columns belonging to the table currently open. Selecting at least one column enables the action, but does not narrow its scope to that column. |
|
||||
|
||||
There is no database-level column action and no synchronization action for an individual column.
|
||||
The selection requirement in the tables and columns views controls whether the action selector can
|
||||
be used; it is not always the same as the synchronization target. The **Sync all** button in the
|
||||
tables view is the direct shortcut for the full-database scope.
|
||||
|
||||
Before starting any synchronization, run **Test connection**. Synchronization is rejected when
|
||||
the binding is unreachable or its tested version is older than the current binding configuration.
|
||||
Only one catalog operation can be active for a database at a time; explicit cleanup shares this
|
||||
exclusion.
|
||||
|
||||
### What a synchronization does
|
||||
|
||||
Every run first creates a durable queued operation and reads a schema snapshot. The scan reports
|
||||
these phases:
|
||||
|
||||
1. Connect to the database.
|
||||
2. Read tables.
|
||||
3. Read columns and primary-key positions.
|
||||
4. Read foreign-key relationships.
|
||||
5. Calculate the planned catalog difference.
|
||||
6. Apply only the requested scope.
|
||||
|
||||
The scan currently reads the complete physical snapshot, including foreign keys, even when the
|
||||
requested scope is only tables or columns. This is required by the introspection contract and is
|
||||
why the log can mention foreign-key reading during a table synchronization. Reading those keys
|
||||
does not by itself create or update catalog relationships:
|
||||
|
||||
- **Synchronize tables** writes table membership and source comments. If a table disappears, its
|
||||
catalog columns and physical relationships are removed through the table cascade.
|
||||
- **Synchronize columns** writes column membership and structural attributes for all tables or for
|
||||
the selected table subset. It does not write physical relationships.
|
||||
- **Synchronize relationships** writes the physical relationships derived from the source foreign
|
||||
keys. It does not create generated or manual logical relationships.
|
||||
- **Synchronize all** applies all three scopes and marks the database schema version as fully
|
||||
synchronized.
|
||||
|
||||
The run scans first and publishes a durable operation. If it detects a destructive difference, it
|
||||
requires confirmation and re-scans before applying. You can cancel before apply; completed and
|
||||
failed runs remain in history. The live log is delivered over SSE with a polling fallback.
|
||||
|
||||
The synchronization history records the requested scope, progress phases, planned changes,
|
||||
confirmation, result counts, and errors. Closing the history drawer does not cancel a running
|
||||
operation; it can be reopened from the database synchronization history control.
|
||||
|
||||
Explicit cleanup is different from source synchronization: administrators can clear selected
|
||||
table/relationship or column/relationship catalog metadata without changing the external source,
|
||||
the connection binding, or secrets. Deleting a table cascades to its columns and relationships.
|
||||
|
||||
@@ -16,13 +16,20 @@ The root catalog has `schema_version: 1` and an ordered list of workspace identi
|
||||
<!-- non-workspace-migration:end -->
|
||||
|
||||
<!-- workspace-descriptor-contract:start -->
|
||||
Schema v3 is the only accepted workspace descriptor. Schema v1 and v2 workspace descriptors are
|
||||
rejected before activation. Each catalog entry must have a matching descriptor at
|
||||
Schema v4 is the only accepted workspace descriptor. Schema v1, v2, and v3 workspace descriptors
|
||||
are rejected before activation. Each catalog entry must have a matching descriptor at
|
||||
`<id>/workspace.yaml` in the same Git commit. The application validates a complete candidate
|
||||
revision and activates it atomically; invalid content leaves the preceding active revision in
|
||||
place.
|
||||
<!-- workspace-descriptor-contract:end -->
|
||||
|
||||
<!-- non-workspace-migration:start -->
|
||||
To convert a v3 descriptor before committing it, set `workspace.schema_version` to `4`, remove
|
||||
`llm_policy`, and remove `semantic_index`. Database, Evidence, diagnostics, and binding data remain
|
||||
unchanged. Validate the resulting v4 repository revision before activation; ThothII never rewrites
|
||||
the curator-owned repository during pull.
|
||||
<!-- non-workspace-migration:end -->
|
||||
|
||||
## Operator sequence
|
||||
|
||||
1. Curate and push a complete repository revision. Do not put DWH passwords, API keys, private
|
||||
@@ -58,7 +65,7 @@ tht --installation "$INSTALLATION" workspace preprocess run \
|
||||
|
||||
The contract gives exact validation, exit code, and JSON rules in
|
||||
[Workspace preprocessing CLI](../contracts/workspace-preprocessing-cli.md). For Evidence source
|
||||
forms and the schema-v3 descriptor contract, see
|
||||
forms and the schema-v4 descriptor contract, see
|
||||
[Workspace Evidence v3](../contracts/workspace-evidence-v3.md).
|
||||
|
||||
## Transport and revision rules
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Installation Model Catalog
|
||||
|
||||
Status: accepted design; implementation not started.
|
||||
Status: implemented on 2026-09-02.
|
||||
|
||||
## Outcome
|
||||
|
||||
@@ -241,5 +241,6 @@ drafts and model filtering, examples, fixtures, and documentation. Existing sess
|
||||
readable and keep their pinned provider/model identity; only resume resolution changes to the new
|
||||
catalog.
|
||||
|
||||
This document authorizes design only. Software implementation begins only after a separate explicit
|
||||
request.
|
||||
Implementation completed after explicit approval. The installation schema, deterministic runtime
|
||||
projections, migration path, model-free workspace schema v4, backend consumers, operator UI,
|
||||
fixtures, and documentation now enforce this contract.
|
||||
|
||||
@@ -288,7 +288,7 @@ inferenza dalla UI.
|
||||
| PRE-03 | Verificare Pi, provider autore, Ollama, Qdrant, DWH e auth | tutti raggiungibili senza esporre credenziali |
|
||||
| PRE-04 | Inventariare `psd-clinical` come gruppo di controllo | commit, ACTIVE, counts per kind e canary `disambiguation` con expected ID salvati |
|
||||
| PRE-05 | Verificare assenza del workspace/collection lab | nessuna collisione; se esistono, fermarsi e identificare il proprietario |
|
||||
| PRE-06 | Aggiungere catalog entry, descriptor lab e `evidence/README.md` tracciato in una revisione candidata | ID, URI e collection distinti; descriptor schema v3 valido; tree Evidence risolvibile anche senza D1-D3 |
|
||||
| PRE-06 | Aggiungere catalog entry, descriptor lab e `evidence/README.md` tracciato in una revisione candidata | ID e URI distinti; descriptor workspace schema v4 valido; tree Evidence risolvibile anche senza D1-D3 |
|
||||
| PRE-07 | Verificare nel processo dell'installazione `THT_WORKSPACE_GIT_BRANCH=test/evidence-lifecycle`, pushare la revisione su quel branch e usare **Update workspace repository** nella UI | la UI mostra ref/commit attesi, candidate valida attivata atomicamente; PSD resta disponibile |
|
||||
| PRE-08 | Selezionare il lab, configurare binding/secret, **Validate workspace source** e **Test workspace connections** | check verdi; secret solo write-only |
|
||||
| PRE-09 | Selezionare il lab come workspace dell'installazione | nuove sessioni usano il lab; nessuna sessione PSD viene mutata |
|
||||
|
||||
Reference in New Issue
Block a user