docs: assess catalog postgres deployment constraints
This commit is contained in:
Codex
@@ -0,0 +1,219 @@
|
||||
# PostgreSQL deployment and failure-isolation constraints for the Metadata Catalog
|
||||
|
||||
Date: 2026-08-23
|
||||
Issue: [#8 — Assess PostgreSQL deployment and failure-isolation constraints](https://github.com/mptyl/ThothII/issues/8)
|
||||
|
||||
## Question
|
||||
|
||||
How can a Metadata Catalog PostgreSQL service be deployed, backed up, diagnosed, and made
|
||||
non-blocking for the core across local and server installations?
|
||||
|
||||
## Recommendation
|
||||
|
||||
Use PostgreSQL as the operational store, but place it behind a **separate Metadata Catalog API
|
||||
process**. The SQL workflow `core` must not receive the catalog DSN, catalog credentials, a client
|
||||
pool, or a Compose dependency on either the catalog API or PostgreSQL.
|
||||
|
||||
The resulting dependency graph is:
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
UI[Frontend]
|
||||
Core[Core workflow API]
|
||||
Semantic[Qdrant and embedding]
|
||||
Catalog[Metadata Catalog API]
|
||||
PG[(Catalog PostgreSQL)]
|
||||
Publish[Publication worker]
|
||||
|
||||
UI -->|workflow routes| Core
|
||||
Core --> Semantic
|
||||
UI -->|catalog routes| Catalog
|
||||
Catalog --> PG
|
||||
PG --> Publish
|
||||
Publish -->|immutable approved Publication| Semantic
|
||||
```
|
||||
|
||||
The core continues to use the last successfully published, revision-qualified semantic snapshot.
|
||||
Unpublished catalog edits are never a runtime dependency. This matches the domain boundary: the
|
||||
Metadata Catalog does not select workflow SQL elements, while a Publication is an immutable value
|
||||
made available to consumers ([`CONTEXT.md`, lines 96–110](../../CONTEXT.md#L96-L110)). It also
|
||||
preserves the existing rule that Qdrant is a derived index rather than a canonical source
|
||||
([`PROJECT_STATE.md`, lines 405–410](../../PROJECT_STATE.md#L405-L410)).
|
||||
|
||||
## Evidence from the current architecture
|
||||
|
||||
1. The mandatory workflow stack is currently `frontend`, `core`, `qdrant`, `embedding`, and the
|
||||
model-init job. `core` waits only for Qdrant and model initialization
|
||||
([`compose.yaml`, lines 50–60](../../compose.yaml#L50-L60)); the frontend waits only for `core`
|
||||
([`compose.yaml`, lines 123–141](../../compose.yaml#L123-L141)). Adding a catalog dependency to
|
||||
either chain would enlarge the workflow failure domain.
|
||||
2. `tht start` runs Compose and then checks an explicit allowlist containing only those five
|
||||
mandatory services. Other Compose services are ignored by the readiness fold
|
||||
([`service.go`, lines 40–57](../../tools/tht/internal/service/service.go#L40-L57),
|
||||
[`service.go`, lines 166–190](../../tools/tht/internal/service/service.go#L166-L190)). Separate
|
||||
catalog services can therefore start with the installation without redefining core readiness.
|
||||
3. `/health` is deliberately a process-liveness endpoint and does not probe external services
|
||||
([`app.ts`, lines 271–284](../../backend/src/app.ts#L271-L284)). Tests preserve a 200 response
|
||||
even when PostgreSQL session storage is configured but not contacted
|
||||
([`health.test.ts`, lines 9–32](../../backend/test/health.test.ts#L9-L32)). Catalog readiness must
|
||||
not be folded into this endpoint.
|
||||
4. ThothII already has a sound server-PostgreSQL precedent: runtime and migrator credentials are
|
||||
distinct; the one-shot migrator is profile-gated and is explicitly not a dependency of `core`
|
||||
([`compose.session-server.yaml.example`, lines 24–49](../../deploy/compose.session-server.yaml.example#L24-L49)).
|
||||
Its documented failure behavior is route-scoped 503 while process liveness remains healthy
|
||||
([`PROJECT_STATE.md`, lines 621–636](../../PROJECT_STATE.md#L621-L636)).
|
||||
5. Existing backup archives contain selected Compose volumes only. Local archives include seven
|
||||
named volumes and server archives include only Qdrant and embedding volumes; no PostgreSQL
|
||||
logical dump exists today ([`create.go`, lines 29–52](../../tools/tht/internal/backup/create.go#L29-L52)).
|
||||
A catalog database therefore requires an explicit backup contract rather than an assumption
|
||||
that current installation backup already covers it.
|
||||
|
||||
## Deployment contract
|
||||
|
||||
### Shared rules
|
||||
|
||||
- Run `catalog-api` as a separate, non-privileged service. The frontend reverse proxy may route
|
||||
`/api/catalog/*` to it, but frontend startup must not depend on catalog readiness.
|
||||
- Give `catalog-api` a bounded pool and bounded operations. As an initial ceiling for this small
|
||||
administrative workload: pool size 5, overflow 0–2, `connect_timeout=3`, `lock_timeout=2s`, and
|
||||
a normal request `statement_timeout=10s`. Long AI calls must not hold database transactions.
|
||||
PostgreSQL documents that an omitted or zero `connect_timeout` waits indefinitely, so an
|
||||
explicit value is required for failure isolation
|
||||
([libpq connection parameters](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-CONNECT-TIMEOUT)).
|
||||
- Use three roles: `catalog_runtime` (CRUD and job claims only), `catalog_migrator` (DDL, used only
|
||||
by a one-shot job), and `catalog_backup` (minimum privileges needed by the approved dump policy).
|
||||
No role is shared with the DWH or `thoth_sessions`, and none is a PostgreSQL superuser.
|
||||
- Store passwords and the CA as separate secret files. For non-local connections default to
|
||||
`sslmode=verify-full`; this follows the current server-session configuration
|
||||
([`compose.session-server.yaml.example`, lines 5–20](../../deploy/compose.session-server.yaml.example#L5-L20))
|
||||
and PostgreSQL's hostname-verifying TLS guidance
|
||||
([libpq SSL parameters](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS)).
|
||||
- Apply ordered, checksummed migrations under one transaction and an advisory lock. Refuse pending,
|
||||
unknown, or checksum-drifted migrations in catalog readiness. The session repository already
|
||||
implements this pattern
|
||||
([`postgres_repository.py`, lines 118–187](../../harness/tht/session/postgres_repository.py#L118-L187));
|
||||
the catalog should have its own migration table, lock key, schema, and roles.
|
||||
|
||||
### Local installation
|
||||
|
||||
Add an installation-owned `catalog-postgres` container on the private `thothii` network, with no
|
||||
published host port and a dedicated `catalog-postgres-data` named volume. Pin the image by version
|
||||
and digest, add a `pg_isready` healthcheck, and let only `catalog-api` use
|
||||
`depends_on: condition: service_healthy`. `core` and `frontend` keep their existing dependency
|
||||
graph. Docker confirms that `service_healthy` delays only the declaring dependent service, while
|
||||
plain container start does not imply application readiness
|
||||
([Compose startup order](https://docs.docker.com/compose/how-tos/startup-order/)).
|
||||
|
||||
The local catalog services can be included in the normal installation Compose set because the
|
||||
host CLI's mandatory-service allowlist excludes them. A separate `catalog-migrate` one-shot profile
|
||||
retains explicit operator control; Docker profiles are intended for selectively activated and
|
||||
one-off services ([Compose profiles](https://docs.docker.com/compose/how-tos/profiles/)).
|
||||
|
||||
### Server installation
|
||||
|
||||
Use an operator-provided PostgreSQL endpoint configured through a reviewed, untracked overlay.
|
||||
Prefer a separate database and dedicated roles. A separate PostgreSQL cluster gives the strongest
|
||||
resource and outage isolation; sharing the existing server cluster is acceptable only when the
|
||||
operator accepts the residual cluster-wide blast radius and enforces connection limits, timeouts,
|
||||
and separate ownership. Merely using another schema does not isolate connection exhaustion,
|
||||
maintenance outages, WAL pressure, or storage failure.
|
||||
|
||||
The server overlay adds secrets and catalog configuration only to `catalog-api` and the one-shot
|
||||
migrator. It must not mount those secrets into `core`. This mirrors the current rule that `core`
|
||||
never receives the session migrator credential
|
||||
([`PROJECT_STATE.md`, lines 623–630](../../PROJECT_STATE.md#L623-L630)).
|
||||
|
||||
## Failure behavior
|
||||
|
||||
| Failure | Catalog behavior | Workflow behavior |
|
||||
| --- | --- | --- |
|
||||
| PostgreSQL unavailable or connection pool exhausted | Catalog routes return sanitized `503 catalog_unavailable` with `Retry-After`; UI remains read-only/unavailable | Existing sessions and new SQL workflow requests continue against the last published snapshot |
|
||||
| Catalog API process unavailable | `/api/catalog/*` fails; main frontend shell and workflow route remain usable | No effect |
|
||||
| Pending or drifted migration | Catalog readiness fails and writes are refused | No effect |
|
||||
| AI generation fails | Job records a retryable/terminal failure; no database transaction remains open | No effect |
|
||||
| Publication indexing fails | Publication remains non-active/failed and can be retried idempotently | Previous active Publication remains in Qdrant |
|
||||
| Qdrant unavailable | Publication is queued/failed without losing canonical catalog state | Existing core readiness rules already govern this independent failure |
|
||||
|
||||
Do not silently fall back to JSON files, dual-write, or read unpublished PostgreSQL rows from the
|
||||
core. Those paths would create split-brain state. The only degraded-mode contract is use of the
|
||||
last successfully activated Publication.
|
||||
|
||||
## Backup and restore
|
||||
|
||||
### Backup
|
||||
|
||||
Extend `tht backup create` or add a catalog-specific command invoked by it. Create a logical
|
||||
`pg_dump --format=custom` archive through a one-shot helper, checksum it, and place it in the
|
||||
installation archive manifest. Do **not** tar a live PostgreSQL data volume: the existing raw-volume
|
||||
backup mechanism was designed for the current named volumes, not database crash consistency.
|
||||
|
||||
`pg_dump` produces a consistent export while the database remains in use, and custom format is
|
||||
compressed and supports selective/reordered restore
|
||||
([PostgreSQL `pg_dump`](https://www.postgresql.org/docs/current/app-pgdump.html)). Because metadata
|
||||
volume is small, favor correctness over parallelism. Run a backup before every migration and on an
|
||||
operator-defined schedule; record the server version, migration head, dump checksum, creation time,
|
||||
and catalog Publication identifiers in the manifest. Never write a dump into the workspace Git
|
||||
repository.
|
||||
|
||||
For an external server database, the operator must supply the dedicated backup credential through
|
||||
the same secret-file boundary. A successful filesystem/Qdrant archive with a failed catalog dump
|
||||
is an incomplete installation backup and must not be reported as success.
|
||||
|
||||
### Restore
|
||||
|
||||
Restore only during an explicit catalog maintenance window into an empty target database (or a
|
||||
deliberately cleaned target), using `pg_restore --exit-on-error --single-transaction`. Validate the
|
||||
archive checksum first, then require:
|
||||
|
||||
1. expected migration history with no checksum drift;
|
||||
2. referential-integrity checks and bounded entity counts;
|
||||
3. Publication content hashes matching the restored records;
|
||||
4. an authenticated catalog smoke test.
|
||||
|
||||
The relevant restore controls and their behavior are documented by
|
||||
[PostgreSQL `pg_restore`](https://www.postgresql.org/docs/current/app-pgrestore.html). After restore,
|
||||
rebuild catalog-derived Qdrant content from the restored active Publication rather than treating
|
||||
the vector index as the source of truth. Keep the old database untouched until this validation
|
||||
passes; rollback is endpoint reversal, not dual-write.
|
||||
|
||||
## Diagnostics and observability
|
||||
|
||||
- Keep `core /health` unchanged. Give `catalog-api` separate `/health/live` (process only) and
|
||||
`/health/ready` (bounded connection, `SELECT 1`, and migration compatibility) endpoints.
|
||||
- Add a named `metadata-catalog` section to `tht doctor`: configuration completeness; secret-file
|
||||
presence and permissions; DNS/TCP/TLS/authentication; `SELECT 1`; migration applied/pending/drifted
|
||||
state; pool saturation; last successful backup age; and oldest pending Publication. Checks are
|
||||
read-only and all errors/DSNs are sanitized. The current doctor already separates Compose service
|
||||
status from container-local workflow checks
|
||||
([`report.go`, lines 275–310](../../tools/tht/internal/doctor/report.go#L275-L310)).
|
||||
- Use `pg_isready` only for transport readiness; its exit codes distinguish accepting, rejecting,
|
||||
no-response, and invalid-parameter states, but it does not prove schema or authorization
|
||||
correctness ([PostgreSQL `pg_isready`](https://www.postgresql.org/docs/current/app-pg-isready.html)).
|
||||
- Expose structured metrics/logs for connection acquisition failures, pool utilization, query
|
||||
timeout/lock timeout, migration head, AI-job backlog, Publication lag, and backup age. Never log
|
||||
connection strings, prompts containing schema metadata, or generated descriptions at info level.
|
||||
- `tht status` may show the catalog as `healthy`, `degraded`, or `disabled`, but catalog state must
|
||||
not change the existing core health result. `tht doctor` may return a failing catalog check for
|
||||
operator visibility without stopping or restarting the workflow stack.
|
||||
|
||||
## Acceptance constraints for later design and implementation
|
||||
|
||||
1. Killing `catalog-postgres` and `catalog-api` must not change `core /health`, stop Pi, interrupt an
|
||||
existing SQL session, or prevent a new session that uses the already active Publication.
|
||||
2. No catalog database secret, client dependency, migration, or network call exists in the harness
|
||||
workflow path.
|
||||
3. Catalog failure is explicit on catalog routes; no JSON fallback or dual-write is introduced.
|
||||
4. A failed Publication leaves the previous active semantic content addressable and retryable.
|
||||
5. Local and server backup tests perform a real dump/restore round trip and prove that a backup is
|
||||
incomplete when the PostgreSQL dump fails.
|
||||
6. Migration credentials are absent from runtime containers; drift and pending migrations are
|
||||
visible in readiness and `tht doctor`.
|
||||
|
||||
## Decision summary
|
||||
|
||||
The Metadata Catalog should be a separately deployed bounded context backed by PostgreSQL. Local
|
||||
installations own a private container and volume; server installations use a TLS-verified external
|
||||
database, preferably on a separate cluster when strict blast-radius isolation is required. Logical
|
||||
dump/restore, one-shot migrations, role separation, bounded connections, component-specific
|
||||
diagnostics, and last-good-Publication semantics make PostgreSQL operationally safe without making
|
||||
it a dependency of the SQL workflow.
|
||||
Reference in New Issue
Block a user