fix: close Pi management final review findings
This commit is contained in:
@@ -78,10 +78,13 @@ maintenance gate before it checks sessions. Without `--drain`, active open sessi
|
|||||||
command. With `--drain`, the command polls the authenticated session inventory until no active
|
command. With `--drain`, the command polls the authenticated session inventory until no active
|
||||||
sessions remain; it never terminates sessions and the wait is bounded.
|
sessions remain; it never terminates sessions and the wait is bounded.
|
||||||
|
|
||||||
Restart retains the exact current image and does not build, pull, select, tag, or upgrade an image.
|
Restart retains the exact current image and never builds, pulls, or upgrades an image. Before any
|
||||||
It recreates only `core` with `--no-deps --force-recreate`; `frontend` and named volumes are not
|
core mutation, it tags the captured running image ID with a transaction-scoped reference and
|
||||||
|
selects that reference through a lifecycle-only Compose override. A configured mutable tag moving
|
||||||
|
after capture therefore cannot change the restarted image. It recreates only `core` with
|
||||||
|
`--no-deps --force-recreate --no-build --pull never`; `frontend` and named volumes are not
|
||||||
recreated. Before reopening admission, it verifies health, the unchanged Pi version, the
|
recreated. Before reopening admission, it verifies health, the unchanged Pi version, the
|
||||||
provider/model/settings smoke, unchanged non-secret rendered configuration, the current image
|
provider/model/settings smoke, unchanged non-secret rendered configuration, the captured image
|
||||||
identity, and the complete persistence-mount fingerprint.
|
identity, and the complete persistence-mount fingerprint.
|
||||||
|
|
||||||
Restart and update keep separate recovery state:
|
Restart and update keep separate recovery state:
|
||||||
@@ -91,10 +94,13 @@ Restart and update keep separate recovery state:
|
|||||||
<projectDirectory>/.thothctl/<installation-id>/update-state.json
|
<projectDirectory>/.thothctl/<installation-id>/update-state.json
|
||||||
```
|
```
|
||||||
|
|
||||||
The files are mode `0600` and share one installation lifecycle lock, so a restart cannot race an
|
The files are mode `0600` and share one installation lifecycle lock, so restart, update, and
|
||||||
update. A restart refuses an incomplete update or restart state. After core mutation, a failure
|
rollback cannot race. Every mutating lifecycle command checks both files. Malformed or non-terminal
|
||||||
leaves admission gated and preserves `restart-state.json`; the operator must use status/logs and
|
restart recovery state blocks update and rollback; malformed or incomplete update recovery state
|
||||||
maintenance recovery rather than deleting state files.
|
blocks restart. A verified terminal restart state is cleaned up safely before a later mutation.
|
||||||
|
After core mutation, a restart failure leaves admission gated and preserves both
|
||||||
|
`restart-state.json` and its exact-image override; the operator must use status/logs and maintenance
|
||||||
|
recovery rather than deleting recovery material.
|
||||||
|
|
||||||
## Updating Pi
|
## Updating Pi
|
||||||
|
|
||||||
@@ -139,7 +145,7 @@ files and never deletes the durable selector.
|
|||||||
Only `core` is recreated, with `--no-deps --force-recreate`; `frontend` is not recreated and no
|
Only `core` is recreated, with `--no-deps --force-recreate`; `frontend` is not recreated and no
|
||||||
volume-replacement flags are used. Verification checks health; exact requested Pi version at all
|
volume-replacement flags are used. Verification checks health; exact requested Pi version at all
|
||||||
three declared boundaries (the candidate executable, `PI_VERSION` environment, and
|
three declared boundaries (the candidate executable, `PI_VERSION` environment, and
|
||||||
`org.opencontainers.image.version` image label); the provider/model/settings smoke; unchanged
|
`io.thothii.pi.version` image label); the provider/model/settings smoke; unchanged
|
||||||
non-secret rendered configuration; and the complete persistence-mount fingerprint.
|
non-secret rendered configuration; and the complete persistence-mount fingerprint.
|
||||||
|
|
||||||
## Recovery, rollback, and maintenance cleanup
|
## Recovery, rollback, and maintenance cleanup
|
||||||
@@ -175,8 +181,9 @@ For a failed update with `update-state.json`, first run:
|
|||||||
thothctl --installation /absolute/path/thothii-installation.yaml pi rollback --yes
|
thothctl --installation /absolute/path/thothii-installation.yaml pi rollback --yes
|
||||||
```
|
```
|
||||||
|
|
||||||
Rollback operates on update state only. A failed restart with `restart-state.json` retains its
|
Rollback restores the image recorded in update state, but it checks restart state before making any
|
||||||
current image and has no candidate image to roll back; first inspect status and logs, repair the
|
change. A failed, pending, or malformed restart state rejects rollback. A failed restart retains its
|
||||||
|
captured image and has no candidate image to roll back; first inspect status and logs, repair the
|
||||||
reported problem, then use maintenance recovery.
|
reported problem, then use maintenance recovery.
|
||||||
|
|
||||||
Inspect and clean a stale durable gate with:
|
Inspect and clean a stale durable gate with:
|
||||||
@@ -186,14 +193,15 @@ thothctl --installation /absolute/path/thothii-installation.yaml pi maintenance
|
|||||||
thothctl --installation /absolute/path/thothii-installation.yaml pi maintenance recover --yes
|
thothctl --installation /absolute/path/thothii-installation.yaml pi maintenance recover --yes
|
||||||
```
|
```
|
||||||
|
|
||||||
`maintenance recover` verifies and removes an interrupted restart state before it processes update
|
`maintenance recover` restores the captured restart image pin and lifecycle override when needed,
|
||||||
state. It completes an interrupted verified-image promotion, safely finalizes a preparation
|
verifies and removes interrupted restart recovery material, and only then processes update state.
|
||||||
interrupted before core mutation, and refuses other pending mutations. For terminal or absent
|
It completes an interrupted verified-image promotion, safely finalizes a preparation interrupted
|
||||||
recovery state, it removes only a stale transaction override, verifies the running installation
|
before core mutation, and refuses other pending mutations. For terminal or absent recovery state,
|
||||||
when the gate is active, and only then removes the durable marker and reopens admission. It never
|
it removes only a stale transaction override, verifies the running installation when the gate is
|
||||||
removes `current-image.yaml`. If rollback or recovery fails, leave the marker in place, preserve
|
active, and only then removes the durable marker and reopens admission. It never removes
|
||||||
the relevant recovery state, repair the reported Docker/configuration issue, and rerun rollback or
|
`current-image.yaml`. If rollback or recovery fails, leave the marker in place, preserve the
|
||||||
maintenance recovery.
|
relevant recovery state and override, repair the reported Docker/configuration issue, and rerun
|
||||||
|
rollback or maintenance recovery.
|
||||||
|
|
||||||
Missing confirmation, invalid arguments, active sessions, and an interrupted transaction exit
|
Missing confirmation, invalid arguments, active sessions, and an interrupted transaction exit
|
||||||
`2`. Docker and verification failures exit nonzero with concise, redacted guidance. Direct
|
`2`. Docker and verification failures exit nonzero with concise, redacted guidance. Direct
|
||||||
|
|||||||
@@ -77,9 +77,12 @@ running application with one confirmed restart:
|
|||||||
|
|
||||||
`--yes` confirms that core will be recreated. Without `--drain`, restart refuses active sessions;
|
`--yes` confirms that core will be recreated. Without `--drain`, restart refuses active sessions;
|
||||||
with it, ThothII closes admission and waits for active sessions to finish without terminating them.
|
with it, ThothII closes admission and waits for active sessions to finish without terminating them.
|
||||||
The wait is bounded. Restart retains the current image: it does not build, pull, select, or upgrade
|
The wait is bounded. Restart retains the exact captured running image: it does not build, pull, or
|
||||||
an image. It will restart only core, then verifies health, Pi version, settings/model smoke,
|
upgrade an image. Before recreating core, it pins that image through transaction-scoped Compose
|
||||||
non-secret rendered configuration, and persistence mounts before reopening admission.
|
override material so a configured tag moving during the operation cannot change the selected
|
||||||
|
image, and Compose is explicitly told never to build or pull. It will restart only core, then
|
||||||
|
verifies health, Pi version, settings/model smoke, non-secret rendered configuration, and
|
||||||
|
persistence mounts before reopening admission.
|
||||||
|
|
||||||
Use `pi restart --yes` when there are already no active sessions. Do not substitute `thothctl stop`
|
Use `pi restart --yes` when there are already no active sessions. Do not substitute `thothctl stop`
|
||||||
and `thothctl start` or raw Compose commands for this reload workflow.
|
and `thothctl start` or raw Compose commands for this reload workflow.
|
||||||
@@ -110,8 +113,8 @@ volumes.
|
|||||||
## Recover a failed lifecycle operation
|
## Recover a failed lifecycle operation
|
||||||
|
|
||||||
If a restart or update fails after core recreation, leave maintenance enabled and preserve the
|
If a restart or update fails after core recreation, leave maintenance enabled and preserve the
|
||||||
reported recovery state. Do not delete `.thothctl`, state files, containers, or volumes. Inspect
|
reported recovery state and transaction override. Do not delete `.thothctl`, state files,
|
||||||
status and sanitized logs:
|
containers, or volumes. Inspect status and sanitized logs:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
"$THTCTL" --installation "$INSTALLATION" pi maintenance status
|
"$THTCTL" --installation "$INSTALLATION" pi maintenance status
|
||||||
|
|||||||
@@ -252,6 +252,9 @@ test("shows a seven-step host-terminal workflow in scrollable platform tabs", as
|
|||||||
expect(linux).toHaveTextContent("~/bin/thothctl --installation ~/thothii-installation.yaml pi restart --yes --drain");
|
expect(linux).toHaveTextContent("~/bin/thothctl --installation ~/thothii-installation.yaml pi restart --yes --drain");
|
||||||
expect(linux).toHaveTextContent("~/bin/thothctl --installation ~/thothii-installation.yaml pi update --version <VERSION> --source build --yes --drain");
|
expect(linux).toHaveTextContent("~/bin/thothctl --installation ~/thothii-installation.yaml pi update --version <VERSION> --source build --yes --drain");
|
||||||
expect(linux).toHaveTextContent("~/bin/thothctl --installation ~/thothii-installation.yaml pi update --version <VERSION> --source pull --image <IMAGE>@sha256:<DIGEST> --yes --drain");
|
expect(linux).toHaveTextContent("~/bin/thothctl --installation ~/thothii-installation.yaml pi update --version <VERSION> --source pull --image <IMAGE>@sha256:<DIGEST> --yes --drain");
|
||||||
|
expect(linux).toHaveTextContent("<VERSION> is a placeholder. Replace it with the Pi release/version you want to install.");
|
||||||
|
expect(linux).toHaveTextContent("~/bin/thothctl --installation ~/thothii-installation.yaml pi maintenance status");
|
||||||
|
expect(linux).toHaveTextContent("~/bin/thothctl --installation ~/thothii-installation.yaml pi logs");
|
||||||
expect(linux).toHaveTextContent("~/bin/thothctl --installation ~/thothii-installation.yaml pi rollback --yes");
|
expect(linux).toHaveTextContent("~/bin/thothctl --installation ~/thothii-installation.yaml pi rollback --yes");
|
||||||
expect(linux).toHaveTextContent("~/bin/thothctl --installation ~/thothii-installation.yaml pi maintenance recover --yes");
|
expect(linux).toHaveTextContent("~/bin/thothctl --installation ~/thothii-installation.yaml pi maintenance recover --yes");
|
||||||
expect(linux).not.toHaveTextContent("~/.pi/agent/");
|
expect(linux).not.toHaveTextContent("~/.pi/agent/");
|
||||||
@@ -274,6 +277,9 @@ test("shows a seven-step host-terminal workflow in scrollable platform tabs", as
|
|||||||
expect(macos).toHaveTextContent("~/bin/thothctl --installation ~/thothii-installation.yaml pi restart --yes --drain");
|
expect(macos).toHaveTextContent("~/bin/thothctl --installation ~/thothii-installation.yaml pi restart --yes --drain");
|
||||||
expect(macos).toHaveTextContent("~/bin/thothctl --installation ~/thothii-installation.yaml pi update --version <VERSION> --source build --yes --drain");
|
expect(macos).toHaveTextContent("~/bin/thothctl --installation ~/thothii-installation.yaml pi update --version <VERSION> --source build --yes --drain");
|
||||||
expect(macos).toHaveTextContent("~/bin/thothctl --installation ~/thothii-installation.yaml pi update --version <VERSION> --source pull --image <IMAGE>@sha256:<DIGEST> --yes --drain");
|
expect(macos).toHaveTextContent("~/bin/thothctl --installation ~/thothii-installation.yaml pi update --version <VERSION> --source pull --image <IMAGE>@sha256:<DIGEST> --yes --drain");
|
||||||
|
expect(macos).toHaveTextContent("<VERSION> is a placeholder. Replace it with the Pi release/version you want to install.");
|
||||||
|
expect(macos).toHaveTextContent("~/bin/thothctl --installation ~/thothii-installation.yaml pi maintenance status");
|
||||||
|
expect(macos).toHaveTextContent("~/bin/thothctl --installation ~/thothii-installation.yaml pi logs");
|
||||||
expect(macos).toHaveTextContent("~/bin/thothctl --installation ~/thothii-installation.yaml pi rollback --yes");
|
expect(macos).toHaveTextContent("~/bin/thothctl --installation ~/thothii-installation.yaml pi rollback --yes");
|
||||||
expect(macos).toHaveTextContent("~/bin/thothctl --installation ~/thothii-installation.yaml pi maintenance recover --yes");
|
expect(macos).toHaveTextContent("~/bin/thothctl --installation ~/thothii-installation.yaml pi maintenance recover --yes");
|
||||||
|
|
||||||
@@ -295,6 +301,9 @@ test("shows a seven-step host-terminal workflow in scrollable platform tabs", as
|
|||||||
expect(windows).toHaveTextContent('& (Resolve-Path "~\\bin\\thothctl-windows-amd64.exe") --installation (Resolve-Path "~\\thothii-installation.yaml") pi restart --yes --drain');
|
expect(windows).toHaveTextContent('& (Resolve-Path "~\\bin\\thothctl-windows-amd64.exe") --installation (Resolve-Path "~\\thothii-installation.yaml") pi restart --yes --drain');
|
||||||
expect(windows).toHaveTextContent('& (Resolve-Path "~\\bin\\thothctl-windows-amd64.exe") --installation (Resolve-Path "~\\thothii-installation.yaml") pi update --version <VERSION> --source build --yes --drain');
|
expect(windows).toHaveTextContent('& (Resolve-Path "~\\bin\\thothctl-windows-amd64.exe") --installation (Resolve-Path "~\\thothii-installation.yaml") pi update --version <VERSION> --source build --yes --drain');
|
||||||
expect(windows).toHaveTextContent('& (Resolve-Path "~\\bin\\thothctl-windows-amd64.exe") --installation (Resolve-Path "~\\thothii-installation.yaml") pi update --version <VERSION> --source pull --image <IMAGE>@sha256:<DIGEST> --yes --drain');
|
expect(windows).toHaveTextContent('& (Resolve-Path "~\\bin\\thothctl-windows-amd64.exe") --installation (Resolve-Path "~\\thothii-installation.yaml") pi update --version <VERSION> --source pull --image <IMAGE>@sha256:<DIGEST> --yes --drain');
|
||||||
|
expect(windows).toHaveTextContent("<VERSION> is a placeholder. Replace it with the Pi release/version you want to install.");
|
||||||
|
expect(windows).toHaveTextContent('& (Resolve-Path "~\\bin\\thothctl-windows-amd64.exe") --installation (Resolve-Path "~\\thothii-installation.yaml") pi maintenance status');
|
||||||
|
expect(windows).toHaveTextContent('& (Resolve-Path "~\\bin\\thothctl-windows-amd64.exe") --installation (Resolve-Path "~\\thothii-installation.yaml") pi logs');
|
||||||
expect(windows).toHaveTextContent('& (Resolve-Path "~\\bin\\thothctl-windows-amd64.exe") --installation (Resolve-Path "~\\thothii-installation.yaml") pi rollback --yes');
|
expect(windows).toHaveTextContent('& (Resolve-Path "~\\bin\\thothctl-windows-amd64.exe") --installation (Resolve-Path "~\\thothii-installation.yaml") pi rollback --yes');
|
||||||
expect(windows).toHaveTextContent('& (Resolve-Path "~\\bin\\thothctl-windows-amd64.exe") --installation (Resolve-Path "~\\thothii-installation.yaml") pi maintenance recover --yes');
|
expect(windows).toHaveTextContent('& (Resolve-Path "~\\bin\\thothctl-windows-amd64.exe") --installation (Resolve-Path "~\\thothii-installation.yaml") pi maintenance recover --yes');
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -126,7 +126,7 @@ const piPlatforms: Array<{ id: PiPlatform; label: string; details: PiPlatformDet
|
|||||||
restartCommand: "~/bin/thothctl --installation ~/thothii-installation.yaml pi restart --yes --drain",
|
restartCommand: "~/bin/thothctl --installation ~/thothii-installation.yaml pi restart --yes --drain",
|
||||||
updateCommand: "~/bin/thothctl --installation ~/thothii-installation.yaml pi update --version <VERSION> --source build --yes --drain",
|
updateCommand: "~/bin/thothctl --installation ~/thothii-installation.yaml pi update --version <VERSION> --source build --yes --drain",
|
||||||
pullCommand: "~/bin/thothctl --installation ~/thothii-installation.yaml pi update --version <VERSION> --source pull --image <IMAGE>@sha256:<DIGEST> --yes --drain",
|
pullCommand: "~/bin/thothctl --installation ~/thothii-installation.yaml pi update --version <VERSION> --source pull --image <IMAGE>@sha256:<DIGEST> --yes --drain",
|
||||||
recoveryCommands: "~/bin/thothctl --installation ~/thothii-installation.yaml pi rollback --yes\n~/bin/thothctl --installation ~/thothii-installation.yaml pi maintenance recover --yes",
|
recoveryCommands: "~/bin/thothctl --installation ~/thothii-installation.yaml pi maintenance status\n~/bin/thothctl --installation ~/thothii-installation.yaml pi logs\n~/bin/thothctl --installation ~/thothii-installation.yaml pi rollback --yes\n~/bin/thothctl --installation ~/thothii-installation.yaml pi maintenance recover --yes",
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
@@ -140,7 +140,7 @@ const piPlatforms: Array<{ id: PiPlatform; label: string; details: PiPlatformDet
|
|||||||
restartCommand: "~/bin/thothctl --installation ~/thothii-installation.yaml pi restart --yes --drain",
|
restartCommand: "~/bin/thothctl --installation ~/thothii-installation.yaml pi restart --yes --drain",
|
||||||
updateCommand: "~/bin/thothctl --installation ~/thothii-installation.yaml pi update --version <VERSION> --source build --yes --drain",
|
updateCommand: "~/bin/thothctl --installation ~/thothii-installation.yaml pi update --version <VERSION> --source build --yes --drain",
|
||||||
pullCommand: "~/bin/thothctl --installation ~/thothii-installation.yaml pi update --version <VERSION> --source pull --image <IMAGE>@sha256:<DIGEST> --yes --drain",
|
pullCommand: "~/bin/thothctl --installation ~/thothii-installation.yaml pi update --version <VERSION> --source pull --image <IMAGE>@sha256:<DIGEST> --yes --drain",
|
||||||
recoveryCommands: "~/bin/thothctl --installation ~/thothii-installation.yaml pi rollback --yes\n~/bin/thothctl --installation ~/thothii-installation.yaml pi maintenance recover --yes",
|
recoveryCommands: "~/bin/thothctl --installation ~/thothii-installation.yaml pi maintenance status\n~/bin/thothctl --installation ~/thothii-installation.yaml pi logs\n~/bin/thothctl --installation ~/thothii-installation.yaml pi rollback --yes\n~/bin/thothctl --installation ~/thothii-installation.yaml pi maintenance recover --yes",
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
@@ -154,7 +154,7 @@ const piPlatforms: Array<{ id: PiPlatform; label: string; details: PiPlatformDet
|
|||||||
restartCommand: '& (Resolve-Path "~\\bin\\thothctl-windows-amd64.exe") --installation (Resolve-Path "~\\thothii-installation.yaml") pi restart --yes --drain',
|
restartCommand: '& (Resolve-Path "~\\bin\\thothctl-windows-amd64.exe") --installation (Resolve-Path "~\\thothii-installation.yaml") pi restart --yes --drain',
|
||||||
updateCommand: '& (Resolve-Path "~\\bin\\thothctl-windows-amd64.exe") --installation (Resolve-Path "~\\thothii-installation.yaml") pi update --version <VERSION> --source build --yes --drain',
|
updateCommand: '& (Resolve-Path "~\\bin\\thothctl-windows-amd64.exe") --installation (Resolve-Path "~\\thothii-installation.yaml") pi update --version <VERSION> --source build --yes --drain',
|
||||||
pullCommand: '& (Resolve-Path "~\\bin\\thothctl-windows-amd64.exe") --installation (Resolve-Path "~\\thothii-installation.yaml") pi update --version <VERSION> --source pull --image <IMAGE>@sha256:<DIGEST> --yes --drain',
|
pullCommand: '& (Resolve-Path "~\\bin\\thothctl-windows-amd64.exe") --installation (Resolve-Path "~\\thothii-installation.yaml") pi update --version <VERSION> --source pull --image <IMAGE>@sha256:<DIGEST> --yes --drain',
|
||||||
recoveryCommands: '& (Resolve-Path "~\\bin\\thothctl-windows-amd64.exe") --installation (Resolve-Path "~\\thothii-installation.yaml") pi rollback --yes\n& (Resolve-Path "~\\bin\\thothctl-windows-amd64.exe") --installation (Resolve-Path "~\\thothii-installation.yaml") pi maintenance recover --yes',
|
recoveryCommands: '& (Resolve-Path "~\\bin\\thothctl-windows-amd64.exe") --installation (Resolve-Path "~\\thothii-installation.yaml") pi maintenance status\n& (Resolve-Path "~\\bin\\thothctl-windows-amd64.exe") --installation (Resolve-Path "~\\thothii-installation.yaml") pi logs\n& (Resolve-Path "~\\bin\\thothctl-windows-amd64.exe") --installation (Resolve-Path "~\\thothii-installation.yaml") pi rollback --yes\n& (Resolve-Path "~\\bin\\thothctl-windows-amd64.exe") --installation (Resolve-Path "~\\thothii-installation.yaml") pi maintenance recover --yes',
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
];
|
];
|
||||||
@@ -199,13 +199,14 @@ function PiInstructionSteps({ details }: { details: PiPlatformDetails }) {
|
|||||||
<li>
|
<li>
|
||||||
<h4 className="font-semibold text-foreground">Update the Pi version</h4>
|
<h4 className="font-semibold text-foreground">Update the Pi version</h4>
|
||||||
<p className="mt-1 text-muted-foreground">Use a build update only when changing the bundled Pi version.</p>
|
<p className="mt-1 text-muted-foreground">Use a build update only when changing the bundled Pi version.</p>
|
||||||
|
<p className="mt-1 text-muted-foreground">{"<VERSION> is a placeholder. Replace it with the Pi release/version you want to install."}</p>
|
||||||
<PiCodeBlock className="mt-2">{details.updateCommand}</PiCodeBlock>
|
<PiCodeBlock className="mt-2">{details.updateCommand}</PiCodeBlock>
|
||||||
<p className="mt-2 text-[11px] text-muted-foreground">Advanced: pull an immutable, digest-pinned image.</p>
|
<p className="mt-2 text-[11px] text-muted-foreground">Advanced: pull an immutable, digest-pinned image.</p>
|
||||||
<PiCodeBlock className="mt-1 text-[11px] text-muted-foreground">{details.pullCommand}</PiCodeBlock>
|
<PiCodeBlock className="mt-1 text-[11px] text-muted-foreground">{details.pullCommand}</PiCodeBlock>
|
||||||
</li>
|
</li>
|
||||||
<li>
|
<li>
|
||||||
<h4 className="font-semibold text-foreground">Recover a failed update</h4>
|
<h4 className="font-semibold text-foreground">Recover a failed update</h4>
|
||||||
<p className="mt-1 text-muted-foreground">For a failed update, use rollback. For restart or update maintenance recovery after repairing the reported problem, use maintenance recovery.</p>
|
<p className="mt-1 text-muted-foreground">Check maintenance status and bounded sanitized Pi logs first. For a failed update, use rollback. For restart or update maintenance recovery after repairing the reported problem, use maintenance recovery.</p>
|
||||||
<PiCodeBlock className="mt-2 text-[11px] text-muted-foreground">{details.recoveryCommands}</PiCodeBlock>
|
<PiCodeBlock className="mt-2 text-[11px] text-muted-foreground">{details.recoveryCommands}</PiCodeBlock>
|
||||||
</li>
|
</li>
|
||||||
</ol>;
|
</ol>;
|
||||||
|
|||||||
@@ -1131,10 +1131,21 @@ NODE
|
|||||||
|
|
||||||
verify_pi_management_guide() {
|
verify_pi_management_guide() {
|
||||||
local guide="$root/docs/install/pi-management.md"
|
local guide="$root/docs/install/pi-management.md"
|
||||||
|
local lifecycle_contract="$root/docs/contracts/thothctl-pi.md"
|
||||||
[[ -f "$guide" ]] || {
|
[[ -f "$guide" ]] || {
|
||||||
echo "missing Pi management guide: docs/install/pi-management.md" >&2
|
echo "missing Pi management guide: docs/install/pi-management.md" >&2
|
||||||
return 1
|
return 1
|
||||||
}
|
}
|
||||||
|
[[ -f "$lifecycle_contract" ]] || {
|
||||||
|
echo "missing Pi lifecycle contract: docs/contracts/thothctl-pi.md" >&2
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
require_text "$lifecycle_contract" "Pi lifecycle contract" \
|
||||||
|
"io.thothii.pi.version"
|
||||||
|
if grep -Fq 'org.opencontainers.image.version' "$lifecycle_contract"; then
|
||||||
|
echo "Pi lifecycle contract must use io.thothii.pi.version, not org.opencontainers.image.version" >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
require_headings "$guide" "Pi management guide" \
|
require_headings "$guide" "Pi management guide" \
|
||||||
"Choose application defaults" \
|
"Choose application defaults" \
|
||||||
"Edit the provider catalog and enabled-model policy" \
|
"Edit the provider catalog and enabled-model policy" \
|
||||||
|
|||||||
@@ -333,7 +333,11 @@ func piCommand(ctx context.Context, installation config.Installation, runner com
|
|||||||
fmt.Fprintf(stdout, "Pi core restarted with the existing image; version %s readiness and smoke checks passed.\n", output.Sanitize(result.Version, secretValues))
|
fmt.Fprintf(stdout, "Pi core restarted with the existing image; version %s readiness and smoke checks passed.\n", output.Sanitize(result.Version, secretValues))
|
||||||
return 0
|
return 0
|
||||||
case "update":
|
case "update":
|
||||||
request, err := parsePiUpdateArgs(args[1:], installation.UpdateStatePath())
|
request, err := parsePiUpdateArgs(
|
||||||
|
args[1:],
|
||||||
|
installation.UpdateStatePath(),
|
||||||
|
installation.RestartStatePath(),
|
||||||
|
)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return commandUsageError(stderr, err.Error())
|
return commandUsageError(stderr, err.Error())
|
||||||
}
|
}
|
||||||
@@ -351,7 +355,13 @@ func piCommand(ctx context.Context, installation config.Installation, runner com
|
|||||||
if len(args) != 2 || args[1] != "--yes" {
|
if len(args) != 2 || args[1] != "--yes" {
|
||||||
return commandUsageError(stderr, "pi rollback requires --yes")
|
return commandUsageError(stderr, "pi rollback requires --yes")
|
||||||
}
|
}
|
||||||
result, err := pi.Rollback(ctx, controlled, installation.UpdateStatePath(), true)
|
result, err := pi.Rollback(
|
||||||
|
ctx,
|
||||||
|
controlled,
|
||||||
|
installation.UpdateStatePath(),
|
||||||
|
installation.RestartStatePath(),
|
||||||
|
true,
|
||||||
|
)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return piFailure(stderr, err, secretValues)
|
return piFailure(stderr, err, secretValues)
|
||||||
}
|
}
|
||||||
@@ -489,8 +499,8 @@ func parsePiConfigureArgs(args []string) (pi.Defaults, error) {
|
|||||||
return value, nil
|
return value, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func parsePiUpdateArgs(args []string, statePath string) (pi.Request, error) {
|
func parsePiUpdateArgs(args []string, statePath, restartStatePath string) (pi.Request, error) {
|
||||||
request := pi.Request{StatePath: statePath}
|
request := pi.Request{StatePath: statePath, RestartStatePath: restartStatePath}
|
||||||
for len(args) > 0 {
|
for len(args) > 0 {
|
||||||
switch args[0] {
|
switch args[0] {
|
||||||
case "--version":
|
case "--version":
|
||||||
@@ -566,7 +576,7 @@ func parsePiRestartArgs(args []string, restartStatePath, updateStatePath string)
|
|||||||
|
|
||||||
func piFailure(stderr io.Writer, err error, secretValues []string) int {
|
func piFailure(stderr io.Writer, err error, secretValues []string) int {
|
||||||
code := 1
|
code := 1
|
||||||
if errors.Is(err, pi.ErrConfirmationRequired) || errors.Is(err, pi.ErrInvalidRequest) || errors.Is(err, pi.ErrActiveSessions) || errors.Is(err, pi.ErrInterruptedUpdate) {
|
if errors.Is(err, pi.ErrConfirmationRequired) || errors.Is(err, pi.ErrInvalidRequest) || errors.Is(err, pi.ErrActiveSessions) || errors.Is(err, pi.ErrInterruptedUpdate) || errors.Is(err, pi.ErrInterruptedRestart) {
|
||||||
code = 2
|
code = 2
|
||||||
}
|
}
|
||||||
var childExit interface{ ExitCode() int }
|
var childExit interface{ ExitCode() int }
|
||||||
|
|||||||
@@ -274,6 +274,59 @@ func TestRunLogsRedactsAnUnlabelledDeclaredSecret(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRunPiLogsRedactsNestedAuthAndBundleScalars(t *testing.T) {
|
||||||
|
fixture := newCLIFixture(t, "")
|
||||||
|
authFile := filepath.Join(fixture.root, "pi-auth.json")
|
||||||
|
bundleFile := filepath.Join(fixture.root, "thothii.secrets")
|
||||||
|
if err := os.WriteFile(authFile, []byte(`{
|
||||||
|
"providers": {
|
||||||
|
"dummy-provider": {
|
||||||
|
"auth": {
|
||||||
|
"key": "dummy-canary-pi-log-json",
|
||||||
|
"nested": {"access": "dummy-canary-pi-log-nested"}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}`), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(bundleFile, []byte(
|
||||||
|
"MODEL_API_KEY=dummy-canary-pi-log-model\n"+
|
||||||
|
"DWH_PASSWORD=dummy-canary-pi-log-dwh\n",
|
||||||
|
), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
fixture.setEnvContents(t, "PI_AUTH_FILE="+authFile+"\nTHT_SECRETS_FILE="+bundleFile+"\n")
|
||||||
|
t.Setenv(
|
||||||
|
"THOTHCTL_FAKE_LOG",
|
||||||
|
"dummy-canary-pi-log-json dummy-canary-pi-log-nested "+
|
||||||
|
"dummy-canary-pi-log-model dummy-canary-pi-log-dwh",
|
||||||
|
)
|
||||||
|
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
exitCode := run(context.Background(), []string{
|
||||||
|
"--installation", fixture.installationPath, "pi", "logs",
|
||||||
|
}, &stdout, &stderr)
|
||||||
|
|
||||||
|
if exitCode != 0 {
|
||||||
|
t.Fatalf("run() exit code = %d, stderr = %q", exitCode, stderr.String())
|
||||||
|
}
|
||||||
|
for _, canary := range []string{
|
||||||
|
"dummy-canary-pi-log-json",
|
||||||
|
"dummy-canary-pi-log-nested",
|
||||||
|
"dummy-canary-pi-log-model",
|
||||||
|
"dummy-canary-pi-log-dwh",
|
||||||
|
} {
|
||||||
|
if strings.Contains(stdout.String()+stderr.String(), canary) {
|
||||||
|
t.Fatalf("pi logs exposed scalar %q: stdout=%q stderr=%q", canary, stdout.String(), stderr.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if strings.Count(stdout.String(), "[REDACTED]") != 4 {
|
||||||
|
t.Fatalf("pi logs = %q, want four scalar redactions", stdout.String())
|
||||||
|
}
|
||||||
|
assertInvocationContains(t, fixture.invocations(t), "logs", "--tail", "200", "core")
|
||||||
|
}
|
||||||
|
|
||||||
func TestRunResolvesComposeDotenvCommentsQuotesAndInterpolationForSecretFiles(t *testing.T) {
|
func TestRunResolvesComposeDotenvCommentsQuotesAndInterpolationForSecretFiles(t *testing.T) {
|
||||||
fixture := newCLIFixture(t, "")
|
fixture := newCLIFixture(t, "")
|
||||||
secretDirectory := filepath.Join(fixture.root, "secret directory")
|
secretDirectory := filepath.Join(fixture.root, "secret directory")
|
||||||
|
|||||||
@@ -2,15 +2,21 @@
|
|||||||
package output
|
package output
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"bytes"
|
||||||
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
|
"io"
|
||||||
"regexp"
|
"regexp"
|
||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"github.com/aritmolab/thothii/tools/thothctl/internal/safeio"
|
"github.com/aritmolab/thothii/tools/thothctl/internal/safeio"
|
||||||
|
"github.com/compose-spec/compose-go/v2/dotenv"
|
||||||
)
|
)
|
||||||
|
|
||||||
var credentialField = regexp.MustCompile(`(?im)(\b[\w.-]*(?:password|token|key)[\w.-]*\s*[:=]\s*)(?:"[^"\r\n]*"|'[^'\r\n]*'|[^\s,;]+)`)
|
var credentialField = regexp.MustCompile(`(?im)((?:"|')?[\w.-]*(?:password|token|key|secret|credential)[\w.-]*(?:"|')?\s*[:=]\s*)(?:"(?:\\.|[^"\\\r\n])*"|'[^'\r\n]*'|[^\s,;}]+)`)
|
||||||
|
|
||||||
|
var dotenvAssignment = regexp.MustCompile(`(?m)^\s*(?:export\s+)?[A-Za-z_][A-Za-z0-9_.-]*\s*=`)
|
||||||
|
|
||||||
const maxSecretFileBytes = 64 * 1024
|
const maxSecretFileBytes = 64 * 1024
|
||||||
|
|
||||||
@@ -18,6 +24,12 @@ const maxSecretSourceFiles = 32
|
|||||||
|
|
||||||
const maxSecretSourceBytes = 256 * 1024
|
const maxSecretSourceBytes = 256 * 1024
|
||||||
|
|
||||||
|
const maxSecretValuesPerFile = 1024
|
||||||
|
|
||||||
|
const maxSecretValues = 4096
|
||||||
|
|
||||||
|
const maxJSONSecretDepth = 32
|
||||||
|
|
||||||
const maxDiagnosticDetailBytes = 512
|
const maxDiagnosticDetailBytes = 512
|
||||||
|
|
||||||
// Sanitize redacts common credential fields and every supplied secret value.
|
// Sanitize redacts common credential fields and every supplied secret value.
|
||||||
@@ -28,6 +40,9 @@ func Sanitize(text string, secretValues []string) string {
|
|||||||
for _, value := range values {
|
for _, value := range values {
|
||||||
if value != "" {
|
if value != "" {
|
||||||
text = strings.ReplaceAll(text, value, "[REDACTED]")
|
text = strings.ReplaceAll(text, value, "[REDACTED]")
|
||||||
|
if encoded, err := json.Marshal(value); err == nil {
|
||||||
|
text = strings.ReplaceAll(text, string(encoded), "[REDACTED]")
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return text
|
return text
|
||||||
@@ -60,7 +75,7 @@ func SecretValuesFromFiles(paths []string) ([]string, error) {
|
|||||||
seen := make(map[string]struct{})
|
seen := make(map[string]struct{})
|
||||||
var totalBytes int64
|
var totalBytes int64
|
||||||
for _, path := range paths {
|
for _, path := range paths {
|
||||||
value, size, err := readSecretFile(path)
|
contents, size, err := readSecretFile(path)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
@@ -68,21 +83,116 @@ func SecretValuesFromFiles(paths []string) ([]string, error) {
|
|||||||
if totalBytes > maxSecretSourceBytes {
|
if totalBytes > maxSecretSourceBytes {
|
||||||
return nil, errors.New("declared secret file could not be read")
|
return nil, errors.New("declared secret file could not be read")
|
||||||
}
|
}
|
||||||
if value != "" {
|
extracted, err := extractSecretValues(contents)
|
||||||
|
if err != nil {
|
||||||
|
return nil, errors.New("declared secret file could not be read")
|
||||||
|
}
|
||||||
|
for _, value := range extracted {
|
||||||
|
if value == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
if _, exists := seen[value]; exists {
|
if _, exists := seen[value]; exists {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
values = append(values, value)
|
values = append(values, value)
|
||||||
seen[value] = struct{}{}
|
seen[value] = struct{}{}
|
||||||
|
if len(values) > maxSecretValues {
|
||||||
|
return nil, errors.New("declared secret file could not be read")
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return values, nil
|
return values, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func readSecretFile(path string) (string, int64, error) {
|
func readSecretFile(path string) ([]byte, int64, error) {
|
||||||
contents, err := safeio.ReadCanonicalRegular(path, maxSecretFileBytes)
|
contents, err := safeio.ReadCanonicalRegular(path, maxSecretFileBytes)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", 0, errors.New("declared secret file could not be read")
|
return nil, 0, errors.New("declared secret file could not be read")
|
||||||
}
|
}
|
||||||
return strings.TrimRight(string(contents), "\r\n"), int64(len(contents)), nil
|
return contents, int64(len(contents)), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func extractSecretValues(contents []byte) ([]string, error) {
|
||||||
|
whole := strings.TrimRight(string(contents), "\r\n")
|
||||||
|
trimmed := bytes.TrimSpace(contents)
|
||||||
|
if len(trimmed) == 0 {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
values := make([]string, 0, 8)
|
||||||
|
if whole != "" {
|
||||||
|
values = append(values, whole)
|
||||||
|
}
|
||||||
|
|
||||||
|
if trimmed[0] == '{' || trimmed[0] == '[' {
|
||||||
|
var document any
|
||||||
|
decoder := json.NewDecoder(bytes.NewReader(trimmed))
|
||||||
|
decoder.UseNumber()
|
||||||
|
if err := decoder.Decode(&document); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
var extra any
|
||||||
|
if err := decoder.Decode(&extra); !errors.Is(err, io.EOF) {
|
||||||
|
if err == nil {
|
||||||
|
return nil, errors.New("secret JSON contains multiple documents")
|
||||||
|
}
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
count := 0
|
||||||
|
if err := collectJSONSecretValues(document, 0, &count, &values); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return values, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
if dotenvAssignment.Match(trimmed) {
|
||||||
|
parsed, err := dotenv.Parse(bytes.NewReader(contents))
|
||||||
|
if err != nil || len(parsed) > maxSecretValuesPerFile {
|
||||||
|
return nil, errors.New("secret dotenv bundle is invalid")
|
||||||
|
}
|
||||||
|
keys := make([]string, 0, len(parsed))
|
||||||
|
for key := range parsed {
|
||||||
|
keys = append(keys, key)
|
||||||
|
}
|
||||||
|
sort.Strings(keys)
|
||||||
|
for _, key := range keys {
|
||||||
|
if parsed[key] != "" {
|
||||||
|
values = append(values, parsed[key])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return values, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func collectJSONSecretValues(value any, depth int, count *int, values *[]string) error {
|
||||||
|
if depth > maxJSONSecretDepth {
|
||||||
|
return errors.New("secret JSON nesting is too deep")
|
||||||
|
}
|
||||||
|
switch typed := value.(type) {
|
||||||
|
case map[string]any:
|
||||||
|
keys := make([]string, 0, len(typed))
|
||||||
|
for key := range typed {
|
||||||
|
keys = append(keys, key)
|
||||||
|
}
|
||||||
|
sort.Strings(keys)
|
||||||
|
for _, key := range keys {
|
||||||
|
if err := collectJSONSecretValues(typed[key], depth+1, count, values); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
case []any:
|
||||||
|
for _, item := range typed {
|
||||||
|
if err := collectJSONSecretValues(item, depth+1, count, values); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
default:
|
||||||
|
*count++
|
||||||
|
if *count > maxSecretValuesPerFile {
|
||||||
|
return errors.New("secret JSON contains too many scalar values")
|
||||||
|
}
|
||||||
|
if scalar, ok := typed.(string); ok && scalar != "" {
|
||||||
|
*values = append(*values, scalar)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ package output
|
|||||||
import (
|
import (
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -34,6 +35,113 @@ func TestSanitizeRedactsSecretFileContents(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestSecretValuesFromFilesRedactsNestedJSONScalars(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
secretFile := filepath.Join(physicalTempDir(t), "pi-auth.json")
|
||||||
|
contents := `{
|
||||||
|
"providers": {
|
||||||
|
"dummy-provider": {
|
||||||
|
"auth": {
|
||||||
|
"key": "dummy-canary-json-key",
|
||||||
|
"tokens": {
|
||||||
|
"access": "dummy-canary-json-access",
|
||||||
|
"refresh": "dummy-canary-json-refresh"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}`
|
||||||
|
if err := os.WriteFile(secretFile, []byte(contents), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
secrets, err := SecretValuesFromFiles([]string{secretFile})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("SecretValuesFromFiles() error = %v", err)
|
||||||
|
}
|
||||||
|
got := Sanitize(
|
||||||
|
"unlabelled dummy-canary-json-key dummy-canary-json-access dummy-canary-json-refresh",
|
||||||
|
secrets,
|
||||||
|
)
|
||||||
|
for _, canary := range []string{
|
||||||
|
"dummy-canary-json-key",
|
||||||
|
"dummy-canary-json-access",
|
||||||
|
"dummy-canary-json-refresh",
|
||||||
|
} {
|
||||||
|
if strings.Contains(got, canary) {
|
||||||
|
t.Fatalf("Sanitize() exposed nested JSON scalar %q: %q", canary, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSecretValuesFromFilesRedactsEveryDotenvBundleValue(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
secretFile := filepath.Join(physicalTempDir(t), "thothii.secrets")
|
||||||
|
contents := "MODEL_API_KEY=dummy-canary-bundle-model\n" +
|
||||||
|
"DWH_PASSWORD='dummy-canary-bundle-dwh'\n" +
|
||||||
|
"SESSION_TOKEN=\"dummy-canary-bundle-session\"\n"
|
||||||
|
if err := os.WriteFile(secretFile, []byte(contents), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
secrets, err := SecretValuesFromFiles([]string{secretFile})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("SecretValuesFromFiles() error = %v", err)
|
||||||
|
}
|
||||||
|
got := Sanitize(
|
||||||
|
"unlabelled dummy-canary-bundle-model dummy-canary-bundle-dwh dummy-canary-bundle-session",
|
||||||
|
secrets,
|
||||||
|
)
|
||||||
|
for _, canary := range []string{
|
||||||
|
"dummy-canary-bundle-model",
|
||||||
|
"dummy-canary-bundle-dwh",
|
||||||
|
"dummy-canary-bundle-session",
|
||||||
|
} {
|
||||||
|
if strings.Contains(got, canary) {
|
||||||
|
t.Fatalf("Sanitize() exposed dotenv bundle scalar %q: %q", canary, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSanitizeRecognizesQuotedCredentialKeys(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
got := Sanitize(`{"key":"dummy-canary-quoted-key","safe":"visible"}`, nil)
|
||||||
|
if strings.Contains(got, "dummy-canary-quoted-key") || !strings.Contains(got, `"safe":"visible"`) {
|
||||||
|
t.Fatalf("Sanitize() = %q, want only the quoted credential field redacted", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSecretValuesFromFilesRejectsMalformedJSONAndExcessiveScalars(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
t.Run("malformed", func(t *testing.T) {
|
||||||
|
secretFile := filepath.Join(physicalTempDir(t), "malformed-auth.json")
|
||||||
|
if err := os.WriteFile(secretFile, []byte(`{"auth":{"key":"dummy-canary-malformed"}`), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := SecretValuesFromFiles([]string{secretFile}); err == nil {
|
||||||
|
t.Fatal("SecretValuesFromFiles() error = nil, want malformed-JSON failure")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("scalar bound", func(t *testing.T) {
|
||||||
|
secretFile := filepath.Join(physicalTempDir(t), "many-auth-values.json")
|
||||||
|
values := make([]string, 1025)
|
||||||
|
for index := range values {
|
||||||
|
values[index] = `"dummy-canary-value"`
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(secretFile, []byte("["+strings.Join(values, ",")+"]"), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := SecretValuesFromFiles([]string{secretFile}); err == nil {
|
||||||
|
t.Fatal("SecretValuesFromFiles() error = nil, want scalar-count failure")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
func TestSecretValuesFromFilesRejectsOversizedFiles(t *testing.T) {
|
func TestSecretValuesFromFilesRejectsOversizedFiles(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
var (
|
var (
|
||||||
errInterruptedRestart = errors.New("a previous Pi restart is incomplete; recover lifecycle maintenance before restarting again")
|
ErrInterruptedRestart = errors.New("a previous Pi restart is incomplete; recover lifecycle maintenance before another Pi lifecycle operation")
|
||||||
errRestartImageDrift = errors.New("core image changed during Pi restart")
|
errRestartImageDrift = errors.New("core image changed during Pi restart")
|
||||||
errRestartConfigurationDrift = errors.New("external endpoint configuration changed during Pi restart")
|
errRestartConfigurationDrift = errors.New("external endpoint configuration changed during Pi restart")
|
||||||
errRestartMountDrift = errors.New("core persistence mount contract changed during Pi restart")
|
errRestartMountDrift = errors.New("core persistence mount contract changed during Pi restart")
|
||||||
@@ -70,14 +70,7 @@ func restartWithHooks(
|
|||||||
} else if err != nil && !errors.Is(err, os.ErrNotExist) {
|
} else if err != nil && !errors.Is(err, os.ErrNotExist) {
|
||||||
return RestartResult{StatePath: request.StatePath}, err
|
return RestartResult{StatePath: request.StatePath}, err
|
||||||
}
|
}
|
||||||
if state, err := readState(request.StatePath); err == nil {
|
if err := prepareLifecycleMutation(request.StatePath, hooks.removeFile); err != nil {
|
||||||
if err := validateRestartRecoveryState(state); err != nil {
|
|
||||||
return RestartResult{StatePath: request.StatePath}, err
|
|
||||||
}
|
|
||||||
if state.MutationStarted {
|
|
||||||
return RestartResult{StatePath: request.StatePath}, errInterruptedRestart
|
|
||||||
}
|
|
||||||
} else if err != nil && !errors.Is(err, os.ErrNotExist) {
|
|
||||||
return RestartResult{StatePath: request.StatePath}, err
|
return RestartResult{StatePath: request.StatePath}, err
|
||||||
}
|
}
|
||||||
clearMaintenance := true
|
clearMaintenance := true
|
||||||
@@ -124,8 +117,13 @@ func restartWithHooks(
|
|||||||
return RestartResult{StatePath: request.StatePath, Version: version}, err
|
return RestartResult{StatePath: request.StatePath, Version: version}, err
|
||||||
}
|
}
|
||||||
previous.ConfigurationSHA = configured.ConfigurationSHA
|
previous.ConfigurationSHA = configured.ConfigurationSHA
|
||||||
|
transaction := lifecycleTransaction(request.StatePath)
|
||||||
|
previous.Reference = lifecycleImageTag(transaction, "restart")
|
||||||
|
if err := tagImage(ctx, runner, previous.ID, previous.Reference, "restart image pin"); err != nil {
|
||||||
|
return RestartResult{StatePath: request.StatePath, Version: version}, err
|
||||||
|
}
|
||||||
state = State{
|
state = State{
|
||||||
Transaction: lifecycleTransaction(request.StatePath),
|
Transaction: transaction,
|
||||||
Phase: PhasePreflight,
|
Phase: PhasePreflight,
|
||||||
Target: Target{Version: version, Source: "restart"},
|
Target: Target{Version: version, Source: "restart"},
|
||||||
Previous: previous,
|
Previous: previous,
|
||||||
@@ -133,6 +131,11 @@ func restartWithHooks(
|
|||||||
if err := hooks.writeState(request.StatePath, state); err != nil {
|
if err := hooks.writeState(request.StatePath, state); err != nil {
|
||||||
return RestartResult{StatePath: request.StatePath, Version: version}, err
|
return RestartResult{StatePath: request.StatePath, Version: version}, err
|
||||||
}
|
}
|
||||||
|
overridePath := lifecycleOverridePath(request.StatePath, transaction)
|
||||||
|
if err := writeLifecycleOverride(overridePath, previous.Reference); err != nil {
|
||||||
|
return RestartResult{StatePath: request.StatePath, Version: version}, err
|
||||||
|
}
|
||||||
|
lifecycle := composeOverrideRunner{Runner: runner, path: overridePath}
|
||||||
if running, err := activeSessions(ctx, runner); err != nil {
|
if running, err := activeSessions(ctx, runner); err != nil {
|
||||||
return RestartResult{StatePath: request.StatePath, Version: version}, err
|
return RestartResult{StatePath: request.StatePath, Version: version}, err
|
||||||
} else if running {
|
} else if running {
|
||||||
@@ -144,23 +147,26 @@ func restartWithHooks(
|
|||||||
}
|
}
|
||||||
mutationStarted = true
|
mutationStarted = true
|
||||||
clearMaintenance = false
|
clearMaintenance = false
|
||||||
if err := recreateCore(ctx, runner); err != nil {
|
if err := recreateCoreWithoutImageChanges(ctx, lifecycle); err != nil {
|
||||||
return RestartResult{StatePath: request.StatePath, Version: version}, recoveryRequired("Pi restart core recreation failed", err)
|
return RestartResult{StatePath: request.StatePath, Version: version}, recoveryRequired("Pi restart core recreation failed", err)
|
||||||
}
|
}
|
||||||
if err := ensureMaintenance(ctx, runner); err != nil {
|
if err := ensureMaintenance(ctx, lifecycle); err != nil {
|
||||||
return RestartResult{StatePath: request.StatePath, Version: version}, recoveryRequired("Pi restart maintenance proof failed", err)
|
return RestartResult{StatePath: request.StatePath, Version: version}, recoveryRequired("Pi restart maintenance proof failed", err)
|
||||||
}
|
}
|
||||||
state.Phase = PhaseRecreated
|
state.Phase = PhaseRecreated
|
||||||
if err := hooks.writeState(request.StatePath, state); err != nil {
|
if err := hooks.writeState(request.StatePath, state); err != nil {
|
||||||
return RestartResult{StatePath: request.StatePath, Version: version}, recoveryRequired("Pi restart recreation state could not be recorded", err)
|
return RestartResult{StatePath: request.StatePath, Version: version}, recoveryRequired("Pi restart recreation state could not be recorded", err)
|
||||||
}
|
}
|
||||||
if err := verifyRestart(ctx, runner, version, previous); err != nil {
|
if err := verifyRestart(ctx, lifecycle, version, previous); err != nil {
|
||||||
return RestartResult{StatePath: request.StatePath, Version: version}, recoveryRequired("Pi restart verification failed", err)
|
return RestartResult{StatePath: request.StatePath, Version: version}, recoveryRequired("Pi restart verification failed", err)
|
||||||
}
|
}
|
||||||
state.Phase = PhaseVerified
|
state.Phase = PhaseVerified
|
||||||
if err := hooks.writeState(request.StatePath, state); err != nil {
|
if err := hooks.writeState(request.StatePath, state); err != nil {
|
||||||
return RestartResult{StatePath: request.StatePath, Version: version}, recoveryRequired("Pi restart verification state could not be recorded", err)
|
return RestartResult{StatePath: request.StatePath, Version: version}, recoveryRequired("Pi restart verification state could not be recorded", err)
|
||||||
}
|
}
|
||||||
|
if err := hooks.removeFile(overridePath); err != nil {
|
||||||
|
return RestartResult{StatePath: request.StatePath, Version: version}, recoveryRequired("Pi restart image override could not be removed", err)
|
||||||
|
}
|
||||||
if err := hooks.removeFile(request.StatePath); err != nil {
|
if err := hooks.removeFile(request.StatePath); err != nil {
|
||||||
return RestartResult{StatePath: request.StatePath, Version: version}, recoveryRequired("Pi restart recovery state could not be removed", err)
|
return RestartResult{StatePath: request.StatePath, Version: version}, recoveryRequired("Pi restart recovery state could not be removed", err)
|
||||||
}
|
}
|
||||||
@@ -168,6 +174,28 @@ func restartWithHooks(
|
|||||||
return RestartResult{StatePath: request.StatePath, Version: version}, nil
|
return RestartResult{StatePath: request.StatePath, Version: version}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func recreateCoreWithoutImageChanges(ctx context.Context, runner Runner) error {
|
||||||
|
result, err := runCompose(
|
||||||
|
ctx,
|
||||||
|
runner,
|
||||||
|
"up",
|
||||||
|
"--detach",
|
||||||
|
"--wait",
|
||||||
|
"--wait-timeout",
|
||||||
|
"45",
|
||||||
|
"--no-deps",
|
||||||
|
"--force-recreate",
|
||||||
|
"--no-build",
|
||||||
|
"--pull",
|
||||||
|
"never",
|
||||||
|
"core",
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
return commandError("core recreation", result, err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func verifyRestart(ctx context.Context, runner Runner, wanted string, previous Image) error {
|
func verifyRestart(ctx context.Context, runner Runner, wanted string, previous Image) error {
|
||||||
if err := Doctor(ctx, runner); err != nil {
|
if err := Doctor(ctx, runner); err != nil {
|
||||||
return err
|
return err
|
||||||
@@ -223,14 +251,25 @@ func RecoverLifecycleMaintenance(
|
|||||||
if err := validateRestartRecoveryState(restartState); err != nil {
|
if err := validateRestartRecoveryState(restartState); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
restartOverride := lifecycleOverridePath(restartStatePath, restartState.Transaction)
|
||||||
if restartState.MutationStarted {
|
if restartState.MutationStarted {
|
||||||
if err := ensureMaintenance(ctx, runner); err != nil {
|
if err := tagImage(ctx, runner, restartState.Previous.ID, restartState.Previous.Reference, "restart recovery image pin"); err != nil {
|
||||||
|
return recoveryRequired("Pi restart recovery image pin could not be restored", err)
|
||||||
|
}
|
||||||
|
if err := writeLifecycleOverride(restartOverride, restartState.Previous.Reference); err != nil {
|
||||||
|
return recoveryRequired("Pi restart recovery image override could not be restored", err)
|
||||||
|
}
|
||||||
|
lifecycle := composeOverrideRunner{Runner: runner, path: restartOverride}
|
||||||
|
if err := ensureMaintenance(ctx, lifecycle); err != nil {
|
||||||
return recoveryRequired("Pi restart maintenance recovery failed", err)
|
return recoveryRequired("Pi restart maintenance recovery failed", err)
|
||||||
}
|
}
|
||||||
if err := verifyRestart(ctx, runner, restartState.Target.Version, restartState.Previous); err != nil {
|
if err := verifyRestart(ctx, lifecycle, restartState.Target.Version, restartState.Previous); err != nil {
|
||||||
return recoveryRequired("Pi restart recovery verification failed", err)
|
return recoveryRequired("Pi restart recovery verification failed", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if err := durableRemove(restartOverride); err != nil {
|
||||||
|
return recoveryRequired("Pi restart recovery image override could not be removed", err)
|
||||||
|
}
|
||||||
if err := durableRemove(restartStatePath); err != nil {
|
if err := durableRemove(restartStatePath); err != nil {
|
||||||
return recoveryRequired("Pi restart recovery state could not be removed", err)
|
return recoveryRequired("Pi restart recovery state could not be removed", err)
|
||||||
}
|
}
|
||||||
@@ -292,3 +331,30 @@ func validateRestartStatePaths(restartStatePath, updateStatePath string) error {
|
|||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func pairedRestartStatePath(updateStatePath string) string {
|
||||||
|
return filepath.Join(filepath.Dir(updateStatePath), "restart-state.json")
|
||||||
|
}
|
||||||
|
|
||||||
|
func prepareLifecycleMutation(restartStatePath string, removeFile func(string) error) error {
|
||||||
|
state, err := readState(restartStatePath)
|
||||||
|
if errors.Is(err, os.ErrNotExist) {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("restart recovery state could not be validated: %w", err)
|
||||||
|
}
|
||||||
|
if err := validateRestartRecoveryState(state); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if state.Phase != PhaseVerified || !state.MutationStarted {
|
||||||
|
return ErrInterruptedRestart
|
||||||
|
}
|
||||||
|
if err := removeFile(lifecycleOverridePath(restartStatePath, state.Transaction)); err != nil {
|
||||||
|
return recoveryRequired("verified restart override could not be cleaned up", err)
|
||||||
|
}
|
||||||
|
if err := removeFile(restartStatePath); err != nil {
|
||||||
|
return recoveryRequired("verified restart state could not be cleaned up", err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|||||||
@@ -55,15 +55,51 @@ func TestRestartDrainsRecreatesOnlyCoreAndRetainsImage(t *testing.T) {
|
|||||||
if sleepCalls != 1 {
|
if sleepCalls != 1 {
|
||||||
t.Fatalf("drain sleep calls = %d, want 1", sleepCalls)
|
t.Fatalf("drain sleep calls = %d, want 1", sleepCalls)
|
||||||
}
|
}
|
||||||
assertCalled(t, fake.calls, "up --detach --wait --wait-timeout 45 --no-deps --force-recreate core")
|
assertCalled(t, fake.calls, "up --detach --wait --wait-timeout 45 --no-deps --force-recreate --no-build --pull never core")
|
||||||
assertNotCalled(t, fake.calls, "build --pull")
|
assertNotCalled(t, fake.calls, "compose build --pull")
|
||||||
assertNotCalled(t, fake.calls, "pull ")
|
for _, call := range fake.calls {
|
||||||
|
if strings.HasPrefix(call, "pull ") {
|
||||||
|
t.Fatalf("restart invoked direct image pull: %s", call)
|
||||||
|
}
|
||||||
|
}
|
||||||
assertNotCalled(t, fake.calls, "frontend")
|
assertNotCalled(t, fake.calls, "frontend")
|
||||||
if _, err := os.Stat(result.StatePath); !errors.Is(err, os.ErrNotExist) {
|
if _, err := os.Stat(result.StatePath); !errors.Is(err, os.ErrNotExist) {
|
||||||
t.Fatalf("successful restart state still exists: %v", err)
|
t.Fatalf("successful restart state still exists: %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRestartPinsCapturedImageWhenConfiguredTagMovesBeforeRecreate(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
fake := newFakeRunner()
|
||||||
|
hooks := defaultLifecycleHooks
|
||||||
|
write := hooks.writeState
|
||||||
|
hooks.writeState = func(path string, state State) error {
|
||||||
|
if state.Phase == PhasePreflight && state.MutationStarted {
|
||||||
|
fake.tags[fake.configuredImage] = "sha256:moved-configured-tag"
|
||||||
|
fake.imageVersions["sha256:moved-configured-tag"] = "9.99.0"
|
||||||
|
}
|
||||||
|
return write(path, state)
|
||||||
|
}
|
||||||
|
restartStatePath := filepath.Join(dir, "restart-state.json")
|
||||||
|
|
||||||
|
result, err := restartWithHooks(context.Background(), fake, RestartRequest{
|
||||||
|
StatePath: restartStatePath,
|
||||||
|
UpdateStatePath: filepath.Join(dir, "update-state.json"),
|
||||||
|
Confirm: true,
|
||||||
|
}, hooks)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("restartWithHooks() error = %v", err)
|
||||||
|
}
|
||||||
|
if result.Version != "0.80.3" || fake.currentImage != "sha256:old" {
|
||||||
|
t.Fatalf("restart result=%+v image=%q; want captured 0.80.3 / sha256:old", result, fake.currentImage)
|
||||||
|
}
|
||||||
|
assertCalled(t, fake.calls, "image tag sha256:old thothii-core:thothctl-")
|
||||||
|
assertCalled(t, fake.calls, "pi-lifecycle-")
|
||||||
|
if matches, globErr := filepath.Glob(filepath.Join(dir, "pi-lifecycle-*.yaml")); globErr != nil || len(matches) != 0 {
|
||||||
|
t.Fatalf("successful restart overrides = %v, error = %v; want safe cleanup", matches, globErr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRestartRefusesActiveSessionsWithoutDrain(t *testing.T) {
|
func TestRestartRefusesActiveSessionsWithoutDrain(t *testing.T) {
|
||||||
dir := t.TempDir()
|
dir := t.TempDir()
|
||||||
fake := newFakeRunner()
|
fake := newFakeRunner()
|
||||||
@@ -193,6 +229,11 @@ func TestRestartPostRecreateFailureKeepsMaintenanceAndRecoveryState(t *testing.T
|
|||||||
if stateErr != nil || !state.MutationStarted {
|
if stateErr != nil || !state.MutationStarted {
|
||||||
t.Fatalf("restart recovery state = %+v, %v; want durable mutation state", state, stateErr)
|
t.Fatalf("restart recovery state = %+v, %v; want durable mutation state", state, stateErr)
|
||||||
}
|
}
|
||||||
|
overridePath := lifecycleOverridePath(statePath, state.Transaction)
|
||||||
|
selected, overrideErr := readLifecycleOverride(overridePath)
|
||||||
|
if overrideErr != nil || selected != state.Previous.Reference || fake.tags[selected] != state.Previous.ID {
|
||||||
|
t.Fatalf("restart override = %q, %v; want retained exact image %q", selected, overrideErr, state.Previous.ID)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRestartMaintenanceClearFailureRestoresRecoveryState(t *testing.T) {
|
func TestRestartMaintenanceClearFailureRestoresRecoveryState(t *testing.T) {
|
||||||
@@ -233,6 +274,16 @@ func TestRecoverLifecycleMaintenanceVerifiesAndClearsRestartState(t *testing.T)
|
|||||||
Previous: previous,
|
Previous: previous,
|
||||||
MutationStarted: true,
|
MutationStarted: true,
|
||||||
})
|
})
|
||||||
|
restartState, err := readState(restartStatePath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
restartOverride := lifecycleOverridePath(restartStatePath, restartState.Transaction)
|
||||||
|
if err := writeLifecycleOverride(restartOverride, restartState.Previous.Reference); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
delete(fake.tags, restartState.Previous.Reference)
|
||||||
|
fake.tags[fake.configuredImage] = "sha256:moved-before-recovery"
|
||||||
candidate := previous
|
candidate := previous
|
||||||
candidate.Reference = "thothii-core:thothctl-recover-candidate"
|
candidate.Reference = "thothii-core:thothctl-recover-candidate"
|
||||||
writeStateForTest(t, updateStatePath, State{
|
writeStateForTest(t, updateStatePath, State{
|
||||||
@@ -254,6 +305,13 @@ func TestRecoverLifecycleMaintenanceVerifiesAndClearsRestartState(t *testing.T)
|
|||||||
if _, err := os.Stat(restartStatePath); !errors.Is(err, os.ErrNotExist) {
|
if _, err := os.Stat(restartStatePath); !errors.Is(err, os.ErrNotExist) {
|
||||||
t.Fatalf("restart recovery state still exists: %v", err)
|
t.Fatalf("restart recovery state still exists: %v", err)
|
||||||
}
|
}
|
||||||
|
if _, err := os.Stat(restartOverride); !errors.Is(err, os.ErrNotExist) {
|
||||||
|
t.Fatalf("restart recovery override still exists: %v", err)
|
||||||
|
}
|
||||||
|
if fake.tags[restartState.Previous.Reference] != restartState.Previous.ID {
|
||||||
|
t.Fatalf("restart recovery pin = %q, want %q", fake.tags[restartState.Previous.Reference], restartState.Previous.ID)
|
||||||
|
}
|
||||||
|
assertCalled(t, fake.calls, restartOverride)
|
||||||
if fake.maintenance {
|
if fake.maintenance {
|
||||||
t.Fatal("maintenance remained active after both lifecycle states were verified")
|
t.Fatal("maintenance remained active after both lifecycle states were verified")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -37,12 +37,13 @@ const (
|
|||||||
|
|
||||||
// Request contains only non-secret operator inputs.
|
// Request contains only non-secret operator inputs.
|
||||||
type Request struct {
|
type Request struct {
|
||||||
StatePath string
|
StatePath string
|
||||||
Version string
|
RestartStatePath string
|
||||||
Source Source
|
Version string
|
||||||
Image string
|
Source Source
|
||||||
Confirm bool
|
Image string
|
||||||
Drain bool
|
Confirm bool
|
||||||
|
Drain bool
|
||||||
}
|
}
|
||||||
|
|
||||||
// Result summarizes the completed, failed, or recovered transaction without command output.
|
// Result summarizes the completed, failed, or recovered transaction without command output.
|
||||||
@@ -72,6 +73,12 @@ func updateWithHooks(ctx context.Context, runner Runner, request Request, hooks
|
|||||||
if request.StatePath == "" {
|
if request.StatePath == "" {
|
||||||
return Result{}, errors.New("update state path is required")
|
return Result{}, errors.New("update state path is required")
|
||||||
}
|
}
|
||||||
|
if request.RestartStatePath == "" {
|
||||||
|
request.RestartStatePath = pairedRestartStatePath(request.StatePath)
|
||||||
|
}
|
||||||
|
if err := validateRestartStatePaths(request.RestartStatePath, request.StatePath); err != nil {
|
||||||
|
return Result{StatePath: request.StatePath}, err
|
||||||
|
}
|
||||||
lock, err := acquireLock(request.StatePath)
|
lock, err := acquireLock(request.StatePath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return Result{StatePath: request.StatePath}, err
|
return Result{StatePath: request.StatePath}, err
|
||||||
@@ -96,6 +103,9 @@ func updateWithHooks(ctx context.Context, runner Runner, request Request, hooks
|
|||||||
}
|
}
|
||||||
request.Image = canonical
|
request.Image = canonical
|
||||||
}
|
}
|
||||||
|
if err := prepareLifecycleMutation(request.RestartStatePath, hooks.removeFile); err != nil {
|
||||||
|
return Result{StatePath: request.StatePath}, err
|
||||||
|
}
|
||||||
if old, err := readState(request.StatePath); err == nil && stateNeedsRecovery(old) {
|
if old, err := readState(request.StatePath); err == nil && stateNeedsRecovery(old) {
|
||||||
return Result{StatePath: request.StatePath}, ErrInterruptedUpdate
|
return Result{StatePath: request.StatePath}, ErrInterruptedUpdate
|
||||||
} else if err != nil && !errors.Is(err, os.ErrNotExist) {
|
} else if err != nil && !errors.Is(err, os.ErrNotExist) {
|
||||||
@@ -254,11 +264,17 @@ func failPreparation(statePath, overridePath string, state State, cause error, h
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Rollback restores the image recorded in durable update state. It is safe for interrupted runs.
|
// Rollback restores the image recorded in durable update state. It is safe for interrupted runs.
|
||||||
func Rollback(ctx context.Context, runner Runner, statePath string, confirm bool) (result Result, retErr error) {
|
func Rollback(ctx context.Context, runner Runner, statePath, restartStatePath string, confirm bool) (result Result, retErr error) {
|
||||||
return rollbackWithHooks(ctx, runner, statePath, confirm, defaultLifecycleHooks)
|
return rollbackWithHooks(ctx, runner, statePath, restartStatePath, confirm, defaultLifecycleHooks)
|
||||||
}
|
}
|
||||||
|
|
||||||
func rollbackWithHooks(ctx context.Context, runner Runner, statePath string, confirm bool, hooks lifecycleHooks) (result Result, retErr error) {
|
func rollbackWithHooks(ctx context.Context, runner Runner, statePath, restartStatePath string, confirm bool, hooks lifecycleHooks) (result Result, retErr error) {
|
||||||
|
if restartStatePath == "" {
|
||||||
|
restartStatePath = pairedRestartStatePath(statePath)
|
||||||
|
}
|
||||||
|
if err := validateRestartStatePaths(restartStatePath, statePath); err != nil {
|
||||||
|
return Result{StatePath: statePath}, err
|
||||||
|
}
|
||||||
lock, err := acquireLock(statePath)
|
lock, err := acquireLock(statePath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return Result{StatePath: statePath}, err
|
return Result{StatePath: statePath}, err
|
||||||
@@ -267,6 +283,9 @@ func rollbackWithHooks(ctx context.Context, runner Runner, statePath string, con
|
|||||||
if !confirm {
|
if !confirm {
|
||||||
return Result{StatePath: statePath}, ErrConfirmationRequired
|
return Result{StatePath: statePath}, ErrConfirmationRequired
|
||||||
}
|
}
|
||||||
|
if err := prepareLifecycleMutation(restartStatePath, hooks.removeFile); err != nil {
|
||||||
|
return Result{StatePath: statePath}, err
|
||||||
|
}
|
||||||
maintenanceErr := ensureMaintenance(ctx, runner)
|
maintenanceErr := ensureMaintenance(ctx, runner)
|
||||||
clearMaintenance := maintenanceErr == nil
|
clearMaintenance := maintenanceErr == nil
|
||||||
defer func() {
|
defer func() {
|
||||||
|
|||||||
@@ -114,7 +114,7 @@ func TestSuccessfulUpdateAndRollbackRemainSelectedOnFreshRecreate(t *testing.T)
|
|||||||
if fake.currentImage != "sha256:candidate" {
|
if fake.currentImage != "sha256:candidate" {
|
||||||
t.Fatalf("fresh recreate image = %q, want verified candidate", fake.currentImage)
|
t.Fatalf("fresh recreate image = %q, want verified candidate", fake.currentImage)
|
||||||
}
|
}
|
||||||
if _, err := Rollback(context.Background(), fake, statePath, true); err != nil {
|
if _, err := Rollback(context.Background(), fake, statePath, pairedRestartStatePath(statePath), true); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
fake.currentImage = "sha256:candidate"
|
fake.currentImage = "sha256:candidate"
|
||||||
@@ -209,7 +209,7 @@ func TestDigestPinnedConfiguredImageIsNeverUsedAsARollbackTagTarget(t *testing.T
|
|||||||
if _, err := Update(context.Background(), fake, Request{StatePath: statePath, Version: "0.81.0", Source: BuildSource, Confirm: true}); err != nil {
|
if _, err := Update(context.Background(), fake, Request{StatePath: statePath, Version: "0.81.0", Source: BuildSource, Confirm: true}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if _, err := Rollback(context.Background(), fake, statePath, true); err != nil {
|
if _, err := Rollback(context.Background(), fake, statePath, pairedRestartStatePath(statePath), true); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
assertNotCalled(t, fake.calls, "image tag sha256:old "+fake.configuredImage)
|
assertNotCalled(t, fake.calls, "image tag sha256:old "+fake.configuredImage)
|
||||||
@@ -306,7 +306,7 @@ func TestSuccessfulLifecycleCommandsPreserveTypedRecoveryErrorFromMaintenanceCle
|
|||||||
Confirm: true,
|
Confirm: true,
|
||||||
})
|
})
|
||||||
} else {
|
} else {
|
||||||
result, err = Rollback(context.Background(), fake, statePath, true)
|
result, err = Rollback(context.Background(), fake, statePath, pairedRestartStatePath(statePath), true)
|
||||||
}
|
}
|
||||||
|
|
||||||
if !fake.maintenance {
|
if !fake.maintenance {
|
||||||
@@ -382,7 +382,7 @@ func TestManualRollbackSurvivesADeadCandidateCore(t *testing.T) {
|
|||||||
fake.coreRunning = false
|
fake.coreRunning = false
|
||||||
fake.maintenance = false
|
fake.maintenance = false
|
||||||
|
|
||||||
result, err := Rollback(context.Background(), fake, statePath, true)
|
result, err := Rollback(context.Background(), fake, statePath, pairedRestartStatePath(statePath), true)
|
||||||
|
|
||||||
if err != nil || result.Phase != PhaseRolledBack {
|
if err != nil || result.Phase != PhaseRolledBack {
|
||||||
t.Fatalf("Rollback() = %+v, %v; want restored previous core", result, err)
|
t.Fatalf("Rollback() = %+v, %v; want restored previous core", result, err)
|
||||||
@@ -657,7 +657,7 @@ func TestRollbackFinalStateWriteFailureKeepsMaintenanceAndOverrideForRecovery(t
|
|||||||
hooks := defaultLifecycleHooks
|
hooks := defaultLifecycleHooks
|
||||||
hooks.writeState = func(string, State) error { return errors.New("injected rollback state write failure") }
|
hooks.writeState = func(string, State) error { return errors.New("injected rollback state write failure") }
|
||||||
|
|
||||||
result, err := rollbackWithHooks(context.Background(), fake, statePath, true, hooks)
|
result, err := rollbackWithHooks(context.Background(), fake, statePath, pairedRestartStatePath(statePath), true, hooks)
|
||||||
if err == nil || result.Phase != PhaseFailed {
|
if err == nil || result.Phase != PhaseFailed {
|
||||||
t.Fatalf("rollbackWithHooks() = %+v, %v; want failed durable finalization", result, err)
|
t.Fatalf("rollbackWithHooks() = %+v, %v; want failed durable finalization", result, err)
|
||||||
}
|
}
|
||||||
@@ -783,7 +783,7 @@ func TestRollbackRestoresInterruptedOrPreviouslyRecordedState(t *testing.T) {
|
|||||||
previous.Reference = "thothii-core:thothctl-test-previous"
|
previous.Reference = "thothii-core:thothctl-test-previous"
|
||||||
fake.tags[previous.Reference] = previous.ID
|
fake.tags[previous.Reference] = previous.ID
|
||||||
writeStateForTest(t, statePath, State{Version: 1, Phase: PhaseRecreated, Previous: previous})
|
writeStateForTest(t, statePath, State{Version: 1, Phase: PhaseRecreated, Previous: previous})
|
||||||
result, err := Rollback(context.Background(), fake, statePath, true)
|
result, err := Rollback(context.Background(), fake, statePath, pairedRestartStatePath(statePath), true)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Rollback() error = %v", err)
|
t.Fatalf("Rollback() error = %v", err)
|
||||||
}
|
}
|
||||||
@@ -805,6 +805,100 @@ func TestUpdateRefusesToOverwriteInterruptedRecoveryState(t *testing.T) {
|
|||||||
assertNotCalled(t, fake.calls, "compose")
|
assertNotCalled(t, fake.calls, "compose")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestFailedRestartBlocksUpdateAndRollbackAndMalformedStateAlsoRejects(t *testing.T) {
|
||||||
|
for _, operation := range []string{"update", "rollback"} {
|
||||||
|
for _, restartState := range []string{"failed", "malformed"} {
|
||||||
|
t.Run(operation+"_"+restartState, func(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
fake := newFakeRunner()
|
||||||
|
updateStatePath := filepath.Join(dir, "update-state.json")
|
||||||
|
restartStatePath := filepath.Join(dir, "restart-state.json")
|
||||||
|
if restartState == "failed" {
|
||||||
|
fake.fail = "health"
|
||||||
|
_, restartErr := Restart(context.Background(), fake, RestartRequest{
|
||||||
|
StatePath: restartStatePath,
|
||||||
|
UpdateStatePath: updateStatePath,
|
||||||
|
Confirm: true,
|
||||||
|
})
|
||||||
|
var recovery *RecoveryRequiredError
|
||||||
|
if !errors.As(restartErr, &recovery) {
|
||||||
|
t.Fatalf("Restart() error = %v, want failed restart recovery state", restartErr)
|
||||||
|
}
|
||||||
|
fake.fail = ""
|
||||||
|
} else if err := os.WriteFile(restartStatePath, []byte("{malformed"), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if operation == "rollback" {
|
||||||
|
previous := stateImageForTest(t, fake)
|
||||||
|
writeStateForTest(t, updateStatePath, State{
|
||||||
|
Transaction: "rollback-target",
|
||||||
|
Phase: PhaseRecreated,
|
||||||
|
Target: Target{Version: fake.version, Source: string(BuildSource)},
|
||||||
|
Previous: previous,
|
||||||
|
MutationStarted: true,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
fake.calls = nil
|
||||||
|
|
||||||
|
var err error
|
||||||
|
if operation == "update" {
|
||||||
|
_, err = Update(context.Background(), fake, Request{
|
||||||
|
StatePath: updateStatePath,
|
||||||
|
RestartStatePath: restartStatePath,
|
||||||
|
Version: "0.81.0",
|
||||||
|
Source: BuildSource,
|
||||||
|
Confirm: true,
|
||||||
|
})
|
||||||
|
} else {
|
||||||
|
_, err = Rollback(context.Background(), fake, updateStatePath, restartStatePath, true)
|
||||||
|
}
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("%s accepted %s restart recovery state", operation, restartState)
|
||||||
|
}
|
||||||
|
if restartState == "failed" && !errors.Is(err, ErrInterruptedRestart) {
|
||||||
|
t.Fatalf("%s error = %v, want ErrInterruptedRestart", operation, err)
|
||||||
|
}
|
||||||
|
assertNotCalled(t, fake.calls, "compose")
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUpdateCleansVerifiedRestartStateBeforeNormalLifecycleWork(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
fake := newFakeRunner()
|
||||||
|
updateStatePath := filepath.Join(dir, "update-state.json")
|
||||||
|
restartStatePath := filepath.Join(dir, "restart-state.json")
|
||||||
|
state := State{
|
||||||
|
Transaction: "verified-restart",
|
||||||
|
Phase: PhaseVerified,
|
||||||
|
Target: Target{Version: fake.version, Source: "restart"},
|
||||||
|
Previous: stateImageForTest(t, fake),
|
||||||
|
MutationStarted: true,
|
||||||
|
}
|
||||||
|
writeStateForTest(t, restartStatePath, state)
|
||||||
|
overridePath := lifecycleOverridePath(restartStatePath, state.Transaction)
|
||||||
|
if err := writeLifecycleOverride(overridePath, state.Previous.Reference); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
result, err := Update(context.Background(), fake, Request{
|
||||||
|
StatePath: updateStatePath,
|
||||||
|
RestartStatePath: restartStatePath,
|
||||||
|
Version: fake.version,
|
||||||
|
Source: BuildSource,
|
||||||
|
Confirm: true,
|
||||||
|
})
|
||||||
|
if err != nil || result.Phase != PhaseNoop {
|
||||||
|
t.Fatalf("Update() = %+v, %v; want normal no-op after verified restart", result, err)
|
||||||
|
}
|
||||||
|
for _, path := range []string{restartStatePath, overridePath} {
|
||||||
|
if _, err := os.Stat(path); !errors.Is(err, os.ErrNotExist) {
|
||||||
|
t.Fatalf("terminal restart artifact %s still exists: %v", path, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRunningImageCapturesServerBindAndNamedMountIdentity(t *testing.T) {
|
func TestRunningImageCapturesServerBindAndNamedMountIdentity(t *testing.T) {
|
||||||
fake := newFakeRunner()
|
fake := newFakeRunner()
|
||||||
fake.mountsJSON = `[{"Type":"bind","Source":"/srv/thothii/data","Destination":"/data","RW":true},{"Type":"bind","Source":"/srv/thothii/pi","Destination":"/home/thoth/.pi","RW":true},{"Type":"volume","Name":"sessions","Source":"/var/lib/docker/volumes/sessions/_data","Destination":"/data/sessions","RW":true}]`
|
fake.mountsJSON = `[{"Type":"bind","Source":"/srv/thothii/data","Destination":"/data","RW":true},{"Type":"bind","Source":"/srv/thothii/pi","Destination":"/home/thoth/.pi","RW":true},{"Type":"volume","Name":"sessions","Source":"/var/lib/docker/volumes/sessions/_data","Destination":"/data/sessions","RW":true}]`
|
||||||
|
|||||||
Reference in New Issue
Block a user