Going Back and Solving Problems

What you see	Go to
3.0 shows an Upgrade needed page instead of the app	3.0 was started before the upgrade
apply says the stores may be in use, or names a flag to add	Apply options
The report lists items under Needs a person	Items that need a person
apply was cut off part way (dropped SSH connection, restart, kill)	If apply was cut off
You want to return to 2.6	Go back to 2.6
You want to return to 2.6, and 3.0 has already started	Go back after 3.0 has started
You want to know what the backup holds, or delete it	Backup sets
Warnings in the log after the upgrade	Warnings after the upgrade

3.0 was started before the upgrade

3.0 never starts on data it was not made for. Started on 2.6 data, on data a newer version wrote, or on stores that are not where it keeps them, it opens nothing for writing, runs no pipeline and opens no connection. It prints what it found and the commands to run, logs one warning with the page's address, and then serves a read-only Upgrade needed page on its HTTP address until you stop it:

• The page says what was found — data an earlier version wrote that needs to be migrated, data a newer version wrote, or data that needs attention (stores in another database engine, or in two places) — that nothing was changed and nothing runs, and links to the upgrade guides. It shows the installed version. It shows no store names, paths, sizes or contents, because nobody has signed in: the commands are in this guide, and in what the server printed. After an interrupted upgrade it shows, for the administrator, the one command that restores the backup the upgrade took, in the form of the way MaestroHub runs (Docker Compose in the image, the binary, or none on Kubernetes).
• Health checks: /health answers 200 (the process is alive, so an orchestrator does not restart it in a loop), and /ready answers 503 with the state (so a load balancer sends it no traffic). On Kubernetes the pod stays running and not ready; reach the page with kubectl port-forward.
• The API answers 503 with the state and the upgrade check command.
• Stopping it (docker compose stop, docker stop, systemctl stop, Ctrl+C) ends it at once. Then run the upgrade: see Upgrade from 2.6.x to 3.0.
• File owners in the Docker image. Before MaestroHub starts, the image's start script gives every entry directly in /data that the app user (uid 1000) does not own to that user, and prints one line for each (maestrohub entrypoint: giving /data/… to maestrohub). No file's contents change. upgrade check never does this.

To exit instead, as 3.0 did before this page existed, set upgrade.page.enabled: false in config.yaml or MAESTROHUB_UPGRADE_PAGE_ENABLED=false. The refused start then prints the commands and exits with 1.

Backup sets

upgrade apply backs up every file-based store it changes before it changes anything. It writes each backup set to one folder, upgrade-backups in your data folder (<storage.dataDir>/upgrade-backups). In the Docker image that is /data/data/upgrade-backups, on your volume. You can choose another folder with --backup-dir. apply checks that the folder has enough free space before it starts. upgrade status lists every backup set it finds there.

Not in the backup set. Back these up yourself, with MaestroHub stopped:

• An external TimescaleDB historian. Its time-series values are never in the upgrade's backup set. Take a pg_dump of the database or a snapshot of its disk.
• Any MaestroHub store kept in PostgreSQL. If the upgrade changes a store in PostgreSQL, apply requires --db-snapshot-taken <your-snapshot-id>. This is your confirmation that you took a snapshot. apply records the ID in the backup set's manifest.json, so you know which snapshot goes with this upgrade.
• History. The backup set holds your settings, not your time-series history.

Protect or delete the set. It holds your data as it was before the upgrade, including the secrets that 2.6 stored in plain text (see Secrets are now encrypted). upgrade status marks each set that holds such secrets and prints the command that deletes it (rm -rf <set>). It also says, for each set, whether it can still be restored (with the restore command), whether its stores already hold what it holds, whether it can no longer be restored and why, or why restore would refuse it now.

In Docker the set is on your volume, so delete it through a container, for example:

docker run --rm -v maestrohub-data:/data --entrypoint rm maestrohub/context-engine:<3.0 version> -rf <set>

A deleted set cannot be restored, so delete it once 3.0 runs well.

Go back to 2.6

You can go back to 2.6 only before 3.0 has started on the upgraded data.

1. Keep MaestroHub stopped.

2. Run the restore command that apply printed, with the 3.0 binary or image and the same config. With Docker Compose:

   docker compose run --rm maestrohub maestrohub-lite upgrade restore --backup <backup folder> --config /config/config.yaml

   With docker run:

   docker run --rm -it -v maestrohub-data:/data maestrohub/context-engine:<3.0 version> upgrade restore --backup <backup folder>

   On the single binary, run ./context-engine upgrade restore --backup <backup folder> from the 3.0 folder.

3. If a store is in PostgreSQL, restore your snapshot of that database with the tool you used to take it. The restore report names it.

4. Start 2.6 again: put back the copy of your Compose file and .env from step 1 of the upgrade (or set the tag back to your 2.6 version), or start the 2.6 binary. If you start the 3.0 image instead, it shows the Upgrade needed page.

Every store is then as it was before apply. Three things stay:

• The backup set. It can be restored again, or deleted (see Backup sets).
• Encryption key files that the upgrade created, in data/secrets. 2.6 ignores them, and 3.0 needs them if you upgrade again. Do not delete them.
• The run's report, in <storage.dataDir>/upgrade-reports/<run id>.json (in the Docker image, /data/data/upgrade-reports). 3.0 keeps it so that System → Upgrades can show the run after the backup set is deleted. 2.6 ignores it.

Once 3.0 has started, restore refuses. It checks every store in the backup set before it puts any back, so a refused restore reports nothing was changed and names each store it refuses, for example the stores that changed after the upgrade. From then on, going back means restoring your own backups.

If writing a store fails part way through a restore, restore stops and names the stores it put back and the ones it did not. The backup set is unchanged: fix the cause and run the same command again, and do not start MaestroHub until it succeeds.

Go back after 3.0 has started

After 3.0 has started on the upgraded data, restore refuses. You go back from the copy of MaestroHub's data you took in step 3 of the upgrade (or from your disk snapshot). This discards everything 3.0 has done since: its configuration changes and its run history.

1. Stop MaestroHub.

2. Put the copy back. This deletes everything in the data volume first, then unpacks the copy:

   With Docker Compose:

   docker compose run --rm --no-deps -T --entrypoint sh maestrohub -c 'find /data -mindepth 1 -delete && tar xzf - -C /data' < maestrohub-data-2.6.tgz

   With docker run:

   docker run --rm -i --entrypoint sh -v maestrohub-data:/data maestrohub/context-engine:<3.0 version> -c 'find /data -mindepth 1 -delete && tar xzf - -C /data' < maestrohub-data-2.6.tgz

   On the single binary, replace the data folder (~/maestrohub/data) with your copy.

3. Restore your external databases from the snapshots you took before the upgrade: 3.0 has written to them too.

4. Start 2.6, as in step 4 of Go back to 2.6.

Warnings after the upgrade

These warnings are expected in the log of the first starts of 3.0. Each says what it means; none stops MaestroHub.

Warning	What it means
config: … is no longer read …	A setting from your 2.6 config.yaml that 3.0 ignores. Remove it when convenient.
SECURITY: at-rest encryption key(s) are publicly known	A stored secret is protected by a key that ships with MaestroHub. Rotate it: upgrade check --show warnings says how.
Parked producer→topic edge: target topic has no graph node yet	Hundreds at the first start are normal while the dependency graph is rebuilt. They resolve on their own.
Zero-output contradiction: run completed with input but dispatched nothing	A pipeline ran and published nothing. Expected for a pipeline that publishes only when a value changes. For a pipeline that should always publish, check it.
Trigger node config does not satisfy its contract	Usually a webhook trigger with no authentication, which upgrade check reported. Open the node and set its headers, or mark it as public.
external broker: backend credential did not resolve to a subject	Matters only if your external MQTT broker hands its authorization decisions to MaestroHub. Otherwise, no action.