Upgrading from 3.0 to 3.1

Move a 3.0 installation to 3.1: get 3.1, check, stop, back up and run upgrade apply once, then start. Buffered messages come through the upgrade.

From

3.0.0

Installs

Docker Compose · docker run · Single binary · Helm

Upgrade step

upgrade apply, once

3.1 is a minor release. It is not a swap of the image tag: 3.1 stores some of your data differently from 3.0, so your data is migrated once, by a command you run. MaestroHub never migrates data on its own. If you start 3.1 without the command, it refuses to start and tells you why (see If you skip the step).

What the command changes in 3.0 data:

• Store and forward: the buffers' records get a version, and buffers that kept the old 24-hour default retention move to keep until delivered (see What changes in 3.1).
• Access: role grants are moved to their 3.1 form.
• Database: the new tables and columns of 3.1 are added.

upgrade check lists each of these for your own data before anything is changed.

The steps

Step	What you do	MaestroHub is
1	Get 3.1	Running on 3.0
2	upgrade check	Running on 3.0
3	Stop MaestroHub	Stopped
4	Back up, then upgrade apply	Stopped
5	Start 3.1	Running on 3.1

MaestroHub is stopped from step 3 to step 5. On a test installation with its stores in PostgreSQL, that took about a minute.

Is this guide for you?

You run	What to do
3.0.0	Follow this guide.
2.6.x	Follow Upgrade from 2.6.x to 3.0 with the 3.1 image or archive. It runs the 3.1 upgrade in the same upgrade apply. Read What changes in 3.1 too.
No existing data	Install 3.1. There is nothing to upgrade.

Which version do you run? Ask MaestroHub, without signing in:

curl http://localhost:8080/api/v1/version

It prints, for example, {"version":"3.0.0",…}.

Before you start

1. Read What changes in 3.1. Most of it is store and forward.
2. Back up your PostgreSQL databases, if MaestroHub keeps stores in PostgreSQL (the single-node bundle does, and so does Helm). upgrade apply backs up MaestroHub's own file stores, not your databases. It asks for the name of your snapshot and refuses without one.
3. Plan a short stop. While MaestroHub is stopped it reads nothing from your devices and sends nothing. Values your devices produce in that time are not buffered. Messages that were already buffered stay buffered (see below).

An outage in progress is fine. If a destination is down and MaestroHub is holding a backlog for it, upgrade anyway. In our customer-twin test, a Kafka link was cut on 3.0.0, the backlog built up, MaestroHub was upgraded to 3.1 during the cut, and 3.1 started with 7,598 messages still held. When the link came back, every one of them reached Kafka. The upgrade lost nothing and duplicated nothing.

How to run an upgrade command

Every upgrade command is the same on every installation. Only the part in front of it differs.

Docker Compose

docker compose run --rm maestrohub context-platform upgrade check --config /config/config.yaml

This is for the 3.0 bundles, where the service is named maestrohub and its command: starts with context-platform --config /config/config.yaml. If yours differs, use your service name and your own command: line. Run it from the folder that holds your Compose file.

docker run

docker run --rm -it -v maestrohub-data:/data maestrohub/context-platform:3.1.0 upgrade check

Use the volume name you gave 3.0. If you start MaestroHub with a config file, mount it and pass --config in every command.

Single binary

./context-platform upgrade check

Run it from the folder where you unpacked 3.1. On Windows, use context-platform.exe. If you start MaestroHub with --config <file>, pass the same flag to every command.

Helm

On the single-node chart the commands run in a pod that has the app's image, config, secrets and data volume. On the Enterprise chart helm upgrade runs them for you. See Helm below.

Step 1. Get 3.1

Docker Compose

Keep a copy of your Compose file and .env, then set the image tag to 3.1.0 (in image: or in your .env) and pull it:

docker compose pull maestrohub

Do not run docker compose up yet. 3.0 keeps running until step 3.

docker run

docker pull maestrohub/context-platform:3.1.0

Single binary

Unpack the 3.1 archive into a new folder, and keep the 3.0 folder. Both use the same data folder. If you changed the 3.0 config.yaml, make the same changes in 3.1's.

Helm

See Helm.

Step 2. Check

Run upgrade check with the prefix for your installation (above). It changes nothing and is safe while 3.0 runs. It works on copies of your stores and reports what apply would do.

Exit code 2 with work listed is what you expect from 3.0 data. If it lists an item that needs a person, read it now: apply asks you about it.

Step 3. Stop MaestroHub

Docker Compose

docker compose stop maestrohub

docker run

docker stop <container name>

Single binary

Stop the running context-platform (Ctrl+C in its window, or stop its service).

Helm

See Helm.

Step 4. Back up and apply

Take your PostgreSQL snapshot first, if you keep stores there. Then run apply. It asks what it needs to know, backs up every file store it changes, migrates your data and verifies the result. If a step fails, it puts every store back as it was.

Docker Compose

docker compose run --rm maestrohub context-platform upgrade apply --config /config/config.yaml

With stores in PostgreSQL, add --db-snapshot-taken <your snapshot name>.

docker run

docker run --rm -it -v maestrohub-data:/data maestrohub/context-platform:3.1.0 upgrade apply

Single binary

./context-platform upgrade apply

Helm

See Helm.

It ends with Upgraded and verified. In a script, give --force --data-not-in-use so it does not ask, and --db-snapshot-taken <name> for PostgreSQL. The flags are in the command reference.

Step 5. Start 3.1

Docker Compose

docker compose up -d maestrohub
docker compose ps maestrohub

The container shows healthy once 3.1 is ready.

docker run

Start the container as you started 3.0, with the 3.1.0 image.

Single binary

Start context-platform from the 3.1 folder.

Helm

See Helm.

Buffers that held messages before the upgrade show them on Store & Forward, and deliver them once their destination answers.

Helm

Single-node chart. Every helm upgrade (and every Argo CD sync) first runs the <release>-upgrade-gate Job: upgrade check in the new image, read-only. On 3.0 data it fails the release with exit code 2, so the running 3.0 pod keeps serving instead of being replaced by one that cannot start. Read its report:

kubectl -n <namespace> logs job/<release>-upgrade-gate

Then:

1. Take a snapshot of the PostgreSQL databases.
2. Scale the app to 0 (with Argo CD, turn auto-sync off first).
3. Run context-platform upgrade apply --config /config/config.yaml --force --data-not-in-use --db-snapshot-taken <name> once, in a pod with the app's image, config, secret and data volume.
4. Upgrade (or sync) again. The gate passes and 3.1 starts.

The chart's README.md (section Upgrade gate) has the commands for each step.

--reuse-values works. A 3.0.0 release upgraded with helm upgrade --reuse-values is no longer refused for engine.durability.storeLimitBytes. The chart now derives the store-and-forward limit from nats.config.jetstream.fileStore.maxSize. A value carried over from 3.0 that differs from it is refused, with the fix: --set engine.durability.storeLimitBytes=0. Keep the value only with an external NATS.

Enterprise chart (one deployment per service). helm upgrade runs the data upgrade for you, in the <release>-upgrade Job, before any 3.1 service starts. You take the database snapshot and give its name; the Job does steps 2 to 5.

1. Run the upgrade as you always do. Give it time, since Helm waits for the Job:

   helm upgrade <release> <3.1 chart> -n <namespace> -f <your values> --timeout 30m

   The Job runs upgrade check against every MaestroHub database. On 3.0 data it stops the upgrade with nothing deployed, and 3.0 keeps running. Its log lists what 3.1 converts in your data, and how to take the snapshot:

   kubectl -n <app namespace> logs job/<release>-upgrade --all-containers

2. Take a snapshot of the MaestroHub databases on your PostgreSQL server with your own tooling: a CloudNativePG backup, a volume snapshot, or pg_dump of each database. With backup.postgres.enabled, the chart's backup job takes one now. The log prints the command with your names; for a release named maestrohub it is:

   kubectl -n maestrohub create job --from=cronjob/maestrohub-backup-postgres pre-3-1

3. Run the same upgrade again with the snapshot's name:

   helm upgrade <release> <3.1 chart> -n <namespace> -f <your values> --timeout 30m \
     --set upgrade.snapshotTaken=<snapshot name>

   The Job stops every MaestroHub service of the release (the databases keep running), runs upgrade apply, and prints what it converted. Then Helm starts the 3.1 services. MaestroHub is stopped from the start of upgrade apply until the 3.1 services are ready. On a test installation of nine services, the whole helm upgrade took under 30 seconds.

Running the upgrade again later is safe: with nothing left to convert, the Job stops nothing and the upgrade goes on. You can leave upgrade.snapshotTaken in your values.

• The image. The Job runs one image, maestrohub/context-platform at the chart's version. It carries the upgrade command, and it also stops the services, through the Kubernetes API with the Job's own service account. If your chart's version is not the version of your services, set upgrade.image.tag to the 3.1 version.
• Documents that need a person. If the log says a document needs a person, read its finding, then add --set upgrade.accept={<id>,<id>}.
• If upgrade apply fails, the log says what was changed and what was put back. MaestroHub stays stopped. helm rollback <release> starts 3.0 again. If the report says a database was left changed, restore your snapshot first.
• Argo CD runs the Job as a PreSync hook. The sync fails on 3.0 data until you set upgrade.snapshotTaken in the Application's values.

The chart's README.md (values upgrade.*) describes each setting.

Air-gapped installs. The upgrade Job needs one image besides the 3.1 service images you already mirror: MaestroHub's own maestrohub/context-platform, at the version you install. Nothing else is pulled.

1. Copy it into your registry, from a machine that reaches Docker Hub and your registry:

   docker pull maestrohub/context-platform:3.1.0
   docker tag maestrohub/context-platform:3.1.0 registry.example.com/maestrohub/context-platform:3.1.0
   docker push registry.example.com/maestrohub/context-platform:3.1.0

2. Point the Job at it in your values. If your registry needs credentials, the Job uses the same pull secret as the rest of the chart, global.imagePullSecrets:

   upgrade:
     image:
       repository: registry.example.com/maestrohub/context-platform
       tag: "3.1.0"
       pullPolicy: IfNotPresent
   global:
     imagePullSecrets:
       - name: my-registry-credentials

3. Run the steps above. To see which image the Job will run before you upgrade, render it:

   helm template <release> <3.1 chart> -f <your values> --show-only templates/job-upgrade.yaml | grep image:

If you skip the step

If you start 3.1 on 3.0 data without upgrade apply, nothing is changed and nothing runs:

• MaestroHub prints what it found and the commands to run, and serves a Data migration needed page on its address.
• Docker shows the container unhealthy. docker inspect shows why: the health check prints MaestroHub did not start and the command to run.
• Kubernetes keeps the pod running and not ready (/ready answers 503 with the reason).
• Helm refuses the release at the upgrade gate, or on the Enterprise chart at the upgrade Job (see above).

Stop it, then go on from step 3. See also 3.0 was started before the upgrade.

If you want to go back

Until 3.1 has started on your data, upgrade restore puts back the backup that apply took. Restore your PostgreSQL snapshot as well, then start 3.0 again. See Upgrade problems.