Skip to main content
Version: 3.0 (next)

How MaestroHub Keeps Upgrades Safe

We ship new versions often. This page explains what makes that safe for the data you already have: what we promise, how an upgrade works, how we test it, and where the limits are.

To upgrade now, go to Upgrade from 2.6.x to 3.0.

What we promise​

  • Your data changes in one place only: the upgrade command. MaestroHub does not rewrite stored data by itself at startup. Your data is converted only when you run upgrade apply, with MaestroHub stopped.
  • You see every change before it happens. upgrade check shows everything apply would change, and changes nothing.
  • Nothing is removed without being reported. If a new version drops something, upgrade check lists it first.
  • MaestroHub refuses to start on data it does not understand. If the data is older than the new version, or newer, MaestroHub stops and prints the command to run. It does not guess. Until you stop it, it serves a read-only page on its usual address with the same steps (see 3.0 was started before the upgrade).
  • Every upgrade is backed up and recorded. apply takes a backup before it changes anything, into one folder in your data folder, and records what it did. upgrade status lists every backup set.
  • Patch releases need no upgrade step. If a release does not change how data is stored, you replace the binary or image and start it.

How it works​

Every stored item carries a version​

Pipelines, connections, dashboards, topics, access grants and the other things MaestroHub stores each carry a schema version. When a new release changes the shape of one of them, it adds a conversion step from the old version to the new one. MaestroHub runs on one version at a time, so it can always tell old data from new.

Released database migrations never change​

Once a release has shipped, its database migrations are frozen. Each migration file of every release, from 2.6.0 on, is recorded by its content hash. An automated check blocks any change that edits, renames or deletes one of those files. A second check blocks new migrations that change data rather than structure: only the upgrade command changes data.

upgrade check: see what will change​

upgrade check runs the whole upgrade on a copy of your data and prints a report. It changes nothing, so you can run it while the earlier version is still running. It works on copies of the stores that apply would change. It reads the other stores in place, only to estimate the first start's work, and never writes them. While the earlier version runs, that read includes its most recent writes; SQLite then records the reader in the store's -shm file, as it does for every reader. The report tells you:

  • Repairs: changes the upgrade makes on its own, such as a topic path gaining its organization.
  • Needs a person: items the upgrade cannot decide for you. Each one names the item and the field, and proposes the exact change. It shows the value before and after.
  • Notes: behaviour that changes in the new version, which you should know about.
  • At first start: long database work that the first start of the new version will do, with the table's rows, how long it may take and how much disk it needs. This includes stores that an earlier version created without recording their database version: the upgrade reads the tables that are there, not only the version table.
  • Stores: every store MaestroHub keeps, where it is now, where the new version keeps it, and its size.

An item that needs a person stays exactly as it was stored. You accept its proposed change with --accept, or fix it on the earlier version first. The new version does not start until every such item is decided.

upgrade apply: change it, all or nothing​

With MaestroHub stopped, upgrade apply:

  1. backs up every store it is about to change;
  2. runs the database changes that only the upgrade may run;
  3. converts each stored item to the new version;
  4. rebuilds derived data, such as search indexes;
  5. checks that nothing old is left, and records the run.

If any step fails, every store is put back as it was. The run is recorded in an upgrade log inside each store it changed, and in the audit trail when MaestroHub next starts.

upgrade restore: go back​

upgrade restore --backup <dir> puts back the backup set that apply wrote, and moves any store that apply moved back to where the earlier version keeps it. You can then start the earlier version again. Restore works only until the new version has started on the data: after that, the data has moved on and restore refuses.

Restore decides for the whole backup set before it puts any store back. If any store would be refused (it is in use, missing, changed since the upgrade, or its backup is damaged), nothing is changed, and every refused store is named. If writing a store fails part way, restore stops, names the stores it put back and the ones it did not, and leaves the backup set as it was: fix the cause and run the same command again to finish.

upgrade status: see where you are​

upgrade status reports whether every store is at this version's, the record of past upgrades kept in each store, and every backup set in the backup folder: its age, its size, whether it holds secrets that the earlier version stored in plain text, whether it can still be restored (with the restore command if it can), and the command that deletes it. It changes nothing. It opens the stores the way MaestroHub's own start does, so run it with MaestroHub stopped: while MaestroHub runs, it refuses because the stores are in use.

The startup check​

Each time MaestroHub starts, it checks the version of the data before it opens anything. If the data is older, it refuses and names upgrade apply. If the data is newer (for example, after going back to an old binary without restore), it refuses too. It also refuses to start if a store is still where an earlier version kept it, instead of creating a new, empty one.

Secrets are encrypted at rest​

MaestroHub encrypts the secrets it stores, such as connector passwords, identity provider secrets, the SMTP password and the keys that sign tokens. Each module uses its own key. When an upgrade finds a secret that an earlier version stored in plain text, apply encrypts it. The report names each one, but never shows its value.

How we test an upgrade​

Before a release, we upgrade real data and compare behaviour, not just data:

  • The released images. We install the published 2.6.0 and 2.6.5 images, fill them with users, roles, connections, pipelines, dashboards, identity providers and secrets through their own API, and then upgrade them. After the upgrade, every account signs in with its old password, every pipeline loads, and every secret decrypts to its original value. Each user's permissions are compared with what the earlier release granted, and any difference must be explained by the report.
  • A large, long-running installation. We upgrade a copy of a large, long-running production-shaped installation. It uses an external TimescaleDB and MQTT broker, and holds tens of thousands of topics and a multi-year audit history. We then replay the same inputs on both versions and compare every pipeline node, topic, dashboard panel and user's permissions. A difference that no release note or report line explains is a bug that we fix before release.
  • Every rehearsal also checks that a second apply changes nothing, that the new version refuses un-upgraded data, and that restore brings the earlier version back.

What the upgrade does not do​

We want you to know the limits before you rely on it:

  • It does not back up an external database's time-series data. If your historian is an external TimescaleDB, its values are not in the backup set. The same applies to any store kept in PostgreSQL: take your own snapshot first. apply asks you to confirm that you did.
  • It does not remove plain-text secrets from the backup set. The backup holds your data as it was before the upgrade, including secrets that the earlier version stored in plain text. Protect it like the data itself, and delete it once the new version runs well.
  • It does not convert history. Execution records and saved versions of pipelines and dashboards stay as they were written. If you revert a pipeline to a version saved before the upgrade, that version is converted when you revert to it. If it cannot be converted, the revert is refused and you are told why.
  • It does not move data between database engines, with one named exception. If a store is in SQLite and the new configuration puts it in PostgreSQL, or the other way round, the upgrade refuses and tells you. The exception is the 2.6 UNS metadata store kept in PostgreSQL next to SQLite stores: apply copies it into SQLite, reads the PostgreSQL database only, and restore removes the copy.
  • It does not upgrade while MaestroHub runs. Old and new versions cannot share the same data. Plan a short stop for apply, plus the first start. See At first start in the report.
  • It cannot undo after the new version has started. From then on, going back means restoring your own backups.
  • check cannot fully check a store in PostgreSQL. On PostgreSQL, check reads without changing anything, so it can count database changes but not run them. It reports the items it could not check. apply checks them.