What Changes in 3.1

Find which 3.0 behaviours change in 3.1, most of them in store and forward, and what to do before and after upgrading.

Release

3.1.0, a minor release

Upgrade step

upgrade apply, once

Most changes

Store and forward

3.1 needs upgrade apply once on 3.0 data. To run the upgrade, go to Upgrading from 3.0 to 3.1. upgrade check reports the changes below that touch your own data before anything is changed.

Find what affects you

Change	It affects you if	What to do
Store-and-forward for UDP, TCP and retention	A buffer kept the 24-hour default retention, or a pipeline sends over UDP, TCP or syslog with store-and-forward on	Read the notes the report gives. Move writes that must survive an outage from UDP to TCP or TLS.
Outages are waited out, not counted	An output buffers on failure, or uses Kafka	Nothing. Remove workarounds such as a 5 minute Kafka Delivery Timeout.
Changing a buffered output keeps its buffer	You edit outputs that have a buffer	Nothing.
The container's health check is readiness	Something reads the container's health (Docker, Compose, Portainer, a proxy such as Traefik)	Expect unhealthy while MaestroHub refuses to start. Reach the upgrade page on MaestroHub's own port.
admin-cli needs MaestroHub stopped on SQLite	You run admin-cli (for example a break-glass password reset) against SQLite stores	Stop MaestroHub first, then start it again.
Store-and-forward names and permissions	A dashboard reads buffer-failure counts, or a workspace role changed the capacity pool or the emergency pause	Read the new keys. Ask a platform administrator for instance-wide settings.

Store-and-forward for UDP, TCP and retention

Three store-and-forward behaviours change on purpose:

• A buffer keeps its messages until they are delivered. 2.6 and 3.0 stored a 24-hour retention in every buffer by default, and dropped older messages. upgrade apply moves a buffer that holds that old default to keep until delivered: the capacity pool bounds the buffer, not an age. check lists each one as a repair. A buffer with another retention keeps it. To restore an age limit, set a retention on the buffer.
• UDP is never buffered. UDP has no acknowledgement and no sign that the receiver is down, so a buffer cannot tell a delivered datagram from a lost one. UDP Send and UDP Request, and syslog Send Message over UDP (the syslog default), are sent directly whatever the node's store-and-forward setting says. Their promise is at most once: what they send while the receiver is down is lost. 3.0 buffered them. check notes every UDP node set to store-and-forward, and MaestroHub logs, once at each start, every write set to store-and-forward that its connection cannot buffer (store-and-forward is set on writes their connection cannot buffer), naming the pipeline, connection and operation. Use TCP, or syslog over TLS or TCP, where an outage must not lose data.
• A TCP write is sent, not confirmed, never delivered. TCP and syslog over TLS or TCP are still buffered while the receiver can't be reached. Once the network takes a write, MaestroHub counts it sent, not confirmed: the protocol has no acknowledgement, so it is never sent again and never called delivered, and one taken just before the receiver fails may be lost. Outage reports, the Overview and notifications show it as its own number.

See Delivery promises.

Outages are waited out, not counted

• An outage costs no attempts. A buffered message is given up, or an In order buffer blocked, only after a destination answered it with errors while it delivered other messages. An outage of any length is held, and the buffer delivers when the destination is back. On 3.0, a long outage could use up a message's attempts and block an In order buffer ("retries ran out after 10 attempts").
• Kafka's Delivery Timeout runs from the moment a produce starts. On 3.0 it ran from the record's own time, so a buffered record older than the timeout expired before it was sent, again on every retry. The default of 20 seconds is right for outages of any length.
• No message is lost at the start of a cut. 3.0 could fail to buffer the first failed writes of a cut, and counted them under internal_error.
• Reconnecting through a proxy is fast. A destination behind a proxy that accepts and drops connections is retried at the fast pace once it answers again: seconds, not minutes.
• Every Kafka record carries its write key (header maestrohub.write-key). A consumer can recognise a copy that an at-least-once resend wrote (see Delivery promise).

Changing a buffered output keeps its buffer

On 3.0, changing an output's store-and-forward mode, delivery order, freshness or destination retired its buffer. For up to five minutes the output then refused writes that needed buffering, and kept them in Failed messages, where the oldest beyond the cap were dropped. In 3.1 the buffer keeps taking writes. A change of connection removes an empty buffer at once and makes the new one. A buffer that still holds messages delivers them to its old connection first. See Changing an output that has a buffer.

In order keeps the order in which the runs hand values to the output. For values in input order from the source to the destination, set Run order to One at a time on every pipeline they pass through (see Delivery order).

The container's health check is readiness

The image's health check is now context-platform health. It asks /ready and prints the answer. While MaestroHub refuses to start (for example, waiting for upgrade apply), the container shows unhealthy, and docker inspect shows why. On 3.0 it showed healthy. A reverse proxy that routes only to healthy containers, such as Traefik, does not route to the upgrade page: open it on MaestroHub's own port. Kubernetes probes are unchanged.

admin-cli needs MaestroHub stopped on SQLite

While it runs, MaestroHub now keeps each SQLite store to itself, so a stalled or frozen data volume stops only its writes, not the whole server (health, sign-in and reads keep answering). Another program, admin-cli included, cannot open those files meanwhile. With storage.engine: sqlite (the default), stop MaestroHub before an admin-cli command and start it afterwards; docker exec into the running container no longer works for those commands. The CLI says so, with the steps for Docker Compose, Linux and Windows, when a store is held. See Stop MaestroHub before using a SQLite store. Postgres stores are unchanged.

Store-and-forward names and permissions

• Store-and-forward reports its refusals under their own names. GET /engine/durability/buffer-failures counts writes refused at the disk floor under byReason.disk_floor (they were under ingestion_paused, the operator's emergency pause), writes a full Reject new buffer refused under rejected_new (they were under capacity_exceeded, whose writes are all kept), writes to a paused buffer under buffer_paused, and writes to an In order output whose run ended before their turn under ordered_wait_expired (they are now kept in Failed messages; before, they were told to the run only). A dashboard or alert that reads those keys should read the new ones. See When Store & Forward itself can't buffer.
• The store-and-forward controls that act on the whole instance are a platform administrator's: the capacity pool's total, the emergency pause (which now asks for a reason) and every workspace's quota, a workspace's own included (permission durability_settings:update). A workspace role that changed them on 2.6 can no longer; members still read their quota.
• A failed message's payload is shown to those who may read its pipeline. Others with access to Store & Forward see the message, its reason and its error, with a note that the payload is hidden.