Experimental feature: progressive releases

by Igor Ljubuncic on 20 May 2020

“No plan survives contact with the enemy.” This is a quote famously attributed to the Prussian field marshal Helmuth von Moltke. It is also quite applicable to software development: “No code survives contact with the user.”

In mission-critical environments, staggered deployments of software are a crucial part of controlled updates, designed to ensure maximum stability of production applications and services. This allows developers to monitor and observe the adoption of new versions of their tools, as well as enable operational teams to meet compliance and security targets. Until recently, the timing of automatic snap updates was mostly governed by the client side refresh schedule. Now, there is a new experimental feature that gives snap developers the ability to fine-tune rollouts of new revisions – progressive releases.

Divide & conquer: progressive releases implementation

The idea behind progressive releases is to allow an incremental portion of the total pool of users of a particular snap to receive the update over time. The developer can increase the percentage as they gain confidence that the particular version is working as expected. In particular, this can be quite effective when testing prospective releases for snaps with large install bases.

At the moment, this is an experimental feature, and some functionality may change. To be able to use progressive releases, you can run snapcraft with the relevant environment variable enabled:

SNAPCRAFT_EXPERIMENTAL_PROGRESSIVE_RELEASES=y snapcraft ...

Once the option is enabled, you can then release a revision of your snap.

snapcraft release <snap-name> <revision> <channel,channel,...> --progressive <percentage>

For example:

snapcraft release candycane 13 stable --progressive 30

The command above will release the revision 5 of candycane snap to the stable channel of the default (latest) track, set to 30% deployment. The devices that will be targeted with the new releases are chosen pseudo-randomly based in part on a hash of their device ID.

Some of the concepts and information behind progressive releases may be a little confusing at first. To that end, let’s have a look at an actual example:

$ snapcraft status candycane

Track     Arch    Channel    Version    Revision    Progress
latest    all     stable     0.6        8           73 → 70%
                             0.7        13          21 → 30%
                  candidate  ↑          ↑           -
                  beta       -          -           -
                  edge       -          -           -

What do we have here?

We can see that progressive release of the stable channel has been set to target 30% of the client systems reporting to the store. The second number that is displayed for the stable channel – 21 denotes the actual percentage of the client systems that have already been offered the revision 13. This second value depends on some development work that it’s still in progress, so if you try the experimental feature today, what you will see is:

$ snapcraft status candycane

Track     Arch    Channel    Version    Revision    Progress
latest    all     stable     0.6        8           70%
                             0.7        13          30%
                  candidate  ↑          ↑           -
                  beta       -          -           -
                  edge       -          -           -

The values currently show the percentage of client systems targeted (by the developer) to be refreshed to the progressive revision. For instance, if you set a progressive release to 25%, this means that roughly every fourth client will get the update when it next requests a refresh. Since the typical refresh cycle is six hours, this means that it can take about six or more hours to reach the set target of 25%.

Release pause & revision bug fixes

There’s another scenario where progressive releases can be quite useful. By offering revisions to a smaller pool of client systems, developers will also have more control when it comes to releases that may have issues or require bug fixes.

For instance, you may release revision 13 (r13) to 20% of the channel users, and then discover that there is an issue based on user feedback or other means. You would then stop any further deployment of that particular revision, fix the issue and then release a new revision (r14) to 20% of the channel users. In this case, the 20% of client systems that received the earlier revision r13 will also be the first to get r14. This allows developers to verify that the bug fix in place is effective, after which they can gradually continue the progressive rollout until they reach 100% of the target base.

Summary

Progressive releases is a highly useful, flexible way to roll out software versions in a controlled, gradual manner, allowing developers to closely monitor any potential problems that may arise in the adoption of new releases, especially in high-risk environments. This article is a preview of this capability, which should be rolled out in snapcraft 4.0 very soon. Meanwhile, if you have any questions or ideas around progressive releases (or other topics), please join our forum for a discussion.

Photo by Andrew Ruiz on Unsplash.