How about this:

1. Add transitional packages libzstd1-dev -> libzstd-dev in Artful and Bionic (no Conflicts).

2. Backport what is in Artful, but drop the transitional libzstd1-dev package, rename libzstd-dev to libzstd1-dev and have it Conflicts with libzstd-dev.

This will land after Zesty EOLs, so don't worry about Zesty.

This will result in:

The zstd CLI from the zstd package will be updated, linked against libzstd1 and using new wire protocol.

libzstd.so.1 will become available in Xenial. libzstd.so.0 will remain available.

libzstd-dev will continue to exist and will not change, with the latest version remaining in the release pocket.

libzstd1-dev will be added. Users will only be able to have either libzstd-dev or libzstd1-dev installed at once. The one installed will determine whether the old API or the new API is used, and whether the link will be against the old or new soname.

Users already building using libzstd-dev locally in Xenial will not break and will continue to be able to rebuild (ie. "stay green") without any required intervention. If they wish, they can choose to use libzstd1-dev in Xenial instead. Or they can switch to use a newer release of Ubuntu or build directly against upstream sources, which I think is more likely for this use case anyway. Switching to libzstd1-dev will require code changes due to the API change, but this way this won't be forced upon them.

Users building against libzstd-dev locally who switch to libzstd1-dev will be automatically transitioned to libzstd-dev again upon upgrade to Artful or Bionic. Users who hardcode the libzstd1-dev name in any automated builds will break after upgrading beyond Bionic, since we'll drop the transitional package at that point, so should really do something like "if release == Xenial: install libzstd1-dev; else install libzstd-dev" if they want to use a distribution provided library on Xenial *and* build against the newer wire protocol and library.

If a security or bugfix updates to libzstd.so.0 becomes required, we'll need to adjust the release pocket source package to build only the libzstd0 binary package at that time.

Yann: would you be happy with this?

Rationale:

There is an SRU policy exception "to adjust to changes in the environment, server protocols, web services, and similar". I think, under this exception, we want an SRU to cause the zstd CLI to produce new format output with backwards compatibility, and for libzstd.so.1 to be made available so that other third parties who link against the library we ship can do that also (we'd update other packages in the Ubuntu release too, but we don't have any in this particular case).

> It is clearly labelled experimental, hence does not provide any API stability guarantee.

Separate from an upstream guarantee though, Ubuntu _does_ try to ensure that stable releases do not change behaviour, except when it is deliberate and the project chooses to make an exception. This guarantee is separate from any upstream guarantees. A blanket policy by a distribution is much easier for users to rely upon versus individual upstream guarantees that may not all match. Part of the point of a distribution that does stable releases is to insulate users in this way.

Perhaps an exception is warranted here, but I think it needs justification independent of the decision to update the CLI to produce and consume the newer wire protocol with backwards compatibility, which falls under one of our standing exceptions.