non-dpkg information and broken format in manifest

[Scott Moser]

Ubuntu images have been accompanied by a 'manifest' file since at least 10.04.
This manifest file was a list of the dpkg installed packages and their versions.
The format was as output by dpkg-query --show.
That format was
  package-name<tab>version

The offending change was added at
https://bazaar.launchpad.net/~ubuntu-core-dev/livecd-rootfs/trunk/revision/1706

The disco images now contain non-dpkg information in them.
There are a few problems with this:
 a.) the format is now changed. Some lines will now have 3 fields rather than 2.
 b.) content is not strictly a list of dpkg information.

I understand the desire to have pre-seeded snap information in this file
but believe that the correct way to add representation of that information
is with new files rather extending in non-backwards compatible ways an
existing file.

[Steve Langasek (vorlon)]

We have never represented the .manifest to be a public machine-parseable interface. Why should we not expect it to be the responsibility of whatever you have that is parsing these files to adapt?

[Dimitri John Ledkov (xnox)]

It's been extended in a backwards compatible manner.

Is this just an observation, or is something broken?

You correctly state what the format was, what the format has become, but I don't see an actual bug report stated anywhere as to anything being broken by this.

Because yes, the format of the manifest has changed. Intentionally.

Failing to find a bugreport in this bug....

[Scott Moser (smoser)]

@Steve,
"We have never represented the .manifest to be a public machine-parseable interface."

Thats clearly not true. Machine-formatted content (the .manifest) advertised in a machine-formatted index (streams) is quite reasonable to be expected to be machine parseable.

One example of something that parses that information is 'mfdiff'. I do not know for sure, but I suspect it was broken. Anyone wanting to determine if a specific/fixed version of a package would have come to the decision to parse this file.

@Dimitri,
Changing the number of fields in a record of a tab delimited file and *additionally* changing the meaning of an existing field is not backwards compatible.

@Both,
Why not do this right? Add a .manifest.json file. You can determine the format of the file.

[Dimitri John Ledkov (xnox)]

Wait, so you are reporting as the bug report of the manifest, not actually as a result of anything broken?

What is mfdiff? https://packages.ubuntu.com/search?suite=bionic&arch=any&searchon=contents&keywords=mfdiff brings up nothing.

[Dimitri John Ledkov (xnox)]

I mean if you cloned some random junk branch from launchpad, maybe contact the owner of said junk branch....

[Scott Moser (smoser)]

It is a bug to change content in backwards incompatible ways. livecd-rootfs produced an artifact that is indexed with known-content. A change broke consumers of that content. Thats a regression.

Pretending otherwise is wasting both of our time.

I pointed at one known consumer (owned by the same team that produces the content) that will be affected by this backwards incompatible change. There are undoubtedly others. Software *does* exist outside of the Ubuntu archive, and even outside of launchpad. Often times that software builds on top of Ubuntu. We generally consider that a *good* thing. It is not a good thing when Ubuntu breaks that other software and gives a lame excuse.

I'll ask again... Why not just add a new file that has the content you want?
Then you don't break existing users *and* you have the content you want.

[Dimitri John Ledkov (xnox)]

No, that's not how it works. Lots of people are free to do whatever they want and consume whatever they want. Consumers, are only those that we care about. I have now been pointed to at least 4 forks of mfdiff, and well, maxsplit=1 should be enough to fix the linesplits there.

The production code that does use manifests is known to not be broken.

Please stay on topic and please help identify things that are broken with the new manifest, and used in production, if any.

The manifest format change to include snaps will not be reverted, based on current information.

Do you have anything else to share that you are aware of to be broken? either private or public?

(in addition to the already mentioned mfdiff, including all the public, and private copies and forks of thereof)

[Scott Moser (smoser)]

> No, that's not how it works.

I'm not sure I know what you were referring to.

Can you give a single justification for breaking existing consumers of a deliverable?

I do not see justification anywhere. Not in the commit message [1], the merge proposal [2] or any of the comments here. This is the sort of thing I'd hope to see discussion of in a merge proposal.

I've pointed out that a change made is backwards incompatible. I've suggested ways you support including information about snap packages (manifest.json). I consider this on topic.

Is this change intended to be SRU'd ? As a cosmic-forward change this could be acceptable, but even then... Why?

[1] https://code.launchpad.net/~codyshepherd/livecd-rootfs/snaps-manifest/+merge/358530
[2] https://bazaar.launchpad.net/~ubuntu-core-dev/livecd-rootfs/trunk/revision/1714

[Steve Langasek (vorlon)]

On Tue, Nov 13, 2018 at 08:07:58PM -0000, Scott Moser wrote:
> It is a bug to change content in backwards incompatible ways. livecd-
> rootfs produced an artifact that is indexed with known-content. A
> change broke consumers of that content. Thats a regression.

The definition of a manifest has changed. This is a consequence of our
image contents having changed; the .manifest file is a reflection of
reality.

That this no longer matches the expectations of consumers (software or
otherwise) of this file is not a bug in the manifest file. It's an
incompatibility between the reality of what constitutes the content of an
Ubuntu image, and the thing that is parsing the file.

Maintaining format-compatibility with parsers that assume the contents of
the manifest will always be a list of debs + versions would do a
*disservice* to those consumers, by leaving them oblivious to the fact that
the definition has changed.

So no, I don't think livecd-rootfs should provide two manifests and I think
this bug is 'wontfix'.