import of virt-manager broken - pristine tar verify "ambiguous argument"

[Christian Ehrhardt ]

Hi,
as discussed in IRC the last two days now documenting since we seem to need to track this longer.

The command to import virt manager fails at pristine-tar verify checks:
$ git ubuntu import -v -d /tmp/import/virt-manager virt-manager
(Note: I also added "--skip-applied" as reported, but that didn't help either.)

We let that run from snap which uses the newer bundled pristine-tar (my xenial one would have no "verify" at all).

For verification I tried:
- Snap on Xenial
- Snap on Artful
- Git on Artful

All three fail me to import.

[Christian Ehrhardt  (paelzer)]

Example log (this time from git in artful) http://paste.ubuntu.com/25436827/

[Christian Ehrhardt  (paelzer)]

Preserving soem of the IRC history on this:
[14:14] <cpaelzer> rbasak: the import is failing me on the pristine-tar of 1.4.0
[14:14] <cpaelzer> rbasak: do you mind running the same import and checking if that is a local issue to me?
[14:16] <rbasak> Trying
[14:17] <rbasak> 08/30/2017 13:16:54 - ERROR:stderr: pristine-tar: Unknown subcommand "verify"
[14:17] <rbasak> 08/30/2017 13:16:54 - ERROR:stderr: pristine-tar: Unknown subcommand "verify"
[14:17] <rbasak> 08/30/2017 13:16:54 - ERROR:stderr: pristine-tar: Unknown subcommand "verify"
[14:17] <rbasak> Aargh.
[14:17] <rbasak> Anyway, I think I'd hit that already - my pristine-tar is too old now.
[14:17] <rbasak> (on Xenial)
[14:18] <cpaelzer> IIRC nacc meant some things only work on zesty
[14:19] <cpaelzer> yep artfuls pristine tar has the verify command
[14:19] <cpaelzer> but I thought the snap would encapsulate such things
[14:20] <rbasak> I'm not using the snap.
[14:20] <cpaelzer> maybe it only does so with the git ubuntu libs but not external executables
[14:20] <cpaelzer> hmm, maybe I currently don't either
[14:20] <cpaelzer> let me check
[14:20] <rbasak> It's kinda hard to develop with the snap directly :)
[14:20] <cpaelzer> well I do use it and it fails more or less the same
[14:20] <cpaelzer> let me run it in an artful container
[14:21] <cpaelzer> rbasak: well maybe the snap does encapsulate it
[14:21] <cpaelzer> rbasak: I don't have it failing for missing "verify"
[14:21] <cpaelzer> rbasak: the actual verification fails for me
[14:22] <cpaelzer> rbasak: nacc: http://paste.ubuntu.com/25431785/
[14:22] <rbasak> cpaelzer: there is a known pristine-tar bug
[14:22] <rbasak> 08/30/2017 14:12:29 - ERROR:stderr: fatal: ambiguous argument '6c61fcfe2c55a9269c94b6ec2fbeac09759bf80b^{tree}': unknown revision or path not in the working tree.
[14:22] <cpaelzer> yep
[14:23] <rbasak> That seems like a bug in git ubuntu though
[14:23] <rbasak> nacc: ^
[14:23] <cpaelzer> I'll work on the upload tag for now and let nacc take a look later then
[16:56] <nacc> cpaelzer: can you do the import with --no-fetch first
[16:56] <nacc> cpaelzer: i'm 95% sure that it's because of us chaning how imports work
[16:57] <nacc> cpaelzer: although you are indicating a future bug is going to be hit
[16:57] <nacc> cpaelzer: pristine-tar can't be used directly, because of copmonent tarballs
[17:11] <cpaelzer> nacc: the --no-fetch fails at pristine-tar verify still
[17:11] <cpaelzer> nacc: just at another step of it I think
[17:11] * cpaelzer comparing logs
[17:12] <cpaelzer> nacc: http://paste.ubuntu.com/25432445/
[17:12] <cpaelzer> nacc: that is the error now
[17:14] <nacc> cpaelzer: that's using the snap?
[17:15] <cpaelzer> nacc: my case is snap, rbasak'S was from git
[17:15] <nacc> cpaelzer: right, from git is probably basically broken in principle, we now need new gbp versions, new pristine-tar functionality. Hence the snap
[17:15] <nacc> cpaelzer: i'm setting up a fresh VM to test the snap, one moment
[17:15] <rbasak> Yeah mine is known broken for a different reason.
[17:15] <cpaelzer> nacc: I'm ok without you hurrying
[17:16] <nacc> i think there's something more fundamental with the snap
[17:16] <nacc> something wonk...

[Christian Ehrhardt  (paelzer)]

THe tree in USDI is still broken/old.
After the last messages on IRC I retried the import with the latest snap.

Status of today:
1. as-is import is still failing on the ambiguous error on pristine-tar verify
   As suggested this might be due to us changing the structure

2. I can only import with (the new) --no-fetch to "ignore what we already have"

I don't really see how to preserve my old upload rich histories.
I could after the import go in and mess it up exchanging things from my local repos, then remove the whole current repo in usdi and then push mine.
But I'm kind of scared breaking too much.

I also have MPs up, but the rich history of those also isn't important.
So an ack there is good and we don't have to care on preserving the (trivial) change.

I'd say we just (and only) remove the current USDI virt-manager repo, then do a full re-import of it. I can on next merge work in the rich history from my local trees.

TL;DR - please ack to the following plan of action:
1. review and hopefully ack my current virt-manager MPs
2. I upload the SRU of the MPs but we don't care on preserving their rich histroy
3. fully remove the currend virt-manager USDI repo
4. do a fresh virt-manager import and push to USDI repo
5. on next merge I carry in the rich history from local repo's

[Andreas Hasenack (ahasenack)]

I'm getting a very similar error when trying to build the source out of a branch: http://pastebin.ubuntu.com/25444631/

09/01/2017 09:48:55 - ERROR:stderr: gbp:error: Pristine-tar couldn't checkout "samba_4.6.7+dfsg.orig.tar.gz": fatal: ambiguous argument '58fe526e6611a874f408d8ccc4fa22a0f6b754d4^{tree}': unknown revision or path not in the working tree.

But there are new errors after that:
09/01/2017 09:49:26 - ERROR:stderr: dpkg-buildpackage: error: fakeroot not found, either install the fakeroot

and

(...)
  File "/snap/git-ubuntu/209/gitubuntu/build.py", line 560, in fetch_orig_and_build
    if must_build:
NameError: name 'must_build' is not defined

[Nish Aravamudan (nacc)]

@Andreas, your issues with build were resolved in a different bug (LP: #1714539) and I believe we decided on IRC, it works now for you (either with your (possibly) buggy old repository or a fresh clone).

@Christian: the hidden flag is --reimport, which will preserve upload tags (but only upload tags).

I recommend the following modes of operation:

git ubuntu import --no-push --reimport --directory /path/somewhere

once done, go make sure upload tags are preserved. If so:

git ubuntu import --reimport --no-fetch --directory /path/somewhere

This will 'keep' the contents of the import algorithm at /path/somewhere (--no-fetch), run the importer (which should be a no-op) and then delete the old repository and push the new one.

This will go into something like the 'admin' page for `git-ubuntu`. A special section of the import manpage on how to use the importer for our purposes.

[Christian Ehrhardt  (paelzer)]

Hi,
upload tags would be preserved - so I went got the "--reimport --no-fetch".

But on the actual reimport I get:

[...]
09/04/2017 08:53:40 - DEBUG:Executing: git gc
09/04/2017 08:53:42 - WARNING:There are linked MPs to the pkg repository, which will be deleted:
Traceback (most recent call last):
  File "/snap/git-ubuntu/221/usr/lib/python3/dist-packages/lazr/restfulclient/resource.py", line 355, in __getattr__
    return self.lp_get_parameter(attr)
  File "/snap/git-ubuntu/221/usr/lib/python3/dist-packages/lazr/restfulclient/resource.py", line 260, in lp_get_parameter
    raise KeyError("No such parameter: %s" % param_name)
KeyError: 'No such parameter: landing_candidate_collection_link'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/snap/git-ubuntu/221/bin/git-ubuntu", line 22, in <module>
    main()
  File "/snap/git-ubuntu/221/gitubuntu/__main__.py", line 213, in main
    args.func(args)
  File "/snap/git-ubuntu/221/gitubuntu/importer.py", line 1328, in main
    for mp in lp_git_repo.landing_candidate_collection_link:
  File "/snap/git-ubuntu/221/usr/lib/python3/dist-packages/lazr/restfulclient/resource.py", line 734, in __getattr__
    return super(Entry, self).__getattr__(name)
  File "/snap/git-ubuntu/221/usr/lib/python3/dist-packages/lazr/restfulclient/resource.py", line 358, in __getattr__
    % (self, attr))
AttributeError: https://api.launchpad.net/devel/~usd-import-team/ubuntu/+source/virt-manager/+git/virt-manager object has no attribute 'landing_candidate_collection_link'
09/04/2017 08:53:42 - INFO:Leaving /tmp/import/virt-manager as directed

[Christian Ehrhardt  (paelzer)]

I removed my MPs and the import worked then.

[Nish Aravamudan (nacc)]

re: c#6, fixed the typo in the source with the latest push to master.

re: c#7 to be clear, prior, I had already reimported -- no harm/foul from doing it again, just an FYI.