Multiple CI Train issues after the abspath refactoring

[Łukasz Zemczak]

This is a blanket bug for the few issues we saw regarding the latest changes made to CI Train. Currently we have soft-reverted to the previous working revision (i.e. 798).

One of the issues is that watch-ppa doesn't seem to watch the manually uploaded source packages (e.g. see ubuntu/landing-000 for an example, it's using cu2d trunk).

The other issue is that CI Train seems to be loosing state of its bzr branches. It happened 3 times at least already. CI Train fails when doing a bzr push of the bzr branch it prepares during build, mentioning that it does not exist. Checking through cyphermox-test: in the first case I saw, the directory with the bzr branch was completely missing (!), in the second case the directory was there but was not a bzr branch at all (no .bzr/ directory).
Example log: https://ci-train.ubuntu.com/job/ubuntu-landing-009-2-publish/45/console

[Łukasz Zemczak (sil2100)]

So, the 'abspath' refactoring is the current main culprit, but it might not even be related. Anyway, let's treat it as a blanket bug for the issues.

[Robert Bruce Park (robru)]

I highly doubt this is caused by abspath refactoring, because changing relative paths to absolute ones shouldn't result in the .bzr directory disappearing from a branch. It might have resulted in the entire branch appearing in an unexpected location, but that's not what we're seeing.

I was poking around at silo ubuntu/15 and what I saw was:

build job successfully branches, merges, builds, & uploads to PPA. Then some time passes, then the publish job fails because SILO_DIR/ubuntu-keyboard/.bzr is missing despite the fact that SILO_DIR/ubuntu-keyboard contains the full source tree. What could possibly delete that? it makes no sense.

Anyway I'm re-running the build job now and periodically checking on that .bzr directory. It still existed after the merge and after the upload to the PPA. if it still exists after the build job exists, that will rule out the build job.

[Robert Bruce Park (robru)]

Here's the log showing the .bzr directory is still present after the completion of the build job:

https://ci-train.ubuntu.com/job/cyphermox-test/377/console

This proves the build job didn't delete that .bzr directory (this is with latest trunk in production). So I'm not sure what deleted that .bzr directory originally but I'm pretty sure it wasn't my refactoring work.

[Robert Bruce Park (robru)]

Latest trunk code was able to publish without incident on the first try after rebuilding:

https://ci-train.ubuntu.com/job/ubuntu-landing-015-2-publish/32/console

No idea how that .bzr directory got deleted, seems to be some kind of isolated disk corruption or something, i'm highly doubtful that my refactoring caused this.

[Robert Bruce Park (robru)]

In an unrelated silo however, rtm-2 is missing unity-scopes-api/.bzr after a rebuild:

https://ci-train.ubuntu.com/job/cyphermox-test/378/consoleFull

wtf??

[Robert Bruce Park (robru)]

Ok, I was able to reproduce the above error in a separate silo so as not to trample over jhodapp's work. I then loaded the code down with crazy amounts of debug logging everywhere that any deletion happens in the entire code. Then I found this:

https://ci-train.ubuntu.com/job/ubuntu-rtm-landing-010-1-build/52/console

2014-11-07 19:29:22,623 DEBUG Deleting!: extracted_generated_source: /var/lib/jenkins/silos/ubuntu-rtm/landing-010/unity-scopes-api: Exists? True

This was quite curious as I was expecting the value to be /var/lib/jenkins/silos/ubuntu-rtm/landing-010/generated/unity-scopes-api. So indeed it turns out that there was a bug in my path handling code that caused the wrong directory to be sent for deletion. The reason for all the confusion was that, as I suspected, there was no code saying 'go delete a .bzr directory', but what was happening was actually the entire source branch was being deleted, and then a different source tree (without the branch) was being copied in it's place by some later code. Of course, how that other build succeeded with the same code is yet another mystery.

So I'm pretty sure this is fixed now, just doing some further tests to confirm it.