Build job uploads stale packages.

[Robert Bruce Park]

Just ran into this:

2015-01-08 03:29:48,169 INFO Uploading packages to the PPA.
2015-01-08 03:29:48,534 INFO
Checking signature on .changes
Good signature on /var/lib/jenkins/silos/ubuntu-rtm/landing-008/autopilot_1.5.0+15.04.20150108~rtm-0ubuntu1_source.changes.
Checking signature on .dsc
Good signature on /var/lib/jenkins/silos/ubuntu-rtm/landing-008/autopilot_1.5.0+15.04.20150108~rtm-0ubuntu1.dsc.
Uploading to ppa (via ftp to ppa.launchpad.net):
  Uploading autopilot_1.5.0+15.04.20150108~rtm-0ubuntu1.dsc: done.
  Uploading autopilot_1.5.0+15.04.20150108~rtm.orig.tar.gz: done.
  Uploading autopilot_1.5.0+15.04.20150108~rtm-0ubuntu1.diff.gz: done.
  Uploading autopilot_1.5.0+15.04.20150108~rtm-0ubuntu1_source.changes: done.
Successfully uploaded packages.

2015-01-08 03:29:48,916 INFO
Checking signature on .changes
Good signature on /var/lib/jenkins/silos/ubuntu-rtm/landing-008/ubuntu/autopilot/autopilot_1.5.0+15.04.20141031~rtm-0ubuntu1_source.changes.
Checking signature on .dsc
Good signature on /var/lib/jenkins/silos/ubuntu-rtm/landing-008/ubuntu/autopilot/autopilot_1.5.0+15.04.20141031~rtm-0ubuntu1.dsc.
Uploading to ppa (via ftp to ppa.launchpad.net):
  Uploading autopilot_1.5.0+15.04.20141031~rtm-0ubuntu1.dsc: done.
  Uploading autopilot_1.5.0+15.04.20141031~rtm.orig.tar.gz: done.
  Uploading autopilot_1.5.0+15.04.20141031~rtm-0ubuntu1.diff.gz: done.
  Uploading autopilot_1.5.0+15.04.20141031~rtm-0ubuntu1_source.changes: done.
Successfully uploaded packages.

2015-01-08 03:29:48,940 INFO Check ppa status in https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu-rtm/landing-008
2015-01-08 03:34:49,474 INFO Checking PPA for source packages...
2015-01-08 03:39:49,819 INFO Checking PPA for source packages...
2015-01-08 03:44:50,287 INFO Checking PPA for source packages...
2015-01-08 03:49:50,840 INFO Checking PPA for source packages...
2015-01-08 03:49:51,226 ERROR Versions in PPA did not match our expectations.
Expected: autopilot 1.5.0+15.04.20141031~rtm-0ubuntu1
Found: autopilot 1.5.0+15.04.20150108~rtm-0ubuntu1
2015-01-08 03:49:51,227 ERROR Build failed: Some packages failed to build.
Build step 'Execute shell' marked build as failure
Finished: FAILURE

What's going on here is, the silo started out as an MP silo, but original build failed due to incorrect changelog contents. Silo was changed to a sync silo, however the previous failing build stuck around, and at the package-upload step, both packages were uploaded. Of course the newer-versioned one won, and the older-versioned sync failed to be accepted into the ppa (silently, after succeeding to upload). Note that the original failing build failed before uploading to the ppa, so the ppa was in clean state when sync build failed.

Solution: build job needs to more aggressively delete stale builds when beginning to build. I know it already has some code for that, but it somehow failed in this case. Further investigation is needed (and I think the logging for file deletions should be changed from logging.debug to logging.info because that info sure would have been handy in this log, but it's not present because nobody was expecting anything to go wrong here).

[Robert Bruce Park (robru)]

I believe this is now fixed with the new build.py landing. Sync builds and MP builds are no longer done in an inconsistent location, so there's no chance for one package to be built in SILO_DIR/ubuntu/foo/ and another in SILO_DIR/foo/. Also, the upload code no longer recursively searches for source.changes files, but instead only looks for glob(SILO_DIR/*_source.changes) which is another measure that would prevent a situation like this.