citrain could use a more accurate way to upgrade from silos

[Brendan Donegan]

Currently the citrain tool uses the command 'apt-get -o Dir::Etc::SourceList=/dev/null update' to disable sources.list and make sure that only the PPA is installed from. This works fine in most cases, but we've encountered a recent case where a successful install required a package that was in the main archive. Thanks to Mirv (Timo) we located this command that does something similar but still allows the main archive to be used.

apt update -o Dir::Etc::sourcelist="sources.list.d/ci-train-ppa-service-ubuntu-rtm-landing-009-14.09.list" -o Dir::Etc::sourceparts="-" -o APT::Get::List-Cleanup="0"

We should update the citrain tool to use that command so that we don't hit speed bumps again in future. The only tricky bit will be getting the filename right - a clever use of find should do the trick though.

ProblemType: Bug
DistroRelease: Ubuntu 14.10
Package: phablet-tools 1.1+14.10.20140929-0ubuntu1
Uname: Linux 3.13.0-031300rc2-generic x86_64
ApportVersion: 2.14.4-0ubuntu2
Architecture: amd64
CurrentDesktop: Unity
Date: Tue Oct 7 10:16:45 2014
InstallationDate: Installed on 2013-11-30 (310 days ago)
InstallationMedia: Ubuntu 13.10 "Saucy Salamander" - Release amd64 (20131016.1)
PackageArchitecture: all
SourcePackage: phablet-tools
UpgradeStatus: Upgraded to utopic on 2014-05-16 (143 days ago)

[Sergio Schvezov (sergiusens)]

This is actually reverting back to the previous behavior we had which had the potential of "upgrading" packages out of silo.

Sure you get the dependencies, but the added load of other packages can come in too.

Using main only is not an option as we also have dependencies in universe, the fact that it worked without upgrading unwanted packages might of been just a coincidence with the time of testing (e.g.; just after a new image came out of the oven so all packages were up to date)

The best solution here is to have people state they have new deps and taking care of them prior to running the tool. This is not ideal and I can see this working without any intervention from the user and without disabling any sources if instead of running dist-upgrade you do the following:

- iterate over each binary package in the silo
- if it's not part of the current install list, discard; else add to the "install_list"
- look at the dependencies, add to the "install_list"
- apt install "install_list"

[Brendan Donegan (brendan-donegan)]

Ah okay - I was sure I'd tested this properly but you're probably right that it was a coincidence. Yeah it looks like we will need to get the install list and compare it to the silo binaries. I have some code that can do that already whipped up.

[Antti Kaijanmäki (kaijanmaki)]

How much effort would it be to actually build pristine images per silo that contain the packages in the silo?

Each time we have to make the images writable and add silo ppa's manually and run apt-get we effectively taint the testing results as they don't match the state that a released image would have.

Simple case: as the root is made writable we would not catch the situation where a component is actually depending on write access which would not be possible under ro root.

[Brendan Donegan (brendan-donegan)]

That's the silver bullet of testing, but I understand that may be some time
in the future. Until then citrain needs to do its job better - but I too
would like an image per silo

On Sat, Oct 11, 2014 at 12:10 PM, Antti Kaijanmäki <
<email address hidden>> wrote:

> How much effort would it be to actually build pristine images per silo
> that contain the packages in the silo?
>
> Each time we have to make the images writable and add silo ppa's
> manually and run apt-get we effectively taint the testing results as
> they don't match the state that a released image would have.
>
> Simple case: as the root is made writable we would not catch the
> situation where a component is actually depending on write access which
> would not be possible under ro root.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1378245
>
> Title:
> citrain could use a more accurate way to upgrade from silos
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/phablet-tools/+bug/1378245/+subscriptions
>

[Michał Sawicz (saviq)]

The command in the description (even if I have no idea what it did :P) worked like a charm, can we put it in the citrain tool asap please?

[Michał Sawicz (saviq)]

Oh wait... it actually upgrades from distro, too... so misses the point?

[Timo Jyrinki (timo-jyrinki)]

The command, when properly given the correct sources.list etc, does only update the PPA archive information, after which apt dist-upgrade should do the correct thing.

I thought citrain actually used this already. If so, I think the main remaining citrain problem for eg. Qt testing is that it (maybe?) uses only 'upgrade' which then fails when there'd be new packages to be installed as new dependencies from the package versions in the PPA.

Or, alternatively to the 'upgrade' possibility (vs. more correct dist-upgrade which should be fine if only PPA packages are considered as the ones to be upgrades), the cause of the problem might be the fact that normal archive information is missing on a fresh flash. And it'd be only obtainable by a full apt update, making it impossible to combine this feature when new packages are needed from the archive.

Nevertheless, the citrain tool is not often usable with a big Qt landing.

[Michał Sawicz (saviq)]

My bad, the -os need to be passed both to the upgrade/dist-upgrade comand rather than to update.

[Michał Sawicz (saviq)]

Any reason why this still has not made it into the citrain tool?

[Robert Bruce Park (robru)]

> Any reason why this still has not made it into the citrain tool?

Probably because this is the first I've heard of this just now.

[Robert Bruce Park (robru)]

> How much effort would it be to actually build pristine images per silo that contain the packages in the silo?

That is actually a planned feature, but don't hold your breath. The roadmap for getting there is quite long.

> Simple case: as the root is made writable we would not catch the situation where a component is actually depending on write access which would not be possible under ro root.

The most recent release of the tool switches the root back to readonly after installing packages, so it's not that far off from released images.

[Robert Bruce Park (robru)]

Everybody that cares about this bug, please try out my new script from the linked branch:

http://bazaar.launchpad.net/~robru/phablet-tools/make-useful-again/view/head:/citrain

[Robert Bruce Park (robru)]

Please test out this newer script now:

http://bazaar.launchpad.net/~robru/phablet-tools/further-fixes/view/head:/citrain