Upstream merger otto nodes need an update_host job

[Francis Ginther]

The otto slave nodes used for upstream merger testing are manually updated. The otto containers should be regularly updated as well as the host (i.e. apt-get dist-upgrade).

This does not need to be automated, in fact, if we are doing update of the host, then tests need to be run to ensure that the slave node is still usable. We should be able to do this by just re-running a small variety of recent jobs. We don't want to update all of the nodes at once in case a bad update makes them all unusable.

Currently the nodes are updated when a problem is encountered due to the lxc containers being too outdated, this happens about once a month.

There is an update_host script in lp:otto that can be utilized but it needs to refactored to make the host update optional. Perhaps the update container bit could be made into a separate script.

[Francis Ginther (fginther)]

Asana comments dump (from: https://app.asana.com/0/8736198969650/9084493553965)

Vincent Ladeuil - https://wiki.canonical.com/UbuntuEngineering/CI/Playbook/UpstreamMerger and https://wiki.canonical.com/UbuntuEngineering/CI/Playbook/Otto#cu2d needs some tweaking (as part of this task I think)

Vincent Ladeuil - I'm pretty sure there are notes somewhere about how we use otto for upstream-merger in a slightly different way than on q-jenkins but I can't find them in the playbook nor the incident logs :-/

Vincent Ladeuil - otto nodes carry uncommitted changes is related but scarce on details :-/

Francis Ginther - Vincent, I would like to discuss an update strategy more with you. The current update_host script also does an apt-get dist-upgrade. Shall we actually update these nodes to trusty and run otto w/o tweaks?

The downside is that we're more likely to be bitten by a bad update. Can we mitigate this by staggering updates such that only one host is updated per day?

Vincent Ladeuil - As discussed, a better strategy is to have an otto node where we can truly test that update_host still allow all test suites to run (well, at least at the time we do run them, further test suites updates can trigger new issues...). So, if the test suites can run an that scractch monkey host, we can update_host the production otto nodes.
Hmpf, now that I've written that, I realize it needs a scratch monkey for every *kind* of otto node :-/ intel, radeon, nvidia :-(