Remote upgrade over aiccu connection failed

[Wookey]

I was upgrading a Lubuntu machine to precise over ssh using a SIXXS tunnel managed by aiccu. The last message I saw was:
Preparing to replace aiccu 20070115-14ubuntu1 (using .../aiccu_20070115-14.1ubuntu3.2_i386.deb) ...
runlevel:/var/run/utmp: No such file or directory

So it looks lot like aiccu dropped te connction during the upgrade. Or possibly failed to upgrade and broke. This machine is owned by a completely non-technical person who has already suffered a week of autage because the upgrade to precise failed (libc6 not upgraded from 2.13 to 2.15 before sysv-rc and grub-pc, leading to about 15 broken essential/required packages (including perl, upstart, module-init-tools, plymouth, libnih). installing libc fixed that and allow apt-get dist-upgrade to proceed, which went well until the above message.

Like ssh, it's very important that network-management tools do not break on upgrade, and preserve connections wherever possible, otherwise remote upgrades become very dangerous to perform. I donlt know if there is some good reason why this is not practical in this case, but it seems to me that expecting my connection to stay up during upgrades is not unreasonable.

[Jeroen Massar (massar)]

> So it looks lot like aiccu dropped te connction during the upgrade.

Replacing a package (and thus the binary) implies restarting AICCU to get the new binary up and running.

The only way to figure out what happened and what failed is to see the log, which is not attached.

> preserve connections wherever possible,

Due to the restart of the binary, that is impossible. In these kind of situations one could say to not upgrade tools like this remotely....

Note that the way that ssh "solves" this is to run an other binary on a different port, this is not possible with AICCU as it is actualy providing the connectivity.

The other work around is of course to use IPv4, at least as a backup, as that connectivity is likely there; thus: ssh into the box using IPv6, set up a reverse SSH over IPv4 as a backup connection (similar to the SSH scenario where the alternative port SSH is the backup) and then get into that one over IPv4 too to be sure to be able to survive the AICCU restart.

In the end though this is a non-bug mostly. Though there could maybe be a huge warning, similar to the SSH case, to note to the user that AICCU (and any other VPN tool like OpenVPN etc) is being used and that it might be that that connection gets lost, in the same way as SSH.

[Wookey (wookey)]

+++ Jeroen Massar [2013-01-13 06:42 -0000]:
> > So it looks lot like aiccu dropped te connction during the upgrade.
>
> Replacing a package (and thus the binary) implies restarting AICCU to
> get the new binary up and running.
>
> The only way to figure out what happened and what failed is to see the
> log, which is not attached.

If/When I manage to get access to the machine again (hopefully soon) I
will find the log and add it here.

> > preserve connections wherever possible,
>
> Due to the restart of the binary, that is impossible. In these kind of
> situations one could say to not upgrade tools like this remotely....

Yes. After more than a decade of remote upgrades 'just working' due to
whatever magic ssh uses I forgot that this is actually a very tricky
thing. It is the first time I have ever used aiccu to get to a remote
box (useful because it does the NAT traversal through the router at the
far end).

> Note that the way that ssh "solves" this is to run an other binary on a
> different port,

OK. I don't understand how that is done transparently, but I guess I
can read about it somewhere.

> this is not possible with AICCU as it is actualy
> providing the connectivity.
>
> The other work around is of course to use IPv4, at least as a backup, as
> that connectivity is likely there; thus: ssh into the box using IPv6,
> set up a reverse SSH over IPv4 as a backup connection (similar to the
> SSH scenario where the alternative port SSH is the backup) and then get
> into that one over IPv4 too to be sure to be able to survive the AICCU
> restart.

Right. I now know about this procedure and will indeed use it in
future. I guess when we have native IPv6 then remote upgrades will not
have this problem and will be at least as reliable as ipv4 ssh upgrades.

> In the end though this is a non-bug mostly. Though there could maybe
> be a huge warning, similar to the SSH case, to note to the user that
> AICCU (and any other VPN tool like OpenVPN etc) is being used and that
> it might be that that connection gets lost, in the same way as SSH.

A warning would have helped me enormously. I did not notice aiccu in
the list of 300MB of packages, but even if I had I probably wouldn't
have thought about the connection-loss risk without a hint.

I accept that this is not really a bug - or at best wishlist bug for a
warning. This filing may help a few other people to avoid tripping
over it.

Wookey
--
Principal hats: Linaro, Emdebian, Wookware, Balloonboard, ARM
http://wookware.org/

[Jeroen Massar (massar)]

> > Note that the way that ssh "solves" this is to run an other binary on a different port,

> OK. I don't understand how that is done transparently, but I guess I can read about it somewhere.

It is a feature of update-manager-core, debian's default "apt-get dist-upgrade" style of upgrading for instance would not do this.

As such, any logic like detecting connectivity (be that AICCU's / SSH / OpenVPN / Tinc etc) would have to be done there for it to be done properly.

Of course, maybe having a "we are upgrading this package, you might lose connectivity" warning might be useful to have as a warning in AICCU.

> I guess when we have native IPv6 then remote upgrades will not have this problem and will be at least as reliable as ipv4 ssh upgrades

Over the many years of having native dual-stack IPv4+IPv6 on a large variety of hosts I noticed that is more reliable as one can break for instance IPv4 with a firewall rule and then still use IPv6 to get in ;)

> A warning would have helped me enormously

I agree.

I've quickly checked, but I don't see either OpenVPN or Tinc having it, even network manager does not seem to have it. Note that this kind of warning would affect every single upgrade of a binary that is providing network connectivity.

> This filing may help a few other people to avoid tripping over it.

I don't think it will help much, because one typically does not check all the bug reports before doing an upgrade...

I am pondering if we could teach apt/dpkg about packages that provide network connectivity, so that when they are being upgraded that that package tag can uniformly inform the user that such an upgrade might cause issues for their connectivity.
Would need to determine the right package for that though, as some people use apt, some aptitude, some synaptic and some directly use dpkg and all these package managers, even if they are front-ends to apt, would need to know about it.