/etc/pm/sleep.d/60aiccu can cause aiccu to terminate

[Caleb Callaway]

[EDIT] Per comment #4, the "local-filesystem" signal is not an issue. New description:

When resuming from suspend on a computer that takes time to re-establish its network connection (e.g. a slow WiFi connection), the script /etc/pm/sleep.d/60aiccu can cause the aiccu daemon to terminate. The script checks for connectivity using ping6, and restarts the daemon if ping6 fails. If network connectivity is still not available when the daemon restarts, the daemon will terminate. Manual intervention or a reboot is required to restart the aiccu daemon.

[Original report]
The "local-filesystems" signal is not generated during a resume after suspend, so aiccu doesn't (re)start correctly during resume. The original reason for the local-filesystem condition to be present was to avoid issues with logging when the local filesystems aren't mounted, but I'd say the conditional checks in the pre-start script are sufficient, and the local-filesystems condition can be safely removed.

[Jeroen Massar (massar)]

> aiccu doesn't (re)start

AICCU should just kept on running during a resume, no need to stop it before and start it after it, nothing is changing for it (except maybe network, and that it can handle).

[Lars Düsing (lars.duesing)]

Celeb, I do not have any problems on resume - even on changing networks (I should try changing from wlan to ethernet or grps)
Do you have any troubles?

[Caleb Callaway (enlightened-despot)]

In my case the issue arises when resuming from suspend on a laptop, using
wlan for network connectivity. After resuming, ifconfig doesn't show the
sixxs interface, and aiccu service status is "stopped". Obviously IPv6
connectivity is unavailable in this situation.

I'll do some poking around to see if I can identify a root cause. Sounds
like the real issue is whatever's causing aiccu to stop.

On Mon, Oct 1, 2012 at 12:15 AM, Lars Düsing <email address hidden> wrote:

> Celeb, I do not have any problems on resume - even on changing networks (I
> should try changing from wlan to ethernet or grps)
> Do you have any troubles?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1058883
>
> Title:
> start-on conditions in Upstart script prevent aiccu from restarting
> during resume from suspend
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/aiccu/+bug/1058883/+subscriptions
>

[Caleb Callaway (enlightened-despot)]

Seems I was mistaken in the initial report: the `local-filesystems` signal
is issued on resume, but it appears the combination of `local-filesystem`
and `net-device-up` can precede the final, working wlan connection (that
is, authenticated to AP, obtained IP address, set routes). After a bit of
head scratching and hair tearing I found the file
`/etc/pm/sleep.d/60aiccu`. This appears to be the offending line:

  $P6 -I $INT -c 1 f.root-servers.net >/dev/null 2>&1 || invoke-rc.d aiccu
restart

If I redirect the output of ping6 to a file I get the following output:

  unknown host

If I comment out the offending line and use the package's original Upstart
script (with the local-filesystem condition intact), the sixxs interface is
still present on resume.

On Mon, Oct 1, 2012 at 12:13 PM, Caleb <email address hidden> wrote:

> In my case the issue arises when resuming from suspend on a laptop, using
> wlan for network connectivity. After resuming, ifconfig doesn't show the
> sixxs interface, and aiccu service status is "stopped". Obviously IPv6
> connectivity is unavailable in this situation.
>
> I'll do some poking around to see if I can identify a root cause. Sounds
> like the real issue is whatever's causing aiccu to stop.
>
>
> On Mon, Oct 1, 2012 at 12:15 AM, Lars Düsing <email address hidden> wrote:
>
>> Celeb, I do not have any problems on resume - even on changing networks
>> (I should try changing from wlan to ethernet or grps)
>> Do you have any troubles?
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/1058883
>>
>> Title:
>> start-on conditions in Upstart script prevent aiccu from restarting
>> during resume from suspend
>>
>> To manage notifications about this bug go to:
>>
>> https://bugs.launchpad.net/ubuntu/+source/aiccu/+bug/1058883/+subscriptions
>>
>
>

[Jeroen Massar (massar)]

> I found the file
`/etc/pm/sleep.d/60aiccu`. This appears to be the offending line:
>
> $P6 -I $INT -c 1 f.root-servers.net >/dev/null 2>&1 || invoke-rc.d aiccu
restart

And the million dollar prize question becomes: why is there a restart there?

> If I comment out the offending line and use the package's original Upstart
> script (with the local-filesystem condition intact), the sixxs interface is
> still present on resume.

Always good to know that the unmodified version works fine ;)

[Caleb Callaway (enlightened-despot)]

I'm not sure why the restart is there. Perhaps it's trying to guarantee the
routing tables are correct. Or perhaps it's intended to start up the aiccu
daemon if the computer is suspended on a network with native IPv6 support,
then resumed on a network without native IPv6 support, where the aiccu
daemon is needed. It also restarts aiccu if ping6 isn't found, not sure why.

Just in case it wasn't clear, the sleep.d script is part of the default
installation as well--I don't have a link, but it can be easily verified by
downloading the aiccu source package and looking for the file `60aiccu` in
the debian/ directory.

On Tue, Oct 2, 2012 at 11:00 PM, Jeroen Massar <email address hidden> wrote:

> > I found the file
> `/etc/pm/sleep.d/60aiccu`. This appears to be the offending line:
> >
> > $P6 -I $INT -c 1 f.root-servers.net >/dev/null 2>&1 || invoke-rc.d
> aiccu
> restart
>
> And the million dollar prize question becomes: why is there a restart
> there?
>
> > If I comment out the offending line and use the package's original
> Upstart
> > script (with the local-filesystem condition intact), the sixxs interface
> is
> > still present on resume.
>
> Always good to know that the unmodified version works fine ;)
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1058883
>
> Title:
> start-on conditions in Upstart script prevent aiccu from restarting
> during resume from suspend
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/aiccu/+bug/1058883/+subscriptions
>

[Jeroen Massar (massar)]

> I'm not sure why the restart is there. Perhaps it's trying to guarantee the
> routing tables are correct.

Why would they not be?

> Or perhaps it's intended to start up the aiccu
> daemon if the computer is suspended on a network with native IPv6 support,
> then resumed on a network without native IPv6 support, where the aiccu
> daemon is needed. It also restarts aiccu if ping6 isn't found, not sure why.

There is no logic in the script or in the default AICCU for this. Thus that cannot be it either.

(Logic for detecting native IPv6 and optionally disabling tunneling, or wel, at least the routes for it, is a wishlist item in upstream AICCU)

> Just in case it wasn't clear, the sleep.d script is part of the default
> installation as well--I don't have a link, but it can be easily verified by
> downloading the aiccu source package and looking for the file `60aiccu` in
> the debian/ directory.

Something in Debian/Ubuntu added it; but it is not in default, or maybe better said, original/upstream AICCU.
This as one should not have to restart AICCU, ever.

[Caleb Callaway (enlightened-despot)]

Looking at the Debian changelog, it appears this resume script has its
origins in this bug report:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=531003

On Wed, Oct 3, 2012 at 1:49 PM, Jeroen Massar <email address hidden> wrote:

> > I'm not sure why the restart is there. Perhaps it's trying to guarantee
> the
> > routing tables are correct.
>
> Why would they not be?
>
> > Or perhaps it's intended to start up the aiccu
> > daemon if the computer is suspended on a network with native IPv6
> support,
> > then resumed on a network without native IPv6 support, where the aiccu
> > daemon is needed. It also restarts aiccu if ping6 isn't found, not sure
> why.
>
> There is no logic in the script or in the default AICCU for this. Thus
> that cannot be it either.
>
> (Logic for detecting native IPv6 and optionally disabling tunneling, or
> wel, at least the routes for it, is a wishlist item in upstream AICCU)
>
> > Just in case it wasn't clear, the sleep.d script is part of the default
> > installation as well--I don't have a link, but it can be easily verified
> by
> > downloading the aiccu source package and looking for the file `60aiccu`
> in
> > the debian/ directory.
>
> Something in Debian/Ubuntu added it; but it is not in default, or maybe
> better said, original/upstream AICCU.
> This as one should not have to restart AICCU, ever.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1058883
>
> Title:
> start-on conditions in Upstart script prevent aiccu from restarting
> during resume from suspend
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/aiccu/+bug/1058883/+subscriptions
>

[Jeroen Massar (massar)]

> Looking at the Debian changelog, it appears this resume script has its
> origins in this bug report:
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=531003

Thus instead of having a log or output or anything else to actually figure out what goes wrong the 'solution' is a restart.... right.

[Jeroen Massar (massar)]

[Caleb: thanks for finding the origin of this "fix"]

So it seems this sleep "fix", by just restarting (I thought Debian was
not an ancient Microsoft product where one just reboots all the time),
which was not tested is now causing issues for people:
https://bugs.launchpad.net/bugs/1058883

Would it not be prudent to revert such a patch, or better, never have
applied it if the patch was not tested or proven working at all?!?

It seems there are no logs or other details included about this issue,
just a "it does not work" and "I restart it now it works" and then a
patch for just restarting it.

Would be great if there actually where logs for this issue or other
details that would demonstrate the problem.

Maybe time to remove this patch?

As I have stated in various places: AICCU does not need to be restarted,
the protocols used should handle it, that is why those protocols, exist.

If the tunnel 'breaks', please provide details of what is broken so that
the real problem can be addressed.... just restarting it does not
resolve such a problem.

Bigger note:
 If anybody has logs which demonstrate breakage please provide them,
 then I can take those situations along in the testing of the new AICCU.

 I've added to the to-check list to check with Virtualbox how a
 acpisleepbutton event affects the state of AICCU, thus lets see what
 the result of such a test is (best I can do, no Linux on a laptop
 anywhere near me at the moment... but this should be similar)

Greets,
 Jeroen

[Caleb Callaway (enlightened-despot)]

I stumbled across another relevant bug report while working on an un-related issue: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=566910

[Caleb Callaway (enlightened-despot)]

I think it's worth noting that a great deal of confusion surrounding aiccu restarts is probably related to inconsistencies in the daemon's behavior.

When the daemon starts, it will terminate if it cannot contact the tunneling service, which is very different from how the daemon behaves once it's started, where it is supposed to handle network outages gracefully, without a restart. This bug report contains logs that illustrate the behavior: https://bugs.launchpad.net/ubuntu/+source/aiccu/+bug/223825. The same behavior is present on my system running a fully patched Ubuntu 12.04 release.

These inconsistencies will lead to behavior that I consider incorrect when a system boots up without network connectivity and later establishes a connection (e.g. booting a laptop while outside of wireless coverage, and later coming into range). Neither aiccu's built-in behavior or the Upstart script will handle such a scenario correctly (recall the Upstart script looks for the local-file-system signal in addition to the net-device-up signal).

Seems to me the best solution to these issues is for the aiccu daemon to behave the same way on startup and while running. Given the choice, I'd vote for handling network outages gracefully in both scenarios: this makes the startup scripting very simple: start aiccu on boot, and let it handle everything from there.

[Jeroen Massar (massar)]

> When the daemon starts, it will terminate if it cannot contact the tunneling service

At failures it will always log the error and terminate. This as that is the way that a user will notice things and start looking into logs. Unfortunately people seem to think that everything needs the Win95 treatment and just restart things instead.

>, which is very different from how the
> daemon behaves once it's started, where it is supposed to handle network outages gracefully, without a restart.

That is correct, but how is that inconsistent?

> This bug report contains logs that illustrate the behavior: https://bugs.launchpad.net/ubuntu/+source/aiccu/+bug/223825.
> The same behavior is present on my system running a fully patched Ubuntu 12.04 release.

That shows as the first line indicates that somebody decided to change the startup method which then caused things to break as there was no network yet. Simple solution: start it (once, btw, and not repeatedly) when your network connectivity is up.

> These inconsistencies

That is not inconsistent. If there is an error condition (broken time, no network) it exits.

Now, if your IP address changes while it is already running, then it will keep on working.

> Neither aiccu's built-in behavior or the Upstart

Please note that Upstart is something from Ubuntu etc, do not start blaming AICCU for the way that that start script handles it.

> (recall the Upstart script looks for the local-file-system signal in addition to the net-device-up signal).

Then fix upstart.

> Seems to me the best solution to these issues is for the aiccu daemon to behave the same way on startup and while running.

As it retrieves it's primary configuration details using TIC, it needs them, in it's current incarnation it will thus properly exit.

> Given the choice, I'd vote for handling network outages gracefully in both scenarios: this makes the startup scripting very
> simple: start aiccu on boot, and let it handle everything from there.

This is what will be likely the case in the next AICCU version, which I want to finish but do not get around to unfortunately, maybe tomorrow will be the right day though....

[Caleb Callaway (enlightened-despot)]

On Fri, Oct 5, 2012 at 9:48 AM, Jeroen Massar <email address hidden> wrote:

> > When the daemon starts, it will terminate if it cannot contact the
> tunneling service
>
> At failures it will always log the error and terminate. This as that is
> the way that a user will notice things and start looking into logs.
> Unfortunately people seem to think that everything needs the Win95
> treatment and just restart things instead.
>
> >, which is very different from how the
> > daemon behaves once it's started, where it is supposed to handle network
> outages gracefully, without a restart.
>
> That is correct, but how is that inconsistent?
>
> > This bug report contains logs that illustrate the behavior:
> https://bugs.launchpad.net/ubuntu/+source/aiccu/+bug/223825.
> > The same behavior is present on my system running a fully patched Ubuntu
> 12.04 release.
>
> That shows as the first line indicates that somebody decided to change
> the startup method which then caused things to break as there was no
> network yet. Simple solution: start it (once, btw, and not repeatedly)
> when your network connectivity is up.
>
> > These inconsistencies
>
> That is not inconsistent. If there is an error condition (broken time,
> no network) it exits.
>
> Now, if your IP address changes while it is already running, then it
> will keep on working.
>

The inconsistency is in the daemon sometimes treating a lack of network
connectivity as an error condition, and sometimes treating it as part of
normal operation (as you said previously, the daemon should _not_ restart
on resume, even though network connectivity might be lost).

I'm not sure the "loud" failure that occurs when the daemon starts without
network connectivity is worth the inconsistency, since a user is very
likely to notice the lack of connectivity in other ways (e.g. can't connect
to IPv6 sites, ping6 failing, and the sixxs interface missing from the
ifconfig output). This is very much a design decision, of course, but I'm
not sure it's optimal (see below for further reasons related to init system
concerns).

>
> > Neither aiccu's built-in behavior or the Upstart
>
> Please note that Upstart is something from Ubuntu etc, do not start
> blaming AICCU for the way that that start script handles it.
>

I do not know how assigning blame is related to the words I wrote. They
simply state that neither piece of software was capable of correctly
handling the described scenario without user intervention.

>
> > (recall the Upstart script looks for the local-file-system signal in
> addition to the net-device-up signal).
>
> Then fix upstart.
>

It's probably quite possible to address the scenario using Upstart, but it
is not a good idea to say the init system is responsible for addressing the
scenario I've described. If init systems must handle such edge cases,
systemd would have to implement a separate, incompatible method for
detecting this scenario, which duplicates effort and leads to
implementation-specific fixes that have to be ported manually. In short, it
violates the principle of not repeating ourselves (I'm sure you have
special appreciation for DRY after the number of times you've had to say
aiccu shouldn...

[Caleb Callaway (enlightened-despot)]

I do agree that the reasons for including the sleep.d script aren't very clear, so I'm definitely in favor of reviewing the script's reason for existence and removing it if removal is the appropriate action. I'll try to establish contact with the Debian maintainer about this issue.

[Caleb Callaway (enlightened-despot)]

Oops, I didn't realize Jeroen had already logged a Debian bug: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=689584

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Lars Düsing (lars.duesing)]

aiccu package should be taken out of distribution due to closing of sixxs.net - there is no running tunnel broker any more.