no ping response on all hosts after upgrade

[Eric Metzler]

[Impact]

The backuppc package struggled with ping response on all hosts after upgrade, because the package on both Bionic and Focal doesn’t contain a valid setting for ping6.
This situation before the patch was applied was really unfortunate for users whose backup hosts are using IPv6.

[Test Plan]

** Reproduction **

Make a container(s) for testing:

First option: (creating focal vm)
$ lxc launch images:ubuntu/focal focal-test
$ lxc shell focal-test

Second option: (creating bionic vm)
$ lxc launch images:ubuntu/bionic bionic-test
$ lxc shell bionic-test

Simply install the backuppc package using ‘apt install’ and check the variables in the config.pl file.

Follow the reproduction steps above to ensure that the variable is properly set.

[Where problems could occur]

First of all, any code change might change the behavior of the package in a specific situation and cause other errors. The regression possibilities here are really low.
It is unlikely but possible that users could hardcode valid setting for ping6. In that case if the same user wants to install backuppc, there is a high risk that a warning message will appear and there is a necessity to unify the setting by removing it and installing again.
Another possible (albeit unlikely) source of regressions is the fact that we will be rebuilding backuppc against newer versions of its build dependencies, especially on Bionic.

---------------original bug report----------------------

Binary package hint: backuppc

After upgrading from 3.1.0 to 3.2.0 as part of the Ubuntu server upgrade from maverick to natty, none of my hosts were backing up, and all (including localhost) were reporting "no ping response". I followed all instructions prior to upgrading, and backuppc was at 3.1.0-9ubuntu2 and working. I discovered the addition of the Ping6Path config variable, which is set to "undef" by default. My config.pl file had the line

$Conf{Ping6Path} = undef;

after both an upgrade to 3.2.0-3ubuntu4 and a clean install after purging the package. This, combined with the added code to Lib.pm caused the $pingPath in $pingCmd to be empty, thus causing the "no ping response" errors.

Changing the Ping6Path line in config.pl to

$Conf{Ping6Path} = '/bin/ping6';

solved the problem. This value should be populated by default for a fresh install if /bin/ping6 is found, just as /bin/ping is populated in $Conf{PingPath}.

# lsb_release -rd
Description: Ubuntu 11.04
Release: 11.04
# apt-cache policy backuppc
backuppc:
  Installed: 3.2.0-3ubuntu4
  Candidate: 3.2.0-3ubuntu4
  Version table:
 *** 3.2.0-3ubuntu4 0
        500 http://us.archive.ubuntu.com/ubuntu/ natty/main amd64 Packages
        100 /var/lib/dpkg/status

[Chuck Short (zulcss)]

So you are using ipv6?

Thanks
chuck

[Eric Metzler (ericlmetzler)]

Chuck,

Actually, I'm not really sure. I installed server 10.10 and I don't think I
strayed much from the defaults. For some reason, either backuppc thinks that
I am using ipv6, or it was installed/enabled during the upgrade to 11.04. Is
it usually installed or enabled by default? How can I tell if I'm using it?
Thanks

Eric

On Mon, May 16, 2011 at 1:59 PM, Chuck Short <email address hidden>wrote:

> So you are using ipv6?
>
> Thanks
> chuck
>
> ** Changed in: backuppc (Ubuntu)
> Importance: Undecided => Low
>
> ** Changed in: backuppc (Ubuntu)
> Status: New => Incomplete
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/782890
>
> Title:
> no ping response on all hosts after upgrade
>
> Status in “backuppc” package in Ubuntu:
> Incomplete
>
> Bug description:
> Binary package hint: backuppc
>
> After upgrading from 3.1.0 to 3.2.0 as part of the Ubuntu server
> upgrade from maverick to natty, none of my hosts were backing up, and
> all (including localhost) were reporting "no ping response". I
> followed all instructions prior to upgrading, and backuppc was at
> 3.1.0-9ubuntu2 and working. I discovered the addition of the Ping6Path
> config variable, which is set to "undef" by default. My config.pl file
> had the line
>
> $Conf{Ping6Path} = undef;
>
> after both an upgrade to 3.2.0-3ubuntu4 and a clean install after
> purging the package. This, combined with the added code to Lib.pm
> caused the $pingPath in $pingCmd to be empty, thus causing the "no
> ping response" errors.
>
> Changing the Ping6Path line in config.pl to
>
> $Conf{Ping6Path} = '/bin/ping6';
>
> solved the problem. This value should be populated by default for a
> fresh install if /bin/ping6 is found, just as /bin/ping is populated
> in $Conf{PingPath}.
>
> # lsb_release -rd
> Description: Ubuntu 11.04
> Release: 11.04
> # apt-cache policy backuppc
> backuppc:
> Installed: 3.2.0-3ubuntu4
> Candidate: 3.2.0-3ubuntu4
> Version table:
> *** 3.2.0-3ubuntu4 0
> 500 http://us.archive.ubuntu.com/ubuntu/ natty/main amd64
> Packages
> 100 /var/lib/dpkg/status
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/ubuntu/+source/backuppc/+bug/782890/+subscribe
>

[amichair (amichai2)]

I was hit by the same bug after upgrade to Natty - the original poser's solution worked (configuring "$Conf{Ping6Path} = '/bin/ping6';".

I totally agree that this should be set by default - there's no reason for others to waste time trying to figure out why backups aren't working (there's no clear indication that this is the cause).

I don't have ipv6 support on LAN nor externally, and have not changed anything in this regard in Ubuntu's internal configuration as far as I know. If there's anything in my configuration that you'd like me to check to help with this, I'd be happy to.

[Chuck Short (zulcss)]

Thanks for the response, ipv6 shouldnt be the default and I will try to get this fixed soon.

Regards
chuck

[txapelgorri (ibon)]

This bug was opened because a Maverick to Natty upgrade, but the Lucid (Server LTS) current version (3.2.0-3ubuntu4~lucid1) of BackupPC is also affected. I see that recent versions of Ubuntu Server have more recent version of BackupPC, and I presume that this bug is solved on that versions, but Lucid version still remains with this bug.

[Danny Sauer (ubuntu-dannysauer)]

This happened on my lucid-to-precise upgrade. I didn't even have the Ping6Path in my config.pl; I would have never thought that maybe the tool would use IPV6 on a network without IPV6.

[Danny Sauer (ubuntu-dannysauer)]

I'm pretty sure this is a bug in BackupPC, since:
sauer@pyro:~$ lsmod | grep -c ipv6
0

[Danny Sauer (ubuntu-dannysauer)]

Oh. I guess that's not a module any more, because I'm behind the times on my legacy IPv4 network. :)

So, to check if this is the problem, you can temporarily disable IPv6:

sauer@pyro:~$ ip addr show lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
sauer@pyro:~$ sudo sysctl net.ipv6.conf.all.disable_ipv6
net.ipv6.conf.all.disable_ipv6 = 0
sauer@pyro:~$ sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.all.disable_ipv6 = 1
sauer@pyro:~$ ip addr show lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo

Note that the IPv6 translation address pops up on all interfaces when the disable setting is 0, and goes away immediately when the disable setting is set to 1. To make this permanent on 12.04, create a file named /etc/sysctl.conf.d/60-disableipv6.conf and add a line that says net.ipv6.conf.all.disable_ipv6 = 1. I'm partial to running:
sauer@pyro:~$ echo "net.ipv6.conf.all.disable_ipv6 = 1" | sudo tee /etc/sysctl.d/60-disableipv6.conf
net.ipv6.conf.all.disable_ipv6 = 1
The setting in that file will take effect when you reboot or reload the sysctl settings by running "sudo start procps".

This is still a bug in Precise, IMHO. If IPv6 is going to be enabled by default on the systems, then the Ping6 command should be set in backuppc.pl

[jhansonxi (jhansonxi)]

This still occurs (at least with localhost) with BackupPC 3.3.0 on *buntu 14.04 (Trusty Tahr). The Ping6Path value is still "undef" by default. Workaround is to set it to /bin/ping6 (or force it true for localhost):
http://sourceforge.net/p/backuppc/mailman/message/27518026/

[Michael Mess (michael-michaelmess)]

This bug is quite old, but I stumbled on a broken backuppc (refusing backup due to ping) after upgrading the backup server. Backuppc now has version 3.3.0 (I am even not sure that this has changed during the upgrade that broke the backup).

ping6 hostname failed, but ping succeeded.

The ping6 configuration was empty, i just added /bin/ping (because /bin/ping6 didn't work).

Now backup seems to work again.

[Robie Basak (racb)]

Thank you for the update. This looks like it should be straightforward to fix, so I've added it to the server team's backlog. But since the workaround is trivial it will be prioritised quite low and we may continue to not get to it any time soon.

If someone can volunteer a patch (in the form of the appropriate change to packaging), that would be most welcome. We'll need to make sure the development version of Ubuntu is fixed first, if it isn't already, and if a patch to upstream is required it would be helpful to send it upstream too.

Hopefully that helps anyone who is able to volunteer some time on this.

[Sergio Durigan Junior (sergiodj)]

I have installed backuppc on Jammy (current development release) and verified that $Conf{Ping6Path} is correctly set to /bin/ping6. I also installed the package on Bionic (our oldest LTS), and verified that Ping6Path is empty there. The same issue happens on Focal.

I think the easiest solution here would be to pass "--bin-path ping6=/bin/ping6" to configure.pl.

[Sergio Durigan Junior (sergiodj)]

This came up again during my daily triage. Nothing has changed since the last time I looked into the bug. I still don't think it's a good candidate for an SRU.

[Michał Małoszewski (michal-maloszewski99)]

I've noticed that none of the releases have the $Conf{Ping6Path} set to /bin/ping6.

[Axel Beckert (xtaran)]

I assume the problem is rather that $Conf{Ping6Path} is not set at all ("undef" means that it is not set) and that hosts have AAAA records, maybe even only AAAA records. From my experience this is though only needed for IPv6-only hosts and not needed for dual-stack hosts. (Which is probably also the reason why it hasn't been added by default.)

The actually required value for $Conf{Ping6Path} might though differ depending on which ping implementation is used:

With legacy inetutils-ping these are still two different binaries and /bin/ping doesn't resolve AAAA records and it needs be set to /bin/ping6.

With modern iputils-ping /bin/ping6 is a (likely transitional) symlink to /bin/ping which I assume will go away somewhen in the future. So in that case it should be set to /bin/ping.

You can set it to /bin/ping6 with iputils-ping, too, but it will break once iputils-ping no more ships the /bin/ping6 symlink.

[Axel Beckert (xtaran)]

Ok, the maintainer of iputils-ping stated that he does "not have any plans to remove [the ping6 symlink] any time soon." So backporting b034914233c05982b524414f548ca3454292f457 should be save wrt. this even in the long run.

[Michał Małoszewski (michal-maloszewski99)]

MP with SRU bug report has been sent to inner review.

[Michał Małoszewski (michal-maloszewski99)]

#15

Axel, I was wrong while testing it. Now everything is fine and my code changes are evaluated in pull request.

[Robie Basak (racb)]

Hello Eric, or anyone else affected,

Accepted backuppc into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/backuppc/3.3.2-3ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Robie Basak (racb)]

Hello Eric, or anyone else affected,

Accepted backuppc into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/backuppc/3.3.1-4ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

The fix for this bug has been awaiting testing feedback in the -proposed repository for bionic for more than 90 days. Please test this fix and update the bug appropriately with the results. In the event that the fix for this bug is still not verified 15 days from now, the package will be removed from the -proposed repository.

[Robie Basak (racb)]

The proposed SRU in Focal has been trumped by a security update, so needs rebasing and reuploading if we still need it.

[Michał Małoszewski (michal-maloszewski99)]

Hi Eric,

I did try to reproduce that problem, but I could not do that. I've read through the backuppc documentation and what I can say is:

First, I have built "backuppc setup" to be able to perform the crucial operations on backuppc.
I have built bionic and focal lxc containers.

I wanted to backuppc focal from bionic what was successful.

Then in config.pl file I have put "undef" to both ping(v4) and pingv6. The output:
Can't connect to server (unix connect: Connection refused)

So it was kind of expected.

To do that I have used:

$ BackupPC_serverMesg backup IP focal1 backuppc 1

The BackupPC_serverMesg command verifies that BackupPC is running. This send a message to BackupPC and prints the response.

Then I have set undef to pingv4 and defined pingv6 ('/bin/ping6') and vice-versa.
I could not reproduce that issue then.
I think that a regular ping in that case handles both IPv4 and IPv6.

Could you please help me identify where I am wrong in my testing? I'd appreciate your answer.

P.S. I have tried more complex tests like sending email after backup is done. At first sight, I thought it was successful, but it turned out not to be as successful as I assumed.

[Łukasz Zemczak (sil2100)]

The version of backuppc in the proposed pocket of Bionic that was purported to fix this bug report has been removed because the bugs that were to be fixed by the upload were not verified in a timely (105 days) fashion.