libc on 2016-05-25 causes Apache not to restart, libm.so.6: symbol __strtold_nan, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

[Michael Lueck]

This morning I applied the following updates to Ubuntu 12.04 webservers:

The following packages are currently pending an upgrade:

 apt 0.8.16~exp12ubuntu10.27
 apt-transport-https 0.8.16~exp12ubuntu10.27
 apt-utils 0.8.16~exp12ubuntu10.27
 libapache2-mod-php5 5.3.10-1ubuntu3.23
 libapt-inst1.4 0.8.16~exp12ubuntu10.27
 libapt-pkg4.12 0.8.16~exp12ubuntu10.27
 php5-cli 5.3.10-1ubuntu3.23
 php5-common 5.3.10-1ubuntu3.23
 php5-curl 5.3.10-1ubuntu3.23
 php5-dev 5.3.10-1ubuntu3.23
 php5-gd 5.3.10-1ubuntu3.23
 php5-mysql 5.3.10-1ubuntu3.23
 php-pear 5.3.10-1ubuntu3.23

Apache fails to restart, siting error:

apache2: Syntax error on line 212 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/php5.load: Cannot load /usr/lib/apache2/modules/libphp5.so into server: /lib/x86_64-linux-gnu/libm.so.6: symbol __strtold_nan, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

Into file /var/log/apache2/error.log

Looks like I am trying to roll back all those php packages... Which I see several updates marked Security Update in the changelogs, thus checking the "security vulnerability" box as rolling back in this case is such. :-(

[Michael Lueck (mlueck)]

Rollback successful on all 12.04 servers to PHP build 5.3.10-1ubuntu3. Websites are again working.

[Marc Deslauriers (mdeslaur)]

Could you please attach your apt upgrade log?

[Michael Lueck (mlueck)]

APT history.log

[Michael Lueck (mlueck)]

APT term.log

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[pepo (pepoviola)]

This bug also affect to my servers, after the upgrade I see this error in apache

[Wed May 25 06:42:23 2016] [notice] Graceful restart requested, doing restart
apache2: Syntax error on line 211 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/php5.load: Cannot load /usr/lib/apache2/modules/libphp5.so into server: /lib/x86_64-linux-gnu/libm.so.6: symbol __strtold_nan, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

Upgrade log

Start-Date: 2016-05-25 06:42:18
Upgrade: libc-bin:amd64 (2.15-0ubuntu10.13, 2.15-0ubuntu10.14), php5:amd64 (5.3.10-1ubuntu3.22, 5.3.10-1ubuntu3.23), libapache2-mod-php5:amd64 (5.3.10-1ubuntu3.22, 5.3.10-1ubuntu3.23), php5-curl:amd64 (5.3.10-1ubuntu3.22, 5.3.10-1ubuntu3.23), multiarch-support:amd64 (2.15-0ubuntu10.13, 2.15-0ubuntu10.14), libc6-dev:amd64 (2.15-0ubuntu10.13, 2.15-0ubuntu10.14), php5-mysql:amd64 (5.3.10-1ubuntu3.22, 5.3.10-1ubuntu3.23), php5-cli:amd64 (5.3.10-1ubuntu3.22, 5.3.10-1ubuntu3.23), libc-dev-bin:amd64 (2.15-0ubuntu10.13, 2.15-0ubuntu10.14), libc6:amd64 (2.15-0ubuntu10.13, 2.15-0ubuntu10.14), php5-common:amd64 (5.3.10-1ubuntu3.22, 5.3.10-1ubuntu3.23)
End-Date: 2016-05-25 06:42:23

---
But, in my case after manually restart it's works again.

Thx!

[Marc Deslauriers (mdeslaur)]

Ah, yes, looks like a manual restart of apache is necessary.

[Jon Bach (jonbach)]

Just adding another data point. Three web servers all went down for me about half hour ago. All 3 Ubuntu 12.04 + Apache2 + PHP5, with security updates auto-applying. All I had to do was start apache again.

Server 1 error.log:
[Wed May 25 06:43:08 2016] [notice] Graceful restart requested, doing restart
apache2: Syntax error on line 210 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/php5.load: Cannot load /usr/lib/apache2/modules/libphp5.so into server: /lib/x86_64-linux-gnu/libm.so.6: symbol __strtold_nan, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

Server 2 error.log:
[Wed May 25 06:48:32 2016] [notice] Graceful restart requested, doing restart
apache2: Syntax error on line 211 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/auth_mysql.load: Cannot load /usr/lib/apache2/modules/mod_auth_mysql.so into server: /lib/x86_64-linux-gnu/libm.so.6: symbol __strtold_nan, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

[musashiXXX (musashi-nefaria)]

Same thing happened to me this morning. A manual restart of apache fixed the problem though.

[musashiXXX (musashi-nefaria)]

Also, here's my term.log

[musashiXXX (musashi-nefaria)]

This seems to have affected perl as well, not just PHP. Here's a log snippet from apache's error.log:

apache2: Syntax error on line 210 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/perl.load: Cannot load /usr/lib/apache2/modules/mod_perl.so into server: /lib/x86_64-linux-gnu/libm.so.6: symbol __strtold_nan, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

Again, a manual restart of Apache _seems_ to have fixed the problem.

[Marc Deslauriers (mdeslaur)]

I suspect this is caused by the eglibc update, not the php5 update. Reassigning bug.

[Michael Lueck (mlueck)]

For our servers (12.04) manually restarting the Apache service would cause the same error. Apache would not manually restart.

[Joi Owen (jlellis)]

This change in libm.so has also broken the pam_mysql.so library, thus my vsftpd service is also broken. My nagios started alarming about this breaking around 6:30 am this morning, dpkg.log shows only 5 packages were updated in this morning's automatic update: man-db, libc-bin, libc6, and multiarch-support (all from amd64 arch.)

The error in auth.log is:

vsftpd: PAM unable to dlopen(pam_mysql.so): /lib/x86_64-linux-gnu/libm.so.6: symbol __strtold_nan, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference.
vsftpd: PAM adding faulty module: pam_mysql.so

(I had to manually type the above, as MS hyperv can't do simple clipboard operations like Xen has managed for over a decade now. So please forgive any typos that might be there.)

I don't think this is an apache or pam bug, as libm.so is provided by libc6.

[Adam Conrad (adconrad)]

@jlellis Does hard restarting vsftpd clear up the issue?

[Joi Owen (jlellis)]

@adconrad As it happens, it did. I just came back to update my comment to say so and found your question.

This suggests the libc6 update (not glibc, my server doesn't have glibc package installed) should have required a reboot? I don't recall if the server mentioned a reboot was required in the motd.

[Trevor Bradley (ck-trevor)]

Just a quick note that this bug affected both my Ubuntu 12.04 and 14.04 web servers today. My 16.04 servers appear to be unaffected.

[BrandonTomlinson (druke)]

We had around 60 12.04 servers impacted today, no 14.04 or newer web servers.

Specifically it looks like an update race condition where libapache2-mod-php (which restarts/reloads apache) is updated before the libc6 package.

We could prove this theory by doing 'apt-get install libc6 && apt-get update' to see, but I don't have the ability to test this myself.

[BrandonTomlinson (druke)]

no 14.04/16.04 servers were impacted, I should say.

[Trevor Bradley (ck-trevor)]

Verified. I swore one of my affected servers was 14.04, but it was actually 12.04.

[Steve Beattie (sbeattie)]

Yes, my apologies, the upstream libc fixes for CVE-2014-9761 did some reworking of functions to eliminate some repeated vulnerable code, using internal functions to do the work instead. Unfortunately, this did introduce new function references between libc and libm, causing the problems seen above. Unfortunately, these changes were applied to libc in Ubuntu 14.04 LTS and Ubuntu 15.10 as well, so I'm surprised the same problem was not seen there, too. There was no update for Ubuntu 16.04 LTS, so no issues should be seen there.

Joi: yes, the reboot motd notification was triggered with this update (and will be for future libc updates). That doesn't help you if you can't log in to see it. :(

[Andreas "Jimmy" Gredler (jimmyrax)]

Steve: I think it does not happen with 14.04 because Apache is not restarted during the upgrade process. Not sure why but that's what my tests with 14.04 showed me.
My tests with 12.04 are not 100% reproducible which makes me wonder if there is a race condition in the postinst script of libapache2-mod-php5. Is there a chance that Apache is reloaded before ldconfig is completely finished? That would also explain why a simple restart of Apache solves the problem.
This is how I test:
# apt-get install libc6=2.15-0ubuntu10.13 libc-bin=2.15-0ubuntu10.13 libapache2-mod-php5=5.3.10-1ubuntu3 php5-common=5.3.10-1ubuntu3
# apt-get upgrade

In some cases Apache does not restart successfully, in some it does.

[Anders Sandblad (arune)]

I just had this on 14.04 this morning due to logrotate restarting apache. Manually starting apache worked.
This are the last lines in apache error.log.1:
[Thu May 26 06:45:09.064664 2016] [mpm_prefork:notice] [pid 62472] AH00171: Graceful restart requested, doing restart
apache2: Syntax error on line 140 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/php5.load: Cannot load /usr/lib/apache2/modules/libphp5.so into server: /lib/x86_64-linux-gnu/libm.so.6: symbol __strtold_nan, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

[Stephen Cox (stephen-cox)]

This has affected one of my 14.04 web servers and a couple of 12.04, out of 40 Ubuntu servers.

All show the following in the Apache error log:
[Thu May 26 02:29:08.076578 2016] [mpm_prefork:notice] [pid 2199] AH00171: Graceful restart requested, doing restart
apache2: Syntax error on line 191 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/php5.load: Cannot load /usr/lib/apache2/modules/libphp5.so into server: /lib/x86_64-linux-gnu/libm.so.6: symbol __strtold_nan, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

Manually restarting Apache fixed the problem.

[Blinker (blinker1985)]

This affects several Ubuntu 12.04 servers of mine as well.
I had the Apache error as stated above but also all the Cron services aren't running anymore.

The following error message is in dmesg:
[9065273.669402] cron[10944]: segfault at 968 ip 00007f29c2ba28f0 sp 00007fff2242f8a0 error 4 in libpthread-2.15.so[7f29c2b9c000+18000]

A restart of the Cron daemon fixes this.

[William Grant (wgrant)]

I've unpublished the affected libc6 versions (2.15-0ubuntu10.14, 2.19-0ubuntu6.8, 2.21-0ubuntu4.2).

For machines that have already upgraded, restarting affected processes or rebooting the whole system should resolve any problems.

[William Grant (wgrant)]

Blinker, do those servers with crashed cron daemons use any unusual PAM modules?

[Michael Lueck (mlueck)]

@Marc #12, perhaps so it was actually eglibc package. I tested reapplying the PHP updates, IPL, and Apache stayed up. Evidently I missed that eglibc and the subsequently required IPL.

We are now good with the latest packages for 12.04. Phew! :-)

[Timur Irmatov (irmatov)]

It definitely affects 14.04. I had to restart apache2 on several 14.04 installations today.

[Blinker (blinker1985)]

@William #27. It is a virtual machine on VMWare with Plesk 12.5 installed.

I checked the /etc/pam.d folder and found a custom plesk librarie:

auth sufficient pam_plesk.so try_first_pass

Thus far all the machines with Cron daemon problems were Ubuntu 12.04 machines in combination with Plesk. So, yes. ;-)

[B. (b-deactivatedaccount-deactivatedaccount)]

Same issue

This morning, a security update for libc has been automatically
deployed on all our servers running 14.04 LTS.

unattended-upgrades-dpkg_2016-05-26_06:36:05.829399.log:Unpacking libc6-dev:amd64
unattended-upgrades-dpkg_2016-05-26_06:36:05.829399.log:Unpacking libc-dev-bin
unattended-upgrades-dpkg_2016-05-26_06:36:05.829399.log:Unpacking libc-bin
unattended-upgrades-dpkg_2016-05-26_06:36:05.829399.log:Unpacking libc6:amd64
unattended-upgrades-dpkg_2016-05-26_06:36:05.829399.log:Unpacking multiarch-support

On one of our web servers (only one??) Apache (2.4.7-1ubuntu4.9) has stopped working
after the unattended upgrades of libc6. When Apache received the signal SIGUSR1.

This did not impact any other web server with identical version of all softwares
(aligned with Ansible) ?! -- at least in the beginning of the day...

Later on at 12:52 we did "apachectl graceful" on all our web servers and all apache servers stopped responding until we restart them with "service apache2 restart".

[Thu May 26 06:35:36.222660 2016] [mpm_worker:notice] [pid 6318:tid 140737354041216]
AH00297: SIGUSR1 received. Doing graceful restart

[Thu May 26 06:35:36.231321 2016] [:alert] [pid 16293:tid 140737354041216]
(4)Interrupted system call: FastCGI: read() from pipe failed (0)

[Thu May 26 06:35:36.231395 2016] [:alert] [pid 16293:tid 140737354041216]
(4)Interrupted system call: FastCGI: the PM is shutting down, Apache seems to have disappeared - bye
apache2: Syntax error on line 140 of /etc/apache2/apache2.conf: Syntax error on line 2 of /etc/apache2/mods-enabled/security2.load: Cannot load libxml2.so.2 into server: /lib/x86_64-linux-gnu/libm.so.6: symbol __strtold_nan, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

[Steve Beattie (sbeattie)]

Blinker (and anyone else), I have eglibc/glibc packages available in the ubuntu-security-proposed ppa https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa that revert the problematic fix that added references from the libm library to the new symbols, while keeping the added __strtol*_nan symbols in libc (so as not to break people who have already updated to the pulled packages). It'd be useful to know if this also addresses the cron/Plesk issue as well.

I've reproduced apache failing to soft restart after updating to the broken libc packages, and have verified that updating to the packages in the security-proposed ppa, a currently running apache2 will soft restart successfully when the upgrade is performed from both the restored packages and from the broken packages. But verification from others is appreciated as well.

(Note that the security-proposed ppa often gets packages for testing, so it's best not to leave it enabled after testing these specific libc packages.)

Thanks for your patience, and again, my apologies.

[Launchpad Janitor (janitor)]

This bug was fixed in the package eglibc - 2.15-0ubuntu10.15

---------------
eglibc (2.15-0ubuntu10.15) precise-security; urgency=medium

  * REGRESSION UPDATE: revert CVE-2014-9761 fix due to added symbol
    dependency from libm to libc (LP: #1585614)
    - debian/patches/any/CVE-2014-9761-2.diff: keep exporting
      __strto*_nan symbols added to libc.

 -- Steve Beattie <email address hidden> Thu, 26 May 2016 00:08:17 -0700

[Launchpad Janitor (janitor)]

This bug was fixed in the package glibc - 2.21-0ubuntu4.3

---------------
glibc (2.21-0ubuntu4.3) wily-security; urgency=medium

  * REGRESSION UPDATE: revert CVE-2014-9761 fix due to added symbol
    dependency from libm to libc (LP: #1585614)
    - debian/patches/any/CVE-2014-9761-2.diff: keep exporting
      __strto*_nan symbols added to libc.

 -- Steve Beattie <email address hidden> Thu, 26 May 2016 01:28:23 -0700