php_network_getaddresses since last update

[Guillaume Blachon]

Hello,

Since last update :

2017-03-21 03:42:53,570 INFO Packages that will be upgraded: libc-bin libc-dev-bin libc6 libc6-dev libfreetype6 libfreetype6-dev locales multiarch-support

We get a lot of these errors on our production and development environment (PHP Application on Codeigniter) on Ubuntu 14.04 on AWS EC2 :

php_network_getaddresses: getaddrinfo failed: Name or service not known ubuntu

Apparently we are not alone today :

http://stackoverflow.com/questions/42926075/pdoexception-sqlstatehy000-2002-php-network-getaddresses-getaddrinfo-faile

http://stackoverflow.com/questions/42925765/pdo-exception-php-network-getaddresses-getaddrinfo-failed-after-changing-dns

http://serverfault.com/questions/839659/php-throwing-server-errors-after-ubuntu-ran-an-unattended-upgrades-session-any

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Guillaume Blachon (guillaumeblachon)]

Probably linked to DNS change here : glibc 2.23-0ubuntu6 (https://launchpad.net/ubuntu/+source/glibc/2.23-0ubuntu6)

[Joern Heissler (joernheissler)]

Same here. But I guess the best fix is to restart php or the whole server.

[Daniel Fariña (danielfh)]

Can you confirm that restarting the server fixes the problem permanently?

[Joern Heissler (joernheissler)]

I restarted php on affected servers, problem didn't come back yet.
What I saw in strace / tcpdump is a dns request with rrtype 0x5100 in the question.

There have been problems with libc6 updates before and there is this debconf thingy "Services that need to be restarted". This indicates a known problem with libc6 updates. php-fpm should be added to this list of services...

I don't fully understand how exactly the bug "works". My working theory goes like this:
php-fpm master process has lots of shared libs mmaped into memory. Those are not affected by a libc6 upgrade that changes libs on the file system.
When php-fpm forks a worker process, it inherits all the maps.
But the worker process does some actual work and loads (diff /proc/{$master,$worker}/maps) /lib/x86_64-linux-gnu/libnss_dns-2.23.so which is the *new* version and not ABI compatible with the already loaded libc6.
If my theory holds, restarting the affected process will indeed fix the issue.

[Steve Beattie (sbeattie)]

Joern, yes, that's what's happening, the new libnss_dns is being dynamically loaded (dlopen'ed) but ends up querying the older libresolv/libc, and there was mistakenly an ABI change introduced to cope with CVE-2015-5180.

There are currently test glibc/eglibc packages building in the https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa/ that revert this change (as it breaks other things, see bug 1674532). Unfortunately, when the reversion is issued, that will likely mean needing to restart you php-fpm processes again. :( I'm terribly sorry about that.

[Ast (nowarry)]

I can confirm, that restart of the whole server have worked, but restart of php-fpm didn't work for us.

[Antoine (antwan.fr)]

This is not php related, the same issue happens with python/psycopg2.

[Salman (ic3man321)]

Restart fixes it for few hours. Servers are still randomly going down.

[Ross Campbell (ross-campbell)]

I see that glibc 2.23-0ubuntu7 was just pushed out...
https://launchpad.net/ubuntu/+source/glibc/2.23-0ubuntu7

glibc (2.23-0ubuntu7) xenial-security; urgency=medium

  * REGRESSION UPDATE: Previous update introduced ABI breakage in
    internal glibc query ABI
    - Revert patches/any/CVE-2015-5180-regression.diff
      (LP: #1674532)

[fariazz (fariazz)]

We are having the same issue in 14.04. This is the libc6 version we have:

```
dpkg -s libc6
Package: libc6
Status: install ok installed
Priority: required
Section: libs
Installed-Size: 10568
Maintainer: Ubuntu Developers <email address hidden>
Architecture: amd64
Multi-Arch: same
Source: glibc
Version: 2.21-0ubuntu4
Replaces: libc6-amd64
Depends: libgcc1
Suggests: glibc-doc, debconf | debconf-2.0, locales
Breaks: hurd (<< 1:0.5.git20140203-1), libtirpc1 (<< 0.2.3), lsb-core (<= 3.2-27), nscd (<< 2.21)
Conflicts: prelink (<= 0.0.20090311-1), tzdata (<< 2007k-1), tzdata-etch
Conffiles:
 /etc/ld.so.conf.d/x86_64-linux-gnu.conf 593ad12389ab2b6f952e7ede67b8fbbf
Description: GNU C Library: Shared libraries
 Contains the standard libraries that are used by nearly all programs on
 the system. This package includes shared versions of the standard C library
 and the standard math library, as well as many others.
Homepage: http://www.gnu.org/software/libc/libc.html
Original-Maintainer: GNU Libc Maintainers <email address hidden>

```

[Zunk (zunk)]

Facing the same issue on 'Ubuntu 14.04.3 LTS' with libc6 2.19:

---
Package: libc6
Status: install ok installed
Priority: required
Section: libs
Installed-Size: 10503
Maintainer: Ubuntu Developers <email address hidden>
Architecture: amd64
Multi-Arch: same
Source: eglibc
Version: 2.19-0ubuntu6.11
Replaces: libc6-amd64
Provides: glibc-2.19-1
---

After restarting PHP-FPM process issue has done. in total affected about 50-60 nodes in different AWS regions.

[Matteo Magni (ilbonzo)]

We have same issue on Ubuntu 14.04.5 LTS
---
$ dpkg -s libc6
Package: libc6
Status: install ok installed
Priority: required
Section: libs
Installed-Size: 10503
Maintainer: Ubuntu Developers <email address hidden>
Architecture: amd64
Multi-Arch: same
Source: eglibc
Version: 2.19-0ubuntu6.10
Replaces: libc6-amd64
Provides: glibc-2.19-1
---

After restarting webserver everything is gonna be ok

[Minardus van der Kooi (minardinho)]

We have the same issue on Ubuntu 14.04.1 LTS

dpkg -s libc6
Package: libc6
Status: install ok installed
Priority: required
Section: libs
Installed-Size: 10503
Maintainer: Ubuntu Developers <email address hidden>
Architecture: amd64
Multi-Arch: same
Source: eglibc
Version: 2.19-0ubuntu6.11
Replaces: libc6-amd64
Provides: glibc-2.19-1

A restart of PHP-FPM fixed it temporarily. Problem came back day later, so that wasn't a permanent fix.
We get this problem while connecting to a Amazon RDS instance via PHP.

[Sean Leach (sean.leach)]

I'm also seeing this issue replicated on 14.04.2 -

$ apt-cache show libc6 | grep Version
Version: 2.19-0ubuntu6.11
Version: 2.19-0ubuntu6

cat /var/log/apt/history.log

...
Start-Date: 2017-03-21 07:11:13
Upgrade: libgnutls-openssl27:amd64 (2.12.23-12ubuntu2.6, 2.12.23-12ubuntu2.7), multiarch-support:amd64 (2.19-0ubuntu6.9, 2.19-0ubuntu6.10), libfreetype6:amd64 (2.5.2-1ubuntu2.5, 2.5.2-1ubuntu2.6), libc-dev-bin:amd64 (2.19-0ubuntu6.9, 2.19-0ubuntu6.10), libc-bin:amd64 (2.19-0ubuntu6.9, 2.19-0ubuntu6.10), libc6:amd64 (2.19-0ubuntu6.9, 2.19-0ubuntu6.10), libgnutls26:amd64 (2.12.23-12ubuntu2.6, 2.12.23-12ubuntu2.7), libc6-dev:amd64 (2.19-0ubuntu6.9, 2.19-0ubuntu6.10)
End-Date: 2017-03-21 07:11:31
...

A system reboot brought back stability for circa 24 hours, but the system went down again this morning. Interestingly, around 18 hours after reboot network activity spiked substantially, and steadily.

At it's peak, a meagre 0.7mb/s upstream PHP stopped responding properly to requests.

[Guillaume Blachon (guillaumeblachon)]

Same for Amazon RDS instance.

Problem came back after reboot.

[David Allen (doctorallen)]

I am also encountering the same issue, on 16.04 trying to connect to an RDS instance.

[Andrew Martin (asmartin)]

I'm also seeing this on 14.04.5 with the following versions:
# apt-cache show libc6 | grep Version
Version: 2.19-0ubuntu6.11
Version: 2.19-0ubuntu6

Restarting php5-fpm temporarily fixes the problem, but it comes back within a few hours. I've noticed this in PHP code that connects to a MySQL server and PHP code that attempts to perform an LDAP bind

[Chris Monahan (cobra-v)]

I would really like to know why Ubuntu feels it can just throw libc6 updates out there without restarting services? It feels like a kid at the controls just pushing buttons. Please remember that your server operating system is used to host important and high traffic websites. At this point, we have to turn off automatic security updates and run them manually. The automatic updates are something we have enjoyed for years!

[Lukas (jossnaz)]

confirmed on 2 servers

[Infra Codigo Cyberlin (infracodigo)]

same here, my temporary solved is downgrade to previous version package. in apt log you can see it.
my list downgrade package:

libc6:amd64=2.23-0ubuntu3 locales:amd64=2.23-0ubuntu3 libc-bin:amd64=2.23-0ubuntu3 multiarch-support:amd64=2.23-0ubuntu3 libfreetype6:amd64=2.6.1-0.1ubuntu2

After downgrade everything gonna be ok! hope newest version can fix this bugs.

[Lukas (jossnaz)]

downgrading did not solve my issue on ubuntu 16.04, neither did restarting the server.

my server is unusable, and its a productive server. How many users like me were just thrown out the window "fuck 'em"

i am very disappointed. No work around, no solution provided even after 5 days. Just "fuck 'em".

[Pavel Vorobiov (pavel-vorobiov)]

We have the same issue. Changed nameserver to Freenom World's IP

[Nacho Vazquez (ivazquez1)]

I can confirm the fix for 14.04 is still not working. Even after apache restart and full server reboots.

[Vinicius Camara (viniciuscamara)]

I can confirm error for ubuntu 18.04. I still receiving this error after ubuntu migration to 14.04 for 18.04.