[SRU] Pinger crashes with segfault in libc

[mr_willem]

[Description]

Malformed ICMP packets were accepted into processing with undefined
and potentially nasty results.

Both sets of flaws can result in pinger segmentation fault and halting
the Squid functionality relying on pinger for correct operation.

A backtrace obtained from a failing guest, shows

#0 0x00007f6e3833cb4a in __strcmp_sse2 () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f6e38369971 in __tzfile_compute (timer=1415395716, use_localtime=<optimized out>, leap_correct=0x7ffff810be00,
    leap_hit=0x7ffff810bdf0, tp=0x7f6e38679de0 <_tmbuf>) at tzfile.c:786
#2 0x00007f6e38368547 in __tz_convert () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007f6e38dc2683 in _db_print(char const*, ...) ()
#4 0x00007f6e38dc300b in Debug::finishDebug() ()
#5 0x00007f6e38dc0581 in IcmpPinger::Recv (this=0x7f6e38fd1680 <control>) at IcmpPinger.cc:190
#6 0x00007f6e38dbf04e in main (argc=<optimized out>, argv=<optimized out>) at pinger.cc:223

Dissecting the trace , it appears that the amount
of read bytes on the IcmpPinger::Recv method is < 0 (error), but
no validation is being performed on the readed data, thus an segfault is being triggered.

This patch handles most of this cases by bound checking all the recv values, also the ICMP type checking routines are improved to properly
handle just existent types.

[Test Case]

- Install latest squid3 from archive.
- Enable ICMP pinger
- Wait for some anomalous ICMP response to come from any origin server,
- Then the pinger process will segfault with an error like this:

Nov 8 06:28:56 gd2mrbp001 kernel: [1543874.494491] pinger[8802]: segfault at 0 ip 00007fd276d6bb4a sp 00007fff11711908 error 4 in libc-2.19.so[7fd276ce4000+1bb000]
- After applying this patch, i have ran this over 10 times without
experiencing this issue anymore.

[Other Customer information]

After upgrading ubuntu over from 12. something to the 14.04 lts release my squid proxy did not work any longer.
It often happens that sites are loading quite long and in the end there is a connection problem.
Just in this moments I can see the following happen in dmesg output

[4611237.325605] pinger[15651]: segfault at 0 ip 00007f6db12aeb4a sp 00007fff2552ad68 error 4 in libc-2.19.so[7f6db1227000+1bb000]
[4611258.022931] init: squid3 main process (32738) killed by ABRT signal
[4611258.022968] init: squid3 main process ended, respawning

I already tried to reinstall squid3 and glibc and I build the squid3 packages on the machine just in case there was something wrong with my glibc.

I am using ubuntu 14.04.1 lts
The squid version is
squid3 3.3.8-1ubuntu6.1

[Jorge Niedbalski (niedbalski)]

Linked Vivid branch, Edited bug description for SRU. Attached Trusty patch.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Iain Lane (laney)]

Uploaded for trusty too (I added the bug closure to the changelog for both)

[Iain Lane (laney)]

and utopic as well

[Launchpad Janitor (janitor)]

This bug was fixed in the package squid3 - 3.3.8-1ubuntu9

---------------
squid3 (3.3.8-1ubuntu9) vivid; urgency=medium

  * Fix various ICMP handling issues in Squid pinger. (LP: #1384943)
 -- Jorge Niedbalski <email address hidden> Tue, 18 Nov 2014 14:47:33 -0300

[Marc Deslauriers (mdeslaur)]

This is a security issue that was assigned CVE-2014-7141 and CVE-2014-7142.

As such, it needs to be published in the security pocket.

I will build it as a security update, and will release it.

[Launchpad Janitor (janitor)]

This bug was fixed in the package squid3 - 3.3.8-1ubuntu6.2

---------------
squid3 (3.3.8-1ubuntu6.2) trusty-security; urgency=medium

  * SECURITY UPDATE: Fix various ICMP handling issues in Squid pinger.
    (LP: #1384943)
    - CVE-2014-7141
    - CVE-2014-7142
 -- Jorge Niedbalski <email address hidden> Tue, 18 Nov 2014 15:03:54 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package squid3 - 3.3.8-1ubuntu8.1

---------------
squid3 (3.3.8-1ubuntu8.1) utopic-security; urgency=medium

  * SECURITY UPDATE: Fix various ICMP handling issues in Squid pinger.
    (LP: #1384943)
    - CVE-2014-7141
    - CVE-2014-7142
 -- Jorge Niedbalski <email address hidden> Tue, 18 Nov 2014 14:47:33 -0300

[Marco Bettio (marco-bettio)]

I download 6.2 source package and recompile it on my ubuntu 14.04 trusty system but now pinger have some problems:
It seems that if pinger receive a Packet too big ICMP v6 packet while is pinging an ipv6 address run into strange errors, here is what I found in the log activating debug:

2014/12/17 11:06:20.182| cc(206) SendEcho: x=40
2014/12/17 11:06:20.182| c(116) Log: pingerLog: 1418810780.182442 [2a00:1450:4002:801::1016] 0
2014/12/17 11:06:20.204| cc(263) Recv: 40 bytes from [2a00:1450:4002:801::1016]
2014/12/17 11:06:20.204| c(116) Log: pingerLog: 1418810780.204457 [2a00:1450:4002:801::1016] 129 Echo Reply 22ms 1 hops
2014/12/17 11:06:20.204| nger.cc(235) SendResult: return result to squid. len=78
2014/12/17 11:06:20.577| cc(263) Recv: 282 bytes from [2001:1418:100:84df:1::1]
2014/12/17 11:06:20.578| cc(306) Recv: [2001:1418:100:84df:1::1] said: 2/0 Packet Too Big
*** Error in `(pinger)': munmap_chunk(): invalid pointer: 0x00007f5550efb260 ***

2014/12/17 11:17:24.223| nger.cc(213) Recv: Pass [2a00:1450:4002:801::1016] off to ICMPv6 module.
2014/12/17 11:17:24.223| cc(194) SendEcho: Send Icmp6 packet to [2a00:1450:4002:801::1016].
2014/12/17 11:17:24.223| cc(206) SendEcho: x=40
2014/12/17 11:17:24.224| c(116) Log: pingerLog: 1418811444.223975 [2a00:1450:4002:801::1016] 0
2014/12/17 11:17:24.246| cc(263) Recv: 40 bytes from [2a00:1450:4002:801::1016]
2014/12/17 11:17:24.246| c(116) Log: pingerLog: 1418811444.246856 [2a00:1450:4002:801::1016] 129 Echo Reply 23ms 1 hops
2014/12/17 11:17:24.247| nger.cc(235) SendResult: return result to squid. len=78
2014/12/17 11:17:24.618| cc(263) Recv: 282 bytes from [2001:1418:100:84df:1::1]
2014/12/17 11:17:24.618| cc(306) Recv: [2001:1418:100:84df:1::1] said: 2/0 Packet Too Big
*** Error in `(pinger)': free(): invalid size: 0x00007f32eb024260 ***

2014/12/17 11:19:27.987| nger.cc(213) Recv: Pass [2a00:1450:4002:801::1015] off to ICMPv6 module.
2014/12/17 11:19:27.987| cc(194) SendEcho: Send Icmp6 packet to [2a00:1450:4002:801::1015].
2014/12/17 11:19:27.987| cc(206) SendEcho: x=40
2014/12/17 11:19:27.987| c(116) Log: pingerLog: 1418811567.987412 [2a00:1450:4002:801::1015] 0
2014/12/17 11:19:28.004| cc(263) Recv: 282 bytes from [2001:1418:100:84df:1::1]
2014/12/17 11:19:28.004| cc(306) Recv: [2001:1418:100:84df:1::1] said: 2/0 Packet Too Big
*** Error in `(pinger)': double free or corruption (out): 0x00007f155f608260 ***

2014/12/17 11:20:14| recv: (111) Connection refused
2014/12/17 11:20:14| Closing Pinger socket on FD 28

after every one of this error pinger exit. If I restart squid I came into the error again since my ipv6 is a tunnel and i can get a lot of this ICMP packet too.

[Jorge Niedbalski (niedbalski)]

@marco-bettio

Could you upload the crash files ?

[Marco Bettio (marco-bettio)]

Hi Jorge,
I recreated the error so the crash report is related to the following log entry:

/var/log/squid3/cache.log

2014/12/18 14:09:24.128| nger.cc(213) Recv: Pass [2a00:1450:4002:801::1016] off to ICMPv6 module.
2014/12/18 14:09:24.128| cc(194) SendEcho: Send Icmp6 packet to [2a00:1450:4002:801::1016].
2014/12/18 14:09:24.128| cc(206) SendEcho: x=40
2014/12/18 14:09:24.128| c(116) Log: pingerLog: 1418908164.128595 [2a00:1450:4002:801::1016] 0
2014/12/18 14:09:24.162| cc(263) Recv: 40 bytes from [2a00:1450:4002:801::1016]
2014/12/18 14:09:24.162| c(116) Log: pingerLog: 1418908164.162822 [2a00:1450:4002:801::1016] 129 Echo Reply 34ms 1 hops
2014/12/18 14:09:24.162| nger.cc(235) SendResult: return result to squid. len=78
2014/12/18 14:09:24.596| cc(263) Recv: 282 bytes from [2001:1418:100:84df:1::1]
2014/12/18 14:09:24.596| cc(306) Recv: [2001:1418:100:84df:1::1] said: 2/0 Packet Too Big
*** Error in `(pinger)': free(): invalid size: 0x00007f4adde3e260 ***

/var/log/kern.log

Dec 18 07:05:07 srvproxy01 kernel: [155746.408903] pinger[21467]: segfault at 7f536c000000 ip 00007f536a5a38ac sp 00007fffa49c1c48 error 4 in libc-2.19.so[7f536a520000+1bb000]

and crash report attached

[Jorge Niedbalski (niedbalski)]

@marco-bettio,

Did you compiled this manually? Could you try with the archive package instead?

I can't get the debugging symbols for your crash

[Marco Bettio (marco-bettio)]

I installed the archive package instead and I have this error in the log:

2014/12/18 16:32:35.829| cc(263) Recv: 32 bytes from [fe80::206:4fff:fe9b:d135]
2014/12/18 16:32:40.831| cc(263) Recv: 24 bytes from [fe80::206:4fff:fe9b:d135]
2014/12/18 16:32:45.830| cc(263) Recv: 32 bytes from [fe80::206:4fff:fe9b:d135]
2014/12/18 16:32:50.458| cc(263) Recv: 282 bytes from [2001:1418:100:84df:1::1]
2014/12/18 16:32:50.458| cc(306) Recv: [2001:1418:100:84df:1::1] said: 2/0 Packet Too Big
*** Error in `(pinger)': munmap_chunk(): invalid pointer: 0x00007fdf07d5c260 ***
2014/12/18 16:38:25| recv: (111) Connection refused
2014/12/18 16:38:25| Closing Pinger socket on FD 18

no entry in kern.log this time

attached you can find crash file

[Jorge Niedbalski (niedbalski)]

@marco-bettio,

I am not being able to backtrace your submitted crash, Did you removed the hand-compiled version?
Could you check that are you using the latest archive package version squid_3.3.8-1ubuntu6.2_amd64 ?

Thanks.

[Marco Bettio (marco-bettio)]

Hi Jorge, I tried removing all and reinstallig all from repository.
the step i followed:

 2056 sudo apt-get remove squidclient
 2057 sudo apt-get remove squid3-common (this also remove squid3)
 2058 sudo apt-get clean squid3-common
 2059 sudo apt-get clean squid3
 2060 sudo apt-get clean squidclient
 2061 sudo apt-get install squid3 squid3-common squidclient

Lettura elenco dei pacchetti... Fatto
Generazione albero delle dipendenze
Lettura informazioni sullo stato... Fatto
I seguenti pacchetti sono stati installati automaticamente e non sono più richiesti:
  linux-headers-3.13.0-40 linux-headers-3.13.0-40-generic
  linux-image-3.13.0-40-generic linux-image-extra-3.13.0-40-generic
Usare "apt-get autoremove" per rimuoverli.
Pacchetti suggeriti:
  squid-cgi squid-purge smbclient winbindd
I seguenti pacchetti NUOVI saranno installati:
  squid3 squid3-common squidclient
0 aggiornati, 3 installati, 0 da rimuovere e 1 non aggiornati.
È necessario scaricare 1.970 kB di archivi.
Dopo quest'operazione, verranno occupati 6.679 kB di spazio su disco.
Scaricamento di:1 http://it.archive.ubuntu.com/ubuntu/ trusty-updates/main squid3-common all 3.3.8-1ubuntu6.2 [153 kB]
Scaricamento di:2 http://it.archive.ubuntu.com/ubuntu/ trusty-updates/main squid3 amd64 3.3.8-1ubuntu6.2 [1.787 kB]
Scaricamento di:3 http://it.archive.ubuntu.com/ubuntu/ trusty-updates/universe squidclient amd64 3.3.8-1ubuntu6.2 [31,1 kB]
Recuperati 1.970 kB in 4s (441 kB/s)
Selezionato il pacchetto squid3-common non precedentemente selezionato.
(Lettura del database... 101279 file e directory attualmente installati.)
Preparing to unpack .../squid3-common_3.3.8-1ubuntu6.2_all.deb ...
Unpacking squid3-common (3.3.8-1ubuntu6.2) ...
Selezionato il pacchetto squid3 non precedentemente selezionato.
Preparing to unpack .../squid3_3.3.8-1ubuntu6.2_amd64.deb ...
Unpacking squid3 (3.3.8-1ubuntu6.2) ...
Selezionato il pacchetto squidclient non precedentemente selezionato.
Preparing to unpack .../squidclient_3.3.8-1ubuntu6.2_amd64.deb ...
Unpacking squidclient (3.3.8-1ubuntu6.2) ...
Processing triggers for ufw (0.34~rc-0ubuntu2) ...
Processing triggers for ureadahead (0.100.0-16) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Configurazione di squid3-common (3.3.8-1ubuntu6.2)...
Configurazione di squid3 (3.3.8-1ubuntu6.2)...
squid3 start/running, process 8150
Skipping profile in /etc/apparmor.d/disable: usr.sbin.squid3
Configurazione di squidclient (3.3.8-1ubuntu6.2)...

/var/log/squid3/cache.log

2014/12/19 13:38:07.606| cc(263) Recv: 282 bytes from [2001:1418:100:84df:1::1]
2014/12/19 13:38:07.606| cc(306) Recv: [2001:1418:100:84df:1::1] said: 2/0 Packet Too Big
*** Error in `(pinger)': munmap_chunk(): invalid pointer: 0x00007f430603a260 ***

crash report attached.

Hope this time we had better luck with this .

[Marco Bettio (marco-bettio)]

Hi Jorge,
sorry but I'm not expert, but reading documentation, maybe i have to prepare crash report before sending it:

here you can find a crash report prepared with
apport-cli -c "/var/crash/_usr_lib_squid3_pinger.0.crash"

hope this help.

[Tuomas Heino (iheino+ub)]

Another similar-looking crash file (3.3.8-1ubuntu6.2 as well) in case it could be useful.

[Marco Bettio (marco-bettio)]

Hi Jorge,
updated Linux kernel to 3.13.0-45-generic which fixes some ipv6 related problems,
but pinger crash is still there.

Here attached the crash file collected during last crash with the new kernel.

Thanks in advance.

[mr_willem (willem-crossbone)]

Hello,

I can confirm that the bug is still present.
Allthough I updated the squid package there is often high cpu load of the pinger process and it often segfaults.

Thus there is still a bug present.
It is also in squid 3.4.9 which I installed from source.

Thanks, Willem

[Jorge Niedbalski (niedbalski)]

Hello,

Does anybody is able to upload a minimal configuration file that
reproduces the issue?

Thanks

[mr_willem (willem-crossbone)]

I use such a configuration
I deleted all lines which were commented out to shorten it here a little bit.

http_port 3128
http_port 192.168.0.100:8080 transparent
cache_mem 8 MB
maximum_object_size 4096 KB
cache_dir ufs /mnt/squid/cache 640 20 256
cache_access_log /var/log/squid/access.log
cache_log /var/log/squid/cache.log
cache_store_log /var/log/squid/store.log
mime_table /usr/share/squid3/mime.conf
pid_filename /var/run/squid.pid
debug_options ALL,1
ftp_user anonymous
unlinkd_program /usr/lib/squid3/unlinkd
redirect_children 15
refresh_pattern . 0 20% 4320
negative_ttl 5 minutes
positive_dns_ttl 1 hours
negative_dns_ttl 5 minutes
connect_timeout 120 seconds
read_timeout 15 minutes
request_timeout 30 seconds
half_closed_clients on
pconn_timeout 120 seconds
acl all src all
acl mypcs src 192.168.100.0-192.168.0.109
acl clients src 192.168.0.110-192.168.0.160
acl localhost src 127.0.0.1/32
http_access deny manager !mypcs
http_access allow mypcs
http_access allow clients
always_direct allow all
icp_access allow all
cache_mgr <email address hidden>
cache_effective_user proxy
visible_hostname mypc.mydomain.org
forwarded_for on
cachemgr_passwd all mypassword
request_header_max_size 30 KB

[mr_willem (willem-crossbone)]

Sorry, there was a typo with the IP adresses,
of course it is not
... acl mypcs src 192.168.100.0 but ... acl mypcs src 192.168.0.0

[felixrising (matt-mu)]

Still not fixed?

root@Tank1:/var/crash# grep pinger /var/log/kern.log
Nov 2 06:29:26 Tank1 kernel: [3082595.625197] pinger[9259]: segfault at 242bfa760098 ip 00007fa4a621d845 sp 00007ffebcbeae70 error 4 in libc-2.19.so[7fa4a619f000+1bb000]
Nov 3 06:49:04 Tank1 kernel: [3170130.901619] pinger[13216]: segfault at 33a09a660098 ip 00007fe9639c0845 sp 00007ffdbad15e90 error 4 in libc-2.19.so[7fe963942000+1bb000]
Nov 4 06:31:29 Tank1 kernel: [3255433.287926] pinger[17673]: segfault at 0 ip 00007f15a54d40ca sp 00007ffd0a580658 error 4 in libc-2.19.so[7f15a544d000+1bb000]
Nov 5 06:35:59 Tank1 kernel: [3342060.820864] pinger[24862]: segfault at 0 ip 00007f4ca8c880ca sp 00007ffd4f2bc2a8 error 4 in libc-2.19.so[7f4ca8c01000+1bb000]
Nov 6 06:43:01 Tank1 kernel: [3428839.839449] pinger[15340]: segfault at 0 ip 00007fd7b4e6f0ca sp 00007ffdf1f26288 error 4 in libc-2.19.so[7fd7b4de8000+1bb000]
Nov 7 06:47:30 Tank1 kernel: [3515466.403573] pinger[20821]: segfault at 0 ip 00007f26aaead0ca sp 00007ffe1081d828 error 4 in libc-2.19.so[7f26aae26000+1bb000]
Nov 8 06:47:49 Tank1 kernel: [3601842.427069] pinger[28592]: segfault at 0 ip 00007f7932e6f0ca sp 00007ffef4cb3a08 error 4 in libc-2.19.so[7f7932de8000+1bb000]

[Pawel Kraszewski (linuxpedia.pl)]

Still not fixed

*** Error in `(pinger)': munmap_chunk(): invalid pointer: 0x000055e3af07c0b0 ***
*** Error in `(pinger)': malloc(): memory corruption: 0x000055e3af07bfe0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x778bf)[0x7f6592a7b8bf]
/lib/x86_64-linux-gnu/libc.so.6(+0x81b5e)[0x7f6592a85b5e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7f6592a87744]
/lib64/ld-linux-x86-64.so.2(+0xe4e7)[0x7f65933744e7]
/lib64/ld-linux-x86-64.so.2(+0x15332)[0x7f659337b332]
/lib64/ld-linux-x86-64.so.2(+0x10334)[0x7f6593376334]
/lib64/ld-linux-x86-64.so.2(+0x14af9)[0x7f659337aaf9]
/lib/x86_64-linux-gnu/libc.so.6(+0x1436cd)[0x7f6592b476cd]
/lib64/ld-linux-x86-64.so.2(+0x10334)[0x7f6593376334]
/lib/x86_64-linux-gnu/libc.so.6(__libc_dlopen_mode+0x54)[0x7f6592b47784]
/lib/x86_64-linux-gnu/libc.so.6(backtrace+0xc1)[0x7f6592b199f1]
/lib/x86_64-linux-gnu/libc.so.6(+0x1f9fa)[0x7f6592a239fa]
/lib/x86_64-linux-gnu/libc.so.6(+0x778bf)[0x7f6592a7b8bf]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x1a8)[0x7f6592a87db8]
(pinger)(+0xbbea)[0x55e3adfe5bea]
(pinger)(+0x8668)[0x55e3adfe2668]
(pinger)(+0x50e1)[0x55e3adfdf0e1]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f6592a24841]
(pinger)(+0x5b19)[0x55e3adfdfb19]
======= Memory map: ========
55e3adfda000-55e3adfec000 r-xp 00000000 fc:00 3051330 /usr/lib/squid/pinger
55e3ae1eb000-55e3ae1ec000 r--p 00011000 fc:00 3051330 /usr/lib/squid/pinger
55e3ae1ec000-55e3ae1ed000 rw-p 00012000 fc:00 3051330 /usr/lib/squid/pinger
55e3ae1ed000-55e3ae201000 rw-p 00000000 00:00 0
55e3af06a000-55e3af09c000 rw-p 00000000 00:00 0 [heap]
7f658c000000-7f658c021000 rw-p 00000000 00:00 0
7f658c021000-7f6590000000 ---p 00000000 00:00 0
7f65926fb000-7f6592803000 r-xp 00000000 fc:00 1708669 /lib/x86_64-linux-gnu/libm-2.23.so
7f6592803000-7f6592a02000 ---p 00108000 fc:00 1708669 /lib/x86_64-linux-gnu/libm-2.23.so
7f6592a02000-7f6592a03000 r--p 00107000 fc:00 1708669 /lib/x86_64-linux-gnu/libm-2.23.so
7f6592a03000-7f6592a04000 rw-p 00108000 fc:00 1708669 /lib/x86_64-linux-gnu/libm-2.23.so
7f6592a04000-7f6592bc4000 r-xp 00000000 fc:00 1708605 /lib/x86_64-linux-gnu/libc-2.23.so
7f6592bc4000-7f6592dc4000 ---p 001c0000 fc:00 1708605 /lib/x86_64-linux-gnu/libc-2.23.so
7f6592dc4000-7f6592dc8000 r--p 001c0000 fc:00 1708605 /lib/x86_64-linux-gnu/libc-2.23.so
7f6592dc8000-7f6592dca000 rw-p 001c4000 fc:00 1708605 /lib/x86_64-linux-gnu/libc-2.23.so
7f6592dca000-7f6592dce000 rw-p 00000000 00:00 0
7f6592dce000-7f6592de4000 r-xp 00000000 fc:00 1708638 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f6592de4000-7f6592fe3000 ---p 00016000 fc:00 1708638 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f6592fe3000-7f6592fe4000 rw-p 00015000 fc:00 1708638 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f6592fe4000-7f6593156000 r-xp 00000000 fc:00 3016716 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f6593156000-7f6593356000 ---p 00172000 fc:00 3016716 ...