[Impact]
* Users that are members of large numbers of groups are unable to run procps commands, but instead are greated with a segfault or "failed to allocate XXXXXXXXX bytes of memory"
* Many enterprise or University users create large numbers of groups in order to effectively manage access in their infrastructure. Often admins end up being added to most of these groups. Admins end up not being able to use procps commands as a result
* before this update procps statically allocated a buffer of 1024 bytes for reading all of /proc/#/status. This file includes the list of gid's that a user is part of. The attached patches backport a45dace4 and 95d01362 from upstream procps-ng which modifies file2str to use a dynamically allocated buffer instead.
[Test Case]
* sudo useradd bob; for i in {1..800}; do sudo groupadd group$i; sudo adduser bob group$i; done; sudo -u bob ps
* The above command returns failed to allocate XXXXXXXXXXX bytes of memory
[Regression Potential]
* Regressions are most likely to manifest in corruption of data that is available from the /proc/<pid>/{stat,statm,status} files. Additionally it is theoretically possible that non-returning proc commands could also be possible.
* The above being said, I feel as though regression potential is fairly minimal as this is mostly a backport of upstream commits.
* This fix is now available as part of this ppa https://launchpad.net/~chiluk/+archive/lp1150413 where the fix has already been verified by users on this thread.
[Other Info]
* procps is a horrible no-good very bad codebase, and we should upgrade to procps-ng as soon as possible. Available https://gitorious.org/procps. Preferably before the next LTS.
----------------------------------Snip--------------------------------------
Both ps and pgrep exit with an error like "xrealloc: realloc(1073741824) failedCannot allocate memory" if there is a process owned by a user with a large number of groups.
I suspect this was introduced with a recent kernel patch which no longer limits the number of groups returned by /proc/<pid>/status to 32. It affects Ubuntu kernel 3.2.0-38.60 and newer.
Links
* related kernel bug report - https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1132789
* kernel patch - https://lists.ubuntu.com/archives/kernel-team/2013-January/024446.html
* kernel 3.2.0-38.60 changelog - https://launchpad.net/ubuntu/precise/+source/linux/3.2.0-38.60
strace ps output where it fails when it gets to the process owned by a user with a large number of groups
open("/proc/11860/status", O_RDONLY) = 6
read(6, "Name:\tsu\nState:\tS (sleeping)\nTgid:\t11860\nPid:\t11860\nPPid:\t7912\nTracerPid:\t0\nUid:\t1001\t1001\t1001\t1001\nGid:\t1351\t1351\t1351\t1351\nFDSize:\t256\nGroups:\t1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 13", 1023) = 1023
close(6) = 0
mmap(NULL, 135168, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fbcc1e4c000
mremap(0x7fbcc1e4c000, 135168, 266240, MREMAP_MAYMOVE) = 0x7fbcc1e0b000
mremap(0x7fbcc1e0b000, 266240, 528384, MREMAP_MAYMOVE) = 0x7fbcc28cf000
mremap(0x7fbcc28cf000, 528384, 1052672, MREMAP_MAYMOVE) = 0x7fbcc27ce000
mremap(0x7fbcc27ce000, 1052672, 2101248, MREMAP_MAYMOVE) = 0x7fbcc1c6c000
mremap(0x7fbcc1c6c000, 2101248, 4198400, MREMAP_MAYMOVE) = 0x7fbcc186b000
mremap(0x7fbcc186b000, 4198400, 8392704, MREMAP_MAYMOVE) = 0x7fbcc106a000
mremap(0x7fbcc106a000, 8392704, 16781312, MREMAP_MAYMOVE) = 0x7fbcc0069000
mremap(0x7fbcc0069000, 16781312, 33558528, MREMAP_MAYMOVE) = 0x7fbcbe068000
mremap(0x7fbcbe068000, 33558528, 67112960, MREMAP_MAYMOVE) = 0x7fbcba067000
mremap(0x7fbcba067000, 67112960, 134221824, MREMAP_MAYMOVE) = 0x7fbcb2066000
mremap(0x7fbcb2066000, 134221824, 268439552, MREMAP_MAYMOVE) = 0x7fbca2065000
mremap(0x7fbca2065000, 268439552, 536875008, MREMAP_MAYMOVE) = 0x7fbc82064000
mremap(0x7fbc82064000, 536875008, 1073745920, MREMAP_MAYMOVE) = -1 EFAULT (Bad address)
mmap(NULL, 1073745920, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
brk(0x4133c000) = 0x1333000
mmap(NULL, 1073876992, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
open("/sys/devices/system/cpu/online", O_RDONLY|O_CLOEXEC) = 6
read(6, "0\n", 8192) = 2
close(6) = 0
mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7fbbfa042000
munmap(0x7fbbfa042000, 33284096) = 0
munmap(0x7fbc00000000, 33824768) = 0
mprotect(0x7fbbfc000000, 135168, PROT_READ|PROT_WRITE) = 0
mmap(NULL, 1073745920, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
write(2, "xrealloc: realloc(1073741824) failed", 36xrealloc: realloc(1073741824) failed) = 36
write(2, "Cannot allocate memory\n", 23Cannot allocate memory
) = 23
exit_group(1) = ?
Steps I was able to use to reproduce the problem with all local users and groups. The number of groups needed to break ps may be different on other systems.
root@alowther-d02:~# for i in $(seq 180); do groupadd group$i ; done
root@alowther-d02:~# useradd user1
root@alowther-d02:~# su - user1 -c ps
No directory, logging in with HOME=/
PID TTY TIME CMD
5182 pts/0 00:00:00 su
5183 pts/0 00:00:00 sh
5185 pts/0 00:00:00 ps
root@alowther-d02:~# for i in $(seq 180); do adduser user1 group$i; done > /dev/null
root@alowther-d02:~# su - user1 -c ps
xrealloc: realloc(1073741824) failedCannot allocate memory
System info - I also tried using the version of procps from Raring, but it still failed
root@alowther-d02:~# lsb_release -rd
Description: Ubuntu 12.04.2 LTS
Release: 12.04
root@alowther-d02:~# uname -a
Linux alowther-d02 3.2.0-38-generic #61-Ubuntu SMP Tue Feb 19 12:18:21 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
root@alowther-d02:~# dpkg -l procps
ii procps 1:3.2.8-11ubuntu6 /proc file system utilities
root@alowther-d02:~# apt-cache policy procps
procps:
Installed: 1:3.2.8-11ubuntu6
Candidate: 1:3.3.3-2ubuntu5
Version table:
1:3.3.3-2ubuntu5 0
500 http://aptproxy/ubuntu/ raring/main amd64 Packages
*** 1:3.2.8-11ubuntu6 0
500 http://aptproxy/ubuntu/ precise/main amd64 Packages
100 /var/lib/dpkg/status
I'm not sure the best way to fix this, but I have located the problem.
I've found this bug to be caused by a Debian patch "ps_supgid_
When any /proc/<pid>/status has more than 1024 characters before the end of the Groups line, an infinite loop is entered because the loop never finds the "\n" character it expected. Looking back at the strace, the read of "/proc/
stat("/proc/1110", {st_mode=
open("/
read(6, "1110 (npcd) S 1 1108 1108 0 -1 4202816 509903 56395239 0 129102 205 836 161563 37679 20 0 1 0 3191 243957760 263 184467440737095
While the loop runs, it keeps allocating more memory for the array of Group IDs that it has found associated with the process until it fails because there isn't enough memory to allocate.
The while loop below from the patch is what runs infinitely. The xrealloc allocates more memory, doubling the amount it wants each time the loop thinks the array is filled.
+ case_Groups:
+ isupgid = 0;
+ if (*S != '\n'){ // Is there any supplementary group ?
+ P->supgid = (int *) xmalloc(0x0004 * sizeof(int));
+ int vctsize = 0x0004;
+ while (S[1] != '\n' && isupgid<INT_MAX){ // There is one blank before '\n'
+ if (isupgid == vctsize){
+ vctsize *= 2;
+ P->supgid = (int *)xrealloc(
+ }
+ P->supgid[
+ P->nsupgid++;
+ }
+ }
+ continue;