PostgreSQL's request for a shared memory segment exceeded your kernel's SHMMAX parameter

[Steve Martinelli]

All the logs: http://logs.openstack.org/53/113553/2/check/check-tempest-dsvm-postgres-full/271cc6b/

Full devstack log: http://logs.openstack.org/53/113553/2/check/check-tempest-dsvm-postgres-full/271cc6b/logs/devstacklog.txt.gz

snippet:
2014-08-12 17:16:59.790 | + restart_service postgresql
2014-08-12 17:16:59.790 | + is_ubuntu
2014-08-12 17:16:59.790 | + [[ -z deb ]]
2014-08-12 17:16:59.790 | + '[' deb = deb ']'
2014-08-12 17:16:59.790 | + sudo /usr/sbin/service postgresql restart
2014-08-12 17:16:59.800 | * Restarting PostgreSQL 9.1 database server
2014-08-12 17:17:01.371 | * The PostgreSQL server failed to start. Please check the log output:
2014-08-12 17:17:01.371 | 2014-08-12 17:17:01 UTC FATAL: could not create shared memory segment: Invalid argument
2014-08-12 17:17:01.371 | 2014-08-12 17:17:01 UTC DETAIL: Failed system call was shmget(key=5432001, size=36954112, 03600).
2014-08-12 17:17:01.371 | 2014-08-12 17:17:01 UTC HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMMAX parameter. You can either reduce the request size or reconfigure the kernel with larger SHMMAX. To reduce the request size (currently 36954112 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections.
2014-08-12 17:17:01.371 | If the request size is already small, it's possible that it is less than your kernel's SHMMIN parameter, in which case raising the request size or reconfiguring SHMMIN is called for.
2014-08-12 17:17:01.371 | The PostgreSQL documentation contains more information about shared memory configuration.
2014-08-12 17:17:01.371 | ...fail!
2014-08-12 17:17:01.372 | + exit_trap

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/113634

[Jeremy Stanley (fungi)]

According to logstash this started just after the daily nodepool image updates completed, and is only happening on devstack-trusty nodes in hpcloud-b1. I have deleted that image and held one of the nodes booted from it (which reports 33554432 from /proc/sys/kernel/shmmax just like nodes booted from the previous day's image). I've gone over the image update log, which didn't contain any obvious configuration/setup errors, but will compare it to an earlier run and other hpcloud provider network versions of the same to see if there's a relevant difference.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/113678
Committed: https://git.openstack.org/cgit/openstack-infra/config/commit/?id=2fa821d2ce4b4f655a130668edefb3a663611202
Submitter: Jenkins
Branch: master

commit 2fa821d2ce4b4f655a130668edefb3a663611202
Author: K Jonathan Harker <email address hidden>
Date: Tue Aug 12 16:20:36 2014 -0700

    Accidentally turned a trusty image into precise

    Closes-Bug: #1355998
    Change-Id: Ic585555d48c374f0441615ba95f416ad5f7f8e37

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/113634
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=d0cd62c9c3eaab7f9b1e2131f8d4f792cacf530b
Submitter: Jenkins
Branch: master

commit d0cd62c9c3eaab7f9b1e2131f8d4f792cacf530b
Author: Davanum Srinivas <email address hidden>
Date: Tue Aug 12 16:07:02 2014 -0400

    Add query for Postgres startup failure

    24 hits in last 24 hours

    Related-Bug: 1355998

    Change-Id: If4b364af11a85ec0ee94a7cae05f6c77fdde0722