Comment 2 for bug 1383741

Revision history for this message
Alexander Bozhenko (alexbozhenko) wrote :

This issue can be reproduced as Evgeny reported.
 "fallocate -l <big number> /var/BIG_FILE" - to fill all the space on /var
After running the command try different actions in fuel: provision new nodes, change configs, etc...
After that reboot, clear the space, reboot again.

I have seen several options after these actions:

1)Corrupted index on postgresql database, can be fixed with something like this
PGPASSWORD=nailgun dockerctl shell postgres su - postgres -c "psql keystone -c 'reindex DATABASE keystone;'"

2)Container that can't start. In case it is stateless container, can be fixed with
dockerctl destroy keystone
dockerctl start keystone

3)Stateful container that can't start. Don't know how to fix this.

In fuel 5.1 (and 6.0 preview) version of docker is:
docker-io-0.10.0-2.mira2.x86_64
but in epel repo there is newer available:
docker-io-1.2.0-3.el6.x86_64

Having this comments from docker developer
https://github.com/docker/docker/issues/6368#issuecomment-46201330
https://github.com/docker/docker/issues/7229#issuecomment-60939980
don't we have a plan to update kernel to recomended 2.6.32-504(shipped with centos 6.6), and update docker package?

Usually /var quickly filled with logs and fuel snapshots.
As workaround, can we put /var/lib/docker/ to separate partition? This will prevent docker images from corruption.