Remove all logfiles from containers

[Bjoern]

Please make all containers more immutable and remove all logfiles from the container root partition, ideally there is not even rsyslog running inside the container.
The goal is to prevent containers from filling up. Otherwise we would have to add disk monitoring for each container which I would like to prevent if possible, just because of the volatile state of most containers

[Jesse Pretorius (jesse-pretorius)]

Setting this as a wishlist item as this is an enhancement request.

[Jesse Pretorius (jesse-pretorius)]

Note that the phrasing of this bug report is requesting a specific solution to solve a specific problem. There are many solutions to solve the same problem and whether the requested solution is the best one is entirely debatable.

The specific solution requested is a fairly significant architecture change which would need to be appropriately planned through the blueprint/spec process.

If alternative solutions are an option as either a tactical solution or perhaps also a long term solution, then it is recommended that the bug report title and content be changed to indicate what the issue is at hand and being less prescriptive about the solutions that could be implemented.

[Bjoern (bjoern-t)]

This is by far not a specific request since it just requests to turn off logging inside the container root filesystem. Filling up neutron agent container is not a specific problem. All OSA operations will have this issue sooner than later, simple as that.

On purpose I did not sketch and solution to leave it up to the assignee. Not implementing this bug will cause continuous operational issues. We already started to write openstack logfiles into a bind mount because of the issues described so I don't think it will be too difficult to proceed going this route.

[Kevin Carter (kevin-carter)]

Removing the logging all together within the containers for various services is a bad idea and not something that I'd recommend. Additionally, all of the logs are being shipped through the use or rsyslog within all containers and hosts. This gives direct log access to a deployer in a centralized location.Part of the rsyslog_client role is to also setup log rotation for all logs that its shipping. Again this is a pattern that I'm recommending we keep as its proven to be quite efficient and beneficial to large scale deployments. Now we might be able to figure out a better way to store the log (bind mount or otherwise) that would remove the issues with a container FS filling up. Note that this issue is unique to having a container backend type that is limits storage, IE BTRFS, LVM, ect... That said, OSA already can support deployments using the host rootfs and if this is an issue that is causing pain currently it may be best to start by enabling that capability or by making the logical volumes larger for the various containers already in service. Note that there are pros and cons for just about any approach however I'd be nice to have a real time conversation regarding what works best for the majority of deployers so that we can come up with the best solution.

[Andre Ruffin (andre-ruffin)]

A real time conversation would be great.

It seems that increasing the size of the logical volume within the container would only delay the file system filling up with logs.

However, perhaps we should entertain the idea of standardizing the existing bind mount within the containers to be /var/log to mitigate the issues we have seen as of late.

This would respect the benefits of centralized logging as well.

[Major Hayden (rackerhacker)]

Could this be brought up in the regular OSA meeting?

  https://wiki.openstack.org/wiki/Meetings/openstack-ansible

They are held every Thursday at 16:00 UTC.

[Bjoern (bjoern-t)]

Yes and we discussed that in the past already. Personally I prefer not to have and accumulating log information in tiny small container root disks and proposed to move all log information to syslog like we already do, without storing it locally.

[Bjoern (bjoern-t)]

Closing in favor of https://bugs.launchpad.net/openstack-ansible/+bug/1588051