Exceeding max connections causes update-status hook error, which can block mojo deploys
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
PostgreSQL Charm |
Triaged
|
Medium
|
Unassigned |
Bug Description
Ran into a weird situation earlier today where a problem in our deployment caused the max allowed database connections to be exceeded. The cause was the rate limit on our app's charm was not configured, which would normally protect the database from getting thrashed by a spike in traffic. (Because this traffic is non-interactive messaging, that's ok)
So we tried to roll out a mojo update to address the issue, but couldn't, because the postgresql charm was in an error state. It seemed to be in an error state because of the connection issue:
2019-01-09 15:02:56 DEBUG update-status Traceback (most recent call last):
2019-01-09 15:02:56 DEBUG update-status File "/var/lib/
2019-01-09 15:02:56 DEBUG update-status main()
2019-01-09 15:02:56 DEBUG update-status File "/usr/local/
2019-01-09 15:02:56 DEBUG update-status bus.dispatch()
2019-01-09 15:02:56 DEBUG update-status File "/usr/local/
2019-01-09 15:02:56 DEBUG update-status _invoke(
2019-01-09 15:02:56 DEBUG update-status File "/usr/local/
2019-01-09 15:02:56 DEBUG update-status handler.invoke()
2019-01-09 15:02:56 DEBUG update-status File "/usr/local/
2019-01-09 15:02:56 DEBUG update-status self._action(*args)
2019-01-09 15:02:56 DEBUG update-status File "/var/lib/
2019-01-09 15:02:56 DEBUG update-status ensure_
2019-01-09 15:02:56 DEBUG update-status File "/usr/local/
2019-01-09 15:02:56 DEBUG update-status return func(*args, **kwargs)
2019-01-09 15:02:56 DEBUG update-status File "/var/lib/
2019-01-09 15:02:56 DEBUG update-status con = postgresql.
2019-01-09 15:02:56 DEBUG update-status File "/var/lib/
2019-01-09 15:02:56 DEBUG update-status host=host, port=port_)
2019-01-09 15:02:56 DEBUG update-status File "/usr/lib/
2019-01-09 15:02:56 DEBUG update-status conn = _connect(dsn, connection_
2019-01-09 15:02:56 DEBUG update-status psycopg2.
We were able to work around this by temporarily downing the app server.
summary: |
- Exceeding max connections causes update-status hook to fail, which can + Exceeding max connections causes update-status hook error, which can block mojo deploys |
If regular DB connections are being used, the default setting of superuser_ reserved_ connections= 3 should avoid this.
If superuser connections are being used, we can't avoid running out of slots.
Need to investigate if running out of slots can be handled better.
Per user connection limits might help, where the clients are using the provided credentials.