wal-e no longer works as of snapd 2.37

Bug #1814484 reported by Paul Collins
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
PostgreSQL Charm
Fix Released
Critical
Unassigned
snapd
Fix Released
Undecided
Unassigned

Bug Description

As of snapd 2.37, wal-e is failing as follows:

cannot create user data directory: /var/lib/postgresql/snap/wal-e/13: Permission denied
2019-02-04 00:45:23 UTC [4636]: [50860-1] db=,user= LOG: archive command failed with exit code 1
2019-02-04 00:45:23 UTC [4636]: [50861-1] db=,user= DETAIL: The failed archive command was: /snap/bin/wal-e.envdir /etc/postgresql/9.5/main/wal-e.env /snap/bin/wal-e wal-push pg_xlog/00000001000003CF0000004D
2019-02-04 00:45:23 UTC [4636]: [50862-1] db=,user= WARNING: archiving transaction log file "00000001000003CF0000004D" failed too many times, will try again later

This is because snap-confine removed a special case that accidentally allowed home directories to live under /var/lib: https://github.com/snapcore/snapd/pull/6446

I've been working around this by downgrading the core snap on affected units, but this doesn't scale well, and will be reverted the next time core is updated.

Tags: canonical-is
Stuart Bishop (stub)
Changed in postgresql-charm:
status: New → Confirmed
importance: Undecided → Critical
Stuart Bishop (stub)
tags: added: canonical-is
Revision history for this message
Jason Hobbs (jason-hobbs) wrote :

I think we are hitting this also.

Feb 4 18:38:38 juju-c811b4-prod-cdo-oil-8 kernel: [ 2452.008965] audit: type=1400 audit(1549305518.994:676): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="/snap/core/6350/usr/lib/snapd/snap-confine" name="/tmp/snap.rootfs_mpbcyx/var/lib/jenkins/" pid=20203 comm="snap-confine" srcname="/srv/volume/var/lib/jenkins/" flags="rw, rbind"

Revision history for this message
Stuart Bishop (stub) wrote :

snapd are working on a fix. There isn't much we can do in the charm or in the wal-e snap. You will need to disable WAL shipping in the charm config temporarily if your disk is in danger of filling.

Revision history for this message
Stuart Bishop (stub) wrote :
Changed in snapd:
status: New → Fix Committed
Stuart Bishop (stub)
Changed in snapd:
status: Fix Committed → Fix Released
Changed in postgresql-charm:
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.