crash importing large database dump into maas-test-db

Bug #2020427 reported by Jacopo Rota
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Fix Committed
Medium
Jacopo Rota

Bug Description

As per summary, here the steps to reproduce:

1) generate a large database with `SAMPLEDATA_MACHINES=250 make sampledata`
2) `sudo cp maasdb.dump /var/snap/maas-test-db/common`
3) `sudo snap run --shell maas-test-db.psql -c 'db-dump restore $SNAP_COMMON/maasdb.dump maassampledata'` -> crash

Here's the error
pg_restore: while PROCESSING TOC:
pg_restore: from TOC entry 4145; 2606 19907 CONSTRAINT maasserver_event maasserver_event_pkey ubuntu
pg_restore: error: could not execute query: ERROR: could not open shared memory segment "/PostgreSQL.1198490976": Permission denied
Command was: ALTER TABLE ONLY public.maasserver_event
    ADD CONSTRAINT maasserver_event_pkey PRIMARY KEY (id);
4:52 PM

The sample database dump (with 1000 machines) can be found in the attachments.

Revision history for this message
Jacopo Rota (r00ta) wrote :
Revision history for this message
Jacopo Rota (r00ta) wrote :

The root cause is that `postgres` creates child processes and they do not have the `SNAP_INSTANCE_NAME` in the stack (see https://github.com/maas/snap-preload/blob/753db287d2980eaae0400076ab1d7a81c188b478/snap-preload.c#L79). We use LD_PRELOAD to change the `shm_path` at runtime for the snap https://github.com/maas/snap-preload/blob/main/snap-preload.c .

Debugging the issue, it is possible to see that the new path is constructed like SNAP_INSTANCE_NAME is not set
```
[pid 49217] openat(AT_FDCWD, "/dev/shm/snap..PostgreSQL.946200062", O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_CLOEXEC, 0600) = -1 EACCES (Permission denied)
```

The LD_PRELOAD library is loaded properly by the child processes
```
(gdb) info sharedlibrary
From To Syms Read Shared Object Library
0x00007ff232e3e180 0x00007ff232e3e4da Yes (*) target:/snap/maas-test-db/131/usr/lib/snap-preload.so
```

A possible fix is to store the result of `secure_getenv("SNAP_INSTANCE_NAME");` in the heap instead of in the stack, because the heap is copied properly instead of the stack.

Revision history for this message
Jacopo Rota (r00ta) wrote :
Changed in maas:
assignee: nobody → Jacopo Rota (r00ta)
Changed in maas:
status: Triaged → In Progress
Jacopo Rota (r00ta)
Changed in maas:
status: In Progress → Fix Committed
milestone: none → 3.4.0
importance: Undecided → Medium
Alberto Donato (ack)
Changed in maas:
milestone: 3.4.0 → 3.4.0-beta2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.