[regression] deja-dup-monitor crashed with SIGSEGV in Gigacage::<lambda()>::operator()

[starkus]

Impact
------
webkit2gtk 2.20 adds a new security feature called the Gigacage that uses an extremely large virtual memory address space (much larger than available physical memory).

Deja Dup's monitor background service had "ulimit -v 1000000" (that's 1 GB) set as a workaround for a memory leak issue that the developer was unable to reproduce.

After upgrading to the new webkit2gtk version, Deja Dup's monitor service will crash because of that virtual memory limit.

Test Case
---------
Install the deja-dup update.
Install the webkit2gtk update from a PPA (not prepared yet).
Log out. Log in.
After a few minutes, check /var/crash/ for any Deja Dup crash reports.

Regression Potential
--------------------
This could reintroduce the memory leak bug, but otherwise this is a minimal fix. Even if that happens, it's better than the service refusing to run.

Other Info
----------
https://errors.ubuntu.com/problem/27441b78823246dd5392ee29ac30546f6464289e

ProblemType: Crash
DistroRelease: Ubuntu 18.04
Package: deja-dup 37.1-1fakesync1
ProcVersionSignature: Ubuntu 4.15.0-10.11-generic 4.15.3
Uname: Linux 4.15.0-10-generic x86_64
ApportVersion: 2.20.8-0ubuntu10
Architecture: amd64
CrashCounter: 1
CurrentDesktop: GNOME
Date: Sat Feb 24 14:30:47 2018
ExecutablePath: /usr/lib/deja-dup/deja-dup-monitor
InstallationDate: Installed on 2017-12-27 (59 days ago)
InstallationMedia: Ubuntu 17.10 "Artful Aardvark" - Release amd64 (20171018)
ProcCmdline: /usr/lib/deja-dup/deja-dup-monitor
ProcEnviron:
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=de_DE.UTF-8
 SHELL=/bin/bash
SegvAnalysis:
 Segfault happened at: 0x7ff1c3dda588: movl $0x0,(%rax)
 PC (0x7ff1c3dda588) ok
 source "$0x0" ok
 destination "(%rax)" (0xbbadbeef) not located in a known VMA region (needed writable region)!
SegvReason: writing unknown VMA
Signal: 11
SourcePackage: deja-dup
StacktraceTop:
 ?? () from /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
 __pthread_once_slow (once_control=0x7ff1c404202c, init_routine=0x7ff1baec0490 <__once_proxy>) at pthread_once.c:116
 Gigacage::ensureGigacage() () from /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
 bmalloc::Heap::Heap(bmalloc::HeapKind, std::lock_guard<bmalloc::StaticMutex>&) () from /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
 bmalloc::PerProcess<bmalloc::PerHeapKind<bmalloc::Heap> >::getSlowCase() () from /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
Title: deja-dup-monitor crashed with SIGSEGV in __pthread_once_slow()
UpgradeStatus: Upgraded to bionic on 2018-02-24 (0 days ago)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo

[Apport retracing service (apport)]

StacktraceTop:
 Gigacage::<lambda()>::operator() (__closure=<optimized out>) at ./Source/bmalloc/bmalloc/Gigacage.cpp:154
 std::__invoke_impl<void, Gigacage::ensureGigacage()::<lambda()> > (__f=...) at /usr/include/c++/7/bits/invoke.h:60
 std::__invoke<Gigacage::ensureGigacage()::<lambda()> > (__fn=...) at /usr/include/c++/7/bits/invoke.h:95
 std::<lambda()>::operator() (__closure=<optimized out>) at /usr/include/c++/7/mutex:672
 std::<lambda()>::operator() (__closure=0x0) at /usr/include/c++/7/mutex:677

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jeremy Bícha (jbicha)]

Ubuntu 18.04 (pre-Beta) recently upgraded to webkit2gtk 2.19.91. We are receiving lots of crash reports from Deja Dup at https://launchpad.net/bugs/1751460

Deja Dup is versioned 37.1.

Stacktrace
----------
#0 0x00007fac7224d588 in () at /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
#1 0x00007fac76c64827 in __pthread_once_slow (once_control=0x7fac724b502c, init_routine=0x7fac69330490 <__once_proxy>) at pthread_once.c:116
        _buffer = {__routine = 0x7fac76c64880 <clear_once_control>, __arg = 0x7fac724b502c, __canceltype = 2008932720, __prev = 0x0}
        val = <optimized out>
        newval = <optimized out>
#2 0x00007fac7224ce0d in Gigacage::ensureGigacage() () at /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
#3 0x00007fac7224dd01 in bmalloc::Heap::Heap(bmalloc::HeapKind, std::lock_guard<bmalloc::StaticMutex>&) () at /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
#4 0x00007fac7224bb10 in bmalloc::PerProcess<bmalloc::PerHeapKind<bmalloc::Heap> >::getSlowCase() () at /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
#5 0x00007fac7224b7d4 in bmalloc::Cache::Cache(bmalloc::HeapKind) () at /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
#6 0x00007fac7224bbf6 in bmalloc::PerThread<bmalloc::PerHeapKind<bmalloc::Cache> >::getSlowCase() () at /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
#7 0x00007fac7224b87f in bmalloc::Cache::allocateSlowCaseNullCache(bmalloc::HeapKind, unsigned long) () at /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
#8 0x00007fac72230f56 in WTF::StringImpl::createFromLiteral(char const*, unsigned int) () at /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
#9 0x00007fac72230fe1 in WTF::StringImpl::createFromLiteral(char const*) () at /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
#10 0x00007fac7223d3c0 in WTF::String::String(WTF::ASCIILiteral) () at /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
#11 0x00007fac729f7557 in () at /usr/lib/x86_64-linux-gnu/libwebkit2gtk-4.0.so.37
#12 0x00007fac779c5733 in call_init (env=0x7ffc3677ae08, argv=0x7ffc3677adf8, argc=1, l=<optimized out>) at dl-init.c:72
        j = <optimized out>
        jm = <optimized out>
        addrs = <optimized out>
        init_array = <optimized out>
        env = 0x7ffc3677ae08
        argv = 0x7ffc3677adf8
        argc = 1
        l = <optimized out>
        preinit_array = <optimized out>
        preinit_array_size = <optimized out>
        i = 6
#13 0x00007fac779c5733 in _dl_init (main_map=0x7fac77bde170, argc=1, argv=0x7ffc3677adf8, env=0x7ffc3677ae08) at dl-init.c:119
        preinit_array = <optimized out>
        preinit_array_size = <optimized out>
        i = 6
#14 0x00007fac779b60ca in _dl_start_user () at /lib64/ld-linux-x86-64.so.2
#15 0x0000000000000001 in ()
#16 0x00007ffc3677b9c3 in ()
#17 0x0000000000000000 in ()

[Jeremy Bícha (jbicha)]

Thread Stacktrace
=================
.
Thread 1 (Thread 0x7fac77b6aac0 (LWP 3178)):
#0 0x00007fac7224d588 in () at /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
#1 0x00007fac76c64827 in __pthread_once_slow (once_control=0x7fac724b502c, init_routine=0x7fac69330490 <__once_proxy>) at pthread_once.c:116
        _buffer = {__routine = 0x7fac76c64880 <clear_once_control>, __arg = 0x7fac724b502c, __canceltype = 2008932720, __prev = 0x0}
        val = <optimized out>
        newval = <optimized out>
#2 0x00007fac7224ce0d in Gigacage::ensureGigacage() () at /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
#3 0x00007fac7224dd01 in bmalloc::Heap::Heap(bmalloc::HeapKind, std::lock_guard<bmalloc::StaticMutex>&) () at /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
#4 0x00007fac7224bb10 in bmalloc::PerProcess<bmalloc::PerHeapKind<bmalloc::Heap> >::getSlowCase() () at /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
#5 0x00007fac7224b7d4 in bmalloc::Cache::Cache(bmalloc::HeapKind) () at /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
#6 0x00007fac7224bbf6 in bmalloc::PerThread<bmalloc::PerHeapKind<bmalloc::Cache> >::getSlowCase() () at /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
#7 0x00007fac7224b87f in bmalloc::Cache::allocateSlowCaseNullCache(bmalloc::HeapKind, unsigned long) () at /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
#8 0x00007fac72230f56 in WTF::StringImpl::createFromLiteral(char const*, unsigned int) () at /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
#9 0x00007fac72230fe1 in WTF::StringImpl::createFromLiteral(char const*) () at /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
#10 0x00007fac7223d3c0 in WTF::String::String(WTF::ASCIILiteral) () at /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
#11 0x00007fac729f7557 in () at /usr/lib/x86_64-linux-gnu/libwebkit2gtk-4.0.so.37
#12 0x00007fac779c5733 in call_init (env=0x7ffc3677ae08, argv=0x7ffc3677adf8, argc=1, l=<optimized out>) at dl-init.c:72
        j = <optimized out>
        jm = <optimized out>
        addrs = <optimized out>
        init_array = <optimized out>
        env = 0x7ffc3677ae08
        argv = 0x7ffc3677adf8
        argc = 1
        l = <optimized out>
        preinit_array = <optimized out>
        preinit_array_size = <optimized out>
        i = 6
#13 0x00007fac779c5733 in _dl_init (main_map=0x7fac77bde170, argc=1, argv=0x7ffc3677adf8, env=0x7ffc3677ae08) at dl-init.c:119
        preinit_array = <optimized out>
        preinit_array_size = <optimized out>
        i = 6
#14 0x00007fac779b60ca in _dl_start_user () at /lib64/ld-linux-x86-64.so.2
#15 0x0000000000000001 in ()
#16 0x00007ffc3677b9c3 in ()
#17 0x0000000000000000 in ()

[Cgarcia-f (cgarcia-f)]

The bt looks similar to the one in https://bugs.webkit.org/show_bug.cgi?id=179914#c39

[Michael Catanzaro (mike-catanzaro)]

The failure occurs here:

            // FIXME: Randomize where this goes.
            // https://bugs.webkit.org/show_bug.cgi?id=175245
            void* base = tryVMAllocate(maxAlignment, totalSize);
            if (!base) {
                if (GIGACAGE_ALLOCATION_CAN_FAIL)
                    return;
                fprintf(stderr, "FATAL: Could not allocate gigacage memory with maxAlignment = %lu, totalSize = %lu.\n", maxAlignment, totalSize);
                BCRASH();
            }

So tryVMAllocate fails. That means bmalloc was unable to allocate virtual memory. That's not supposed to fail (obviously). Implementation is here:

inline void* tryVMAllocate(size_t vmSize)
{
    vmValidate(vmSize);
    void* result = mmap(0, vmSize, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON | BMALLOC_NORESERVE, BMALLOC_VM_TAG, 0);
    if (result == MAP_FAILED)
        return nullptr;
    return result;
}

So the problem boils down to this mmap call. It's very strange that this is only happening with Deja Dup. Other applications are unaffected?

[Michael Catanzaro (mike-catanzaro)]

Jeremy, here's some debug you could try adding to Source/bmalloc/bmalloc/VMAllocate.h:

// At the top of the file, before the bmalloc namespace
#include <cstring>
#include <errno.h>

inline void* tryVMAllocate(size_t vmSize)
{
    vmValidate(vmSize);
    void* result = mmap(0, vmSize, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON | BMALLOC_NORESERVE, BMALLOC_VM_TAG, 0);
    if (result == MAP_FAILED)
{
WTFLogAlways("%s: mmap failed: %d (%s)", __FUNCTION__, errno, strerror(errno));
        return nullptr;
}
    return result;
}

That would tell us which of the many possible errors are occurring here.

And if you need an immediate workaround, you can of course build with -DUSE_SYSTEM_MALLOC=ON. That will be bad, so I can't recommend that... but you're already disabling GStreamerGL and web fonts.... :P

[Michael Catanzaro (mike-catanzaro)]

(In reply to Michael Catanzaro from comment #3)
> fprintf(stderr, "FATAL: Could not allocate gigacage memory
> with maxAlignment = %lu, totalSize = %lu.\n", maxAlignment, totalSize);

This might be helpful too. That should also print before the crash.

(In reply to Michael Catanzaro from comment #4)
> inline void* tryVMAllocate(size_t vmSize)
> {
> vmValidate(vmSize);
> void* result = mmap(0, vmSize, PROT_READ | PROT_WRITE, MAP_PRIVATE |
> MAP_ANON | BMALLOC_NORESERVE, BMALLOC_VM_TAG, 0);
> if (result == MAP_FAILED)
> {
> WTFLogAlways("%s: mmap failed: %d (%s)", __FUNCTION__, errno,
> strerror(errno));
> return nullptr;
> }
> return result;
> }

I forgot we can't use WTF here. Got to use fprintf:

fprintf(stderr, "%s: mmap failed: %d (%s)", __FUNCTION__, errno, strerror(errno));

[Ysuzuki-z (ysuzuki-z)]

(In reply to Michael Catanzaro from comment #4)
> Jeremy, here's some debug you could try adding to
> Source/bmalloc/bmalloc/VMAllocate.h:
>
> // At the top of the file, before the bmalloc namespace
> #include <cstring>
> #include <errno.h>
>
> inline void* tryVMAllocate(size_t vmSize)
> {
> vmValidate(vmSize);
> void* result = mmap(0, vmSize, PROT_READ | PROT_WRITE, MAP_PRIVATE |
> MAP_ANON | BMALLOC_NORESERVE, BMALLOC_VM_TAG, 0);
> if (result == MAP_FAILED)
> {
> WTFLogAlways("%s: mmap failed: %d (%s)", __FUNCTION__, errno,
> strerror(errno));
> return nullptr;
> }
> return result;
> }
>
> That would tell us which of the many possible errors are occurring here.
>
> And if you need an immediate workaround, you can of course build with
> -DUSE_SYSTEM_MALLOC=ON. That will be bad, so I can't recommend that... but
> you're already disabling GStreamerGL and web fonts.... :P

The immediate fix is disabling Gigacage by setting GIGACAGE_ENABLED 0 in bmalloc/Gigacage.h.
This keeps bmalloc, but disables Gigacage.

My guess is that Linux fails to mmap regions and returns MAP_FAILED if the size is very large.
But I'm not sure right now since it is working on my environment...
Anyway, @mcatanzaro, do you know the way to allocate virtual memory region which does not have actual backing pages?

[Ysuzuki-z (ysuzuki-z)]

(In reply to Yusuke Suzuki from comment #6)
> (In reply to Michael Catanzaro from comment #4)
> > Jeremy, here's some debug you could try adding to
> > Source/bmalloc/bmalloc/VMAllocate.h:
> >
> > // At the top of the file, before the bmalloc namespace
> > #include <cstring>
> > #include <errno.h>
> >
> > inline void* tryVMAllocate(size_t vmSize)
> > {
> > vmValidate(vmSize);
> > void* result = mmap(0, vmSize, PROT_READ | PROT_WRITE, MAP_PRIVATE |
> > MAP_ANON | BMALLOC_NORESERVE, BMALLOC_VM_TAG, 0);
> > if (result == MAP_FAILED)
> > {
> > WTFLogAlways("%s: mmap failed: %d (%s)", __FUNCTION__, errno,
> > strerror(errno));
> > return nullptr;
> > }
> > return result;
> > }
> >
> > That would tell us which of the many possible errors are occurring here.
> >
> > And if you need an immediate workaround, you can of course build with
> > -DUSE_SYSTEM_MALLOC=ON. That will be bad, so I can't recommend that... but
> > you're already disabling GStreamerGL and web fonts.... :P
>
> The immediate fix is disabling Gigacage by setting GIGACAGE_ENABLED 0 in
> bmalloc/Gigacage.h.
> This keeps bmalloc, but disables Gigacage.
>
> My guess is that Linux fails to mmap regions and returns MAP_FAILED if the
> size is very large.
> But I'm not sure right now since it is working on my environment...
> Anyway, @mcatanzaro, do you know the way to allocate virtual memory region
> which does not have actual backing pages?

My guess is that mmap with READ/WRITE starts populating backing pages and it fails.
I think we potentially have two ways to fix this issue.

The first way is,
1. first allocate region with mmap NO_PROTO
2. set read/write if necessary

But since Darwin is not requiring this mechanism, I think we need to change the current code a bit.
For example, vmAllocate() assumes that returned memory is accessible. But this assumption is slightly broken in Linux if we take this way. We need a bit additional code for Linux.

The second way is
1. first allocate region with mmap NO_PROTO
2. deallocate backing pages
3. set read/write for this region immediately

I'm not sure it works. But if it works, the change should be minimum.

[Michael Catanzaro (mike-catanzaro)]

One thing I just noticed, from https://bugzilla.gnome.org/show_bug.cgi?id=794056, is that this stack trace is exactly the same as the one valgrind hits when trying to use valgrind to debug epiphany since gigacage was enabled. That's probably expected, since valgrind attempts to allocate shadow bytes to keep track of the entire address space, which isn't practical under gigacage.

(In reply to Yusuke Suzuki from comment #6)
> Anyway, @mcatanzaro, do you know the way to allocate virtual memory region
> which does not have actual backing pages?

I have no clue about this. I thought actual backing pages were only employed once the memory region is actually used.

[Michael Catanzaro (mike-catanzaro)]

(In reply to Yusuke Suzuki from comment #6)
> My guess is that Linux fails to mmap regions and returns MAP_FAILED if the
> size is very large.

Keep in mind, we've had gigacage enabled and working fine for half a year....

[Ysuzuki-z (ysuzuki-z)]

(In reply to Michael Catanzaro from comment #8)
> One thing I just noticed, from
> https://bugzilla.gnome.org/show_bug.cgi?id=794056, is that this stack trace
> is exactly the same as the one valgrind hits when trying to use valgrind to
> debug epiphany since gigacage was enabled. That's probably expected, since
> valgrind attempts to allocate shadow bytes to keep track of the entire
> address space, which isn't practical under gigacage.

It sounds reasonable assumption to me.

>
> (In reply to Yusuke Suzuki from comment #6)
> > Anyway, @mcatanzaro, do you know the way to allocate virtual memory region
> > which does not have actual backing pages?
>
> I have no clue about this. I thought actual backing pages were only employed
> once the memory region is actually used.

(In reply to Michael Catanzaro from comment #9)
> (In reply to Yusuke Suzuki from comment #6)
> > My guess is that Linux fails to mmap regions and returns MAP_FAILED if the
> > size is very large.
>
> Keep in mind, we've had gigacage enabled and working fine for half a year....

Yeah, right. If it returns MAP_FAILED due to this issue, we should more constantly see this issue in our daily development. So my guess seems wrong.

[Michael Catanzaro (mike-catanzaro)]

Something could be different in Ubuntu, or in Deja Dup specifically?

I'm actually really surprised that Deja Dup now depends on WebKit. Must be via gnome-online-accounts.

[Jeremy Bícha (jbicha)]

(In reply to Michael Catanzaro from comment #11)
> I'm actually really surprised that Deja Dup now depends on WebKit. Must be
> via gnome-online-accounts.

Good question. I don't see any webkit code in Deja Dup but it does use GOA.

[dino99 (9d9)]

The journalctl log:

org.gnome.DejaDup.Monitor.desktop[2536]: FATAL: Could not allocate gigacage memory with maxAlignment = 34359738368, totalSize = 120259084288.

kernel: deja-dup-monito[2536]: segfault at bbadbeef ip 00007f067db65588 sp 00007ffff2df82c0 error 6 in libjavascriptcoregtk-4.0.so.18.7.6[7f067cdad000+fc4000

[SyKeY-XAM (ubuntu-xam)]

z

[Daniel van Vugt (vanvugt)]

Also tracking in:
https://errors.ubuntu.com/problem/27441b78823246dd5392ee29ac30546f6464289e

[vasilisc (vasilisc)]

[Ubuntu release]
Release: 18.04
Codename: bionic

[uname]
Linux vasilisc 4.15.0-12-generic #13-Ubuntu SMP Thu Mar 8 06:24:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[Kernel command line]
BOOT_IMAGE=/vmlinuz-4.15.0-12-generic root=UUID=6d8ea6da-b2d9-406c-a812-2b21a2925731 ro rootflags=subvol=@ elevator=cfq

deja-dup:
  Installed: 37.1-1fakesync1
  Candidate: 37.1-1fakesync1
  Version table:
 *** 37.1-1fakesync1 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages
        100 /var/lib/dpkg/status

[geez (geez)]

This bug appears to be crashing nautilus every time I try to open a file (by double clicking). As such, it is quite critical for me, as it effectively rendered the machine with 18.04 beta impossible to work on. Is anyone else experiencing this?

[Togo28 (togo28)]

I confirm what geez has written, the crashing is reproduceable

[cubells (cubells)]

@geez I think they are two different bugs.

Today I've opened this:
https://bugs.launchpad.net/ubuntu/+source/nautilus/+bug/1754564

[stephen saines (stephensaines)]

The most reproduceable crash for my system is running Firefox, any version,
including ESR, with no extensions. Sometimes it last longer than others,
almost always the initial opening holds, it's when a a new tab is initiated
it goes down. Rest of OS seems to operate unaffected, but FF is
irretrievable , and the crash report is generated, Never happened with
Chrome, or from my little use, Brave.

On Fri, Mar 9, 2018 at 3:48 AM, Togo28 <email address hidden> wrote:

> I confirm what geez has written, the crashing is reproduceable
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1753122).
> https://bugs.launchpad.net/bugs/1751460
>
> Title:
> [regression] deja-dup-monitor crashed with SIGSEGV in
> Gigacage::<lambda()>::operator()
>
> Status in Déjà Dup:
> New
> Status in WebKit:
> Confirmed
> Status in deja-dup package in Ubuntu:
> Confirmed
> Status in webkit2gtk package in Ubuntu:
> Confirmed
> Status in deja-dup source package in Bionic:
> Confirmed
> Status in webkit2gtk source package in Bionic:
> Confirmed
>
> Bug description:
> https://errors.ubuntu.com/problem/27441b78823246dd5392ee29ac3054
> 6f6464289e
>
> ProblemType: Crash
> DistroRelease: Ubuntu 18.04
> Package: deja-dup 37.1-1fakesync1
> ProcVersionSignature: Ubuntu 4.15.0-10.11-generic 4.15.3
> Uname: Linux 4.15.0-10-generic x86_64
> ApportVersion: 2.20.8-0ubuntu10
> Architecture: amd64
> CrashCounter: 1
> CurrentDesktop: GNOME
> Date: Sat Feb 24 14:30:47 2018
> ExecutablePath: /usr/lib/deja-dup/deja-dup-monitor
> InstallationDate: Installed on 2017-12-27 (59 days ago)
> InstallationMedia: Ubuntu 17.10 "Artful Aardvark" - Release amd64
> (20171018)
> ProcCmdline: /usr/lib/deja-dup/deja-dup-monitor
> ProcEnviron:
> PATH=(custom, no user)
> XDG_RUNTIME_DIR=<set>
> LANG=de_DE.UTF-8
> SHELL=/bin/bash
> SegvAnalysis:
> Segfault happened at: 0x7ff1c3dda588: movl $0x0,(%rax)
> PC (0x7ff1c3dda588) ok
> source "$0x0" ok
> destination "(%rax)" (0xbbadbeef) not located in a known VMA region
> (needed writable region)!
> SegvReason: writing unknown VMA
> Signal: 11
> SourcePackage: deja-dup
> StacktraceTop:
> ?? () from /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
> __pthread_once_slow (once_control=0x7ff1c404202c,
> init_routine=0x7ff1baec0490 <__once_proxy>) at pthread_once.c:116
> Gigacage::ensureGigacage() () from /usr/lib/x86_64-linux-gnu/
> libjavascriptcoregtk-4.0.so.18
> bmalloc::Heap::Heap(bmalloc::HeapKind, std::lock_guard<bmalloc::StaticMutex>&)
> () from /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
> bmalloc::PerProcess<bmalloc::PerHeapKind<bmalloc::Heap>
> >::getSlowCase() () from /usr/lib/x86_64-linux-gnu/
> libjavascriptcoregtk-4.0.so.18
> Title: deja-dup-monitor crashed with SIGSEGV in __pthread_once_slow()
> UpgradeStatus: Upgraded to bionic on 2018-02-24 (0 days ago)
> UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/deja-dup/+bug/1751460/+subscriptions
>

--
stephen saines
cell 647 631 0711

[dino99 (9d9)]

Feedback

Until now, deja-dup was crashing every time a session was opened (after cold boot). But it have stopped crashing on my system with a gnome-shell on xorg session, since you got that upgrade:

sane-backends (1.0.27-1~experimental3ubuntu2) bionic; urgency=medium

  * debian/rules: Drop timestamps from conflicting multiarch:same files
    hwdb.d/20-sane.hwdb and rules.d/60-libsane1.rules (closes: #880391)

 -- Adam Conrad <email address hidden> Sat, 03 Feb 2018 14:26:39 -0700

I cant see the relationship with that report, but the fact is the crash is gone. If there is no relationship at all, then only new gcc-7 & 8 have been upgraded and could explain that.

Is there still something to fix here, does not know !!!

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1751460

[venturia (alessandro-venturini)]

This bug happens every time I startup the PC. I've a notice that dejadup crash, than nautilus double click is no more able to to open a file with appropriate appplication.

Nautilus do not open the file with right click and choosing the app.

A-

[dino99 (9d9)]

Feedback2

commenting on #15 above:
in fact crash happens again but not every time; so we have now a random failure, shortly after opening a session (gnome on org). Does that suggest a race loading process, or a missing event answer ?

[Norbert (nrbrtx)]

Got this bug on fresh installation of Ubuntu 18.04 LTS with MATE DE.

[Luca Ciavatta (cialu)]

Installation of Ubuntu 18.04 LTS updating from 17.10.

This bug usually happens (but not every time) when I turn on the PC.
Dejadup crash by itself, last time I was only browsing some sites, no other activities.

[h9000 (h9000)]

after every reboot it did crash here

[venturia (alessandro-venturini)]

Same as #17

[Manfred Hampl (m-hampl)]

@venturia: The issue of not being able to start an executable from within nautilus has nothing to do with this deja-dup failure. That problem has been reported as Bug #1747711

[Kain Zhou (kainzhou)]

just happened to me again. deja-dup running on the background, no backup setup

[Daniel van Vugt (vanvugt)]

Workaround:

sudo apt remove deja-dup

[Luca Ciavatta (cialu)]

> Nautilus do not open the file with right click and choosing the app.

In my case, if I use the 'Open With Other Application' option (also if I select the same app of the 'Open With App' option), it works.

Still not working with the double-click or with 'Open With App' option.

[emk2203 (emk2203)]

Am 13.03.2018 9:26 vorm. schrieb "Daniel van Vugt" <
<email address hidden>>:

Workaround:

sudo apt remove deja-dup

Best idea so far. This program crashes consistently in every Ubuntu
distribution for years. I won't touch it with a ten-foot pole. Who would
trust it for something important like a backup?

[Manfred Hampl (m-hampl)]

@Luca Ciavatta:
Also your problem has nothing to do with deja-dup and webkit and is probably caused by bug #1754564

[tachiorz (tachiorz)]

temp workaround GIGACAGE_ENABLED=no environment variable

[Michael Catanzaro (mike-catanzaro)]

*** Bug 183594 has been marked as a duplicate of this bug. ***

[Michael Catanzaro (mike-catanzaro)]

So now we have:

 * Our first bug reports against applications other than Deja Dup
 * Our first bug report from a distro other than Ubuntu

Seems like a serious problem.

[Michael Catanzaro (mike-catanzaro)]

(In reply to Michael Catanzaro from comment #4)
> Jeremy, here's some debug you could try adding to
> Source/bmalloc/bmalloc/VMAllocate.h

^ This remains the next step here. We need to know why the mmap() call is failing.

[Shahab (shahab178)]

System monitor is crashing.

[Jeremy Bícha (jbicha)]

(In reply to Michael Catanzaro from comment #4)
> Jeremy, here's some debug you could try adding to
> Source/bmalloc/bmalloc/VMAllocate.h:

webkit2gtk failed to build for me with those lines.

https://launchpad.net/~jbicha/+archive/ubuntu/tempdeja/+sourcepub/8853741/+listing-archive-extra

Build log excerpt
=================

In file included from /<<PKGBUILDDIR>>/Source/bmalloc/bmalloc/SmallPage.h:32:0,
                 from /<<PKGBUILDDIR>>/Source/bmalloc/bmalloc/Deallocator.h:31,
                 from /<<PKGBUILDDIR>>/Source/bmalloc/bmalloc/Cache.h:31,
                 from /<<PKGBUILDDIR>>/Source/bmalloc/bmalloc/bmalloc.h:29,
                 from /<<PKGBUILDDIR>>/Source/bmalloc/bmalloc/BMalloced.h:28,
                 from /<<PKGBUILDDIR>>/Source/bmalloc/bmalloc/IsoHeapImpl.h:28,
                 from /<<PKGBUILDDIR>>/Source/bmalloc/bmalloc/AllIsoHeaps.h:28,
                 from /<<PKGBUILDDIR>>/Source/bmalloc/bmalloc/AllIsoHeaps.cpp:26:
/<<PKGBUILDDIR>>/Source/bmalloc/bmalloc/VMAllocate.h: In function ‘void* tryVMAllocate(size_t)’:
/<<PKGBUILDDIR>>/Source/bmalloc/bmalloc/VMAllocate.h:48:5: error: ‘vmValidate’ was not declared in this scope
     vmValidate(vmSize);
     ^~~~~~~~~~
/<<PKGBUILDDIR>>/Source/bmalloc/bmalloc/VMAllocate.h:49:85: error: ‘BMALLOC_NORESERVE’ was not declared in this scope
     void* result = mmap(0, vmSize, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON | BMALLOC_NORESERVE, BMALLOC_VM_TAG, 0);
                                                                                     ^~~~~~~~~~~~~~~~~
/<<PKGBUILDDIR>>/Source/bmalloc/bmalloc/VMAllocate.h:49:104: error: ‘BMALLOC_VM_TAG’ was not declared in this scope
     void* result = mmap(0, vmSize, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON | BMALLOC_NORESERVE, BMALLOC_VM_TAG, 0);

[Michael Catanzaro (mike-catanzaro)]

(In reply to Jeremy Bicha from comment #16)
> (In reply to Michael Catanzaro from comment #4)
> > Jeremy, here's some debug you could try adding to
> > Source/bmalloc/bmalloc/VMAllocate.h:
>
> webkit2gtk failed to build for me with those lines.

Do: keep the two new #includes where you added them, at the top, above the bmalloc namespace

Don't: add a new definition of tryVMAllocate. Just add the print to the one that already exists, lower in the file. Watch the braces.

[cyril rosik (roscyril-deactivatedaccount)]

deja-dup monitor crash in start ubuntu mate 18.04

[Jeremy Bícha (jbicha)]

Please do not add comments here saying that you are affected by this bug.

Instead, you can click the button at the top of this page that says "This bug affects ___ people".

[Michael Catanzaro (mike-catanzaro)]

Here's a debug patch:

diff --git a/Source/bmalloc/bmalloc/VMAllocate.h b/Source/bmalloc/bmalloc/VMAllocate.h
index 713e9c9fbb6..5c9c8553b1a 100644
--- a/Source/bmalloc/bmalloc/VMAllocate.h
+++ b/Source/bmalloc/bmalloc/VMAllocate.h
@@ -35,6 +35,9 @@
 #include <sys/mman.h>
 #include <unistd.h>

+#include <cstdio>
+#include <errno.h>
+
 #if BOS(DARWIN)
 #include <mach/vm_page_size.h>
 #include <mach/vm_statistics.h>
@@ -123,7 +126,10 @@ inline void* tryVMAllocate(size_t vmSize)
     vmValidate(vmSize);
     void* result = mmap(0, vmSize, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON | BMALLOC_NORESERVE, BMALLOC_VM_TAG, 0);
     if (result == MAP_FAILED)
+{
+fprintf(stderr, "%s: mmap failed: vmSize=%zu errno=%d (%s)", __FUNCTION__, vmSize, errno, strerror(errno));
         return nullptr;
+}
     return result;
 }

[Tpopela (tpopela)]

(In reply to Michael Catanzaro from comment #18)
> Here's a debug patch:

It's missing <cstring> include for strerror..

diff -up webkitgtk-2.20.0/Source/bmalloc/bmalloc/VMAllocate.h.bmalloc_debug webkitgtk-2.20.0/Source/bmalloc/bmalloc/VMAllocate.h
--- webkitgtk-2.20.0/Source/bmalloc/bmalloc/VMAllocate.h.bmalloc_debug 2018-02-19 08:45:33.000000000 +0100
+++ webkitgtk-2.20.0/Source/bmalloc/bmalloc/VMAllocate.h 2018-03-14 09:26:14.977674938 +0100
@@ -35,6 +35,10 @@
 #include <sys/mman.h>
 #include <unistd.h>

+#include <cstdio>
+#include <errno.h>
+#include <cstring>
+
 #if BOS(DARWIN)
 #include <mach/vm_page_size.h>
 #include <mach/vm_statistics.h>
@@ -122,8 +126,10 @@ inline void* tryVMAllocate(size_t vmSize
 {
     vmValidate(vmSize);
     void* result = mmap(0, vmSize, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON | BMALLOC_NORESERVE, BMALLOC_VM_TAG, 0);
- if (result == MAP_FAILED)
- return nullptr;
+ if (result == MAP_FAILED) {
+ fprintf(stderr, "%s: mmap failed: vmSize=%zu errno=%d (%s)", __FUNCTION__, vmSize, errno, strerror(errno));
+ return nullptr;
+ }
     return result;
 }

[Andy (dj23rus)]

SAME PROBLEM

[stephen saines (stephensaines)]

Supply a link! I like many others offered to help. I'm suddenly getting
emails that I might have inadvertently signed up for, but lacks specificity
in how to respond. May I suggest you find a way to link these responses to
your "page" wherever that is? Or do I just block the incoming emails?

On Tue, Mar 13, 2018 at 10:31 PM, Jeremy Bicha <email address hidden> wrote:

> Please do not add comments here saying that you are affected by this
> bug.
>
> Instead, you can click the button at the top of this page that says
> "This bug affects ___ people".
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1753122).
> https://bugs.launchpad.net/bugs/1751460
>
> Title:
> [regression] deja-dup-monitor crashed with SIGSEGV in
> Gigacage::<lambda()>::operator()
>
> Status in Déjà Dup:
> New
> Status in WebKit:
> Confirmed
> Status in deja-dup package in Ubuntu:
> Confirmed
> Status in webkit2gtk package in Ubuntu:
> Confirmed
> Status in deja-dup source package in Bionic:
> Confirmed
> Status in webkit2gtk source package in Bionic:
> Confirmed
>
> Bug description:
> https://errors.ubuntu.com/problem/27441b78823246dd5392ee29ac3054
> 6f6464289e
>
> ProblemType: Crash
> DistroRelease: Ubuntu 18.04
> Package: deja-dup 37.1-1fakesync1
> ProcVersionSignature: Ubuntu 4.15.0-10.11-generic 4.15.3
> Uname: Linux 4.15.0-10-generic x86_64
> ApportVersion: 2.20.8-0ubuntu10
> Architecture: amd64
> CrashCounter: 1
> CurrentDesktop: GNOME
> Date: Sat Feb 24 14:30:47 2018
> ExecutablePath: /usr/lib/deja-dup/deja-dup-monitor
> InstallationDate: Installed on 2017-12-27 (59 days ago)
> InstallationMedia: Ubuntu 17.10 "Artful Aardvark" - Release amd64
> (20171018)
> ProcCmdline: /usr/lib/deja-dup/deja-dup-monitor
> ProcEnviron:
> PATH=(custom, no user)
> XDG_RUNTIME_DIR=<set>
> LANG=de_DE.UTF-8
> SHELL=/bin/bash
> SegvAnalysis:
> Segfault happened at: 0x7ff1c3dda588: movl $0x0,(%rax)
> PC (0x7ff1c3dda588) ok
> source "$0x0" ok
> destination "(%rax)" (0xbbadbeef) not located in a known VMA region
> (needed writable region)!
> SegvReason: writing unknown VMA
> Signal: 11
> SourcePackage: deja-dup
> StacktraceTop:
> ?? () from /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
> __pthread_once_slow (once_control=0x7ff1c404202c,
> init_routine=0x7ff1baec0490 <__once_proxy>) at pthread_once.c:116
> Gigacage::ensureGigacage() () from /usr/lib/x86_64-linux-gnu/
> libjavascriptcoregtk-4.0.so.18
> bmalloc::Heap::Heap(bmalloc::HeapKind, std::lock_guard<bmalloc::StaticMutex>&)
> () from /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
> bmalloc::PerProcess<bmalloc::PerHeapKind<bmalloc::Heap>
> >::getSlowCase() () from /usr/lib/x86_64-linux-gnu/
> libjavascriptcoregtk-4.0.so.18
> Title: deja-dup-monitor crashed with SIGSEGV in __pthread_once_slow()
> UpgradeStatus: Upgraded to bionic on 2018-02-24 (0 days ago)
> UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/deja-dup/+bug/1751460/+subscriptions
>

--
stephen saines
cell 647 63...

[Emanuel Pravato (emanuelpravato)]

I'm just trying to contribute to system development by reporting bugs. I do
not work with programming

2018-03-14 9:30 GMT-03:00 stephen saines <email address hidden>:

> Supply a link! I like many others offered to help. I'm suddenly getting
> emails that I might have inadvertently signed up for, but lacks specificity
> in how to respond. May I suggest you find a way to link these responses to
> your "page" wherever that is? Or do I just block the incoming emails?
>
> On Tue, Mar 13, 2018 at 10:31 PM, Jeremy Bicha <email address hidden> wrote:
>
> > Please do not add comments here saying that you are affected by this
> > bug.
> >
> > Instead, you can click the button at the top of this page that says
> > "This bug affects ___ people".
> >
> > --
> > You received this bug notification because you are subscribed to a
> > duplicate bug report (1753122).
> > https://bugs.launchpad.net/bugs/1751460
> >
> > Title:
> > [regression] deja-dup-monitor crashed with SIGSEGV in
> > Gigacage::<lambda()>::operator()
> >
> > Status in Déjà Dup:
> > New
> > Status in WebKit:
> > Confirmed
> > Status in deja-dup package in Ubuntu:
> > Confirmed
> > Status in webkit2gtk package in Ubuntu:
> > Confirmed
> > Status in deja-dup source package in Bionic:
> > Confirmed
> > Status in webkit2gtk source package in Bionic:
> > Confirmed
> >
> > Bug description:
> > https://errors.ubuntu.com/problem/27441b78823246dd5392ee29ac3054
> > 6f6464289e
> >
> > ProblemType: Crash
> > DistroRelease: Ubuntu 18.04
> > Package: deja-dup 37.1-1fakesync1
> > ProcVersionSignature: Ubuntu 4.15.0-10.11-generic 4.15.3
> > Uname: Linux 4.15.0-10-generic x86_64
> > ApportVersion: 2.20.8-0ubuntu10
> > Architecture: amd64
> > CrashCounter: 1
> > CurrentDesktop: GNOME
> > Date: Sat Feb 24 14:30:47 2018
> > ExecutablePath: /usr/lib/deja-dup/deja-dup-monitor
> > InstallationDate: Installed on 2017-12-27 (59 days ago)
> > InstallationMedia: Ubuntu 17.10 "Artful Aardvark" - Release amd64
> > (20171018)
> > ProcCmdline: /usr/lib/deja-dup/deja-dup-monitor
> > ProcEnviron:
> > PATH=(custom, no user)
> > XDG_RUNTIME_DIR=<set>
> > LANG=de_DE.UTF-8
> > SHELL=/bin/bash
> > SegvAnalysis:
> > Segfault happened at: 0x7ff1c3dda588: movl $0x0,(%rax)
> > PC (0x7ff1c3dda588) ok
> > source "$0x0" ok
> > destination "(%rax)" (0xbbadbeef) not located in a known VMA region
> > (needed writable region)!
> > SegvReason: writing unknown VMA
> > Signal: 11
> > SourcePackage: deja-dup
> > StacktraceTop:
> > ?? () from /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
> > __pthread_once_slow (once_control=0x7ff1c404202c,
> > init_routine=0x7ff1baec0490 <__once_proxy>) at pthread_once.c:116
> > Gigacage::ensureGigacage() () from /usr/lib/x86_64-linux-gnu/
> > libjavascriptcoregtk-4.0.so.18
> > bmalloc::Heap::Heap(bmalloc::HeapKind, std::lock_guard<bmalloc::
> StaticMutex>&)
> > () from /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
> > bmalloc::PerProcess<bmalloc::PerHeapKind<bmalloc::Heap>
> > >::getSlowCase() () from /usr/lib/x86_64-linux-gnu/
> > libjavascriptcoregtk-4.0.so.18
> > Title: deja-d...

[Jeremy Bícha (jbicha)]

Stephen, there is a link in the sidebar of https://launchpad.net/bugs/1751460 to mute bug mail for this bug. Or you can edit your subscription there to stop receiving comments for this bug.

[Michael Catanzaro (mike-catanzaro)]

I swear I tested it, but I tested on trunk, so maybe cstring was getting pulled in elsewhere.

[Tpopela (tpopela)]

(In reply to Michael Catanzaro from comment #20)
> I swear I tested it, but I tested on trunk, so maybe cstring was getting
> pulled in elsewhere.

Nevermind, see Jiri's response in https://bugzilla.redhat.com/show_bug.cgi?id=1554873#c6. Updating mutter fixed the issue..

[Michael Catanzaro (mike-catanzaro)]

(In reply to Michael Catanzaro from comment #15)
> ^ This remains the next step here. We need to know why the mmap() call is
> failing.

Jeremy discovered Deja Dup is setting a virtual memory limit in its desktop file:

Exec=sh -c "ulimit -v 1000000; exec @pkglibexecdir@/deja-dup-monitor"

which is incompatible with Gigacage. It needs to not do that. This is NOTABUG. Good find, Jeremy!

(In reply to Tomas Popela from comment #21)
> Nevermind, see Jiri's response in
> https://bugzilla.redhat.com/show_bug.cgi?id=1554873#c6. Updating mutter
> fixed the issue..

The mutter update would definitely fix the problem described (app fails to launch another app), but I don't see how it could possibly be related to the ensureGigacage() backtrace. Anyway, that bug is closed now too... we can always revisit if it becomes a problem again.

[Jeremy Bícha (jbicha)]

To give some more context about what Deja Dup was doing…

See
https://salsa.debian.org/gnome-team/deja-dup/blob/debian/master/data/org.gnome.DejaDup.Monitor.desktop.in

which points to https://launchpad.net/bugs/1302416

And here's the output when I ran the command via webkit2gtk 2.20 built with the debugging patch proposed earlier here

$ ulimit -v 1000000; /usr/lib/deja-dup/deja-dup-monitor
tryVMAllocate: mmap failed: vmSize=154618822656 errno=12 (Cannot allocate memory)
FATAL: Could not allocate gigacage memory with
 maxAlignment = 34359738368, totalSize = 120259084288.
Segmentation fault (core dumped)

[Michael Catanzaro (mike-catanzaro)]

Seb from Ubuntu is worried that people will have set a vmlimit for the entire desktop session, and requests that we improve the error message here.

[Michael Catanzaro (mike-catanzaro)]

Created attachment 335768
Patch

[Tpopela (tpopela)]

(In reply to Michael Catanzaro from comment #25)
> Created attachment 335768 [details]
> Patch

Informal r+ as we spoke together on IRC..

[Michael Catanzaro (mike-catanzaro)]

Filip, Geoffrey, does this error message look OK to you? Want any adjustments?

[tachiorz (tachiorz)]

ulimit -a returned virtual memory unlimited, but I figured out that this error was triggered because I explicitly disabled overcommit by
vm.overcommit_memory = 2
setting it to 1 fixed the issue

[Fpizlo (fpizlo)]

Comment on attachment 335768
Patch

You might want to set GIGACAEG_ALLOCATION_CAN_FAIL to true on Linux until that's fixed.

[Michael Catanzaro (mike-catanzaro)]

(In reply to dytynenko.roman from comment #28)
> ulimit -a returned virtual memory unlimited, but I figured out that this
> error was triggered because I explicitly disabled overcommit by
> vm.overcommit_memory = 2
> setting it to 1 fixed the issue

Oh boy...

(In reply to Filip Pizlo from comment #29)
> Comment on attachment 335768 [details]
> Patch
>
> You might want to set GIGACAEG_ALLOCATION_CAN_FAIL to true on Linux until
> that's fixed.

I'm not sure what we should do here. We definitely don't need to change anything for Deja Dup; it's easier to fix Deja Dup than to update WebKit, anyway. Disabling overcommit is worth considering, though, and the connection between overcommit and browser security is not obvious.

 * If we leave things unchanged, WebKit will crash with overcommit disabled.
 * If we set GIGACAGE_ALLOCATION_CAN_FAIL, WebKit will not crash, but an important security feature will be lost, and users will not notice.

My inclination is to say Gigacage is sufficiently important that WebKit should crash if overcommit is disabled. Perhaps that's not very kind, though. Thoughts?

[Marcos Nascimento (wstlmn)]

This error message is often displayed after the system restarts ubuntu 18.04.

[tachiorz (tachiorz)]

Gigacage::ensureGigacage() will crash if virtual memory (ulimit -v) isn't unlimited or overcommit (vm.overcommit_memory) disabled.

[Wild Man (wildmanne39)]

Deja-dub crashes on startup every time I boot my laptop, I upgraded from Ubuntu 17.10 to 18.04. I am using kernel 4.16rc4.

[Michael Catanzaro (mike-catanzaro)]

Comment on attachment 335768
Patch

Let's keep this error message. System administrators should consider the needs of the software they are running before setting virtual memory limits or disabling overcommit.

[Commit-queue (commit-queue)]

Comment on attachment 335768
Patch

Clearing flags on attachment: 335768

Committed r229669: <https://trac.webkit.org/changeset/229669>

[Commit-queue (commit-queue)]

All reviewed patches have been landed. Closing bug.

[Webkit-bug-importer (webkit-bug-importer)]

<rdar://problem/38548430>

[Ysuzuki-z (ysuzuki-z)]

(In reply to dytynenko.roman from comment #28)
> ulimit -a returned virtual memory unlimited, but I figured out that this
> error was triggered because I explicitly disabled overcommit by
> vm.overcommit_memory = 2
> setting it to 1 fixed the issue

OVERCOMMIT_NEVER (2) makes mmap's flag MAP_NORESERVE meaningless.
It makes Gigacage address space allocation failed.
https://github.com/torvalds/linux/blob/master/mm/mmap.c#L1475
So it sounds reasonable to me that vm.overcommit_memory=0 or 1 fixes this issue.

[Launchpad Janitor (janitor)]

This bug was fixed in the package deja-dup - 37.1-2fakesync1

---------------
deja-dup (37.1-2fakesync1) bionic-proposed; urgency=medium

  * Fake sync due to mismatching orig tarball.

deja-dup (37.1-2) experimental; urgency=medium

  * Add 0002-don-t-use-ulimit.patch:
    - Stop using ulimit since it is incompatible with webkit2gtk 2.20
      (LP: #1751460)

 -- Jeremy Bicha <email address hidden> Fri, 16 Mar 2018 19:31:21 -0400

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Kain Zhou (kainzhou)]

Thank you for releasing a fix

[Vej (vej)]

You fixed by removing a fix without taking measurements to prevent the problem the fix was for?
Are you going to handle that as well or are we just going to see the OOM bugs again?

If you are handling this in another bug: Please link it here.

Thanks

Vej

[Jeremy Bícha (jbicha)]

Vej, please file a separate bug if you can reproduce the original memory consumption bug.

[Jeremy Bícha (jbicha)]

I am closing the webkit2gtk part of this bug.

It was mentioned that someone might set a virtual memory limit for their entire system. I don't think setting that makes sense on a desktop and it's my understanding that doing so will also break Java apps for a similar reason.

To test the impact of this, I added this to my ~/.profile

ulimit -v 4000000

After logging out and logging back in, I was still able to run GNOME Shell (although presumably the Captive Portal feature won't work). Any webkit using apps won't work (and that includes Epiphany, gnome-control-center, evolution, etc.).

At least the systemd journal records this basic error message:

org.gnome.Epiphany.desktop[12949]: FATAL: Could not allocate gigacage
memory with maxAlignment = 34359738368, totalSize = 103079215104.

.

The next upstream release of webkit2gtk will also add this to that error message: "Make sure you have not set a virtual memory limit."

So the remaining task here is to provide a deja-dup update to all supported Ubuntu releases so that we can safely provide webkit2gtk security updates there.

[Manfred Hampl (m-hampl)]

Looking at Jeremy Bicha's error message

org.gnome.Epiphany.desktop[12949]: FATAL: Could not allocate gigacage memory with maxAlignment = 34359738368, totalSize = 103079215104.

"totalSize = 103079215104" stands for a memory allocation of 96 GiB.
What the heck is webkit2gtk doing with so much memory?

[Jeremy Bícha (jbicha)]

Manfred, it is virtual memory space for an ASLR type security hardening feature. It doesn't actually allocate that much memory at all.

[Eduard Hasenleithner (eduard-hasenleithner)]

It is really a sad affair here. As far as I know Linux does not provide a explicit means to limit the amount of memory an application can use. The only reliable ulimit one can set is the virtual size limit. It feels also strange that DejaDup pulls a full web-browser into its background daemon. And furthermore that this web browser is not initialized on demand, but unconditionally.

[Vej (vej)]

Hello!

I noticed that the recent duplicate bug #1760233 does use deja-dup 37.1-2fakesync1, although this bug had been marked as fixed for this package.

Should we reset this to "Triaged"?

[Vej (vej)]

Resetting to Triaged, because the duplicates bug #1756817 and bug #1756776 show the same characteristics as described in comment #50.

[JGuire (jdmcguire)]

I think this one is all good now for me, thanks!

On Sat, Mar 31, 2018, 7:20 AM Vej, <email address hidden> wrote:

> Hello!
>
> I noticed that the recent duplicate bug #1760233 does use deja-dup
> 37.1-2fakesync1, although this bug had been marked as fixed for this
> package.
>
> Should we reset this to "Triaged"?
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1754195).
> https://bugs.launchpad.net/bugs/1751460
>
> Title:
> [regression] deja-dup-monitor crashed with SIGSEGV in
> Gigacage::<lambda()>::operator()
>
> Status in Déjà Dup:
> New
> Status in WebKit:
> Fix Released
> Status in deja-dup package in Ubuntu:
> Fix Released
> Status in webkit2gtk package in Ubuntu:
> Invalid
> Status in deja-dup source package in Xenial:
> Triaged
> Status in deja-dup source package in Artful:
> Triaged
> Status in deja-dup source package in Bionic:
> Fix Released
>
> Bug description:
> Impact
> ------
> webkit2gtk 2.20 adds a new security feature called the Gigacage that
> uses an extremely large virtual memory address space (much larger than
> available physical memory).
>
> Deja Dup's monitor background service had "ulimit -v 1000000" (that's
> 1 GB) set as a workaround for a memory leak issue that the developer
> was unable to reproduce.
>
> After upgrading to the new webkit2gtk version, Deja Dup's monitor
> service will crash because of that virtual memory limit.
>
> Test Case
> ---------
> Install the deja-dup update.
> Install the webkit2gtk update from a PPA (not prepared yet).
> Log out. Log in.
> After a few minutes, check /var/crash/ for any Deja Dup crash reports.
>
> Regression Potential
> --------------------
> This could reintroduce the memory leak bug, but otherwise this is a
> minimal fix. Even if that happens, it's better than the service refusing to
> run.
>
> Other Info
> ----------
>
> https://errors.ubuntu.com/problem/27441b78823246dd5392ee29ac30546f6464289e
>
> ProblemType: Crash
> DistroRelease: Ubuntu 18.04
> Package: deja-dup 37.1-1fakesync1
> ProcVersionSignature: Ubuntu 4.15.0-10.11-generic 4.15.3
> Uname: Linux 4.15.0-10-generic x86_64
> ApportVersion: 2.20.8-0ubuntu10
> Architecture: amd64
> CrashCounter: 1
> CurrentDesktop: GNOME
> Date: Sat Feb 24 14:30:47 2018
> ExecutablePath: /usr/lib/deja-dup/deja-dup-monitor
> InstallationDate: Installed on 2017-12-27 (59 days ago)
> InstallationMedia: Ubuntu 17.10 "Artful Aardvark" - Release amd64
> (20171018)
> ProcCmdline: /usr/lib/deja-dup/deja-dup-monitor
> ProcEnviron:
> PATH=(custom, no user)
> XDG_RUNTIME_DIR=<set>
> LANG=de_DE.UTF-8
> SHELL=/bin/bash
> SegvAnalysis:
> Segfault happened at: 0x7ff1c3dda588: movl $0x0,(%rax)
> PC (0x7ff1c3dda588) ok
> source "$0x0" ok
> destination "(%rax)" (0xbbadbeef) not located in a known VMA region
> (needed writable region)!
> SegvReason: writing unknown VMA
> Signal: 11
> SourcePackage: deja-dup
> StacktraceTop:
> ?? () from /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
> __pthread_once_slow (once_control=0x7ff1c404202c,
> init_routine=...

[Jeremy Bícha (jbicha)]

Vej, I'm reclosing this bug for bionic. Maybe you just hadn't rebooted since applying the update. Apport sometimes reports bugs late too.

[Jeremy Bícha (jbicha)]

deja-dup 38.0 was released with the ulimit workaround removed for compatibility with the latest webkit release.

[Tpopela (tpopela)]

So it looks like there is a way of avoiding this in most cases. Florian Weimer suggested the following in https://bugzilla.redhat.com/show_bug.cgi?id=1570021:

mmap(NULL, 154618822656, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)

MAP_NORESERVE does not have the intended effect with vm.overcommit_memory=2 (which enables the beancounter/disables overcommit).

The standard way of reserving address space is to use a PROT_NONE mapping and make parts readable/writable as needed using mprotect later. This will work with vm.overcommit_memory=2 as well because initially PROT_NONE mappings do not count towards the commit limit.

Yusuke, would you mind looking at this? You seem to know this low level stuff best.

[Michael Catanzaro (mike-catanzaro)]

Of course he notices due to Emacs... OK, reopening.

[Launchpad Janitor (janitor)]

This bug was fixed in the package deja-dup - 36.3-0ubuntu0.2

---------------
deja-dup (36.3-0ubuntu0.2) artful-security; urgency=medium

  * Add 0002-don-t-use-ulimit.patch:
    - Stop using ulimit since it is incompatible with webkit2gtk 2.20
      (LP: #1751460)

 -- Marc Deslauriers <email address hidden> Fri, 27 Apr 2018 07:48:04 -0400

[Jeremy Bícha (jbicha)]

I closed the Xenial task since deja-dup in Xenial doesn't use ulimit.

[Daniel Kondor (kondor-dani)]

Hi,

I just started having problems due to this issue recently on Ubuntu 16.04.4 with libjavascriptcoregtk-4.0-18 version 2.20.1-0ubuntu0.16.04.1. Several Gnome 3.18 applications will crash on startup (including Rhythmbox 3.3, yelp, gnome-control-center, unity-control-center 15.04.0+16.04.20171130-0ubuntu1, zenity, but basically anything that depends on WebKit I guess). I get the following (or similar) error message:

FATAL: Could not allocate gigacage memory with maxAlignment = 34359738368, totalSize = 103079215104.
(Make sure you have not set a virtual memory limit.)

Setting the environment variable GIGACAGE_ENABLED=0 is a good workaround.

I have overcommit_memory = 2, and I prefer to have it that way (short reason: I develop and run various programs for scientific computations, which sometimes need a lot of memory -- in my case, it is clearly preferable to have a memory allocation fail, producing a meaningful error message than having a seemingly random crash later or invoking the OOM killer). Nevertheless, I still use Gnome desktop applications, which depend on WebKit, producing this issue.

My understanding is that the only way to fix this is to use PROT_NONE when allocating address space and later use mprotect() before actually using it. I understand that depending the structure of the code, that could be a lot of work to do for a somewhat specialized use case.

Let me know if I can provide any more info.

Stack trace:

Thread 1 (Thread 0x7ffff7edda80 (LWP 16799)):
#0 0x00007fffeccdb905 in ?? ()
   from /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
#1 0x00007ffff66b4a99 in __pthread_once_slow (once_control=0x7fffecf49008,
    init_routine=0x7fffe8aceac0 <__once_proxy>) at pthread_once.c:116
#2 0x00007fffeccdb243 in Gigacage::ensureGigacage() ()
   from /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
#3 0x00007fffecce05a8 in bmalloc::mapToActiveHeapKind(bmalloc::HeapKind) ()
   from /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
#4 0x00007fffeccd9d3e in bmalloc::Cache::allocateSlowCaseNullCache(bmalloc::HeapKind, unsigned long) ()
   from /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
#5 0x00007fffeccbf6f6 in WTF::StringImpl::createFromLiteral(char const*, unsigned int) () from /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
#6 0x00007fffeccbf781 in WTF::StringImpl::createFromLiteral(char const*) ()
   from /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
#7 0x00007fffecccbc20 in WTF::String::String(WTF::ASCIILiteral) ()
   from /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18
#8 0x00007ffff3865a17 in ?? ()
   from /usr/lib/x86_64-linux-gnu/libwebkit2gtk-4.0.so.37
#9 0x00007ffff7de76ba in call_init (l=<optimized out>, argc=argc@entry=1,
    argv=argv@entry=0x7fffffffe608, env=env@entry=0x7fffffffe618)
    at dl-init.c:72
#10 0x00007ffff7de77cb in call_init (env=0x7fffffffe618, argv=0x7fffffffe608,
    argc=1, l=<optimized out>) at dl-init.c:30
#11 _dl_init (main_map=0x7ffff7ffe168, argc=1, argv=0x7fffffffe608,
    env=0x7fffffffe618) at dl-init.c:120
#12 0x00007ffff7dd7c6a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#13 0x0000000000000001 in ?? (...

[Michael Catanzaro (mike-catanzaro)]

(In reply to Daniel Kondor from comment #38)
> My understanding is that the only way to fix this is to use PROT_NONE when
> allocating address space and later use mprotect() before actually using it.
> I understand that depending the structure of the code, that could be a lot
> of work to do for a somewhat specialized use case.

Yeah, that's exactly what needs to happen here. I took a look at this, but wasn't sure where the mprotect()s were needed.

[Ysuzuki-z (ysuzuki-z)]

(In reply to Michael Catanzaro from comment #39)
> (In reply to Daniel Kondor from comment #38)
> > My understanding is that the only way to fix this is to use PROT_NONE when
> > allocating address space and later use mprotect() before actually using it.
> > I understand that depending the structure of the code, that could be a lot
> > of work to do for a somewhat specialized use case.
>
> Yeah, that's exactly what needs to happen here. I took a look at this, but
> wasn't sure where the mprotect()s were needed.

I'm a bit worried about this approach in terms of performance...

[Daniel Kondor (kondor-dani)]

Just another thought on a potential further implication:
if PROT_NONE was the default and mprotect() used to make individual pages writable / readable, any out of bounds accesses (due to e.g. any bugs / exploits that might come up later) have some chance of triggering a segfault if it falls to another page not reserved with mprotect().

Not being familiar with the code, I'm not sure if this would be a feature or if guard pages like this are already implemented. Just to point out that any change will probably have more implications than I originally would have thought :)

[Ysuzuki-z (ysuzuki-z)]

Created attachment 340850
Patch

[Michael Catanzaro (mike-catanzaro)]

Comment on attachment 340850
Patch

Well I'm not sure if we should do this, because Gigacage is an important security feature. There are some notes at https://labs.mwrinfosecurity.com/blog/some-brief-notes-on-webkit-heap-hardening/ about all the hardening features that this will silently disable.

But I guess not crashing is good, so r=me.

Did you try Florian's suggestion? Were there problems with performance?

[Ysuzuki-z (ysuzuki-z)]

(In reply to Michael Catanzaro from comment #43)
> Comment on attachment 340850 [details]
> Patch
>
> Well I'm not sure if we should do this, because Gigacage is an important
> security feature. There are some notes at
> https://labs.mwrinfosecurity.com/blog/some-brief-notes-on-webkit-heap-
> hardening/ about all the hardening features that this will silently disable.
>
> But I guess not crashing is good, so r=me.
>
> Did you try Florian's suggestion? Were there problems with performance?

Currently, I'm not trying this.
I think currently this patch is OK since it avoids crashing. And users can enable gigacage by enabling overcommit :)

[Ysuzuki-z (ysuzuki-z)]

Committed r232059: <https://trac.webkit.org/changeset/232059>