produces empty core dumps

[Martin Pitt]

Binary package hint: apport

Various reports, like bug 122470 or bug 122511 have an empty CoreDump: and empty Stacktraces.

[Martin Pitt (pitti)]

Can people who are affected by this please do the following:

  sudo rm /var/crash/*
  sudo rm /var/log/apport.log
  /usr/share/apport/testsuite/run-tests 2>&1 | tee /tmp/apport-test.out

and attach /tmp/apport-test.out and /var/log/apport.log to bug 122688?

Thanks a lot!

[Jordan Hall (jordan-hall)]

I will attach the files you requested. It thought it may also be useful to know that your final command ended as follows:

[snip]
Test search_bug_patterns(). ... ok
Test standard_title(). ... ok

======================================================================
FAIL: Test add_gdb_info() with a script.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/share/python-support/python-apport/apport/report.py", line 1179, in test_add_gdb_info_script
    self.assert_('libc.so' in pr['Stacktrace'])
AssertionError

[Jordan Hall (jordan-hall)]

It appears after running those commands that /var/log/apport.log does not exist. I have attached the closest match (/var/log/apport.log.1), although I guess this is just a backup or older log. I assume the failure to create /var/log/apport.log was caused by the AssertionError generated by the 3rd command.

[Martin Pitt (pitti)]

Thanks, folks, for the logs so far, this gave me an initial idea. Can you please now do the following:

  /usr/share/apport/testsuite/test-apport kernel 2>&1 | tee /tmp/test-kernel.out

And attach /tmp/test-kernel.out and /var/log/apport.log now? If /var/log/apport.log does not exist, don't bother attaching apport.log.1, that does not help.

There seems to be something fundamentally wrong with gdb & co on i386. However, the tests should get a little further if you install the package 'build-essential' (this will pull in some dependencies like gcc and libc6-dev). After that, can you please do the /usr/share/apport/testsuite/run-tests test again?

Thanks a lot in advance!

[Roberto Gradini (robertogradini)]

There's no /var/log/apport.log

[Roberto Gradini (robertogradini)]

I've install 'build-essential'

[Martin Pitt (pitti)]

not a gnome-speech bug, closing this task.

[Martin Pitt (pitti)]

RobertGradini,

sorry, I forgot to mention that you have to clean /var/crash/ before running the test suite. Can you please do

  sudo rm /var/crash/*
  /usr/share/apport/testsuite/test-apport kernel 2>&1 | tee /tmp/test-kernel.out

instead? Thank you!

[Roberto Gradini (robertogradini)]

I'm sorry but there's a problem:

roberto@pearljam:~$ /usr/share/apport/testsuite/test-apport kernel 2>&1 | tee /tmp/test-kernel.out
* Check test process creation/killing with apport
* Check that a subsequent crash does not alter unseen report
* Check that a subsequent crash alters seen report
* Check that report has required fields
* Check that dumped environment only has insensitive variables
* Check that collected system groups has nonempty user groups information
* Check that collected system groups are not those from root
* Check that non-packaged executables do not create a report
* Check that apport ignores SIGQUIT
* Check that existence of user-inaccessible files does not leak
* Check that non-packaged scripts do not create a report
* Test limitation of flooding: iteration 0 1 2 3
Traceback (most recent call last):
  File "/usr/share/apport/testsuite/test-apport", line 290, in <module>
    'no reports present after deleting (present: %s)' % str(apport.fileutils.get_all_reports())
AssertionError: no reports present after deleting (present: ['/var/crash/_usr_share_apport_testsuite_test-apport.1000.crash'])

[Shirish Agarwal (shirishag75)]

FAIL: Test add_gdb_info() with a script.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/share/python-support/python-apport/apport/report.py", line 1179, in test_add_gdb_info_script
    self.assert_('libc.so' in pr['Stacktrace'])
AssertionError

----------------------------------------------------------------------
Ran 16 tests in 16.125s

FAILED (failures=1)

last output while doing the tests.

[useResa (rdrijsen)]

Performed the actions as requested in Bug #122771, being:
"Actually apport should work on this kernel again. Can you please do the
following:

 sudo rm /var/crash/*
 sudo rm /var/log/apport.log
 /usr/share/apport/testsuite/run-tests 2>&1 | tee /tmp/apport-test.out

and attach /tmp/apport-test.out and /var/log/apport.log to bug 122688?"

[useResa (rdrijsen)]

The second file /var/log/apport.log was not present.
Please find attached the results of the last command as provided in the instructions:

rdrijsen@rdrijsen-gutsy:~$ /usr/share/apport/testsuite/run-tests 2>&1 | tee /tmp/apport-test.out
+ python -tt /usr/share/python-support/python-problem-report/problem_report.py -v
Test adding information to an existing report. ... ok
Test basic creation and operation. ... ok
Test writing and re-decoding a big random file. ... ok
Test non-default constructor arguments. ... ok
Test ProblemReport iteration. ... ok
Test load() with various formatting. ... ok
Test reading, modifying fields, and writing back. ... ok
Test reading a report with binary data. ... ok
Test reading a report with binary data (legacy bzip2 compression). ... ok
Test various error conditions. ... ok
Test writing and a big random file with a size limit key. ... ok
Test new_keys() and write() with only_new=True. ... ok
Test write() and proper formatting. ... ok
Test write() with appending to an existing file. ... ok
Test writing a report with binary file data. ... ok
Test writing a report with a pointer to a file-like object. ... ok
Test write_mime() for binary values and file references. ... ok
Test write_mime() with extra headers. ... ok
Test write_mime() for text values. ... ok

----------------------------------------------------------------------
Ran 19 tests in 5.018s

OK
+ python -tt /usr/share/python-support/python-apport/apport/fileutils.py -v
Test check_files_md5(). ... ok
Test delete_report(). ... ok
Test find_file_package(). ... ok
Test find_package_desktopfile(). ... ok
Test get_all_reports(). ... ok
Test get_recent_crashes(). ... ok
Test get_all_system_reports() and get_new_system_reports(). ... ok
Test likely_packaged(). ... ok
Test make_report_path(). ... ok
Test get_new_reports() and seen_report(). ... ok

----------------------------------------------------------------------
Ran 10 tests in 4.158s

OK
+ python -tt /usr/share/python-support/python-apport/apport/report.py -v
Test add_gdb_info() with core dump file reference. ... Failed to read a valid object file image from memory.
warning: Memory read failed for corefile section, 4096 bytes at 0xffffe000.
ok
Test add_gdb_info() with inline core dump. ... Failed to read a valid object file image from memory.
warning: Memory read failed for corefile section, 4096 bytes at 0xffffe000.
ok
Test add_gdb_info() with a script. ... Failed to read a valid object file image from memory.
warning: Memory read failed for corefile section, 4096 bytes at 0xffffe000.
FAIL
Test add_hooks_info(). ... ok
Test add_os_info(). ... ok
Test add_package_info(). ... ok
Test add_proc_info(). ... ok
Test add_user_info(). ... ok
Test _check_interpreted(). ... ok
Test crash_signature(). ... ok
Test _gen_stacktrace_top(). ... ok
Test has_useful_stacktrace(). ... ok
Test mark_ignore() and check_ignored(). ... ok
Test obsolete_packages(). ... ok
Test search_bug_patterns(). ... ok
Test standard_title(). ... ok

======================================================================
FAIL: Test add_gdb_info() with a script.
----------------------------------------------------------------------
Traceback (most r...

[Shirish Agarwal (shirishag75)]

tried /usr/share/apport/testsuite/test-apport kernel 2>&1 | tee /tmp/test-kernel.out

this happens

shirish@ubuntu:~$ sudo rm /var/crash/*
[sudo] password for shirish:
rm: cannot remove `/var/crash/*': No such file or directory
shirish@ubuntu:~$ /usr/share/apport/testsuite/test-apport kernel 2>&1 | tee /tmp/test-kernel.out
* Check test process creation/killing with apport
Traceback (most recent call last):
  File "/usr/share/apport/testsuite/test-apport", line 165, in <module>
    check_crash()
  File "/usr/share/apport/testsuite/test-apport", line 60, in check_crash
    assert not os.path.exists('core'), 'no core dump in current directory'
AssertionError: no core dump in current directory
shirish@ubuntu:~$

and there is a crash report at bug 122819

[Shirish Agarwal (shirishag75)]

Here is the whole thing in case if people are interested

shirish@ubuntu:~$ /usr/share/apport/testsuite/run-tests 2>&1 | tee /tmp/apport-test.out
+ python -tt /usr/share/python-support/python-problem-report/problem_report.py -v
Test adding information to an existing report. ... ok
Test basic creation and operation. ... ok
Test writing and re-decoding a big random file. ... ok
Test non-default constructor arguments. ... ok
Test ProblemReport iteration. ... ok
Test load() with various formatting. ... ok
Test reading, modifying fields, and writing back. ... ok
Test reading a report with binary data. ... ok
Test reading a report with binary data (legacy bzip2 compression). ... ok
Test various error conditions. ... ok
Test writing and a big random file with a size limit key. ... ok
Test new_keys() and write() with only_new=True. ... ok
Test write() and proper formatting. ... ok
Test write() with appending to an existing file. ... ok
Test writing a report with binary file data. ... ok
Test writing a report with a pointer to a file-like object. ... ok
Test write_mime() for binary values and file references. ... ok
Test write_mime() with extra headers. ... ok
Test write_mime() for text values. ... ok

----------------------------------------------------------------------
Ran 19 tests in 4.366s

OK
+ python -tt /usr/share/python-support/python-apport/apport/fileutils.py -v
Test check_files_md5(). ... ok
Test delete_report(). ... ok
Test find_file_package(). ... ok
Test find_package_desktopfile(). ... ok
Test get_all_reports(). ... ok
Test get_recent_crashes(). ... ok
Test get_all_system_reports() and get_new_system_reports(). ... ok
Test likely_packaged(). ... ok
Test make_report_path(). ... ok
Test get_new_reports() and seen_report(). ... ok

----------------------------------------------------------------------
Ran 10 tests in 2.198s

OK
+ python -tt /usr/share/python-support/python-apport/apport/report.py -v
Test add_gdb_info() with core dump file reference. ... Failed to read a valid object file image from memory.
warning: Memory read failed for corefile section, 4096 bytes at 0xffffe000.
ok
Test add_gdb_info() with inline core dump. ... Failed to read a valid object file image from memory.
warning: Memory read failed for corefile section, 4096 bytes at 0xffffe000.
ok
Test add_gdb_info() with a script. ... Failed to read a valid object file image from memory.
warning: Memory read failed for corefile section, 4096 bytes at 0xffffe000.
FAIL
Test add_hooks_info(). ... ok
Test add_os_info(). ... ok
Test add_package_info(). ... ok
Test add_proc_info(). ... ok
Test add_user_info(). ... ok
Test _check_interpreted(). ... ok
Test crash_signature(). ... ok
Test _gen_stacktrace_top(). ... ok
Test has_useful_stacktrace(). ... ok
Test mark_ignore() and check_ignored(). ... ok
Test obsolete_packages(). ... ok
Test search_bug_patterns(). ... ok
Test standard_title(). ... ok

======================================================================
FAIL: Test add_gdb_info() with a script.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/share/python-support/python-apport/apport/report.py", line 1...

[Shirish Agarwal (shirishag75)]

After talking with martin pitti I removed the core file :-

shirish@ubuntu:~$ rm core
shirish@ubuntu:~$ /usr/share/apport/testsuite/test-apport kernel 2>&1 | tee /tmp/test-kernel.out
* Check test process creation/killing with apport
* Check that a subsequent crash does not alter unseen report
* Check that a subsequent crash alters seen report
* Check that report has required fields
* Check that dumped environment only has insensitive variables
* Check that collected system groups has nonempty user groups information
* Check that collected system groups are not those from root
* Check that non-packaged executables do not create a report
* Check that apport ignores SIGQUIT
* Check that existence of user-inaccessible files does not leak
* Check that non-packaged scripts do not create a report
* Test limitation of flooding: iteration 0 1 2
* Check that core dump works for non-writable cwds
Traceback (most recent call last):
  File "/usr/share/apport/testsuite/test-apport", line 296, in <module>
    check_crash()
  File "/usr/share/apport/testsuite/test-apport", line 60, in check_crash
    assert not os.path.exists('core'), 'no core dump in current directory'
AssertionError: no core dump in current directory
shirish@ubuntu:~$

There is a 1040 bytes test-kernel.out which I am attaching

[Shirish Agarwal (shirishag75)]

Here's the /var/log/apport.log for that session.

[Martin Pitt (pitti)]

Thanks for the logs so far. So we just found out that bug 74691 is the reason why 'run-tests' fails on i386. It works fine on amd64 and powerpc (the platforms I'm testing on).

I will install i386 now to have a look at this myself.

[Shirish Agarwal (shirishag75)]

shirish@ubuntu:~$ LC_ALL=en_US apt-cache policy linux-image-`uname -r`
linux-image-2.6.22-7-generic:
  Installed: 2.6.22-7.14
  Candidate: 2.6.22-7.14
  Version table:
 *** 2.6.22-7.14 0
        500 http://in.archive.ubuntu.com gutsy/main Packages
        100 /var/lib/dpkg/status

[Shirish Agarwal (shirishag75)]

this is output from /tmp

shirish@ubuntu:~$ /usr/share/apport/testsuite/test-apport kernel 2>&1 | tee /tmp/test-kernel.out
Traceback (most recent call last):
  File "/usr/share/apport/testsuite/test-apport", line 147, in <module>
    assert apport.fileutils.get_all_reports() == [], 'no reports already present'
AssertionError: no reports already present

[Shirish Agarwal (shirishag75)]

after talking with pitti, I did the following things :-

1. go to /var/crash :- rm *
2. then copied test apport to /tmp
 cp /usr/share/apport/testsuite/test-apport /tmp
3. ran the test more than 15 times /tmp/test-apport kernel 2>&1 | tee /tmp/test-kernel.out

I get the same output as in https://bugs.launchpad.net/ubuntu/+source/apport/+bug/122688/comments/20

[Martin Pitt (pitti)]

BTW, the effect of filing useless bugs is mitigated with

apport (0.86) gutsy; urgency=low

  * test-apport: Check that apport does not create reports for emtpy core
    dumps.
  * problem_report.py: Introduce a fourth optional parameter "fail_on_empty"
    to file pointer tuples which causes write() to raise an IOError if no data
    was read. Add test cases.
  * bin/apport: Enforce non-emptyness of CoreDump.
  * problem_report.py: Add test case for delayed piping of data passed as file
    object pointers. This was supposed to explain the reason for getting bugs
    with zero-byte core dumps, but already works correctly.
  * apport/report.py, check_ignored(): round the mtime to an int (just like
    mark_ignore() does), to not get wrong results on file systems that support
    subsecond file timestamps. This fixes running the test suite on the live
    CD.
  * test-apport: Clarify assertion message if /var/crash is not empty.

 -- Martin Pitt <email address hidden> Thu, 28 Jun 2007 19:14:36 +0200

It does not actually solve the original problem, though, that apport sometimes generates crash reports with empty core dumps.

[Martin Pitt (pitti)]

Removing milestone, since it does not generate bug spam any more. It does, however, prevent bugs to be filed under some circumstances.

[miked (miked11)]

root@2HewittRand-desktop:~# /usr/share/apport/testsuite/test-apport kernel 2>&1 | tee /tmp/test-kernel.out
* Check that empty core dumps do not generate a report
* Check test process creation/killing with apport
* Check that a subsequent crash does not alter unseen report
* Check that a subsequent crash alters seen report
* Check that report has required fields
* Check that dumped environment only has insensitive variables
* Check that collected system groups has nonempty user groups information
* Check that collected system groups are not those from root
* Check that only one apport instance is ran at a time
* Check that existing .lock file as dangling symlink does not create the file
* Check that non-packaged executables do not create a report
* Check that apport ignores SIGQUIT
* Check that existence of user-inaccessible files does not leak
* Check that non-packaged scripts do not create a report
* Test limitation of flooding: iteration 0 1 2
* Check that core dump works for non-writable cwds
* Check that non-packaged executables create core dumps on proper ulimits
* Check that non-packaged executables create core dumps on proper ulimits for SIGABRT
* Check that packaged executables create core dumps on proper ulimits
* Check that crashes create core dumps with an existing crash report
* Check that packaged executables create core dumps on proper ulimits for SIGABRT
* Check that core dumps are capped on available memory size....
* Check that binary blacklisting works
root@2HewittRand-desktop:~#
root@2HewittRand-desktop:~# gksudo cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=8.04
DISTRIB_CODENAME=hardy
DISTRIB_DESCRIPTION="Ubuntu hardy (development branch)"
root@2HewittRand-desktop:~#

[miked (miked11)]

Added Info

[Martin Pitt (pitti)]

miked, your run looks good, the test suite passed. I haven't seen this anywhere else on Hardy any more, so I'm closing this.

[miked (miked11)]

cool, thanks.

[linuxatico88 (simo88-47)]

(gdb) info registers
eax 0xfffffdfc -516
ecx 0x6 6
edx 0x63 99
ebx 0x87d1398 142414744
esp 0xbfdeb484 0xbfdeb484
ebp 0xbfdeb4a8 0xbfdeb4a8
esi 0x63 99
edi 0xb6e84ff4 -1226289164
eip 0xb7f29410 0xb7f29410 <__kernel_vsyscall+16>
eflags 0x200246 [ PF ZF IF ID ]
cs 0x73 115
ss 0x7b 123
ds 0x7b 123
es 0x7b 123
fs 0x0 0
gs 0x33 51
(gdb) thread apply all backtrace

Thread 9 (Thread 0xb2fd4b90 (LWP 9479)):
#0 0xb7f29410 in __kernel_vsyscall ()
#1 0xb6e06697 in poll () from /lib/tls/i686/cmov/libc.so.6
#2 0xb6f2d176 in ?? () from /usr/lib/libglib-2.0.so.0
#3 0x0888b0b0 in ?? ()
#4 0x00000005 in ?? ()
#5 0xffffffff in ?? ()
#6 0x0888b0b0 in ?? ()
#7 0x00000005 in ?? ()
#8 0xb6fa05f8 in ?? () from /usr/lib/libglib-2.0.so.0
#9 0xb6fa0620 in ?? () from /usr/lib/libglib-2.0.so.0
#10 0xb2fd42d4 in ?? ()
#11 0x00000001 in ?? ()
#12 0x00000001 in ?? ()
#13 0x088599f0 in ?? ()
#14 0x0888b0b0 in ?? ()
#15 0xb6e06620 in ?? () from /lib/tls/i686/cmov/libc.so.6
#16 0xb7c98df0 in ?? () from /lib/tls/i686/cmov/libpthread.so.0
#17 0xb7c97520 in ?? () from /lib/tls/i686/cmov/libpthread.so.0
#18 0xb6f51a62 in g_thread_self () from /usr/lib/libglib-2.0.so.0
#19 0xb6f2d527 in g_main_loop_run () from /usr/lib/libglib-2.0.so.0
#20 0xb77d2ce0 in ?? () from /usr/lib/libORBit-2.so.0
#21 0x088979c0 in ?? ()
#22 0xb6fa0248 in ?? () from /usr/lib/libglib-2.0.so.0
#23 0xb2fd4358 in ?? ()
#24 0xb6f51fef in ?? () from /usr/lib/libglib-2.0.so.0
#25 0x00000000 in ?? ()

Thread 7 (Thread 0xb37d5b90 (LWP 9447)):
#0 0xb7f29410 in __kernel_vsyscall ()
#1 0xb7c99aa5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/cmov/libpthread.so.0
#2 0xb6f04e52 in ?? () from /usr/lib/libglib-2.0.so.0
#3 0xb38012d0 in ?? ()
#4 0x081fe560 in ?? ()
#5 0xb37d5298 in ?? ()
#6 0xb6fa0248 in ?? () from /usr/lib/libglib-2.0.so.0
#7 0xb38016d8 in ?? ()
#8 0x00000000 in ?? ()

Thread 1 (Thread 0xb6526720 (LWP 9355)):
#0 0xb7f29410 in __kernel_vsyscall ()
#1 0xb6e06697 in poll () from /lib/tls/i686/cmov/libc.so.6
#2 0xb6f2d176 in ?? () from /usr/lib/libglib-2.0.so.0
#3 0x087d1398 in ?? ()
#4 0x00000006 in ?? ()
#5 0x00000063 in ?? ()
#6 0x087d1398 in ?? ()
#7 0x00000009 in ?? ()
#8 0xb6fa05f8 in ?? () from /usr/lib/libglib-2.0.so.0
#9 0xb6fa0620 in ?? () from /usr/lib/libglib-2.0.so.0
#10 0xbfdeb4f4 in ?? ()
#11 0x00000001 in ?? ()
#12 0x00000001 in ?? ()
#13 0x0814e480 in ?? ()
#14 0x087d1398 in ?? ()
#15 0xb6e06620 in ?? () from /lib/tls/i686/cmov/libc.so.6
#16 0xb7c98df0 in ?? () from /lib/tls/i686/cmov/libpthread.so.0
#17 0xb7c97520 in ?? () from /lib/tls/i686/cmov/libpthread.so.0
#18 0xb6f51a62 in g_thread_self () from /usr/lib/libglib-2.0.so.0
#19 0xb6f2d527 in g_main_loop_run () from /usr/lib/libglib-2.0.so.0
#20 0xb7534044 in gtk_main () from /usr/lib/libgtk-x11-2.0.so.0
#21 0x080627b0 in main ()

[Martin Pitt (pitti)]

linuxatico88, you attached a test suite run which is invalid because you ran it as root. Please run it as normal user.

Do you also have the problem? Your reply did not have any explanations or context.

[Craig Maloney (craig-decafbad)]

I can confirm this is an issue with an upgrade from Gutsy to Hardy. I am experiencing exactly the same behavior mentioned above. I'm not receiving any crash reports, even though apport is supposedly running.

[Martin Pitt (pitti)]

Hi,

Craig Maloney [2008-05-31 3:34 -0000]:
> I can confirm this is an issue with an upgrade from Gutsy to Hardy. I am
> experiencing exactly the same behavior mentioned above. I'm not
> receiving any crash reports, even though apport is supposedly running.

NB that apport is turned off by default in Hardy final, this time by
completely disabling it in /etc/default/apport. This avoids wasting a
lot of disk access and delays after a program crashed.

So if you do not get any crash report at all, it is behaving as
intended. If you do get crash reports, but the core dump is empty, you
triggered this bug.

[Craig Maloney (craig-decafbad)]

I agree, but this is the settings in /etc/default/apport. I'm not sure how more enabled I can make it, yet I'm still not receiving anything in /var/crash when I crash an application.

# set this to 0 to disable apport, or to 1 to enable it
enabled=1

# set maximum core dump file size (default: 209715200 bytes == 200 MB)
maxsize=209715200

[Martin Pitt (pitti)]

Craig Maloney [2008-06-03 2:05 -0000]:
> enabled=1

OK, that should work. After you get a crash, what does
/var/log/apport.log contain? You can easily generate a "fake" crash
with opening a Terminal and typing

  bash -c 'kill -SEGV $$'

[Craig Maloney (craig-decafbad)]

OK, that seemed to work. It must be something else then, because I'm trying to get a crash report for Rhythmbox, and while the app disappears repeatedly when I perform a certain task, I can't seem to generate a crash dump off of that.

Thanks for the pointers!