mongodb fails assertion on ppc64el with 64k pagesize

[James Page]

The output of running the command is:

ubuntu@rockne-02:/etc/init$ sudo /usr/bin/mongod --auth
--dbpath=/home/ubuntu/.juju/local/db --sslOnNormalPorts --sslPEMKeyFile
'/home/ubuntu/.juju/local/server.pem' --sslPEMKeyPassword ignored
--bind_ip 0.0.0.0 --port 37017 --noprealloc --smallfiles
warning: bind_ip of 0.0.0.0 is unnecessary; listens on all ips by default
note: noprealloc may hurt performance in many applications
Wed Mar 19 01:05:53.146 [initandlisten] MongoDB starting : pid=31599
port=37017 dbpath=/home/ubuntu/.juju/local/db 64-bit host=rockne-02
Wed Mar 19 01:05:53.146 [initandlisten] db version v2.4.9
Wed Mar 19 01:05:53.146 [initandlisten] git version: nogitversion
Wed Mar 19 01:05:53.146 [initandlisten] build info: Linux fisher01
3.13.0-8-generic #28-Ubuntu SMP Tue Feb 11 17:58:07 UTC 2014 ppc64le
BOOST_LIB_VERSION=1_54
Wed Mar 19 01:05:53.146 [initandlisten] allocator: tcmalloc
Wed Mar 19 01:05:53.147 [initandlisten] options: { auth: true, bind_ip:
"0.0.0.0", dbpath: "/home/ubuntu/.juju/local/db", noprealloc: true,
port: 37017, smallfiles: true, sslOnNormalPorts: true, sslPEMKeyFile:
"/home/ubuntu/.juju/local/server.pem", sslPEMKeyPassword: "<password>" }
Wed Mar 19 01:05:53.148 [initandlisten] journal
dir=/home/ubuntu/.juju/local/db/journal
Wed Mar 19 01:05:53.148 [initandlisten] recover : no journal files
present, no recovery needed
Wed Mar 19 01:05:53.150 [initandlisten] Fatal Assertion 16143
0x106d0848 0x106a27f8 0x106833a8 0x106a7f28 0x1035fdfc 0x103600fc
0x10360b14 0x10354e20 0x101a1a54 0x101a3838 0x10179138 0x3fffada74b80
0x3fffada74d78
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x58) [0x106d0848]
 /usr/bin/mongod(_ZN5mongo10logContextEPKc+0x98) [0x106a27f8]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa8) [0x106833a8]
 /usr/bin/mongod(_ZN5mongo7LogFile17synchronousAppendEPKvm+0x358)
[0x106a7f28]
 /usr/bin/mongod(_ZN5mongo3dur7Journal5_openEv+0x27c) [0x1035fdfc]
 /usr/bin/mongod(_ZN5mongo3dur7Journal4openEv+0x5c) [0x103600fc]
 /usr/bin/mongod(_ZN5mongo3dur16preallocateFilesEv+0x94) [0x10360b14]
 /usr/bin/mongod(_ZN5mongo3dur7startupEv+0x60) [0x10354e20]
 /usr/bin/mongod(_ZN5mongo14_initAndListenEi+0x974) [0x101a1a54]
 /usr/bin/mongod(_ZN5mongo13initAndListenEi+0x28) [0x101a3838]
 /usr/bin/mongod(main+0x368) [0x10179138]
 /lib/powerpc64le-linux-gnu/libc.so.6(+0x24b80) [0x3fffada74b80]
 /lib/powerpc64le-linux-gnu/libc.so.6(__libc_start_main+0xc8)
[0x3fffada74d78]
Wed Mar 19 01:05:53.155 [initandlisten]

The line of code is here:

https://github.com/mongodb/mongo/blob/v2.6/src/mongo/util/logfile.cpp#L241

and is:

void LogFile::synchronousAppend(const void *b, size_t len) {
  const char *buf = static_cast<const char *>( b );
  ssize_t charsToWrite = static_cast<ssize_t>( len );

  fassert( 16144, charsToWrite >= 0 );
  fassert( 16142, _fd >= 0 );
  fassert( 16143, reinterpret_cast<ssize_t>( buf ) %
g_minOSPageSizeBytes == 0 ); // aligned

It is the last line there that is asserting. It seems that something is
misalligned at the OS level. It could well be that the VM is emulating
power 8, and is doing something weird.

[James Page (james-page)]

It seems the pages size of rockne-02 is

ubuntu@rockne-02:~/go/src/launchpad.net/juju-core$ getconf PAGESIZE
65536

but winton-02 has:

ubuntu@winton-02:~$ getconf PAGESIZE
4096

this may be it... no idea why mongo does the alignment check anyway,
seems like crack

[James Page (james-page)]

for context this problem is seen on rockne-02, but not on winton-02.

[Mark Ramm (mark-ramm)]

This bug is also impacting the juju-mongodb package, and therefore juju on ppc64el.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Adam Conrad (adconrad)]

This should fixed in mongodb 1:2.4.9-1ubuntu2 and juju-mongodb 2.4.9-0ubuntu3

[Antonio Rosales (arosales)]

This looks to be fixed in recent testing of Juju on the latest mongodb:

ubuntu@wolfe-01:~$ dpkg -l | grep mongo
ii juju-mongodb 2.4.9-0ubuntu3 ppc64el MongoDB object/document-oriented database for Juju
ubuntu@wolfe-01:~$ getconf PAGESIZE
65536
ubuntu@wolfe-01:~$ juju --version
1.17.6-trusty-ppc64
ubuntu@wolfe-01:~$ juju switch
local
ubuntu@wolfe-01:~$ juju bootstrap
. . .
Starting MongoDB server (juju-db-ubuntu-local)
Bootstrapping Juju machine agent
Starting Juju machine agent (juju-agent-ubuntu-local)
2014-03-28 00:25:32 INFO juju.cmd supercommand.go:306 command finished
ubuntu@wolfe-01:~$ echo $?
0
ubuntu@wolfe-01:~/charms/trusty$ juju stat
environment: local
machines:
  "0":
    agent-state: started
    agent-version: 1.17.6.1
    dns-name: localhost
    instance-id: localhost
    series: trusty
services: {}

I was also able to sucessfully deploy Apache2 and HAProxy with Juju in the same enviornment.

-thanks,
Antonio

[Kalpana S Shetty (kalshett)]

I'm using mongodb 2.6.2 code base and hit the same issue on Ubuntu LE 14.10 VM. Could someone tell me what is the fix go into mongodb?