[arm64] gccgo runtime crashes with CONFIG_ARM64_PGTABLE_LEVELS=4

[dann frazier]

[Impact]
Various go-based packages crash on startup on arm64 when booted with a kernel where CONFIG_ARM64_PGTABLE_LEVELS=4. LXD, docker and juju are examples.

Ubuntu does not ship with this config, so it is currently not impacted. However, we would like to enable this config to add new hardware support in 15.10. The 15.10 kernel will eventually be provided in trusty as an lts backport, at which point this will begin to break users.

[Test Case]
Boot an arm64 system with CONFIG_ARM64_PGTABLE_LEVELS=4 enabled in the kernel. Install docker.io and run "docker help". If it fails, you'll see "fatal error: runtime_lfstackpush" followed by a stack trace. If it succeeds, you'll get normal help output.

[Regression Risk]
The risk of regression is lessened by testing this release on existing Ubuntu kernels w/ 3 level page tables.

[dann frazier (dannf)]

This issue persists with lxd 0.13-0ubuntu1. Note that it doesn't fail on all arm64 systems, but failures seems to be correlated w/ a high core count.

[Michael Hudson-Doyle (mwhudson)]

The fix I've sent upstream is https://go-review.googlesource.com/#/c/13037/. It hasn't been reviewed yet, but I've confirmed it works to my own satisfaction on arm64.

[Michael Hudson-Doyle (mwhudson)]

Filed upstream as https://gcc.gnu.org/bugzilla/show_bug.cgi?id=67508

[Launchpad Janitor (janitor)]

This bug was fixed in the package gcc-5 - 5.2.1-17ubuntu1

---------------
gcc-5 (5.2.1-17ubuntu1) wily; urgency=medium

  * Merge with Debian; remaining changes:
    - Build from upstream sources.

 -- Matthias Klose <email address hidden> Fri, 11 Sep 2015 03:24:48 +0200

[Michael Hudson-Doyle (mwhudson)]

Here's a patch for trusty, based on the gccgo-4.9 from ~ubuntu-toolchain-r/+archive/ubuntu/ppa: http://people.canonical.com/~mwh/gccgo-4.9_4.9.3-0ubuntu3_4.9.3-0ubuntu4.diff (not tested yet).

[Michael Hudson-Doyle (mwhudson)]

Oh yes, and dann and I confirmed yesterday that the bug is no longer present in wily.

[Brian Murray (brian-murray)]

Hello dann, or anyone else affected,

Accepted gccgo-5 into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/gccgo-5/5.1~rc1-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[dann frazier (dannf)]

= vivid verification =

# My kernel has the problematic config
ubuntu@cvm1:~$ grep CONFIG_ARM64_PGTABLE_LEVELS /boot/config-$(uname -r)
CONFIG_ARM64_PGTABLE_LEVELS=4

# Build a clean vivid chroot
ubuntu@cvm1:~$ mkdir lp1472650
ubuntu@cvm1:~$ sudo debootstrap --components 'main,universe' --include docker.io vivid lp1472650 http://ports.ubuntu.com/ubuntu-ports > /dev/null 2>&1

# docker needs proc to be mounted
ubuntu@cvm1:~$ sudo mount proc lp1472650/proc -t proc
ubuntu@cvm1:~$ sudo chroot lp1472650 docker help
p=0xffff88459000
fatal error: runtime_lfstackpush: invalid pointer
[--trim--]

# It's buggy. Let's install the gccgo update
ubuntu@cvm1:~$ echo 'deb http://ports.ubuntu.com/ubuntu-ports vivid-proposed main universe' | sudo tee -a lp1472650/etc/apt/sources.list > /dev/null
ubuntu@cvm1:~$ sudo chroot lp1472650 apt-get update > /dev/null 2>&1
ubuntu@cvm1:~$ sudo chroot lp1472650 apt-get install libgo7
Reading package lists... Done
Building dependency tree... Done
The following extra packages will be installed:
  gcc-5-base libgcc1
The following packages will be upgraded:
  gcc-5-base libgcc1 libgo7
3 upgraded, 0 newly installed, 0 to remove and 8 not upgraded.
Need to get 6243 kB of archives.
After this operation, 2048 B of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://ports.ubuntu.com/ubuntu-ports/ vivid-proposed/universe libgo7 arm64 5.1~rc1-0ubuntu1.1 [6198 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports/ vivid-proposed/main gcc-5-base arm64 5.1~rc1-0ubuntu1.1 [14.4 kB]
Get:3 http://ports.ubuntu.com/ubuntu-ports/ vivid-proposed/main libgcc1 arm64 1:5.1~rc1-0ubuntu1.1 [31.0 kB]
Fetched 4733 kB in 28s (166 kB/s)
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
 LANGUAGE = (unset),
 LC_ALL = (unset),
 LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
E: Can not write log (Is /dev/pts mounted?) - posix_openpt (2: No such file or directory)
(Reading database ... 11950 files and directories currently installed.)
Preparing to unpack .../libgo7_5.1~rc1-0ubuntu1.1_arm64.deb ...
Unpacking libgo7:arm64 (5.1~rc1-0ubuntu1.1) over (5.1~rc1-0ubuntu1) ...
Preparing to unpack .../gcc-5-base_5.1~rc1-0ubuntu1.1_arm64.deb ...
Unpacking gcc-5-base:arm64 (5.1~rc1-0ubuntu1.1) over (5.1~rc1-0ubuntu1) ...
Setting up gcc-5-base:arm64 (5.1~rc1-0ubuntu1.1) ...
(Reading database ... 11950 files and directories currently installed.)
Preparing to unpack .../libgcc1_1%3a5.1~rc1-0ubuntu1.1_arm64.deb ...
Unpacking libgcc1:arm64 (1:5.1~rc1-0ubuntu1.1) over (1:5.1~rc1-0ubuntu1) ...
Setting up libgcc1:arm64 (1:5.1~rc1-0ubuntu1.1) ...
Setting up libgo7:arm64 (5.1~rc1-0ubuntu1.1) ...
Processing triggers for libc-bin (2.21-0ubuntu4) ...
ubuntu@cvm1:~$ sudo chroot lp1472650 docker help
Usage: docker [OPTIONS] COMMAND [arg...]

A self-sufficient runt...

[Launchpad Janitor (janitor)]

This bug was fixed in the package gccgo-5 - 5.1~rc1-0ubuntu1.1

---------------
gccgo-5 (5.1~rc1-0ubuntu1.1) vivid; urgency=medium

  * libgo: Fix PR go/67508, rewrite lfstack packing/unpacking to look more
    like that in Go (Michael Hudson). LP: #1472650.

 -- dann frazier <email address hidden> Wed, 16 Sep 2015 13:56:48 +0000

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for gccgo-5 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Steve Langasek (vorlon)]

Hello dann, or anyone else affected,

Accepted gccgo-4.9 into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/gccgo-4.9/4.9.3-0ubuntu4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[dann frazier (dannf)]

Verified:

ubuntu@cvm2s:~$ juju help
p=0xffffa6947000
fatal error: runtime_lfstackpush: invalid pointer

runtime stack:

 :0

 :0

 :0

 :0

 :0

 :0

 :0

 :0

 :0

 :0

goroutine 1 [garbage collection]:
github.com_juju_cmd.DefaultContext
 /build/juju-core-qiO5uV/juju-core-1.24.7/src/github.com/juju/cmd/cmd.go:261
main.Main
 /build/juju-core-qiO5uV/juju-core-1.24.7/src/github.com/juju/juju/cmd/juju/main.go:54
main.main
 /build/juju-core-qiO5uV/juju-core-1.24.7/src/github.com/juju/juju/cmd/juju/main.go:233

goroutine 3 [syscall]:
 goroutine in C code; stack unavailable

goroutine 12 [runnable]:
ubuntu@cvm2s:~$ sudo apt-get install libgo5 -y
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  devio libuuid-perl linux-base
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  gcc-4.9-base libgcc1
The following packages will be upgraded:
  gcc-4.9-base libgcc1 libgo5
3 upgraded, 0 newly installed, 0 to remove and 17 not upgraded.
Need to get 2,337 kB of archives.
After this operation, 17.4 kB of additional disk space will be used.
Get:1 http://ports.ubuntu.com/ubuntu-ports/ trusty-proposed/main libgo5 arm64 4.9.3-0ubuntu4 [2,291 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports/ trusty-proposed/main gcc-4.9-base arm64 4.9.3-0ubuntu4 [15.2 kB]
Get:3 http://ports.ubuntu.com/ubuntu-ports/ trusty-proposed/main libgcc1 arm64 1:4.9.3-0ubuntu4 [30.7 kB]
Fetched 2,337 kB in 11s (212 kB/s)
(Reading database ... 70242 files and directories currently installed.)
Preparing to unpack .../libgo5_4.9.3-0ubuntu4_arm64.deb ...
Unpacking libgo5:arm64 (4.9.3-0ubuntu4) over (4.9.1-0ubuntu1) ...
Preparing to unpack .../gcc-4.9-base_4.9.3-0ubuntu4_arm64.deb ...
Unpacking gcc-4.9-base:arm64 (4.9.3-0ubuntu4) over (4.9.1-0ubuntu1) ...
Setting up gcc-4.9-base:arm64 (4.9.3-0ubuntu4) ...
(Reading database ... 70242 files and directories currently installed.)
Preparing to unpack .../libgcc1_1%3a4.9.3-0ubuntu4_arm64.deb ...
Unpacking libgcc1:arm64 (1:4.9.3-0ubuntu4) over (1:4.9.1-0ubuntu1) ...
Setting up libgcc1:arm64 (1:4.9.3-0ubuntu4) ...
Setting up libgo5:arm64 (4.9.3-0ubuntu4) ...
Processing triggers for libc-bin (2.19-0ubuntu6.6) ...
ubuntu@cvm2s:~$ juju help
Juju -- devops distilled
https://juju.ubuntu.com/

Juju provides easy, intelligent service orchestration on top of environments
such as Amazon EC2, HP Cloud, OpenStack, MaaS, or your own local machine.

Basic commands:
  juju init generate boilerplate configuration for juju environments
  juju bootstrap start up an environment from scratch

  juju deploy deploy a new service
  juju add-relation add a relation between two services
  juju expose expose a service

  juju help bootstrap more help on e.g. bootstrap command
  juju help commands list all commands
  juju help glossary glossary of terms
  juju help topics list all help topics

Provider information:
  juju help azure-provider use on Windows Azure
  juju help ec2-provider use on Amazon EC2...

[Launchpad Janitor (janitor)]

This bug was fixed in the package gccgo-4.9 - 4.9.3-0ubuntu4

---------------
gccgo-4.9 (4.9.3-0ubuntu4) trusty-proposed; urgency=medium

  * Upload the final GCC 4.9.3 release.
  * Merge changes from gcc-4.9 4.9.3-2ubuntu1, including:
    - Fix PR go/60931, garbage collector issue with non 4kB system page size.
      LP: #1304754.
    - Fix wrong-code issue in the little endian vector API (ppc64el).
      LP: #1311128.
    - Fix ABI incompatibility between POWER and Z HTM builtins and intrinsics.
      LP: #1320292.
    - Fix an ICE with invalid code. PR c++/61046. LP: #1313102.
    - gccgo: Don't overwrite memory if an archive has a bad file name.
    - Fix PR rtl-optimization/61801, sched2 miscompiles syscall sequence with -g.
    - libgo: Fix unexpected GC interfering with closure passing.
    - libgo: Fix PR other/61895, ignore small argv[0] file for backtrace.
    - libgo: Fix unexpected GC interfering with closure passing.
    - Fix ICE in the Go frontend. LP: #1362906.
    - Enable cgo on AArch64 (Michael Hudson). LP: #1361940.
    - libgo: runtime: Use the clone system call on GNU/Linux.
    - libgo: Support arm64 relocations (Michael Hudson).
  * Include the cc1 binary into the gccgo-4.9 package.
  * Do not build-depend on sdt-systemtap for the trusty upload.
  * Warn about ppc ELFv2 ABI issues, which will change in GCC 5.
  * libgo: Fix PR go/67508, rewrite lfstack packing/unpacking to look more
    like that in Go (Michael Hudson). LP: #1472650.

gcc-4.9 (4.9.3-2ubuntu1) wily; urgency=medium

  * Merge with Debian; remaining changes:
    - Build from upstream sources.

gcc-4.9 (4.9.3-2) unstable; urgency=medium

  * Update to SVN 20150714 (r225765) from the gcc-4_9-branch.
  * Update the Linaro support to the 4.9-2015.06 release.

gcc-4.9 (4.9.3-1ubuntu1) wily; urgency=medium

  * Merge with Debian; remaining changes:
    - Build from upstream sources.

gcc-4.9 (4.9.3-1) unstable; urgency=medium

  * GCC 4.9.3 release.
  * Update to SVN 20150629 (r225135) from the gcc-4_9-branch.
  * Refresh patches.

gcc-4.9 (4.9.2-22ubuntu1) wily; urgency=medium

  * Merge with Debian; remaining changes:
    - Build from upstream sources.

gcc-4.9 (4.9.2-22) unstable; urgency=medium

  * Update to SVN 20150623 (r224833) from the gcc-4_9-branch.
  * Fix PR target/66483, taken from the trunk. Closes: #787689.
  * Fix "empty-binary-package" lintian warnings.
  * Fix PR tree-optimization/66233. Closes: #788812.

gcc-4.9 (4.9.2-21) unstable; urgency=medium

  * Update to SVN 20150611 (r224436) from the gcc-4_9-branch.
  * Build libstdc++6 when building the common libraries.

gcc-4.9 (4.9.2-20ubuntu1) wily; urgency=medium

  * Merge with Debian; remaining changes:
    - Build from upstream sources.

gcc-4.9 (4.9.2-20) unstable; urgency=medium

  * Update to SVN 20150531 (r223898) from the gcc-4_9-branch.
  * Fix PR rtl-optimization/63843, taken from the trunk. Closes: #785475.

gcc-4.9 (4.9.2-19ubuntu1) wily; urgency=medium

  * Merge with Debian; remaining changes:
    - Build from upstream sources.

gcc-4.9 (4.9.2-19) unstable; urgency=medium

  * Update to SVN 20150529 (r223861) from the gcc-4_9-branch.
  * Make symbols file symlinking for cross bu...

[dann frazier (dannf)]

Marking as impacting juju-core/trusty. This is easily reproduced by attempting to bootstrap a juju-local environment in trusty with juju-core 1.24.7-0ubuntu1~14.04.1 when using a kernel w/ 4 level page tables. This will be a supported config once the xenial-lts kernel is available in trusty.

A no-change rebuild of juju-core trusty would be sufficient to resolve this issue.

ubuntu@cvm2s2:~$ juju bootstrap --verbose
Bootstrapping environment "local"
Starting new instance for initial state server
Building tools to upload (1.24.7.1-trusty-arm64)
Bootstrap failed, destroying environment
ERROR failed to bootstrap environment: cannot upload bootstrap tools: cannot get version from "/tmp/juju-tools525417221/jujud": exit status 2; p=0xffff9354f000
fatal error: runtime_lfstackpush: invalid pointer

runtime stack:

 :0

 :0

 :0

 :0

 :0

 :0

 :0

 :0

 :0

 :0

goroutine 1 [garbage collection]:
github.com_juju_cmd.DefaultContext
 /build/juju-core-qiO5uV/juju-core-1.24.7/src/github.com/juju/cmd/cmd.go:261
main.Main
 /build/juju-core-qiO5uV/juju-core-1.24.7/src/github.com/juju/juju/cmd/jujud/main.go:166
main.MainWrapper
 /build/juju-core-qiO5uV/juju-core-1.24.7/src/github.com/juju/juju/cmd/jujud/main.go:151
main.main
 /build/juju-core-qiO5uV/juju-core-1.24.7/src/github.com/juju/juju/cmd/jujud/main_nix.go:13

goroutine 3 [syscall]:
 goroutine in C code; stack unavailable

[Anastasia (anastasia-macmood)]

From mwhudson: we don't use gccgo on arm64 any more.