ISST-LTE:pVM: golang does not support huge amount (>256) CPUs machines (as E870)

[bugproxy]

Docker and golang packages are not able to run on e870 machine due to a lot of CPU on these machines.

The current machine has 400 CPUs and shows the following error:

runtime stack:
runtime.throw(0x654c60, 0x17)
        /usr/lib/go/src/runtime/panic.go:530 +0x8c
runtime.procresize(0x190, 0x0)
        /usr/lib/go/src/runtime/proc.go:3158 +0xbd4
runtime.schedinit()
        /usr/lib/go/src/runtime/proc.go:452 +0x140
runtime.rt0_go(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2, 0x3fffd512de59, ...)
        /usr/lib/go/src/runtime/asm_ppc64x.s:72 +0xac
/var/lib/dpkg/info/golang-go.postinst: line 18: 100 + : syntax error: operand expected (error token is "+ ")
dpkg: error processing package golang-go (--configure):
 subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of golang-doc:
 golang-doc depends on golang-go; however:
  Package golang-go is not configured yet.

dpkg: error processing package golang-doc (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of golang:
 golang depends on golang-doc (>= 2:1.6-1ubuntu1); however:
  Package golang-doc is not configured yet.
 golang depends on golang-go (>= 2:1.6-1ubuntu1); however:
  Package golang-go is not configured yet.

dpkg: error processing package golang (--configure):
 dependency problems - leaving unconfigured
No apport report written because the error message indicates its a followup error from a previous failure.
                                                                                                          No apport report written because the error message indicates its a followup error from a previous failure.
                                                      Errors were encountered while processing:
 golang-go
 golang-doc
 golang
E: Sub-process /usr/bin/dpkg returned an error code (1)

Looking at the code I see:

_MaxGomaxprocs = 1 << 8

        if old < 0 || old > _MaxGomaxprocs || nprocs <= 0 || nprocs > _MaxGomaxprocs {
                throw("procresize: invalid arg")
        }

If I decrease the amount of CPU to < 256, I don't see a problem.

[bugproxy (bugproxy)]

Default Comment by Bridge

[Breno Leitão (breno-leitao)]

I am going to ask to send the following patch upstream. Let see what the upstream community will say about it

commit 17857791d91cf31944d266cb1dfd9322cea8933f
Author: Breno Leitao <email address hidden>
Date: Mon Apr 4 16:09:55 2016 -0400

    Increase _MaxGomaxprocs to support big machines

    Currently go does not support machines that contains > 256 CPUs, as IBM's E880
    that could host 1536 hardware thread, when configured with 192 CPU cores and
    SMT (Symmetric Multi Thread) 8.

    For example, when running a go program on this machine, I got, the following
    problem[1]:

      procresize: invalid arg

    This is because of the following code:

      _MaxGomaxprocs = 1 << 8

      if old < 0 || old > _MaxGomaxprocs || nprocs <= 0 || nprocs > _MaxGomaxprocs {
          throw("procresize: invalid arg")
      }

    This patch just redefine _MAxGomaxprocs to 1 << 12.

diff --git a/src/runtime/runtime2.go b/src/runtime/runtime2.go
index e0137f7..508cecb 100644
--- a/src/runtime/runtime2.go
+++ b/src/runtime/runtime2.go
@@ -472,7 +472,7 @@ type p struct {
 const (
  // The max value of GOMAXPROCS.
  // There are no fundamental restrictions on the value.
- _MaxGomaxprocs = 1 << 8
+ _MaxGomaxprocs = 1 << 12
 )

 type schedt struct {

[Michael Hudson-Doyle (mwhudson)]

Yes, this should be fixed upstream, or at least discussed there before we fix this in Ubuntu.

For clarity, which go packages are you installing on which release? I don't understand why postinst is failing.

[bugproxy (bugproxy)]

------- Comment on attachment From <email address hidden> 2016-04-05 09:57 EDT-------

Issue is seen on Ubuntu16.04 , kernel: 4.4.0-16-generic

Following are go packages got installed, but docker installation is failing still:
---------------
root@dockerlp1:~# dpkg -l |grep -i go
ii gir1.2-glib-2.0:ppc64el 1.46.0-3ubuntu1 ppc64el Introspection data for GLib, GObject, Gio and GModule
ii golang 2:1.6-1ubuntu4 all Go programming language compiler - metapackage
ii golang-1.6 1.6-0ubuntu4 all Go programming language compiler - metapackage
ii golang-1.6-doc 1.6-0ubuntu4 all Go programming language - documentation
ii golang-1.6-go 1.6-0ubuntu4 ppc64el Go programming language compiler, linker, compiled stdlib
ii golang-1.6-src 1.6-0ubuntu4 ppc64el Go programming language - source files
ii golang-doc 2:1.6-1ubuntu4 all Go programming language - documentation
ii golang-go 2:1.6-1ubuntu4 ppc64el Go programming language compiler, linker, compiled stdlib
ii golang-src 2:1.6-1ubuntu4 ppc64el Go programming language - source files
ii libalgorithm-diff-perl 1.19.03-1 all module to find differences between files
ii libalgorithm-diff-xs-perl 0.04-4build1 ppc64el module to find differences between files (XS accelerated)
ii libalgorithm-merge-perl 0.08-3 all Perl module for three-way merge of textual data
ii libfribidi0:ppc64el 0.19.7-1 ppc64el Free Implementation of the Unicode BiDi algorithm
ii libgirepository-1.0-1:ppc64el 1.46.0-3ubuntu1 ppc64el Library for handling GObject introspection data (runtime library)
ii libgomp1:ppc64el 5.3.1-13ubuntu3 ppc64el GCC OpenMP (GOMP) support library
ii liblz4-1:ppc64el 0.0~r131-2 ppc64el Fast LZ compression algorithm library - runtime
ii libpolkit-gobject-1-0:ppc64el 0.105-14.1 ppc64el PolicyKit Authorization API
ii python3-gi 3.20.0-0ubuntu1 ppc64el Python 3 bindings for gobject-introspection libraries
root@dockerlp1:~#

Please refer to attachment showing more details on the error..

Thanks,
Manju

[Breno Leitão (breno-leitao)]

Upstream discussion is happening into github:

https://github.com/golang/go/issues/15131

[Launchpad Janitor (janitor)]

This bug was fixed in the package golang-1.6 - 1.6-0ubuntu5

---------------
golang-1.6 (1.6-0ubuntu5) xenial; urgency=medium

  * Fix GOVER calculation in debian/rules to only pull out MAJOR.MINOR.
  * Two patches backported from upstream tip:
    - d/patches/0002-maxcpus-fix.patch (LP: #1565978)
    - d/patches/0003-crypto-dsa-eliminate-invalid-PublicKey-early.patch

 -- Michael Hudson-Doyle <email address hidden> Fri, 08 Apr 2016 13:54:12 +1200

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-04-09 07:41 EDT-------
Hi (In reply to comment #15)
> This bug was fixed in the package golang-1.6 - 1.6-0ubuntu5
>
> ---------------
> golang-1.6 (1.6-0ubuntu5) xenial; urgency=medium
>
> * Fix GOVER calculation in debian/rules to only pull out MAJOR.MINOR.
> * Two patches backported from upstream tip:
> - d/patches/0002-maxcpus-fix.patch (LP: #1565978)
> - d/patches/0003-crypto-dsa-eliminate-invalid-PublicKey-early.patch
>
> -- Michael Hudson-Doyle <email address hidden> Fri, 08 Apr 2016
> 13:54:12 +1200

Hello,

docker .io installation fails with golang-1.6-0ubuntu5 :

root@dockerlp1:~# dpkg -l |grep golang-1.6 | awk -F' ' '{print $1 "\t" $2 "\t" $3}'
ii golang-1.6 1.6-0ubuntu5
ii golang-1.6-doc 1.6-0ubuntu5
ii golang-1.6-go 1.6-0ubuntu5
ii golang-1.6-src 1.6-0ubuntu5

root@dockerlp1:~# apt-get install docker.io
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
aufs-tools lxc rinse zfs-fuse | zfsutils
The following NEW packages will be installed:
docker.io
0 upgraded, 1 newly installed, 0 to remove and 68 not upgraded.
Need to get 0 B/7,981 kB of archives.
After this operation, 44.6 MB of additional disk space will be used.
Selecting previously unselected package docker.io.
(Reading database ... 104201 files and directories currently installed.)
Preparing to unpack .../docker.io_1.10.3-0ubuntu4_ppc64el.deb ...
Unpacking docker.io (1.10.3-0ubuntu4) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for systemd (229-3ubuntu2) ...
Setting up docker.io (1.10.3-0ubuntu4) ...
addgroup: The group `docker' already exists as a system group. Exiting.
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.
invoke-rc.d: initscript docker, action "start" failed.
dpkg: error processing package docker.io (--configure):
subprocess installed post-installation script returned error exit status 1
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for systemd (229-3ubuntu2) ...
Errors were encountered while processing:
docker.io
E: Sub-process /usr/bin/dpkg returned an error code (1)
root@dockerlp1:~#
root@dockerlp1:~#
root@dockerlp1:~# systemctl status docker.service
? docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2016-04-09 06:35:25 CDT; 5s ago
Docs: https://docs.docker.com
Main PID: 13778 (code=exited, status=2)

Apr 09 06:35:25 dockerlp1 docker[13778]: runtime.procresize(0x190, 0x0)
Apr 09 06:35:25 dockerlp1 docker[13778]: /usr/lib/go-1.6/src/runtime/proc.go:3158 +0xbd4
Apr 09 06:35:25 dockerlp1 docker[13778]: runtime.schedinit()
Apr 09 06:35:25 dockerlp1 docker[13778]: /usr/lib/go-1.6/src/runtime/proc.go:452 +0x140
Apr 09 06:35:25 dockerlp1 docker[13778]: runtime.rt0_go(0x0, 0x0, 0x0, 0x3fffffbf12a0, 0xa01749f27c975be7, 0x3fff94e67300, 0xa01749f217e37ba7, 0x0, 0x0, 0x0,
Apr 09 06:3...

[Michael Hudson-Doyle (mwhudson)]

Docker needs to be rebuilt to pick up the fix (yay static linking). Will do
that early next week.
On 9/04/2016 11:55 pm, "bugproxy" <email address hidden> wrote:

> ------- Comment From <email address hidden> 2016-04-09 07:41 EDT-------
> Hi (In reply to comment #15)
> > This bug was fixed in the package golang-1.6 - 1.6-0ubuntu5
> >
> > ---------------
> > golang-1.6 (1.6-0ubuntu5) xenial; urgency=medium
> >
> > * Fix GOVER calculation in debian/rules to only pull out MAJOR.MINOR.
> > * Two patches backported from upstream tip:
> > - d/patches/0002-maxcpus-fix.patch (LP: #1565978)
> > - d/patches/0003-crypto-dsa-eliminate-invalid-PublicKey-early.patch
> >
> > -- Michael Hudson-Doyle <email address hidden> Fri, 08 Apr 2016
> > 13:54:12 +1200
>
> Hello,
>
> docker .io installation fails with golang-1.6-0ubuntu5 :
>
> root@dockerlp1:~# dpkg -l |grep golang-1.6 | awk -F' ' '{print $1 "\t" $2
> "\t" $3}'
> ii golang-1.6 1.6-0ubuntu5
> ii golang-1.6-doc 1.6-0ubuntu5
> ii golang-1.6-go 1.6-0ubuntu5
> ii golang-1.6-src 1.6-0ubuntu5
>
> root@dockerlp1:~# apt-get install docker.io
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Suggested packages:
> aufs-tools lxc rinse zfs-fuse | zfsutils
> The following NEW packages will be installed:
> docker.io
> 0 upgraded, 1 newly installed, 0 to remove and 68 not upgraded.
> Need to get 0 B/7,981 kB of archives.
> After this operation, 44.6 MB of additional disk space will be used.
> Selecting previously unselected package docker.io.
> (Reading database ... 104201 files and directories currently installed.)
> Preparing to unpack .../docker.io_1.10.3-0ubuntu4_ppc64el.deb ...
> Unpacking docker.io (1.10.3-0ubuntu4) ...
> Processing triggers for man-db (2.7.5-1) ...
> Processing triggers for ureadahead (0.100.0-19) ...
> Processing triggers for systemd (229-3ubuntu2) ...
> Setting up docker.io (1.10.3-0ubuntu4) ...
> addgroup: The group `docker' already exists as a system group. Exiting.
> Job for docker.service failed because the control process exited with
> error code. See "systemctl status docker.service" and "journalctl -xe" for
> details.
> invoke-rc.d: initscript docker, action "start" failed.
> dpkg: error processing package docker.io (--configure):
> subprocess installed post-installation script returned error exit status 1
> Processing triggers for ureadahead (0.100.0-19) ...
> Processing triggers for systemd (229-3ubuntu2) ...
> Errors were encountered while processing:
> docker.io
> E: Sub-process /usr/bin/dpkg returned an error code (1)
> root@dockerlp1:~#
> root@dockerlp1:~#
> root@dockerlp1:~# systemctl status docker.service
> ? docker.service - Docker Application Container Engine
> Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor
> preset: enabled)
> Active: failed (Result: exit-code) since Sat 2016-04-09 06:35:25 CDT; 5s
> ago
> Docs: https://docs.docker.com
> Main PID: 13778 (code=exited, status=2)
>
> Apr 09 06:35:25 dockerlp1 docker[13778]: runtime.procresize(0x190, 0x0)
> Apr 09 06:35:25 dockerlp1 docker[13778]:
> /usr/lib/go-1.6/src/runtime/proc.go:3158 +0xbd4
> Apr 09 06:35:25 dockerlp1 docker...

[Michael Hudson-Doyle (mwhudson)]

I've uploaded https://launchpad.net/ubuntu/+source/docker.io/1.10.3-0ubuntu5 which should now work on your system (you may have to enable proposed to get it, it should migrate to -release in an hour or so).

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-04-11 08:16 EDT-------
(In reply to comment #19)
> I've uploaded https://launchpad.net/ubuntu/+source/docker.io/1.10.3-0ubuntu5
> which should now work on your system (you may have to enable proposed to get
> it, it should migrate to -release in an hour or so).

Great !! Looks like fix is available in -release..

Waited for few hours and tested:

root@dockerlp1:~# nproc
400

root@dockerlp1:~# apt-get install docker.io
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
aufs-tools rinse zfs-fuse | zfsutils
The following NEW packages will be installed:
docker.io
0 upgraded, 1 newly installed, 0 to remove and 9 not upgraded.
Need to get 7,982 kB of archives.
After this operation, 44.6 MB of additional disk space will be used.
Get:1 http://ports.ubuntu.com/ubuntu-ports xenial/universe ppc64el docker.io ppc64el 1.10.3-0ubuntu5 [7,982 kB]
Fetched 7,982 kB in 14s (535 kB/s)
Selecting previously unselected package docker.io.
(Reading database ... 104670 files and directories currently installed.)
Preparing to unpack .../docker.io_1.10.3-0ubuntu5_ppc64el.deb ...
Unpacking docker.io (1.10.3-0ubuntu5) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for systemd (229-4ubuntu1) ...
Processing triggers for ureadahead (0.100.0-19) ...
Setting up docker.io (1.10.3-0ubuntu5) ...
addgroup: The group `docker' already exists as a system group. Exiting.
Processing triggers for systemd (229-4ubuntu1) ...
Processing triggers for ureadahead (0.100.0-19) ...
root@dockerlp1:~#

docker.io gets installed now on system having more than 256 CPUs.

Thanks,
Manju

[Breno Leitão (breno-leitao)]

Documented this limitation at https://wiki.ubuntu.com/ppc64el/Recommendations#Golang_applications_on_big_machines

[Michael Hudson-Doyle (mwhudson)]

On 21 April 2016 at 01:38, Breno Leitão <email address hidden> wrote:
> Documented this limitation at
> https://wiki.ubuntu.com/ppc64el/Recommendations#Golang_applications_on_big_machines

Well, the problem is fixed in the golang in Xenial, so I don't think
that warning really belongs there?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-04-27 09:13 EDT-------
> Well, the problem is fixed in the golang in Xenial, so I don't think
> that warning really belongs there?

Right. I added it before we had the fix, now that we have this one fixed, I just removed that documentation. Thanks