btw. This has nothing to do with ssh, it is to do with the length of
time the juju cli process is running for. Normally this is a very
short period, but juju ssh forks ssh as a child then waits for it to
finish. The crash is caused by a bug in the previous runtime (compiled
statically into the binary) when the garbage collector had been
running for long enough to call madvise(DONT_NEED, ...) with the wrong
page size.

On Sat, Jul 26, 2014 at 8:22 AM, Matt Bruzek
<email address hidden> wrote:
> With sinzui's help I have compiled the latest version of Juju 1.20.2
> with the gccgo-4.9_4.9.1-1ubuntu3_ppc64el.deb.
>
> NOT able to reproduce the “juju ssh” panic/problem having several ssh
> sessions open simultaneously and having output streaming for over 2
> hours.
>
> For clarity and review here are the details of how we build the deb
> packages for Juju with the gccgo 4.9.1 compiler on the CI machine
> stilson-09 (a P8 64k page size machine).
>
> ubuntu@stilson-09:~⟫ uname -a
> Linux stilson-09 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:29:18 UTC 2014 ppc64le ppc64le ppc64le GNU/Linux
> ubuntu@stilson-09:~⟫ getconf PAGE_SIZE
> 65536
> ubuntu@stilson-09:~⟫ more /proc/cpuinfo
> processor       : 0
> cpu             : POWER8E (raw), altivec supported
> clock           : 4116.000000MHz
> revision        : 2.0 (pvr 004b 0200)
>
> processor       : 1
> cpu             : POWER8E (raw), altivec supported
> clock           : 4116.000000MHz
> revision        : 2.0 (pvr 004b 0200)
>
> timebase        : 512000000
> platform        : pSeries
> model           : IBM pSeries (emulated by qemu)
> machine         : CHRP IBM pSeries (emulated by qemu)
>
> Downloaded the following packages from:
> https://launchpad.net/ubuntu/utopic/ppc64el/gccgo-4.9/4.9.1-1ubuntu3
>
> binutils_2.24.51.20140709-1ubuntu1_ppc64el.deb
> cpp-4.9_4.9.1-1ubuntu3_ppc64el.deb
> gcc-4.9_4.9.1-1ubuntu3_ppc64el.deb
> gcc-4.9-base_4.9.1-1ubuntu3_ppc64el.deb
> gccgo-4.9_4.9.1-1ubuntu3_ppc64el.deb
> libatomic1_4.9.1-1ubuntu3_ppc64el.deb
> libgcc1_4.9.1-1ubuntu3_ppc64el.deb
> libgcc-4.9-dev_4.9.1-1ubuntu3_ppc64el.deb
> libgo5_4.9.1-1ubuntu3_ppc64el.deb
> libgomp1_4.9.1-1ubuntu3_ppc64el.deb
> libitm1_4.9.1-1ubuntu3_ppc64el.deb
>
> Installed the packages on stilson-09:
>
> ubuntu@stilson-09:~/gccgo⟫ sudo dpkg -i *.deb
>
> ubuntu@stilson-09:~/gccgo⟫ dpkg -l | grep gcc
> ii  gcc                                    4:4.8.2-1ubuntu6              ppc64el      GNU C compiler
> ii  gcc-4.8                                4.8.3-3ubuntu0.2              ppc64el      GNU C compiler
> ii  gcc-4.8-base:ppc64el                   4.8.3-3ubuntu0.2              ppc64el      GCC, the GNU Compiler Collection (base package)
> ii  gcc-4.9                                4.9.1-1ubuntu3                ppc64el      GNU C compiler
> ii  gcc-4.9-base:ppc64el                   4.9.1-1ubuntu3                ppc64el      GCC, the GNU Compiler Collection (base package)
> ii  gccgo                                  4:4.9-1ubuntu6                ppc64el      Go compiler, based on the GCC backend
> ii  gccgo-4.9                              4.9.1-1ubuntu3                ppc64el      GNU Go compiler
> ii  gccgo-go                               1.2.1-0ubuntu1                ppc64el      Go tool for use with gccgo
> ii  libgcc-4.8-dev:ppc64el                 4.8.3-3ubuntu0.2              ppc64el      GCC support library (development files)
> ii  libgcc-4.9-dev:ppc64el                 4.9.1-1ubuntu3                ppc64el      GCC support library (development files)
> ii  libgcc1:ppc64el                        1:4.9.1-1ubuntu3              ppc64el      GCC support library
>
> Built Juju from source on stilson-09:
>
> <sinzui> ~/Work/juju-release-tools/make-source-packages.bash stable juju-core_1.20.2.tar.gz 'Curtis Hovey <email address hidden>' 1340893 1345014 1344940 1345832 1307434 1347371 1337091 1342747 1343301 1341589
> <sinzui>  ~/Work/juju-release-tools/build-package-with-dsc.bash ubuntu@stilson-09 '' /home/curtis/Downloads/tmp.WUZeXOfFz1/juju-core_1.20.2-0ubuntu1~14.04.1~juju1.dsc
>
> Resulting in:
>
> ubuntu@stilson-09:~⟫ ls -l juju-build
> total 17332
> -rw-r--r-- 1 ubuntu ubuntu     6944 Jul 25 15:05 juju_1.20.2-0ubuntu1~14.04.1~juju1_all.deb
> drwxrwxr-x 7 ubuntu ubuntu     4096 Jul 25 15:04 juju-core-1.20.2
> -rw------- 1 ubuntu ubuntu     8356 Jul 25 15:04 juju-core_1.20.2-0ubuntu1~14.04.1~juju1.debian.tar.gz
> -rw------- 1 ubuntu ubuntu     1183 Jul 25 15:04 juju-core_1.20.2-0ubuntu1~14.04.1~juju1.dsc
> -rw-rw-r-- 1 ubuntu ubuntu     3133 Jul 25 15:05 juju-core_1.20.2-0ubuntu1~14.04.1~juju1_ppc64el.changes
> -rw-r--r-- 1 ubuntu ubuntu 14341816 Jul 25 15:05 juju-core_1.20.2-0ubuntu1~14.04.1~juju1_ppc64el.deb
> -rw-rw-r-- 1 ubuntu ubuntu  3354392 Jul 25 15:03 juju-core_1.20.2.orig.tar.gz
> -rw-r--r-- 1 ubuntu ubuntu     7020 Jul 25 15:05 juju-local_1.20.2-0ubuntu1~14.04.1~juju1_all.deb
> -rw-r--r-- 1 ubuntu ubuntu     7022 Jul 25 15:05 juju-local-kvm_1.20.2-0ubuntu1~14.04.1~juju1_all.deb
>
> I copied these deb packages to stilson-01:
> ubuntu@stilson-01:~$ ls -l juju-build
> total 14032
> -rw-r--r-- 1 ubuntu ubuntu     6944 Jul 25 15:21 juju_1.20.2-0ubuntu1~14.04.1~juju1_all.deb
> -rw-r--r-- 1 ubuntu ubuntu 14341816 Jul 25 15:22 juju-core_1.20.2-0ubuntu1~14.04.1~juju1_ppc64el.deb
> -rw-r--r-- 1 ubuntu ubuntu     7020 Jul 25 15:22 juju-local_1.20.2-0ubuntu1~14.04.1~juju1_all.deb
> -rw-r--r-- 1 ubuntu ubuntu     7022 Jul 25 15:22 juju-local-kvm_1.20.2-0ubuntu1~14.04.1~juju1_all.deb
>
> Installed the deb packages on stilson-01:
>
> sudo dpkg -i *.deb
>
> ubuntu@stilson-01:~$ dpkg -l | grep juju
> ii  juju                                   1.20.2-0ubuntu1~14.04.1~juju1     all          next generation service orchestration system
> ii  juju-core                              1.20.2-0ubuntu1~14.04.1~juju1     ppc64el      Juju is devops distilled - client
> ii  juju-deployer                          0.3.6-0ubuntu2                    all          Deploy complex stacks of services using Juju
> ii  juju-jitsu                             0.20-1                            all          external tools to enhance juju
> ii  juju-local                             1.20.2-0ubuntu1~14.04.1~juju1     all          dependency package for the Juju local provider
> ii  juju-local-kvm                         1.20.2-0ubuntu1~14.04.1~juju1     all          dependency package for the Juju local provider
> ii  juju-mongodb                           2.4.9-0ubuntu3                    ppc64el      MongoDB object/document-oriented database for Juju
> ii  juju-quickstart                        1.4.1+bzr88+ppa25~ubuntu14.04.1   all          Easy configuration of Juju environments
> ii  python-jujuclient                      0.17.5-0ubuntu2                   all          Python API client for juju
>
> Examined the strings in juju and jujud:
>
> ubuntu@stilson-01:~$ strings $(which juju) | grep gcc
> __gccgo_personality_v0
> libgcc_s.so.1
> -gccgoflags=-static-libgo
> gccgo
> ubuntu@stilson-01:~$ strings $(which jujud) | grep gcc
> libgcc_s.so.1
> gccgo
> -gccgoflags=-static-libgo
> gccgo
> xgcc (Ubuntu 4.9.1-1ubuntu3) 4.9.1
>
>
> Doing some testing on power that involves running the “juju test” command which bootstraps an environment deploys waits for success/fail and then destroys the environment.  The test run will attempt to run 141 times and is on 9 iterations without a juju failure.  I am confident this gccgo fix is good.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1347322
>
> Title:
>   juju ssh results in a panic: runtime error
>
> Status in juju-core:
>   Triaged
> Status in “juju-core” package in Ubuntu:
>   Confirmed
>
> Bug description:
>   I am using Juju on Power 8 hardware and I get a panic when I use juju
>   to ssh to a system.  The reproduction steps are
>
>   juju init
>   juju bootstrap -e local
>   juju switch local
>   juju deploy local:trusty/ubuntu
>   juju ssh ubuntu/0
>
>   (use the session to the local machine and after some time the panic
>   happens and the screen is unreadable.)
>
>   ubuntu@ubuntu-local-machine-1:~$ panic: runtime error: invalid memory address or nil pointer dereference
>      [signal 0xb code=0x1 addr=0x8]
>
>                                    goroutine 11 [running]:
>                                                           code.google.com_p_go.net_websocket.Send.N40_code.
>   google.com_p_go.net_websocket.Codec
>                                           /build/buildd/juju-core-1.18.1/src/code.google.com/p/go.net/webso
>   cket/websocket.go:293
>                            launchpad.net_juju_core_rpc_jsoncodec.Send.N48_launchpad.net_juju_core_rpc_jsonc
>   odec.wsJSONConn
>                           /build/buildd/juju-core-1.18.1/src/launchpad.net/juju-core/rpc/jsoncodec/conn.go:
>   21
>         launchpad.net_juju_core_rpc_jsoncodec.WriteMessage.pN43_launchpad.net_juju_core_rpc_jsoncodec.Codec
>           /build/buildd/juju-core-1.18.1/src/launchpad.net/juju-core/rpc/jsoncodec/codec.go:178
>                                                                                                launchpad.ne
>   t_juju_core_rpc.send.pN32_launchpad.net_juju_core_rpc.Conn
>                                                                   /build/buildd/juju-core-1.18.1/src/launch
>   pad.net/juju-core/rpc/client.go:72
>                                         launchpad.net_juju_core_rpc.Go.pN32_launchpad.net_juju_core_rpc.Con
>   n
>           /build/buildd/juju-core-1.18.1/src/launchpad.net/juju-core/rpc/client.go:174
>                                                                                       launchpad.net_juju_co
>   re_rpc.Call.pN32_launchpad.net_juju_core_rpc.Conn
>                                                           /build/buildd/juju-core-1.18.1/src/launchpad.net/
>   juju-core/rpc/client.go:148
>                                  launchpad.net_juju_core_state_api.Call.pN39_launchpad.net_juju_core_state_
>   api.State
>                   /build/buildd/juju-core-1.18.1/src/launchpad.net/juju-core/state/api/apiclient.go:168
>   launchpad.net_juju_core_state_api.Ping.pN39_launchpad.net_juju_core_state_api.State
>                                                                                           /build/buildd/juj
>   u-core-1.18.1/src/launchpad.net/juju-core/state/api/apiclient.go:158
>                                                                           launchpad.net_juju_core_state_api
>   .heartbeatMonitor.pN39_launchpad.net_juju_core_state_api.State
>                                                                           /build/buildd/juju-core-1.18.1/sr
>   c/launchpad.net/juju-core/state/api/apiclient.go:149
>                                                           created by launchpad.net_juju_core_state_api.Open
>           /build/buildd/juju-core-1.18.1/src/launchpad.net/juju-core/state/api/apiclient.go:143
>
>                                                                                                goroutine 1
>   [syscall]:
>                   goroutine in C code; stack unavailable
>
>                                                         goroutine 3 [syscall]:
>                                                                                   goroutine in C code; stac
>   k unavailable
>
>                    goroutine 10 [IO wait]:
>                                           code.google.com_p_go.net_websocket.ReadByte.N57_code.google.com_p
>   _go.net_websocket.hybiFrameReaderFactory
>                                                   /build/buildd/juju-core-1.18.1/src/code.google.com/p/go.n
>   et/websocket/hybi.go:113
>                               code.google.com_p_go.net_websocket.NewFrameReader.N57_code.google.com_p_go.ne
>   t_websocket.hybiFrameReaderFactory
>                                           /build/buildd/juju-core-1.18.1/src/code.google.com/p/go.net/webso
>   cket/hybi.go:126
>                       code.google.com_p_go.net_websocket.Receive.N40_code.google.com_p_go.net_websocket.Cod
>   ec
>           /build/buildd/juju-core-1.18.1/src/code.google.com/p/go.net/websocket/websocket.go:314
>                                                                                                 launchpad.n
>   et_juju_core_rpc_jsoncodec.Receive.N48_launchpad.net_juju_core_rpc_jsoncodec.wsJSONConn
>                                                                                                   /build/bu
>   ildd/juju-core-1.18.1/src/launchpad.net/juju-core/rpc/jsoncodec/conn.go:25
>                                                                                 launchpad.net_juju_core_rpc
>   _jsoncodec.ReadHeader.pN43_launchpad.net_juju_core_rpc_jsoncodec.Codec
>                                                                                   /build/buildd/juju-core-1
>   .18.1/src/launchpad.net/juju-core/rpc/jsoncodec/codec.go:113
>                                                                   launchpad.net_juju_core_rpc.loop.pN32_lau
>   nchpad.net_juju_core_rpc.Conn
>                                           /build/buildd/juju-core-1.18.1/src/launchpad.net/juju-core/rpc/se
>   rver.go:344
>                  launchpad.net_juju_core_rpc.input.pN32_launchpad.net_juju_core_rpc.Conn
>                                                                                           /build/buildd/juj
>   u-core-1.18.1/src/launchpad.net/juju-core/rpc/server.go:317
>                                                                  created by launchpad.net_juju_core_rpc.Sta
>   rt.pN32_launchpad.net_juju_core_rpc.Conn
>                                                   /build/buildd/juju-core-1.18.1/src/launchpad.net/juju-cor
>   e/rpc/server.go:200
>
>   Here are the specifics on the host system.
>
>   ubuntu@stilson-01:~$ uname -a
>   Linux stilson-01 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:50:31 UTC 2014 ppc64le ppc64le ppc64le GN
>   U/Linux
>   ubuntu@stilson-01:~$ dpkg -l | grep juju
>   ii  juju                                   1.18.1-0ubuntu1.1                 all          next generation
>    service orchestration system
>   ii  juju-core                              1.18.1-0ubuntu1.1                 ppc64el      Juju is devops
>   distilled - client
>   ii  juju-deployer                          0.3.6-0ubuntu2                    all          Deploy complex
>   stacks of services using Juju
>   ii  juju-jitsu                             0.20-1                            all          external tools
>   to enhance juju
>   ii  juju-local                             1.18.1-0ubuntu1.1                 all          dependency pack
>   age for the Juju local provider
>   ii  juju-mongodb                           2.4.9-0ubuntu3                    ppc64el      MongoDB object/
>   document-oriented database for Juju
>   ii  juju-quickstart                        1.4.1+bzr88+ppa25~ubuntu14.04.1   all          Easy configurat
>   ion of Juju environments
>   ii  python-jujuclient                      0.17.5-0ubuntu2                   all          Python API clie
>   nt for juju
>   ubuntu@stilson-01:~$ getconf PAGE_SIZE
>   65536
>
>   I got this error when testing a fix for another bug:
>   https://bugs.launchpad.net/ubuntu/+source/gccgo-4.9/+bug/1304754
>
>   Since the dmesg looks different we believe this is a new problem.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1347322/+subscriptions