juju-core vivid ppc64el fails to build

[Curtis Hovey]

The vivid ppc64el builds are failing. The last success was on 2015-01.20. their first failure was on 2015-01-23.

We suspect that golang-go sets the wrong GOARCH in vivid.

We cannot see the build logs because Lp is a punk. The builds were in the ~juju-package stable and devel ppas
    https://launchpad.net/~juju-packaging/+archive/ubuntu/stable/+packages
    https://launchpad.net/~juju-packaging/+archive/ubuntu/devel/+packages?field.name_filter=juju-core&field.status_filter=superseded&field.series_filter=vivid

Using the source packages at
    https://launchpad.net/~juju-packaging/+archive/ubuntu/stable/+packages?field.name_filter=juju-core&field.status_filter=published&field.series_filter=vivid

When we build in a vivid ppc64el container on (stilson-07). the build failed at the first call to use the built juju. The first call is to generate the man pages using "juju version" and "juju help commands" debian/rules sets the GOPATH and then adds the GOPATH/bin to PATH. The build binaries were placed in $GOPATH/bin/linux__ppc64le, not $GOPATH/bin!

The "dpkg --print-architecture" for both the host and the container are "ppc64el".
The host'ss "go env" shows GOARCH="ppc64"
The container's "go env" shows GOARCH="ppc64le"

I can contrive a successful build doing this
    GOARCH=ppc64 dpkg-buildpackage -us -uc

So we could change debian/rules, but we first need to understand if GOARCH for vivid ppc64el is sane.

[Curtis Hovey (sinzui)]

We captured a snippet of the private build log and we can see the issue is the same as the report. The first call to access bin/juju fails because it is not there, it is in $GOPATH/bin/linux__ppc64le. VIvid's gccgo-go thinks it is cross-compiling.

mkdir -p /build/buildd/juju-core-1.21.2/bin/linux_ppc64le/
cp $WORK/github.com/juju/juju/cmd/plugins/juju-restore/_obj/exe/a.out /build/buildd/juju-core-1.21.2/bin/linux_ppc64le/juju-restore
go install -x -v -work -gccgoflags -static-libgo github.com/juju/juju/cmd/jujud

....

    doc_generator = GENERATORS[args[1]]()
  File "/build/buildd/juju-core-1.21.2/src/github.com/juju/juju/scripts/jujuman.py", line 11, in __init__
    self.version = self._version()
  File "/build/buildd/juju-core-1.21.2/src/github.com/juju/juju/scripts/jujuman.py", line 22, in _version
    juju_version = self.run_juju('version')
  File "/build/buildd/juju-core-1.21.2/src/github.com/juju/juju/scripts/jujuman.py", line 19, in run_juju
    return subprocess.check_output(cmd).strip()
  File "/usr/lib/python2.7/subprocess.py", line 566, in check_output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
debian/rules:43: recipe for target 'override_dh_auto_install' failed
make[1]: *** [override_dh_auto_install] Error 1
make[1]: Leaving directory '/build/buildd/juju-core-1.21.2'
debian/rules:22: recipe for target 'binary-arch' failed
make: *** [binary-arch] Error 2
dpkg-buildpackage: error: /usr/bin/fakeroot debian/rules binary-arch gave error exit status 2

[Dimiter Naydenov (dimitern)]

There's an updated gccgo-go version for vivid - https://launchpad.net/ubuntu/+source/gccgo-go/1.2.1-0ubuntu7 which includes fixes for a number of issues we're seeing (tests failing, builds failing due to duplicate symbols, random instabilities). Why is this 1.2.1-0ubuntu7 not backported to trusty (even precise?). I think it should be to solve those issues and use it on the ppc64 CI jobs among others. Having CI blockers due to a flaky ppc64 compiler version which already has a fixed version is NOT OK.

Related bug #1393825, another one - #1381671.

[Dimiter Naydenov (dimitern)]

OK, jamespage confirmed a port to precise is not happening, as there's no ppc64 support in there. That's fine - let's see what we can do for trusty though.

[Dimiter Naydenov (dimitern)]

It turns out the vivid package does not yet have the needed patch from https://go-review.googlesource.com/#/c/1840/ we'll have to wait for an upstream release I guess.

[Matthias Klose (doko)]

the default gccgo in vivid is gccgo-5. Please check that all needed patches are available in vivid. the gccgo-go package should be removed, or just contain symlinks pointing to go-5 and gofmt-5.

[Matthias Klose (doko)]

I wouldn't mind a backport of gccgo-5 to trusty, if needed.

[Matthias Klose (doko)]

It is my understanding that GOARCH doesn't include the endian information on any architecture. So the assumption to use ppc64el seems to be outdated.

[Curtis Hovey (sinzui)]

The last 2 Lp builds of juju packages succeeded, as does my own builds. I think this issue resolved itself since I did not change packaging rules. I will do a little more investigation, but hope to close this bug in a few hours.

[Curtis Hovey (sinzui)]

There has been a lot confusion about this bug. As the reporter, I am happy to say vivid packaging (or cloud/builder images) were fixed last week. Without any changes to juju packaging. Launchpad and lxc can again built vivid ppc64el packages without explicitly setting the GOARCH to match the dpkg architecture.

And to be clear, this issue was only ever about a mismatch of deb architecture which caused gccgo to think it is cross-compiling. The binaries worked fine if you could find them.

These packages are among the changes seen in my stale container when I issued the dist-upgrade.
     g++-4.9 gcc-4.9 gcc-4.9-base gcc-5-base gccgo gccgo-5 gccgo-go libgo5 libgo7

We can see from Lp that the fix arrived on or before 2015-02-17.