xml/marshal.go:10:2: cannot find package "encoding"

[Curtis Hovey]

Juju cannot be compiled on ppc64el because the "encoding" package cannot be found.

go version xgcc (Ubuntu 4.9.1-13ubuntu1) 4.9.1 linux/ppc64
+ make build
skipping godeps
go build github.com/juju/juju/...
../xml/marshal.go:10:2: cannot find package "encoding" in any of:
 /usr/src/pkg/encoding (from $GOROOT)
 /home/ubuntu/juju-core_1.24-alpha1/src/encoding (from $GOPATH)
make: *** [build] Error 1

The first occurrence of of this error is seen in the ppc unit-test run in lxc:
    http://reports.vapour.ws/releases/2483/job/run-unit-tests-trusty-ppc64el-lxc/attempt/355
We want to run on lxc, but the ppc unit test never pass in lxc, so the test is non-voting.

[Curtis Hovey (sinzui)]

Note that ppc packages packages were built for ppc on stilson-07, but stilson-08 cannot build for tests. Maybe test deps are different, or there is a dep installed on one stilson, but not the other.

[Nate Finch (natefinch)]

This is a problem with the build environment - it sounds like the go installation is messed up somehow. The error shows that go can't find the "encoding" package from the standard library... presumably because it's looking in /usr/src/pkg/encoding and should be looking in /usr/lib/go/pkg/encoding

I don't know why it's looking in the wrong place probably worth just reinstalling the golang package to see if that fixes it.

[Curtis Hovey (sinzui)]

On the ppc64el machine that use gccgo-4.9, we can see the encoding package is available.

/usr/lib/powerpc64le-linux-gnu/go/4.9/encoding
/usr/lib/powerpc64le-linux-gnu/go/4.9/encoding.gox
/usr/lib/powerpc64le-linux-gnu/go/4.9/encoding/ascii85.gox
/usr/lib/powerpc64le-linux-gnu/go/4.9/encoding/asn1.gox
/usr/lib/powerpc64le-linux-gnu/go/4.9/encoding/base32.gox
/usr/lib/powerpc64le-linux-gnu/go/4.9/encoding/base64.gox
/usr/lib/powerpc64le-linux-gnu/go/4.9/encoding/binary.gox
/usr/lib/powerpc64le-linux-gnu/go/4.9/encoding/csv.gox
/usr/lib/powerpc64le-linux-gnu/go/4.9/encoding/gob.gox
/usr/lib/powerpc64le-linux-gnu/go/4.9/encoding/hex.gox
/usr/lib/powerpc64le-linux-gnu/go/4.9/encoding/json.gox
/usr/lib/powerpc64le-linux-gnu/go/4.9/encoding/pem.gox
/usr/lib/powerpc64le-linux-gnu/go/4.9/encoding/xml.gox

[Curtis Hovey (sinzui)]

The go stdlib is provoided by libgo5 with is installed and provides the encoding packages.

[Curtis Hovey (sinzui)]

Per Ian Lance Taylor 25 days ago at https://github.com/golang/go/issues/10092
I think the "import cycle not allowed" problem happens when gccgo sees packages that have the same name as packages in GOROOT. Because gccgo has no GOROOT, it gets confused and thinks those packages are the ones in GOROOT. That is, if GOPATH has a time package, gccgo thinks that is the package it is supposed to use for import "time", even though really it will use the standard time package. This doesn't happen with the gc compiler because it sees the GOROOT time package and knows that one will be used first. Since gccgo has no GOROOT, this does not happen.

Per Dave Cheney 12 days ago at https://github.com/golang/go/issues/10258
We had to work around this for gccgo-4.9 shipped in Ubuntu. We had to
provide a modified version of the go tool which knew the list of packages
which made up gccgo's version of $GOROOT and always reported that they were
up to date in the dependency graph,

[Nate Finch (natefinch)]

The thing I see is, it builds fine on my machine using the standard ubuntu golang package and gccgo package... so I don't know how it can be a bug in the tool itself. The bugs linked don't seem to have anything to do with the error we're seeing.

[Curtis Hovey (sinzui)]

I have lowered this bug to High. We are playing musical ppc servers. We are doubling up on the machine that builds packages because it was happy.

[Curtis Hovey (sinzui)]

Juju cannot be compiled on stilson-07 (which was used for packaging) or stilson-08 (which was near pristine, used for manual deployments). The logs for run-unit-tests-trusty-ppc64el are at http://reports.vapour.ws/releases/2524. They are no more informative than the first set of logs.

run-unit-tests-trusty-ppc64el-lxc (non-voting) also fails the same way. the LXC test runs are 100% pristine, using a new trusty instance and installing only what the Makefile wants.

I suspect the Makefile is at fault since we still saw this same build make packages using different rules.

[Curtis Hovey (sinzui)]

Or the issue is the deps needed for testing break the build.

[Curtis Hovey (sinzui)]

Reviewing http://data.vapour.ws/juju-ci/products/version-2524/publish-revision/build-1677/consoleText we can see that the binary package can be installed. We don't see a specific call to 'go build'. Instead we see calls to 'go install' each binary required by the package.

All builds of the packages warn about the orig files left in the tree:
dpkg-source: info: using source format `3.0 (quilt)'
dpkg-source: info: building juju-core using existing ./juju-core_1.24-alpha1.orig.tar.gz
dpkg-source: warning: ignoring deletion of file src/github.com/juju/xml/xml.go.orig
dpkg-source: warning: ignoring deletion of file src/github.com/juju/xml/read_test.go.orig
dpkg-source: warning: ignoring deletion of file src/github.com/juju/xml/marshal.go.orig
dpkg-source: warning: ignoring deletion of file src/github.com/juju/xml/marshal_test.go.orig
dpkg-source: info: building juju-core in juju-core_1.24-alpha1-0ubuntu1~14.04.1~juju1.debian.tar.gz
dpkg-source: info: building juju-core in juju-core_1.24-alpha1-0ubuntu1~14.04.1~juju1.dsc

Could gccgo be trying the compile the .orig files which running 'make build ./...'? The calls to install just the needed binaries may be working with a subset of the source code that is sane.

A successful binary packages makes these calls 4 calls involving marshall

gccgo -I $WORK -I /home/ubuntu/juju-build/juju-core-1.24-alpha1/pkg/gccgo_linux_ppc64 -c -g -fgo-pkgpath=github.com/juju/juju/tools -fgo-relative-import-path=_/home/ubuntu/juju-build/juju-core-1.24-alpha1/src/github.com/juju/juju/tools -o $WORK/github.com/juju/juju/tools/_obj/tools.o ./list.go ./marshal.go ./tools.go

gccgo -I $WORK -I /home/ubuntu/juju-build/juju-core-1.24-alpha1/pkg/gccgo_linux_ppc64 -c -g -fgo-pkgpath=gopkg.in/macaroon.v1 -fgo-relative-import-path=_/home/ubuntu/juju-build/juju-core-1.24-alpha1/src/gopkg.in/macaroon.v1 -o $WORK/gopkg.in/macaroon.v1/_obj/macaroon.o ./crypto.go ./macaroon.go ./marshal.go ./packet.go

gccgo -I $WORK -I /home/ubuntu/juju-build/juju-core-1.24-alpha1/pkg/gccgo_linux_ppc64 -c -g -fgo-pkgpath=github.com/juju/juju/environs/imagemetadata -fgo-relative-import-path=_/home/ubuntu/juju-build/juju-core-1.24-alpha1/src/github.com/juju/juju/environs/imagemetadata -o $WORK/github.com/juju/juju/environs/imagemetadata/_obj/imagemetadata.o ./generate.go ./marshal.go ./simplestreams.go ./urls.go ./validation.go

gccgo -I $WORK -I /home/ubuntu/juju-build/juju-core-1.24-alpha1/pkg/gccgo_linux_ppc64 -c -g -fgo-pkgpath=github.com/juju/juju/environs/tools -fgo-relative-import-path=_/home/ubuntu/juju-build/juju-core-1.24-alpha1/src/github.com/juju/juju/environs/tools -o $WORK/github.com/juju/juju/environs/tools/_obj/tools.o ./build.go ./marshal.go ./simplestreams.go ./storage.go ./tools.go ./urls.go ./validation.go

[Curtis Hovey (sinzui)]

I have no idea why gccgo cannot import the encoding pkg. It is clearly looking in the wrong place, setting GOROOT is unhelpful because go only wants src, not gox files used by gccgo.

golang is not installable on ppc64el, and maybe we shouldn't. Since this is gccgo-4.9, we probably need to use its' copy of the go source. I verified that I can get the source package create a goroot and compile where gccgo will find the source code.

cd workspace
apt-get source gccgo-4.9
cd gccgo-4.9-4.9.1/
tar -xf gcc-4.9.1.tar.xz
mkdir -p goroot/src
ln -s ~/workspace/gccgo-4.9-4.9.1/gcc-4.9.1/libgo/go $HOME/workspace/goroot/src/pkg
export GOROOT=$HOME/workspace/goroot
go build -v github.com/juju/juju/...

Should the Makefile do this because the install-dependencies target is no longer valid for gccgo?
Should run-unit-tests do they everytime?
We could do a once time setup of the goroot on every stilson, but then it is not obvious that this is a requirement for anyone developing on a ppc64el machine.

[Curtis Hovey (sinzui)]

I have added a goroot to the ubuntu user's home dir on the stilsons. run-unit-tests will set the GOROOT when that dir exists. This is allowing the tests to run. We cannot run tests in lxc though because we don't have a means yet for the script or for the Makefile to make a goroot from the gccgo-4.9 source package as needed.

[Curtis Hovey (sinzui)]

stilson-07 cannot build PPC64el debs. This means Lp/packagers cannot build juju on PPC64el as well.

go install -x -v -work github.com/juju/juju/cmd/juju
src/github.com/juju/govmomi/vim25/xml/marshal.go:10:2: cannot find package "encoding" in any of:
 /usr/src/pkg/encoding (from $GOROOT)
 /home/ubuntu/juju-build/juju-core-1.24-alpha1/src/encoding (from $GOPATH)

We cannot hack our builders for this case. The release tarfile but contain the encoding src or pkg, or we avoid using github.com/juju/govmomi/vim25/xml.

[Curtis Hovey (sinzui)]

Hi Martin. I don't see fix in the release tarball that ensures gccgo finds an encoding package.

[Curtis Hovey (sinzui)]

While we could make packages, but not test a recent change now prevents packages from building. The first failure to build packages happened in http://reports.vapour.ws/releases/2563 These are the 3 commits are the additions from the previous build.

Commit 1735d50 Merge pull request #2123 from ericsnowcurrently/vsphere-provider …
Commit 735ef1c Merge pull request #2126 from bz2/fmt_vs_provider …
Commit d97bb4f Merge pull request #2125 from bz2/skip_test_uniterupgrade_ppc …

[Richard Harding (rharding)]

This was done to fix a xml issue serving out icon svgs from charms. The upstream Go bug is locted at:
https://github.com/golang/go/issues/10158

We're hoping to see some traction on this in go 1.5.

I'll have Madison pull the commit in our fork. It's in the fork here:

https://github.com/juju/xml/commits/master

[Martin Packman (gz)]

I did fix this... then the addition of govmomi regressed this by including *another* broken copy of encoding/xml. Fortunately there's the easy fix of making govmomi use juju/xml as that exists anyway.

[Eric Snow (ericsnowcurrently)]

Do not build the vsphere provider under GCCGo: http://reviews.vapour.ws/r/1510/

[Eric Snow (ericsnowcurrently)]

@Martin, unfortunately making our govmomi fork use our hacked XML won't work once we drop the fork (once Go 1.4 lands in trusty/vivid in a few days).

[Nate Finch (natefinch)]

For the record, the tests seem to be passing just fine? It compiled fine yesterday, just hit some timeout issues in the tests, nothing to do with encoding or anything. Are we sure this is still a bug?

[Nate Finch (natefinch)]

For reference about the above:

http://data.vapour.ws/juju-ci/products/version-2568/run-unit-tests-trusty-ppc64el/build-2863/consoleText

[Curtis Hovey (sinzui)]

Nate, this bug was complicated by two xml packages. I was not aware of the *two* packages when I reopened this bug. I would have reported a separate one.

"go test" and "go install" hit or miss the xml bug differently. Per the original conversation about juju/xml, "go install" used by deb packges worked, but "go test" did not we create a goroot from the gccgo-go source package to make the tests pass. The govm*/xml package also passed because the test machines have a goroot from the unpacked source package. But "go install" failed. Packages are built in a fakeroot where we cannot contrive a goroot created from the source of the same binary package the build depends on.