new compiler breaks ppc64el unit tests in many ways

[Curtis Hovey]

The tests for http://juju-ci.vapour.ws:8080/job/run-unit-tests-trusty-ppc64el/ last passed for commit 5ecf58fb. The tests fail starting from commit 3d20d690

The common failures are
FAIL: local_test.go:227: TestBootstrapFailsWhenPublicIPError.pN61_github_com_juju_juju_provider_openstack_test.localServerSuite
TestCollectInstances.pN61_github_com_juju_juju_provider_openstack_test.localServerSuite
TestInstancesBuildSpawning.pN61_github_com_juju_juju_provider_openstack_test.localServerSuite
TestStopInstanceSecurityGroupNotDeleted.pN61_github_com_juju_juju_provider_openstack_test.localServerSuite

After additional commits were added, we see additional failures
TearDownTest.pN49_github_com_juju_juju_apiserver_test.debugLogSuite
TestAddRemoveSet.pN42_github_com_juju_juju_replicaset.MongoSuite

The failing code has not changed.

We can see that the gcc-4.9 compiler was updated between runs. When the last passing tarball is run with the current compiler, it fails the same way. The bad compiler is
go version xgcc (Ubuntu 4.9.1-12ubuntu2) 4.9.1 linux/ppc64

That last know good compiler was
go version xgcc (Ubuntu 4.9.1-10ubuntu3) 4.9.1 linux/ppc64

[Curtis Hovey (sinzui)]

This is the log form the latest run from commit d5e7a1d1

[Curtis Hovey (sinzui)]

The suspected commits are:
e72e63d Merge pull request #630 from katco-/lp-1345553-harvest
4aa871b Merge pull request #657 from davecheney/192-mongo-dont-depend-on-apiserver-params
3ee3751 Merge pull request #670 from davecheney/193-apiserver-params-remove-status-data-type
6654869 Merge pull request #660 from mjs/UpgradeInfo
ea24962 Merge pull request #664 from axw/api-provisioner-test-mocks
cd6fe1f Merge pull request #665 from mattyw/add-metric-workers
3d20d69 Merge pull request #672 from wallyworld/floating-ip-address-updates-trunk

[Ian Booth (wallyworld)]

The failing tests all utilitse the ControlHook which looks at the callstack to monkey patch in code required for the tests to pass. The build machine recent started running gccgo 4.9.1-12ubuntu2 whereas I think previously it was 4.9.1-10ubuntu2

I strongly suspect this is a gccgo compiler issue.

The snippet of test code in question is looks like the below. This code had not changed for months. New tests were added recently which use this code, and there are also existing tests from long ago. The return fmt.Error() is just not being run because the compiler doesn't hook it in and so the test breaks.

 cleanup := s.srv.Service.Nova.RegisterControlPoint(
  "addFloatingIP",
  func(sc hook.ServiceControl, args ...interface{}) error {
   return fmt.Errorf("failed on purpose")
  },
 )

[Ian Booth (wallyworld)]

Confirmed with Curtis that running the older gccgo passes the tests.