EOF uploading charm

[Aaron Bentley]

As seen here:

http://reports.vapour.ws/releases/issue/56b8ca60749a56411766ed37

Juju is EOFing uploading a charm. This is an intermittent failure.

It looks timing-related. We have recently switched this test to new hardware, which may have changed timing.

[Cheryl Jennings (cherylj)]

It looks like this is the same underlying cause as bug #1538583. Both the manual provider and maas provider change the mongo replicaset after bootstrap completes, causing the apiserver to drop all its connections. Here's how it happened here:

Client logs in fine:
2016-02-08 15:12:47 DEBUG juju.apiserver apiserver.go:287 -> [11] user-admin@local 57.72784ms {"RequestId":1,"Response":"'body redacted'"} Admin[""].Login

Client does a ModelGet just fine (part of the deploy command)
2016-02-08 15:12:47 DEBUG juju.apiserver apiserver.go:273 <- [11] user-admin@local {"RequestId":3,"Type":"Client","Version":1,"Request":"ModelGet","Params":"'params redacted'"}
2016-02-08 15:12:47 DEBUG juju.apiserver apiserver.go:287 -> [11] user-admin@local 3.237235ms {"RequestId":3,"Response":"'body redacted'"} Client[""].ModelGet

But then we see the kiss of death for all api connections:
2016-02-08 15:12:48 INFO juju.apiserver apiserver.go:504 got error pinging mongo: EOF

After this point the apiserver will die and kill all its connections, so the next time the deploy command tries to use the API connection it has, it will get an error that it is using a closed connection ("connection is shut down").

[Cheryl Jennings (cherylj)]

Although, this one in particular is even more awesome, because we get a panic as a result of trying to store the charm after the connection is shut down:

2016-02-08 15:12:49 DEBUG juju.storage managedstorage.go:294 managed resource entry created with path "buckets/b35686de-5f09-4e7e-8f9f-dae388a3d5c0/charms/local:trusty/dummy-sink-0-95f8f94d-f7d0-4f0f-8c4f-b7426a8efaa1" -> "1aa8fc4f7ab2ee3c0c00fbd9846fd89336d70710247e5d7a548a8e65698031d23269598f92c5c0279a97113a6b912afd"
2016/02/08 15:12:49 http: panic serving 10.0.210.1:57604: Session already closed
goroutine 865 [running]:
net/http.func·009()
 /usr/lib/go/src/pkg/net/http/server.go:1093 +0xae
runtime.panic(0x16617a0, 0xc210794e30)
 /usr/lib/go/src/pkg/runtime/panic.c:248 +0x106
gopkg.in/mgo%2ev2.(*Session).cluster(0xc21025f540, 0x1c)
 /workspace/juju-core-2.0-alpha2/src/gopkg.in/mgo.v2/session.go:1549 +0x66
gopkg.in/mgo%2ev2.copySession(0xc21025f540, 0x424e01, 0xc21000f000)
 /workspace/juju-core-2.0-alpha2/src/gopkg.in/mgo.v2/session.go:519 +0x32
gopkg.in/mgo%2ev2.(*Session).Copy(0xc21025f540, 0x4d544b0)
 /workspace/juju-core-2.0-alpha2/src/gopkg.in/mgo.v2/session.go:1515 +0x41
github.com/juju/juju/mongo.CollectionFromName(0xc210349520, 0x2156ee0, 0x6, 0x6, 0xc2103504a8, ...)
 /workspace/juju-core-2.0-alpha2/src/github.com/juju/juju/mongo/collections.go:12 +0x35
github.com/juju/juju/state.(*database).GetCollection(0xc2102fad40, 0x2156ee0, 0x6, 0x0, 0x0, ...)
 /workspace/juju-core-2.0-alpha2/src/github.com/juju/juju/state/database.go:182 +0x3c0
github.com/juju/juju/state.(*State).getCollection(0xc2100a44d0, 0x2156ee0, 0x6, 0xc210622108, 0x6bcc1d, ...)
 /workspace/juju-core-2.0-alpha2/src/github.com/juju/juju/state/collection.go:22 +0x44
github.com/juju/juju/state.(*State).UpdateUploadedCharm(0xc2100a44d0, 0x7f8ef8029230, 0xc2106aca50, 0xc21038e000, 0xc210690910, ...)
 /workspace/juju-core-2.0-alpha2/src/github.com/juju/juju/state/state.go:1074 +0x6f
github.com/juju/juju/apiserver/service.StoreCharmArchive(0xc2100a44d0, 0xc21038e000, 0x7f8ef8029230, 0xc2106aca50, 0x7f8efb6e7618, ...)
 /workspace/juju-core-2.0-alpha2/src/github.com/juju/juju/apiserver/service/charmstore.go:137 +0x206
github.com/juju/juju/apiserver.(*charmsHandler).repackageAndUploadCharm(0xc2102a2340, 0xc2100a44d0, 0xc2106aca50, 0xc21038e000, 0x0, ...)
 /workspace/juju-core-2.0-alpha2/src/github.com/juju/juju/apiserver/charms.go:387 +0x561
github.com/juju/juju/apiserver.(*charmsHandler).processPost(0xc2102a2340, 0xc210873820, 0xc2100a44d0, 0x0, 0x0, ...)
 /workspace/juju-core-2.0-alpha2/src/github.com/juju/juju/apiserver/charms.go:253 +0x71a
github.com/juju/juju/apiserver.(*charmsHandler).servePost(0xc2102a2340, 0x7f8efb6e7350, 0xc2102138c0, 0xc210873820, 0xc2106ac701, ...)
 /workspace/juju-core-2.0-alpha2/src/github.com/juju/juju/apiserver/charms.go:66 +0xa7
github.com/juju/juju/apiserver.(*charmsHandler).ServeHTTP(0xc2102a2340, 0x7f8efb6e7350, 0xc2102138c0, 0xc210873820)
 /workspace/juju-core-2.0-alpha2/src/github.com/juju/juju/apiserver/charms.go:48 +0x177
github.com/bmizerany/pat.(*PatternServeMux).ServeHTTP(0xc21018b410, 0x7f8efb6e7350, 0xc2102138c0, 0xc210873820)
 /workspace/juju-core-2.0-alpha2/src/github.com/bmizerany...

[Cheryl Jennings (cherylj)]

Master PR: https://github.com/juju/juju/pull/4373