Canonical Juju

Panic upgradeSuite.TestLoginsDuringUpgrade error pinging mongo: Closed explicitly

Bug #1642311 reported by Curtis Hovey
4
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
Fix Released
Medium
Unassigned

Bug Description

As seen in
    http://reports.vapour.ws/releases/issue/582c68d5749a5607bf4bc1dd

PANIC: <autogenerated>:995: upgradeSuite.TearDownTest

... Panic: error pinging mongo: Closed explicitly (PC=0x45F70E)

/usr/lib/go/src/runtime/panic.go:426
  in gopanic
/home/ubuntu/juju-core_2.1-beta1/src/github.com/juju/juju/provider/dummy/environs.go:358
  in environState.destroyLocked
/home/ubuntu/juju-core_2.1-beta1/src/github.com/juju/juju/provider/dummy/environs.go:334
  in environState.destroy
/home/ubuntu/juju-core_2.1-beta1/src/github.com/juju/juju/provider/dummy/environs.go:313
  in Reset
/home/ubuntu/juju-core_2.1-beta1/src/github.com/juju/juju/juju/testing/conn.go:648
  in JujuConnSuite.tearDownConn
/home/ubuntu/juju-core_2.1-beta1/src/github.com/juju/juju/juju/testing/conn.go:133
  in JujuConnSuite.TearDownTest
/usr/lib/go/src/reflect/value.go:303
  in Value.Call
/usr/lib/go/src/runtime/asm_amd64.s:1998
  in goexit

----------------------------------------------------------------------
PANIC: upgrade_test.go:94: upgradeSuite.TestLoginsDuringUpgrade

Revision history for this message
John A Meinel (jameinel) wrote :
Download full text (3.4 KiB)

I got a different failure in this same test recently.
http://ci.jujucharms.com/job/github-merge-juju/101/console
LOG] 0:00.983 ERROR juju.worker.dependency "api-server" manifold worker returned unexpected error: listen tcp :44094: bind: address already in use
[LOG] 0:00.983 DEBUG juju.worker.dependency stack trace:
listen tcp :44094: bind: address already in use
github.com/juju/juju/worker/apiserver/worker.go:140:
github.com/juju/juju/worker/apiserver/manifold.go:163:
[LOG] 0:03.667 DEBUG juju.worker.apicaller failed to connect
[LOG] 0:03.667 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: cannot open api: try was stopped
[LOG] 0:03.726 DEBUG juju.worker.apicaller connecting with current password
[LOG] 0:03.726 DEBUG juju.api looked up localhost -> [127.0.0.1]
[LOG] 0:04.040 DEBUG juju.worker.dependency "api-server" manifold worker stopped: listen tcp :44094: bind: address already in use
[LOG] 0:04.040 ERROR juju.worker.dependency "api-server" manifold worker returned unexpected error: listen tcp :44094: bind: address already in use
[LOG] 0:04.040 DEBUG juju.worker.dependency stack trace:
listen tcp :44094: bind: address already in use
github.com/juju/juju/worker/apiserver/worker.go:140:
github.com/juju/juju/worker/apiserver/manifold.go:163:
[LOG] 0:06.726 DEBUG juju.worker.apicaller failed to connect
[LOG] 0:06.726 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: cannot open api: try was stopped
[LOG] 0:06.726 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: cannot open api: try was stopped
[LOG] 0:06.726 DEBUG juju.worker.dependency stack trace:
try was stopped
github.com/juju/juju/api/apiclient.go:720:
github.com/juju/juju/api/apiclient.go:596:
github.com/juju/juju/api/apiclient.go:196:
github.com/juju/juju/worker/apicaller/connect.go:153:
github.com/juju/juju/worker/apicaller/connect.go:196:
github.com/juju/juju/worker/apicaller/manifold.go:82: cannot open api
[LOG] 0:07.043 DEBUG juju.worker.dependency "api-server" manifold worker stopped: listen tcp :44094: bind: address already in use
[LOG] 0:07.043 ERROR juju.worker.dependency "api-server" manifold worker returned unexpected error: listen tcp :44094: bind: address already in use
[LOG] 0:07.043 DEBUG juju.worker.dependency stack trace:
listen tcp :44094: bind: address already in use
github.com/juju/juju/worker/apiserver/worker.go:140:
github.com/juju/juju/worker/apiserver/manifold.go:163:
[LOG] 0:09.784 DEBUG juju.worker.apicaller connecting with current password
[LOG] 0:09.785 DEBUG juju.api looked up localhost -> [127.0.0.1]
[LOG] 0:10.103 DEBUG juju.worker.dependency "api-server" manifold worker stopped: listen tcp :44094: bind: address already in use
[LOG] 0:10.103 ERROR juju.worker.dependency "api-server" manifold worker returned unexpected error: listen tcp :44094: bind: address already in use
[LOG] 0:10.103 DEBUG juju.worker.dependency stack trace:
listen tcp :44094: bind: address already in use
github.com/juju/juju/worker/apiserver/worker.go:140:
github.com/juju/juju/worker/apiserver/manifold.go:163:
upgrade_test.go:141:
    c.Assert(waitForUpgradeToStart(upgradeCh), jc.IsTrue)
... obtained bo...

Read more...

Revision history for this message
John A Meinel (jameinel) wrote : Re: [Bug 1642311] Re: Panic upgradeSuite.TestLoginsDuringUpgrade error pinging mongo: Closed explicitly
Download full text (5.9 KiB)

Something like this might have happened again.
http://ci.jujucharms.com/job/github-merge-juju/374/testReport/junit/github/com_juju_juju_featuretests/TestPackage/

In this case we got some sort of 25min deadlock triggering the test suite
timeout.
The stack trace seems to indicate
goroutine 24709 [chan receive, 23 minutes]:
gopkg.in/tomb%2ev1.(*Tomb).Wait(0xc42129ad30, 0xc42201f968, 0x5957a9)
/workspace/src/gopkg.in/tomb.v1/tomb.go:113 +0x4a
github.com/juju/juju/cmd/jujud/agent.(*MachineAgent).Stop(0xc42129ad20,
0x37a9ae0, 0xc42201f978)
/workspace/src/github.com/juju/juju/cmd/jujud/agent/machine.go:371 +0x41
github.com/juju/juju/featuretests.(*upgradeSuite).TestLoginsDuringUpgrade.func3(0xc4202f8d20,
0xc42129ad20)
/workspace/src/github.com/juju/juju/featuretests/upgrade_test.go:139 +0x2b
github.com/juju/juju/featuretests.(*upgradeSuite).TestLoginsDuringUpgrade(0xc4203fbd40,
0xc4202f8d20)
/workspace/src/github.com/juju/juju/featuretests/upgrade_test.go:165 +0x8aa
reflect.Value.call(0x3564220, 0xc4203fbd40, 0x9e13, 0x3627083, 0x4,
0xc420eeef70, 0x1, 0x1, 0x5698e60, 0x3523e00, ...)
/snap/go/1739/src/reflect/value.go:447 +0x969
reflect.Value.Call(0x3564220, 0xc4203fbd40, 0x9e13, 0xc421526f70, 0x1, 0x1,
0xc4202f8e10, 0xc421526f07, 0x3)
/snap/go/1739/src/reflect/value.go:308 +0xa4
gopkg.in/check%2ev1.(*suiteRunner).forkTest.func1(0xc4202f8d20)
/workspace/src/gopkg.in/check.v1/check.go:772 +0x575
gopkg.in/check%2ev1.(*suiteRunner).forkCall.func1(0xc420c80300,
0xc4202f8d20, 0xc4214850c0)
/workspace/src/gopkg.in/check.v1/check.go:666 +0x7c
created by gopkg.in/check%2ev1.(*suiteRunner).forkCall
/workspace/src/gopkg.in/check.v1/check.go:663 +0x20c

or possibly
goroutine 24826 [chan receive, 23 minutes]:
github.com/juju/juju/featuretests.(*upgradeSuite).SetUpTest.func1(0xc4200ee8a0)
/workspace/src/github.com/juju/juju/featuretests/upgrade_test.go:88 +0x49
created by github.com/juju/juju/featuretests.(*upgradeSuite).SetUpTest
/workspace/src/github.com/juju/juju/featuretests/upgrade_test.go:87 +0x24a

On Wed, Feb 21, 2018 at 11:21 AM, John A Meinel <email address hidden>
wrote:

> I got a different failure in this same test recently.
> http://ci.jujucharms.com/job/github-merge-juju/101/console
> LOG] 0:00.983 ERROR juju.worker.dependency "api-server" manifold worker
> returned unexpected error: listen tcp :44094: bind: address already in use
> [LOG] 0:00.983 DEBUG juju.worker.dependency stack trace:
> listen tcp :44094: bind: address already in use
> github.com/juju/juju/worker/apiserver/worker.go:140:
> github.com/juju/juju/worker/apiserver/manifold.go:163:
> [LOG] 0:03.667 DEBUG juju.worker.apicaller failed to connect
> [LOG] 0:03.667 DEBUG juju.worker.dependency "api-caller" manifold worker
> stopped: cannot open api: try was stopped
> [LOG] 0:03.726 DEBUG juju.worker.apicaller connecting with current password
> [LOG] 0:03.726 DEBUG juju.api looked up localhost -> [127.0.0.1]
> [LOG] 0:04.040 DEBUG juju.worker.dependency "api-server" manifold worker
> stopped: listen tcp :44094: bind: address already in use
> [LOG] 0:04.040 ERROR juju.worker.dependency "api-server" manifold worker
> returned unexpected error: listen tcp :44094: bind: address already in...

Read more...

Revision history for this message
John A Meinel (jameinel) wrote :
Download full text (6.2 KiB)

And again:
http://ci.jujucharms.com/job/github-merge-juju/634/consoleText

On Thu, Apr 19, 2018 at 8:01 PM, John Meinel <email address hidden> wrote:

> Something like this might have happened again.
> http://ci.jujucharms.com/job/github-merge-juju/374/
> testReport/junit/github/com_juju_juju_featuretests/TestPackage/
>
> In this case we got some sort of 25min deadlock triggering the test suite
> timeout.
> The stack trace seems to indicate
> goroutine 24709 [chan receive, 23 minutes]:
> gopkg.in/tomb%2ev1.(*Tomb).Wait(0xc42129ad30, 0xc42201f968, 0x5957a9)
> /workspace/src/gopkg.in/tomb.v1/tomb.go:113 +0x4a
> github.com/juju/juju/cmd/jujud/agent.(*MachineAgent).Stop(0xc42129ad20,
> 0x37a9ae0, 0xc42201f978)
> /workspace/src/github.com/juju/juju/cmd/jujud/agent/machine.go:371 +0x41
> github.com/juju/juju/featuretests.(*upgradeSuite).
> TestLoginsDuringUpgrade.func3(0xc4202f8d20, 0xc42129ad20)
> /workspace/src/github.com/juju/juju/featuretests/upgrade_test.go:139 +0x2b
> github.com/juju/juju/featuretests.(*upgradeSuite).TestLoginsDuringUpgrade(
> 0xc4203fbd40, 0xc4202f8d20)
> /workspace/src/github.com/juju/juju/featuretests/upgrade_test.go:165
> +0x8aa
> reflect.Value.call(0x3564220, 0xc4203fbd40, 0x9e13, 0x3627083, 0x4,
> 0xc420eeef70, 0x1, 0x1, 0x5698e60, 0x3523e00, ...)
> /snap/go/1739/src/reflect/value.go:447 +0x969
> reflect.Value.Call(0x3564220, 0xc4203fbd40, 0x9e13, 0xc421526f70, 0x1,
> 0x1, 0xc4202f8e10, 0xc421526f07, 0x3)
> /snap/go/1739/src/reflect/value.go:308 +0xa4
> gopkg.in/check%2ev1.(*suiteRunner).forkTest.func1(0xc4202f8d20)
> /workspace/src/gopkg.in/check.v1/check.go:772 +0x575
> gopkg.in/check%2ev1.(*suiteRunner).forkCall.func1(0xc420c80300,
> 0xc4202f8d20, 0xc4214850c0)
> /workspace/src/gopkg.in/check.v1/check.go:666 +0x7c
> created by gopkg.in/check%2ev1.(*suiteRunner).forkCall
> /workspace/src/gopkg.in/check.v1/check.go:663 +0x20c
>
> or possibly
> goroutine 24826 [chan receive, 23 minutes]:
> github.com/juju/juju/featuretests.(*upgradeSuite).
> SetUpTest.func1(0xc4200ee8a0)
> /workspace/src/github.com/juju/juju/featuretests/upgrade_test.go:88 +0x49
> created by github.com/juju/juju/featuretests.(*upgradeSuite).SetUpTest
> /workspace/src/github.com/juju/juju/featuretests/upgrade_test.go:87 +0x24a
>
>
> On Wed, Feb 21, 2018 at 11:21 AM, John A Meinel <email address hidden>
> wrote:
>
>> I got a different failure in this same test recently.
>> http://ci.jujucharms.com/job/github-merge-juju/101/console
>> LOG] 0:00.983 ERROR juju.worker.dependency "api-server" manifold worker
>> returned unexpected error: listen tcp :44094: bind: address already in use
>> [LOG] 0:00.983 DEBUG juju.worker.dependency stack trace:
>> listen tcp :44094: bind: address already in use
>> github.com/juju/juju/worker/apiserver/worker.go:140:
>> github.com/juju/juju/worker/apiserver/manifold.go:163:
>> [LOG] 0:03.667 DEBUG juju.worker.apicaller failed to connect
>> [LOG] 0:03.667 DEBUG juju.worker.dependency "api-caller" manifold worker
>> stopped: cannot open api: try was stopped
>> [LOG] 0:03.726 DEBUG juju.worker.apicaller connecting with current
>> password
>> [LOG] 0:03.726 DEBUG juju.api looked up localhost -> [127.0.0.1]
>> [LOG] 0:04.040 DE...

Read more...

Revision history for this message
Anastasia (anastasia-macmood) wrote :

Interestingly, I cannot get this test to fail at all locally.. not even under stress. We must have fixed these issues as part of other work in the area.

Changed in juju:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.