IOError: Socket closed while talking to rabbitmq
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ubuntu CI Engine |
Confirmed
|
Medium
|
Unassigned |
Bug Description
Encountered after fixing bug #1290320 filing here as an independent bug.
The bsbuiler fails (didn't check the others but they probably fail in the same way) with:
Traceback (most recent call last):
File "./branch-
amqp_
File "/srv/bsb_
conn = connection(config)
File "/srv/bsb_
password=
File "/usr/lib/
(10, 30), # tune
File "/usr/lib/
self.
File "/usr/lib/
self.
File "/usr/lib/
raise m
IOError: Socket closed
Changed in ubuntu-ci-services-itself: | |
status: | Confirmed → In Progress |
Changed in uci-engine: | |
importance: | Undecided → Medium |
milestone: | none → backlog |
status: | New → Confirmed |
no longer affects: | ubuntu-ci-services-itself |
summary: |
- IOError: Socket closed while taking to rabbitmq + IOError: Socket closed while talking to rabbitmq |
<doanac> 1) rabbit might not be ready, so we could just add more retries (and maybe increase the delay betwen them)
<doanac> 2) we have some weird network restriction in hpcloud between the 2 nodes for some reason
<doanac> #1 seems more likely
<vila> doanac: argh, I've been chasing ghosts >-/ I was fooled by the last line: 'KeyboardInterrupt' but in fact, in my current deployment I have *another* line after that: INFO:root:Waiting for messages. ^C to exit. main__: exiting
<vila> and if I restart the worker: I get:
<vila> INFO:__
<vila> INFO:root:Waiting for messages. ^C to exit.
<doanac> vila: oh - so things are working "properly" then?
<vila> doanac: so the messages were for previous attempts to connect to the server and I may have encounter a case where it took too long and then failed to properly identify it
<plars> how is it getting a ^C?
<vila> doanac: seems so
<vila> plars: it doesn't, it says gimme one
<plars> ah, I thought it was exiting
<plars> gotcha
<doanac> plars: upstart is configured to send ctrl-c to the job when it restarts it
<plars> ack
<vila> plars: well, yeah, what doanac said, it did received one
<doanac> vila: don't be sad, that's one bug down! :)
<doanac> also note: with the MP i landed last night, you can now view from the webui what services have their workers attached
<vila> doanac: will change the bug to ask for early connections to be trapped internally so we put stuff in logs only when it takes really too long
<doanac> so its much more obvious now
<vila> doanac: rats, I should have thought about that too
<doanac> vila: ack, and lets move it into the backlog also.
<doanac> vila: there are cases where that exception is important (ie if rabbit goes offline)
<doanac> so the logging needs to be smarter