IOError: Socket closed while talking to rabbitmq

Bug #1290386 reported by Vincent Ladeuil on 2014-03-10
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu CI Engine
Medium
Unassigned

Bug Description

Encountered after fixing bug #1290320 filing here as an independent bug.

The bsbuiler fails (didn't check the others but they probably fail in the same way) with:

Traceback (most recent call last):
  File "./branch-source-builder/bsbuilder/run_worker.py", line 91, in <module>
    amqp_utils.process_queue(config, 'bsbuilder', on_message)
  File "/srv/bsb_worker/ci-utils/ci_utils/amqp_utils.py", line 101, in process_queue
    conn = connection(config)
  File "/srv/bsb_worker/ci-utils/ci_utils/amqp_utils.py", line 48, in connection
    password=rabbit_config.AMQP_PASSWORD
  File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 144, in __init__
    (10, 30), # tune
  File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 95, in wait
    self.channel_id, allowed_methods)
  File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 202, in _wait_method
    self.method_reader.read_method()
  File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method
    raise m
IOError: Socket closed

Vincent Ladeuil (vila) on 2014-03-10
Changed in ubuntu-ci-services-itself:
status: Confirmed → In Progress
Vincent Ladeuil (vila) wrote :

<doanac> 1) rabbit might not be ready, so we could just add more retries (and maybe increase the delay betwen them)
<doanac> 2) we have some weird network restriction in hpcloud between the 2 nodes for some reason
<doanac> #1 seems more likely

<vila> doanac: argh, I've been chasing ghosts >-/ I was fooled by the last line: 'KeyboardInterrupt' but in fact, in my current deployment I have *another* line after that: INFO:root:Waiting for messages. ^C to exit.
<vila> and if I restart the worker: I get:
<vila> INFO:__main__:exiting
<vila> INFO:root:Waiting for messages. ^C to exit.
<doanac> vila: oh - so things are working "properly" then?
<vila> doanac: so the messages were for previous attempts to connect to the server and I may have encounter a case where it took too long and then failed to properly identify it
<plars> how is it getting a ^C?
<vila> doanac: seems so
<vila> plars: it doesn't, it says gimme one
<plars> ah, I thought it was exiting
<plars> gotcha
<doanac> plars: upstart is configured to send ctrl-c to the job when it restarts it
<plars> ack
<vila> plars: well, yeah, what doanac said, it did received one
<doanac> vila: don't be sad, that's one bug down! :)
<doanac> also note: with the MP i landed last night, you can now view from the webui what services have their workers attached
<vila> doanac: will change the bug to ask for early connections to be trapped internally so we put stuff in logs only when it takes really too long
<doanac> so its much more obvious now
<vila> doanac: rats, I should have thought about that too
<doanac> vila: ack, and lets move it into the backlog also.
<doanac> vila: there are cases where that exception is important (ie if rabbit goes offline)
<doanac> so the logging needs to be smarter

Changed in ubuntu-ci-services-itself:
milestone: phase-0 → backlog
assignee: Vincent Ladeuil (vila) → nobody
status: In Progress → Confirmed
importance: Critical → Medium
Ursula Junque (ursinha) on 2014-03-26
Changed in uci-engine:
importance: Undecided → Medium
milestone: none → backlog
status: New → Confirmed
Evan (ev) on 2014-03-31
no longer affects: ubuntu-ci-services-itself
Vincent Ladeuil (vila) on 2014-04-01
summary: - IOError: Socket closed while taking to rabbitmq
+ IOError: Socket closed while talking to rabbitmq
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers