Encountered after fixing bug #1290320 filing here as an independent bug.
The bsbuiler fails (didn't check the others but they probably fail in the same way) with:
Traceback (most recent call last):
File "./branch-source-builder/bsbuilder/run_worker.py", line 91, in <module>
amqp_utils.process_queue(config, 'bsbuilder', on_message)
File "/srv/bsb_worker/ci-utils/ci_utils/amqp_utils.py", line 101, in process_queue
conn = connection(config)
File "/srv/bsb_worker/ci-utils/ci_utils/amqp_utils.py", line 48, in connection
password=rabbit_config.AMQP_PASSWORD
File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 144, in __init__
(10, 30), # tune
File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 95, in wait
self.channel_id, allowed_methods)
File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 202, in _wait_method
self.method_reader.read_method()
File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method
raise m
IOError: Socket closed
<doanac> 1) rabbit might not be ready, so we could just add more retries (and maybe increase the delay betwen them)
<doanac> 2) we have some weird network restriction in hpcloud between the 2 nodes for some reason
<doanac> #1 seems more likely
<vila> doanac: argh, I've been chasing ghosts >-/ I was fooled by the last line: 'KeyboardInterrupt' but in fact, in my current deployment I have *another* line after that: INFO:root:Waiting for messages. ^C to exit.
<vila> and if I restart the worker: I get:
<vila> INFO:__
<vila> INFO:root:Waiting for messages. ^C to exit.
<doanac> vila: oh - so things are working "properly" then?
<vila> doanac: so the messages were for previous attempts to connect to the server and I may have encounter a case where it took too long and then failed to properly identify it
<plars> how is it getting a ^C?
<vila> doanac: seems so
<vila> plars: it doesn't, it says gimme one
<plars> ah, I thought it was exiting
<plars> gotcha
<doanac> plars: upstart is configured to send ctrl-c to the job when it restarts it
<plars> ack
<vila> plars: well, yeah, what doanac said, it did received one
<doanac> vila: don't be sad, that's one bug down! :)
<doanac> also note: with the MP i landed last night, you can now view from the webui what services have their workers attached
<vila> doanac: will change the bug to ask for early connections to be trapped internally so we put stuff in logs only when it takes really too long
<doanac> so its much more obvious now
<vila> doanac: rats, I should have thought about that too
<doanac> vila: ack, and lets move it into the backlog also.
<doanac> vila: there are cases where that exception is important (ie if rabbit goes offline)
<doanac> so the logging needs to be smarter