glance-simplestreams-sync doesn't work with 3 rabbits

[Andreas Hasenack]

I deployed landscape 16.03 from the 16.03 series, ceph/swift, with the image sync feature flag enabled. It failed because no images were ever downloaded. The gs3 charm was deployed with no errors, but the cron job was backtracing:

ERROR * 04-01 23:09:03 [PID:21584] * root * Exception during do_sync
Traceback (most recent call last):
  File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 541, in main
    "message": "Sync starting."})
  File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 467, in send_message
    with self.conn.Producer(exchange=self.exchange) as producer:
  File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 660, in Producer
    return Producer(channel or self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 82, in __init__
    self.revive(self._channel)
  File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 216, in revive
    self.declare()
  File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 102, in declare
    self.exchange.declare()
  File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 163, in declare
    return self.channel.exchange_declare(
  File "/usr/lib/python2.7/dist-packages/kombu/abstract.py", line 115, in channel
    channel = self._channel = channel()
  File "/usr/lib/python2.7/dist-packages/kombu/utils/__init__.py", line 405, in __call__
    value = self.__value__ = self.__contract__()
  File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 203, in <lambda>
    channel = ChannelPromise(lambda: connection.default_channel)
  File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 766, in default_channel
    self.connection
  File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 751, in connection
    self._connection = self._establish_connection()
  File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 710, in _establish_connection
    conn = self.transport.establish_connection()
  File "/usr/lib/python2.7/dist-packages/kombu/transport/librabbitmq.py", line 119, in establish_connection
    conn = self.Connection(**opts)
  File "/usr/lib/python2.7/dist-packages/librabbitmq/__init__.py", line 184, in __init__
    self.connect()
ConnectionError: Error opening socket: Bad file descriptor

Further debugging showed that it was trying to connect to a rabbitmq-host with the ip "1" (yes, just 1). Guess why:

in glance-simplestreams-sync.py, StatusExchange() class, _setup_connection() method:
(Pdb) p hosts
'10.245.201.15,10.245.201.253,10.245.201.34' <--- these are my rabbit hosts
(Pdb) l
429
430 id_conf = read_conf(ID_CONF_FILE_NAME)
431
432 import pdb; pdb.set_trace()
433 hosts = id_conf.get('rabbit_hosts', None)
434 -> if hosts is not None:
435 host = hosts[0]
436 else:
437 host = id_conf.get('rabbit_host', None)
438
439 if host is None:
(Pdb) type(hosts)
<type 'str'>
(Pdb)

It works for a single rabbit (or a simple 3 node ceph/ceph deployment) because config variables are different between single and multiple rabbit units.
When deployment has only one rabbit unit: config variable is "rabbit_host", singular, and glance_simplestreams_sync.py gets the variable as is, get("rabbit_host", None), which works.
When deployment has more than one rabbit unit: config variable is "rabbit_hosts", plural, and after retrieving the value it tries to get the first element of the hosts list, which is a char given hosts is a string. A split(",") is missing there.

It's a bug in the charm, but looks like we are using our own copy. Since we will need to change landscape to at least bump the charm revision, I'm filing the bug against landscape first.

[Ursula Junque (ursinha)]

It works for a single rabbit (or a simple 3 node ceph/ceph deployment) because codepath is different as config variables are different for single and multiple rabbit units.

When deployment has only one rabbit unit, config variable is called "rabbit_host", singular, and glance_simplestreams_sync.py gets the variable as is, get("rabbit_host", None), which works.
When deployment has more than one rabbit unit, config variable is called "rabbit_hosts", plural, and after retrieving the value it tries to get the first element of the hosts list, which is a char given hosts is a string. A split(",") is missing there.