rax-nova-agent does not drain the event queue, before starting

Bug #1765057 reported by Dimitri John Ledkov on 2018-04-18
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
rax-nova-agent (Ubuntu)
Undecided
Unassigned

Bug Description

A single iteration of event processing is not sufficient to drain all of the events on firstboot provisioning in Xen based clouds.

Specifically, instead of receiving a list of 7 events, one might only receive 1 event at a time, spaced 1 or 2 seconds apart, until all initial events are processed.

To work around the fact that supervisor is not batching all events together, do the following:

* process events until there are none
* ensure that there were at least three iterations without any events
* then notify that daemon as ready, as no more early boot events are expected
* continue polling any interactive triggered events going forward

Dimitri John Ledkov (xnox) wrote :
Download full text (3.3 KiB)

Tested on a self-created image:

2018-04-18 14:43:48,129 [INFO ] Skipping os.fork as directed by arguments
2018-04-18 14:43:48,129 [INFO ] Starting actions for novaagent.libs.debian
2018-04-18 14:43:48,130 [INFO ] Checking for existence of /dev/xen/xenbus
2018-04-18 14:43:48,131 [INFO ] Provider: Rackspace
2018-04-18 14:43:48,144 [INFO ] Event: 77bc84f1-1810-4089-946e-668ccb3d1e86 -> version
2018-04-18 14:43:48,145 [INFO ] Returning {"message": "2.1.13", "returncode": "0"}
2018-04-18 14:43:49,148 [INFO ] Event: 5fd73e82-fe51-4463-be63-e73928cf05be -> features
2018-04-18 14:43:49,149 [INFO ] Returning {"message": "kmsactivate,resetnetwork,version,keyinit,features,password,injectfile", "returncode": "0"}
2018-04-18 14:43:50,151 [INFO ] Event: b0f171aa-c1ab-416d-a51c-0a72a18ee4ea -> injectfile
2018-04-18 14:43:50,153 [INFO ] Returning {"message": "", "returncode": "0"}
2018-04-18 14:43:51,155 [INFO ] Event: ce18fecb-128d-4ed3-b88f-79ea05bb2c50 -> keyinit
2018-04-18 14:43:51,156 [INFO ] Returning {"message": "63332560236465421491477949819793", "returncode": "D0"}
2018-04-18 14:43:52,160 [INFO ] Event: 8a591e8d-b24f-48a8-9bc0-e75c32a947b8 -> password
2018-04-18 14:43:52,201 [INFO ] Returning {"message": "", "returncode": "0"}
2018-04-18 14:43:53,204 [INFO ] Event: d9edad5f-737d-482f-8744-c48384a1e89d -> resetnetwork
2018-04-18 14:43:53,205 [INFO ] Hostname: xnox-test-counter
2018-04-18 14:43:53,205 [INFO ] Backing up -> /etc/hostname (1524062633.2054577.bak)
2018-04-18 14:43:53,212 [ERROR] Error using hostnamectl: b'Failed to create bus connection: No such file or directory\n'
2018-04-18 14:43:53,218 [INFO ] Interface BC764E20680D: {'label': 'private', 'broadcast': '10.209.95.255', 'ips': [{'ip': '10.209.64.42', 'netmask': '255.255.224.0', 'enabled': '1', 'gateway': None}], 'mac': 'BC:76:4E:20:68:0D', 'routes': [{'route': '10.208.0.0', 'netmask': '255.240.0.0', 'gateway': '10.209.64.1'}, {'route': '10.176.0.0', 'netmask': '255.240.0.0', 'gateway': '10.209.64.1'}], 'gateway': None}
2018-04-18 14:43:53,219 [INFO ] Interface BC764E206804: {'ip6s': [{'ip': '2001:4802:7805:101:be76:4eff:fe20:6804', 'netmask': 64, 'enabled': '1', 'gateway': 'fe80::def'}], 'label': 'public', 'broadcast': '104.239.175.255', 'ips': [{'ip': '104.239.175.86', 'netmask': '255.255.255.0', 'enabled': '1', 'gateway': '104.239.175.1'}], 'mac': 'BC:76:4E:20:68:04', 'gateway_v6': 'fe80::def', 'dns': ['69.20.0.164', '69.20.0.196'], 'gateway': '104.239.175.1'}
2018-04-18 14:43:53,444 [INFO ] Returning {"message": "", "returncode": "0"}
2018-04-18 14:43:54,446 [INFO ] No event received
2018-04-18 14:43:55,448 [INFO ] No event received
2018-04-18 14:43:56,450 [INFO ] No event received

and

Apr 18 14:43:47 ubuntu systemd[1]: Starting Nova Agent for xenstore...
Apr 18 14:43:52 ubuntu passwd[729]: pam_unix(passwd:chauthtok): password changed for root
Apr 18 14:43:57 xnox-test-counter systemd[1]: Started Nova Agent for xenstore.
Apr 18 14:43:57 xnox-test-counter systemd[1]: Starting Initial cloud-init job (pre-networking)...
Apr 18 14:43:58 xnox-test-counter cloud-init[763]: Cloud-init v. 18.2 running 'init-local' at Wed, 18 Apr 2018 14:43:58 +0000. Up 16.60 seconds.
Apr 18 14:43:58 xnox-...

Read more...

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package rax-nova-agent - 2.1.13-0ubuntu3

---------------
rax-nova-agent (2.1.13-0ubuntu3) bionic; urgency=medium

  * Drain the events queue, until there are no more events, for at least 3
    iterations, before sending started notification. LP: #1765057

 -- Dimitri John Ledkov <email address hidden> Wed, 18 Apr 2018 14:59:45 +0100

Changed in rax-nova-agent (Ubuntu):
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers