salt-minion test needs fixing

[Scott Moser]

cloud-init-integration-lxd-b test 200 failed.
Looking at the console log at [1] shows
  boot instance encountered error: ssh: after 120s instance is not reachable
at 2018-06-26 11:23:47,693 which was indeed 120 seconds after starting the
salt minion test.

The summary of the failure is just that salt failed to restart due to dns
lookup problems of a host 'salt'.

The corresponding cloud-init log [2] progresses well until

  2018-06-26 11:22:16,726 - util.py[DEBUG]: Running command ['service', 'salt-minion', 'restart'] with allowed return codes [0] (shell=False, capture=False)
  2018-06-26 11:23:47,032 - handlers.py[DEBUG]: finish: modules-final/config-salt-minion: SUCCESS: config-salt-minion ran successfully

So 'service salt-minion restart' took ~ 90 seconds.
 $ journalctl --file=system.journal --unit=salt-minion.service | pastebinit
 http://paste.ubuntu.com/p/jjkrs3nhYF/

Lines of most interest there are 3 times:
  Jun 26 07:22:17 [1241]: [ERROR ] DNS lookup or connection check of 'salt' failed.
  Jun 26 07:22:17 [1241]: [ERROR ] Master hostname: 'salt' not found or not responsive. Retrying in 30 seconds

[1] https://jenkins.ubuntu.com/server/job/cloud-init-integration-lxd-b/200/consoleFull
[2] https://jenkins.ubuntu.com/server/job/cloud-init-integration-lxd-b/200/artifact/cloud-init/results/lxd/bionic/modules/salt_minion/cloud-init.log

[Scott Moser (smoser)]

Interesting, this does not seem to always fail.
Here is a
journalctl --file=log2 --unit=salt-minion.service
http://paste.ubuntu.com/p/wkBVJtW586/

from a run of the same test on the day before (build 199).
https://jenkins.ubuntu.com/server/job/cloud-init-integration-lxd-b/199/artifact/cloud-init/results/lxd/bionic/modules/

[Scott Moser (smoser)]

Wow, heres another error log of a passing (cloud-init-intergration-lxd-b/202)
$ journalctl --file=passed.journal --unit=salt-minion.service | pastebinit
http://paste.ubuntu.com/p/WYRPkfvVDH/

I really am kind of thinking this is a salt-minion issue at this point.

this is at best strange behavior out of the gate for 'apt-get install salt-minion'.

[Scott Moser (smoser)]

This bug is fixed with commit 58581362 to cloud-init on branch master.

To view that commit see the following URL:
https://git.launchpad.net/cloud-init/commit/?id=58581362

[Scott Moser (smoser)]

This bug is believed to be fixed in cloud-init in version 18.4. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3198