install-undercloud.log missing ansible logs

[Emilien Macchi]

Probably related to the switch to Ansible runner but now the install-undercloud.log file doesn't have the Ansible logs like we had before:

2020-01-10 13:51:36.760 961513 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] ** Downloading undercloud ansible.. **
2020-01-10 13:51:46.062 961513 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] Generating default ansible config file /home/vagrant/ansible.cfg
2020-01-10 14:14:16.483 961513 ERROR tripleoclient.utils.utils [ ] Ansible execution failed. playbook: deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2
2020-01-10 14:14:16.483 961513 ERROR tripleoclient.v1.tripleo_deploy.Deploy [ ] Exception: Ansible execution failed. playbook: deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2
2020-01-10 14:14:16.485 961513 ERROR tripleoclient.v1.tripleo_deploy.Deploy [ ] None
2020-01-10 14:14:17.132 961513 ERROR tripleoclient.v1.tripleo_deploy.Deploy [ ] ** Found ansible errors for undercloud deployment! **
2020-01-10 14:14:17.133 961513 ERROR tripleoclient.v1.tripleo_deploy.Deploy [ ] [
 [
  "Enable and start systemd service for neutron_dhcp",
  {
   "invocation": {
    "module_args": {
     "no_block": false,
     "force": null,
     "name": "tripleo_neutron_dhcp.service",
     "daemon_reexec": false,
     "enabled": true,
     "daemon_reload": true,
     "state": "restarted",
     "user": null,
     "scope": null,
     "masked": null
    }
   },
   "msg": "Unable to start service tripleo_neutron_dhcp.service: Job for tripleo_neutron_dhcp.service failed because a configured resource limit was exceeded. See \"systemctl status tripleo_neutron_dhcp.service\" and \"journalctl -xe\" for details.\n",
   "changed": false,
   "_ansible_no_log": false
  }
 ]
]
2020-01-10 14:14:17.133 961513 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] Not cleaning working directory /home/vagrant/tripleo-heat-installer-templates
2020-01-10 14:14:17.155 961513 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] Not cleaning ansible directory /home/vagrant/undercloud-ansible-LHiho4
2020-01-10 14:14:17.175 961513 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] Install artifact is located at /home/vagrant/undercloud-install-20200110141416.tar.bzip2
2020-01-10 14:14:17.175 961513 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ]
########################################################

Deployment successful!

########################################################

2020-01-10 14:14:17.175 961513 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] Writing the stack virtual update mark file /var/lib/tripleo-heat-installer/update_mark_undercloud
2020-01-10 14:14:17.242 961513 ERROR tripleoclient.utils.utils [ ] reset failed: reset: standard error: Inappropriate ioctl for device

2020-01-10 14:14:17.242 961513 WARNING tripleoclient.utils.utils [ ] Unable to reset command line. Try manually running "reset" if the command line is broken.
2020-01-10 14:14:17.324 961266 WARNING tripleoclient.v1.undercloud.UpgradeUndercloud [ ]
##########################################################

The Undercloud has been successfully upgraded.

Useful files:

Password file is at ~/undercloud-passwords.conf
The stackrc file is at ~/stackrc

Use these files to interact with OpenStack services, and
ensure they are secured.

##########################################################

[Emilien Macchi (emilienm)]

Note that ansible logs are now in ~/ansible.log which we need to bring back to install-undercloud.log if possible.

[Cédric Jeanneret (cjeanner)]

If it can help, the logs are in ~/ansible.log - iirc there were some issues with the way ansible-runner manages the callback (output), probably related.

[Cédric Jeanneret (cjeanner)]

So after some more digging, there's static ANSIBLE_LOG_PATH set to that ~/ansible.log.

It might be wise to make it more dynamic, for instance using the "undercloud_log_file" internal var when we actually deploy the undercloud.
Beware, I'm not sure how ansible-runner|ansible-playbook manage the log file, i.e. not sure if it's overridden or just appending content.

[Cédric Jeanneret (cjeanner)]

We can probably use the "extra_env_variables" dict for "run_ansible_playbook" method in the deploy. It allows to override things.

Will try to get something with that.

[wes hayutin (weshayutin)]

Thanks guys!

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/702212

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/702212
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=585e41d6b3191accd704f3c750ddf0dca81641e3
Submitter: Zuul
Branch: master

commit 585e41d6b3191accd704f3c750ddf0dca81641e3
Author: Cédric Jeanneret <email address hidden>
Date: Fri Jan 10 16:12:35 2020 +0100

    Inject ansible-runner logs in the install-undercloud.log file

    The move to ansible-runner lib created a regression where ansible runs
    were logged in a separated file (ansible.log) instead of being appended
    to the install-undercloud.log.

    Change-Id: I0ceb90abfd6fcfbb3cbff439f8c348da2a7bfe2f
    Closes-Bug: #1859183

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 13.1.0 release.