Failed to deploy on ironic_upload_images and clear_node_reservations_for_conductor on cluster with Ironic, TLS and Nova Quotas

[Kyrylo Romanenko]

Detailed bug description:
Failed ironic_upload_images task and clear_node_reservations_for_conductor on cluster deployment with TLS and Nova Quotas.

Steps to reproduce:
1) Setup cluster settings with following configuration:
1 Controller+Cinder
1 Compute
1 Ironic Conductor

Neutron VLAN, QEMU hypervisor.

Set "Nova Qotas", "Debug Logging", "TLS for OpenStack public endpoints", "HTTPS for Horizon"
2) Start deployment.

Expected results:
  Deployed cluster expected.
Actual result:
  Cluster failed to deploy.

Snippet from ironic conductor log:
http://paste.openstack.org/show/516879/
From Glance API:
http://paste.openstack.org/show/516880/

Diagnostic snapshot: https://drive.google.com/a/mirantis.com/file/d/0B6E70aHvCcRQLWphTkJVQWlTeFU/view?usp=sharing

cat /etc/fuel_build_id:
 490
cat /etc/fuel_build_number:
 490
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6349.noarch
 fuel-misc-9.0.0-1.mos8460.noarch
 python-packetary-9.0.0-1.mos140.noarch
 fuel-bootstrap-cli-9.0.0-1.mos285.noarch
 fuel-migrate-9.0.0-1.mos8460.noarch
 rubygem-astute-9.0.0-1.mos750.noarch
 fuel-mirror-9.0.0-1.mos140.noarch
 shotgun-9.0.0-1.mos90.noarch
 fuel-openstack-metadata-9.0.0-1.mos8743.noarch
 fuel-notify-9.0.0-1.mos8460.noarch
 nailgun-mcagents-9.0.0-1.mos750.noarch
 python-fuelclient-9.0.0-1.mos325.noarch
 fuel-9.0.0-1.mos6349.noarch
 fuel-utils-9.0.0-1.mos8460.noarch
 fuel-setup-9.0.0-1.mos6349.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8743.noarch
 fuel-library9.0-9.0.0-1.mos8460.noarch
 network-checker-9.0.0-1.mos74.x86_64
 fuel-agent-9.0.0-1.mos285.noarch
 fuel-ui-9.0.0-1.mos2717.noarch
 fuel-ostf-9.0.0-1.mos936.noarch
 fuelmenu-9.0.0-1.mos274.noarch
 fuel-nailgun-9.0.0-1.mos8743.noarch

[Andrey Shestakov (ashestakov)]

Astute log:
2016-06-17 11:27:01 NOTICE (Scope(Class[main])) MODULAR: ironic_upload_images
2016-06-17 11:30:11 NOTICE (/Stage[main]/Main/Exec[ironic_upload_images_shell]/returns) executed successfully

Puppet log:
2016-06-17 11:29:12 DEBUG [1630] Node[1]: Node 1: task ironic_upload_images, task status running
2016-06-17 11:29:12 WARNING [1630] Puppet agent 1 didn't respond within the allotted time
2016-06-17 11:29:12 DEBUG [1630] Task time summary: ironic_upload_images with status failed on node 1 took 00:03:00

Images uploads more then 3 minutes which caused error by timeout.
In manual run this task takes around 1 minute.

Looks like virtual env was overloaded at that time.

[Kyrylo Romanenko (kromanenko)]

Ok, i`ll recheck on RC2 ISO.

[Kyrylo Romanenko (kromanenko)]

Really it looks to be timeout failure on slow env.