Failure to execute create_swift_temp_url_key.sh during undercloud upgrade

Bug #1768328 reported by Emilien Macchi
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Emilien Macchi

Bug Description

When upgrading a non-containerized undercloud (queens) to a containerized undercloud (rocky), ironic-conductor container fails to start because create_swift_temp_url_key.sh requires Keystone to be running but it seems like it's not the case at the time the script is running:

"Error running ['docker', 'run', '--name', 'create_swift_temp_url_key', '--label', 'config_id=tripleo_step4', '--label', 'container_name=create_swift_temp_url_key', '--label', 'managed_by=paunch', '--label', 'config_data={\"start_order\": 70, \"command\": \"/usr/bin/bootstrap_host_exec ironic_conductor /create_swift_temp_url_key.sh\", \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/lib/config-data/puppet-generated/ironic/etc/ironic:/etc/ironic:ro\", \"/var/lib/docker-config-scripts/create_swift_temp_url_key.sh:/create_swift_temp_url_key.sh:ro\"], \"image\": \"docker.io/tripleomaster/centos-binary-ironic-conductor:757e1326921026fb4ce23ede1e4742d586b65ab2_642db7af\", \"detach\": false, \"net\": \"host\"}', '--net=host', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/var/lib/config-data/puppet-generated/ironic/etc/ironic:/etc/ironic:ro', '--volume=/var/lib/docker-config-scripts/create_swift_temp_url_key.sh:/create_swift_temp_url_key.sh:ro', 'docker.io/tripleomaster/centos-binary-ironic-conductor:757e1326921026fb4ce23ede1e4742d586b65ab2_642db7af', '/usr/bin/bootstrap_host_exec', 'ironic_conductor', '/create_swift_temp_url_key.sh']. [1]",
        "stdout: Check if a temporary URL key already exists",
        "Creating a new temporary URL for project service",
        "stderr: Failed to discover available identity versions when contacting http://192.168.24.3:5000. Attempting to parse version from URL.",
        "Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Service Unavailable (HTTP 503)",

We probably want to make sure ironic-conductor container starts after Keystone is actually ready to operate.

Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

I strongly believe adding containers start-up/deployment dependencies is a no go. Could we instead modify those to become post-deploy steps?

Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

I wonder, if https://review.openstack.org/#/c/565543 fixes this as well?

Revision history for this message
Emilien Macchi (emilienm) wrote :

I added more logs here: http://paste.openstack.org/show/a4vRzHRlTCBPnC1mHawy/

Note: I can consistently reproduce it. I'll keep debugging but any help is welcome.

Revision history for this message
Emilien Macchi (emilienm) wrote :

so I'm having issues to reproduce it in CI, I wonder if that happens on my laptop because it's so sloooow ? I'll keep digging.

Revision history for this message
Emilien Macchi (emilienm) wrote :

I don't hit this one anymore.

Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.