StructuredDeployment fails to delete with Property error: default: could not convert string to float

[Marius Cornea]

Upgrading an all-in-one controller node overcloud from Ocata to Pike fails with:

(undercloud) [stack@undercloud-0 ~]$ openstack stack failures list overcloud
overcloud.AllNodesDeploySteps.ControllerUpgradeBatch_Step0:
  resource_type: OS::Heat::SoftwareDeploymentGroup
  physical_resource_id: 3dc2087b-447a-4029-912b-a4388fad8829
  status: CREATE_FAILED
  status_reason: |
    CREATE aborted
overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 51321724-e524-45f7-8288-2bd6314c5f39
  status: DELETE_FAILED
  status_reason: |
    ValueError: resources[0]: Property error: default: could not convert string to float:
  deploy_stdout: |
    ...
    Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::service::begin]: Triggered 'refresh' from 2 events
    Notice: /Stage[main]/Neutron::Agents::Dhcp/Service[neutron-dhcp-service]: Triggered 'refresh' from 1 events
    Notice: /Stage[main]/Neutron::Server/Service[neutron-server]: Triggered 'refresh' from 1 events
    Notice: /Stage[main]/Neutron::Agents::L3/Service[neutron-l3]: Triggered 'refresh' from 1 events
    Notice: /Stage[main]/Neutron::Agents::Metadata/Service[neutron-metadata]: Triggered 'refresh' from 1 events
    Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Service[neutron-ovs-agent-service]/ensure: ensure changed 'stopped' to 'running'
    Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::service::end]: Triggered 'refresh' from 5 events
    Notice: /Stage[main]/Nova::Cron::Archive_deleted_rows/Cron[nova-manage db archive_deleted_rows]/ensure: created
    Notice: /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]/returns: executed successfully
    Notice: Applied catalog in 147.59 seconds
    (truncated, view all with --long)
  deploy_stderr: |
    ...
    Warning: You cannot collect exported resources without storeconfigs being set; the collection will be ignored at /etc/puppet/modules/panko/manifests/api.pp:110:5
    Warning: Scope(Class[Nova::Compute]): compute_manager is marked as deprecated in Nova but still needed when Ironic is used. It will be removed once Nova removes it.
    Warning: Scope(Oslo::Messaging::Rabbit[cinder_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
    Warning: Scope(Oslo::Messaging::Rabbit[keystone_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
    Warning: Scope(Oslo::Messaging::Rabbit[glance_api_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
    Warning: Scope(Oslo::Messaging::Rabbit[glance_registry_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
    Warning: Scope(Oslo::Messaging::Rabbit[neutron_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
    Warning: Scope(Oslo::Messaging::Rabbit[ceilometer_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
    Warning: Scope(Oslo::Messaging::Rabbit[aodh_config]): The oslo_messaging rabbit_host, rabbit_hosts, rabbit_port, rabbit_userid, rabbit_password, rabbit_virtual_host parameters have been deprecated by the [DEFAULT]\transport_url. Please use oslo::messaging::default::transport_url instead.
    Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.
    (truncated, view all with --long)

Upgrade command:

(undercloud) [stack@undercloud-0 ~]$ cat overcloud_upgrade_docker.sh
source ~/stackrc
export THT=/usr/share/openstack-tripleo-heat-templates/

openstack overcloud deploy --templates $THT \
-r ~/openstack_deployment/roles/roles_data.yaml \
-e $THT/environments/network-isolation.yaml \
-e $THT/environments/network-management.yaml \
-e $THT/environments/tls-endpoints-public-ip.yaml \
-e ~/openstack_deployment/environments/nodes.yaml \
-e ~/openstack_deployment/environments/network-environment.yaml \
-e ~/openstack_deployment/environments/public_vip.yaml \
-e ~/openstack_deployment/environments/enable-tls.yaml \
-e ~/openstack_deployment/environments/inject-trust-anchor.yaml \
-e ~/openstack_deployment/environments/neutron-settings.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps-docker.yaml \
-e ~/containers-default-parameters.yaml

[Artem Hrechanychenko (ahrechan)]

Reproduced!
 export THT=/usr/share/openstack-tripleo-heat-templates
    #workaround for VLAN 10 rules
    ./overcloud-prep-network.sh
    openstack overcloud deploy \
    --libvirt-type qemu \
    --ntp-server clock.redhat.com \
    --control-scale 1 \
    --control-flavor oooq_control \
    --compute-flavor oooq_compute \
    --templates $THT \
    -e $THT/environments/low-memory-usage.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
    -e /home/stack/network-environment.yaml \
    -e $THT/environments/docker.yaml \
    -e $THT/environments/major-upgrade-composable-steps-docker.yaml \
    -e ~/containers-default-parameters.yaml \
    -e ~/containers-upgrade-repos.yaml

paste.openstack.org/show/612398/

[Jiří Stránský (jistr)]

An important thing is that the error happens on delete action, not create/update:

overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 51321724-e524-45f7-8288-2bd6314c5f39
  status: DELETE_FAILED
  status_reason: |
    ValueError: resources[0]: Property error: default: could not convert string to float:

I don't think Heat should ever fail to delete a software deployment like this, it's probably a Heat bug.

[Jiří Stránský (jistr)]

Full failures list from mcornea http://paste.openstack.org/show/612387/

[Thomas Herve (therve)]

Do you a heat traceback for that error?

[Marius Cornea (mcornea)]

Attaching heat-engine log.

[Thomas Herve (therve)]

That looks like the issue may have been introduced by https://review.openstack.org/#/c/462440/ . We used to not have a schema for inputs, and setting one seems to create issues. But that should only be triggered if a config as inputs with both type and default (and a number type) which seems pretty rare in tht.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/475817

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/475817
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=f911ebbd670babc8052ec13bf829f6b3216db9c3
Submitter: Jenkins
Branch: master

commit f911ebbd670babc8052ec13bf829f6b3216db9c3
Author: Thomas Herve <email address hidden>
Date: Tue Jun 20 16:21:11 2017 +0200

    Ignore string default for software config input

    We recently (I87edc0d8f1fba2fb61276c682a60c1b2241b5705) enforced default
    type for software config input, whereas it was always a string before.
    It causes issues with upgrades, as we used to store "" as a default for
    all configs regarless of their type, and it failed when we started
    validating.

    This identifies this case and skip validation.

    Change-Id: I8a4f4e834b0862ecef4b97f208f4b03be3572e86
    Closes-Bug: #1697627

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 9.0.0.0b3 development milestone.