upgrade from newton to ocata fails (probably because of change in network params)

[Jan Provaznik]

Supposing I have an OSP10 (newton) overcloud which was deployed with following command:
openstack overcloud deploy --debug \
--templates \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
--control-scale 1 \
--control-flavor controller \--compute-scale 1 \
--compute-flavor compute \
--environment-file /usr/share/openstack-tripleo-heat-templates/environments/manila-netapp-config.yaml \
--neutron-network-type vxlan \
--neutron-tunnel-types vxlan \
-e /home/stack/virt/network/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/CUSTOM_TEMPLATE.yaml \
--log-file overcloud_deployment_12.log

And then I want to upgrade to OSP11 (ocata) so make following changes into the deployment command because --neutron-network-type, --neutron-tunnel-types and '-e' arguments are not accepted by "openstack overcloud deploy" in the new version:

openstack overcloud deploy --debug \
--templates \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
--control-scale 1 \
--control-flavor controller \--compute-scale 1 \
--compute-flavor compute \
--environment-file /usr/share/openstack-tripleo-heat-templates/environments/manila-netapp-config.yaml \
--environment-file /home/stack/virt/network/network-environment.yaml \
--environment-file /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
--environment-file /home/stack/virt/hostnames.yml \
--environment-file /home/stack/virt/CUSTOM_TEMPLATE.yaml \
--environment-file /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps.yaml \
--environment-file /home/stack/overcloud-repos.yaml \
--log-file overcloud_deployment_12.log

and add into parameter_defaults of /home/stack/virt/network/network-environment.yaml:
    NeutronNetworkType: 'vxlan'
    NeutronTunnelTypes: 'vxlan'

Then stack update fails with error:
GET call to orchestration for http://192.168.24.1:8004/v1/98ec47664d6a4aa087acb2016fa66fa2/stacks/overcloud/b3b364d8-a42a-4af0-a46b-511b9ee0d598/events?marker=d056bbc0-405e-43e3-b43a-05dc5d4e96b1&nested_depth=2&sort_dir=asc used request id req-2d36602f-90fc-4288-a690-81e77b1afb39
2017-06-08 09:27:07Z [ServiceNetMap]: UPDATE_IN_PROGRESS state changed
2017-06-08 09:27:07Z [overcloud-ServiceNetMap-o2m2rtahsnhn]: UPDATE_IN_PROGRESS Stack UPDATE started
2017-06-08 09:27:07Z [DefaultPasswords]: UPDATE_IN_PROGRESS state changed
2017-06-08 09:27:07Z [Networks]: UPDATE_IN_PROGRESS state changed
2017-06-08 09:27:07Z [overcloud-Networks-mxuyizmuxa4i]: UPDATE_IN_PROGRESS Stack UPDATE started
2017-06-08 09:27:08Z [overcloud-ServiceNetMap-o2m2rtahsnhn]: UPDATE_COMPLETE Stack UPDATE completed successfully
2017-06-08 09:27:08Z [overcloud-Networks-mxuyizmuxa4i.TenantNetwork]: UPDATE_IN_PROGRESS state changed
2017-06-08 09:27:08Z [overcloud-Networks-mxuyizmuxa4i.InternalNetwork]: UPDATE_IN_PROGRESS state changed
2017-06-08 09:27:08Z [overcloud-Networks-mxuyizmuxa4i-TenantNetwork-mckupko5ntfe]: UPDATE_IN_PROGRESS Stack UPDATE started
2017-06-08 09:27:08Z [overcloud-Networks-mxuyizmuxa4i-InternalNetwork-3vb5uk3tl7oh]: UPDATE_IN_PROGRESS Stack UPDATE started
2017-06-08 09:27:08Z [overcloud-Networks-mxuyizmuxa4i.ExternalNetwork]: UPDATE_IN_PROGRESS state changed
2017-06-08 09:27:09Z [overcloud-Networks-mxuyizmuxa4i.StorageNetwork]: UPDATE_IN_PROGRESS state changed
2017-06-08 09:27:09Z [overcloud-Networks-mxuyizmuxa4i-ExternalNetwork-dhj7rvrd56ti]: UPDATE_IN_PROGRESS Stack UPDATE started
2017-06-08 09:27:09Z [ServiceNetMap]: UPDATE_COMPLETE state changed
2017-06-08 09:27:09Z [overcloud-Networks-mxuyizmuxa4i-TenantNetwork-mckupko5ntfe.TenantNetwork]: CREATE_IN_PROGRESS state changed
2017-06-08 09:27:09Z [DefaultPasswords]: UPDATE_COMPLETE state changed
2017-06-08 09:27:09Z [overcloud-Networks-mxuyizmuxa4i-StorageNetwork-44kitbzxkyvm]: UPDATE_IN_PROGRESS Stack UPDATE started
2017-06-08 09:27:09Z [overcloud-Networks-mxuyizmuxa4i.StorageMgmtNetwork]: UPDATE_IN_PROGRESS state changed
2017-06-08 09:27:09Z [overcloud-Networks-mxuyizmuxa4i-InternalNetwork-3vb5uk3tl7oh.InternalApiNetwork]: CREATE_IN_PROGRESS state changed
2017-06-08 09:27:09Z [overcloud-Networks-mxuyizmuxa4i-ExternalNetwork-dhj7rvrd56ti.ExternalNetwork]: CREATE_IN_PROGRESS state changed
2017-06-08 09:27:09Z [overcloud-Networks-mxuyizmuxa4i-StorageMgmtNetwork-ytghtp73jheg]: UPDATE_IN_PROGRESS Stack UPDATE started
2017-06-08 09:27:09Z [overcloud-Networks-mxuyizmuxa4i-InternalNetwork-3vb5uk3tl7oh.InternalApiNetwork]: CREATE_FAILED Conflict: resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use.
Neutron server returns request_ids: ['req-6b50a8df-3145-46de-9948-9359942365d5']
2017-06-08 09:27:09Z [overcloud-Networks-mxuyizmuxa4i-InternalNetwork-3vb5uk3tl7oh]: UPDATE_FAILED Conflict: resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use.
Neutron server returns request_ids: ['req-6b50a8df-3145-46de-9948-9359942365d5']
2017-06-08 09:27:09Z [overcloud-Networks-mxuyizmuxa4i-StorageNetwork-44kitbzxkyvm.StorageNetwork]: CREATE_IN_PROGRESS state changed
2017-06-08 09:27:09Z [overcloud-Networks-mxuyizmuxa4i-TenantNetwork-mckupko5ntfe.TenantNetwork]: CREATE_FAILED Conflict: resources.TenantNetwork: Unable to create the flat network. Physical network tenant is in use.
Neutron server returns request_ids: ['req-885fc34e-ed39-4f5b-b717-c2ab8166528a']
2017-06-08 09:27:09Z [overcloud-Networks-mxuyizmuxa4i-TenantNetwork-mckupko5ntfe]: UPDATE_FAILED Conflict: resources.TenantNetwork: Unable to create the flat network. Physical network tenant is in use.
Neutron server returns request_ids: ['req-885fc34e-ed39-4f5b-b717-c2ab8166528a']
2017-06-08 09:27:09Z [overcloud-Networks-mxuyizmuxa4i-StorageMgmtNetwork-ytghtp73jheg.StorageMgmtNetwork]: CREATE_IN_PROGRESS state changed
2017-06-08 09:27:09Z [overcloud-Networks-mxuyizmuxa4i-StorageNetwork-44kitbzxkyvm.StorageNetwork]: CREATE_FAILED Conflict: resources.StorageNetwork: Unable to create the flat network. Physical network storage is in use.
Neutron server returns request_ids: ['req-d71a35e5-46ef-475f-a228-dd0a34b621de']
2017-06-08 09:27:09Z [overcloud-Networks-mxuyizmuxa4i-StorageNetwork-44kitbzxkyvm]: UPDATE_FAILED Conflict: resources.StorageNetwork: Unable to create the flat network. Physical network storage is in use.
Neutron server returns request_ids: ['req-d71a35e5-46ef-475f-a228-dd0a34b621de']
2017-06-08 09:27:10Z [overcloud-Networks-mxuyizmuxa4i-ExternalNetwork-dhj7rvrd56ti.ExternalNetwork]: CREATE_FAILED Conflict: resources.ExternalNetwork: Unable to create the flat network. Physical network external is in use.
Neutron server returns request_ids: ['req-f3e3ecbe-ea39-49d1-a4dd-056cf89a1321']
2017-06-08 09:27:10Z [overcloud-Networks-mxuyizmuxa4i-ExternalNetwork-dhj7rvrd56ti]: UPDATE_FAILED Conflict: resources.ExternalNetwork: Unable to create the flat network. Physical network external is in use.
Neutron server returns request_ids: ['req-f3e3ecbe-ea39-49d1-a4dd-056cf89a1321']
2017-06-08 09:27:10Z [overcloud-Networks-mxuyizmuxa4i-StorageMgmtNetwork-ytghtp73jheg.StorageMgmtNetwork]: CREATE_FAILED Conflict: resources.StorageMgmtNetwork: Unable to create the flat network. Physical network storage_mgmt is in use.
Neutron server returns request_ids: ['req-a0374093-a9fe-4cd4-9182-66c4da04fb67']
2017-06-08 09:27:10Z [overcloud-Networks-mxuyizmuxa4i-StorageMgmtNetwork-ytghtp73jheg]: UPDATE_FAILED Conflict: resources.StorageMgmtNetwork: Unable to create the flat network. Physical network storage_mgmt is in use.
Neutron server returns request_ids: ['req-a0374093-a9fe-4cd4-9182-66c4da04fb67']
2017-06-08 09:27:10Z [overcloud-Networks-mxuyizmuxa4i.InternalNetwork]: UPDATE_FAILED resources.InternalNetwork: Conflict: resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use.
Neutron server returns request_ids: ['req-6b50a8df-3145-46de-9948-9359942365d5']
2017-06-08 09:27:10Z [overcloud-Networks-mxuyizmuxa4i.TenantNetwork]: UPDATE_FAILED UPDATE aborted
2017-06-08 09:27:10Z [overcloud-Networks-mxuyizmuxa4i.ExternalNetwork]: UPDATE_FAILED UPDATE aborted
2017-06-08 09:27:10Z [overcloud-Networks-mxuyizmuxa4i.StorageNetwork]: UPDATE_FAILED UPDATE aborted
2017-06-08 09:27:10Z [overcloud-Networks-mxuyizmuxa4i.StorageMgmtNetwork]: UPDATE_FAILED UPDATE aborted
2017-06-08 09:27:10Z [overcloud-Networks-mxuyizmuxa4i]: UPDATE_FAILED resources.InternalNetwork: Conflict: resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use.
Neutron server returns request_ids: ['req-6b50a8df-3145-46de-9948-9359942365d5']
2017-06-08 09:27:11Z [Networks]: UPDATE_FAILED resources.Networks: resources.InternalNetwork: Conflict: resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use.
Neutron server returns request_ids: ['req-6b50a8df-3145-46de-9948-9359942365d5']
2017-06-08 09:27:11Z [overcloud]: UPDATE_FAILED resources.Networks: resources.InternalNetwork: Conflict: resources.InternalApiNetwork: Unable to create the flat network. Physical network internal_api is in use.
Neutron server returns request_ids: ['req-6b50a8df-3145-46de-9948-9359942365d5']

 Stack overcloud UPDATE_FAILED

It looks like network configuration has changed it tries to recreate networks (which of course is not desired state).

[Dan Sneddon (dsneddon)]

I'm not sure what the issue is here, but I don't think this is happening because of the network type parameters being added to network-environment.yaml.

The networks that Heat is trying to rebuild (Internal API, etc.) are not overcloud networks, those are networks defined on the Undercloud instance of Neutron. The NeutronNetworkType and TunnelTypes parameters only affect the overcloud Neutron.

So something is trying to rebuild all the internal networks. The instructions that I've seen for upgrading specify to use "openstack overcloud update" rather than "openstack overcloud deploy". Could that be causing the issue? I know this is for a previous version, but see here:
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/9/html-single/upgrading_red_hat_openstack_platform/

[Emilien Macchi (emilienm)]

This is an automated cleanup. This bug report has been closed because it
is older than 18 months and there is no open code change to fix this.
After this time it is unlikely that the circumstances which lead to
the observed issue can be reproduced.

If you can reproduce the bug, please:
* reopen the bug report (set to status "New")
* AND add the detailed steps to reproduce the issue (if applicable)
* AND leave a comment "CONFIRMED FOR: <RELEASE_NAME>"
  Only still supported release names are valid (FUTURE, PIKE, QUEENS, ROCKY, STEIN).
  Valid example: CONFIRMED FOR: FUTURE