Activity log for bug #1831935

Date Who What changed Old value New value Message
2019-06-06 19:31:18 Jeff Hillman bug added bug
2019-06-07 04:44:36 Frode Nordahl information type Public Private
2019-06-07 07:16:28 Frode Nordahl information type Private Public
2019-06-07 07:19:55 Frode Nordahl bug added subscriber OpenStack Charmers
2019-06-07 07:19:59 Frode Nordahl information type Public Private
2019-06-07 07:22:26 Frode Nordahl removed subscriber OpenStack Charmers
2019-06-07 07:25:24 Frode Nordahl description In the scenario where their is no Nuetron Gateway, we're only using provider networking. Specifically VLAN provider networking. The one network and subnet created look as follows: openstack network show mgmt-1 +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | UP | | availability_zone_hints | zone1, zone2, zone3 | | availability_zones | zone1, zone2 | | created_at | 2019-06-06T17:31:19Z | | description | | | dns_domain | | | id | 724aef7a-54a2-4daf-9aa3-98f008215b55 | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | None | | is_vlan_transparent | None | | mtu | 9000 | | name | mgmt-1 | | port_security_enabled | True | | project_id | 143294d60ce54454b451214026857bc9 | | provider:network_type | vlan | | provider:physical_network | physnet1 | | provider:segmentation_id | 1030 | | qos_policy_id | None | | revision_number | 3 | | router:external | Internal | | segments | None | | shared | False | | status | ACTIVE | | subnets | 5d0cf549-4bca-410d-8514-90b805276324 | | tags | | | updated_at | 2019-06-06T17:31:20Z | +---------------------------+--------------------------------------+ $ openstack subnet show mgmt-1 +-------------------+--------------------------------------+ | Field | Value | +-------------------+--------------------------------------+ | allocation_pools | 10.243.160.10-10.243.160.100 | | cidr | 10.243.160.0/24 | | created_at | 2019-06-06T17:31:20Z | | description | | | dns_nameservers | | | enable_dhcp | True | | gateway_ip | 10.243.160.254 | | host_routes | | | id | 5d0cf549-4bca-410d-8514-90b805276324 | | ip_version | 4 | | ipv6_address_mode | None | | ipv6_ra_mode | None | | name | mgmt-1 | | network_id | 724aef7a-54a2-4daf-9aa3-98f008215b55 | | project_id | 143294d60ce54454b451214026857bc9 | | revision_number | 0 | | segment_id | None | | service_types | | | subnetpool_id | None | | tags | | | updated_at | 2019-06-06T17:31:20Z | +-------------------+--------------------------------------+ neutron-openvswitch has enable-local-dhcp-and-metadata set to True. This is verified with: $ openstack network agent list | egrep -i 'dhcp|meta' | 08586252-fe88-48e7-afd8-f95a3500dee1 | Metadata agent | compute15 | None | :-) | UP | neutron-metadata-agent | | 09f469c9-0f02-4e20-9725-3e98120ae704 | DHCP agent | compute1 | zone1 | :-) | UP | neutron-dhcp-agent | | 0a9bc81d-ff79-43a8-b279-ef22290d36d7 | Metadata agent | compute3 | None | :-) | UP | neutron-metadata-agent | | 0c0ecf6a-aa56-4095-9fff-4f22ab99d00c | Metadata agent | compute5 | None | :-) | UP | neutron-metadata-agent | | 170fb80a-f782-4aa7-8ac6-750d00ec0125 | Metadata agent | compute14 | None | :-) | UP | neutron-metadata-agent | | 1e0f1511-11a4-4e43-bdca-722bde6b6c11 | DHCP agent | compute3 | zone1 | :-) | UP | neutron-dhcp-agent | | 22a110e4-f928-4388-87ca-e8ebc9f15554 | DHCP agent | compute2 | zone1 | :-) | UP | neutron-dhcp-agent | | 236387f3-3f24-4075-9cb7-e12280d36438 | DHCP agent | compute5 | zone1 | :-) | UP | neutron-dhcp-agent | | 2d643308-cd4d-4d07-a54a-7ab99167c1b0 | Metadata agent | compute6 | None | :-) | UP | neutron-metadata-agent | | 2f35c4bd-cf44-4187-920a-db5beeab05cf | DHCP agent | compute14 | zone3 | :-) | UP | neutron-dhcp-agent | | 3db2dcbf-bdea-444a-afc6-75b49f0132a8 | DHCP agent | compute12 | zone3 | :-) | UP | neutron-dhcp-agent | | 44a09595-eded-46eb-9d39-036870731433 | Metadata agent | compute8 | None | :-) | UP | neutron-metadata-agent | | 44e02602-f0e3-45f0-9887-9739aaaef3de | DHCP agent | compute8 | zone2 | :-) | UP | neutron-dhcp-agent | | 468e7408-8af1-433c-a77b-e474bc16f9f9 | Metadata agent | compute13 | None | :-) | UP | neutron-metadata-agent | | 51ba61b8-72b5-49cb-a33e-17afe72a4a0b | DHCP agent | compute6 | zone2 | :-) | UP | neutron-dhcp-agent | | 56f99f61-3a6f-48d2-931d-6128d0877486 | DHCP agent | compute10 | zone2 | :-) | UP | neutron-dhcp-agent | | 61df9ce0-7ad7-4955-8d59-88fe45b24ff7 | DHCP agent | compute15 | zone3 | :-) | UP | neutron-dhcp-agent | | 6897a6d1-8e56-4958-b300-9642fd895ad0 | Metadata agent | compute10 | None | :-) | UP | neutron-metadata-agent | | 6ab71669-379a-44ab-a804-0805da23630e | Metadata agent | compute7 | None | :-) | UP | neutron-metadata-agent | | 6ba578ee-ed55-4930-af7e-3c3dade6db79 | Metadata agent | compute12 | None | :-) | UP | neutron-metadata-agent | | 7feaf689-05b7-483a-a812-3ffb129717ee | DHCP agent | compute4 | zone1 | :-) | UP | neutron-dhcp-agent | | 80e7c6d1-9748-4803-8e54-1b7df807cb69 | Metadata agent | compute11 | None | :-) | UP | neutron-metadata-agent | | a8a70e20-910b-4f5c-8414-01d4dcac42cf | Metadata agent | compute9 | None | :-) | UP | neutron-metadata-agent | | b6a751e7-c86a-4347-8f5b-079fd019fcaf | DHCP agent | compute7 | zone2 | :-) | UP | neutron-dhcp-agent | | bd95e896-496f-4b37-84bc-9e916a1cd313 | Metadata agent | compute4 | None | :-) | UP | neutron-metadata-agent | | d5ec415a-f8c9-43b0-b128-0f07da1b2625 | Metadata agent | compute1 | None | :-) | UP | neutron-metadata-agent | | dba5e158-6222-4cba-b9d3-19165918261b | DHCP agent | compute11 | zone3 | :-) | UP | neutron-dhcp-agent | | f4640a2e-fc7c-415c-8f3d-aa0c68f6110c | Metadata agent | compute2 | None | :-) | UP | neutron-metadata-agent | | fdde9c2b-2352-4876-8598-ef877eda724d | DHCP agent | compute13 | zone3 | :-) | UP | neutron-dhcp-agent | | ff340680-8eff-4cb3-8685-36a9b0459141 | DHCP agent | compute9 | zone2 | :-) | UP | neutron-dhcp-agent | The instance, when booting is giving the message: [WARNING]: No active metadata service found If i create an instance using config-drive (ignoring metadata server), it starts up fine, and has a routing table with 169.254.169.254 via one of the qdhcp namespaces. Inside of this instance i can ping both the IP of the namespace and the 169.254.169.254 address, but I cannot curl it. Inside of the namespace if I curl http://169.254.169.254/ I get the following: --- # curl http://169.254.169.254 <html> <head> <title>404 Not Found</title> </head> <body> <h1>404 Not Found</h1> The resource could not be found.<br /><br /> </body> --- Doing some googling, I found a suggestion of verify that the following rule existed in the iptables of the namespace: -A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8775 I have no such rule, but this is likely because there is no NGW in their environment. It is also worth noting, that in the console output of the instance, it never attempts to call http://169.254.169.254 Bundle can be found at: https://git.launchpad.net/cpe-deployments/tree/config/bundle.yaml?h=2019-05-27-Telefonica-OCS-OP-152907 Console log of last boot found at: https://pastebin.canonical.com/p/Jyk22sfGT8/ In the scenario where their is no Nuetron Gateway, we're only using provider networking. Specifically VLAN provider networking. The one network and subnet created look as follows:  openstack network show mgmt-1 +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | UP | | availability_zone_hints | zone1, zone2, zone3 | | availability_zones | zone1, zone2 | | created_at | 2019-06-06T17:31:19Z | | description | | | dns_domain | | | id | 724aef7a-54a2-4daf-9aa3-98f008215b55 | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | None | | is_vlan_transparent | None | | mtu | 9000 | | name | mgmt-1 | | port_security_enabled | True | | project_id | 143294d60ce54454b451214026857bc9 | | provider:network_type | vlan | | provider:physical_network | physnet1 | | provider:segmentation_id | 1030 | | qos_policy_id | None | | revision_number | 3 | | router:external | Internal | | segments | None | | shared | False | | status | ACTIVE | | subnets | 5d0cf549-4bca-410d-8514-90b805276324 | | tags | | | updated_at | 2019-06-06T17:31:20Z | +---------------------------+--------------------------------------+ $ openstack subnet show mgmt-1 +-------------------+--------------------------------------+ | Field | Value | +-------------------+--------------------------------------+ | allocation_pools | 10.243.160.10-10.243.160.100 | | cidr | 10.243.160.0/24 | | created_at | 2019-06-06T17:31:20Z | | description | | | dns_nameservers | | | enable_dhcp | True | | gateway_ip | 10.243.160.254 | | host_routes | | | id | 5d0cf549-4bca-410d-8514-90b805276324 | | ip_version | 4 | | ipv6_address_mode | None | | ipv6_ra_mode | None | | name | mgmt-1 | | network_id | 724aef7a-54a2-4daf-9aa3-98f008215b55 | | project_id | 143294d60ce54454b451214026857bc9 | | revision_number | 0 | | segment_id | None | | service_types | | | subnetpool_id | None | | tags | | | updated_at | 2019-06-06T17:31:20Z | +-------------------+--------------------------------------+ neutron-openvswitch has enable-local-dhcp-and-metadata set to True. This is verified with: $ openstack network agent list | egrep -i 'dhcp|meta' | 08586252-fe88-48e7-afd8-f95a3500dee1 | Metadata agent | compute15 | None | :-) | UP | neutron-metadata-agent | | 09f469c9-0f02-4e20-9725-3e98120ae704 | DHCP agent | compute1 | zone1 | :-) | UP | neutron-dhcp-agent | | 0a9bc81d-ff79-43a8-b279-ef22290d36d7 | Metadata agent | compute3 | None | :-) | UP | neutron-metadata-agent | | 0c0ecf6a-aa56-4095-9fff-4f22ab99d00c | Metadata agent | compute5 | None | :-) | UP | neutron-metadata-agent | | 170fb80a-f782-4aa7-8ac6-750d00ec0125 | Metadata agent | compute14 | None | :-) | UP | neutron-metadata-agent | | 1e0f1511-11a4-4e43-bdca-722bde6b6c11 | DHCP agent | compute3 | zone1 | :-) | UP | neutron-dhcp-agent | | 22a110e4-f928-4388-87ca-e8ebc9f15554 | DHCP agent | compute2 | zone1 | :-) | UP | neutron-dhcp-agent | | 236387f3-3f24-4075-9cb7-e12280d36438 | DHCP agent | compute5 | zone1 | :-) | UP | neutron-dhcp-agent | | 2d643308-cd4d-4d07-a54a-7ab99167c1b0 | Metadata agent | compute6 | None | :-) | UP | neutron-metadata-agent | | 2f35c4bd-cf44-4187-920a-db5beeab05cf | DHCP agent | compute14 | zone3 | :-) | UP | neutron-dhcp-agent | | 3db2dcbf-bdea-444a-afc6-75b49f0132a8 | DHCP agent | compute12 | zone3 | :-) | UP | neutron-dhcp-agent | | 44a09595-eded-46eb-9d39-036870731433 | Metadata agent | compute8 | None | :-) | UP | neutron-metadata-agent | | 44e02602-f0e3-45f0-9887-9739aaaef3de | DHCP agent | compute8 | zone2 | :-) | UP | neutron-dhcp-agent | | 468e7408-8af1-433c-a77b-e474bc16f9f9 | Metadata agent | compute13 | None | :-) | UP | neutron-metadata-agent | | 51ba61b8-72b5-49cb-a33e-17afe72a4a0b | DHCP agent | compute6 | zone2 | :-) | UP | neutron-dhcp-agent | | 56f99f61-3a6f-48d2-931d-6128d0877486 | DHCP agent | compute10 | zone2 | :-) | UP | neutron-dhcp-agent | | 61df9ce0-7ad7-4955-8d59-88fe45b24ff7 | DHCP agent | compute15 | zone3 | :-) | UP | neutron-dhcp-agent | | 6897a6d1-8e56-4958-b300-9642fd895ad0 | Metadata agent | compute10 | None | :-) | UP | neutron-metadata-agent | | 6ab71669-379a-44ab-a804-0805da23630e | Metadata agent | compute7 | None | :-) | UP | neutron-metadata-agent | | 6ba578ee-ed55-4930-af7e-3c3dade6db79 | Metadata agent | compute12 | None | :-) | UP | neutron-metadata-agent | | 7feaf689-05b7-483a-a812-3ffb129717ee | DHCP agent | compute4 | zone1 | :-) | UP | neutron-dhcp-agent | | 80e7c6d1-9748-4803-8e54-1b7df807cb69 | Metadata agent | compute11 | None | :-) | UP | neutron-metadata-agent | | a8a70e20-910b-4f5c-8414-01d4dcac42cf | Metadata agent | compute9 | None | :-) | UP | neutron-metadata-agent | | b6a751e7-c86a-4347-8f5b-079fd019fcaf | DHCP agent | compute7 | zone2 | :-) | UP | neutron-dhcp-agent | | bd95e896-496f-4b37-84bc-9e916a1cd313 | Metadata agent | compute4 | None | :-) | UP | neutron-metadata-agent | | d5ec415a-f8c9-43b0-b128-0f07da1b2625 | Metadata agent | compute1 | None | :-) | UP | neutron-metadata-agent | | dba5e158-6222-4cba-b9d3-19165918261b | DHCP agent | compute11 | zone3 | :-) | UP | neutron-dhcp-agent | | f4640a2e-fc7c-415c-8f3d-aa0c68f6110c | Metadata agent | compute2 | None | :-) | UP | neutron-metadata-agent | | fdde9c2b-2352-4876-8598-ef877eda724d | DHCP agent | compute13 | zone3 | :-) | UP | neutron-dhcp-agent | | ff340680-8eff-4cb3-8685-36a9b0459141 | DHCP agent | compute9 | zone2 | :-) | UP | neutron-dhcp-agent | The instance, when booting is giving the message: [WARNING]: No active metadata service found If i create an instance using config-drive (ignoring metadata server), it starts up fine, and has a routing table with 169.254.169.254 via one of the qdhcp namespaces. Inside of this instance i can ping both the IP of the namespace and the 169.254.169.254 address, but I cannot curl it. Inside of the namespace if I curl http://169.254.169.254/ I get the following: --- # curl http://169.254.169.254 <html>  <head>   <title>404 Not Found</title>  </head>  <body>   <h1>404 Not Found</h1>   The resource could not be found.<br /><br />  </body> --- Doing some googling, I found a suggestion of verify that the following rule existed in the iptables of the namespace: -A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8775 I have no such rule, but this is likely because there is no NGW in their environment. It is also worth noting, that in the console output of the instance, it never attempts to call http://169.254.169.254 Bundle can be found at: Console log of last boot found at: https://pastebin.canonical.com/p/Jyk22sfGT8/
2019-06-07 07:25:54 Frode Nordahl information type Private Public
2019-06-07 07:51:34 Chris MacNaughton attachment added bundle.yaml https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1831935/+attachment/5269266/+files/bundle.yaml
2019-06-07 13:14:19 Jeff Hillman bug added subscriber Canonical Field Critical
2019-06-07 16:36:23 David Ames charm-neutron-openvswitch: status New In Progress
2019-06-07 16:36:25 David Ames charm-neutron-openvswitch: importance Undecided Critical
2019-06-07 16:36:27 David Ames charm-neutron-openvswitch: assignee David Ames (thedac)
2019-06-07 16:36:30 David Ames charm-neutron-openvswitch: milestone 19.07
2019-06-10 07:33:37 OpenStack Infra charm-neutron-openvswitch: status In Progress Fix Committed
2019-07-15 10:13:38 James Page charm-neutron-openvswitch: status Fix Committed Fix Released
2020-01-25 00:24:38 David Ames charm-neutron-openvswitch: status Fix Released Triaged
2020-01-25 00:24:46 David Ames charm-neutron-openvswitch: milestone 19.07 20.02
2020-01-25 00:24:57 David Ames bug task added charm-helpers
2020-01-25 00:25:10 David Ames bug task added charm-neutron-gateway
2020-01-25 00:25:19 David Ames charm-helpers: status New Triaged
2020-01-25 00:25:22 David Ames charm-neutron-gateway: status New Triaged
2020-01-25 00:25:25 David Ames charm-helpers: importance Undecided High
2020-01-25 00:25:27 David Ames charm-neutron-gateway: importance Undecided High
2020-01-25 00:25:29 David Ames charm-neutron-openvswitch: importance Critical High
2020-01-25 00:25:32 David Ames charm-helpers: assignee David Ames (thedac)
2020-01-25 00:25:34 David Ames charm-neutron-gateway: assignee David Ames (thedac)
2020-01-25 00:25:40 David Ames charm-neutron-gateway: milestone 20.02
2020-01-25 00:36:50 OpenStack Infra charm-neutron-openvswitch: status Triaged In Progress
2020-01-27 23:05:04 OpenStack Infra charm-neutron-gateway: status Triaged In Progress
2020-02-17 17:28:44 Liam Young charm-neutron-openvswitch: milestone 20.02 20.05
2020-02-17 17:28:54 Liam Young charm-neutron-gateway: milestone 20.02 20.05
2020-02-28 00:12:06 Nikolay Vinogradov bug added subscriber Nikolay Vinogradov
2020-03-03 10:49:14 David Ames charm-helpers: status Triaged Fix Released
2020-03-03 10:49:17 David Ames charm-neutron-gateway: status In Progress Fix Released
2020-03-03 10:49:19 David Ames charm-neutron-openvswitch: status In Progress Fix Released