Bug #1825976 “THT: wrong indentation (?) in placement-api-contai...” : Bugs : tripleo

Revision history for this message

Luca Miccini (lmiccini2) wrote on 2019-04-23:

#1

Download full text (3.6 KiB)

# Generated by iptables-save v1.4.21 on Tue Apr 23 03:42:47 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -m comment --comment "000 accept related established rules ipv4" -j ACCEPT
-A INPUT -p icmp -m state --state NEW -m comment --comment "001 accept all icmp ipv4" -j ACCEPT
-A INPUT -i lo -m state --state NEW -m comment --comment "002 accept all to lo interface ipv4" -j ACCEPT
-A INPUT -s 192.168.24.0/24 -p tcp -m multiport --dports 22 -m state --state NEW -m comment --comment "003 accept ssh from controlplane ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 873,3123,3306,4444,4567,4568,9200 -m state --state NEW -m comment --comment "104 mysql galera-bundle ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 3124,6379,26379 -m state --state NEW -m comment --comment "108 redis-bundle ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 3122,4369,5672,25672 -m state --state NEW -m comment --comment "109 rabbitmq-bundle ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 5000,13000,35357 -m state --state NEW -m comment --comment "111 keystone ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 9292,13292 -m state --state NEW -m comment --comment "112 glance_api ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 8774,13774 -m state --state NEW -m comment --comment "113 nova_api ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 9696,13696 -m state --state NEW -m comment --comment "114 neutron api ipv4" -j ACCEPT
-A INPUT -p udp -m multiport --dports 4789 -m state --state NEW -m comment --comment "118 neutron vxlan networks ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 8776,13776 -m state --state NEW -m comment --comment "119 cinder ipv4" -j ACCEPT
-A INPUT -p udp -m multiport --dports 6081 -m state --state NEW -m comment --comment "119 neutron geneve networks ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 3260 -m state --state NEW -m comment --comment "120 iscsi initiator ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 6641,6642 -m state --state NEW -m comment --comment "121 OVN DB server ports ipv4" -j ACCEPT
-A INPUT -s 172.17.0.0/24 -p tcp -m multiport --dports 11211 -m state --state NEW -m comment --comment "121 memcached 172.17.0.0/24 ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 8080,13808 -m state --state NEW -m comment --comment "122 swift proxy ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 873,6000,6001,6002 -m state --state NEW -m comment --comment "123 swift storage ipv4" -j ACCEPT
-A INPUT -s 192.168.24.0/24 -p udp -m multiport --dports 161 -m state --state NEW -m comment --comment "124 snmp 192.168.24.0/24 ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 8004,13004 -m state --state NEW -m comment --comment "125 heat_api ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 8000,13800 -m state --state NEW -m comment --comment "125 heat_cfn ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 80,443 -m state --state NEW -m comment --comment "126 horizon ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 80,443 -m state --state NEW -m comment --comment "127 horizon ipv4" -j ACCEPT
-A INPUT -p tcp ...

# Generated by iptables-save v1.4.21 on Tue Apr 23 03:42:47 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -m comment --comment "000 accept related established rules ipv4" -j ACCEPT
-A INPUT -p icmp -m state --state NEW -m comment --comment "001 accept all icmp ipv4" -j ACCEPT
-A INPUT -i lo -m state --state NEW -m comment --comment "002 accept all to lo interface ipv4" -j ACCEPT
-A INPUT -s 192.168.24.0/24 -p tcp -m multiport --dports 22 -m state --state NEW -m comment --comment "003 accept ssh from controlplane ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 873,3123,3306,4444,4567,4568,9200 -m state --state NEW -m comment --comment "104 mysql galera-bundle ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 3124,6379,26379 -m state --state NEW -m comment --comment "108 redis-bundle ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 3122,4369,5672,25672 -m state --state NEW -m comment --comment "109 rabbitmq-bundle ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 5000,13000,35357 -m state --state NEW -m comment --comment "111 keystone ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 9292,13292 -m state --state NEW -m comment --comment "112 glance_api ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 8774,13774 -m state --state NEW -m comment --comment "113 nova_api ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 9696,13696 -m state --state NEW -m comment --comment "114 neutron api ipv4" -j ACCEPT
-A INPUT -p udp -m multiport --dports 4789 -m state --state NEW -m comment --comment "118 neutron vxlan networks ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 8776,13776 -m state --state NEW -m comment --comment "119 cinder ipv4" -j ACCEPT
-A INPUT -p udp -m multiport --dports 6081 -m state --state NEW -m comment --comment "119 neutron geneve networks ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 3260 -m state --state NEW -m comment --comment "120 iscsi initiator ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 6641,6642 -m state --state NEW -m comment --comment "121 OVN DB server ports ipv4" -j ACCEPT
-A INPUT -s 172.17.0.0/24 -p tcp -m multiport --dports 11211 -m state --state NEW -m comment --comment "121 memcached 172.17.0.0/24 ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 8080,13808 -m state --state NEW -m comment --comment "122 swift proxy ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 873,6000,6001,6002 -m state --state NEW -m comment --comment "123 swift storage ipv4" -j ACCEPT
-A INPUT -s 192.168.24.0/24 -p udp -m multiport --dports 161 -m state --state NEW -m comment --comment "124 snmp 192.168.24.0/24 ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 8004,13004 -m state --state NEW -m comment --comment "125 heat_api ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 8000,13800 -m state --state NEW -m comment --comment "125 heat_cfn ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 80,443 -m state --state NEW -m comment --comment "126 horizon ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 80,443 -m state --state NEW -m comment --comment "127 horizon ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 2224,3121,21064 -m state --state NEW -m comment --comment "130 pacemaker tcp ipv4" -j ACCEPT
-A INPUT -p udp -m multiport --dports 5405 -m state --state NEW -m comment --comment "131 pacemaker udp ipv4" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 6080,13080 -m state --state NEW -m comment --comment "137 nova_vnc_proxy ipv4" -j ACCEPT
-A INPUT -m state --state NEW -m limit --limit 20/min --limit-burst 15 -m comment --comment "998 log all ipv4" -j LOG
COMMIT
# Completed on Tue Apr 23 03:42:47 2019

description:

updated

Revision history for this message

Luca Miccini (lmiccini2) wrote on 2019-04-23:

#2

I failed miserably at formatting :(

here a couple of links:
https://github.com/openstack/tripleo-heat-templates/blob/7cd0e8ff7b724e7ac71dca0aafa018a56081cbbb/deployment/placement/placement-api-container-puppet.yaml#L104

https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/nova/nova-api-container-puppet.yaml#L182

Revision history for this message

Luca Miccini (lmiccini2) wrote on 2019-04-23:

#3

adding indentation results in such hieradata:

   "tripleo::placement::firewall_rules": {
        "138 placement": {
            "dport": [
                8778,
                13778
            ]
        }
    },

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-04-23: Fix proposed to tripleo-heat-templates (master)

#4

Fix proposed to branch: master
Review: https://review.opendev.org/655120

Changed in tripleo:
assignee:	nobody → Luca Miccini (lmiccini2)
status:	New → In Progress

yatin (yatinkarel) on 2019-04-24

Changed in tripleo:
importance:	Undecided → Critical

Revision history for this message

yatin (yatinkarel) wrote on 2019-04-24:

#5

So i was tracking down promotion job failures and reached to this bug, This bug is affecting all ovb jobs in both upstream master check[1] and master promotion jobs[2], issue started happening with https://review.opendev.org/#/c/630644/43 merged as placement iptables rules are not there[3], still a doubt as PS37 has job passing but those logs are deleted, someone from nova can confirm this. Marking the bug as critical and promotion blocker.

[1] https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035&job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001&branch=master&branch=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053&branch=tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039
[2] https://trunk-primary.rdoproject.org/api-centos-master-uc/api/civotes_detail.html?commit_hash=0ce75cbd14493d7ff1636937ce89ae11768672f1&distro_hash=6089bcdbc70a171dc02a608863bf57efb9dac5aa
[3] https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/c32fa29/logs/overcloud-controller-0/etc/sysconfig/iptables.gz

tags:

added: alert ci promotion-blocker

Revision history for this message

Martin Schuppert (mschuppert) wrote on 2019-04-24:

#6

Download full text (10.5 KiB)

I have checked failing tempest from [1]

1) just for reference, the instance gets created ok:
* Instance 521734e6-27d9-4867-bc60-ceacc55aa22b

On compute log [2] we see the instance created ok on the compute:
2019-04-23 23:25:27.525 7 INFO nova.compute.manager [-] [instance: 521734e6-27d9-4867-bc60-ceacc55aa22b] VM Started (Lifecycle Event)

- instance build took ~13s
2019-04-23 23:25:31.070 7 INFO nova.compute.manager [req-20654e60-709d-48f0-be45-4b977e50965b 45acace926eb4efaa977376e17a1b50a 59fb6d91d57e443a8b4887d94e9653f4 - default default] [instance: 521734e6-27d9-4867-bc60-ceacc55aa22b] Took 13.25 seconds to build instance.

- also we see the floating ip in the network_info:
2019-04-23 23:25:38.504 7 DEBUG nova.network.base_api [req-48530c65-4b1e-43ec-9f93-713010f4ca42 2dd4f26ca21548979a1a33f220ac8262 8377d6b2752c49698d063799ed61ee8e - default default] [instance: 521734e6-27d9-4867-bc60-ceacc55aa22b] Updating instance_info_cache with network_info: [{"profile": {}, "ovs_interfaceid": "bac228d8-c5a2-451f-85fe-d29815323d54", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [{"meta": {}, "version": 4, "type": "floating", "address": "10.0.0.117"}], "address": "10.100.0.12"}], "version": 4, "meta": {"dhcp_server": "10.100.0.2"}, "dns": [], "routes": [], "cidr": "10.100.0.0/28", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "10.100.0.1"}}], "meta": {"injected": false, "tunneled": true, "tenant_id": "59fb6d91d57e443a8b4887d94e9653f4", "physical_network": null, "mtu": 1292}, "id": "4b2a43c3-5820-4532-906f-5067bbf97c82", "label": "tempest-network-smoke--1490756850"}, "devname": "tapbac228d8-c5", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:6d:1f:c9", "active": true, "type": "ovs", "id": "bac228d8-c5a2-451f-85fe-d29815323d54", "qbg_params": null}] update_instance_cache_with_nw_info /usr/lib/python2.7/site-packages/nova/network/base_api.py:48

- the allocation in placement is correct as well
2019-04-23 23:26:08.962 7 DEBUG nova.compute.resource_tracker [req-8928dcb0-8883-4551-bfb6-20bd2295f8cf - - - - -] Instance 521734e6-27d9-4867-bc60-ceacc55aa22b actively managed on this compute host and has allocations in placement: {u'resources': {u'VCPU': 1, u'MEMORY_MB': 64, u'DISK_GB': 1}}. _remove_deleted_instances_allocations /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:1308

- later delete of instance
2019-04-23 23:30:56.677 7 INFO nova.compute.manager [req-813e8b5a-4e4c-48cf-a5d7-234de6fed75d 45acace926eb4efaa977376e17a1b50a 59fb6d91d57e443a8b4887d94e9653f4 - default default] [instance: 521734e6-27d9-4867-bc60-ceacc55aa22b] Terminating instance
...
2019-04-23 23:30:57.593 7 INFO nova.virt.libvirt.driver [-] [instance: 521734e6-27d9-4867-bc60-ceacc55aa22b] Instance destroyed successfully.

BUT the issue is on the metadata side:

2) we see errors in the ovn metadata agent log [3] for a request for the instance (10.100.0.12):

2019-04-23 23:27:42.513 30579 DEBUG networking_ovn.agent.metadata.server [-] Request: GET /2009-04-04/meta-data/instance-id HTTP/1.0
...

I have checked failing tempest from [1]

1) just for reference, the instance gets created ok:
* Instance 521734e6-27d9-4867-bc60-ceacc55aa22b

On compute log [2] we see the instance created ok on the compute:
2019-04-23 23:25:27.525 7 INFO nova.compute.manager [-] [instance: 521734e6-27d9-4867-bc60-ceacc55aa22b] VM Started (Lifecycle Event)

- instance build took ~13s 
2019-04-23 23:25:31.070 7 INFO nova.compute.manager [req-20654e60-709d-48f0-be45-4b977e50965b 45acace926eb4efaa977376e17a1b50a 59fb6d91d57e443a8b4887d94e9653f4 - default default] [instance: 521734e6-27d9-4867-bc60-ceacc55aa22b] Took 13.25 seconds to build instance.

- also we see the floating ip in the network_info:
2019-04-23 23:25:38.504 7 DEBUG nova.network.base_api [req-48530c65-4b1e-43ec-9f93-713010f4ca42 2dd4f26ca21548979a1a33f220ac8262 8377d6b2752c49698d063799ed61ee8e - default default] [instance: 521734e6-27d9-4867-bc60-ceacc55aa22b] Updating instance_info_cache with network_info: [{"profile": {}, "ovs_interfaceid": "bac228d8-c5a2-451f-85fe-d29815323d54", "preserve_on_delete": false, "network": {"bridge": "br-int", "subnets": [{"ips": [{"meta": {}, "version": 4, "type": "fixed", "floating_ips": [{"meta": {}, "version": 4, "type": "floating", "address": "10.0.0.117"}], "address": "10.100.0.12"}], "version": 4, "meta": {"dhcp_server": "10.100.0.2"}, "dns": [], "routes": [], "cidr": "10.100.0.0/28", "gateway": {"meta": {}, "version": 4, "type": "gateway", "address": "10.100.0.1"}}], "meta": {"injected": false, "tunneled": true, "tenant_id": "59fb6d91d57e443a8b4887d94e9653f4", "physical_network": null, "mtu": 1292}, "id": "4b2a43c3-5820-4532-906f-5067bbf97c82", "label": "tempest-network-smoke--1490756850"}, "devname": "tapbac228d8-c5", "vnic_type": "normal", "qbh_params": null, "meta": {}, "details": {"port_filter": true}, "address": "fa:16:3e:6d:1f:c9", "active": true, "type": "ovs", "id": "bac228d8-c5a2-451f-85fe-d29815323d54", "qbg_params": null}] update_instance_cache_with_nw_info /usr/lib/python2.7/site-packages/nova/network/base_api.py:48

- the allocation in placement is correct as well
2019-04-23 23:26:08.962 7 DEBUG nova.compute.resource_tracker [req-8928dcb0-8883-4551-bfb6-20bd2295f8cf - - - - -] Instance 521734e6-27d9-4867-bc60-ceacc55aa22b actively managed on this compute host and has allocations in placement: {u'resources': {u'VCPU': 1, u'MEMORY_MB': 64, u'DISK_GB': 1}}. _remove_deleted_instances_allocations /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:1308

- later delete of instance
2019-04-23 23:30:56.677 7 INFO nova.compute.manager [req-813e8b5a-4e4c-48cf-a5d7-234de6fed75d 45acace926eb4efaa977376e17a1b50a 59fb6d91d57e443a8b4887d94e9653f4 - default default] [instance: 521734e6-27d9-4867-bc60-ceacc55aa22b] Terminating instance
...
2019-04-23 23:30:57.593 7 INFO nova.virt.libvirt.driver [-] [instance: 521734e6-27d9-4867-bc60-ceacc55aa22b] Instance destroyed successfully.

BUT the issue is on the metadata side:

2) we see errors in the ovn metadata agent log [3] for a request for the instance (10.100.0.12):

2019-04-23 23:27:42.513 30579 DEBUG networking_ovn.agent.metadata.server [-] Request: GET /2009-04-04/meta-data/instance-id HTTP/1.0
Accept: */*
Connection: close
Content-Type: text/plain
Host: 169.254.169.254
User-Agent: curl/7.24.0 (x86_64-pc-linux-gnu) libcurl/7.24.0 OpenSSL/1.0.0j zlib/1.2.6
X-Forwarded-For: 10.100.0.12
X-Ovn-Network-Id: 6a2c1465-129d-4eba-b24d-856327a0ba47 __call__ /usr/lib/python2.7/site-packages/networking_ovn/agent/metadata/server.py:64
2019-04-23 23:27:42.514 30579 DEBUG networking_ovn.agent.metadata.server [-] Request to Nova at overcloud.internalapi.localdomain:8775 _proxy_request /usr/lib/python2.7/site-packages/networking_ovn/agent/metadata/server.py:101
2019-04-23 23:27:42.514 30579 DEBUG networking_ovn.agent.metadata.server [-] {'X-Forwarded-For': '10.100.0.12', 'X-Instance-ID-Signature': '3ab84648c2acfc5963a8ec7ebc9af0edddc76348de1aab25dd0db579c6f85667', 'X-Tenant-ID': '59fb6d91d57e443a8b4887d94e9653f4', 'X-Instance-ID': '521734e6-27d9-4867-bc60-ceacc55aa22b'} _proxy_request /usr/lib/python2.7/site-packages/networking_ovn/agent/metadata/server.py:102
2019-04-23 23:27:47.107 30404 DEBUG ovsdbapp.backend.ovs_idl.event [-] SbGlobalUpdateEvent : Matched SB_Global, ('update',), None None matches /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/event.py:40
2019-04-23 23:27:47.108 30404 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): UpdateChassisExtIdsCommand(if_exists=True, external_ids={'neutron:ovn-metadata-sb-cfg': '49'}, name=893bfbb4-63c7-40b0-a236-6ebc2313f593) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2019-04-23 23:27:47.449 30404 DEBUG ovsdbapp.backend.ovs_idl.event [-] SbGlobalUpdateEvent : Matched SB_Global, ('update',), None None matches /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/event.py:40
2019-04-23 23:27:47.450 30404 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): UpdateChassisExtIdsCommand(if_exists=True, external_ids={'neutron:ovn-metadata-sb-cfg': '50'}, name=893bfbb4-63c7-40b0-a236-6ebc2313f593) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2019-04-23 23:27:47.592 30579 ERROR networking_ovn.agent.metadata.server [-] Unexpected error.: error: [Errno 110] ETIMEDOUT
2019-04-23 23:27:47.592 30579 ERROR networking_ovn.agent.metadata.server Traceback (most recent call last):
2019-04-23 23:27:47.592 30579 ERROR networking_ovn.agent.metadata.server   File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata/server.py", line 68, in __call__
2019-04-23 23:27:47.592 30579 ERROR networking_ovn.agent.metadata.server     return self._proxy_request(instance_id, project_id, req)
2019-04-23 23:27:47.592 30579 ERROR networking_ovn.agent.metadata.server   File "/usr/lib/python2.7/site-packages/networking_ovn/agent/metadata/server.py", line 119, in _proxy_request
2019-04-23 23:27:47.592 30579 ERROR networking_ovn.agent.metadata.server     body=req.body)
2019-04-23 23:27:47.592 30579 ERROR networking_ovn.agent.metadata.server   File "/usr/lib/python2.7/site-packages/httplib2/__init__.py", line 1621, in request
2019-04-23 23:27:47.592 30579 ERROR networking_ovn.agent.metadata.server     (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
2019-04-23 23:27:47.592 30579 ERROR networking_ovn.agent.metadata.server   File "/usr/lib/python2.7/site-packages/httplib2/__init__.py", line 1363, in _request
2019-04-23 23:27:47.592 30579 ERROR networking_ovn.agent.metadata.server     (response, content) = self._conn_request(conn, request_uri, method, body, headers)
2019-04-23 23:27:47.592 30579 ERROR networking_ovn.agent.metadata.server   File "/usr/lib/python2.7/site-packages/httplib2/__init__.py", line 1284, in _conn_request
2019-04-23 23:27:47.592 30579 ERROR networking_ovn.agent.metadata.server     conn.connect()
2019-04-23 23:27:47.592 30579 ERROR networking_ovn.agent.metadata.server   File "/usr/lib/python2.7/site-packages/httplib2/__init__.py", line 934, in connect
2019-04-23 23:27:47.592 30579 ERROR networking_ovn.agent.metadata.server     raise socket.error, msg
2019-04-23 23:27:47.592 30579 ERROR networking_ovn.agent.metadata.server error: [Errno 110] ETIMEDOUT
2019-04-23 23:27:47.592 30579 ERROR networking_ovn.agent.metadata.server 
2019-04-23 23:27:47.596 30579 INFO eventlet.wsgi.server [-] Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 578, in handle_one_response
    write(b''.join(towrite))
  File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 519, in write
    wfile.flush()
  File "/usr/lib64/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 401, in sendall
    tail = self.send(data, flags)
  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 395, in send
    return self._send_loop(self.fd.send, data, flags)
  File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 382, in _send_loop
    return send_method(data, *args)
error: [Errno 32] Broken pipe

We don't see any incoming request to the nova metadata api service on any of the controllers.

From haproxy.log of the different controllers we only see always the local metadata service backend as up, which still should be enough to server the request:
- controller-0
Apr 23 23:18:07 overcloud-controller-0 haproxy[12]: Server nova_metadata/overcloud-controller-0.internalapi.localdomain is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 2ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.

- controller-1:
Apr 23 23:18:01 overcloud-controller-1 haproxy[13]: Server nova_metadata/overcloud-controller-1.internalapi.localdomain is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 1ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.

- controller-2:
Apr 23 23:18:01 overcloud-controller-2 haproxy[13]: Server nova_metadata/overcloud-controller-2.internalapi.localdomain is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 1ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.

The issue is that we miss the nova_metadata api iptables rules (port 8775/13775) [5] , e.g. [4]. [5] references 'tripleo::nova_placement::firewall_rules' which is wrong.

I'll create a new LP for this and submit a patch.

Just a side note: designate use the same ID 139 [6]

[1] http://logs.rdoproject.org/20/655120/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053/37cc755/logs/tempest.html.gz
[2] http://logs.rdoproject.org/20/655120/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053/37cc755/logs/overcloud-novacompute-0/var/log/containers/nova/nova-compute.log.txt.gz
[3] http://logs.rdoproject.org/20/655120/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053/37cc755/logs/overcloud-novacompute-0/var/log/containers/neutron/ovn-metadata-agent.log.txt.gz
[4] http://logs.rdoproject.org/20/655120/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053/37cc755/logs/overcloud-controller-0/etc/sysconfig/iptables.gz
[5] https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/nova/nova-metadata-container-puppet.yaml#L135
[6] https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/experimental/designate/designate-api-container-puppet.yaml#L97-L101

Revision history for this message

Martin Schuppert (mschuppert) wrote on 2019-04-24:

#7

for reference, I filed https://bugs.launchpad.net/tripleo/+bug/1826178

wes hayutin (weshayutin) on 2019-04-24

Changed in tripleo:
milestone:	none → train-1

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-04-24: Fix merged to tripleo-heat-templates (master)

#8

Reviewed: https://review.opendev.org/655120
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=8e60f83615cb166e04f9de1a9a4331245424c2db
Submitter: Zuul
Branch: master

commit 8e60f83615cb166e04f9de1a9a4331245424c2db
Author: Luca Miccini <email address hidden>
Date: Tue Apr 23 14:18:35 2019 +0200

Properly indent placement::firewall_rules

placement::firewall_rules is missing proper indentation.

This leads to hieradata like:

    {
        "138 placement": {
            "dport": [
                8778,
                13778
            ]
        },

that is not picked up by puppet (hence no firewall rules are created).

This commit adds one level of indentation, resulting in proper hieradata:

    "tripleo::placement::firewall_rules": {
           "138 placement": {
               "dport": [
                   8778,
                   13778
               ]
           }
       },

Change-Id: I168863d6187ddf485546dbbc7e65bd45ef56ea38
Closes-bug: #1825976

Changed in tripleo:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-06-07: Fix included in openstack/tripleo-heat-templates 11.0.0

#9

This issue was fixed in the openstack/tripleo-heat-templates 11.0.0 release.

tripleo

THT: wrong indentation (?) in placement-api-container-puppet.yaml leads to wrong hieradata (and failed deployment)

Bug Description

Other bug subscribers

Remote bug watches