tripleo

Overcloud deployments fail with FileNotFound errors during os-net-config

Bug #1896829 reported by Lance Bragstad
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Rabi Mishra
tripleo victoria-rc1 "tripleo-victoria-rc1"

Bug Description

The overcloud deployment fails with the following error using the latest master packages from RDO:

    2020-09-23 18:58:59.447757 | fa163e76-26da-e8d8-8e88-000000000ad2 | TASK | Run tripleo_os_net_config_module with network_config
    n3.6/site-packages/os_net_config/cli.py\", line 343, in main\n provider.add_object(obj)\n File \"/usr/lib/python3.6/site-packages/os_net_config/__init__.py\", line 64, in add_object\n self.add_bridge(obj)\n File \"/usr/lib/python3.6/site-packages/os_net_config/impl_ifcfg.py\", line 807, in add_bridge\n data = self._add_common(bridge)\n File \"/usr/lib/python3.6/site-packages/os_net_config/impl_ifcfg.py\", line 475, in _add_common\n mac = utils.interface_mac(base_opt.primary_interface_name)\n File \"/usr/lib/python3.6/site-packages/os_net_config/utils.py\", line 120, in interface_mac\n with open(_SYS_CLASS_NET + '/%s/address' % name, 'r') as f:\nFileNotFoundError: [Errno 2] No such file or directory: '/sys/class/net/interface_name/address'\n", "stderr_lines": ["[2020/09/23 06:58:59 PM] [INFO] Using config file at: /etc/os-net-config/config.json", "[2020/09/23 06:58:59 PM] [INFO] Ifcfg net config provider created.", "[2020/09/23 06:58:59 PM] [INFO] Not using any mapping file.", "[2020/09/23 06:58:59 PM] [DEBUG] network_config JSON: [{'addresses': [{'ip_netmask': '192.168.24.20/24'}], 'dns_servers': ['192.168.1.57'], 'domain': ['ooo.test', 'nightly-latest-tripleo-ipa-ds.test'], 'members': [{'name': 'interface_name', 'primary': True, 'type': 'interface'}], 'name': 'bridge_name', 'routes': [{'default': True, 'next_hop': '192.168.24.14'}], 'type': 'ovs_bridge', 'use_dhcp': False}]", "[2020/09/23 06:59:00 PM] [INFO] Finding active nics", "[2020/09/23 06:59:00 PM] [INFO] eth1 is an embedded active nic", "[2020/09/23 06:59:00 PM] [INFO] lo is not an active nic", "[2020/09/23 06:59:00 PM] [INFO] eth2 is an embedded active nic", "[2020/09/23 06:59:00 PM] [INFO] eth0 is an embedded active nic", "[2020/09/23 06:59:00 PM] [INFO] No DPDK mapping available in path (/var/lib/os-net-config/dpdk_mapping.yaml)", "[2020/09/23 06:59:00 PM] [INFO] Active nics are ['eth0', 'eth1', 'eth2']", "[2020/09/23 06:59:00 PM] [INFO] nic3 mapped to: eth2", "[2020/09/23 06:59:00 PM]2020-09-23 18:59:03.368174 | fa163e76-26da-e8d8-8e88-000000000ab8 | FATAL | Run tripleo_os_net_config_module with network_config | controller-0 | error={"ansible_job_id": "329351640964.7889", "changed": false, "finished": 1, "msg": "Running os-net-config --config-file /etc/os-net-config/config.json --debug --detailed-exit-codes failed with return code 1.", "rc": 1, "stderr": "[2020/09/23 06:58:59 PM] [INFO] Using config file at: /etc/os-net-config/config.json\n[2020/09/23 06:58:59 PM] [INFO] Ifcfg net config provider created.\n[2020/09/23 06:58:59 PM] [INFO] Not using any mapping file.\n[2020/09/23 06:58:59 PM] [DEBUG] network_config JSON: [{'addresses': [{'ip_netmask': '192.168.24.20/24'}], 'dns_servers': ['192.168.1.57'], 'domain': ['ooo.test', 'nightly-latest-tripleo-ipa-ds.test'], 'members': [{'name': 'interface_name', 'primary': True, 'type': 'interface'}], 'name': 'bridge_name', 'routes': [{'default': True, 'next_hop': '192.168.24.14'}], 'type': 'ovs_bridge', 'use_dhcp': False}]\n[2020/09/23 06:59:00 PM] [INFO] Finding active nics\n[2020/09/23 06:59:00 PM] [INFO] eth1 is an embedded active nic\n[2020/09/23 06:59:00 PM] [INFO] lo is not an active nic\n[2020/09/23 06:59:00 PM] [INFO] eth2 is an embedded active nic\n[2020/09/23 06:59:00 PM] [INFO] eth0 is an embedded active nic\n[2020/09/23 06:59:00 PM] [INFO] No DPDK mapping available in path (/var/lib/os-net-config/dpdk_mapping.yaml)\n[2020/09/23 06:59:00 PM] [INFO] Active nics are ['eth0', 'eth1', 'eth2']\n[2020/09/23 06:59:00 PM] [INFO] nic3 mapped to: eth2\n[2020/09/23 06:59:00 PM] [INFO] nic1 mapped to: eth0\n[2020/09/23 06:59:00 PM] [INFO] nic2 mapped to: eth1\n[2020/09/23 06:59:00 PM] [INFO] adding bridge: bridge_name\n[2020/09/23 06:59:00 PM] [ERROR] Unable to read mac address: interface_name\nTraceback (most recent call last):\n File \"/bin/os-net-config\", line 10, in <module>\n sys.exit(main())\n File \"/usr/lib/python3.6/site-packages/os_net_config/cli.py\", line 343, in main\n provider.add_object(obj)\n File \"/usr/lib/python3.6/site-packages/os_net_config/__init__.py\", line 64, in add_object\n self.add_bridge(obj)\n File \"/usr/lib/python3.6/site-packages/os_net_config/impl_ifcfg.py\", line 807, in add_bridge\n data = self._add_common(bridge)\n File \"/usr/lib/python3.6/site-packages/os_net_config/impl_ifcfg.py\", line 475, in _add_common\n mac = utils.interface_mac(base_opt.primary_interface_name)\n File \"/usr/lib/python3.6/site-packages/os_net_config/utils.py\", line 120, in interface_mac\n with open(_SYS_CLASS_NET + '/%s/address' % name, 'r') as f:\nFileNotFoundError: [Errno 2] No such file or directory: '/sys/class/net/interface_name/address'\n", "stderr_lines": ["[2020/09/23 06:58:59 PM] [INFO] Using config file at: /etc/os-net-config/config.json", "[2020/09/23 06:58:59 PM] [INFO] Ifcfg net config provider created.", "[2020/09/23 06:58:59 PM] [INFO] Not using any mapping file.", "[2020/09/23 06:58:59 PM] [DEBUG] network_config JSON: [{'addresses': [{'ip_netmask': '192.168.24.20/24'}], 'dns_servers': ['192.168.1.57'], 'domain': ['ooo.test', 'nightly-latest-tripleo-ipa-ds.test'], 'members': [{'name': 'interface_name', 'primary': True, 'type': 'interface'}], 'name': 'bridge_name', 'routes': [{'default': True, 'next_hop': '192.168.24.14'}], 'type': 'ovs_bridge', 'use_dhcp': False}]", "[2020/09/23 06:59:00 PM] [INFO] Finding active nics", "[2020/09/23 06:59:00 PM] [INFO] eth1 is an embedded active nic", "[2020/09/23 06:59:00 PM] [INFO] lo is not an active nic", "[2020/09/23 06:59:00 PM] [INFO] eth2 is an embedded active nic", "[2020/09/23 06:59:00 PM] [INFO] eth0 is an embedded active nic", "[2020/09/23 06:59:00 PM] [INFO] No DPDK mapping available in path (/var/lib/os-net-config/dpdk_mapping.yaml)", "[2020/09/23 06:59:00 PM] [INFO] Active nics are ['eth0', 'eth1', 'eth2']", "[2020/09/23 06:59:00 PM] [INFO] nic3 mapped to: eth2", "[2020/09/23 06:59:00 PM] [INFO] nic1 mapped to: eth0", "[2020/09/23 06:59:00 PM] [INFO] nic2 mapped to: eth1", "[2020/09/23 06:59:00 PM] [INFO] adding bridge: bridge_name", "[2020/09/23 06:59:00 PM] [ERROR] Unable to read mac address: interface_name", "Traceback (most recent call last):", " File \"/bin/os-net-config\", line 10, in <module>", " sys.exit(main())", " File \"/usr/lib/python3.6/site-packages/os_net_config/cli.py\", line 343, in main", " provider.add_object(obj)", " File \"/usr/lib/python3.6/site-packages/os_net_config/__init__.py\", line 64, in add_object", " self.add_bridge(obj)", " File \"/usr/lib/python3.6/site-packages/os_net_config/impl_ifcfg.py\", line 807, in add_bridge", " data = self._add_common(bridge)", " File \"/usr/lib/python3.6/site-packages/os_net_config/impl_ifcfg.py\", line 475, in _add_common", " mac = utils.interface_mac(base_opt.primary_interface_name)", " File \"/usr/lib/python3.6/site-packages/os_net_config/utils.py\", line 120, in interface_mac", " with open(_SYS_CLASS_NET + '/%s/address' % name, 'r') as f:", "FileNotFoundError: [Errno 2] No such file or directory: '/sys/class/net/interface_name/address'"], "stdout": "", "stdout_lines": []}
    2020-09-23 18:59:03.370301 | fa163e76-26da-e8d8-8e88-000000000ab8 | TIMING | tripleo_network_config : Run tripleo_os_net_config_module with network_config | controller-0 | 0:06:54.190699 | 4.55s
    .6/site-packages/os_net_config/cli.py\", line 343, in main\n provider.add_object(obj)\n File \"/usr/lib/python3.6/site-packages/os_net_config/__init__.py\", line 64, in add_object\n self.add_bridge(obj)\n File \"/usr/lib/python3.6/site-packages/os_net_config/impl_ifcfg.py\", line 807, in add_bridge\n data = self._add_common(bridge)\n File \"/usr/lib/python3.6/site-packages/os_net_config/impl_ifcfg.py\", line 475, in _add_common\n mac = utils.interface_mac(base_opt.primary_interface_name)\n File \"/usr/lib/python3.6/site-packages/os_net_config/utils.py\", line 120, in interface_mac\n with open(_SYS_CLASS_NET + '/%s/address' % name, 'r') as f:\nFileNotFoundError: [Errno 2] No such file or directory: '/sys/class/net/interface_name/address'\n", "stderr_lines": ["[2020/09/23 06:59:01 PM] [INFO] Using config file at: /etc/os-net-config/config.json", "[2020/09/23 06:59:01 PM] [INFO] Ifcfg net config provider created.", "[2020/09/23 06:59:01 PM] [INFO] Not using any mapping file.", "[2020/09/23 06:59:01 PM] [DEBUG] network_config JSON: [{'addresses': [{'ip_netmask': '192.168.24.18/24'}], 'dns_servers': ['192.168.1.57'], 'domain': ['ooo.test', 'nightly-latest-tripleo-ipa-ds.test'], 'members': [{'name': 'interface_name', 'primary': True, 'type': 'interface'}], 'name': 'bridge_name', 'routes': [{'default': True, 'next_hop': '192.168.24.14'}], 'type': 'ovs_bridge', 'use_dhcp': False}]", "[2020/09/23 06:59:01 PM] [INFO] Finding active nics", "[2020/09/23 06:59:01 PM] [INFO] eth1 is an embedded active nic", "[2020/09/23 06:59:01 PM] [INFO] lo is not an active nic", "[2020/09/23 06:59:01 PM] [INFO] eth2 is an embedded active nic", "[2020/09/23 06:59:01 PM] [INFO] eth0 is an embedded active nic", "[2020/09/23 06:59:01 PM] [INFO] No DPDK mapping available in path (/var/lib/os-net-config/dpdk_mapping.yaml)", "[2020/09/23 06:59:01 PM] [INFO] Active nics are ['eth0', 'eth1', 'eth2']", "[2020/09/23 06:59:01 PM] [INFO] nic2 mapped to: eth1", "[2020/09/23 06:59:01 PM] [2020-09-23 18:59:03.758682 | fa163e76-26da-e8d8-8e88-000000000ad2 | FATAL | Run tripleo_os_net_config_module with network_config | compute-0 | error={"ansible_job_id": "835714799551.11154", "changed": false, "finished": 1, "msg": "Running os-net-config --config-file /etc/os-net-config/config.json --debug --detailed-exit-codes failed with return code 1.", "rc": 1, "stderr": "[2020/09/23 06:59:01 PM] [INFO] Using config file at: /etc/os-net-config/config.json\n[2020/09/23 06:59:01 PM] [INFO] Ifcfg net config provider created.\n[2020/09/23 06:59:01 PM] [INFO] Not using any mapping file.\n[2020/09/23 06:59:01 PM] [DEBUG] network_config JSON: [{'addresses': [{'ip_netmask': '192.168.24.18/24'}], 'dns_servers': ['192.168.1.57'], 'domain': ['ooo.test', 'nightly-latest-tripleo-ipa-ds.test'], 'members': [{'name': 'interface_name', 'primary': True, 'type': 'interface'}], 'name': 'bridge_name', 'routes': [{'default': True, 'next_hop': '192.168.24.14'}], 'type': 'ovs_bridge', 'use_dhcp': False}]\n[2020/09/23 06:59:01 PM] [INFO] Finding active nics\n[2020/09/23 06:59:01 PM] [INFO] eth1 is an embedded active nic\n[2020/09/23 06:59:01 PM] [INFO] lo is not an active nic\n[2020/09/23 06:59:01 PM] [INFO] eth2 is an embedded active nic\n[2020/09/23 06:59:01 PM] [INFO] eth0 is an embedded active nic\n[2020/09/23 06:59:01 PM] [INFO] No DPDK mapping available in path (/var/lib/os-net-config/dpdk_mapping.yaml)\n[2020/09/23 06:59:01 PM] [INFO] Active nics are ['eth0', 'eth1', 'eth2']\n[2020/09/23 06:59:01 PM] [INFO] nic2 mapped to: eth1\n[2020/09/23 06:59:01 PM] [INFO] nic3 mapped to: eth2\n[2020/09/23 06:59:01 PM] [INFO] nic1 mapped to: eth0\n[2020/09/23 06:59:01 PM] [INFO] adding bridge: bridge_name\n[2020/09/23 06:59:01 PM] [ERROR] Unable to read mac address: interface_name\nTraceback (most recent call last):\n File \"/bin/os-net-config\", line 10, in <module>\n sys.exit(main())\n File \"/usr/lib/python3.6/site-packages/os_net_config/cli.py\", line 343, in main\n provider.add_object(obj)\n File \"/usr/lib/python3.6/site-packages/os_net_config/__init__.py\", line 64, in add_object\n self.add_bridge(obj)\n File \"/usr/lib/python3.6/site-packages/os_net_config/impl_ifcfg.py\", line 807, in add_bridge\n data = self._add_common(bridge)\n File \"/usr/lib/python3.6/site-packages/os_net_config/impl_ifcfg.py\", line 475, in _add_common\n mac = utils.interface_mac(base_opt.primary_interface_name)\n File \"/usr/lib/python3.6/site-packages/os_net_config/utils.py\", line 120, in interface_mac\n with open(_SYS_CLASS_NET + '/%s/address' % name, 'r') as f:\nFileNotFoundError: [Errno 2] No such file or directory: '/sys/class/net/interface_name/address'\n", "stderr_lines": ["[2020/09/23 06:59:01 PM] [INFO] Using config file at: /etc/os-net-config/config.json", "[2020/09/23 06:59:01 PM] [INFO] Ifcfg net config provider created.", "[2020/09/23 06:59:01 PM] [INFO] Not using any mapping file.", "[2020/09/23 06:59:01 PM] [DEBUG] network_config JSON: [{'addresses': [{'ip_netmask': '192.168.24.18/24'}], 'dns_servers': ['192.168.1.57'], 'domain': ['ooo.test', 'nightly-latest-tripleo-ipa-ds.test'], 'members': [{'name': 'interface_name', 'primary': True, 'type': 'interface'}], 'name': 'bridge_name', 'routes': [{'default': True, 'next_hop': '192.168.24.14'}], 'type': 'ovs_bridge', 'use_dhcp': False}]", "[2020/09/23 06:59:01 PM] [INFO] Finding active nics", "[2020/09/23 06:59:01 PM] [INFO] eth1 is an embedded active nic", "[2020/09/23 06:59:01 PM] [INFO] lo is not an active nic", "[2020/09/23 06:59:01 PM] [INFO] eth2 is an embedded active nic", "[2020/09/23 06:59:01 PM] [INFO] eth0 is an embedded active nic", "[2020/09/23 06:59:01 PM] [INFO] No DPDK mapping available in path (/var/lib/os-net-config/dpdk_mapping.yaml)", "[2020/09/23 06:59:01 PM] [INFO] Active nics are ['eth0', 'eth1', 'eth2']", "[2020/09/23 06:59:01 PM] [INFO] nic2 mapped to: eth1", "[2020/09/23 06:59:01 PM] [INFO] nic3 mapped to: eth2", "[2020/09/23 06:59:01 PM] [INFO] nic1 mapped to: eth0", "[2020/09/23 06:59:01 PM] [INFO] adding bridge: bridge_name", "[2020/09/23 06:59:01 PM] [ERROR] Unable to read mac address: interface_name", "Traceback (most recent call last):", " File \"/bin/os-net-config\", line 10, in <module>", " sys.exit(main())", " File \"/usr/lib/python3.6/site-packages/os_net_config/cli.py\", line 343, in main", " provider.add_object(obj)", " File \"/usr/lib/python3.6/site-packages/os_net_config/__init__.py\", line 64, in add_object", " self.add_bridge(obj)", " File \"/usr/lib/python3.6/site-packages/os_net_config/impl_ifcfg.py\", line 807, in add_bridge", " data = self._add_common(bridge)", " File \"/usr/lib/python3.6/site-packages/os_net_config/impl_ifcfg.py\", line 475, in _add_common", " mac = utils.interface_mac(base_opt.primary_interface_name)", " File \"/usr/lib/python3.6/site-packages/os_net_config/utils.py\", line 120, in interface_mac", " with open(_SYS_CLASS_NET + '/%s/address' % name, 'r') as f:", "FileNotFoundError: [Errno 2] No such file or directory: '/sys/class/net/interface_name/address'"], "stdout": "", "stdout_lines": []}
    2020-09-23 18:59:03.760033 | fa163e76-26da-e8d8-8e88-000000000ad2 | TIMING | tripleo_network_config : Run tripleo_os_net_config_module with network_config | compute-0 | 0:06:54.580439 | 4.31s

These are the networking templates that I'm using for a deployed server environment:

[stack@undercloud ~]$ cat networking.yaml
# Ansible managed

resource_registry:
  OS::TripleO::Controller::Net::SoftwareConfig: /usr/share/openstack-tripleo-heat-templates/net-config-static-bridge.yaml
  OS::TripleO::Compute::Net::SoftwareConfig: /usr/share/openstack-tripleo-heat-templates/net-config-static-bridge.yaml
  OS::TripleO::DeployedServer::ControlPlanePort: /usr/share/openstack-tripleo-heat-templates/deployed-server/deployed-neutron-port.yaml
  OS::TripleO::Network::Ports::ControlPlaneVipPort: /usr/share/openstack-tripleo-heat-templates/deployed-server/deployed-neutron-port.yaml
  OS::TripleO::Network::Ports::RedisVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::Network::Ports::OVNDBsVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml

parameter_defaults:
  ControlPlaneDefaultRoute: 192.168.24.14
  NeutronPublicInterface: eth1
[stack@undercloud ~]$ cat hostname-map.yaml
# Ansible managed

parameter_defaults:
  # This maps the hostname values from each role to the hostname for each server.
  # For example, the default value for hostnames in the Controller role is
  # overcloud-controller-$N, where $N is the controller number (e.g., 0, 1, 2).
  # That value needs to map to the actual host name specified in Upshift. If you
  # supply an override to the hostname through role data, you need to update this
  # map. It's setup to assume the defaults. If this mapping isn't correct, the
  # overcloud install will fail because the hosts aren't discoverable.
  HostnameMap:
    overcloud-controller-0: controller-0
    overcloud-novacompute-0: compute-0

  # This is a list of all the infrastructure we want to use for the
  # pre-provisioned deployment. Each IP address is associated to the
  # provisioning netowrk in Upshift. This ensures the overcloud and undercloud
  # are using the same interface to communicate.
  DeployedServerPortMap:
    control_virtual_ip:
      fixed_ips:
        - ip_address: 192.168.24.100
      subnets:
        - cidr: 192.168.24.0/24
      network:
        tags:
          - 192.168.24.0/24
    controller-0-ctlplane:
      fixed_ips:
        - ip_address: 192.168.24.20
      subnets:
        - cidr: 192.168.24.0/24
      network:
        tags:
          - 192.168.24.0/24
    compute-0-ctlplane:
      fixed_ips:
        - ip_address: 192.168.24.18
      subnets:
        - cidr: 192.168.24.0/24
      network:
        tags:
          - 192.168.24.0/24

It appears that the following change doesn't include the necessary changes for the interface name refactoring:

https://review.opendev.org/#/c/749523/22

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.opendev.org/753832

Changed in tripleo:
assignee: nobody → Alex Schultz (alex-schultz)
status: New → In Progress
Alex Schultz (alex-schultz)
Changed in tripleo:
importance: Undecided → Critical
milestone: none → victoria-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (master)

Change abandoned by Alex Schultz (<email address hidden>) on branch: master
Review: https://review.opendev.org/753832
Reason: https://review.opendev.org/#/c/751720/ I think addressed this. It left master broken for about a day since it landed so much later than the first

Revision history for this message
Alex Schultz (alex-schultz) wrote :

Closing because https://review.opendev.org/#/c/751720/ should fix this up. please reopen it if it's still broke

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.opendev.org/753932

Rabi Mishra (rabi)
Changed in tripleo:
status: Fix Released → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-ansible (master)

Fix proposed to branch: master
Review: https://review.opendev.org/753957

Changed in tripleo:
assignee: Alex Schultz (alex-schultz) → Rabi Mishra (rabi)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-ansible (master)

Change abandoned by Rabi Mishra (<email address hidden>) on branch: master
Review: https://review.opendev.org/753957

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/753932
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=b2d3c05f4e6926413a657fd5579f39bf5e1cb4be
Submitter: Zuul
Branch: master

commit b2d3c05f4e6926413a657fd5579f39bf5e1cb4be
Author: Rabi Mishra <email address hidden>
Date: Thu Sep 24 09:14:56 2020 +0530

    Replace bridge_name and interface_name in config

    The new ansible os_net_config does not replace the interface_name
    and bridge_name variables in net config. This replaces those
    variables beforehand.

    Though it would have been good to handle it in module itself, in
    the proposed way to generate these configs with ansible would
    allow us to replace them with group_vars.

    Change-Id: I2049d3ce2ad10507fd846d3876e20cc3ec4b9fbe
    Closes-Bug: #1896829

Changed in tripleo:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.