tripleo

Determining IP information for ens3... failed on cs9 molecule job with delegated driver

Bug #1979807 reported by chandan kumar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Confirmed
High
Unassigned
tripleo zed-1

Bug Description

https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ansible-centos-stream-molecule-tripleo_update_trusted_cas&project=openstack/tripleo-ansible is failing while porting it on centos-stream 9. 839688: Use ubi9-init image as base | https://review.opendev.org/c/openstack/tripleo-ansible/+/839688

This job uses delegated driver, It means the molecule job runs on centos-stream 9 host and below is the failure[1].
```
TASK [apply network config] ****************************************************
2022-06-24 08:49:34.635338 | centos-9-stream | fatal: [instance]: FAILED! => changed=false
2022-06-24 08:49:34.635361 | centos-9-stream | msg: Running os-net-config --config-file /tmp/os-net-configlmwogybt --debug failed with return code 1.
2022-06-24 08:49:34.635384 | centos-9-stream | rc: 1
2022-06-24 08:49:34.635407 | centos-9-stream | stderr: |-
2022-06-24 08:49:34.635429 | centos-9-stream | Traceback (most recent call last):
2022-06-24 08:49:34.635452 | centos-9-stream | File "/bin/os-net-config", line 10, in <module>
2022-06-24 08:49:34.635475 | centos-9-stream | sys.exit(main())
2022-06-24 08:49:34.635498 | centos-9-stream | File "/usr/lib/python3.9/site-packages/os_net_config/cli.py", line 340, in main
2022-06-24 08:49:34.635521 | centos-9-stream | files_changed = provider.apply(cleanup=opts.cleanup,
2022-06-24 08:49:34.635543 | centos-9-stream | File "/usr/lib/python3.9/site-packages/os_net_config/impl_ifcfg.py", line 1944, in apply
2022-06-24 08:49:34.635566 | centos-9-stream | raise os_net_config.ConfigurationError(message)
2022-06-24 08:49:34.635589 | centos-9-stream | os_net_config.ConfigurationError: Failure(s) occurred when applying configuration
2022-06-24 08:49:34.635632 | centos-9-stream | stderr_lines: <omitted>
2022-06-24 08:49:34.635661 | centos-9-stream | stdout: |-
2022-06-24 08:49:34.635683 | centos-9-stream | 2022-06-24 08:48:33.356 INFO os_net_config.main Using config file at: /tmp/os-net-configlmwogybt
2022-06-24 08:49:34.635707 | centos-9-stream | 2022-06-24 08:48:33.357 INFO os_net_config.impl_ifcfg.__init__ Ifcfg net config provider created.
2022-06-24 08:49:34.635729 | centos-9-stream | 2022-06-24 08:48:33.357 INFO os_net_config.main Not using any mapping file.
2022-06-24 08:49:34.635752 | centos-9-stream | 2022-06-24 08:48:33.358 DEBUG os_net_config.main network_config: [{'type': 'interface', 'name': 'ens3', 'use_dhcp': True}]
2022-06-24 08:49:34.635775 | centos-9-stream | 2022-06-24 08:48:33.574 INFO os_net_config.utils._ordered_nics Finding active nics
2022-06-24 08:49:34.635809 | centos-9-stream | 2022-06-24 08:48:33.575 INFO os_net_config.utils._ordered_nics ens3 is an active nic
2022-06-24 08:49:34.635834 | centos-9-stream | 2022-06-24 08:48:33.575 INFO os_net_config.utils._ordered_nics lo is not an active nic
2022-06-24 08:49:34.635856 | centos-9-stream | 2022-06-24 08:48:33.575 INFO os_net_config.utils._ordered_nics No DPDK mapping available in path (/var/lib/os-net-config/dpdk_mapping.yaml)
2022-06-24 08:49:34.635892 | centos-9-stream | 2022-06-24 08:48:33.576 INFO os_net_config.utils._ordered_nics Active nics are ['ens3']
2022-06-24 08:49:34.635916 | centos-9-stream | 2022-06-24 08:48:33.576 INFO os_net_config.objects.mapped_nics nic1 mapped to: ens3
2022-06-24 08:49:34.635955 | centos-9-stream | 2022-06-24 08:48:33.576 INFO os_net_config.impl_ifcfg.add_interface adding interface: ens3
2022-06-24 08:49:34.635982 | centos-9-stream | 2022-06-24 08:48:33.576 DEBUG os_net_config.impl_ifcfg.add_interface interface data: # This file is autogenerated by os-net-config
2022-06-24 08:49:34.636005 | centos-9-stream | DEVICE=ens3
2022-06-24 08:49:34.636029 | centos-9-stream | ONBOOT=yes
2022-06-24 08:49:34.636051 | centos-9-stream | HOTPLUG=no
2022-06-24 08:49:34.636074 | centos-9-stream | NM_CONTROLLED=no
2022-06-24 08:49:34.636104 | centos-9-stream | BOOTPROTO=dhcp
2022-06-24 08:49:34.636143 | centos-9-stream |
2022-06-24 08:49:34.636170 | centos-9-stream | 2022-06-24 08:48:33.576 INFO os_net_config.impl_ifcfg.apply applying network configs...
2022-06-24 08:49:34.636194 | centos-9-stream | 2022-06-24 08:48:33.577 DEBUG os_net_config.utils.diff Diff file data:
2022-06-24 08:49:34.636217 | centos-9-stream | # Automatically generated, do not edit
2022-06-24 08:49:34.636239 | centos-9-stream | DEVICE=ens3
2022-06-24 08:49:34.636262 | centos-9-stream | BOOTPROTO=static
2022-06-24 08:49:34.636285 | centos-9-stream | HWADDR=fa:16:3e:e8:21:0a
2022-06-24 08:49:34.636309 | centos-9-stream | ONBOOT=yes
2022-06-24 08:49:34.636331 | centos-9-stream | NM_CONTROLLED=yes
2022-06-24 08:49:34.636354 | centos-9-stream | TYPE=Ethernet
2022-06-24 08:49:34.636376 | centos-9-stream | IPADDR=198.72.124.76
2022-06-24 08:49:34.636400 | centos-9-stream | NETMASK=255.255.255.0
2022-06-24 08:49:34.636423 | centos-9-stream | DEFROUTE=yes
2022-06-24 08:49:34.636446 | centos-9-stream | GATEWAY=198.72.124.1
2022-06-24 08:49:34.636469 | centos-9-stream |
2022-06-24 08:49:34.636492 | centos-9-stream | 2022-06-24 08:48:33.577 DEBUG os_net_config.utils.diff Diff data:
2022-06-24 08:49:34.636515 | centos-9-stream | # This file is autogenerated by os-net-config
2022-06-24 08:49:34.636538 | centos-9-stream | DEVICE=ens3
2022-06-24 08:49:34.636561 | centos-9-stream | ONBOOT=yes
2022-06-24 08:49:34.636583 | centos-9-stream | HOTPLUG=no
2022-06-24 08:49:34.636606 | centos-9-stream | NM_CONTROLLED=no
2022-06-24 08:49:34.636693 | centos-9-stream | BOOTPROTO=dhcp
2022-06-24 08:49:34.636719 | centos-9-stream |
2022-06-24 08:49:34.636742 | centos-9-stream | 2022-06-24 08:48:33.577 DEBUG os_net_config.impl_ifcfg.ifcfg_requires_restart Original ifcfg file:
2022-06-24 08:49:34.636765 | centos-9-stream | # Automatically generated, do not edit
2022-06-24 08:49:34.636788 | centos-9-stream | DEVICE=ens3
2022-06-24 08:49:34.636811 | centos-9-stream | BOOTPROTO=static
2022-06-24 08:49:34.636834 | centos-9-stream | HWADDR=fa:16:3e:e8:21:0a
2022-06-24 08:49:34.636857 | centos-9-stream | ONBOOT=yes
2022-06-24 08:49:34.636886 | centos-9-stream | NM_CONTROLLED=yes
2022-06-24 08:49:34.636899 | centos-9-stream | TYPE=Ethernet
2022-06-24 08:49:34.636911 | centos-9-stream | IPADDR=198.72.124.76
2022-06-24 08:49:34.636933 | centos-9-stream | NETMASK=255.255.255.0
2022-06-24 08:49:34.636946 | centos-9-stream | DEFROUTE=yes
2022-06-24 08:49:34.636959 | centos-9-stream | GATEWAY=198.72.124.1
2022-06-24 08:49:34.636971 | centos-9-stream |
2022-06-24 08:49:34.636984 | centos-9-stream | 2022-06-24 08:48:33.577 DEBUG os_net_config.impl_ifcfg.ifcfg_requires_restart New ifcfg file:
2022-06-24 08:49:34.636996 | centos-9-stream | # This file is autogenerated by os-net-config
2022-06-24 08:49:34.637009 | centos-9-stream | DEVICE=ens3
2022-06-24 08:49:34.637021 | centos-9-stream | ONBOOT=yes
2022-06-24 08:49:34.637034 | centos-9-stream | HOTPLUG=no
2022-06-24 08:49:34.637046 | centos-9-stream | NM_CONTROLLED=no
2022-06-24 08:49:34.637059 | centos-9-stream | BOOTPROTO=dhcp
2022-06-24 08:49:34.637071 | centos-9-stream |
2022-06-24 08:49:34.637084 | centos-9-stream | 2022-06-24 08:48:33.577 DEBUG os_net_config.impl_ifcfg.ifcfg_requires_restart DHCP on BOOTPROTO requires restart
2022-06-24 08:49:34.637097 | centos-9-stream | 2022-06-24 08:48:33.578 DEBUG os_net_config.utils.diff Diff file data:
2022-06-24 08:49:34.637109 | centos-9-stream |
2022-06-24 08:49:34.637126 | centos-9-stream | 2022-06-24 08:48:33.578 DEBUG os_net_config.utils.diff Diff data:
2022-06-24 08:49:34.637149 | centos-9-stream |
2022-06-24 08:49:34.637165 | centos-9-stream | 2022-06-24 08:48:33.578 DEBUG os_net_config.utils.diff Diff file data:
2022-06-24 08:49:34.637185 | centos-9-stream |
2022-06-24 08:49:34.637199 | centos-9-stream | 2022-06-24 08:48:33.578 DEBUG os_net_config.utils.diff Diff data:
2022-06-24 08:49:34.637212 | centos-9-stream |
2022-06-24 08:49:34.637224 | centos-9-stream | 2022-06-24 08:48:33.578 DEBUG os_net_config.utils.diff Diff file data:
2022-06-24 08:49:34.637236 | centos-9-stream |
2022-06-24 08:49:34.637253 | centos-9-stream | 2022-06-24 08:48:33.578 DEBUG os_net_config.utils.diff Diff data:
2022-06-24 08:49:34.637266 | centos-9-stream |
2022-06-24 08:49:34.637279 | centos-9-stream | 2022-06-24 08:48:33.578 INFO os_net_config.execute running ifdown on interface: ens3
2022-06-24 08:49:34.637300 | centos-9-stream | 2022-06-24 08:48:33.777 INFO os_net_config.write_config Writing config /etc/sysconfig/network-scripts/ifcfg-ens3
2022-06-24 08:49:34.637315 | centos-9-stream | 2022-06-24 08:48:33.778 INFO os_net_config.execute running ifup on interface: ens3
2022-06-24 08:49:34.637327 | centos-9-stream | 2022-06-24 08:49:34.014 ERROR os_net_config.impl_ifcfg.apply Failure(s) occurred when applying configuration
2022-06-24 08:49:34.637340 | centos-9-stream | 2022-06-24 08:49:34.015 ERROR os_net_config.impl_ifcfg.apply stdout:
2022-06-24 08:49:34.637352 | centos-9-stream | Determining IP information for ens3... failed.
2022-06-24 08:49:34.637368 | centos-9-stream | , stderr: WARN : [ifup] You are using 'ifup' script provided by 'network-scripts', which are now deprecated.
2022-06-24 08:49:34.637381 | centos-9-stream | WARN : [ifup] 'network-scripts' will be removed from distribution in near future.
2022-06-24 08:49:34.637394 | centos-9-stream | WARN : [ifup] It is advised to switch to 'NetworkManager' instead - it provides 'ifup/ifdown' scripts as well.
2022-06-24 08:49:34.637406 | centos-9-stream | stdout_lines: <omitted>
```
[1]. https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ad3/839688/82/check/tripleo-ansible-centos-stream-molecule-tripleo_update_trusted_cas/ad32eb1/job-output.txt

Here is the codebase at which it is failing:
https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/ansible_plugins/tests/molecule/tripleo_os_net_config/converge.yml#L65

It might be related to the tripleo_os_net_config module itself.
It is blocking the porting so marking this job as a non-voting and later it can be fixed.

Revision history for this message
chandan kumar (chkumar246) wrote :

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8c4/839688/84/check/tripleo-ansible-centos-stream-molecule-tripleo-modules/8c4359a/job-output.txt is also impacted by this sometimes.

Revision history for this message
Harald Jensås (harald-jensas) wrote :

Are we sure there is a DHCP service available at all?

What configured the ens3 interface in the first place, used a static config and it was not os-net-config. And the static IP address on ens3 is in a public IP range, looks like a cloud-provider (iweb.com) range?

2022-06-24 08:49:34.636765 | centos-9-stream | # Automatically generated, do not edit
2022-06-24 08:49:34.636788 | centos-9-stream | DEVICE=ens3
2022-06-24 08:49:34.636811 | centos-9-stream | BOOTPROTO=static
<...>
2022-06-24 08:49:34.636911 | centos-9-stream | IPADDR=198.72.124.76

os-net-config uses this header: "# This file is autogenerated by os-net-config"

I think we may have to remove this part of the test:
https://opendev.org/openstack/tripleo-ansible/blame/branch/master/tripleo_ansible/ansible_plugins/tests/molecule/tripleo_os_net_config/converge.yml#L27-L64

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/839688
Committed: https://opendev.org/openstack/tripleo-ansible/commit/4c9233d07c0886871719b46b2801c7b0565f26cc
Submitter: "Zuul (22348)"
Branch: master

commit 4c9233d07c0886871719b46b2801c7b0565f26cc
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Mon May 30 09:45:42 2022 +0530

    Use ubi9-init image as base

    CentOS-8 no longer exists and existing jobs are running
    on centos-stream-8. There will be no openstack master
    rpm packages for centos-stream-8.

    In order to use ubi9-init image in molecule jobs,
    we need to include following workarounds:
    - redhat-release package(RH specific package) is installed in ubi-init
      which does not work with centos-stream package.
      On removing redhat-release, dnf ansible breaks as they did not find
      --releasever during package installation.
      - By adding releasever in the package module fixes the issue.
    - In Molecule job, we can use any hostname either of centos or ubi both
      will work.
    - It changes the nodeset and image to ubi9.
    - Install pip < 20.3.0 to avoid https://pip.pypa.io/en/stable/topics/dependency-resolution/#backtracking
      and upstream requirements file to avoid cyclic dependency
      issue.
    - Set enable_fips: true in collectd jobs as in molecule it is not
      needed otherwise job will enter into retry_limit.

    We cannot use centos:stream9 image as, init support is removed
    from there.

    Below is the list of following jobs marked non-voting
    due to following bugs.
    - tripleo-ansible-centos-stream-molecule-tripleo_update_trusted_cas
      - https://bugs.launchpad.net/tripleo/+bug/1979807
    - tripleo-ansible-centos-stream-molecule-tripleo-modules
      - https://bugs.launchpad.net/tripleo/+bug/1979807
    - tripleo-ansible-centos-stream-molecule-tripleo_container_manage
      - https://bugs.launchpad.net/tripleo/+bug/1979810
    - tripleo-ansible-centos-stream-molecule-tripleo_redhat_enforce
      - https://bugs.launchpad.net/tripleo/+bug/1979971
    - tripleo-ansible-centos-stream-molecule-tripleo_collectd
      - https://bugs.launchpad.net/tripleo/+bug/1979972
    - tripleo-ansible-centos-stream-molecule-backup_and_restore
      - https://bugs.launchpad.net/tripleo/+bug/1980198

    Related-Bug: #1979807
    Related-Bug: #1979810
    Related-Bug: #1979971
    Related-Bug: #1979972
    Related-Bug: #1980198

    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>
    Change-Id: I75a27ca496fb7670ba22a6a22f62f001c8afe6a9

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-ansible (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/848810

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-ansible (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/848810
Committed: https://opendev.org/openstack/tripleo-ansible/commit/b666c78954fbbe7edfd50b6b949c4921fc6c5518
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit b666c78954fbbe7edfd50b6b949c4921fc6c5518
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Mon May 30 09:45:42 2022 +0530

    Use ubi9-init image as base

    CentOS-8 no longer exists and existing jobs are running
    on centos-stream-8. There will be no openstack master
    rpm packages for centos-stream-8.

    In order to use ubi9-init image in molecule jobs,
    we need to include following workarounds:
    - redhat-release package(RH specific package) is installed in ubi-init
      which does not work with centos-stream package.
      On removing redhat-release, dnf ansible breaks as they did not find
      --releasever during package installation.
      - By adding releasever in the package module fixes the issue.
    - In Molecule job, we can use any hostname either of centos or ubi both
      will work.
    - It changes the nodeset and image to ubi9.
    - Install pip < 20.3.0 to avoid https://pip.pypa.io/en/stable/topics/dependency-resolution/#backtracking
      and upstream requirements file to avoid cyclic dependency
      issue.
    - Set enable_fips: true in collectd jobs as in molecule it is not
      needed otherwise job will enter into retry_limit.

    We cannot use centos:stream9 image as, init support is removed
    from there.

    Below is the list of following jobs marked non-voting
    due to following bugs.
    - tripleo-ansible-centos-stream-molecule-tripleo_update_trusted_cas
      - https://bugs.launchpad.net/tripleo/+bug/1979807
    - tripleo-ansible-centos-stream-molecule-tripleo-modules
      - https://bugs.launchpad.net/tripleo/+bug/1979807
    - tripleo-ansible-centos-stream-molecule-tripleo_container_manage
      - https://bugs.launchpad.net/tripleo/+bug/1979810
    - tripleo-ansible-centos-stream-molecule-tripleo_redhat_enforce
      - https://bugs.launchpad.net/tripleo/+bug/1979971
    - tripleo-ansible-centos-stream-molecule-tripleo_collectd
      - https://bugs.launchpad.net/tripleo/+bug/1979972
    - tripleo-ansible-centos-stream-molecule-backup_and_restore
      - https://bugs.launchpad.net/tripleo/+bug/1980198

    Note: It is not a clean cherry pick.
    - Remove cap from molecule dependency
      to fix No module named 'ansiblelint.prerun'
    - Populate wallaby repos using tripleo-repos tool
    - This includes the commit 0bc2fa808d34235ffe7cf8d36920f6d96f0685b5
      which fixed the issue with the original change in master.
    - Marking tripleo-ansible-centos-stream-molecule-tripleo_packages
      to non-voting due to
      https://bugs.launchpad.net/tripleo/+bug/1983347

    Related-Bug: #1979807
    Related-Bug: #1979810
    Related-Bug: #1979971
    Related-Bug: #1979972
    Related-Bug: #1980198
    Related-Bug: #1983347

    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>
    Change-Id: I75a27ca496fb7670ba22a6a22f62f001c8afe6a9
    (cherry picked from commit 4c9233d07c0886871719b46b2801c7b0565f26cc)

tags: added: in-stable-wallaby
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.