CentOS-8 NetworkManager restarts and drops connection in tripleo-ci jobs

Bug #1885701 reported by Sagi (Sergey) Shnaidman on 2020-06-30
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Critical
yatin

Bug Description

We lose connection to upstream hosts, example console logs are:

tripleo-ci-centos-8-scenario004-standalone: pastebin.com/cKZAfAY2
tripleo-ci-centos-7-containers-multinode-train: https://pastebin.com/96mftQM4
tripleo-build-containers-centos-8-ussuri: https://pastebin.com/yfz0HpWm

It causes a lot of retry_limits in jobs.

wes hayutin (weshayutin) wrote :
Download full text (3.2 KiB)

2020-07-01 14:16:26.798304 | fa163efa-da5e-9f7e-6dbf-000000000078 | TIMING | include_role : tripleo_image_serve | 0:02:02.916 | 0.20s
2020-07-01 14:16:29.172223 | fa163efa-da5e-9f7e-6dbf-0000000005c5 | FATAL | ensure apache is installed | standalone | error={"changed": false, "msg": "Failed to download packages: Cannot download Packages/apr-util-1.6.1-6.el8.x86_64.rpm: All mirrors were tried", "results": []}

Jul 1 14:16:09 centos-8-inap-mtl01-0017550599 ansible-dnf[49123]: Invoked with name=['lvm2'] state=latest allow_downgrade=False autoremove=False bugfix=False disable_gpg_check=False disable_plugin=[] disablerepo=[] download_only=False enable_plugin=[] enablerepo=[] exclude=[] installroot=/ install_repoquery=True install_weak_deps=True security=False skip_broken=False update_cache=False update_only=False validate_certs=True lock_timeout=30 conf_file=None disable_excludes=None download_dir=None list=None releasever=None
Jul 1 14:16:11 centos-8-inap-mtl01-0017550599 NetworkManager[1023]: <warn> [1593612971.1051] dhcp4 (ens3): request timed out
Jul 1 14:16:11 centos-8-inap-mtl01-0017550599 NetworkManager[1023]: <info> [1593612971.1055] dhcp4 (ens3): state changed unknown -> timeout
Jul 1 14:16:11 centos-8-inap-mtl01-0017550599 NetworkManager[1023]: <info> [1593612971.1056] device (ens3): state change: ip-config -> failed (reason 'ip-config-unavailable', sys-iface-state: 'managed')
Jul 1 14:16:11 centos-8-inap-mtl01-0017550599 NetworkManager[1023]: <info> [1593612971.1062] manager: NetworkManager state is now DISCONNECTED
Jul 1 14:16:11 centos-8-inap-mtl01-0017550599 NetworkManager[1023]: <warn> [1593612971.1072] device (ens3): Activation: failed for connection 'Wired connection 1'
Jul 1 14:16:11 centos-8-inap-mtl01-0017550599 NetworkManager[1023]: <info> [1593612971.1075] device (ens3): state change: failed -> disconnected (reason 'none', sys-iface-state: 'managed')
Jul 1 14:16:11 centos-8-inap-mtl01-0017550599 NetworkManager[1023]: <info> [1593612971.1181] dhcp4 (ens3): canceled DHCP transaction
Jul 1 14:16:11 centos-8-inap-mtl01-0017550599 NetworkManager[1023]: <info> [1593612971.1182] dhcp4 (ens3): state changed timeout -> done
Jul 1 14:16:11 centos-8-inap-mtl01-0017550599 NetworkManager[1023]: <info> [1593612971.1195] policy: set-hostname: current hostname was changed outside NetworkManager: 'standalone.localdomain'
Jul 1 14:16:11 centos-8-inap-mtl01-0017550599 NetworkManager[1023]: <info> [1593612971.1196] policy: auto-activating connection 'Wired connection 1' (c073f3b1-0f9c-3f80-a77b-6ccff1be72e5)
Jul 1 14:16:11 centos-8-inap-mtl01-0017550599 NetworkManager[1023]: <info> [1593612971.1206] device (ens3): Activation: starting connection 'Wired connection 1' (c073f3b1-0f9c-3f80-a77b-6ccff1be72e5)
Jul 1 14:16:11 centos-8-inap-mtl01-0017550599 NetworkManager[1023]: <info> [1593612971.1207] device (ens3): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Jul 1 14:16:11 centos-8-inap-mtl01-0017550599 NetworkManager[1023]: <info> [1593612971.1212] manager: NetworkManager state is now CONNECTING
Jul 1 14:16:11 centos-8-inap-mtl01-0017550599 NetworkManager[1023]: <info> [15...

Read more...

Changed in tripleo:
milestone: none → victoria-1
summary: - Node failures on upstream infra, hosts are lost
+ CentOS-8 NetworkManager restarts and drops connection in tripleo-ci jobs
tags: added: promotion-blocker
wes hayutin (weshayutin) wrote :
Changed in tripleo:
assignee: nobody → Sagi (Sergey) Shnaidman (sshnaidm)
status: Triaged → In Progress
Changed in tripleo:
assignee: Sagi (Sergey) Shnaidman (sshnaidm) → yatin (yatinkarel)
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers