Multicloud :: Azure OnPrem :: OnPrem GW network restart fails.

[Ritam Gangopadhyay]

I see network restart failing intermittently on the OnPrem GW and I have to reboot the node to recover it from this state.

Ansible Play:- ansible/gateway/playbooks/deploy_and_run_all.yml
Ansible File:- ansible/gateway/playbooks/roles/loopback/tasks/main.yaml
Ansible Task:-

  - name: Restart networking Ubuntu
    service:
      name: networking
      state: restarted
    when: ansible_distribution == "Ubuntu"

Ansible ERROR:-

2018-10-26 09:37:15,958 p=18479 u=root | fatal: [192.168.2.1]: FAILED! => {"changed": false, "failed": true, "msg": "Unable to restart service networking: Job for networking.service failed because the control process exited with error code. See \"systemctl status networking.service\" and \"journalctl -xe\" for details.\n"}

Manually if I try I see this:-

root@5c3s1-node4:~# service networking status
* networking.service - Raise network interfaces
   Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
  Drop-In: /run/systemd/generator/networking.service.d
           `-50-insserv.conf-$network.conf
   Active: failed (Result: exit-code) since Fri 2018-10-26 02:37:13 PDT; 23min ago
     Docs: man:interfaces(5)
  Process: 36080 ExecStop=/sbin/ifdown -a --read-environment --exclude=lo (code=exited, status=0/SUCCESS)
  Process: 36244 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=1/FAILURE)
  Process: 36236 ExecStartPre=/bin/sh -c [ "$CONFIGURE_INTERFACES" != "no" ] && [ -n "$(ifquery --read-environment --list --exclude=lo)" ] && udevadm settle (code=exited, status=0/SUCCESS)
 Main PID: 36244 (code=exited, status=1/FAILURE)

Oct 26 02:37:13 5c3s1-node4 ifup[36244]: /etc/network/interfaces.d/interfaces: Is a directory
Oct 26 02:37:13 5c3s1-node4 ifup[36244]: /etc/network/interfaces.d/interfaces: Is a directory
Oct 26 02:37:13 5c3s1-node4 ifup[36244]: sh: echo: I/O error
Oct 26 02:37:13 5c3s1-node4 ifup[36244]: Waiting for a slave to join bond0 (will timeout after 60s)
Oct 26 02:37:13 5c3s1-node4 ifup[36244]: RTNETLINK answers: File exists
Oct 26 02:37:13 5c3s1-node4 ifup[36244]: Failed to bring up bond0.
Oct 26 02:37:13 5c3s1-node4 systemd[1]: networking.service: Main process exited, code=exited, status=1/FAILURE
Oct 26 02:37:13 5c3s1-node4 systemd[1]: Failed to start Raise network interfaces.
Oct 26 02:37:13 5c3s1-node4 systemd[1]: networking.service: Unit entered failed state.
Oct 26 02:37:13 5c3s1-node4 systemd[1]: networking.service: Failed with result 'exit-code'.
root@5c3s1-node4:~# service networking start
Job for networking.service failed because the control process exited with error code. See "systemctl status networking.service" and "journalctl -xe" for details.
root@5c3s1-node4:~# systemctl status networking.service
* networking.service - Raise network interfaces
   Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
  Drop-In: /run/systemd/generator/networking.service.d
           `-50-insserv.conf-$network.conf
   Active: failed (Result: exit-code) since Fri 2018-10-26 03:01:20 PDT; 15s ago
     Docs: man:interfaces(5)
  Process: 36080 ExecStop=/sbin/ifdown -a --read-environment --exclude=lo (code=exited, status=0/SUCCESS)
  Process: 37499 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=1/FAILURE)
  Process: 37493 ExecStartPre=/bin/sh -c [ "$CONFIGURE_INTERFACES" != "no" ] && [ -n "$(ifquery --read-environment --list --exclude=lo)" ] && udevadm settle (code=exited, status=0/SUCCESS)
 Main PID: 37499 (code=exited, status=1/FAILURE)

Oct 26 03:01:19 5c3s1-node4 sh[37493]: /etc/network/interfaces.d/interfaces: Is a directory
Oct 26 03:01:19 5c3s1-node4 ifup[37499]: /etc/network/interfaces.d/interfaces: Is a directory
Oct 26 03:01:20 5c3s1-node4 ifup[37499]: sh: echo: I/O error
Oct 26 03:01:20 5c3s1-node4 ifup[37499]: Waiting for a slave to join bond0 (will timeout after 60s)
Oct 26 03:01:20 5c3s1-node4 ifup[37499]: RTNETLINK answers: File exists
Oct 26 03:01:20 5c3s1-node4 ifup[37499]: Failed to bring up bond0.
Oct 26 03:01:20 5c3s1-node4 systemd[1]: networking.service: Main process exited, code=exited, status=1/FAILURE
Oct 26 03:01:20 5c3s1-node4 systemd[1]: Failed to start Raise network interfaces.
Oct 26 03:01:20 5c3s1-node4 systemd[1]: networking.service: Unit entered failed state.
Oct 26 03:01:20 5c3s1-node4 systemd[1]: networking.service: Failed with result 'exit-code'.
root@5c3s1-node4:~#

[Ritam Gangopadhyay (ritam)]

To recover I have to reboot the node:-

root@5c3s1-node4:~# reboot
root@5c3s1-node4:~# Connection to 10.87.74.132 closed by remote host.
Connection to 10.87.74.132 closed.
[ritam@nodem3 ~]$ ssh root@10.87.74.132
root@10.87.74.132's password:
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-87-generic x86_64)

 * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/advantage
Last login: Fri Oct 26 02:59:44 2018 from 10.204.216.92
root@5c3s1-node4:~# systemctl status networking.service
* networking.service - Raise network interfaces
   Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
  Drop-In: /run/systemd/generator/networking.service.d
           `-50-insserv.conf-$network.conf
   Active: active (exited) since Fri 2018-10-26 03:16:23 PDT; 2min 22s ago
     Docs: man:interfaces(5)
  Process: 1036 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=0/SUCCESS)
  Process: 985 ExecStartPre=/bin/sh -c [ "$CONFIGURE_INTERFACES" != "no" ] && [ -n "$(ifquery --read-environment --list --exclude=lo)" ] && udevadm settle (code=exited, status=0/SUCCESS)
 Main PID: 1036 (code=exited, status=0/SUCCESS)
    Tasks: 0
   Memory: 0B
      CPU: 0
   CGroup: /system.slice/networking.service

Oct 26 03:16:16 5c3s1-node4 systemd[1]: Starting Raise network interfaces...
Oct 26 03:16:16 5c3s1-node4 sh[985]: /etc/network/interfaces.d/interfaces: Is a directory
Oct 26 03:16:16 5c3s1-node4 ifup[1036]: /etc/network/interfaces.d/interfaces: Is a directory
Oct 26 03:16:17 5c3s1-node4 ifup[1036]: /sbin/ifup: waiting for lock on /run/network/ifstate.em1
Oct 26 03:16:23 5c3s1-node4 systemd[1]: Started Raise network interfaces.
root@5c3s1-node4:~#

[Sanju Abraham (asanju)]

This is not seen on other servers in the lab and deployments done by dev team and solutions.

Since it is not reproducible on other setups, Please provide access to the setup if it is reprodicuble and I will surely try to check and fix it approriately.