Backup/restore master node failed. Docker containers logs: Socket service systemd-udevd.service not loaded, refusing

[Egor Kotko]

Sometimes Backup/restore master node failed. Docker containers logs: Socket service systemd-udevd.service not loaded, refusing

Scenario:
1. Setup master node
2. Create new environment
3. Choose Neutron, TUN
4. Add 3 controller
5. Add 2 compute
6. Verify networks
7. Deploy the environment
8. Verify networks
9. Run OSTF tests
10. Backup master node with default path for archive(#dockerctl backup)
11. Verify networks
12. Run OSTF tests
13. Add 1 compute node
14. Restore master node with default path for archive(#dockerctl restore)

The majority container logs contain:
#dockerctl logs nginx
...
Socket service systemd-udevd.service not loaded, refusing.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "478"
  build_id: "478"
  fuel-nailgun_sha: "ae949905142507f2cb446071783731468f34a572"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "481ed135de2cb5060cac3795428625befdd1d814"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "420c6fa5f8cb51f3322d95113f783967bde9836e"
  fuel-ostf_sha: "ab5fd151fc6c1aa0b35bc2023631b1f4836ecd61"
  fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
  fuelmenu_sha: "fac143f4dfa75785758e72afbdc029693e94ff2b"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "6c6b088a3d52dd0eaf43d59f3a3a149c93a07e7e"

Snapshot:
https://drive.google.com/file/d/0BzWDM1PONYEuV2xyZnJoZ2pqbnM/view?usp=sharing

[Matthew Mosesohn (raytrac3r)]

Workaround is to reboot Fuel Master and verify that containers started okay. It will go in Known Issues of Fuel 8.0 release notes.

[Alexander Kurenyshev (akurenyshev)]

The same error with restore with this scenario:

1 Setup master node
2 Create new environment
3 Choose Neutron, TUN
4 Add 3 controller
5 Add 2 compute
6 Verify networks
7 Deploy the environment
8 Verify networks
9 Run OSTF tests
10 Backup master node with default path for archive(#dockerctl backup)
11 Verify networks
12 Run OSTF tests
13 Reset environment
14 Restore master node with default path for archive(#dockerctl restore <path_to_the_backup_file>)

The reboot didn't solve a problem:
[root@nailgun ~]# last| grep reboot
reboot system boot 3.10.0-229.20.1. Tue Jan 26 12:23 - 12:29 (00:06)
reboot system boot 3.10.0-229.20.1. Mon Jan 25 11:20 - 12:23 (1+01:02)
reboot system boot 3.10.0-229.20.1. Mon Jan 25 10:53 - 12:23 (1+01:29)

[root@nailgun ~]# fuel node
500 Server Error: Internal Server Error (argument 2 to map() must support iteration)

[root@nailgun ~]# dockerctl shell nginx
[root@nailgun /]#
[root@nailgun /]# systemctl status systemd-udevd-control.socket
systemd-udevd-control.socket - udev Control Socket
   Loaded: loaded (/usr/lib/systemd/system/systemd-udevd-control.socket; static)
   Active: inactive (dead)
     Docs: man:systemd-udevd.service(8)
           man:udev(7)
   Listen: /run/udev/control (SequentialPacket)

[Alexander Kurenyshev (akurenyshev)]

Snapshot ^^

[Matthew Mosesohn (raytrac3r)]

Let's move this to mos-linux. There are systemd services on Fuel Master crashing, causing containers to fail.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/273479

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ivan Suzdal (<email address hidden>) on branch: master
Review: https://review.openstack.org/273479
Reason: https://review.openstack.org/#/c/273037/