[ubuntu-focal] freezer_scheduler container stuck in restart loop

[joel waddell]

**Bug Report**

What happened:

enabling freezer in globals.yml results in a deployment that deploys the freezer containers but freezer_scheduler is stuck in a restart loop.

What you expected to happen:

expected freezer to deploy with both containers to be up and running.

How to reproduce it (minimal and precise):

Enable freezer in /etc/kolla/globals.yml, then ./kolla-ansible/tools/kolla-ansible -i multinode deploy

**Environment**:
* OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="20.04.1 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.1 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

* Kernel (e.g. `uname -a`):

Linux poc-kolla-deployer 5.4.0-48-generic #52-Ubuntu SMP Thu Sep 10 10:58:49 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

* Docker version if applicable (e.g. `docker version`):

root@poc-openstack-01:~# docker version
Client: Docker Engine - Community
 Version: 19.03.13
 API version: 1.40
 Go version: go1.13.15
 Git commit: 4484c46d9d
 Built: Wed Sep 16 17:02:52 2020
 OS/Arch: linux/amd64
 Experimental: false

Server: Docker Engine - Community
 Engine:
  Version: 19.03.13
  API version: 1.40 (minimum version 1.12)
  Go version: go1.13.15
  Git commit: 4484c46d9d
  Built: Wed Sep 16 17:01:20 2020
  OS/Arch: linux/amd64
  Experimental: false
 containerd:
  Version: 1.3.7
  GitCommit: 8fba4e9a7d01810a393d5d25a3621dc101981175
 runc:
  Version: 1.0.0-rc10
  GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version: 0.18.0
  GitCommit: fec3683

* Kolla-Ansible version (e.g. `git head or tag or stable branch` or pip package version if using release):

abed85e7f

* Docker image Install type (source/binary):

source

* Docker image distribution:

ubuntu

* Are you using official images from Docker Hub or self built?

official images
* If self built - Kolla version and environment used to build:
* Share your inventory file, globals.yml and other configuration files if relevant

[joel waddell (maasmaaster)]

/var/log/kolla/freezer/freezer-scheduler.log shows this:

102640 2020-10-27 12:50:09.625 6 ERROR freezer-scheduler
102641 2020-10-27 12:51:13.074 7 CRITICAL freezer-scheduler [-] Unhandled error: OSError: [Errno 88] Socket operation on non-socket
102642 2020-10-27 12:51:13.074 7 ERROR freezer-scheduler Traceback (most recent call last):
102643 2020-10-27 12:51:13.074 7 ERROR freezer-scheduler File "/var/lib/kolla/venv/bin/freezer-scheduler", line 8, in <module>
102644 2020-10-27 12:51:13.074 7 ERROR freezer-scheduler sys.exit(main())
102645 2020-10-27 12:51:13.074 7 ERROR freezer-scheduler File "/var/lib/kolla/venv/lib/python3.8/site-packages/freezer/scheduler/freezer_scheduler.py", line 253, in main
102646 2020-10-27 12:51:13.074 7 ERROR freezer-scheduler daemon.start()
102647 2020-10-27 12:51:13.074 7 ERROR freezer-scheduler File "/var/lib/kolla/venv/lib/python3.8/site-packages/freezer/scheduler/daemon.py", line 178, in start
102648 2020-10-27 12:51:13.074 7 ERROR freezer-scheduler with DaemonContext(pidfile=pidfile, signal_map=self.signal_map,
102649 2020-10-27 12:51:13.074 7 ERROR freezer-scheduler File "/var/lib/kolla/venv/lib/python3.8/site-packages/freezer/lib/pep3143daemon/daemon.py", line 132, in __init__
102650 2020-10-27 12:51:13.074 7 ERROR freezer-scheduler self.detach_process = detach_required()
102651 2020-10-27 12:51:13.074 7 ERROR freezer-scheduler File "/var/lib/kolla/venv/lib/python3.8/site-packages/freezer/lib/pep3143daemon/daemon.py", line 416, in detach_required
102652 2020-10-27 12:51:13.074 7 ERROR freezer-scheduler if parent_is_inet() or parent_is_init():
102653 2020-10-27 12:51:13.074 7 ERROR freezer-scheduler File "/var/lib/kolla/venv/lib/python3.8/site-packages/freezer/lib/pep3143daemon/daemon.py", line 394, in parent_is_inet
102654 2020-10-27 12:51:13.074 7 ERROR freezer-scheduler sock = socket.fromfd(
102655 2020-10-27 12:51:13.074 7 ERROR freezer-scheduler File "/usr/lib/python3.8/socket.py", line 544, in fromfd
102656 2020-10-27 12:51:13.074 7 ERROR freezer-scheduler return socket(family, type, proto, nfd)
102657 2020-10-27 12:51:13.074 7 ERROR freezer-scheduler File "/usr/lib/python3.8/socket.py", line 231, in __init__
102658 2020-10-27 12:51:13.074 7 ERROR freezer-scheduler _socket.socket.__init__(self, family, type, proto, fileno)
102659 2020-10-27 12:51:13.074 7 ERROR freezer-scheduler OSError: [Errno 88] Socket operation on non-socket
102660 2020-10-27 12:51:13.074 7 ERROR freezer-scheduler

[joel waddell (maasmaaster)]

running a 3 control 3 compute cluster in virtual machines, but I have been able to reproduce this issue running on baremetal. When running ubuntu containers, the freezer_scheduler container is always stuck in restart loop.

[Radosław Piliszek (yoctozepto)]

This is master/victoria.

[Radosław Piliszek (yoctozepto)]

Will test.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/759933

[Radosław Piliszek (yoctozepto)]

So, basically, the issue is that Freezer does not seem to work under Python 3.8.

[Mark Goddard (mgoddard)]

Not a kolla bug then - retarget to freezer?

[Radosław Piliszek (yoctozepto)]

Checking scope and will likely retarget.

[Radosław Piliszek (yoctozepto)]

Work:
CentOS 8 - py36
Debian 10 - py37

Not work:
Ubuntu Focal - py38

Either platform or python to blame here.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Radosław Piliszek (<email address hidden>) on branch: master
Review: https://review.opendev.org/759933

[Boris Lukashev (rageltman)]

This is still an issue - freezer_scheduler container is restarting incessantly and the logs are flooded with:
```
2021-08-03 03:49:53.247 11 CRITICAL freezer-scheduler [-] Unhandled error: OSError: [Errno 88] Socket operation on non-socket
2021-08-03 03:49:53.247 11 ERROR freezer-scheduler Traceback (most recent call last):
2021-08-03 03:49:53.247 11 ERROR freezer-scheduler File "/var/lib/kolla/venv/bin/freezer-scheduler", line 8, in <module>
2021-08-03 03:49:53.247 11 ERROR freezer-scheduler sys.exit(main())
2021-08-03 03:49:53.247 11 ERROR freezer-scheduler File "/var/lib/kolla/venv/lib/python3.8/site-packages/freezer/scheduler/freezer_scheduler.py", line 255, in main
2021-08-03 03:49:53.247 11 ERROR freezer-scheduler daemon.start()
2021-08-03 03:49:53.247 11 ERROR freezer-scheduler File "/var/lib/kolla/venv/lib/python3.8/site-packages/freezer/scheduler/daemon.py", line 178, in start
2021-08-03 03:49:53.247 11 ERROR freezer-scheduler with DaemonContext(pidfile=pidfile, signal_map=self.signal_map,
2021-08-03 03:49:53.247 11 ERROR freezer-scheduler File "/var/lib/kolla/venv/lib/python3.8/site-packages/freezer/lib/pep3143daemon/daemon.py", line 132, in __init__
2021-08-03 03:49:53.247 11 ERROR freezer-scheduler self.detach_process = detach_required()
2021-08-03 03:49:53.247 11 ERROR freezer-scheduler File "/var/lib/kolla/venv/lib/python3.8/site-packages/freezer/lib/pep3143daemon/daemon.py", line 416, in detach_required
2021-08-03 03:49:53.247 11 ERROR freezer-scheduler if parent_is_inet() or parent_is_init():
2021-08-03 03:49:53.247 11 ERROR freezer-scheduler File "/var/lib/kolla/venv/lib/python3.8/site-packages/freezer/lib/pep3143daemon/daemon.py", line 394, in parent_is_inet
2021-08-03 03:49:53.247 11 ERROR freezer-scheduler sock = socket.fromfd(
2021-08-03 03:49:53.247 11 ERROR freezer-scheduler File "/usr/lib/python3.8/socket.py", line 544, in fromfd
2021-08-03 03:49:53.247 11 ERROR freezer-scheduler return socket(family, type, proto, nfd)
2021-08-03 03:49:53.247 11 ERROR freezer-scheduler File "/usr/lib/python3.8/socket.py", line 231, in __init__
2021-08-03 03:49:53.247 11 ERROR freezer-scheduler _socket.socket.__init__(self, family, type, proto, fileno)
2021-08-03 03:49:53.247 11 ERROR freezer-scheduler OSError: [Errno 88] Socket operation on non-socket
2021-08-03 03:49:53.247 11 ERROR freezer-scheduler

```
while syslog is full of
```
level=error msg="copy shim log" error="read /proc/self/fd/439: file already closed"
```
from `containerd`

[Radosław Piliszek (yoctozepto)]

I am afraid Freezer is a dead project. I have seen no issue getting resolved neither here nor on the mailing list.

[Pierre Riteau (priteau)]

@rageltman: What version were you running in your latest test? This may fix it: https://review.opendev.org/c/openstack/freezer/+/795715

[Boris Lukashev (rageltman)]

Thanks for the updates gentlemen. Coming back into OpenStack world after a few years out of it, this is really sad to see - there's a much greater need for a robust FOSS private cloud ecosystem and so much talent has been snapped-up under contractual terms forbidding contribution.
Could anyone point me to some decent documentation on how one applies hot-patches to kolla container images? Thanks again