bindmounting /dev in containers might break /dev/pts on the host
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tripleo |
Fix Released
|
High
|
Cédric Jeanneret |
Bug Description
Related: https:/
Description of problem:
When forking a pty as regular user, we get a failure to read/write on /dev/ptmx since it's just a symlink to /dev/pts/ptmx.
In fedora, /dev/ptmx is its own node.
[1] centos 9
[2] fedora
[3] strace
[4] reproducer script
After doing further investivation, Openshift product has seen a similar issue in Bug 1950408 that resulted in this article [a] and this doc change [b]
Since this issues starts to happen only after the undercloud installation is completed, we can presume that some openstack containers are having bad mounts [5]
In Bugzilla 1950408 comment 18 [c], it's recommended that, if we need to bind mount /dev, we should also add a mount type=devpts,
It appears like tripleo doesn't have the hability to pass mounttypes.
[a] https:/
[b] https:/
[c] https:/
Steps to Reproduce:
1. Run script [4]
Workaround:
rm /dev/ptmx
mknod -m 666 /dev/ptmx c 5 2
or
chmod 666 /dev/pts/ptmx
Additional info:
[1]
~~~
(undercloud) [stack@undercloud-0 ~]$ ls -tlra /dev/ptmx
lrwxrwxrwx. 1 root root 8 Nov 7 12:34 /dev/ptmx -> pts/ptmx
~~~
[2]
~~~
[dvd@fedora ~]$ ls -tlra /dev/ptmx
crw-rw-rw-. 1 root tty 5, 2 Nov 8 09:39 /dev/ptmx
[dvd@fedora ~]$ ls -tlra /dev/pts/ptmx
c---------. 1 root root 5, 2 Oct 25 16:50 /dev/pts/ptmx
~~~
[3]
~~~
14:48:20.542970 openat(AT_FDCWD, "/dev/ptmx", O_RDWR) = -1 EACCES (Permission denied)
14:48:20.543061 write(2, "Traceback (most recent call last):\n", 35Traceback (most recent call last):
) = 35
14:48:20.543116 write(2, " File \"/home/
) = 53
14:48:20.543168 openat(AT_FDCWD, "/home/
14:48:20.543216 newfstatat(3, "", {st_mode=
14:48:20.543266 ioctl(3, TCGETS, 0x7ffe22bd1f30) = -1 ENOTTY (Inappropriate ioctl for device)
14:48:20.543310 lseek(3, 0, SEEK_CUR) = 0
14:48:20.543361 fcntl(3, F_DUPFD_CLOEXEC, 0) = 4
14:48:20.543403 fcntl(4, F_GETFL) = 0x8000 (flags O_RDONLY|
14:48:20.543445 newfstatat(4, "", {st_mode=
14:48:20.543493 read(4, "#!/usr/bin/env python\n# Python program to explain os.openpty() method \n \n# importing os module \nimport os\n \n \n# open new pseudo-terminal pair\n# using os.openpty() method\nmaster, slave = os.openpty()\n \n \n# Get the terminal device\n# name associated with\n# file descriptor master \nname = os.ttyname(
14:48:20.543542 close(4) = 0
14:48:20.543584 lseek(3, 0, SEEK_SET) = 0
14:48:20.543636 read(3, "#!/usr/bin/env python\n# Python program to explain os.openpty() method \n \n# importing os module \nimport os\n \n \n# open new pseudo-terminal pair\n# using os.openpty() method\nmaster, slave = os.openpty()\n \n \n# Get the terminal device\n# name associated with\n# file descriptor master \nname = os.ttyname(
14:48:20.543689 close(3) = 0
14:48:20.543735 write(2, " master, slave = os.openpty()\n", 33 master, slave = os.openpty()
) = 33
14:48:20.543794 write(2, "PermissionError: [Errno 13] Permission denied\n", 46PermissionError: [Errno 13] Permission denied
) = 46
~~~
[4]
~~~
#!/usr/bin/env python
import os
master, slave = os.openpty()
~~~
[5]
~~~
[root@undercloud-0 ~]# podman ps -q | while read l;do echo "$l";podman inspect $l | jq -r '.[].Mounts[] | [.Source, .Destination] | @tsv' | grep -P "/dev[^\/]";done
229b0319bb29
560fa91d2da3
82bccf849215
fe0cdd41acb8
/dev /dev/
c7c11279eacb
7f842c077598
c22873137bfe
2ad7c661e2c0
8a703f6f40f9
0546bb29d745
88df372ab2af
8100522a45d4
/dev /dev
d4725a57676b
9940f74ff548
63a170016303
6d978a2fc69a
0b7824b15fe9
[root@undercloud-0 ~]# podman ps | grep -P "fe0cdd41acb8|
fe0cdd41acb8 undercloud-
8100522a45d4 undercloud-
~~~
Changed in tripleo: | |
status: | New → In Progress |
I think We are also seeing similar issue on CS9 multinode jobs /logserver. rdoproject. org/26/ 34926/9/ check/periodic- tripleo- ci-centos- 9-containers- multinode- master/ 3bc334a/ logs/undercloud /home/zuul- worker/ overcloud_ deploy. log.txt. gz
https:/
``` tripleo- playbooks [00m utils.utils [-] Temporary directory [ /tmp/tripleo8h9 4vv6b ] cleaned up[00m utils.utils [-] Temporary directory [ /tmp/tripleo4c4 oe5y1 ] cleaned up[00m v1.overcloud_ deploy. DeployOvercloud [-] Exception occured while running the command: OSError: out of pty devices v1.overcloud_ deploy. DeployOvercloud Traceback (most recent call last): v1.overcloud_ deploy. DeployOvercloud File "/usr/lib/ python3. 9/site- packages/ tripleoclient/ command. py", line 34, in run v1.overcloud_ deploy. DeployOvercloud super(Command, self).run( parsed_ args) v1.overcloud_ deploy. DeployOvercloud File "/usr/lib/ python3. 9/site- packages/ osc_lib/ command/ command. py", line 39, in run v1.overcloud_ deploy. DeployOvercloud return super(Command, self).run( parsed_ args) v1.overcloud_ deploy. DeployOvercloud File "/usr/lib/ python3. 9/site- packages/ cliff/command. py", line 186, in run v1.overcloud_ deploy. DeployOvercloud return_code = self.take_ action( parsed_ args) or 0 v1.overcloud_ deploy. DeployOvercloud File "/usr/lib/ python3. 9/site- packages/ tripleoclient/ v1/overcloud_ deploy. py", line 1132, in take_action v1.overcloud_ deploy. DeployOvercloud created_env_files = self.create_ env_files( v1.overcloud_ deploy. DeployOvercloud File "/usr/lib/ python3. 9/site- packages/ tripleoclient/ v1/overcloud_ deploy. py", line 304, in create_env_files v1.overcloud_ deploy. DeployOvercloud self._provision _networks( parsed_ args, new_tht_root, v1.overcloud_ deploy. DeployOvercloud File "/usr/lib/ python3. 9/site- packages/ tripleoclient/ v1/overcloud_ deploy. py", line 555, in _provision_networks v1.overcloud_ deploy. DeployOvercloud utils.run_ ansible_ playbook( v1.overcloud_ deploy. DeployOvercloud File "/usr/lib/ python3. ..
/ansible/
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 INFO tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 INFO tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.
2021-11-08 07:18:47 | 2021-11-08 07:18:47.814 100991 ERROR tripleoclient.