Comment 5 for bug 1901449

A runtime certificate update restarts containerd. Due to systemd depeendencies docker is restarted which triggers ceph-preshutdown.sh that unmounts the RBD devices.

2020-10-21T12:47:46.537 Info: 2020-10-21 12:47:46 +0000 /Stage[pre]/Platform::Config::Certs::Ssl_ca/File[create-ssl-ca-cert]: Filebucketed /etc/pki/ca-trust/source/anchors/ca-cert.pem to puppet with sum ed886921b19e510f522bb5005cf5a4c2
2020-10-21T12:47:46.539 Notice: 2020-10-21 12:47:46 +0000 /Stage[pre]/Platform::Config::Certs::Ssl_ca/File[create-ssl-ca-cert]/content: content changed '{md5}ed886921b19e510f522bb5005cf5a4c2' to '{md5}b30884f73776aab90df0fe8a831c3b44'
2020-10-21T12:47:46.542 Info: 2020-10-21 12:47:46 +0000 /Stage[pre]/Platform::Config::Certs::Ssl_ca/File[create-ssl-ca-cert]: Scheduling refresh of Exec[update-ca-trust ]
2020-10-21T12:47:46.544 Info: 2020-10-21 12:47:46 +0000 /Stage[pre]/Platform::Config::Certs::Ssl_ca/File[create-ssl-ca-cert]: Scheduling refresh of Exec[restart containerd]
2020-10-21T12:47:46.547 Debug: 2020-10-21 12:47:46 +0000 /Stage[pre]/Platform::Config::Certs::Ssl_ca/File[create-ssl-ca-cert]: The container Class[Platform::Config::Certs::Ssl_ca] will propagate my refresh event
2020-10-21T12:47:46.549 Debug: 2020-10-21 12:47:46 +0000 Exec[update-ca-trust ](provider=posix): Executing 'update-ca-trust'
2020-10-21T12:47:46.557 Debug: 2020-10-21 12:47:46 +0000 Executing: 'update-ca-trust'
2020-10-21T12:47:47.037 Notice: 2020-10-21 12:47:47 +0000 /Stage[pre]/Platform::Config::Certs::Ssl_ca/Exec[update-ca-trust ]: Triggered 'refresh' from 1 events
2020-10-21T12:47:47.039 Debug: 2020-10-21 12:47:47 +0000 /Stage[pre]/Platform::Config::Certs::Ssl_ca/Exec[update-ca-trust ]: The container Class[Platform::Config::Certs::Ssl_ca] will propagate my refresh event
2020-10-21T12:47:47.042 Debug: 2020-10-21 12:47:47 +0000 Exec[restart containerd](provider=posix): Executing 'pmon-restart containerd'
2020-10-21T12:47:47.044 Debug: 2020-10-21 12:47:47 +0000 Executing: 'pmon-restart containerd'
2020-10-21T12:47:47.072 [10651.00109] controller-0 pmond mon pmonMsg.cpp ( 701) pmon_service_inbox : Info : containerd process-restart ; by request
2020-10-21T12:47:48.072 [10651.00110] controller-0 pmond mon pmonHdlr.cpp (1107) unregister_process : Info : containerd Unregister (2438)
2020-10-21T12:47:48.072 [10651.00111] controller-0 pmond mon pmonHdlr.cpp ( 946) kill_running_process : Warn : containerd Killed (2438)
2020-10-21T12:47:48.072 [10651.00112] controller-0 pmond mon pmonHdlr.cpp (1311) respawn_process : Info : containerd Spawn (661307)

2020-10-21T12:47:48.077 controller-0 systemd[1]: info Stopping Docker Application Container Engine...

2020-10-21T12:47:48.086 [10651.00113] controller-0 pmond mon pmonHdlr.cpp ( 303) manage_process_failure :Error : dockerd failed (2473) (p:1 a:0)

2020-10-21T12:47:48.000 controller-0 ceph-preshutdown.sh: notice Unmapped /dev/rbd0
2020-10-21T12:47:48.000 controller-0 ceph-preshutdown.sh: notice Unmapped /dev/rbd1
2020-10-21T12:47:48.000 controller-0 ceph-preshutdown.sh: notice Unmounted /dev/rbd0
2020-10-21T12:47:48.000 controller-0 ceph-preshutdown.sh: notice Unmounted /dev/rbd1
2020-10-21T12:47:48.000 controller-0 ceph-preshutdown.sh: notice Unmounting /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/kube-rbd-image-kubernetes-dynamic-pvc-4ee653e1-1378-11eb-b4ec-4ebf037698b7
2020-10-21T12:47:48.000 controller-0 ceph-preshutdown.sh: notice Unmounting /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/kube-rbd-image-kubernetes-dynamic-pvc-8dee859f-1378-11eb-b4ec-4ebf037698b7
2020-10-21T12:47:48.000 controller-0 ceph-preshutdown.sh: notice Unmounting /var/lib/kubelet/pods/08b16390-834b-4681-af2f-5f9bbd4ea259/volumes/kubernetes.io~rbd/pvc-2278fda0-759b-4d48-bb14-fa8f6a620069
2020-10-21T12:47:48.000 controller-0 ceph-preshutdown.sh: notice Unmounting /var/lib/kubelet/pods/e0169287-cae6-416b-9c24-33975beaec72/volumes/kubernetes.io~rbd/pvc-1bbcce8b-8658-436a-80e6-24d36c677541

2020-10-21T12:47:48.102 [10651.00114] controller-0 pmond com nodeUtil.cpp (1899) get_system_state : Info : systemctl reports host in 'degraded' state (0)
2020-10-21T12:47:48.102 [10651.00115] controller-0 pmond mon pmonHdlr.cpp (1547) manage_alarm : Info : dockerd process has failed ; Auto recovery in progress.
2020-10-21T12:47:48.102 [10651.00116] controller-0 pmond mon pmonMsg.cpp ( 328) pmon_send_event : Info : controller-0 pmon log sent

2020-10-21T12:47:48.236 controller-0 kernel: err [ 3127.702215] Aborting journal on device rbd0-8.
2020-10-21T12:47:48.236 controller-0 kernel: err [ 3127.707177] Buffer I/O error on dev rbd0, logical block 491520, lost sync page write
2020-10-21T12:47:48.236 controller-0 kernel: err [ 3127.715832] JBD2: Error -5 detected when updating journal superblock for rbd0-8.
2020-10-21T12:47:48.244 controller-0 kernel: warning [ 3127.724180] EXT4-fs (rbd0): discard request in group:17 block:20517 count:1 failed with -5

2020-10-21T12:47:48.471 [10651.00117] controller-0 pmond mon pmonHdlr.cpp (1007) process_running : Info : dockerd process not running
2020-10-21T12:47:48.471 [10651.00118] controller-0 pmond mon pmonHdlr.cpp (1311) respawn_process : Info : dockerd Spawn (661452)

2020-10-21T12:47:49.237 controller-0 kernel: crit [ 3128.688532] EXT4-fs error (device rbd0): ext4_find_entry:1318: inode #131076: comm elasticsearch[m: reading directory lblock 0
2020-10-21T12:47:49.237 controller-0 kernel: crit [ 3128.701321] EXT4-fs error (device rbd0): ext4_read_inode_bitmap:163: comm elasticsearch[m: Cannot read inode bitmap - block_group = 16, inode_bitmap = 524304
2020-10-21T12:47:49.253 controller-0 kernel: crit [ 3128.717070] EXT4-fs error (device rbd0): ext4_journal_check_start:56: Detected aborted journal
2020-10-21T12:47:49.253 controller-0 kernel: crit [ 3128.724185] EXT4-fs (rbd0): Remounting filesystem read-only
2020-10-21T12:47:49.253 controller-0 kernel: warning [ 3128.724843] EXT4-fs warning (device rbd0): __ext4_read_dirblock:903: error reading directory block (ino 131076, block 0)

2020-10-21T12:47:53.073 [10651.00119] controller-0 pmond mon pmonFsm.cpp ( 616) pmon_passive_handler : Info : containerd Restarted (661449)
2020-10-21T12:47:53.073 [10651.00120] controller-0 pmond mon pmonHdlr.cpp (1142) register_process : Info : containerd Registered (661449)
2020-10-21T12:47:53.472 [10651.00121] controller-0 pmond mon pmonFsm.cpp ( 624) pmon_passive_handler : Info : dockerd Monitor (661499)

2020-10-21T12:48:04.543 controller-0 kernel: warning [ 3144.021105] EXT4-fs warning (device rbd1): __ext4_read_dirblock:903: error reading directory block (ino 8651014, block 0)
2020-10-21T12:48:14.543 controller-0 kernel: warning [ 3154.020465] EXT4-fs warning (device rbd1): __ext4_read_dirblock:903: error reading directory block (ino 8651014, block 0)

2020-10-21T12:48:17.072 [10651.00122] controller-0 pmond mon pmonFsm.cpp ( 659) pmon_passive_handler : Info : dockerd Stable (661499)
2020-10-21T12:48:17.572 [10651.00123] controller-0 pmond mon pmonFsm.cpp ( 731) pmon_passive_handler : Info : dockerd Recovered (661499)
2020-10-21T12:48:17.572 [10651.00124] controller-0 pmond mon pmonHdlr.cpp (1142) register_process : Info : dockerd Registered (661499)

2020-10-21T12:48:24.543 controller-0 kernel: warning [ 3164.019870] EXT4-fs warning (device rbd1): __ext4_read_dirblock:903: error reading directory block (ino 8651014, block 0)
2020-10-21T12:48:34.543 controller-0 kernel: warning [ 3174.019415] EXT4-fs warning (device rbd1): __ext4_read_dirblock:903: error reading directory block (ino 8651014, block 0)
2020-10-21T12:48:44.543 controller-0 kernel: warning [ 3184.018766] EXT4-fs warning (device rbd1): __ext4_read_dirblock:903: error reading directory block (ino 8651014, block 0)
2020-10-21T12:48:54.543 controller-0 kernel: warning [ 3194.018375] EXT4-fs warning (device rbd1): __ext4_read_dirblock:903: error reading directory block (ino 8651014, block 0)
2020-10-21T12:49:04.543 controller-0 kernel: warning [ 3204.017813] EXT4-fs warning (device rbd1): __ext4_read_dirblock:903: error reading directory block (ino 8651014, block 0)
2020-10-21T12:49:14.544 controller-0 kernel: warning [ 3214.017395] EXT4-fs warning (device rbd1): __ext4_read_dirblock:903: error reading directory block (ino 8651014, block 0)
2020-10-21T12:49:24.544 controller-0 kernel: warning [ 3224.016965] EXT4-fs warning (device rbd1): __ext4_read_dirblock:903: error reading directory block (ino 8651014, block 0)
2020-10-21T12:49:34.544 controller-0 kernel: warning [ 3234.016326] EXT4-fs warning (device rbd1): __ext4_read_dirblock:903: error reading directory block (ino 8651014, block 0)