Brief Description
-----------------
Failed to unlock controller-1 after system bootup.
Severity
--------
Minor
Steps to Reproduce
------------------
1. Install a ubuntu 16.04
2. Star install duplex aio controller-0 in virtualized environment.
3. Install and config controller-0
4. When controller-0 is in available mode, run virsh start duplex-controller-1 from ubuntu host
5. controller-1 start to install and done
6. Make config in controller-0 and unlock controller-1
7. controller-1 failed to unlock.
Expected Behavior
------------------
controller-1 come into available
Actual Behavior
----------------
kubelet failed to start
Reproducibility
---------------
Reproducible
System Configuration
--------------------
duplex aio in virtualized environment
Branch/Pull Time/Commit
-----------------------
2020-07-30 UTC+8
Last Pass
---------
No
Timestamp/Logs
--------------
/var/log/daemon.log
2020-07-30T09:36:53.199 controller-1 systemd[1]: notice Unit kubelet.service entered failed state.
2020-07-30T09:36:53.199 controller-1 systemd[1]: warning kubelet.service failed.
2020-07-30T09:37:03.272 controller-1 systemd[1]: info kubelet.service holdoff time over, scheduling restart.
2020-07-30T09:37:03.273 controller-1 systemd[1]: warning Cannot add dependency job for unit dev-hugepages.mount, ignoring: Unit is masked.
2020-07-30T09:37:03.274 controller-1 systemd[1]: info Stopped Kubernetes Kubelet Server.
2020-07-30T09:37:03.287 controller-1 systemd[1]: info Starting Kubernetes Kubelet Server...
2020-07-30T09:37:03.000 controller-1 root: info /usr/bin/kubelet-cgroup-setup.sh(95318): Nothing to do, already configured: /sys/fs/cgroup/pids/k8s-infra.
2020-07-30T09:37:03.305 controller-1 systemd[1]: info Started Kubernetes Kubelet Server.
2020-07-30T09:37:03.522 controller-1 kubelet[95323]: info Flag --feature-gates has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
2020-07-30T09:37:03.522 controller-1 kubelet[95323]: info Flag --cpu-manager-policy has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
2020-07-30T09:37:03.522 controller-1 kubelet[95323]: info Flag --topology-manager-policy has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
2020-07-30T09:37:03.522 controller-1 kubelet[95323]: info Flag --system-reserved-cgroup has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
2020-07-30T09:37:03.522 controller-1 kubelet[95323]: info Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
2020-07-30T09:37:03.522 controller-1 kubelet[95323]: info Flag --kube-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
2020-07-30T09:37:03.523 controller-1 kubelet[95323]: info F0730 09:37:03.523269 95323 server.go:196] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory
2020-07-30T09:37:03.527 controller-1 systemd[1]: notice kubelet.service: main process exited, code=exited, status=255/n/a
2020-07-30T09:37:03.539 controller-1 systemd[1]: notice Unit kubelet.service entered failed state.
2020-07-30T09:37:03.539 controller-1 systemd[1]: warning kubelet.service failed.
/var/log/puppet/latest/puppet.log
2020-07-30T09:29:04.515 Debug: 2020-07-30 09:29:04 +0000 Scope(Drbd::Resource[drbd-dockerdistribution]): Retrieving template drbd/secondary-resource.res.erb
2020-07-30T09:29:04.519 Debug: 2020-07-30 09:29:04 +0000 template[/usr/share/puppet/modules/drbd/templates/secondary-resource.res.erb]: Bound template variables for /usr/share/puppet/modules/drbd/templates/secondary-resource.res.erb in 0.00 seconds
2020-07-30T09:29:04.523 Debug: 2020-07-30 09:29:04 +0000 template[/usr/share/puppet/modules/drbd/templates/secondary-resource.res.erb]: Interpolated template /usr/share/puppet/modules/drbd/templates/secondary-resource.res.erb in 0.00 seconds
2020-07-30T09:29:04.526 Debug: 2020-07-30 09:29:04 +0000 Scope(Drbd::Resource[drbd-cephmon]): Retrieving template drbd/header.res.erb
2020-07-30T09:29:04.530 Debug: 2020-07-30 09:29:04 +0000 template[/usr/share/puppet/modules/drbd/templates/header.res.erb]: Bound template variables for /usr/share/puppet/modules/drbd/templates/header.res.erb in 0.00 seconds
2020-07-30T09:29:04.534 Debug: 2020-07-30 09:29:04 +0000 template[/usr/share/puppet/modules/drbd/templates/header.res.erb]: Interpolated template /usr/share/puppet/modules/drbd/templates/header.res.erb in 0.00 seconds
2020-07-30T09:29:04.537 Debug: 2020-07-30 09:29:04 +0000 Scope(Drbd::Resource[drbd-cephmon]): Retrieving template drbd/primary-resource.res.erb
2020-07-30T09:29:04.541 Debug: 2020-07-30 09:29:04 +0000 template[/usr/share/puppet/modules/drbd/templates/primary-resource.res.erb]: Bound template variables for /usr/share/puppet/modules/drbd/templates/primary-resource.res.erb in 0.00 seconds
2020-07-30T09:29:04.544 Debug: 2020-07-30 09:29:04 +0000 template[/usr/share/puppet/modules/drbd/templates/primary-resource.res.erb]: Interpolated template /usr/share/puppet/modules/drbd/templates/primary-resource.res.erb in 0.00 seconds
2020-07-30T09:29:04.548 Debug: 2020-07-30 09:29:04 +0000 Scope(Drbd::Resource[drbd-cephmon]): Retrieving template drbd/secondary-resource.res.erb
2020-07-30T09:29:04.551 Debug: 2020-07-30 09:29:04 +0000 template[/usr/share/puppet/modules/drbd/templates/secondary-resource.res.erb]: Bound template variables for /usr/share/puppet/modules/drbd/templates/secondary-resource.res.erb in 0.00 seconds
2020-07-30T09:29:04.555 Debug: 2020-07-30 09:29:04 +0000 template[/usr/share/puppet/modules/drbd/templates/secondary-resource.res.erb]: Interpolated template /usr/share/puppet/modules/drbd/templates/secondary-resource.res.erb in 0.00 seconds
2020-07-30T09:29:04.558 Debug: 2020-07-30 09:29:04 +0000 Scope(Haproxy::Config[haproxy]): Retrieving template haproxy/haproxy-base.cfg.erb
2020-07-30T09:29:04.561 Debug: 2020-07-30 09:29:04 +0000 template[/usr/share/openstack-puppet/modules/haproxy/templates/haproxy-base.cfg.erb]: Bound template variables for /usr/share/openstack-puppet/modules/haproxy/templates/haproxy-base.cfg.erb in 0.00 seconds
2020-07-30T09:29:04.566 Debug: 2020-07-30 09:29:04 +0000 template[/usr/share/openstack-puppet/modules/haproxy/templates/haproxy-base.cfg.erb]: Interpolated template /usr/share/openstack-puppet/modules/haproxy/templates/haproxy-base.cfg.erb in 0.00 seconds
2020-07-30T09:29:04.570 Debug: 2020-07-30 09:29:04 +0000 importing '/usr/share/openstack-puppet/modules/haproxy/manifests/globals.pp' in environment production
2020-07-30T09:29:04.576 Debug: 2020-07-30 09:29:04 +0000 Automatically imported haproxy::globals from haproxy/globals into production
2020-07-30T09:29:04.580 Debug: 2020-07-30 09:29:04 +0000 Performing a hiera indirector lookup of haproxy::globals::sort_options_alphabetic with options {:variables=>Scope(Class[Haproxy::Globals]), :merge=>#<Puppet::Pops::FirstFoundStrategy:0x00000005a77600 @options_t=#<Puppet::Pops::Types::PStructType:0x00000005d959b8 @elements=[#<Puppet::Pops::Types::PStructElement:0x00000005d95a08 @key_type=#<Puppet::Pops::Types::POptionalType:0x00000005d95a80 @type=#<Puppet::Pops::Types::PStringType:0x00000005d95cb0 @size_type=nil, @values=["strategy"]>>, @value_type=#<Puppet::Pops::Types::POptionalType:0x00000005d95e90 @type=#<Puppet::Pops::Types::PPatternType:0x00000005d95f08 @patterns=[#<Puppet::Pops::Types::PRegexpType:0x00000005d960c0 @pattern="first", @regexp=/first/>]>>>]>, @options={}>}
2020-07-30T09:29:04.584 Debug: 2020-07-30 09:29:04 +0000 hiera(): Looking up haproxy::globals::sort_options_alphabetic in YAML backend
2020-07-30T09:29:04.588 Debug: 2020-07-30 09:29:04 +0000 hiera(): Looking for data source runtime
2020-07-30T09:29:04.592 Debug: 2020-07-30 09:29:04 +0000 hiera(): Cannot find datafile /tmp/puppet/hieradata/runtime.yaml, skipping
2020-07-30T09:29:04.595 Debug: 2020-07-30 09:29:04 +0000 hiera(): Looking for data source host
2020-07-30T09:29:04.599 Debug: 2020-07-30 09:29:04 +0000 hiera(): Looking for data source secure_system
2020-07-30T09:29:04.602 Debug: 2020-07-30 09:29:04 +0000 hiera(): Looking for data source system
2020-07-30T09:29:04.606 Debug: 2020-07-30 09:29:04 +0000 hiera(): Looking for data source secure_static
2020-07-30T09:29:04.609 Debug: 2020-07-30 09:29:04 +0000 hiera(): Looking for data source static
2020-07-30T09:29:04.612 Debug: 2020-07-30 09:29:04 +0000 hiera(): Looking for data source personality
2020-07-30T09:29:04.616 Debug: 2020-07-30 09:29:04 +0000 hiera(): Looking for data source global
2020-07-30T09:29:04.620 Debug: 2020-07-30 09:29:04 +0000 Scope(Haproxy::Frontend[docker-registry]): Retrieving template haproxy/haproxy_frontend_block.erb
2020-07-30T09:29:04.623 Debug: 2020-07-30 09:29:04 +0000 template[/usr/share/openstack-puppet/modules/haproxy/templates/haproxy_frontend_block.erb]: Bound template variables for /usr/share/openstack-puppet/modules/haproxy/templates/haproxy_frontend_block.erb in 0.00 seconds
2020-07-30T09:29:04.626 Debug: 2020-07-30 09:29:04 +0000 Scope(Haproxy::Frontend[docker-registry]): Retrieving template haproxy/fragments/_bind.erb
2020-07-30T09:29:04.630 Debug: 2020-07-30 09:29:04 +0000 template[/usr/share/openstack-puppet/modules/haproxy/templates/fragments/_bind.erb]: Bound template variables for /usr/share/openstack-puppet/modules/haproxy/templates/fragments/_bind.erb in 0.00 seconds
2020-07-30T09:29:04.635 Debug: 2020-07-30 09:29:04 +0000 template[/usr/share/openstack-puppet/modules/haproxy/templates/fragments/_bind.erb]: Interpolated template /usr/share/openstack-puppet/modules/haproxy/templates/fragments/_bind.erb in 0.00 seconds
2020-07-30T09:29:04.640 Debug: 2020-07-30 09:29:04 +0000 Scope(Haproxy::Frontend[docker-registry]): Retrieving template haproxy/fragments/_mode.erb
2020-07-30T09:29:04.644 Debug: 2020-07-30 09:29:04 +0000 template[/usr/share/openstack-puppet/modules/haproxy/templates/fragments/_mode.erb]: Bound template variables for /usr/share/openstack-puppet/modules/haproxy/templates/fragments/_mode.erb in 0.00 seconds
2020-07-30T09:29:04.649 Debug: 2020-07-30 09:29:04 +0000 template[/usr/share/openstack-puppet/modules/haproxy/templates/fragments/_mode.erb]: Interpolated template /usr/share/openstack-puppet/modules/haproxy/templates/fragments/_mode.erb in 0.00 seconds
2020-07-30T09:29:04.653 Debug: 2020-07-30 09:29:04 +0000 Scope(Haproxy::Frontend[docker-registry]): Retrieving template haproxy/fragments/_options.erb
2020-07-30T09:29:04.658 Debug: 2020-07-30 09:29:04 +0000 template[/usr/share/openstack-puppet/modules/haproxy/templates/fragments/_options.erb]: Bound template variables for /usr/share/openstack-puppet/modules/haproxy/templates/fragments/_options.erb in 0.00 seconds
2020-07-30T09:29:04.663 Debug: 2020-07-30 09:29:04 +0000 template[/usr/share/openstack-puppet/modules/haproxy/templates/fragments/_options.erb]: Interpolated template /usr/share/openstack-puppet/modules/haproxy/templates/fragments/_options.erb in 0.00 seconds
2020-07-30T09:29:04.667 Debug: 2020-07-30 09:29:04 +0000 template[/usr/share/openstack-puppet/modules/haproxy/templates/haproxy_frontend_block.erb]: Interpolated template /usr/share/openstack-puppet/modules/haproxy/templates/haproxy_frontend_block.erb in 0.00 seconds
2020-07-30T09:29:04.671 Debug: 2020-07-30 09:29:04 +0000 importing '/usr/share/openstack-puppet/modules/haproxy/manifests/listen.pp' in environment production
Test Activity
-------------
Evaluation
Workaround
----------
no workaround yet
Closing as stx.3.0 is EOL as of Dec 2020