util.py[WARNING]: Failed to disable password for user centos

[Anil Shashikumar Belur]

We are running into below errors (from /var/log/cloud-init-outputs.log) with cloud-init on centos7 before it could get to run the initialization test script passed as user_data, while bring up openstack instance using simple heat template.

Here is the link to the heat template used also has the user_data:
https://github.com/opendaylight/releng-builder/blob/master/openstack-hot/generic-server.yaml

The script is copied onto `/var/lib/cloud/instance/scripts/part-001`

    Cloud-init v. 0.7.5 running 'init-local' at Thu, 18 May 2017 23:00:31 +0000. Up 8.39 seconds.
    Cloud-init v. 0.7.5 running 'init' at Thu, 18 May 2017 23:00:52 +0000. Up 28.93 seconds.
    ....
    2017-05-18 23:00:53,931 - util.py[WARNING]: Failed forking and calling callback NoneType
    2017-05-18 23:00:54,059 - util.py[WARNING]: Failed to disable password for user centos
    2017-05-18 23:00:54,062 - util.py[WARNING]: Running users-groups (<module 'cloudinit.config.cc_users_groups' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_users_groups.pyc'>) failed

However, the user_data script never gets executed, which fails on running the initialization script.

Here is the [code snippet](https://github.com/number5/cloud-init/blob/master/cloudinit/distros/__init__.py#L545-L561) which is causing the failure. What is causing this issue ?

Thanks.

[Anil Shashikumar Belur (askb23)]

 cat /var/log/cloud-init.log
May 18 04:36:25 centos-7---java-builder---20170517-2244 cloud-init: Cloud-init v. 0.7.5 running 'init-local' at Thu, 18 May 2017 04:36:25 +0000. Up 8.38 seconds.
May 18 04:36:46 centos-7---java-builder---20170517-2244 cloud-init: Cloud-init v. 0.7.5 running 'init' at Thu, 18 May 2017 04:36:46 +0000. Up 28.95 seconds.
May 18 04:36:46 centos-7---java-builder---20170517-2244 cloud-init: ci-info: ++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++
May 18 04:36:46 centos-7---java-builder---20170517-2244 cloud-init: ci-info: +--------+------+--------------+---------------+-------------------+
May 18 04:36:46 centos-7---java-builder---20170517-2244 cloud-init: ci-info: | Device | Up | Address | Mask | Hw-Address |
May 18 04:36:46 centos-7---java-builder---20170517-2244 cloud-init: ci-info: +--------+------+--------------+---------------+-------------------+
May 18 04:36:46 centos-7---java-builder---20170517-2244 cloud-init: ci-info: | lo: | True | 127.0.0.1 | 255.0.0.0 | . |
May 18 04:36:46 centos-7---java-builder---20170517-2244 cloud-init: ci-info: | eth0: | True | 10.29.12.230 | 255.255.252.0 | bc:76:4e:04:92:56 |
May 18 04:36:46 centos-7---java-builder---20170517-2244 cloud-init: ci-info: +--------+------+--------------+---------------+-------------------+
May 18 04:36:46 centos-7---java-builder---20170517-2244 cloud-init: ci-info: +++++++++++++++++++++++++++++++Route info+++++++++++++++++++++++++++++++
May 18 04:36:46 centos-7---java-builder---20170517-2244 cloud-init: ci-info: +-------+-------------+------------+---------------+-----------+-------+
May 18 04:36:46 centos-7---java-builder---20170517-2244 cloud-init: ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
May 18 04:36:46 centos-7---java-builder---20170517-2244 cloud-init: ci-info: +-------+-------------+------------+---------------+-----------+-------+
May 18 04:36:46 centos-7---java-builder---20170517-2244 cloud-init: ci-info: | 0 | 0.0.0.0 | 10.29.12.1 | 0.0.0.0 | eth0 | UG |
May 18 04:36:46 centos-7---java-builder---20170517-2244 cloud-init: ci-info: | 1 | 10.29.12.0 | 0.0.0.0 | 255.255.252.0 | eth0 | U |
May 18 04:36:46 centos-7---java-builder---20170517-2244 cloud-init: ci-info: +-------+-------------+------------+---------------+-----------+-------+
May 18 04:36:47 centos-7---java-builder---20170517-2244 cloud-init: 2017-05-18 04:36:47,944 - util.py[WARNING]: Failed forking and calling callback NoneType
May 18 04:36:48 centos-7---java-builder---20170517-2244 cloud-init: 2017-05-18 04:36:48,071 - util.py[WARNING]: Failed to disable password for user centos
May 18 04:36:48 centos-7---java-builder---20170517-2244 cloud-init: 2017-05-18 04:36:48,074 - util.py[WARNING]: Running users-groups (<module 'cloudinit.config.cc_users_groups' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_users_groups.pyc'>) failed

[Anil Shashikumar Belur (askb23)]

I built new rpm with version 0.7.9 and tested it with centos7. Attached cloud-init*.log. Also noticed that the user data script does not get coped under /var/lib/cloud/scripts/*

[Anil Shashikumar Belur (askb23)]

The new rpm built for 0.7.9 on CentOS7 is available here:
https://copr.fedorainfracloud.org/coprs/askb23/cloud-init/repo/epel-7/askb23-cloud-init-epel-7.repo

# uname -a
Linux sandbox-32442-4-java-builder-0 3.10.0-514.16.1.el7.x86_64 #1 SMP Wed Apr 12 15:04:24 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/redhat-release
CentOS Linux release 7.3.1611 (Core)

rpm -qa | grep cloud-init
cloud-init-0.7.9-1.el7.centos.noarch

[Anil Shashikumar Belur (askb23)]

Tested with latest src (cloud-init-0.7.9+148.g9e01cca-1.el7.centos.noarch) from cloud-init master. Issue is reproducible.

[Joshua Powers (powersj)]

Anil,

Because you mentioned that the initial deploy fails I figured it was worth taking a look at the heat template. Given I have no prior experience working with heat templates I decided to see if there was a way to validate the template itself. This was my result:

root@a:~# ./validate-templates .
Got error validating ./generic-server.yaml , Missing value auth-url required for auth plugin password

I got the validate-templates scripts from:
https://raw.githubusercontent.com/openstack/heat-templates/master/tools/validate-templates

Given it is complaining about missing parameter for the auth plugin it makes me suspect given the errors you are seeing with cloud-init.

Has this heat template ever worked?

[Joshua Powers (powersj)]

This may also be as simple as:
https://ask.openstack.org/en/question/98041/missing-value-auth-url-required-for-auth-plugin-password/

If you could run the validate-templates command with your OS_AUTH_URL that would be good to.

[Anil Shashikumar Belur (askb23)]

This error "Missing value auth-url required for auth plugin password" most likely seen because the auth url has to be sourced through 1. ~/.config/openstack/clouds.yaml or 2. RC file or 3. an env variable. The template validate script does not show any errors for us since its able to resolve the auth url value.

As discussed on IRC, will dig a bit deeper into differences in the dependencies on both images and provide an update.

[Anil Shashikumar Belur (askb23)]

Updating the cloud-init.log for the rpm built from cloud-init master branch as requested smoser.

[Anil Shashikumar Belur (askb23)]

smoser: As requested, now I have uploaded the logs, dmesg and audi.log for the run with 0.7.5-10.

0.7.5_logs/
0.7.5_logs/cloud-init.log
0.7.5_logs/cloud-init-output.log
0.7.5_logs/dmesg.txt
0.7.5_logs/audit.log

[Anil Shashikumar Belur (askb23)]

smoser: As requested, now I have uploaded the logs, dmesg and audi.log for the run with 0.7.5-10.

0.7.5_logs/
0.7.5_logs/cloud-init.log
0.7.5_logs/cloud-init-output.log
0.7.5_logs/dmesg.txt
0.7.5_logs/audit.log

[Scott Moser (smoser)]

For ease of viewing, from Anil's logs:

$ grep passwd audit.log
type=AVC msg=audit(1495768802.146:100): avc: denied { create } for pid=2487 comm="passwd" name="shadow+" scontext=system_u:system_r:passwd_t:s0 tcontext=system_u:object_r:etc_t:s0 tclass=file
type=SYSCALL msg=audit(1495768802.146:100): arch=c000003e syscall=2 success=no exit=-13 a0=7fd8a86ecb80 a1=c2 a2=0 a3=2 items=0 ppid=2406 pid=2487 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="passwd" exe="/usr/bin/passwd" subj=system_u:system_r:passwd_t:s0 key=(null)
type=USER_CHAUTHTOK msg=audit(1495768802.147:101): pid=2487 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:passwd_t:s0 msg='op=lock password id=1000 exe="/usr/bin/passwd" hostname=? addr=? terminal=? res=failed'

[Lars Kellogg-Stedman (larsks)]

Anil: I suspect from the errors that you are seeing that this is not a cloud-init bug but is instead due to a problem with the image you are using. Is there any chance you can share that image so that we can take a closer look?

[Anil Shashikumar Belur (askb23)]

Scott, Below output returned by `/var/lib/cloud/data/status.json` which provides similar information which could potentially be the issue with the selinux permissions.

# cat /var/lib/cloud/data/status.jso

{
 "v1": {
  "init": {
   "start": 1496116240.008911,
   "finished": 1496116241.228474,
   "errors": [
    "('users-groups', ProcessExecutionError('Unexpected error while running command.\\nCommand: [\\'passwd\\', \\'-l\\', \\'centos\\']\\nExit code: 255\\nReason: -\\nStdout: \\'Locking password for user centos.\\\\npasswd: Error (password not set?)\\\\n\\'\\nStderr: \"passwd: Libuser error at line: 124 - error creating `/etc/shadow+\\': Permission denied.\\\\n\"',))"
   ],
   "end": null
  },
  "datasource": "DataSourceNone",
  "modules-config": {
   "start": null,
   "errors": [],
   "end": null
  },
  "modules-final": {
   "start": null,
   "errors": [],
   "end": null
  },
  "init-local": {
   "start": 1496116219.609383,
   "finished": 1496116219.764872,
   "errors": [],
   "end": null
  },
  "stage": null
 }
}

From this we can see that /etc/shadow is set with incorrect permissions set during instance boot up.

# ls -Z /etc/shadow

----------. root root system_u:object_r:etc_t:s0 /etc/shadow

The images we use on Rackspace can be accessed only our private keys, I am afraid, idk how to get around that to get access into the instances.

I have also tried setting selinux to "disabled" or "permissive" on the instance which still does not work with cloud-init, tells me we might be missing something here. Would it be useful to look at the logs with selinux disabled ?

[Scott Moser (smoser)]

Anil,

As Lars pointed out, it looks like at this point that your image is built improperly.

It seems that this is the only issue that cloud-init reported, so you can probably manage to use user-scripts ('#!/bin/sh') through user-data or some other mechanism to work around this.

That said, I suggest you contact whoever provided you with the image and tell them that something is not created right.

It would seem that simply running 'passwd -l USERNAME' would recreate the issue outside of cloud-init.

[Launchpad Janitor (janitor)]

[Expired for cloud-init because there has been no activity for 60 days.]

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2899