When adding ceilometer to the pipeline, swift-proxy cannot start

[David Hill]

When adding ceilometer to the pipeline, swift-proxy cannot start as it cannot read /etc/ceilometer/ceilometer.conf. The solution is to chown/chgrp/chmod 644 /etc/ceilometer/ceilometer.conf and restart swift-proxy. The problem I have with this solution is that if we need to have another service reading /etc/ceilometer/ceilometer.conf, it wont be possible to do that.

IE: Nova, Swift and Ceilometer trying to access that file? Unless they have a common group like "adm"....

Dec 17 23:09:27 z4 proxy-server UNCAUGHT EXCEPTION#012Traceback (most recent call last):#012 File "/usr/bin/swift-proxy-server", line 22, in <module>#012 run_wsgi(conf_file, 'proxy-server', default_port=8080, **options)#012 File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 187, in run_wsgi#012 run_server()#012 File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 149, in run_server#012 global_conf=
{'log_name': log_name}
)#012 File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp#012 return loadobj(APP, uri, name=name, **kw)#012 File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj#012 return context.create()#012 File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create#012 return self.object_type.invoke(self)#012 File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 207, in invoke#012 app = filter(app)#012 File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift_middleware.py", line 171, in ceilometer_filter#012 return CeilometerMiddleware(app, conf)#012 File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift_middleware.py", line 80, in _init#012 service.prepare_service()#012 File "/usr/lib/python2.7/dist-packages/ceilometer/service.py", line 91, in prepare_service#012 cfg.CONF(argv[1:], project='ceilometer')#012 File "/usr/lib/python2.7/dist-packages/oslo/config/cfg.py", line 1180, in __call_#012 self._parse_config_files()#012 File "/usr/lib/python2.7/dist-packages/oslo/config/cfg.py", line 1655, in _parse_config_files#012 raise ConfigFilesNotFoundError(not_read_ok)#012ConfigFilesNotFoundError: Failed to read some config files: /etc/ceilometer/ceilometer.conf

[Julien Danjou (jdanjou)]

I'm setting this as won't fix as we can't do anything with that in Ceilometer. It's only a deployment problem.

[David Hill (david-hill-ubisoft)]

Deployment problem?

Swift runs as swift and ceilometer as ceilometer...

[David Hill (david-hill-ubisoft)]

Well, there's something weird I agree... I put the user swift in the ceilometer group and it still can't access the file.

[David Moreau Simard (dmsimard)]

I wrote that in a kind of confusing way.. and I can't edit the comment. Here's a less confusing version:

====

I'm also having this issue so let me weigh in with maybe some additional info. Are you willing to reconsider at all ?
I tried digging into this issue but I am not familiar enough with the code and the daemons to tell where exactly the problem is.

Now, in my setup, the situation looks like this:
- /etc/ceilometer is 750, ceilometer:ceilometer
- /etc/ceilometer/ceilometer.conf is 640, ceilometer:ceilometer
- User swift is a member of the ceilometer group:
# sudo -u swift id
uid=1001(swift) gid=1001(swift) groups=1001(swift),4(adm),115(ceilometer)

On paper, this looks good. To confirm, I did a "su" under swift and I am able to read the ceilometer files with the swift user.

However, the swift-proxy process seems unable to read /etc/ceilometer/ceilometer.conf unless either /etc/ceilometer the FOLDER is in 755 OR /etc/ceilometer is owned by ceilometer:swift. /etc/ceilometer/ceilometer.conf can remain in 640.

It doesn't look like the swift-proxy process is trying to access /etc/ceilometer/ceilometer.conf as the user swift (effective id).
Doing a strace -fp on swift-proxy-server, you'll have the following:
[pid 19619] stat("/etc/ceilometer/ceilometer.conf", 0x7fffa4540430) = -1 EACCES (Permission denied)

The error in syslog when swift-proxy is unable to access the file is:
Jan 14 17:51:55 proxy01 proxy-server STDOUT: 2014-01-14 17:51:55 CRITICAL [ceilometer] coercing to Unicode: need string or buffer, NoneType found

====

[David Moreau Simard (dmsimard)]

I did some more digging. The fact that the swift-proxy error messages change when I change the permissions on /etc/ceilometer lead me in the wrong direction. The real error is indeed about swift-proxy not being able to write to /var/log/ceilometer.

The permissions for /var/log/ceilometer are 750, ceilometer:adm.
The permissions for /var/log/ceilometer/swift-proxy-server.log are 644, root:root.
The user swift is a member of the adm group - it is able to read from that directory but not write to the logfile.

I guess in that context it is kind of related to the deployment.

[David Moreau Simard (dmsimard)]

Nevermind part of my last comment... (sorry, looks like i'm tired today !)

Let's see if I can get it right this time... I will try to be as precise as possible.

The user swift needs to: Read the /etc/ceilometer/ceilometer.conf file
====
-- /etc/ceilometer is by default 750, ceilometer:ceilometer
-- /etc/ceilometer/ceilometer.conf is by default 640, ceilometer:ceilometer

If swift does not have access to /etc/ceilometer, the error message is:
proxy-server STDOUT: 2014-01-13 22:10:48 CRITICAL [ceilometer] coercing to Unicode: need string or buffer, NoneType found"

If swift has access to /etc/ceilometer but NOT /etc/ceilometer/ceilometer.conf, the error message is:
proxy-server UNCAUGHT EXCEPTION#012Traceback (most recent call last):#012 File "/usr/bin/swift-proxy-server", line 22, in <module>#012 run_wsgi(conf_file, 'proxy-server', default_port=8080, **options)#012 File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 187, in run_wsgi#012 run_server()#012 File "/usr/lib/python2.7/dist-packages/swift/common/wsgi.py", line 149, in run_server#012 global_conf={'log_name': log_name})#012 File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp#012 return loadobj(APP, uri, name=name, **kw)#012 File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj#012 return context.create()#012 File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create#012 return self.object_type.invoke(self)#012 File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 207, in invoke#012 app = filter(app)#012 File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift_middleware.py", line 171, in ceilometer_filter#012 return CeilometerMiddleware(app, conf)#012 File "/usr/lib/python2.7/dist-packages/ceilometer/objectstore/swift_middleware.py", line 80, in __init__#012 service.prepare_service()#012 File "/usr/lib/python2.7/dist-packages/ceilometer/service.py", line 95, in prepare_service#012 cfg.CONF(argv[1:], project='ceilometer')#012 File "/usr/lib/python2.7/dist-packages/oslo/config/cfg.py", line 1180, in __call__#012 self._parse_config_files()#012 File "/usr/lib/python2.7/dist-packages/oslo/config/cfg.py", line 1655, in _parse_config_files#012 raise ConfigFilesNotFoundError(not_read_ok)#012ConfigFilesNotFoundError: Failed to read some config files: /etc/ceilometer/ceilometer.conf

The problem (bug?):
- I would expect adding the swift user to the ceilometer group to be sufficient. However, it does not work. I either need to:
-- chown the /etc/ceilometer folder AND the /etc/ceilometer/ceilometer.conf file to ceilometer:swift
-- chmod the /etc/ceilometer folder to 755 AND chmod the /etc/ceilometer/ceilometer.conf to 644.

750 and 640 both grant read permission bits to the ceilometer group. I have tried to understand why, swift being in the ceilometer group, is not sufficient here.. to no avail.

====

The user swift needs to: Write to /var/log/ceilometer/swift-proxy-server.log
-- /var/log/ceilometer is by default 750, ceilometer:adm
-- /var/log/ceilometer/swift-proxy-server.log is by default 644, root:root

If...

[David Moreau Simard (dmsimard)]

It could or could not be related but an additional test I have done is to change the PRIMARY group of the swift user to ceilometer, e.g,:
usermod -g ceilometer -G swift swift

# sudo -u swift id
uid=1001(swift) gid=115(ceilometer) groups=115(ceilometer),1001(swift)

After that, swift-proxy was able to read into /etc/ceilometer and /etc/ceilometer/ceilometer.conf.

If I make swift primary and ceilometer secondary, it breaks again:
usermod -g swift -G ceilometer swift

# sudo -u swift id
uid=1001(swift) gid=1001(swift) groups=1001(swift),115(ceilometer)

[Hans Petrie (hhus)]

The issue appears to me to be in the /usr/share/pyshared/swift/common/utils.py:

I hacked up the code below to make it work by adding the ceilusr stuff just to make it work with mode 640 on ceilometer.conf. The issue is the uid and gid get set to swift user id and swift group id but the groups don't get set for the process so its not really a member of the ceilometer group. This is why you can either make the gid for swift user the gid for "ceilometer" and it will work or you can do a sudo -u swift swift-init proxy restart and it will work or you can run swift-init as root with the changed code below and it will work. But if you swift-init as root and swift user has swift group the swift-proxy process's aren't part of the ceilometer group. At least I'm 90% sure this is what is happening.

def drop_privileges(user):
    """
    Sets the userid/groupid of the current process, get session leader, etc.

    :param user: User name to change privileges to
    """
    user = pwd.getpwnam(user)
    ceilusr = pwd.getpwnam("ceilometer")
    if os.geteuid() == 0:
        os.setgroups([ceilusr[3]])
    os.setgid(user[3])
    os.setuid(user[2])
    os.environ['HOME'] = user[5]
    try:
        os.setsid()
    except OSError:
        pass
    os.chdir('/') # in case you need to rmdir on where you started the daemon
    os.umask(022) # ensure files are created with the correct privileges

[David Moreau Simard (dmsimard)]

Awesome Hans, I knew I wasn't crazy but I wasn't able to find exactly where privilege escalation/drop was taking place.

Would this be a bug to file under swift, then ?

[Hans Petrie (hhus)]

I fully believe this is a bug in swift and should be fixed. Here is a more elegant solution that appends all existing member groups of a given user to the process so that swift-proxy will respect group memberships set by the admin rather than effectively "clearing" the group memberships. I propose the following patch to /usr/share/pyshared/swift/common/utils.py:
cloudscaling@zm-meth:~$ diff -c utils.py newutils.py

*** utils.py 2014-01-15 07:13:29.500844967 -0800
--- newutils.py 2014-01-15 07:12:57.736974152 -0800
***************
*** 18,24 ****
  import errno
  import fcntl
  import os
! import pwd
  import sys
  import time
  import functools
--- 18,24 ----
  import errno
  import fcntl
  import os
! import pwd, grp
  import sys
  import time
  import functools
***************
*** 849,857 ****

      :param user: User name to change privileges to
      """
      user = pwd.getpwnam(user)
      if os.geteuid() == 0:
! os.setgroups([])
      os.setgid(user[3])
      os.setuid(user[2])
      os.environ['HOME'] = user[5]
--- 849,859 ----

      :param user: User name to change privileges to
      """
+
+ groups = [g.gr_gid for g in grp.getgrall() if user in g.gr_mem]
      user = pwd.getpwnam(user)
      if os.geteuid() == 0:
! os.setgroups(groups)
      os.setgid(user[3])
      os.setuid(user[2])
      os.environ['HOME'] = user[5]

[David Moreau Simard (dmsimard)]

I submitted https://bugs.launchpad.net/swift/+bug/1269473.