Ceilometer cannot support IPv6 network

[Dazhao Yu]

In ceilometer.conf. in [api] part, add "host=::" to let ceilometer to listen ipv6 address, and start ceilometer-api service, will get the "gaierror: [Errno -9] Address family for hostname not supported".

The root cause is in ceilometer/api/app.py, it will call python wsgiref/simple_server.py to bind host, and simple_server.py will call /python2.6/SocketServer.py to do it. and the python TCPServer implementation is hard-coded to use ipv4, the python standard library is lacking in this regard. so when binding IPv6 host, ceilometer will throw "gaierror: [Errno -9] Address family for hostname not supported".

I have opened one bug to Python community--bug like: http://bugs.python.org/issue20215, but them consider it should be handled as new function, and want to handle it in python 3.5, whether we can accept this suggest? If we need to support IPv6 in ceilometer in this release, we need to make workaround in ceilometer to fix it

the log message as below:

2014-01-03 03:35:10.662 8609 CRITICAL ceilometer [-] [Errno -9] Address family for hostname not supported
2014-01-03 03:35:10.662 8609 TRACE ceilometer Traceback (most recent call last):
2014-01-03 03:35:10.662 8609 TRACE ceilometer File "./ceilometer-api", line 10, in <module>
2014-01-03 03:35:10.662 8609 TRACE ceilometer sys.exit(start())
2014-01-03 03:35:10.662 8609 TRACE ceilometer File "/usr/lib/python2.6/site-packages/ceilometer/api/app.py", line 118, in start
2014-01-03 03:35:10.662 8609 TRACE ceilometer srv = simple_server.make_server(host, port, root)
2014-01-03 03:35:10.662 8609 TRACE ceilometer File "/usr/lib64/python2.6/wsgiref/simple_server.py", line 181, in make_server
2014-01-03 03:35:10.662 8609 TRACE ceilometer server = server_class((host, port), handler_class)
2014-01-03 03:35:10.662 8609 TRACE ceilometer File "/usr/lib64/python2.6/SocketServer.py", line 412, in __init__
2014-01-03 03:35:10.662 8609 TRACE ceilometer self.server_bind()
2014-01-03 03:35:10.662 8609 TRACE ceilometer File "/usr/lib64/python2.6/wsgiref/simple_server.py", line 50, in server_bind
2014-01-03 03:35:10.662 8609 TRACE ceilometer HTTPServer.server_bind(self)
2014-01-03 03:35:10.662 8609 TRACE ceilometer File "/usr/lib64/python2.6/BaseHTTPServer.py", line 108, in server_bind
2014-01-03 03:35:10.662 8609 TRACE ceilometer SocketServer.TCPServer.server_bind(self)
2014-01-03 03:35:10.662 8609 TRACE ceilometer File "/usr/lib64/python2.6/SocketServer.py", line 424, in server_bind
2014-01-03 03:35:10.662 8609 TRACE ceilometer self.socket.bind(self.server_address)
2014-01-03 03:35:10.662 8609 TRACE ceilometer File "<string>", line 1, in bind
2014-01-03 03:35:10.662 8609 TRACE ceilometer gaierror: [Errno -9] Address family for hostname not supported
2014-01-03 03:35:10.662 8609 TRACE ceilometer
[root@localhost bin]# gaierror: [Errno -9] Address family for hostname not supported

Can you give me some suggests about it? whether we need to make the workaround or wait Python to fix it in python 3.5?

[Doug Hellmann (doug-hellmann)]

The simple server is really only meant to be used for development, and not for production deployments. Production systems can deploy the ceilometer WSGI app behind any number of WSGI servers, most or all of which would be able to handle IPv6 addressing.

Do you have a development environment without IPv4 access?

[Dazhao Yu (dzyu)]

Hi Doug, I do not have the development environment without IPv6 access. I just have IPv4 and IPv6 dual-stack environment.

Do you mean in production systems do not have such problem?

Anyway, I think we should to fix it anyway in development env or not?

[Doug Hellmann (doug-hellmann)]

That's correct, production systems should not have any issue with using IPv6 because they should deploy the WSGI app in a real WSGI server (mod_wsgi, gunicorn, etc.) rather than using the development server used by a devstack installation.

[Dazhao Yu (dzyu)]

So, you think we do not need to fix this issue in our local development env, is it right?

I think this issue was blocking our ceilometer in local development env.

[Julien Danjou (jdanjou)]

If you have a fix, go ahead and send it. Otherwise we'll live with it.

[Dazhao Yu (dzyu)]

Thanks Julien, I will fix it later since it still block development env

Doug, I still little confuse about wsgiref and mod_wsgi. I know the wsgiref is a simple pure Python implementation of WSGI server based on simple HTTP server from Python distributions. But little confuse is for:

I saw in ceilometer, in ceilometer/api/app.py, it import wsgiref simple server directly, so I think it will call wsgiref for ceilomenter functions, is it right? If I want to use mod_wsgi in openstack, what I should to do？how can controll to use mod_wsgi rather than wsgiref? I am new for it, can you give me some explanation about it?

[Doug Hellmann (doug-hellmann)]

A production deployment should use the VersionSelectorApp class to create a WSGI app. The exact way to do that depends on the WSGI server container being used for deployment, but it usually involves creating a small python file that imports the class, and then either provides a callable function that returns an instance or creates an instance with a specific variable name that the WSGI server is going to use to find the object in the module. (We need to document this better, but I think that's a separate bug from this one.)

[Dazhao Yu (dzyu)]

Doug, I still consider it as bug.

You mean VersionSelectorApp can choose whether use wsgiref or mod_wsgi? I do not understand the work principle, from code, I see the VersionSelectorApp depends on ceilometer configuration file, so can you tell me if I want to use mod_wsgi(not wsgiref ), what I should to do? how to configure the ceilometer files? From app.py code in line 118 https://github.com/openstack/ceilometer/blob/master/ceilometer/api/app.py#L118. the ceilometer will call wsgiref.simple_server directly, and will not use mod_wsgi, it is not possible to call mod_wsgi.

BTW, even the code can call mod_wsgi, in development or individual environment, when use simple server, the code still would throw error, so I made one patch about it, please review it

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/67786

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/67786
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=c843acb4f603018615fe62cbb07f3f4936840cd9
Submitter: Jenkins
Branch: master

commit c843acb4f603018615fe62cbb07f3f4936840cd9
Author: Dazhao <email address hidden>
Date: Mon Jan 20 15:32:45 2014 +0800

    Make sure use IPv6 sockets for ceilometer in IPv6 environment

    In IPv6 management network environment, ceilometer command will
    throw 'Address family for hostname not support' when use WSGI
    simple server. The root cause is python TCPServer implementation
    is hard-coded to use IPv4, even in IPv6 environments. Python
    community want to fix it in Python3.5, so we need to make a
    workaround for it.

    Change-Id: I3260a346a095b0f3e57093c18b1920dffe5ac55d
    Closes-Bug: #1269243