collectd errors w/cannot assign requested address

[Shannon McFarland]

On each of my nodes running g.0 the syslog reports:

Jul 8 14:30:53 swift-proxy01 collectd[27570]: connecting to build-server.example.com:2003
Jul 8 14:30:53 swift-proxy01 collectd[27570]: Initialization complete, entering read-loop.
Jul 8 14:46:36 swift-proxy01 collectd[27713]: bind: Cannot assign requested address
Jul 8 14:46:36 swift-proxy01 collectd[27713]: network plugin: network_config_add_listen: sockent_open failed.

Collectd is running on the build-server node.

The /etc/collectd/collectd.conf file looks ok to me. It uses the correct interface and points to the correct listener (build-server.example.com port 9000) and that name is fully resolvable.

[Mark T. Voelker (mvoelker)]

Is graphite actually showing any stats coming in from these hosts?

[Shannon McFarland (shmcfarl)]

I can see Nagios3 info just fine but nothing on Graphite.

On Tue, Jul 9, 2013 at 12:04 AM, Mark T. Voelker <<email address hidden>
> wrote:

> Is graphite actually showing any stats coming in from these hosts?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1199012
>
> Title:
> collectd errors w/cannot assign requested address
>
> Status in Openstack @ Cisco:
> Triaged
>
> Bug description:
> On each of my nodes running g.0 the syslog reports:
>
>
> Jul 8 14:30:53 swift-proxy01 collectd[27570]: connecting to
> build-server.example.com:2003
> Jul 8 14:30:53 swift-proxy01 collectd[27570]: Initialization complete,
> entering read-loop.
> Jul 8 14:46:36 swift-proxy01 collectd[27713]: bind: Cannot assign
> requested address
> Jul 8 14:46:36 swift-proxy01 collectd[27713]: network plugin:
> network_config_add_listen: sockent_open failed.
>
> Collectd is running on the build-server node.
>
> The /etc/collectd/collectd.conf file looks ok to me. It uses the
> correct interface and points to the correct listener (build-
> server.example.com port 9000) and that name is fully resolvable.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/openstack-cisco/+bug/1199012/+subscriptions
>

[Mark T. Voelker (mvoelker)]

I just did a fresh build and can confirm that I *am* seeing stats coming into Graphite. Just to be sure we're on the same page:

If you click on the "Tree" tab in Graphite and expand the "Graphite" folder, you should see a folder for each of the hosts running collectd. Expand any of those and you should see lots of subfolders for the metrics being collected (see attached screenshot). Those folders all get created in the UI the first time the corresponding stat is received by carbon (e.g. it's not preprogrammed to expect certain metrics; it just starts showing whatever it receives), so the mere existence of the folders indicates that you've at least gotten some metrics.

You're not even seeing the folders?

[Shannon McFarland (shmcfarl)]

Sorry for the delay Mark. I am seeing metrics come in from Graphite (my earlier comment about seeing nothing in Graphite was do to a proxy issue with it being set for the machine I was testing from for Graphite). So I am not sure why it keeps complaining about not connecting to the server. In fact, even on the build-server itself it whines about the carbon server:
[2013-07-11 10:12:46] [warning] carbon_writer: no connection to carbon server

So, I confirmed that I can see stats on each node, but also each node continuously logs errors about connecting to the server.

[Mark T. Voelker (mvoelker)]

OK, thanks for confirming. I'll see if I can get to the bottom of the error message.

[Mark T. Voelker (mvoelker)]

The root of the problem seems to be this:

We set up a couple of collectd plugins, two of which are the caborn_writer plugin and the network plugin. Typically, the network plugin is used to either received data from other sources or to send data to a remote data sink (e.g. something like a Graphite instance). In our case, the data sink we want our nodes sending data to is the Graphite instance running on the build node. However, that's taken care of by the carbon_writer plugin and therefore the Network plugin isn't actually doing anything useful for us here. The error message stems from the fact that the Network plugin is actually misconfigured: it's setting up a socket to listen, but the address is that of the build node rather than the local node (which won't work).

The fix here is to simply remove the config for the network plugin. We don't need it, as is evidenced by the fact that Graphite still shows stats even though the network plugin is erroring out.

[Mark T. Voelker (mvoelker)]

Pull request to fix:
https://github.com/CiscoSystems/puppet-collectd/pull/5