fluentd unable to parse gnocchi-metricd.log

[Krzysztof Klimonda]

Seeing a lot of warnings related to gnocchi-metricd.log file, like:

```
2019-05-14 08:58:53 +0000 [warn]: #0 got incomplete line before first line from /var/log/kolla/gnocchi/gnocchi-metricd.log: "RuntimeError: Set changed size during iteration\n"
2019-05-14 08:58:53 +0000 [warn]: #0 got incomplete line before first line from /var/log/kolla/gnocchi/gnocchi-metricd.log: "2019-05-14 08:58:53,253 [32] ERROR gnocchi.chef: Error processing new measures\n"
2019-05-14 08:58:53 +0000 [warn]: #0 got incomplete line before first line from /var/log/kolla/gnocchi/gnocchi-metricd.log: "Traceback (most recent call last):\n"
2019-05-14 08:58:53 +0000 [warn]: #0 got incomplete line before first line from /var/log/kolla/gnocchi/gnocchi-metricd.log: "Traceback (most recent call last):\n"
2019-05-14 08:58:53 +0000 [warn]: #0 got incomplete line before first line from /var/log/kolla/gnocchi/gnocchi-metricd.log: " File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/gnocchi/chef.py\", line 153, in process_new_measures_for_sack\n"
2019-05-14 08:58:53 +0000 [warn]: #0 got incomplete line before first line from /var/log/kolla/gnocchi/gnocchi-metricd.log: " with self.incoming.process_measures_for_sack(sack) as measures:\n"
2019-05-14 08:58:53 +0000 [warn]: #0 got incomplete line before first line from /var/log/kolla/gnocchi/gnocchi-metricd.log: " File \"/usr/lib/python2.7/contextlib.py\", line 17, in __enter__\n"
2019-05-14 08:58:53 +0000 [warn]: #0 got incomplete line before first line from /var/log/kolla/gnocchi/gnocchi-metricd.log: " return self.gen.next()\n"
2019-05-14 08:58:53 +0000 [warn]: #0 got incomplete line before first line from /var/log/kolla/gnocchi/gnocchi-metricd.log: " File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/gnocchi/incoming/redis.py\", line 157, in process_measures_for_sack\n"
2019-05-14 08:58:53 +0000 [warn]: #0 got incomplete line before first line from /var/log/kolla/gnocchi/gnocchi-metricd.log: " results = self._scripts['process_measures_for_sack'](keys=[str(sack)])\n"
2019-05-14 08:58:53 +0000 [warn]: #0 got incomplete line before first line from /var/log/kolla/gnocchi/gnocchi-metricd.log: " File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/redis/client.py\", line 3575, in __call__\n"
2019-05-14 08:58:53 +0000 [warn]: #0 got incomplete line before first line from /var/log/kolla/gnocchi/gnocchi-metricd.log: " return client.evalsha(self.sha, len(keys), *args)\n"
2019-05-14 08:58:53 +0000 [warn]: #0 got incomplete line before first line from /var/log/kolla/gnocchi/gnocchi-metricd.log: " File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/redis/client.py\", line 2761, in evalsha\n"
2019-05-14 08:58:53 +0000 [warn]: #0 got incomplete line before first line from /var/log/kolla/gnocchi/gnocchi-metricd.log: " return self.execute_command('EVALSHA', sha, numkeys, *keys_and_args)\n"
2019-05-14 08:58:53 +0000 [warn]: #0 got incomplete line before first line from /var/log/kolla/gnocchi/gnocchi-metricd.log: " File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/redis/client.py\", line 772, in execute_command\n"
2019-05-14 08:58:53 +0000 [warn]: #0 got incomplete line before first line from /var/log/kolla/gnocchi/gnocchi-metricd.log: " connection = pool.get_connection(command_name, **options)\n"
2019-05-14 08:58:53 +0000 [warn]: #0 got incomplete line before first line from /var/log/kolla/gnocchi/gnocchi-metricd.log: " File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/redis/connection.py\", line 994, in get_connection\n"
2019-05-14 08:58:53 +0000 [warn]: #0 got incomplete line before first line from /var/log/kolla/gnocchi/gnocchi-metricd.log: " connection.connect()\n"
2019-05-14 08:58:53 +0000 [warn]: #0 got incomplete line before first line from /var/log/kolla/gnocchi/gnocchi-metricd.log: " File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/redis/sentinel.py\", line 45, in connect\n"
2019-05-14 08:58:53 +0000 [warn]: #0 got incomplete line before first line from /var/log/kolla/gnocchi/gnocchi-metricd.log: " self.connect_to(self.connection_pool.get_master_address())\n"
2019-05-14 08:58:53 +0000 [warn]: #0 got incomplete line before first line from /var/log/kolla/gnocchi/gnocchi-metricd.log: " File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/redis/sentinel.py\", line 107, in get_master_address\n"
2019-05-14 08:58:53 +0000 [warn]: #0 got incomplete line before first line from /var/log/kolla/gnocchi/gnocchi-metricd.log: " self.disconnect()\n"
2019-05-14 08:58:53 +0000 [warn]: #0 got incomplete line before first line from /var/log/kolla/gnocchi/gnocchi-metricd.log: " File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/redis/connection.py\", line 1040, in disconnect\n"
2019-05-14 08:58:53 +0000 [warn]: #0 got incomplete line before first line from /var/log/kolla/gnocchi/gnocchi-metricd.log: " for connection in all_conns:\n"
2019-05-14 08:58:53 +0000 [warn]: #0 got incomplete line before first line from /var/log/kolla/gnocchi/gnocchi-metricd.log: "RuntimeError: Set changed size during iteration\n"
```

[Mark Goddard (mgoddard)]

Where are you seeing these messages?

[Krzysztof Klimonda (kklimonda)]

In the output of `docker logs fluentd` on controllers.

[Krzysztof Klimonda (kklimonda)]

Also, it seems that once fluentd sees a message it cannot parse, it stops parsing other log files on the host.