[kafka plugin] firewall task is ran after zookeeper and kafka being launched

[Nikita Karpin]

On MOS 9.2 with next environment:

fuel nodes
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---+---------+------------------+---------+-----------+-------------------+------------------------------------------------------------------------+---------------+--------+---------
 3 | error | Untitled (ae:3e) | 1 | 10.20.0.5 | 52:54:00:2f:ae:3e | kafka | | 1 | 1
 1 | stopped | Untitled (56:ae) | 1 | 10.20.0.4 | 52:54:00:8d:56:ae | controller, kafka | | 1 | 1
 4 | ready | Untitled (32:20) | 1 | 10.20.0.6 | 00:25:90:7f:32:20 | cinder, compute | | 1 | 1
 2 | error | Untitled (67:04) | 1 | 10.20.0.3 | 52:54:00:24:67:04 | elasticsearch_kibana, influxdb_grafana, infrastructure_alerting, kafka | | 1 | 1

deployment fails with errors:

2017-03-30 13:46:23 +0000 Puppet (notice): Compiled catalog for node-3.domain.tld in environment production in 0.60 seconds
2017-03-30 13:46:26 +0000 /Stage[main]/Main/Exec[run_/tmp/create-topics.sh]/returns (notice): + /opt/kafka/bin/kafka-topics.sh --create --if-not-exists --zookeeper 192.168.0.6:2181 --replication-factor 3 --partitions 2 --topic notifications.sample
2017-03-30 13:46:26 +0000 /Stage[main]/Main/Exec[run_/tmp/create-topics.sh]/returns (notice): WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
2017-03-30 13:46:26 +0000 /Stage[main]/Main/Exec[run_/tmp/create-topics.sh]/returns (notice): Error while executing topic command : replication factor: 3 larger than available brokers: 2
2017-03-30 13:46:26 +0000 /Stage[main]/Main/Exec[run_/tmp/create-topics.sh]/returns (notice): [2017-03-30 13:46:25,824] ERROR kafka.admin.AdminOperationException: replication factor: 3 larger than available brokers: 2
2017-03-30 13:46:26 +0000 /Stage[main]/Main/Exec[run_/tmp/create-topics.sh]/returns (notice): at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:117)
2017-03-30 13:46:26 +0000 /Stage[main]/Main/Exec[run_/tmp/create-topics.sh]/returns (notice): at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:403)
2017-03-30 13:46:26 +0000 /Stage[main]/Main/Exec[run_/tmp/create-topics.sh]/returns (notice): at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:110)
2017-03-30 13:46:26 +0000 /Stage[main]/Main/Exec[run_/tmp/create-topics.sh]/returns (notice): at kafka.admin.TopicCommand$.main(TopicCommand.scala:61)
2017-03-30 13:46:26 +0000 /Stage[main]/Main/Exec[run_/tmp/create-topics.sh]/returns (notice): at kafka.admin.TopicCommand.main(TopicCommand.scala)
2017-03-30 13:46:26 +0000 /Stage[main]/Main/Exec[run_/tmp/create-topics.sh]/returns (notice): (kafka.admin.TopicCommand$)

Tasks order:

[root@fuel ~]# for i in 3 4 5; do ssh 10.20.0.$i grep 'kafka:' /var/log/puppet.log; done
Warning: Permanently added '10.20.0.3' (ECDSA) to the list of known hosts.
2017-03-30 13:09:18 +0000 Scope(Class[main]) (notice): fuel-plugin-kafka: hiera_override.pp
2017-03-30 13:09:59 +0000 Scope(Class[main]) (notice): fuel-plugin-kafka: check_environment_configuration.pp
2017-03-30 13:11:05 +0000 Scope(Class[main]) (notice): fuel-plugin-kafka: zookeeper.pp
2017-03-30 13:11:36 +0000 Scope(Class[main]) (notice): fuel-plugin-kafka: kafka.pp
2017-03-30 13:11:54 +0000 Scope(Class[main]) (notice): kafka: firewall.pp
Warning: Permanently added '10.20.0.4' (ECDSA) to the list of known hosts.
2017-03-30 13:39:57 +0000 Scope(Class[main]) (notice): fuel-plugin-kafka: hiera_override.pp
2017-03-30 13:40:00 +0000 Scope(Class[main]) (notice): fuel-plugin-kafka: check_environment_configuration.pp
2017-03-30 13:45:24 +0000 Scope(Class[main]) (notice): fuel-plugin-kafka: zookeeper.pp
2017-03-30 13:45:58 +0000 Scope(Class[main]) (notice): fuel-plugin-kafka: kafka.pp
Warning: Permanently added '10.20.0.5' (ECDSA) to the list of known hosts.
2017-03-30 13:07:04 +0000 Scope(Class[main]) (notice): fuel-plugin-kafka: hiera_override.pp
2017-03-30 13:07:06 +0000 Scope(Class[main]) (notice): fuel-plugin-kafka: check_environment_configuration.pp
2017-03-30 13:07:09 +0000 Scope(Class[main]) (notice): fuel-plugin-kafka: zookeeper.pp
2017-03-30 13:07:27 +0000 Scope(Class[main]) (notice): fuel-plugin-kafka: kafka.pp
2017-03-30 13:07:43 +0000 Scope(Class[main]) (notice): kafka: firewall.pp

as we can see on node-2 and node-3 firewall task was placed after kafka installation while on node-1 this task wasn't even executed, it was still in pending state when create-topics task was launched

[Alex Kholkin (akholkin)]

https://review.openstack.org/#/c/452105