Kubernetes client is incompatible with evenlet and breaks the periodic tasks

[Spyros Trigazis]

After kubernetes client 4.0.0 there is this issue [1].
Magnum has three periodic tasks, one to sync the magnum service, one to update the cluster status and one send cluster metrics
The send_metrics task uses the kubernetes client for kubernetes clusters and it crashes the sync_cluster_status and send_cluster_metrics tasks.

[1] https://github.com/eventlet/eventlet/issues/147

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/529098
Committed: https://git.openstack.org/cgit/openstack/magnum/commit/?id=cf8468394027ffb1db420a72312b6a9f59b78381
Submitter: Zuul
Branch: master

commit cf8468394027ffb1db420a72312b6a9f59b78381
Author: Kirsten G <email address hidden>
Date: Tue Dec 19 07:57:55 2017 -0800

    Add send_cluster_metrics configuration parameter

    Added configuration parameter, send_cluster_metrics, to magnum.conf
    with default value of True. If set to True, periodic tasks will pull
    COE data and send to ceilometer. This parameter can be set to False to
    disable periodic collection of data to avoid unnecessary load from the
    cluster.

    Closes-Bug: #1668330
    Related-Bug: #1746510

    Change-Id: I9945293e7b2b52731f6e220d0925c1f6ad097caa

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/542198

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/542198
Committed: https://git.openstack.org/cgit/openstack/magnum/commit/?id=710192a63fd36186e9669cc2ac00dca2cc4fcc81
Submitter: Zuul
Branch: master

commit 710192a63fd36186e9669cc2ac00dca2cc4fcc81
Author: Spyros Trigazis <email address hidden>
Date: Thu Feb 8 13:58:45 2018 +0000

    Add issue to reno for the incompatible k8s client

    Advice users to disable the periodic task collection which
    uses the incompatible python kubernetes client.

    Change-Id: Ifb77774cf70e1391f97d9bc6f4cf45756913db47
    Related-Bug: #1746510

[Spyros Trigazis (strigazi)]

The client also breaks the kubernetes scale manager, disabling it until the bug is fixed.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/547016

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/547094

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/547016
Committed: https://git.openstack.org/cgit/openstack/magnum/commit/?id=1eb9bf16f5d60c5024b76fdc6691f0dc82e05a81
Submitter: Zuul
Branch: master

commit 1eb9bf16f5d60c5024b76fdc6691f0dc82e05a81
Author: Spyros Trigazis <email address hidden>
Date: Thu Feb 22 14:47:24 2018 +0000

    kuberntes: Disable the scale_manager for scale down

    Due to bug #1746510, the kubernetes scale manager needs to be
    disabled to not break the scale down command completely. Note, that when
    magnum scales down the cluster will pick the nodes to scale randomly.

    Related-Bug: #1746510
    Change-Id: I8c3505ec6d155323288217e0c8ea54adabdff1c3

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/547094
Committed: https://git.openstack.org/cgit/openstack/magnum/commit/?id=de39220cac079da6e14d6742b2d69b04002e5303
Submitter: Zuul
Branch: stable/queens

commit de39220cac079da6e14d6742b2d69b04002e5303
Author: Spyros Trigazis <email address hidden>
Date: Thu Feb 22 14:47:24 2018 +0000

    kuberntes: Disable the scale_manager for scale down

    Due to bug #1746510, the kubernetes scale manager needs to be
    disabled to not break the scale down command completely. Note, that when
    magnum scales down the cluster will pick the nodes to scale randomly.

    Related-Bug: #1746510
    Change-Id: I8c3505ec6d155323288217e0c8ea54adabdff1c3

[Feilong Wang (flwang)]

@Spyros, we probably should consider to close this bug because https://github.com/eventlet/eventlet/issues/172#issuecomment-379421165 has been merged and I have verified it locally and it works.

[Feilong Wang (flwang)]

After upgrade to eventlet 0.23, I can see below error

 from (pid=16692) request /usr/local/lib/python2.7/dist-packages/kubernetes/client/rest.py:219
2018-06-01 15:11:09.948 DEBUG kubernetes.client.rest [req-12a903dc-b5d6-4e1b-8098-aa97eea70417 None None] response body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/default/pods","resourceVersion":"1339250"},"items":[]}
 from (pid=16692) request /usr/local/lib/python2.7/dist-packages/kubernetes/client/rest.py:219
Exception AttributeError: "'Queue' object has no attribute 'not_empty'" in <Finalize object, dead> ignored

But there is no impact for Magnum functions so far.