TypeError: remove_event() takes exactly 3 arguments (4 given)

[Angus Salkeld]

I have just started getting this in heat engine logs, not sure on the cause.

2014-10-09 00:10:24.534 INFO heat.engine.stack [-] Stack DELETE COMPLETE (one): Stack DELETE completed successfully
2014-10-09 00:10:24.556 DEBUG heat.engine.scheduler [-] Task update_task from Stack "one" [9a0da6eb-d437-4977-be45-601997190247] complete from (pid=15777) step /opt/stack/heat/heat/engine/scheduler.py:216
2014-10-09 00:10:24.562 DEBUG heat.engine.stack_lock [-] Engine 62bff557-7ea5-4dc8-848f-7561b696209a released lock on stack 9a0da6eb-d437-4977-be45-601997190247 from (pid=15777) release /opt/stack/heat/heat/engine/stack_lock.py:122
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 455, in fire_timers
    timer()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__
    cb(*args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 219, in main
    self._resolve_links()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 230, in _resolve_links
    f(self, *ca, **ckw)
TypeError: remove_event() takes exactly 3 arguments (4 given)

[Angus Salkeld (asalkeld)]

seems to be introduced by this patch: https://github.com/openstack/heat/commit/6f13c2d0a4ff2004654a690c1e84a0c9d40ed577

[Angus Salkeld (asalkeld)]

Added some debug:
- def remove_event(self, stack_id, event):
+ def remove_event(self, stack_id, event, *args, **kwargs):
+ LOG.error('remove_event args:%s, kwargs:%s' % (args, kwargs))

I think eventlet (or something) passes some extra args (then event object):
heat.engine.service [-] remove_event args:(<eventlet.event.Event object at 0x7fcc14222110>,), kwargs:{}

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/127096

[Angus Salkeld (asalkeld)]

The callback has the wrong signature: http://eventlet.net/doc/modules/greenthread.html

 link(func, *curried_args, **curried_kwargs)

    Set up a function to be called with the results of the GreenThread.

    The function must have the following signature:

    def func(gt, [curried args/kwargs]):

since the patch that introduced this is lined up for stable, I do the same for this.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/127096
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=c73d607246b6bec3f5c79fdcee57adf5cfa3cc19
Submitter: Jenkins
Branch: master

commit c73d607246b6bec3f5c79fdcee57adf5cfa3cc19
Author: Angus Salkeld <email address hidden>
Date: Thu Oct 9 15:35:34 2014 +1000

    Add missing extra "greenthread" arg to remove_event()

    See: http://eventlet.net/doc/modules/greenthread.html
    the callback gets passed a "gt" arg.

    Change-Id: I9bd44857662e45a1da2d4287017eb966dea8276d
    Closes-bug: #1379072

[Zane Bitter (zaneb)]

Changed priority to High, since this means the memory leak is not fixed :(

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: proposed/juno
Review: https://review.openstack.org/127383

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/127383
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=170069d540342ced4dc3fdb3df6619395dd56c6f
Submitter: Jenkins
Branch: proposed/juno

commit 170069d540342ced4dc3fdb3df6619395dd56c6f
Author: Angus Salkeld <email address hidden>
Date: Thu Oct 9 15:35:34 2014 +1000

    Add missing extra "greenthread" arg to remove_event()

    See: http://eventlet.net/doc/modules/greenthread.html
    the callback gets passed a "gt" arg.

    Change-Id: I9bd44857662e45a1da2d4287017eb966dea8276d
    Closes-bug: #1379072
    (cherry picked from commit c73d607246b6bec3f5c79fdcee57adf5cfa3cc19)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/128908

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/128908
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=91c0e2329982df57e9520c23cda44930fa1b2cf4
Submitter: Jenkins
Branch: master

commit c06772571a81383aa37336a78d72d455067b75e8
Author: Zane Bitter <email address hidden>
Date: Fri Oct 10 15:43:16 2014 -0400

    Don't update a LoadBalancer under autoscaling control

    Unfortunately, Autoscaling currently uses the update() method of a
    LoadBalancer resource to do updates, with the result that the current
    member list gets persisted (good) and used to compare in the event of
    future *stack* updates (bad).

    With this patch, we assume that LoadBalancers under the control of
    Autoscaling will never have a members list property supplied in the
    template. We then ignore any updates to Autoscaling LoadBalancers that
    don't actually modify the template.

    The test changes revert the changes made in order to be able to merge
    d32370233eaf2a5c32888f269bd1dc5e0e787467, before which LoadBalancers were
    behaving correctly.

    Change-Id: I9c02ab3d3dfbee0a8a90dd0ba345a5acdaf8a610
    Closes-Bug: #1379619
    (cherry picked from commit 7506d2c22fcd3d6e825ee660a4b20c5947cc7e9a)

commit 5aa66555209eb5a59abdb189b0a5d1224e73b566
Author: Angus Salkeld <email address hidden>
Date: Fri Oct 10 12:00:37 2014 +1000

    Make sure that the properties are stored on updates

    Previously properties_data was only stored on creation, now
    this is getting passed to the update mechanism too.
    Later we can look at reworking this into a single mechanism.

    Closes-bug: #1377681
    Change-Id: If3d476f34b9e61a3c99f63ba33734a875353c8fc
    (cherry picked from commit d32370233eaf2a5c32888f269bd1dc5e0e787467)

commit 6e1ad898d887514267e3a429c291b2a067ea7e03
Author: huangtianhua <email address hidden>
Date: Wed Sep 24 15:19:24 2014 +0800

    Do not attempt a stack update when it is deleting

    If a stack is in any delete state(inprogress, failed),
    don't attempt to update it.

    Closes-bug: #1379113
    Change-Id: I1de99702a385ac8ddeffc568270d2d3b51674323

commit 1e9b2cdd9004c5e1fa3bb1a27830d75ebaebe20c
Author: Brant Knudson <email address hidden>
Date: Wed Oct 8 20:15:54 2014 -0500

    Update references to auth_token middleware

    There's references to the auth_token middleware in keystoncelient.
    The auth_token middleware has been moved to keystonemiddleware and
    the version in keystoneclient shouldn't be used anymore.

    If these references aren't updated, then when options are changed in
    keystonemiddleware.auth_token the heat-api will fail to start because
    there's duplicate options in keystoneclient.middleware.auth_token.

    Change-Id: I04573aa5ff967afe3e00329f797fcc71b779e7b3
    Closes-Bug: #1379082

commit 170069d540342ced4dc3fdb3df6619395dd56c6f
Author: Angus Salkeld <email address hidden>
Date: Thu Oct 9 15:35:34 2014 +1000

    Add missing extra "greenthread" arg to remove_event()

    See: http://eventlet.net/doc/modules/greenthread.html
    the callback gets passed a "gt" arg.

    Change-Id: I9bd44857662e45a1da2d4287017e...