[MuranoPL] pselect exits on first exception

Bug #1449497 reported by Stan Lagun on 2015-04-28
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Murano
Critical
Stan Lagun
Juno
Undecided
Unassigned
Kilo
Critical
Stan Lagun
Liberty
Critical
Stan Lagun

Bug Description

When several threads are running in parallel using pselect function or Parallel macro and when of those threads raise exception
pselect exits (with exception) without waiting for other threads to complete. As a result uncaught exception causes deployment to finish while there are still running threads. Those threads can still report to UI even if no deployment is running from user perspective. Because after deployment local package cache is wiped if such thread accesses its resource file an exception (file not found) is raised and that exception is reported to UI. And if such thread will try to send execution plan that was built earlier it will restart agent listener and it will steal messages from RabbitMQ queues causing subsequent deployments hang

Steps:
1. Add any Docker Application to environment with k8s cluster
2. Deploy environment

Actual result:
Sometimes deployment fails with errors in deployment logs in dashboard:
2015-04-21 13:07:55 - Configuring etcd master node
2015-04-21 13:07:55 - Error: [Errno 2] No such file or directory: '/tmp/murano-packages-cache/8ac61cc5-5f2b-43ff-9b84-6bccd8a1a95b/io.murano.apps.docker.kubernetes.KubernetesCluster'
2015-04-21 13:07:55 - Creating Kubernetes Master
2015-04-21 13:07:55 - Creating Kubernetes Minion
2015-04-21 13:07:55 - Creating Gateway node for Kubernetes services
2015-04-21 13:07:55 - Error: [Errno 2] No such file or directory: '/tmp/murano-packages-cache/8ac61cc5-5f2b-43ff-9b84-6bccd8a1a95b/io.murano.apps.docker.kubernetes.KubernetesCluster'
Full deployment hystory: http://paste.openstack.org/show/204972/
This can be caused by http://paste.openstack.org/show/208822/ , for example.

Changed in murano:
status: New → Confirmed
milestone: none → kilo-rc2
description: updated
Stan Lagun (slagun) on 2015-04-28
description: updated
description: updated

Reviewed: https://review.openstack.org/179036
Committed: https://git.openstack.org/cgit/openstack/murano/commit/?id=6a027bf11e756c57ab76f81d55fc9c26eb555bb2
Submitter: Jenkins
Branch: stable/kilo

commit 6a027bf11e756c57ab76f81d55fc9c26eb555bb2
Author: Stan Lagun <email address hidden>
Date: Thu Apr 30 15:21:28 2015 +0300

    pselect exited on first exception

    Due to eventlet bug in GreenPool implementation when first
    spawned thread threw exception imap method exited immediately
    without waiting for other spawned threads to accomplish.

    Issue in eventlet github:
    https://github.com/eventlet/eventlet/issues/232

    This commits wraps executed functions so that no exception can be
    raised on imap boundary and then unwraps result after execution

     Closes-Bug: #1449497

    Change-Id: I0e99de8e0711071c4a009e37a58ec22761e90e6c

Changed in murano:
status: Fix Committed → Fix Released
Changed in murano:
milestone: kilo-rc2 → 2015.1.0

Reviewed: https://review.openstack.org/179532
Committed: https://git.openstack.org/cgit/openstack/murano/commit/?id=fba9ed643de02a353d8f30918d373458cc09815e
Submitter: Jenkins
Branch: master

commit fba9ed643de02a353d8f30918d373458cc09815e
Author: Stan Lagun <email address hidden>
Date: Thu Apr 30 15:21:28 2015 +0300

    pselect exited on first exception

    Due to eventlet bug in GreenPool implementation when first
    spawned thread threw exception imap method exited immediately
    without waiting for other spawned threads to accomplish.

    Issue in eventlet github:
    https://github.com/eventlet/eventlet/issues/232

    This commits wraps executed functions so that no exception can be
    raised on imap boundary and then unwraps result after execution

     Closes-Bug: #1449497

    Change-Id: I0e99de8e0711071c4a009e37a58ec22761e90e6c
    (cherry picked from commit 6a027bf11e756c57ab76f81d55fc9c26eb555bb2)

tags: added: juno-backport-potential
tags: added: in-stable-juno
tags: removed: in-stable-juno
Changed in murano:
status: Fix Committed → Fix Released

Reviewed: https://review.openstack.org/196032
Committed: https://git.openstack.org/cgit/openstack/murano/commit/?id=82458d8e52533af6e3ee62d055ee861907f8a6bd
Submitter: Jenkins
Branch: stable/juno

commit 82458d8e52533af6e3ee62d055ee861907f8a6bd
Author: Stan Lagun <email address hidden>
Date: Thu Apr 30 15:21:28 2015 +0300

    pselect exited on first exception

    Due to eventlet bug in GreenPool implementation when first
    spawned thread threw exception imap method exited immediately
    without waiting for other spawned threads to accomplish.

    Issue in eventlet github:
    https://github.com/eventlet/eventlet/issues/232

    This commits wraps executed functions so that no exception can be
    raised on imap boundary and then unwraps result after execution

     Closes-Bug: #1449497

    Change-Id: I0e99de8e0711071c4a009e37a58ec22761e90e6c
    (cherry picked from commit 6a027bf11e756c57ab76f81d55fc9c26eb555bb2)
    (cherry picked from commit fba9ed643de02a353d8f30918d373458cc09815e)

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers