[MuranoPL] pselect exits on first exception

Bug #1449635 reported by Victor Ryzhenkin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Fix Released
Critical
Stan Lagun

Bug Description

When several threads are running in parallel using pselect function or Parallel macro and when of those threads raise exception
pselect exits (with exception) without waiting for other threads to complete. As a result uncaught exception causes deployment to finish while there are still running threads. Those threads can still report to UI even if no deployment is running from user perspective. Because after deployment local package cache is wiped if such thread accesses its resource file an exception (file not found) is raised and that exception is reported to UI. And if such thread will try to send execution plan that was built earlier it will restart agent listener and it will steal messages from RabbitMQ queues causing subsequent deployments hang

Steps:
1. Deploy OS, CentOS, neutron-gre, Murano, All Ceph, HA (3 KVM controllers, 1 Hardware Compute)
2. Add any Docker Application to environment with k8s cluster
3. Deploy env

Aclual result:
Sometimes deployment failed with error message:
http://paste.openstack.org/show/208822/
And [OS Errno 2]: No such file in UI Reports

The same bug in upstream: https://bugs.launchpad.net/murano/+bug/1449497

Tags: murano
Changed in mos:
importance: Undecided → Critical
status: New → Confirmed
assignee: nobody → Stan Lagun (slagun)
milestone: none → 6.1
tags: added: murano
description: updated
Changed in mos:
status: Confirmed → In Progress
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/murano (openstack-ci/fuel-6.1/2014.2)

Fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: Stan Lagun <email address hidden>
Review: https://review.fuel-infra.org/6377

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/murano (openstack-ci/fuel-6.1/2014.2)

Reviewed: https://review.fuel-infra.org/6377
Submitter: Ekaterina Chernova <email address hidden>
Branch: openstack-ci/fuel-6.1/2014.2

Commit: 64b077e5f59503f5a9740364e212c75ee619a86a
Author: Stan Lagun <email address hidden>
Date: Thu Apr 30 16:34:48 2015

pselect exited on first exception

Due to eventlet bug in GreenPool implementation when first
spawned thread threw exception imap method exited immediately
without waiting for other spawned threads to accomplish.

Issue in eventlet github:
https://github.com/eventlet/eventlet/issues/232

This commits wraps executed functions so that no exception can be
raised on imap boundary and then unwraps result after execution

Change-Id: Icdcfa8e09becb272627b09ce258109eec2c9dd5b
Closes-Bug: #1449635

Changed in mos:
status: In Progress → Fix Committed
Changed in mos:
status: Fix Committed → Fix Released
Revision history for this message
Victor Ryzhenkin (vryzhenkin) wrote :

Verified on
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2.2-6.1"
  api: "1.0"
  build_number: "432"
  build_id: "2015-05-18_03-43-53"
  nailgun_sha: "076566b5df37f681c3fd5b139c966d680d81e0a5"
  python-fuelclient_sha: "38765563e1a7f14f45201fd47cf507393ff5d673"
  astute_sha: "cb655a9a9ad26848bcd9d9ace91857b6f4a0ec15"
  fuel-library_sha: "1621cb350af744f497c35f2b3bb889c2041465d8"
  fuel-ostf_sha: "9ce1800749081780b8b2a4a7eab6586583ffaf33"
  fuelmain_sha: "0e970647a83d9a7d336c4cc253606d4dd0d59a60"

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/murano (openstack-ci/fuel-7.0/2015.1.0)

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Stan Lagun <email address hidden>
Review: https://review.fuel-infra.org/8204

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/murano (openstack-ci/fuel-7.0/2015.1.0)

Change abandoned by Kirill Zaitsev <email address hidden> on branch: openstack-ci/fuel-7.0/2015.1.0
Review: https://review.fuel-infra.org/8204

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.