[provision] master iso smoke test failed with anaconda error

Bug #1324297 reported by Aleksandra Fedorova on 2014-05-28
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Medium
Fuel QA Team
4.1.x
High
MOS Linux
5.0.x
High
MOS Linux
5.1.x
High
MOS Linux
6.0.x
Medium
MOS Linux
6.1.x
Medium
MOS Linux
7.0.x
Medium
MOS Linux

Bug Description

Smoke test for fuel-master-228 iso fails on setup slave-02_compute node with anaconda traceback:

anaconda 13.21.215 exception report

<skip>

DBusException: org.freedesktop.DBus.Error.UnknownMethod: Method "Get" with signature "ss" on interface "org.freedesktop.DBus.Properties" doesn't exist

Aleksandra Fedorova (bookwar) wrote :
Aleksandra Fedorova (bookwar) wrote :
Changed in fuel:
milestone: none → 5.0.1
assignee: nobody → Fuel Library Team (fuel-library)
Nastya Urlapova (aurlapova) wrote :

Can't reproduce.

Changed in fuel:
status: New → Invalid
Artem Panchenko (apanchenko-8) wrote :

{"build_id": "2014-05-29_00-52-53", "mirantis": "yes", "build_number": "330", "nailgun_sha": "0828946af6a2e757fb018c2f69f7f81f0ea4b7f1", "ostf_sha": "3690709d95c1f42506b6d955b44221910cec8527", "fuelmain_sha": "7fbe99c86a4fffcb5ebc5b86046dcf1c20776f61", "astute_sha": "55df06b2e84fa5d71a1cc0e78dbccab5db29d968", "release": "4.1B", "fuellib_sha": "c64ffdc5e7bfe7fa8eb971be1fea2260ee849ab5"}

This bug was reproduced on ISO # 330 (4.1). Slave node gets stuck on provisioning and becomes offline. You can find anaconda log in attachments.

Artem Panchenko (apanchenko-8) wrote :
Artem Panchenko (apanchenko-8) wrote :
Changed in fuel:
status: Invalid → New
Artem Panchenko (apanchenko-8) wrote :

{"build_id": "2014-06-04_09-16-08", "mirantis": "yes", "build_number": "341", "nailgun_sha": "a828d6b7610f872980d5a2113774f1cda6f6810b", "ostf_sha": "c959aa55f83fe2555cf2d382559271c7a9b17467", "fuelmain_sha": "7ed0f85acc0bab4b9157703a618b8cc9fd7de3e1", "astute_sha": "55df06b2e84fa5d71a1cc0e78dbccab5db29d968", "release": "4.1B", "fuellib_sha": "0e96fc5a340cd57f75c454ea8536471379299494"}

Reproduced once again on ISO # 341.

Changed in fuel:
status: New → Invalid
Vladimir Kuklin (vkuklin) wrote :

it is anaconda bug which we will not fix in 4.x series

Dmitry Borodaenko (angdraug) wrote :

Why is this marked Invalid for 5.0.x?

Changed in fuel:
milestone: 5.0.1 → 5.1
assignee: Fuel Library Team (fuel-library) → Artem Panchenko (apanchenko-8)
Artem Panchenko (apanchenko-8) wrote :

Before I reproduced this issue for 4.1.1 it was set as assigned to 5.1 only and had invalid state. After adding my first comment I added to issue 4.x series and mistakenly changed status of 5.1 to new, but for 4.1.1 it still was invalid. So, when I posted my previous comment I discovered this and changed status for 5.1 back to invalid and for 4.1.1 to new.

Changed in fuel:
assignee: Artem Panchenko (apanchenko-8) → Fuel Library Team (fuel-library)
Anastasia Palkina (apalkina) wrote :

Reproduced this bug on ISO #262
"build_id": "2014-06-20_00-31-14",
"mirantis": "yes",
"build_number": "262",
"ostf_sha": "2f30e5cab5bec1f1e2fd80e26e4da771a8ffe2d4",
"nailgun_sha": "0c5e3b94fdd6bc9a50d5f840bf5151f95a23d908",
"production": "docker",
"api": "1.0",
"fuelmain_sha": "4f547561532baf5f26733bf66db692dc5b61806d",
"astute_sha": "694b5a55695e01e1c42185bfac9cc7a641a9bd48",
"release": "5.1",
"fuellib_sha": "25eb618a33a2ec87bc56f6bad16dc25b1837f0f0"
1. Create new environment (CentOS, HA mode)
2. Choose nova-network, VLAN manager
3. Choose both Ceph
4. Choose Ceilometer
5. Add 3 controller+mongo, 1 compute, 3 ceph
6. Start deployment
7. At the end of provisioning ceph node has failed with analogue error see screen)
8. Deployment has failed

Changed in fuel:
status: Invalid → Confirmed
Anastasia Palkina (apalkina) wrote :
Anastasia Palkina (apalkina) wrote :
Bogdan Dobrelya (bogdando) wrote :

If this bug is anaconda related, should we reassign it to the Fuel Linux Hardening team then?

Dmitry Borodaenko (angdraug) wrote :

Bogdan: good point, bug reassigned.

Fuel Linux Hardening team: looks like we need to at least backport the fix to the bugzilla bug above to the anaconda version in Fuel 5.0/5.1. If possible, the preferred solution would be to upgrade whole anaconda to the latest version, as long as it's still compatible with the kernel versions we're using for CentOS (2.6.32 and 3.10) and with out kickstart scripts.

Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Fuel Linux Hardening Team (fuel-linux)
Artem Panchenko (apanchenko-8) wrote :

api: '1.0'
astute_sha: 694b5a55695e01e1c42185bfac9cc7a641a9bd48
build_id: 2014-06-22_00-31-14
build_number: '264'
fuellib_sha: dc2713b3ba20ccff2816cf61e8481fe2f17ed69b
fuelmain_sha: 4394ca9be6540d652cc3919556633d9381e0db64
mirantis: 'yes'
nailgun_sha: eaabb2c9bbe8e921aaa231960dcda74a7bc86213
ostf_sha: 429c373fb79b1073aa336bc62c6aad45a8f93af6
production: docker
release: '5.1'

This issue is still being reproduced on master branch iso image and causes failing of 'smoke' system test on CI:

http://jenkins-product.srt.mirantis.net:8080/job/fuel_master_iso_smoke/958/

Changed in fuel:
assignee: Registry Administrators (registry) → MOS Linux (mos-linux)
tags: added: mos-linux
Changed in fuel:
importance: Critical → High
Albert Syriy (asyriy) wrote :

The anaconda bug #980576 was fixed for anaconda-rhel7 and anaconda-f20 branches only, what corresponds to Fedora-20 and CentOS 7 distributives. This is git commit fix, as an example:
----------------------------------- RHEL 7 branch ---------------------------
$ git log | grep -w8 980576
commit 57c029ae7be92bf9d8a7e8f2ec552b4294d23f6c
Author: Radek Vykydal <email address hidden>
Date: Tue Sep 10 10:21:50 2013 +0200

    Catch race of network device state vs reading its config properties (#919478, #1005681)

    Fedora bz: #980576
--------------------------------------------------------------------------------

CentOS 6.5 based on the anaconda-rhel4 branch, what unfortunately havn't got fixed for this bug.
The most plausible reason is sources are different.

The bug fix was applied to the file nm.py
---
 pyanaconda/nm.py | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)
---

This file pyanaconda/nm.py is absent in the anaconda-rhel4 branch at all and as the result in anaconda using in CentOS 6.5

My summary is:
=======================
Migration to CentOS 7 defentely will fix the issue.
Merely apply the bug fix from (Fedora/Centos 7) to Centos 6.5 is not possible, the sources are slightly different.
=======================

Any comments/suggestion?

I'll check if that fix can be backported

Actually I doubt this bug has anything to do with https://bugzilla.redhat.com/show_bug.cgi?id=980576 (for one, the version of anaconda in CentOS 6.5 is much older and does not contain the affected code).

Also the error message is just way to generic (just google it). We need more information in order to find out its cause
(like the complete backtrace, installation log, etc., see https://bugzilla.redhat.com/show_bug.cgi?id=980576 )

Artem Panchenko (apanchenko-8) wrote :

Alexei, I attached full anaconda log to the my previous comment:

https://bugs.launchpad.net/fuel/+bug/1324297/+attachment/4122676/+files/anaconda_error.log

Albert Syriy (asyriy) wrote :

More info:
1. The patch could be find on the anaconda-rhel7 branch. To get the code you should type:

git clone https://git.fedorahosted.org/git/anaconda.git -b rhel6-branch

2. The patch could be taken by the command (commit 57c029a) :

git diff 87ec02c 57c029a

I don't think commit 57c029a actually solves the problem (the code being changed is absent in rhel6-branch).
It looks like the correct patch is 22611be ('network: handle race condition of disappearing active connection (#1073424)').
I'm working on backporting it to rhel6-branch.

Bug was reproduced in system tests
{

    "build_id": "2014-07-13_02-01-14",
    "ostf_sha": "9863db951a6e159f4fa6e6861c8331e1af069cf8",
    "build_number": "316",
    "auth_required": false,
    "api": "1.0",
    "nailgun_sha": "638f4e09c5696d549482fabd0116f6008973291a",
    "production": "docker",
    "fuelmain_sha": "f32b958788e9d7acd904b31962b6bb71f9b66766",
    "astute_sha": "d68cc542290b3176b7a556eb9accba5a38a88519",
    "feature_groups": [
        "mirantis"
    ],
    "release": "5.1",
    "fuellib_sha": "1b17a7999969d87babbc7a07ba1c5c805d1588d8"

}
Cluster configuration - Centos, Simple mode, Neutron GRE, 1 controller, 2 compute nodes
Logs are attached

Preliminary version of the patch. Be forewarned: it hasn't been tested in any way (i.e. it might even completely break things).
In particular I'm not NetworkManager expert and I'm not quite sure what is the proper way to check if the connection
is actually active (is this isys.NM_DEVICE_STATE_ACTIVATED or something else?)

Also I'm not quite sure what is the fastest/easiest way to make an installation media with the updated anaconda version.

Dmitry Ilyin (idv1985) on 2014-07-15
summary: - [smoke] master iso smoke test failed with anaconda error
+ [provision] master iso smoke test failed with anaconda error
Michael Semenov (msemenov) wrote :

It's too risky to aply the patch in 5.1. So, moved to 6.0.

Changed in fuel:
milestone: 5.1 → 6.0
Mike Scherbakov (mihgen) on 2014-08-21
no longer affects: fuel
Mike Scherbakov (mihgen) on 2014-09-01
tags: added: release-notes
no longer affects: fuel
Michael Semenov (msemenov) wrote :

anaconda is not needed in 7.0 and newer versions, because IBP is being used for deployment. So, moving to 8.0.

Dmitry Pyzhov (dpyzhov) on 2015-10-12
Changed in fuel:
milestone: 7.0 → 8.0
no longer affects: fuel/8.0.x
Dmitry Pyzhov (dpyzhov) on 2015-10-22
tags: added: area-linux

Please recheck this behavior with latest CentOS 7 based image:

http://lists.openstack.org/pipermail/openstack-dev/2015-December/081421.html

Changed in fuel:
status: Confirmed → Incomplete
assignee: MOS Linux (mos-linux) → Aleksandra Fedorova (afedorova)
Aleksandra Fedorova (bookwar) wrote :

QA team, please take a look, if this issue is reproducible

Changed in fuel:
assignee: Aleksandra Fedorova (afedorova) → Fuel QA Team (fuel-qa)
Tatyanka (tatyana-leontovich) wrote :

move to invalid for 7.0 according issue is not reproduces there https://product-ci.infra.mirantis.net/job/8.0.ubuntu.smoke_neutron/

Changed in fuel:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.