[provision] master iso smoke test failed with anaconda error

[Aleksandra Fedorova]

Smoke test for fuel-master-228 iso fails on setup slave-02_compute node with anaconda traceback:

anaconda 13.21.215 exception report

<skip>

DBusException: org.freedesktop.DBus.Error.UnknownMethod: Method "Get" with signature "ss" on interface "org.freedesktop.DBus.Properties" doesn't exist

[Dmitry Burmistrov (dburmistrov)]

It seems to be anaconda bug https://bugzilla.redhat.com/show_bug.cgi?id=980576

[Nastya Urlapova (aurlapova)]

Can't reproduce.

[Artem Panchenko (apanchenko-8)]

{"build_id": "2014-05-29_00-52-53", "mirantis": "yes", "build_number": "330", "nailgun_sha": "0828946af6a2e757fb018c2f69f7f81f0ea4b7f1", "ostf_sha": "3690709d95c1f42506b6d955b44221910cec8527", "fuelmain_sha": "7fbe99c86a4fffcb5ebc5b86046dcf1c20776f61", "astute_sha": "55df06b2e84fa5d71a1cc0e78dbccab5db29d968", "release": "4.1B", "fuellib_sha": "c64ffdc5e7bfe7fa8eb971be1fea2260ee849ab5"}

This bug was reproduced on ISO # 330 (4.1). Slave node gets stuck on provisioning and becomes offline. You can find anaconda log in attachments.

[Artem Panchenko (apanchenko-8)]

{"build_id": "2014-06-04_09-16-08", "mirantis": "yes", "build_number": "341", "nailgun_sha": "a828d6b7610f872980d5a2113774f1cda6f6810b", "ostf_sha": "c959aa55f83fe2555cf2d382559271c7a9b17467", "fuelmain_sha": "7ed0f85acc0bab4b9157703a618b8cc9fd7de3e1", "astute_sha": "55df06b2e84fa5d71a1cc0e78dbccab5db29d968", "release": "4.1B", "fuellib_sha": "0e96fc5a340cd57f75c454ea8536471379299494"}

Reproduced once again on ISO # 341.

[Vladimir Kuklin (vkuklin)]

it is anaconda bug which we will not fix in 4.x series

[Dmitry Borodaenko (angdraug)]

Why is this marked Invalid for 5.0.x?

[Artem Panchenko (apanchenko-8)]

Before I reproduced this issue for 4.1.1 it was set as assigned to 5.1 only and had invalid state. After adding my first comment I added to issue 4.x series and mistakenly changed status of 5.1 to new, but for 4.1.1 it still was invalid. So, when I posted my previous comment I discovered this and changed status for 5.1 back to invalid and for 4.1.1 to new.

[Anastasia Palkina (apalkina)]

Reproduced this bug on ISO #262
"build_id": "2014-06-20_00-31-14",
"mirantis": "yes",
"build_number": "262",
"ostf_sha": "2f30e5cab5bec1f1e2fd80e26e4da771a8ffe2d4",
"nailgun_sha": "0c5e3b94fdd6bc9a50d5f840bf5151f95a23d908",
"production": "docker",
"api": "1.0",
"fuelmain_sha": "4f547561532baf5f26733bf66db692dc5b61806d",
"astute_sha": "694b5a55695e01e1c42185bfac9cc7a641a9bd48",
"release": "5.1",
"fuellib_sha": "25eb618a33a2ec87bc56f6bad16dc25b1837f0f0"
1. Create new environment (CentOS, HA mode)
2. Choose nova-network, VLAN manager
3. Choose both Ceph
4. Choose Ceilometer
5. Add 3 controller+mongo, 1 compute, 3 ceph
6. Start deployment
7. At the end of provisioning ceph node has failed with analogue error see screen)
8. Deployment has failed

[Bogdan Dobrelya (bogdando)]

If this bug is anaconda related, should we reassign it to the Fuel Linux Hardening team then?

[Dmitry Borodaenko (angdraug)]

Bogdan: good point, bug reassigned.

Fuel Linux Hardening team: looks like we need to at least backport the fix to the bugzilla bug above to the anaconda version in Fuel 5.0/5.1. If possible, the preferred solution would be to upgrade whole anaconda to the latest version, as long as it's still compatible with the kernel versions we're using for CentOS (2.6.32 and 3.10) and with out kickstart scripts.

[Artem Panchenko (apanchenko-8)]

api: '1.0'
astute_sha: 694b5a55695e01e1c42185bfac9cc7a641a9bd48
build_id: 2014-06-22_00-31-14
build_number: '264'
fuellib_sha: dc2713b3ba20ccff2816cf61e8481fe2f17ed69b
fuelmain_sha: 4394ca9be6540d652cc3919556633d9381e0db64
mirantis: 'yes'
nailgun_sha: eaabb2c9bbe8e921aaa231960dcda74a7bc86213
ostf_sha: 429c373fb79b1073aa336bc62c6aad45a8f93af6
production: docker
release: '5.1'

This issue is still being reproduced on master branch iso image and causes failing of 'smoke' system test on CI:

http://jenkins-product.srt.mirantis.net:8080/job/fuel_master_iso_smoke/958/

[Albert Syriy (asyriy)]

The anaconda bug #980576 was fixed for anaconda-rhel7 and anaconda-f20 branches only, what corresponds to Fedora-20 and CentOS 7 distributives. This is git commit fix, as an example:
----------------------------------- RHEL 7 branch ---------------------------
$ git log | grep -w8 980576
commit 57c029ae7be92bf9d8a7e8f2ec552b4294d23f6c
Author: Radek Vykydal <email address hidden>
Date: Tue Sep 10 10:21:50 2013 +0200

    Catch race of network device state vs reading its config properties (#919478, #1005681)

    Fedora bz: #980576
--------------------------------------------------------------------------------

CentOS 6.5 based on the anaconda-rhel4 branch, what unfortunately havn't got fixed for this bug.
The most plausible reason is sources are different.

The bug fix was applied to the file nm.py
---
 pyanaconda/nm.py | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)
---

This file pyanaconda/nm.py is absent in the anaconda-rhel4 branch at all and as the result in anaconda using in CentOS 6.5

My summary is:
=======================
Migration to CentOS 7 defentely will fix the issue.
Merely apply the bug fix from (Fedora/Centos 7) to Centos 6.5 is not possible, the sources are slightly different.
=======================

Any comments/suggestion?

[Alexei Sheplyakov (asheplyakov)]

I'll check if that fix can be backported

[Alexei Sheplyakov (asheplyakov)]

Actually I doubt this bug has anything to do with https://bugzilla.redhat.com/show_bug.cgi?id=980576 (for one, the version of anaconda in CentOS 6.5 is much older and does not contain the affected code).

Also the error message is just way to generic (just google it). We need more information in order to find out its cause
(like the complete backtrace, installation log, etc., see https://bugzilla.redhat.com/show_bug.cgi?id=980576 )

[Artem Panchenko (apanchenko-8)]

Alexei, I attached full anaconda log to the my previous comment:

https://bugs.launchpad.net/fuel/+bug/1324297/+attachment/4122676/+files/anaconda_error.log

[Albert Syriy (asyriy)]

More info:
1. The patch could be find on the anaconda-rhel7 branch. To get the code you should type:

git clone https://git.fedorahosted.org/git/anaconda.git -b rhel6-branch

2. The patch could be taken by the command (commit 57c029a) :

git diff 87ec02c 57c029a

[Alexei Sheplyakov (asheplyakov)]

I don't think commit 57c029a actually solves the problem (the code being changed is absent in rhel6-branch).
It looks like the correct patch is 22611be ('network: handle race condition of disappearing active connection (#1073424)').
I'm working on backporting it to rhel6-branch.

[Andrey Sledzinskiy (asledzinskiy)]

Bug was reproduced in system tests
{

    "build_id": "2014-07-13_02-01-14",
    "ostf_sha": "9863db951a6e159f4fa6e6861c8331e1af069cf8",
    "build_number": "316",
    "auth_required": false,
    "api": "1.0",
    "nailgun_sha": "638f4e09c5696d549482fabd0116f6008973291a",
    "production": "docker",
    "fuelmain_sha": "f32b958788e9d7acd904b31962b6bb71f9b66766",
    "astute_sha": "d68cc542290b3176b7a556eb9accba5a38a88519",
    "feature_groups": [
        "mirantis"
    ],
    "release": "5.1",
    "fuellib_sha": "1b17a7999969d87babbc7a07ba1c5c805d1588d8"

}
Cluster configuration - Centos, Simple mode, Neutron GRE, 1 controller, 2 compute nodes
Logs are attached

[Alexei Sheplyakov (asheplyakov)]

Preliminary version of the patch. Be forewarned: it hasn't been tested in any way (i.e. it might even completely break things).
In particular I'm not NetworkManager expert and I'm not quite sure what is the proper way to check if the connection
is actually active (is this isys.NM_DEVICE_STATE_ACTIVATED or something else?)

Also I'm not quite sure what is the fastest/easiest way to make an installation media with the updated anaconda version.

[Michael Semenov (msemenov)]

It's too risky to aply the patch in 5.1. So, moved to 6.0.

[Michael Semenov (msemenov)]

anaconda is not needed in 7.0 and newer versions, because IBP is being used for deployment. So, moving to 8.0.

[Aleksander Mogylchenko (amogylchenko)]

Please recheck this behavior with latest CentOS 7 based image:

http://lists.openstack.org/pipermail/openstack-dev/2015-December/081421.html

[Aleksandra Fedorova (bookwar)]

QA team, please take a look, if this issue is reproducible

[Tatyanka (tatyana-leontovich)]

move to invalid for 7.0 according issue is not reproduces there https://product-ci.infra.mirantis.net/job/8.0.ubuntu.smoke_neutron/