qemu2.0.0 should be upgraded to 2.4 on Jenkins Product CI

[Dennis Dmitriev]

There are several types of issues faced using qemu2.0.0:

1) Fuel admin node freezes after suspend / resume sequence (even without taking a snapshot) eating 100% CPU: [1], [2]
Reproduced in ~70% system tests when snapshot is taken after deploy of cluster.
We use workaround for this issue in system tests: destroy and start Fuel admin node if it inaccessible after revert. Duration of each job that perform system tests can be increased more than on an hour because of this issue.

2) Docker containers on Fuel master node can be completely broken after destroy Fuel admin node that freezes by the reason from the issue №1 : [3]
No workaround here because there is an unrecoverable data loss. Tests are failed.

3) Sometimes libvirt unable to take a snapshot because a volume can be busy for unknown reason at the moment of taking the snapshot: [4]
No workaround here. Tests are failed.

To investigate the issue, was performed the following actions:
  - Stop docker containers before taking a snapshot. Issue №1 is reproduced.
  - Reduce IO on Fuel admin node by stopping docker containers and rsyslogd. Issue №1 is reproduced.
  - Snapshot slave nodes first in order to get Fuel admin node snapshoted when there is no other activity on the host. Issue №1 is reproduced.
  - Force closing all SSH sessions on Fuel admin node to reduce the amount of ESTABLISHED connections before taking a snapshot. Issue №1 is reproduced.
  - All the above methods at the same time. No success.

Issue №1 can be not reproduced or reproducing constantly on the same host, when no other activity on the host was performed.

--------------

As the possible solution, we can test qemu v2.3 on Jenkins, which doesn't show issues with snapshoting or suspending on local hosts, and upgrade to qemu2.3 in case if the tests did not detect new problems.

[1] https://bugs.launchpad.net/fuel/+bug/1418204
[2] https://bugs.launchpad.net/fuel/+bug/1450508
[3] https://bugs.launchpad.net/fuel/+bug/1457802
[4] https://bugs.launchpad.net/fuel/+bug/1415079

[Alexander Evseev (aevseev)]

BTW 2.4.0 just released...

[Igor Shishkin (teran)]

QEMU 2.3 is not present in Ubuntu repository.
Reassigning to build team.

[Dmitry Borodaenko (angdraug)]

You may be able to take qemu 1:2.3+dfsg-6a from Debian/sid as is:
https://packages.debian.org/sid/qemu

[Aleksandra Fedorova (bookwar)]

This issue becomes urgent. We have several failures on CI due to peak CPU usage during vm's bootstrap process.

 This load is not seen on zabbix graph, but can reach about 12 according to atop data.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: master
Change author: Sergey Otpuschennikov <email address hidden>
Review: https://review.fuel-infra.org/11169

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/11169
Submitter: Andrey Nikitin <email address hidden>
Branch: master

Commit: 7878dc95c82bb0086b52a52c635b76a8f33160c7
Author: Sergey Otpuschennikov <email address hidden>
Date: Sat Sep 5 11:12:33 2015

Added new repository

- fuel-infra-packages/libseccomp

Change-Id: Ic3f64fe90fe8ba479456cbda628887b10cb4ac98
Partial-Bug: #1484080

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: master
Change author: Sergey Otpuschennikov <email address hidden>
Review: https://review.fuel-infra.org/11248

[Sergey Otpuschennikov (sotpuschennikov)]

New versions of qemu successfully built in temporary repositories
qemu-2.3 http://obs-1.mirantis.com:82/qa-ubuntu-10882/
qemu-2.4 http://obs-1.mirantis.com:82/qa-ubuntu-11241/

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: master
Change author: Sergey Otpuschennikov <email address hidden>
Review: https://review.fuel-infra.org/11274

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/11274
Submitter: Igor Shishkin <email address hidden>
Branch: master

Commit: 9ae0e11a8ad13680270f411f8bb26f2a70aae3fa
Author: Sergey Otpuschennikov <email address hidden>
Date: Tue Sep 8 13:52:47 2015

Added new project

- fuel-infra-packages/libiscsi
- fuel-infra-packages/openbios

We need upgrade version libiscsi, openbios-ppc and openbios-sparc for QEMU 2.4

Change-Id: Ib1d61dd1ba0146c15ea5d47ededaee9aeae71b47
Partial-Bug: #1484080

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: master
Change author: Sergey Otpuschennikov <email address hidden>
Review: https://review.fuel-infra.org/11303

[Dennis Dmitriev (ddmitriev)]

I've tested qemu2.4 on Ubuntu 14.04 LTS, using these repos:

qemu2.4: http://obs-1.mirantis.com:82/qa-ubuntu-11241/ubuntu/
openbios_1.1+svn1334: http://obs-1.mirantis.com:82/qa-ubuntu-11303/ubuntu/
,and the package http://launchpadlibrarian.net/199538382/seabios_1.7.5-1ubuntu1_all.deb

Several system tests were passed on qemu2.4 without any issues.

[Dennis Dmitriev (ddmitriev)]

Please provide any CI server (or 2-3 servers if it is possible), from old or new CI, where in console output of system tests appeared such message:

"From first time admin isn't reverted: [Errno 113] No route to host"

Could be found, for example, almost in any run of https://product-ci.infra.mirantis.net/view/7.0_swarm/job/7.0.system_test.ubuntu.thread_3/

[Dennis Dmitriev (ddmitriev)]

Upd. to #13 - please upgrade qemu to 2.4 on the provided server(s) so we could test it during acceptance.

[Dennis Dmitriev (ddmitriev)]

Servers for testing are requested in https://bugs.launchpad.net/fuel/+bug/1494278

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: master
Change author: Sergey Otpuschennikov <email address hidden>
Review: https://review.fuel-infra.org/11435

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/11435
Submitter: Igor Shishkin <email address hidden>
Branch: master

Commit: ca3cf047b56377fc5afcc34bcb876349f66b5b12
Author: Sergey Otpuschennikov <email address hidden>
Date: Fri Sep 11 10:25:59 2015

Added new project

-fuel-infra-packages/seabios

We need update version seabios for upgrade qemu to 2.4
Partial-Bug: #1484080

Change-Id: I8558792d34836729016dda59c9e64071237287ae

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: master
Change author: Sergey Otpuschennikov <email address hidden>
Review: https://review.fuel-infra.org/11518

[Dennis Dmitriev (ddmitriev)]

qemu2.4 was testing during two weeks (see https://bugs.launchpad.net/fuel/+bug/1494278).

Any of the issues described in this bug description wasn't appeared.

qemu2.4 was working quite stable for snapshoting/reverting; for running a HA environment for days without unexpected issues; for deploying multiple environments in the same time.

I propose to update qemu to the version 2.4 on all Jenkins slaves that run tests for Fuel 8.0.

[Igor Shishkin (teran)]

Assigning back to Build team because of -2 from CI in qemu review.
So Built team work is not done yet.

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/11303
Submitter: Igor Shishkin <email address hidden>
Branch: master

Commit: ee0dfd289aeed397993e2b6f0de1b46104e62825
Author: Sergey Otpuschennikov <email address hidden>
Date: Tue Sep 8 15:20:33 2015

Upgrade packages openbios-ppc and openbios-sparc for qemu

Change-Id: I72897bd531f224e61ff00d958c2b715280cb33a5
Partial-Bug: #1484080

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/11518
Submitter: Igor Shishkin <email address hidden>
Branch: master

Commit: 3a35bac6a7007d0c65cd507cdc6fbe660619b17a
Author: Sergey Otpuschennikov <email address hidden>
Date: Mon Sep 14 07:17:08 2015

Added seabios 1.7.5 for upgrade qemu to 2.4

Change-Id: Icd85cf43dbe81355a78f271ee5959e0f418d40aa
Partial-Bug: #1484080

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/11248
Submitter: Igor Shishkin <email address hidden>
Branch: master

Commit: 02bfad12bb3c64cb462c9f7a034f99419213e278
Author: Sergey Otpuschennikov <email address hidden>
Date: Mon Oct 12 16:33:18 2015

Upgrade version to 2.1.1 for qemu 2.4

Change-Id: I29011fd0b7bf1463a40249ce6f4f2462cd19cdb0
Partial-Bug: #1484080

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/11241
Submitter: Igor Shishkin <email address hidden>
Branch: master

Commit: cc8384be6405e5aba69d46dc411e29179ccb9057
Author: Sergey Otpuschennikov <email address hidden>
Date: Mon Oct 12 15:14:30 2015

Update version to 2.4

Change-Id: I74107272e0707369f21bdb4d9968dcf68616013e
Closes-Bug: #1484080

[Yuriy Shamray (ishamrai)]

https://trello.com/c/cyJqWA5j/993-update-ci-slaves-to-qemu-2-4

[Andrey Nikitin (heos)]

We've faced the problem is described in the bug: https://bugs.launchpad.net/fuel/+bug/1506582. So, we should be sure that it's not affected our tests to start update the slaves on new version of qemu.

[Andrey Nikitin (heos)]

That's why current bug is hold on time of investigation.

[Dmitry Klenov (dklenov)]

Andrey, can you please tell the latest status? This bug is still marked as a swarm blocker.

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: master
Change author: Igor Shishkin <email address hidden>
Review: https://review.fuel-infra.org/13683

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/13683
Submitter: Igor Shishkin <email address hidden>
Branch: master

Commit: 1e2a8f7b1bc7bb5efd3afa61a350ad2da06db8ab
Author: Igor Shishkin <email address hidden>
Date: Thu Nov 5 15:34:07 2015

Adding stub for qemu=2.4 missed modprobe file

Change-Id: I05d4c3cc79b98c7d72f51cae3dd1d47c5b8acb03
Closes-Bug: #1506582
Closes-Bug: #1513394
Related-Bug: #1484080
Signed-off-by: Igor Shishkin <email address hidden>

[Mateusz Matuszkowiak (mmatuszkowiak)]

Three servers left from these connected to product-ci (3 remains of 35 in total).

srv39-bud.infra.mirantis.net
srv74-bud.infra.mirantis.net
srv75-bud.infra.mirantis.net

^^ these cannot be now upgraded because still jobs are running on them.

[Mateusz Matuszkowiak (mmatuszkowiak)]

Hi.

All of the remaining servers are done. This means all product-ci slaves are running now qemu2.4 with nested=1.

Regards,
Mateusz

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Sergey Otpuschennikov <email address hidden> on branch: master
Review: https://review.fuel-infra.org/10882