nailgun agent: segfault due to ruby-sigar bug

[Anastasia Palkina]

1. Create new environment (Ubuntu)
2. Choose Neutron, GRE
3. Choose Ceilometer
4. Add 1 controller, 1 compute, 1 cinder, 2 mongo
5. Move Storage network to eth1
6. Move Management network to eth2 and untag it
7. Deploy the environment. It was successful
8. Run OSTF tests. It was successful
9. But controller (node-1) has gone away and back online (more than 100 notifications after deployment) - see screen

From /var/log/nailgun-agent.log
http://paste.openstack.org/show/281390/

Logs are here: https://drive.google.com/a/mirantis.com/file/d/0B6SjzarTGFxaQWZULS1LSE15RE0/view?usp=sharing

"build_id": "2015-06-08_06-13-27", "build_number": "521", "release_versions": {"2014.2.2-6.1": {"VERSION": {"build_id": "2015-06-08_06-13-27", "build_number": "521", "api": "1.0", "fuel-library_sha": "f43c2ae1af3b493ee0e7810eab7bb7b50c986c7d", "nailgun_sha": "4340d55c19029394cd5610b0e0f56d6cb8cb661b", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "4fc55db0265bbf39c369df398b9dc7d6469ba13b", "astute_sha": "7766818f079881e2dbeedb34e1f67e517ed7d479", "fuel-ostf_sha": "7c938648a246e0311d05e2372ff43ef1eb2e2761", "release": "6.1", "fuelmain_sha": "bcc909ffc5dd5156ba54cae348b6a07c1b607b24"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "f43c2ae1af3b493ee0e7810eab7bb7b50c986c7d", "nailgun_sha": "4340d55c19029394cd5610b0e0f56d6cb8cb661b", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "4fc55db0265bbf39c369df398b9dc7d6469ba13b", "astute_sha": "7766818f079881e2dbeedb34e1f67e517ed7d479", "fuel-ostf_sha": "7c938648a246e0311d05e2372ff43ef1eb2e2761", "release": "6.1", "fuelmain_sha": "bcc909ffc5dd5156ba54cae348b6a07c1b607b24"

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/190225

[Vladimir Sharshov (vsharshov)]

Looks like this version of Ruby p484 has many problem with SIG-FAULT. Workaround with GC.disable has not helped.

Another similar bugs:

* https://bugs.ruby-lang.org/issues/9679
* https://communities.vmware.com/message/2349694

Looks like we have 2 possible solution:

* we can update ohai gem to 6.16 instead of 6.14;
* we can update it to Ruby 1.9.3-p545 and check it again.

After ohai update it start works fine. I rollback it to 6.14 and it is still works fine. But SIG-FAULT has very random nature.

Additional info:

Obtained service url from config file: 'http://10.20.0.2:8000/api'
/usr/lib/ruby/vendor_ruby/ohai/mash.rb:192: [BUG] rb_gc_mark(): unknown data type 0x0(0x1fbf7e0) non object
ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux]

[Andrey Maximov (maximov)]

I prefer to limit changes at this moment and update of gem looks better.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dmitry Ilyin (<email address hidden>) on branch: master
Review: https://review.openstack.org/190225

[Alexei Sheplyakov (asheplyakov)]

> * we can update it to Ruby 1.9.3-p545 and check it again

No, we can't. Rebuilding mcollective, puppet, and all other ruby software in Ubuntu is not an option.

[Alexei Sheplyakov (asheplyakov)]

> Another similar bugs:

> * https://bugs.ruby-lang.org/issues/9679

Could you elaborate, please:

a) What is exactly similar here (other than being terminated by SIGSEGV, which is pretty common consequence of memory management errors)
b) Are you sure this is ruby runtime bug (rather than bug of chef itself)?
c) Are you sure Ubuntu version of ruby 1.9.1 is the same/older?

[Alexei Sheplyakov (asheplyakov)]

Which extension, exactly?
How do I reproduce the error (a snippet of ruby code, please)?

[Alexei Sheplyakov (asheplyakov)]

I've started packaging ohai 6.16, but I don't think the update can possibly solve the problem. ohai itself is purely ruby code, so I doubt it can segfault.

[Alexei Sheplyakov (asheplyakov)]

As far as I understand the only C extensions which used by ohai are ruby-yajl, ruby-sigar.

The problem might be caused by https://github.com/hyperic/sigar/issues/28 (which is not solved yet)

[Alexei Sheplyakov (asheplyakov)]

I guess we need to wrap every sigar usage in ohai like this: https://github.com/m1093782566/bosh-cloudstack-cpi/commit/7adcca3a43fd5a63ec2297531f580f5eb32b762b

[Alexei Sheplyakov (asheplyakov)]

The following code segfaults for me:

require 'sigar'
require 'logger'

GC.disable

sigars = []

100.times do
  sigar = Sigar.new
  #sigar.logger = Logger.new(STDOUT)
  sigars << sigar
end

puts "Go"
GC.enable
GC.start

sleep 10

sigarbug.rb:17: [BUG] rb_gc_mark(): unknown data type 0x10(0x7f811c5817c8) non object
ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0004 p:---- s:0010 b:0010 l:000009 d:000009 CFUNC :start
c:0003 p:0101 s:0007 b:0007 l:0019b8 d:002550 EVAL sigarbug.rb:17
c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH
c:0001 p:0000 s:0002 b:0002 l:0019b8 d:0019b8 TOP

-- Ruby level backtrace information ----------------------------------------
sigarbug.rb:17:in `<main>'
sigarbug.rb:17:in `start'

-- C level backtrace information -------------------------------------------
/usr/lib/libruby-1.9.1.so.1.9(+0x155bb9) [0x7f811c6ddbb9] ../sysdeps/x86_64/multiarch/../strcmp.S:1007
/usr/lib/libruby-1.9.1.so.1.9(+0x5966d) [0x7f811c5e166d] vfscanf.c:2350
/usr/lib/libruby-1.9.1.so.1.9(rb_bug+0xb3) [0x7f811c5e1de3] vfscanf.c:1602
/usr/lib/libruby-1.9.1.so.1.9(+0x6ef90) [0x7f811c5f6f90] iogetdelim.c:56
/usr/lib/libruby-1.9.1.so.1.9(+0x6f0e1) [0x7f811c5f70e1] iogetdelim.c:90
/usr/lib/libruby-1.9.1.so.1.9(rb_gc+0x1a) [0x7f811c5f830a] ioseekpos.c:34
/usr/lib/libruby-1.9.1.so.1.9(rb_gc_start+0x9) [0x7f811c5f8349] ioseekpos.c:49
/usr/lib/libruby-1.9.1.so.1.9(+0x14c1d3) [0x7f811c6d41d3] ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S:1374
/usr/lib/libruby-1.9.1.so.1.9(+0x146a95) [0x7f811c6cea95] ../sysdeps/x86_64/multiarch/memcpy-ssse3.S:1444
/usr/lib/libruby-1.9.1.so.1.9(+0x149d8e) [0x7f811c6d1d8e] ../sysdeps/x86_64/multiarch/memcpy-ssse3.S:2078
/usr/lib/libruby-1.9.1.so.1.9(rb_iseq_eval_main+0x9f) [0x7f811c6da85f] ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S:2561
/usr/lib/libruby-1.9.1.so.1.9(+0x5ca1a) [0x7f811c5e4a1a] vfscanf.c:1956
/usr/lib/libruby-1.9.1.so.1.9(ruby_exec_node+0x1d) [0x7f811c5e585d] ../sysdeps/posix/tempname.c:203
/usr/lib/libruby-1.9.1.so.1.9(ruby_run_node+0x1e) [0x7f811c5e730e] vfprintf.c:1422
ruby1.9.1() [0x40086b]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f811c1e4ec5] libc-start.c:287
ruby1.9.1() [0x400899]

-- Other runtime information -----------------------------------------------

* Loaded script: sigarbug.rb

* Loaded features:

    0 enumerator.so
    1 /usr/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so
    2 /usr/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so
    3 /usr/lib/ruby/1.9.1/rubygems/defaults.rb
    4 /usr/lib/ruby/1.9.1/x86_64-linux/rbconfig.rb
    5 /usr/lib/ruby/1.9.1/rubygems/deprecate.rb
    6 /usr/lib/ruby/1.9.1/rubygems/exceptions.rb
    7 /usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb
    8 /usr/lib/ruby/1.9.1/rubygems/custom_require.rb
    9 /usr/lib/ruby/1.9.1/rubygems.rb
   10 /usr/lib/ruby/vendor_ruby/1.9.1/x86_64-linux/sigar.so
   11 /usr/lib/ruby/1.9.1/thread.rb
   12 /usr/lib/ruby/1.9.1/monitor.rb
   13 /usr/lib/ruby/1.9.1/logger.rb

* Process memory map:

00400000-00401000 r-xp 00000000 fc:00 30718 /usr/bin/ruby1.9.1
00600000-0060100...

[Alexei Sheplyakov (asheplyakov)]

The segfault can be reproduced with ruby 2.1.5 and 2.0.0.484, so upgrading ruby won't help anyway (provided that I've correctly identified the root cause).

[Ihor Kalnytskyi (ikalnytskyi)]

Guys, there's no doubts that we do have the issue. Still, the issue is rare and we didn't see it on CI, so why do we want to fix it ASAP in 6.1? It looks to me as a good candidate to be moved to 7.0, because I really don't see a *safe* fix right now.

[Alexei Sheplyakov (asheplyakov)]

The attached patch should solve the problem. However it breaks ohai unit tests (and might be totally wrong)

[Vladimir Sharshov (vsharshov)]

Alexei Sheplyakov (asheplyakov), i try to use GC.disable in main script on problem node and it has not helped. I still got this error.

In other hands after installation new version of ohai or reinstall old version of ohai problem disappear.

[Vladimir Sharshov (vsharshov)]

Hmm, let try to use patch. +1 for me. Also i suggest to merge Dmitry Ilyin changes.

[Vladimir Sharshov (vsharshov)]

I talk about this changes: https://review.openstack.org/#/c/190225/

[Alexei Sheplyakov (asheplyakov)]

GC.disable is not necessary (and it won't help since other modules can turn it on).

[Vladimir Sharshov (vsharshov)]

Reproduced again: https://bugs.launchpad.net/fuel/+bug/1475290

[Vladimir Sharshov (vsharshov)]

After discussion with Alexei Sheplyakov we need fix tests which broken if we apply patch. Assigned to myself.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dmitry Pyzhov (<email address hidden>) on branch: master
Review: https://review.openstack.org/190225
Reason: No activity for more than a month

[Vladimir Kozhukalov (kozhukalov)]

Guys, the same error was reproduced on scale lab 24 July 2015, so it is still actual. Is there anything preventing us from applying this patch with SigarBox suggested by Alexey?

[Vladimir Kozhukalov (kozhukalov)]

Another suggestion is to disable network_listeners plugin since it gathers the information about network connections and we don't need this in nailgun-agent

https://github.com/stackforge/fuel-nailgun-agent/blob/master/agent#L163

[Vladimir Sharshov (vsharshov)]

I've checked Vladimir Kozhukalov. Sorry, but this will not work, because Sugar is used in many places including CPU, Networks, Disks and in many others.

[Vladimir Sharshov (vsharshov)]

Fix tests. Now can safely apply this patch and include it in our ohai package.

[Vladimir Sharshov (vsharshov)]

Fix tests. Now can safely apply this patch and include it in our ohai package.

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: master
Change author: Alexei Sheplyakov <email address hidden>
Review: https://review.fuel-infra.org/10374

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/10374
Submitter: Andrey Nikitin <email address hidden>
Branch: master

Commit: 17a594c444a57087ccdf863b253bcdf8d3b51ffa
Author: Alexei Sheplyakov <email address hidden>
Date: Wed Aug 12 14:19:17 2015

Add ohai custom package for Ubuntu

The original ohai package suffers from a segfault in a C extension
(ruby-sigar). This segfault causes false alarms (the node goes offline
from the nailgun point of the view) and sporadic tests failures.
Apparently fixing ruby-sigar itself is rather complicated, it's much
easier to work around the problem in ohai itself. Hence a custom
package.

Change-Id: Idab24a61df562c312ea47234aa02012223b7b0a0
Related-Bug: #1463835

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 7.0
Change author: Alexei Sheplyakov <email address hidden>
Review: https://review.fuel-infra.org/10377

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/10377
Submitter: Michael Semenov <email address hidden>
Branch: 7.0

Commit: 7219d5622e11e8a49f5e4713d7405986ea63c176
Author: Alexei Sheplyakov <email address hidden>
Date: Wed Aug 12 15:34:32 2015

Work around the segfault in ruby-sigar

ruby-sigar attempts to GC a pointer which is not ruby object which yields
a segfault [1]. This segfault causes false alarms (the node goes offline
from the nailgun point of the view) and sporadic tests failures. Apparently
fixing ruby-sigar itself is rather complicated, it's much easier to work
around the problem in ohai itself, hence a custom package.

[1] https://github.com/hyperic/sigar/issues/28

The original source has been downloaded from
http://archive.ubuntu.com/ubuntu/pool/universe/o/ohai/ohai_6.14.0.orig.tar.gz
http://archive.ubuntu.com/ubuntu/pool/universe/o/ohai/ohai_6.14.0-2.debian.tar.gz

Closes-Bug: #1463835
Change-Id: Icf0da3a829863530b95ba88cd671bdc82fe08f51

[Alexei Sheplyakov (asheplyakov)]

Reassigning bug MOS 6.1 to MOS sustaining team (they are the right guys to decide if the fix needs to be backported to MOS 6.1)

[Anastasia Palkina (apalkina)]

Verified on ISO #219 availability of the needed package.
I also did not see this error on latest ISOs

"build_id": "2015-08-23_15-01-12", "build_number": "219", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "2015-08-23_15-01-12", "build_number": "219", "api": "1.0", "fuel-library_sha": "3a3ea6d9849bc1ba35c1bd882f0a0678b20d2e51", "nailgun_sha": "7790ce872512ecdf21689e6a5f970dd7119febdb", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "4c2ab9d6c623d345086c6e2874d1df81fd96a942", "production": "docker", "python-fuelclient_sha": "fc7b63aa6900fe3b2c183108ba6a13e868bc0472", "astute_sha": "53c86cba593ddbac776ce5a3360240274c20738c", "fuel-ostf_sha": "16839cbf471b7142b04c0d2c2d94786bc486fefe", "release": "7.0", "fuelmain_sha": "a494e6628319abfef57e1754f6453cf8f1a4bc65"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "3a3ea6d9849bc1ba35c1bd882f0a0678b20d2e51", "nailgun_sha": "7790ce872512ecdf21689e6a5f970dd7119febdb", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "4c2ab9d6c623d345086c6e2874d1df81fd96a942", "production": "docker", "python-fuelclient_sha": "fc7b63aa6900fe3b2c183108ba6a13e868bc0472", "astute_sha": "53c86cba593ddbac776ce5a3360240274c20738c", "fuel-ostf_sha": "16839cbf471b7142b04c0d2c2d94786bc486fefe", "release": "7.0", "fuelmain_sha": "a494e6628319abfef57e1754f6453cf8f1a4bc65"

[Anastasia Palkina (apalkina)]

Reproduced on ISO #228

"build_id": "2015-08-25_03-34-40", "build_number": "228", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "2015-08-25_03-34-40", "build_number": "228", "api": "1.0", "fuel-library_sha": "1d23fa825a73564bb6c715993ea9df8a66249a93", "nailgun_sha": "9f227f2cc0ca276715cb098c73b2bd231b103eae", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "4c2ab9d6c623d345086c6e2874d1df81fd96a942", "production": "docker", "python-fuelclient_sha": "fc7b63aa6900fe3b2c183108ba6a13e868bc0472", "astute_sha": "53c86cba593ddbac776ce5a3360240274c20738c", "fuel-ostf_sha": "3ad03d076c46347691cc3480dd19d34e37b73df4", "release": "7.0", "fuelmain_sha": "28d4bfcff7a0fb1b37504dbcac4998789df17935"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "1d23fa825a73564bb6c715993ea9df8a66249a93", "nailgun_sha": "9f227f2cc0ca276715cb098c73b2bd231b103eae", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "4c2ab9d6c623d345086c6e2874d1df81fd96a942", "production": "docker", "python-fuelclient_sha": "fc7b63aa6900fe3b2c183108ba6a13e868bc0472", "astute_sha": "53c86cba593ddbac776ce5a3360240274c20738c", "fuel-ostf_sha": "3ad03d076c46347691cc3480dd19d34e37b73df4", "release": "7.0", "fuelmain_sha": "28d4bfcff7a0fb1b37504dbcac4998789df17935"

Test case:
1. Create new environment
2. Choose Neutron, VLAN
3. Choose Ceph for volumes
4. Choose Murano
5. Add 3 controller, 1 compute3 ceph
6. Start deployment
7. Stop deployment during provisioning
8. Wait until nodes become 'Pending addition'
9. Deploy the environment again. It was successful
10. But controller (node-3) has gone away and back online many times

[Anastasia Palkina (apalkina)]

Issue is fixed but we have another problem https://bugs.launchpad.net/fuel/+bug/1488844

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 6.1
Change author: Denis Puchkin <email address hidden>
Review: https://review.fuel-infra.org/13248

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Denis Puchkin <email address hidden> on branch: 6.1
Review: https://review.fuel-infra.org/13248
Reason: MOS 6.1 is based on Ubuntu 14.04 (trusty).

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 6.1
Change author: Alexei Sheplyakov <email address hidden>
Review: https://review.fuel-infra.org/13513

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 6.1
Change author: Denis Puchkin <email address hidden>
Review: https://review.fuel-infra.org/13609

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Denis Puchkin <email address hidden> on branch: 6.1
Review: https://review.fuel-infra.org/13513

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/13609
Submitter: Vitaly Sedelnik <email address hidden>
Branch: 6.1

Commit: 54bcb1dac1c9e35ea1fbf7b88fcef96220c57841
Author: Denis Puchkin <email address hidden>
Date: Tue Nov 3 11:23:34 2015

Work around the segfault in ruby-sigar

ruby-sigar attempts to GC a pointer which is not ruby object which yields
a segfault [1]. This segfault causes false alarms (the node goes offline
from the nailgun point of the view) and sporadic tests failures. Apparently
fixing ruby-sigar itself is rather complicated, it's much easier to work
around the problem in ohai itself, hence a custom package.

[1] https://github.com/hyperic/sigar/issues/28

The original source has been downloaded from
http://archive.ubuntu.com/ubuntu/pool/universe/o/ohai/ohai_6.14.0.orig.tar.gz
http://archive.ubuntu.com/ubuntu/pool/universe/o/ohai/ohai_6.14.0-2.debian.tar.gz

This is a backport of 7219d5622e11e8a49f5e4713d7405986ea63c176 for mos6.1

Closes-Bug: #1463835
Change-Id: I8e05d0125b790c619140b4c982d3333806a4e4e2

[Denis Puchkin (dpuchkin)]

The ohai package can't be build for mos 6.1, because OBS/6.1 CI can't use 'trusty/universe' repo.
in this repo located necessary packages to meet the build dependencies

To resolve unmet dependencies issue we should re-build 7 packages from 'trusty/universe' for mos 6.1, it is :
ruby-mixlib-config
ruby-mixlib-log
ruby-mixlib-cli
ruby-systemu
ruby-yajl
ruby-sigar
ruby-ipaddress

[Alexei Sheplyakov (asheplyakov)]

> The ohai package can't be build for mos 6.1, because OBS/6.1 CI can't use 'trusty/universe' repo.

Please fix the CI to lift this silly restriction.

> To resolve unmet dependencies issue we should re-build 7 packages from 'trusty/universe' for mos 6.1

ruby-mixlib-config
ruby-mixlib-log
ruby-mixlib-cli
ruby-systemu
ruby-yajl
ruby-sigar
ruby-ipaddress

Most of ruby gems are in the `universe' section, rebuilding them to make CI happy is plain wrong.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 6.1
Change author: Denis Puchkin <email address hidden>
Review: https://review.fuel-infra.org/14441

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 6.1
Change author: Denis Puchkin <email address hidden>
Review: https://review.fuel-infra.org/14444

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 6.1
Change author: Denis Puchkin <email address hidden>
Review: https://review.fuel-infra.org/14448

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 6.1
Change author: Denis Puchkin <email address hidden>
Review: https://review.fuel-infra.org/14450

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 6.1
Change author: Denis Puchkin <email address hidden>
Review: https://review.fuel-infra.org/14451

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 6.1
Change author: Denis Puchkin <email address hidden>
Review: https://review.fuel-infra.org/14465

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Denis Puchkin <email address hidden> on branch: 6.1
Review: https://review.fuel-infra.org/14441

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Denis Puchkin <email address hidden> on branch: 6.1
Review: https://review.fuel-infra.org/14445

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Denis Puchkin <email address hidden> on branch: 6.1
Review: https://review.fuel-infra.org/14446

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Denis Puchkin <email address hidden> on branch: 6.1
Review: https://review.fuel-infra.org/14450

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Denis Puchkin <email address hidden> on branch: 6.1
Review: https://review.fuel-infra.org/14444

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Denis Puchkin <email address hidden> on branch: 6.1
Review: https://review.fuel-infra.org/14448

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Denis Puchkin <email address hidden> on branch: 6.1
Review: https://review.fuel-infra.org/14451

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Denis Puchkin <email address hidden> on branch: 6.1
Review: https://review.fuel-infra.org/14465