ERR after adding/remove controller: swift-ring-builder /etc/swift/object.builder create 10 3 1 returned 1 instead of one of [0]

[Anastasia Palkina]

1. Create new environment (by default)
2. Add 3 controllers and 2 computes
3. Start deployment. It was successful
4. Run OSTF test. It was successful
5. Add 1 cinder
6. Redeploy environment. It was successful
7. Run OSTF test. It was successful
8. Remove the 'cinder' node and remove a 'controller' node
9. Add 'controller'+'cinder' node
10. Start deployment. It was successful
11. There is error on primary controller (node-6):

2015-11-19 16:14:00 ERR (/Stage[main]/Swift::Ringbuilder/Swift::Ringbuilder::Create[object]/Exec[create_object]/returns) change from notrun to 0 failed: swift-ring-builder /etc/swift/object.builder create 10 3 1 returned 1 instead of one of [0]

12. Start OSTF tests. Test "Check that required services are running" has failed.

root@node-6:~# nova-manage service list
No handlers could be found for logger "oslo_config.cfg"
Binary Host Zone Status State Updated_At
nova-cert node-5.domain.tld internal enabled XXX 2015-11-19 15:09:12
nova-consoleauth node-5.domain.tld internal enabled XXX 2015-11-19 15:08:33
nova-scheduler node-5.domain.tld internal enabled XXX 2015-11-19 15:08:54
nova-conductor node-5.domain.tld internal enabled XXX 2015-11-19 15:09:17
nova-cert node-6.domain.tld internal enabled XXX 2015-11-19 17:13:57
nova-consoleauth node-6.domain.tld internal enabled XXX 2015-11-20 03:29:06
nova-scheduler node-6.domain.tld internal enabled XXX 2015-11-19 17:13:50
nova-conductor node-6.domain.tld internal enabled XXX 2015-11-19 17:44:22
nova-cert node-7.domain.tld internal enabled XXX 2015-11-19 17:14:00
nova-consoleauth node-7.domain.tld internal enabled XXX 2015-11-20 03:29:08
nova-scheduler node-7.domain.tld internal enabled XXX 2015-11-20 03:29:17
nova-conductor node-7.domain.tld internal enabled XXX 2015-11-19 17:44:21
nova-compute node-2.domain.tld nova enabled XXX 2015-11-20 03:29:15
nova-compute node-1.domain.tld nova enabled :-) 2015-11-20 12:31:02
nova-cert node-9.domain.tld internal enabled XXX 2015-11-19 17:32:46
nova-scheduler node-9.domain.tld internal enabled :-) 2015-11-20 12:31:13
root@node-6:~#

Controllers: node-6, 7
Controller+cinder: node-9
Computes: node-1,2

Logs are here: https://drive.google.com/a/mirantis.com/file/d/0B6SjzarTGFxaeVFrc3RoanhPamc/view?usp=sharing

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  openstack_version: "2015.1.0-8.0"
  api: "1.0"
  build_number: "180"
  build_id: "180"
  fuel-nailgun_sha: "e5600e8a87745323c2a9c524b7c0647c9829ebb9"
  python-fuelclient_sha: "e685d68c1c0d0fa0491a250f07d9c3a8d0f9608c"
  fuel-agent_sha: "6faa1e0ba836ef114b3c1a6c4d12469fc66ae402"
  fuel-nailgun-agent_sha: "db1738a65012d2a1d2b7e83cc2a44a196e4290b4"
  astute_sha: "c8400f51b0b92254da206de55ef89d17fdf35393"
  fuel-library_sha: "f9281f9c3b08ed16b2fb411e30f5d48809446030"
  fuel-ostf_sha: "7e24fc802a95d2f0627512e585f8977f587aea18"
  fuel-createmirror_sha: "22a7aacd95bbdca69f9e0f08b70facabdec8fb28"
  fuelmenu_sha: "d12061b1aee82f81b3d074de74ea27a6e962a686"
  shotgun_sha: "c377d163519f6d10b69a654019d6086ba5f14edc"
  network-checker_sha: "a57e1d69acb5e765eb22cab0251c589cd76f51da"
  fuel-upgrade_sha: "1e894e26d4e1423a9b0d66abd6a79505f4175ff6"
  fuelmain_sha: "cd084cf5c4372a46184fb7c2f24568da4e030be2"

[Kyrylo Galanov (kgalanov)]

Hi,

The error was caused by file access error. That needs to be investigated.
http://paste.openstack.org/show/479599/

However, next try was successful
http://paste.openstack.org/show/479601/

--
Kyrylo

[Kyrylo Galanov (kgalanov)]

Related bug: https://bugs.launchpad.net/fuel/+bug/1405755

[Kyrylo Galanov (kgalanov)]

It's a regression of https://review.openstack.org/#/c/144127/1

[Kyrylo Galanov (kgalanov)]

A patch submitted to upstream https://review.openstack.org/#/c/248150/

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: mos-8.0
Change author: Kyrylo Galanov <email address hidden>
Review: https://review.fuel-infra.org/14325

[Ivan Berezovskiy (iberezovskiy)]

https://review.fuel-infra.org/#/c/14325/

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/14325
Submitter: Sergey Kolekonov <email address hidden>
Branch: mos-8.0

Commit: bec2e750a0887b28d27e21f59f3a0d0d544cfd19
Author: Kyrylo Galanov <email address hidden>
Date: Tue Dec 1 12:36:12 2015

Create ring builders under swift user

Since we're running ring rebalance under 'swift' user, we also
need to create ring builders under the same user.

Change-Id: I04972e883b8f5f6355737545525f6dc32582563d
Related-bug: #1518305

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/252394

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ivan Berezovskiy (<email address hidden>) on branch: master
Review: https://review.openstack.org/252394

[Denis Egorenko (degorenko)]

This bug don't needed fix for fuel-library, only a new tag for Librarian. It was done in separate commit: https://review.openstack.org/#/c/248147/

[Ivan Berezovskiy (iberezovskiy)]

First run of swift-ring-builder was with error, but second was ok. It doesn't break cluster or swift deployment. that's why priority is medium

[Nastya Urlapova (aurlapova)]

This issue can has Medium status, because it is related to cluster scalability.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/252938
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=04693990cb186a50fca817ff701160c924347983
Submitter: Jenkins
Branch: master

commit 04693990cb186a50fca817ff701160c924347983
Author: Alexey Stepanov <email address hidden>
Date: Mon Nov 2 16:39:38 2015 +0300

    Test for postdeploy plugin install and enable.

    Related-bug: #1519050
    Related-bug: #1518305
    Closes-bug: #1527123

    Change-Id: I54418e328049e5a885cad783a3c3941996334005

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/261003

[Denis Egorenko (degorenko)]

Can't reproduce this bug.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "429"
  build_id: "429"
  fuel-nailgun_sha: "12b15b2351e250af41cc0b10d63a50c198fe77d8"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "df16d41cd7a9445cf82ad9fd8f0d53824711fcd8"
  fuel-nailgun-agent_sha: "92ebd5ade6fab60897761bfa084aefc320bff246"
  astute_sha: "c7ca63a49216744e0bfdfff5cb527556aad2e2a5"
  fuel-library_sha: "3eaf4f4a9b88b287a10cc19e9ce6a62298cc4013"
  fuel-ostf_sha: "214e794835acc7aa0c1c5de936e93696a90bb57a"
  fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
  fuelmenu_sha: "85de57080a18fda18e5325f06eaf654b1b931592"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "e8e36cff332644576d7853c80b8a53d5b955420a"

[Anastasia Palkina (apalkina)]

Also can't reproduce on ISO #429

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "429"
  build_id: "429"
  fuel-nailgun_sha: "12b15b2351e250af41cc0b10d63a50c198fe77d8"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "df16d41cd7a9445cf82ad9fd8f0d53824711fcd8"
  fuel-nailgun-agent_sha: "92ebd5ade6fab60897761bfa084aefc320bff246"
  astute_sha: "c7ca63a49216744e0bfdfff5cb527556aad2e2a5"
  fuel-library_sha: "3eaf4f4a9b88b287a10cc19e9ce6a62298cc4013"
  fuel-ostf_sha: "214e794835acc7aa0c1c5de936e93696a90bb57a"
  fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
  fuelmenu_sha: "85de57080a18fda18e5325f06eaf654b1b931592"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "e8e36cff332644576d7853c80b8a53d5b955420a"

[ElenaRossokhina (esolomina)]

Bug was reproduced
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "478"
  build_id: "478"

1. Deploy any environment with 3 controllers, 1 compute and NeutronTUN or NeutronVLAN
2. Shut down the primary controller node, then remove it from the cluste
3. Add another node to the cluster as the third controller
4. Deploy changes
5. Run OSTF

Step#5 fails:
Check that required services are running
Some nova services have not been started.. Please refer to OpenStack logs for more details.

nova-manage service list
No handlers could be found for logger "oslo_config.cfg"
Binary Host Zone Status State Updated_At
nova-scheduler node-1.test.domain.local internal enabled XXX 2016-01-27 09:08:40
nova-cert node-1.test.domain.local internal enabled XXX 2016-01-27 09:08:59
nova-conductor node-1.test.domain.local internal enabled XXX 2016-01-27 09:08:53
nova-consoleauth node-1.test.domain.local internal enabled XXX 2016-01-27 09:08:59
nova-consoleauth node-2.test.domain.local internal enabled :-) 2016-01-27 12:25:47
nova-scheduler node-2.test.domain.local internal enabled :-) 2016-01-27 12:25:47
nova-conductor node-2.test.domain.local internal enabled :-) 2016-01-27 12:25:47
nova-cert node-2.test.domain.local internal enabled :-) 2016-01-27 12:25:42
nova-consoleauth node-3.test.domain.local internal enabled :-) 2016-01-27 12:25:50
nova-scheduler node-3.test.domain.local internal enabled :-) 2016-01-27 12:25:43
nova-conductor node-3.test.domain.local internal enabled :-) 2016-01-27 12:25:54
nova-cert node-3.test.domain.local internal enabled :-) 2016-01-27 12:25:45
nova-compute node-4.test.domain.local nova enabled :-) 2016-01-27 12:25:11
nova-consoleauth node-5.test.domain.local internal enabled :-) 2016-01-27 12:25:14
nova-scheduler node-5.test.domain.local internal enabled :-) 2016-01-27 12:25:14
nova-conductor node-5.test.domain.local internal enabled :-) 2016-01-27 12:25:15
nova-cert node-5.test.domain.local internal enabled :-) 2016-01-27 12:25:17

here is snapshot https://drive.google.com/a/mirantis.com/file/d/0B2ag_Bf-ShtTOV9BMG45ckFUakE/view?usp=sharing

[Denis Egorenko (degorenko)]

I don't see any errors in puppet logs on every controller node, so if there is problem only with OSTF it is not related to this bug actually.

The problem is that after removing some controller node from cluster we don't clean nova db and there still exists services from removed controller.

[Denis Egorenko (degorenko)]

Well, i've missed error in logs. It is really present.

Here is short log from puppet:

http://paste.openstack.org/show/485164/

After two first tries we have error:

2016-01-27 09:49:19 +0000 Exec[create_object](provider=posix) (debug): Executing 'swift-ring-builder /etc/swift/object.builder create 10 3 1'
2016-01-27 09:49:19 +0000 Puppet (debug): Executing 'swift-ring-builder /etc/swift/object.builder create 10 3 1'
2016-01-27 09:49:19 +0000 /Stage[main]/Swift::Ringbuilder/Swift::Ringbuilder::Create[object]/Exec[create_object]/returns (notice): Traceback (most recent call last):
2016-01-27 09:49:19 +0000 /Stage[main]/Swift::Ringbuilder/Swift::Ringbuilder::Create[object]/Exec[create_object]/returns (notice): File "/usr/bin/swift-ring-builder", line 24, in <module>
2016-01-27 09:49:19 +0000 /Stage[main]/Swift::Ringbuilder/Swift::Ringbuilder::Create[object]/Exec[create_object]/returns (notice): sys.exit(main())
2016-01-27 09:49:19 +0000 /Stage[main]/Swift::Ringbuilder/Swift::Ringbuilder::Create[object]/Exec[create_object]/returns (notice): File "/usr/lib/python2.7/dist-packages/swift/cli/ringbuilder.py", line 1187, in main
2016-01-27 09:49:19 +0000 /Stage[main]/Swift::Ringbuilder/Swift::Ringbuilder::Create[object]/Exec[create_object]/returns (notice): Commands.__dict__.get(command, Commands.unknown.im_func)()
2016-01-27 09:49:19 +0000 /Stage[main]/Swift::Ringbuilder/Swift::Ringbuilder::Create[object]/Exec[create_object]/returns (notice): File "/usr/lib/python2.7/dist-packages/swift/cli/ringbuilder.py", line 413, in create
2016-01-27 09:49:19 +0000 /Stage[main]/Swift::Ringbuilder/Swift::Ringbuilder::Create[object]/Exec[create_object]/returns (notice): '%d.' % time() + basename(builder_file)))
2016-01-27 09:49:19 +0000 /Stage[main]/Swift::Ringbuilder/Swift::Ringbuilder::Create[object]/Exec[create_object]/returns (notice): File "/usr/lib/python2.7/dist-packages/swift/common/ring/builder.py", line 1572, in save
2016-01-27 09:49:19 +0000 /Stage[main]/Swift::Ringbuilder/Swift::Ringbuilder::Create[object]/Exec[create_object]/returns (notice): with open(builder_file, 'wb') as f:
2016-01-27 09:49:19 +0000 /Stage[main]/Swift::Ringbuilder/Swift::Ringbuilder::Create[object]/Exec[create_object]/returns (notice): IOError: [Errno 13] Permission denied: '/etc/swift/backups/1453888159.object.builder'
2016-01-27 09:49:19 +0000 Puppet (err): swift-ring-builder /etc/swift/object.builder create 10 3 1 returned 1 instead of one of [0]

And then we have changing permissions on backups directory:

2016-01-27 09:49:20 +0000 /Stage[main]/Openstack::Swift::Proxy/File[/etc/swift/backups] (info): Starting to evaluate the resource
2016-01-27 09:49:20 +0000 /Stage[main]/Openstack::Swift::Proxy/File[/etc/swift/backups]/mode (notice): mode changed '0644' to '2770'
2016-01-27 09:49:20 +0000 /Stage[main]/Openstack::Swift::Proxy/File[/etc/swift/backups] (debug): The container Class[Openstack::Swift::Proxy] will propagate my refresh event
2016-01-27 09:49:20 +0000 /Stage[main]/Openstack::Swift::Proxy/File[/etc/swift/backups] (info): Evaluated in 0.00 seconds

[Denis Egorenko (degorenko)]

Looks like we have executing of ringbuild::create [1,2] before changing permissions [3]:

[1] https://github.com/openstack/puppet-swift/blob/master/manifests/ringbuilder/create.pp#L47
[2] https://github.com/openstack/fuel-library/blob/master/deployment/puppet/openstack/manifests/swift/proxy.pp#L188
[3] https://github.com/openstack/fuel-library/blob/master/deployment/puppet/openstack/manifests/swift/proxy.pp#L70-L77

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/273152

[Denis Egorenko (degorenko)]

As was discussed with library folks, this bug should be fixed on package side: add this directory directly to swift package.

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Mikhail Ivanov <email address hidden>
Review: https://review.fuel-infra.org/16563

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/16551
Submitter: Pkgs Jenkins <email address hidden>
Branch: master

Commit: 993719a2c63440bc9be549f9023b748666677ab2
Author: Mikhail Ivanov <email address hidden>
Date: Fri Jan 29 16:00:03 2016

Add 'backups' folder to swift package

Add folder with proper permissions. This is necessary
because without this folder there are errors with the file access

Related-Bug:#1518305

Change-Id: Ieefc546189ea96b4c64b9fa6243293ed4790d376

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Mikhail Ivanov <email address hidden>
Review: https://review.fuel-infra.org/16574

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Mikhail Ivanov <email address hidden> on branch: openstack-ci/fuel-8.0/liberty
Review: https://review.fuel-infra.org/16563
Reason: Moved to: https://review.fuel-infra.org/#/c/16574/

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/16574
Submitter: Pkgs Jenkins <email address hidden>
Branch: openstack-ci/fuel-8.0/liberty

Commit: 30386b04b176efaec73b62789cbb3ad2d26a2b67
Author: Mikhail Ivanov <email address hidden>
Date: Mon Feb 1 10:56:35 2016

Add 'backups' folder to swift package

Add folder with proper permissions. This is necessary
because without this folder there are errors with the file access

Related-Bug:#1518305

Change-Id: Ieefc546189ea96b4c64b9fa6243293ed4790d376

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/273152
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=c6307068fe8f0e317a32cb88e4c1e26d6258aec7
Submitter: Jenkins
Branch: master

commit c6307068fe8f0e317a32cb88e4c1e26d6258aec7
Author: Denis Egorenko <email address hidden>
Date: Wed Jan 27 19:59:23 2016 +0300

    Remove puppet resource for '/etc/swift/backups' directory

    This directory is maintained by Swift package now, hence no need
    to manage it through puppet.

    Change-Id: Icb00e509907b09eff37adc2c8eaae09fbf7c4b7b
    Closes-bug: #1518305

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/275082

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/275082
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=c9106adb30bea0b5d46c6c3472f29d1afe1090ad
Submitter: Jenkins
Branch: stable/8.0

commit c9106adb30bea0b5d46c6c3472f29d1afe1090ad
Author: Denis Egorenko <email address hidden>
Date: Wed Jan 27 19:59:23 2016 +0300

    Remove puppet resource for '/etc/swift/backups' directory

    This directory is maintained by Swift package now, hence no need
    to manage it through puppet.

    Change-Id: Icb00e509907b09eff37adc2c8eaae09fbf7c4b7b
    Closes-bug: #1518305
    (cherry picked from commit c6307068fe8f0e317a32cb88e4c1e26d6258aec7)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/276853

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/276853
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=1a5e0e726f83687f5c14b9ae0e616e3acb83b236
Submitter: Jenkins
Branch: master

commit 1a5e0e726f83687f5c14b9ae0e616e3acb83b236
Author: Denis Egorenko <email address hidden>
Date: Fri Feb 5 20:48:33 2016 +0300

    Add a new swift_backups section in rsync conf

    On primary controller we are executing all stuff (like ring create, rebalance and etc).
    Swift Ring Builder requires folder '/etc/swift/backups' with proper permissions (2770).

    On another controllers, we just run rsync from primary, for '/etc/swift/backups' directory.
    The problem is that we are using settings from swift_server section (which comes from
    swift upstream module). This section has default incoming_chmod/outgoing_chmod = '0644'.
    Hense all another controllers have these permissions for '/etc/swift/backups'.
    In case, when we want to destroy old primary + add new controller - we faced with wrong
    permissions for '/etc/swift/backups' on all controller nodes.

    Change-Id: I47ee344e6ea6d65eb984941287ec9c0ed3e28a5b
    Related-bug: #1518305
    Closes-bug: #1542281

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/8.0
Review: https://review.openstack.org/277505

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/277505
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=abd2eea1c620e0ce72c24c711dcbbc5cf8c224b5
Submitter: Jenkins
Branch: stable/8.0

commit abd2eea1c620e0ce72c24c711dcbbc5cf8c224b5
Author: Denis Egorenko <email address hidden>
Date: Fri Feb 5 20:48:33 2016 +0300

    Add a new swift_backups section in rsync conf

    On primary controller we are executing all stuff (like ring create, rebalance and etc).
    Swift Ring Builder requires folder '/etc/swift/backups' with proper permissions (2770).

    On another controllers, we just run rsync from primary, for '/etc/swift/backups' directory.
    The problem is that we are using settings from swift_server section (which comes from
    swift upstream module). This section has default incoming_chmod/outgoing_chmod = '0644'.
    Hense all another controllers have these permissions for '/etc/swift/backups'.
    In case, when we want to destroy old primary + add new controller - we faced with wrong
    permissions for '/etc/swift/backups' on all controller nodes.

    Change-Id: I47ee344e6ea6d65eb984941287ec9c0ed3e28a5b
    Related-bug: #1518305
    Closes-bug: #1542281
    (cherry picked from commit 1a5e0e726f83687f5c14b9ae0e616e3acb83b236)

[Evgeny Sikachev (esikachev)]

verified on 529 iso

[Sofiia Andriichenko (sandriichenko)]

OSTF tests "Check that required services are running" has failed

After "nova-manage service list" system freezes
root@node-13:~# nova-manage service list
Option "verbose" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future.
Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications".
Option "notification_topics" from group "DEFAULT" is deprecated. Use option "topics" from group "oslo_messaging_notifications".
DEPRECATED: Use the nova service-* commands from python-novaclient instead or the os-services REST resource. The service subcommand will be removed in the 14.0 release.
^C

[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 101
cat /etc/fuel_build_number:
 101
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 liberty-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6316.noarch
 rubygem-astute-9.0.0-1.mos724.noarch
 fuel-library9.0-9.0.0-1.mos8173.noarch
 shotgun-9.0.0-1.mos85.noarch
 fuel-agent-9.0.0-1.mos264.noarch
 fuel-ui-9.0.0-1.mos2604.noarch
 fuel-nailgun-9.0.0-1.mos8561.noarch
 fuel-migrate-9.0.0-1.mos8173.noarch
 nailgun-mcagents-9.0.0-1.mos724.noarch
 fuel-misc-9.0.0-1.mos8173.noarch
 python-packetary-9.0.0-1.mos128.noarch
 fuel-bootstrap-cli-9.0.0-1.mos264.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8561.noarch
 network-checker-9.0.0-1.mos72.x86_64
 fuel-mirror-9.0.0-1.mos128.noarch
 fuel-openstack-metadata-9.0.0-1.mos8561.noarch
 fuel-notify-9.0.0-1.mos8173.noarch
 fuel-setup-9.0.0-1.mos6316.noarch
 python-fuelclient-9.0.0-1.mos292.noarch
 fuelmenu-9.0.0-1.mos263.noarch
 fuel-9.0.0-1.mos6316.noarch
 fuel-utils-9.0.0-1.mos8173.noarch
 fuel-ostf-9.0.0-1.mos919.noarch
[root@nailgun ~]#

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Denis Egorenko (degorenko)]

Sofiia, i don't see any reason for reopening this bug with your report. Provide please more info.