Fuel for OpenStack

Creating floating IP range with nova-network fails due to missing admin role for admin user

Series 5.1.x
Bug #1407675

Bug #1407675 reported by Andrey Sledzinskiy on 2015-01-05

This bug affects 2 people

	Status	Importance	Assigned to	Milestone
Fuel for OpenStack	Fix Released	High	Vladimir Kuklin	Fuel for OpenStack 6.1
5.0.x	Fix Committed	High	Bogdan Dobrelya	Fuel for OpenStack 5.0-updates
5.1.x	Fix Committed	High	Bogdan Dobrelya	Fuel for OpenStack 5.1.1-updates
6.0.x	Fix Committed	High	Bartłomiej Piotrowski	Fuel for OpenStack 6.0-updates
6.1.x	Fix Released	High	Vladimir Kuklin	Fuel for OpenStack 6.1

Bug Description

{

    "build_id": "2015-01-02_22-54-45",
    "ostf_sha": "249574cdda0279dc8ec4957a5979651439476e8a",
    "build_number": "40",
    "auth_required": true,
    "api": "1.0",
    "nailgun_sha": "4b325a95b0217a26a17f526cb734b3748cb03e12",
    "production": "docker",
    "fuelmain_sha": "cc47eef01622b8fdf2d8f290cb8dfb46738dc7f5",
    "astute_sha": "18be5cd3b819f3cad4c970ce5f72d3fb211a0969",
    "feature_groups": [
        "mirantis"
    ],
    "release": "6.1",
    "release_versions": {
        "2014.2-6.0": {
            "VERSION": {
                "build_id": "2015-01-02_22-54-45",
                "ostf_sha": "249574cdda0279dc8ec4957a5979651439476e8a",
                "build_number": "40",
                "api": "1.0",
                "nailgun_sha": "4b325a95b0217a26a17f526cb734b3748cb03e12",
                "production": "docker",
                "fuelmain_sha": "cc47eef01622b8fdf2d8f290cb8dfb46738dc7f5",
                "astute_sha": "18be5cd3b819f3cad4c970ce5f72d3fb211a0969",
                "feature_groups": [
                    "mirantis"
                ],
                "release": "6.1",
                "fuellib_sha": "42df19509c40e2cdc9ede9d89b42188ea27c1b7e"
            }
        }
    },
    "fuellib_sha": "42df19509c40e2cdc9ede9d89b42188ea27c1b7e"

}

Steps:
1. Create next cluster - Simple, CentOS, flat nova-network, 1 controller, 1 compute
2. Run deployment of the cluster

Actual result - deployment got stuck on 57% progress with errors in puppet log (node-2):

(/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:91:in `execute'

2015-01-03 02:27:24 ERR

(/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:137:in `run'

2015-01-03 02:27:24 ERR

(/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'

2015-01-03 02:27:24 ERR

(/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:478:in `exit_on_fail'

2015-01-03 02:27:24 ERR

(/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'

2015-01-03 02:27:24 ERR

(/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:470:in `plugin_hook'

2015-01-03 02:27:24 ERR

(/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'

2015-01-03 02:27:24 ERR

(/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:146:in `run_command'

2015-01-03 02:27:24 ERR

(/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:218:in `main'

2015-01-03 02:27:24 ERR

(/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /etc/puppet/modules/nova/lib/puppet/provider/nova_floating_range/nova_manage.rb:15:in `exists?'

2015-01-03 02:27:24 ERR

(/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /etc/puppet/modules/nova/lib/puppet/provider/nova_floating_range/nova_manage.rb:49:in `operate_range'

2015-01-03 02:27:24 ERR

(/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /usr/lib/ruby/gems/1.8/gems/openstack-1.1.2/lib/openstack/compute/connection.rb:476:in `get_floating_ips_bulk'

2015-01-03 02:27:24 ERR

(/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /usr/lib/ruby/gems/1.8/gems/openstack-1.1.2/lib/openstack/connection.rb:207:in `req'

2015-01-03 02:27:24 ERR

(/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) /usr/lib/ruby/gems/1.8/gems/openstack-1.1.2/lib/openstack/connection.rb:512:in `raise_exception'

2015-01-03 02:27:24 ERR

(/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) Could not evaluate: The server returned status 403

Logs are attached

Tags:

Revision history for this message

Andrey Sledzinskiy (asledzinskiy) wrote on 2015-01-05:

fail_error_deploy_simple_flat-2015_01_03__04_24_46.tar.gz Edit (7.6 MiB, application/x-tar)

Revision history for this message

Aleksey Kasatkin (alekseyk-ru) wrote on 2015-01-05:

Download full text (3.5 KiB)

Successfully deployed on

{"build_id": "2014-12-26_14-25-46", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "58", "auth_required": true, "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "81d38d6f2903b5a8b4bee79ca45a54b76c1361b8", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-26_14-25-46", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "58", "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "81d38d6f2903b5a8b4bee79ca45a54b76c1361b8", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "fde8ba5e11a1acaf819d402c645c731af450aff0"}}}, "fuellib_sha": "fde8ba5e11a1acaf819d402c645c731af450aff0"}

I took your network configuration:

2015-01-03 02:14:08.486 DEBUG [7fe67715f740] (logger) Request PUT /api/clusters/1/network_configuration/nova_network from 172.17.42.1:44197 {"networking_parameters": {"dns_nameservers": ["8.8.4.4", "8.8.8.8"], "net_manager": "FlatDHCPManager", "fixed_networks_vlan_start": 103, "fixed_networks_amount": 1, "floating_ranges": [["10.108.31.128", "10.108.31.254"]], "fixed_network_size": 256, "fixed_networks_cidr": "10.1.0.0/24"}, "networks": [{"name": "fixed", "ip_ranges": [], "gateway": null, "meta": {"ext_net_data": ["fixed_networks_vlan_start", "fixed_networks_amount"], "name": "fixed", "notation": null, "render_type": null, "assign_vip": false, "map_priority": 2, "use_gateway": false, "vlan_start": null, "render_addr_mask": null, "configurable": false}, "vlan_start": null, "cidr": null, "group_id": 1, "id": 5}, {"name": "public", "ip_ranges": [["10.108.31.2", "10.108.31.127"]], "gateway": "10.108.31.1", "meta": {"name": "public", "notation": "ip_ranges", "render_type": null, "assign_vip": true, "map_priority": 1, "use_gateway": true, "vlan_start": null, "render_addr_mask": "public", "cidr": "172.16.0.0/24", "configurable": true, "gateway": "172.16.0.1", "ip_range": ["172.16.0.2", "172.16.0.127"]}, "vlan_start": null, "cidr": "10.108.31.0/24", "group_id": 1, "id": 2}, {"name": "management", "ip_ranges": [["10.108.32.2", "10.108.32.254"]], "gateway": "10.108.32.1", "meta": {"name": "management", "notation": "cidr", "render_type": "cidr", "assign_vip": true, "map_priority": 2, "use_gateway": false, "vlan_start": 101, "render_addr_mask": "internal", "cidr": "192.168.0.0/24", "configurable": true}, "vlan_start": null, "cidr": "10.108.32.0/24", "group_id": 1, "id": 3}, {"name": "storage", "ip_ranges": [["10.108.34.2", "10.108.34.254"]], "gateway": "10.108.34.1", "meta": {"name": "storage", "notation": "cidr", "render_type": "cidr", "assign_vip": false, "map_priority": 2, "use_gateway": false, "vlan_start": 102, "render_addr_mask": "storage", "cidr": "192.168.1.0/24", "configurable": true}, "vlan_start": null, "cidr": "10.108.34.0/24", "group_id": 1, "id": 4}, {"name": "fuelweb_admin", "ip_ranges": [["10.108.30.3", "10.108.30.254"]], "gateway": null, "meta":...

Successfully deployed on

I took your network configuration:

and set vlan_id=222 for management network (as I have just 3 NICs per node I could not leave it untagged). Other parameters were left unchanged.

Revision history for this message

Aleksey Kasatkin (alekseyk-ru) wrote on 2015-01-05:

fuel-snapshot-2015-01-05_17-12-26.tgz Edit (5.7 MiB, application/x-tar)

Ryan Moe (rmoe) on 2015-01-05

Changed in fuel:
status:	New → Confirmed

Revision history for this message

Ryan Moe (rmoe) wrote on 2015-01-05:

nova-api returned a 403 because compute_extension:floating_ips_bulk requires the admin role. The novaSimpleFlat user (admin user from settings tab) did not have the admin role added until 7 seconds after the attempt to configure the floating IPs.

2015-01-03T02:27:22.950945+00:00 info: (/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) Starting to evaluate the resource

2015-01-03T02:27:29.742637+00:00 notice: (/Stage[main]/Keystone::Roles::Admin/Keystone_user_role[novaSimpleFlat@novaSimpleFlat]/roles) roles changed ['_member_'] to 'admin'

Bogdan Dobrelya (bogdando) on 2015-01-07

Changed in fuel:
status:	Confirmed → Triaged
tags:	added: low-hanging-fruit

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-03: Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/152674

Changed in fuel:
assignee:	Fuel Library Team (fuel-library) → Ryan Moe (rmoe)
status:	Triaged → In Progress

Revision history for this message

Stanislaw Bogatkin (sbogatkin) wrote on 2015-02-06: Re: Deployment of simple centos cluster failed with (/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) Could not evaluate: The server returned status 403

We don't support simple mode in 6.1 anymore.

Revision history for this message

Ryan Moe (rmoe) wrote on 2015-02-06:

The way to reproduce this in simple mode is simply by changing the admin username to something besides admin. This causes things to run out of order (I don't understand why) and fail because there is no dependency (in both simple and HA modes) between the admin role and floating ip pool creation.

So I have two questions:

1) Is simple mode going to be removed completely or will it stay in experimental mode as it now?
2) Is it possible that a similar small and seemingly unrelated change could cause the same failure in HA mode?

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2015-02-13:

The simple mode was deprecated for 6.0 and removed in 6.1. We should not consider the issue with deprecated deployment mode as a high one

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-16: Fix proposed to fuel-library (stable/5.0)

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/156180

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-16: Fix proposed to fuel-library (stable/5.1)

#10

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/156181

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-16: Change abandoned on fuel-library (master)

#11

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: master
Review: https://review.openstack.org/152674
Reason: Chis change is abandoned as related bug is won't fix.

Bogdan Dobrelya (bogdando) on 2015-02-20

Changed in fuel:
status:	In Progress → Won't Fix

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-23: Fix merged to fuel-library (stable/5.1)

#12

Reviewed: https://review.openstack.org/156181
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=1fb61716e1d8975e267342614044b9892319cfd2
Submitter: Jenkins
Branch: stable/5.1

commit 1fb61716e1d8975e267342614044b9892319cfd2
Author: Ryan Moe <email address hidden>
Date: Tue Feb 3 12:36:58 2015 -0800

Ensure admin role is assigned prior to creating floating range

Nova API will return a 403 if a non-admin user attempts to create
a floating IP range.

    Change-Id: I60ad318ba3bb91c863d58755f2b03e73099efc2b
    Closes-bug: #1407675
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-26: Fix merged to fuel-library (stable/5.0)

#13

Reviewed: https://review.openstack.org/156180
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=25bdfff6f25c32204ddf83dcf392ced3dd4743df
Submitter: Jenkins
Branch: stable/5.0

commit 25bdfff6f25c32204ddf83dcf392ced3dd4743df
Author: Ryan Moe <email address hidden>
Date: Tue Feb 3 12:36:58 2015 -0800

Ensure admin role is assigned prior to creating floating range

Nova API will return a 403 if a non-admin user attempts to create
a floating IP range.

    Change-Id: I60ad318ba3bb91c863d58755f2b03e73099efc2b
    Closes-bug: #1407675
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Revision history for this message

Ryan Moe (rmoe) wrote on 2015-03-02: Re: Deployment of simple centos cluster failed with (/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254]) Could not evaluate: The server returned status 403

#14

Moving to confirmed for 6.1 and 6.0 as this bug was reproduced in an HA deployment here: https://bugs.launchpad.net/fuel/+bug/1427194

Changed in fuel:
status:	Won't Fix → Confirmed

OpenStack Infra (hudson-openstack) on 2015-03-02

Changed in fuel:
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-03-05: Fix proposed to fuel-library (stable/6.0)

#15

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/161701

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-03-09: Fix merged to fuel-library (stable/6.0)

#16

Reviewed: https://review.openstack.org/161701
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=591d5d51ebccb07df6ec8030b9b664347bcf71f0
Submitter: Jenkins
Branch: stable/6.0

commit 591d5d51ebccb07df6ec8030b9b664347bcf71f0
Author: Ryan Moe <email address hidden>
Date: Tue Feb 3 12:36:58 2015 -0800

Ensure admin role is assigned prior to creating floating range

Nova API will return a 403 if a non-admin user attempts to create
a floating IP range.

    Change-Id: I60ad318ba3bb91c863d58755f2b03e73099efc2b
    Closes-bug: #1407675
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit 1fb61716e1d8975e267342614044b9892319cfd2)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-03-09: Fix proposed to fuel-library (master)

#17

Fix proposed to branch: master
Review: https://review.openstack.org/162647

Changed in fuel:
assignee:	Ryan Moe (rmoe) → Vladimir Kuklin (vkuklin)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-03-09: Change abandoned on fuel-library (master)

#18

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: master
Review: https://review.openstack.org/152674
Reason: The fix for master branch superseded by https://review.openstack.org/162647

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-03-09: Fix merged to fuel-library (master)

#19

Reviewed: https://review.openstack.org/162647
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=62e68af896887ebe18944e6a0a9721e269119ad4
Submitter: Jenkins
Branch: master

commit 62e68af896887ebe18944e6a0a9721e269119ad4
Author: Vladimir Kuklin <email address hidden>
Date: Mon Mar 9 17:53:08 2015 +0300

Ensure admin role is assigned prior to creating floating range

Nova API will return a 403 if a non-admin user attempts to create
a floating IP range.

Change-Id: I5458f9e261ecf90554db264f7b3d3a8b4f0ce468
Closes-bug: #1407675

Changed in fuel:
status:	In Progress → Fix Committed

Ryan Moe (rmoe) on 2015-03-16

summary:

- Deployment of simple centos cluster failed with
- (/Stage[main]/Osnailyfacter::Cluster_simple/Nova_floating_range[10.108.31.128-10.108.31.254])
- Could not evaluate: The server returned status 403
+ Creating floating IP range with nova-network fails due to missing admin
+ role for admin user

Revision history for this message

Anastasia Palkina (apalkina) wrote on 2015-03-23:

#20

Verified on ISO #216 for non-default and default admin username. Both deployments were successful

"build_id": "2015-03-22_22-54-44", "ostf_sha": "b4d284e9364e30bf5162975c2ba0be6ca0f14ebd", "build_number": "216", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-03-22_22-54-44", "ostf_sha": "b4d284e9364e30bf5162975c2ba0be6ca0f14ebd", "build_number": "216", "api": "1.0", "nailgun_sha": "51974b50c3961be3ed0fdc7859570db2eeb83e9c", "production": "docker", "python-fuelclient_sha": "b223dcaf5fdad2f714cd245958fefe03995d6207", "astute_sha": "4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "f52e4442df55a2b62637a2cf4038a24ba6f37b6f", "fuellib_sha": "a636c680e3c7d8cc66ed3e03645f38250beb8970"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "51974b50c3961be3ed0fdc7859570db2eeb83e9c", "production": "docker", "python-fuelclient_sha": "b223dcaf5fdad2f714cd245958fefe03995d6207", "astute_sha": "4a117a1ca6bdcc34fe4d086959ace1a6d18eeca9", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "f52e4442df55a2b62637a2cf4038a24ba6f37b6f", "fuellib_sha": "a636c680e3c7d8cc66ed3e03645f38250beb8970"