Keystone doen't work through public VIP

[Roger Törnström]

Cloud deployed with MOS 6.0 in HA mode with 3 controllers.

It is not possible to use the keystone command from outside through the management VIP address. All other openstack CLI-commands works. It is possible to use the keystone command from inside the cloud on a controller. So it appears that something is misconfigured in keystone since it looks like the internal IP is exposed to the outside.

This is what happens from the outside (OS_AUTH_URL=http://10.80.84.4:5000/v2.0):

$ keystone --debug tenant-list
DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://10.80.84.4:5000/v2.0/tokens
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 10.80.84.4
DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 4366
DEBUG:keystoneclient.session:REQ: curl -g -i -X GET http://192.168.0.2:35357/v2.0/tenants -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}1362926ee9e5652dcf8f19b14b70820bd6aee8ad"
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 192.168.0.2
Unable to establish connection to http://192.168.0.2:35357/v2.0/tenants

If I source the openrc file on a controller and run the same command it works (OS_AUTH_URL=http://192.168.0.2:5000/v2.0/):

:~# keystone tenant-list
+----------------------------------+-----------+---------+
| id | name | enabled |
+----------------------------------+-----------+---------+
| 1ab2038ab4554253b6e188cad8fe3ba1 | admin | True |
| df61f8c5b8a3455d83d6685e59fbf985 | tenant-1 | True |
| a11522807d4c4c02a1e6e399816c9825 | tenant-2 | True |
| 7cf7f21595534c96a72e19bbc3605f70 | services | True |
+----------------------------------+-----------+---------+

# fuel --fuel-version
api: '1.0'
astute_sha: 16b252d93be6aaa73030b8100cf8c5ca6a970a91
auth_required: true
build_id: 2014-12-26_14-25-46
build_number: '58'
feature_groups:
- mirantis
fuellib_sha: fde8ba5e11a1acaf819d402c645c731af450aff0
fuelmain_sha: 81d38d6f2903b5a8b4bee79ca45a54b76c1361b8
nailgun_sha: 5f91157daa6798ff522ca9f6d34e7e135f150a90
ostf_sha: a9afb68710d809570460c29d6c3293219d3624d4
production: docker
release: '6.0'
release_versions:
  2014.2-6.0:
    VERSION:
      api: '1.0'
      astute_sha: 16b252d93be6aaa73030b8100cf8c5ca6a970a91
      build_id: 2014-12-26_14-25-46
      build_number: '58'
      feature_groups:
      - mirantis
      fuellib_sha: fde8ba5e11a1acaf819d402c645c731af450aff0
      fuelmain_sha: 81d38d6f2903b5a8b4bee79ca45a54b76c1361b8
      nailgun_sha: 5f91157daa6798ff522ca9f6d34e7e135f150a90
      ostf_sha: a9afb68710d809570460c29d6c3293219d3624d4
      production: docker
      release: '6.0'

[Bogdan Dobrelya (bogdando)]

Please try to rich the keystone endpoint via public VIP instead, would it have succeeded?

[Roger Törnström (roger-tornstrom)]

That is what I used, i.e. management VIP is public VIP. I just called it management, sorry for the confusion :)

[Bogdan Dobrelya (bogdando)]

Could you provide a diagnostic logs snapshot?

[Roger Törnström (roger-tornstrom)]

I will see if we can get the logs.

But FYI we have just installed another cloud, without HA, and the same issue is present there. Keystone works locally on the controller but not from the outside through the public ip.

:~$ keystone --debug endpoint-list
DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://10.80.84.132:5000/v2.0/tokens
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 10.80.84.132
DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 3724
DEBUG:keystoneclient.session:REQ: curl -g -i -X GET http://192.168.0.4:35357/v2.0/endpoints -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}f2480ee157fcc6a010c67b5c75ef1b93cd7daea4"
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 192.168.0.4
Unable to establish connection to http://192.168.0.4:35357/v2.0/endpoints

Can you run keystone commands towards a real cloud environment deployed by MOS 6.0? It sure looks like something is not configured properly by Fuel.

[Bogdan Dobrelya (bogdando)]

Thank you for clarification @Roger, if it possible please attach logs from any type of affected environment. I'm also asking our QA team to reproduce this.

[Roger Törnström (roger-tornstrom)]

I tried to generate a diagnostic snapshot from Fuel but when I went back to check this is printed next to the button in Fuel:

"exit code 139: stderr: sh: line 1: 408 Segmentation fault shotgun -c /tmp/dump_config >> /var/log/dump.log 2>&1"

And in /var/log/messages I see this:

Mar 16 09:24:35 fuel kernel: shotgun[20256]: segfault at 58 ip 00007f6690628ee5 sp 00007f65e6de9770 error 4 in libpython2.6.so.1.0[7f6690558000+15d000]

So it doesn't look like I can provide the requested logs.

If you have specific logs or other files that you want to see I can probably provide them.

[Bogdan Dobrelya (bogdando)]

This bug could be a dup of https://bugs.launchpad.net/mos/+bug/1434088

@Roger, could you check which default route is specified for controller nodes inside of the haproxy namspace? Does it go via 240.0.0.1? For example:
root@node-1:~# ip netns exec haproxy ip ro
default via 240.0.0.1 dev hapr-p

Would the issue disappear if you replaced this route via your gateway (see grep gateway /etc/astute.yaml) ?

[Roger Törnström (roger-tornstrom)]

This is how it looks like on the HA-cloud:

root@node-84:~# ip netns exec haproxy ip ro
default dev hapr-p scope link metric 10
default via 240.0.0.1 dev hapr-ns metric 10000
10.80.84.0/28 dev hapr-p proto kernel scope link src 10.80.84.4
240.0.0.0/30 dev hapr-ns proto kernel scope link src 240.0.0.2

root@node-84:~# grep gateway /etc/astute.yaml
        gateway: 192.168.111.1
        gateway: 10.80.84.1
      gateway: 10.80.84.1
      default_gateway: true

But in our case the public VIP works fine. Horizon works fine, other CLI
commands (nova, neutron etc.). It is just the keystone command that isn't
working. And the same issue exists in a non-HA cloud. So it looks like
something that is specific to keystone. I haven't had much time to look
further into how keystone is configured but i'm suspecting that the problem
lies there somewhere.

/Roger

2015-03-20 14:56 GMT+01:00 Bogdan Dobrelya <email address hidden>:

> This bug could be a dup of https://bugs.launchpad.net/mos/+bug/1434088
>
> @Roger, could you check which default route is specified for controller
> nodes inside of the haproxy namspace? Does it go via 240.0.0.1? For example:
> root@node-1:~# ip netns exec haproxy ip ro
> default via 240.0.0.1 dev hapr-p
>
> Would the issue disappear if you replaced this route via your gateway
> (see grep gateway /etc/astute.yaml) ?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1430309
>
> Title:
> Keystone doen't work through public VIP
>
> Status in Fuel: OpenStack installer that works:
> Incomplete
>
> Bug description:
> Cloud deployed with MOS 6.0 in HA mode with 3 controllers.
>
> It is not possible to use the keystone command from outside through
> the management VIP address. All other openstack CLI-commands works. It
> is possible to use the keystone command from inside the cloud on a
> controller. So it appears that something is misconfigured in keystone
> since it looks like the internal IP is exposed to the outside.
>
> This is what happens from the outside
> (OS_AUTH_URL=http://10.80.84.4:5000/v2.0):
>
> $ keystone --debug tenant-list
> DEBUG:keystoneclient.auth.identity.v2:Making authentication request to
> http://10.80.84.4:5000/v2.0/tokens
> INFO:requests.packages.urllib3.connectionpool:Starting new HTTP
> connection (1): 10.80.84.4
> DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens
> HTTP/1.1" 200 4366
> DEBUG:keystoneclient.session:REQ: curl -g -i -X GET
> http://192.168.0.2:35357/v2.0/tenants -H "User-Agent:
> python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token:
> {SHA1}1362926ee9e5652dcf8f19b14b70820bd6aee8ad"
> INFO:requests.packages.urllib3.connectionpool:Starting new HTTP
> connection (1): 192.168.0.2
> Unable to establish connection to http://192.168.0.2:35357/v2.0/tenants
>
> If I source the openrc file on a controller and run the same command
> it works (OS_AUTH_URL=http://192.168.0.2:5000/v2.0/):
>
> :~# keystone tenant-list
> +----------------------------------+-----------+---------+
> | id | name | enabled |
> +----...

[Vladimir Kuklin (vkuklin)]

Roger, are you trying to connect to keystone using Public IP or Management IP? Can you post info about telnet <IP>:5000 and <IP>:35357 commands output?

[Roger Törnström (roger-tornstrom)]

In case of HA it's the public VIP, in non-HA the public IP of the
controller.

As I said, all other CLI commands (nova, neutron, glance, cinder... etc.)
works fine. It's just the keystone command that doesn't work. It looks like
this command is exposing the internal IP to the client outside. It only
works if you run it internally in the cloud on a controller by sourcing
openrc.

Anyway....here is the output you requested:

:~$ telnet 10.80.84.4 5000 -e §
Telnet escape character is '\247'.
Trying 10.80.84.4...
Connected to 10.80.84.4.
Escape character is '\247'.
HTTP/1.0 408 Request Time-out
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html><body><h1>408 Request Time-out</h1>
Your browser didn't send a complete request in time.
</body></html>
Connection closed by foreign host.

:~$ telnet 10.80.84.4 35357 -e §
Telnet escape character is '\247'.
Trying 10.80.84.4...
Connected to 10.80.84.4.
Escape character is '\247'.
HTTP/1.0 408 Request Time-out
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html><body><h1>408 Request Time-out</h1>
Your browser didn't send a complete request in time.
</body></html>
Connection closed by foreign host.

/Roger

2015-03-30 18:56 GMT+02:00 Vladimir Kuklin <email address hidden>:

> Roger, are you trying to connect to keystone using Public IP or
> Management IP? Can you post info about telnet <IP>:5000 and <IP>:35357
> commands output?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1430309
>
> Title:
> Keystone doen't work through public VIP
>
> Status in Fuel: OpenStack installer that works:
> Incomplete
>
> Bug description:
> Cloud deployed with MOS 6.0 in HA mode with 3 controllers.
>
> It is not possible to use the keystone command from outside through
> the management VIP address. All other openstack CLI-commands works. It
> is possible to use the keystone command from inside the cloud on a
> controller. So it appears that something is misconfigured in keystone
> since it looks like the internal IP is exposed to the outside.
>
> This is what happens from the outside
> (OS_AUTH_URL=http://10.80.84.4:5000/v2.0):
>
> $ keystone --debug tenant-list
> DEBUG:keystoneclient.auth.identity.v2:Making authentication request to
> http://10.80.84.4:5000/v2.0/tokens
> INFO:requests.packages.urllib3.connectionpool:Starting new HTTP
> connection (1): 10.80.84.4
> DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens
> HTTP/1.1" 200 4366
> DEBUG:keystoneclient.session:REQ: curl -g -i -X GET
> http://192.168.0.2:35357/v2.0/tenants -H "User-Agent:
> python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token:
> {SHA1}1362926ee9e5652dcf8f19b14b70820bd6aee8ad"
> INFO:requests.packages.urllib3.connectionpool:Starting new HTTP
> connection (1): 192.168.0.2
> Unable to establish connection to http://192.168.0.2:35357/v2.0/tenants
>
> If I source the openrc file on a controller and run the same command
> it works (OS_AUTH_URL=http://192.168.0.2:5000/v2.0/):
>
> :~# keystone tenant-list
> +----------------------------------+-----------+---------+
> ...

[Vladimir Kuklin (vkuklin)]

@Roger

can you post output of command:

`export; keystone --debug tenant-list`

It seems you might have been using some wrong environment variables which keystone client is consuming. For example, endpoint URL

[Roger Törnström (roger-tornstrom)]

Here is the output you requested:

:~$ export; keystone --debug tenant-list
declare -x HOME="/home/nilrog"
declare -x LANG="en_US.UTF-8"
declare -x LC_ADDRESS="sv_SE.UTF-8"
declare -x LC_COLLATE="C"
declare -x LC_IDENTIFICATION="sv_SE.UTF-8"
declare -x LC_MEASUREMENT="sv_SE.UTF-8"
declare -x LC_MESSAGES="C"
declare -x LC_MONETARY="sv_SE.UTF-8"
declare -x LC_NAME="sv_SE.UTF-8"
declare -x LC_NUMERIC="sv_SE.UTF-8"
declare -x LC_PAPER="sv_SE.UTF-8"
declare -x LC_TELEPHONE="sv_SE.UTF-8"
declare -x LC_TIME="sv_SE.UTF-8"
declare -x LESSCLOSE="/usr/bin/lesspipe %s %s"
declare -x LESSOPEN="| /usr/bin/lesspipe %s"
declare -x LOGNAME="nilrog"
declare -x LS_COLORS="rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:"
declare -x MAIL="/var/mail/nilrog"
declare -x OLDPWD
declare -x OS_AUTH_URL="http://10.80.84.4:5000/v2.0"
declare -x OS_PASSWORD="XXXXXX"
declare -x OS_TENANT_NAME="admin"
declare -x OS_USERNAME="admin"
declare -x PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
declare -x PS1="\\[\\e]0;\\u@\\h: \\w\\a\\]\\u@\\h[admin]:\\w\$ "
declare -x PWD="/home/nilrog"
declare -x QT_QPA_PLATFORMTHEME="appmenu-qt5"
declare -x SHELL="/bin/bash"
declare -x SHLVL="1"
declare -x SSH_CLIENT="10.0.2.2 57288 22"
declare -x SSH_CONNECTION="10.0.2.2 57288 10.0.2.15 22"
declare -x SSH_TTY="/dev/pts/6"
declare -x TERM="xterm"
declare -x USER="nilrog"
declare -x XDG_RUNTIME_DIR="/run/user/1000"
declare -x XDG_SESSION_ID="2"
DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://10.80.84.4:5000/v2.0/tokens
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 10.80.84.4
DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 4316
DEBUG:keystoneclient.session:REQ: curl -g -i -X GET http://192.168.0.2:35357/v2.0/tenants -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}55ef76a681f09f...

[Vladimir Kuklin (vkuklin)]

Roger. So as far as I see you are using internal URL whenever you do not have access to it. You should use public one or perform commands from the place where you have access to your management network.

 Please consult with http://docs.openstack.org/cli-reference/content/keystoneclient_commands.html
and use correct --endpoint-type arguments.

I will mark this bug as Invalid. If you think this is not valid, please provide your comments

[Roger Törnström (roger-tornstrom)]

I disagree with your conclusion!

We ARE using the public IP. We HAVE NOT modified the keystone endpoints.
That part is configured by Fuel!

So if the endpoints are wrong the fault is in Fuel, and that is the bug!

/Roger

tisdag 14 april 2015 skrev Vladimir Kuklin <email address hidden>:

> Roger. So as far as I see you are using internal URL whenever you do not
> have access to it. You should use public one or perform commands from
> the place where you have access to your management network.
>
> Please consult with
> http://docs.openstack.org/cli-reference/content/keystoneclient_commands.html
> and use correct --endpoint-type arguments.
>
> I will mark this bug as Invalid. If you think this is not valid, please
> provide your comments
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1430309
>
> Title:
> Keystone doen't work through public VIP
>
> Status in Fuel: OpenStack installer that works:
> Confirmed
>
> Bug description:
> Cloud deployed with MOS 6.0 in HA mode with 3 controllers.
>
> It is not possible to use the keystone command from outside through
> the management VIP address. All other openstack CLI-commands works. It
> is possible to use the keystone command from inside the cloud on a
> controller. So it appears that something is misconfigured in keystone
> since it looks like the internal IP is exposed to the outside.
>
> This is what happens from the outside
> (OS_AUTH_URL=http://10.80.84.4:5000/v2.0):
>
> $ keystone --debug tenant-list
> DEBUG:keystoneclient.auth.identity.v2:Making authentication request to
> http://10.80.84.4:5000/v2.0/tokens
> INFO:requests.packages.urllib3.connectionpool:Starting new HTTP
> connection (1): 10.80.84.4
> DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens
> HTTP/1.1" 200 4366
> DEBUG:keystoneclient.session:REQ: curl -g -i -X GET
> http://192.168.0.2:35357/v2.0/tenants -H "User-Agent:
> python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token:
> {SHA1}1362926ee9e5652dcf8f19b14b70820bd6aee8ad"
> INFO:requests.packages.urllib3.connectionpool:Starting new HTTP
> connection (1): 192.168.0.2
> Unable to establish connection to http://192.168.0.2:35357/v2.0/tenants
>
> If I source the openrc file on a controller and run the same command
> it works (OS_AUTH_URL=http://192.168.0.2:5000/v2.0/):
>
> :~# keystone tenant-list
> +----------------------------------+-----------+---------+
> | id | name | enabled |
> +----------------------------------+-----------+---------+
> | 1ab2038ab4554253b6e188cad8fe3ba1 | admin | True |
> | df61f8c5b8a3455d83d6685e59fbf985 | tenant-1 | True |
> | a11522807d4c4c02a1e6e399816c9825 | tenant-2 | True |
> | 7cf7f21595534c96a72e19bbc3605f70 | services | True |
> +----------------------------------+-----------+---------+
>
> # fuel --fuel-version
> api: '1.0'
> astute_sha: 16b252d93be6aaa73030b8100cf8c5ca6a970a91
> auth_required: true
> build_id: 2014-12-26_14-25-46
> build_number: '58'
> feature_groups:
> - mirantis
> fuellib_sha: fde8ba5e11a1a...

[Roger Törnström (roger-tornstrom)]

@vkuklin
Btw, if you read the documentation you linked to you can see that there is
no "--endpoint-type" argument available, except for the "keystone
endpoint-get" command.

Did you _actually_ test what I describe? With a cloud deployed with
Mirantis 6.0 without patching it? I have described the observed behaviour
over and over and the most useful response I have gotten is #7, but
unfortunately it did not help.

So, once again, if you look at the outputs I have included many
times...keystone successfully connect to the public IP (10.80.84.4) but for
some reason it receives a response that triggers it to try to connect to a
private IP (192.168.0.2).

/Roger

2015-04-14 15:49 GMT+02:00 Roger Törnström <email address hidden>:

> I disagree with your conclusion!
>
> We ARE using the public IP. We HAVE NOT modified the keystone endpoints.
> That part is configured by Fuel!
>
> So if the endpoints are wrong the fault is in Fuel, and that is the bug!
>
> /Roger
>
>
> tisdag 14 april 2015 skrev Vladimir Kuklin <email address hidden>:
>
>> Roger. So as far as I see you are using internal URL whenever you do not
>> have access to it. You should use public one or perform commands from
>> the place where you have access to your management network.
>>
>> Please consult with
>> http://docs.openstack.org/cli-reference/content/keystoneclient_commands.html
>> and use correct --endpoint-type arguments.
>>
>> I will mark this bug as Invalid. If you think this is not valid, please
>> provide your comments
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/1430309
>>
>> Title:
>> Keystone doen't work through public VIP
>>
>> Status in Fuel: OpenStack installer that works:
>> Confirmed
>>
>> Bug description:
>> Cloud deployed with MOS 6.0 in HA mode with 3 controllers.
>>
>> It is not possible to use the keystone command from outside through
>> the management VIP address. All other openstack CLI-commands works. It
>> is possible to use the keystone command from inside the cloud on a
>> controller. So it appears that something is misconfigured in keystone
>> since it looks like the internal IP is exposed to the outside.
>>
>> This is what happens from the outside
>> (OS_AUTH_URL=http://10.80.84.4:5000/v2.0):
>>
>> $ keystone --debug tenant-list
>> DEBUG:keystoneclient.auth.identity.v2:Making authentication request to
>> http://10.80.84.4:5000/v2.0/tokens
>> INFO:requests.packages.urllib3.connectionpool:Starting new HTTP
>> connection (1): 10.80.84.4
>> DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens
>> HTTP/1.1" 200 4366
>> DEBUG:keystoneclient.session:REQ: curl -g -i -X GET
>> http://192.168.0.2:35357/v2.0/tenants -H "User-Agent:
>> python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token:
>> {SHA1}1362926ee9e5652dcf8f19b14b70820bd6aee8ad"
>> INFO:requests.packages.urllib3.connectionpool:Starting new HTTP
>> connection (1): 192.168.0.2
>> Unable to establish connection to http://192.168.0.2:35357/v2.0/tenants
>>
>> If I source the openrc file on a controller and run the same command
>> it works (OS_AUTH_URL=http:...

[Bogdan Dobrelya (bogdando)]

I will try to reproduce it and provide a triage feedback ASAP

[Bogdan Dobrelya (bogdando)]

I will provide details about 6.0.1 once I have a deployed environment

[Bogdan Dobrelya (bogdando)]

For the 6.1, ubuntu trusty HA lab, I cannot reproduce it, keystone is reachable via public IP.
Here are details http://paste.openstack.org/show/203957/ (note, I connected from external node to public VIP)
As you can see from requests.packages.urllib3.connectionpool debug, it successfully communicates both via public and management VIPs

[Bogdan Dobrelya (bogdando)]

My bad, the internal IP was as well reachable from my external node :)
Once I restricted internal VIP from my node under test (sudo iptables -I INPUT -s 10.109.2.2/32 -j REJECT)
The requests.packages.urllib3.connectionpool stopped to process connections at the step:

DEBUG:keystoneclient.session:REQ: curl -i -X GET http://10.109.2.2:35357/v2.0/tenants -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: 29a94aa73ecc4b799820bac48b529c41"
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 10.109.2.2

So the issue is confirmed for the 6.1 and likely for the 6.0.1 as well

[Bogdan Dobrelya (bogdando)]

Note, nova list also hangs for given case, so the issue likely affects *all* Openstack clients

[Bogdan Dobrelya (bogdando)]

Nova list debug output is here http://paste.openstack.org/show/203958/

[Roger Törnström (roger-tornstrom)]

Good that you can reproduce it :)

But for us all other commands (nova, neutron, glance, cinder etc) works
fine. It is only the keystone command that fails.

/Roger

2015-04-15 12:03 GMT+02:00 Bogdan Dobrelya <email address hidden>:

> Nova list debug output is here http://paste.openstack.org/show/203958/
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1430309
>
> Title:
> Keystone doen't work through public VIP
>
> Status in Fuel: OpenStack installer that works:
> Confirmed
> Status in Fuel for OpenStack 6.0.x series:
> Confirmed
>
> Bug description:
> Cloud deployed with MOS 6.0 in HA mode with 3 controllers.
>
> It is not possible to use the keystone command from outside through
> the management VIP address. All other openstack CLI-commands works. It
> is possible to use the keystone command from inside the cloud on a
> controller. So it appears that something is misconfigured in keystone
> since it looks like the internal IP is exposed to the outside.
>
> This is what happens from the outside
> (OS_AUTH_URL=http://10.80.84.4:5000/v2.0):
>
> $ keystone --debug tenant-list
> DEBUG:keystoneclient.auth.identity.v2:Making authentication request to
> http://10.80.84.4:5000/v2.0/tokens
> INFO:requests.packages.urllib3.connectionpool:Starting new HTTP
> connection (1): 10.80.84.4
> DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens
> HTTP/1.1" 200 4366
> DEBUG:keystoneclient.session:REQ: curl -g -i -X GET
> http://192.168.0.2:35357/v2.0/tenants -H "User-Agent:
> python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token:
> {SHA1}1362926ee9e5652dcf8f19b14b70820bd6aee8ad"
> INFO:requests.packages.urllib3.connectionpool:Starting new HTTP
> connection (1): 192.168.0.2
> Unable to establish connection to http://192.168.0.2:35357/v2.0/tenants
>
> If I source the openrc file on a controller and run the same command
> it works (OS_AUTH_URL=http://192.168.0.2:5000/v2.0/):
>
> :~# keystone tenant-list
> +----------------------------------+-----------+---------+
> | id | name | enabled |
> +----------------------------------+-----------+---------+
> | 1ab2038ab4554253b6e188cad8fe3ba1 | admin | True |
> | df61f8c5b8a3455d83d6685e59fbf985 | tenant-1 | True |
> | a11522807d4c4c02a1e6e399816c9825 | tenant-2 | True |
> | 7cf7f21595534c96a72e19bbc3605f70 | services | True |
> +----------------------------------+-----------+---------+
>
> # fuel --fuel-version
> api: '1.0'
> astute_sha: 16b252d93be6aaa73030b8100cf8c5ca6a970a91
> auth_required: true
> build_id: 2014-12-26_14-25-46
> build_number: '58'
> feature_groups:
> - mirantis
> fuellib_sha: fde8ba5e11a1acaf819d402c645c731af450aff0
> fuelmain_sha: 81d38d6f2903b5a8b4bee79ca45a54b76c1361b8
> nailgun_sha: 5f91157daa6798ff522ca9f6d34e7e135f150a90
> ostf_sha: a9afb68710d809570460c29d6c3293219d3624d4
> production: docker
> release: '6.0'
> release_versions:
> 2014.2-6.0:
> VERSION:
> api: '1.0'
> astute_sha: 16b252d93be6aaa73030b8100cf8c5ca6a970a...

[Bogdan Dobrelya (bogdando)]

@Roger, indeed - once I replaced all internalURL to publicURL in my openrc, nova stopped to hang on list request. But keystone still does. So, we're now at the same page

[Roger Törnström (roger-tornstrom)]

Great! :)

2015-04-15 13:04 GMT+02:00 Bogdan Dobrelya <email address hidden>:

> @Roger, indeed - once I replaced all internalURL to publicURL in my
> openrc, nova stopped to hang on list request. But keystone still does.
> So, we're now at the same page
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1430309
>
> Title:
> Keystone doen't work through public VIP
>
> Status in Fuel: OpenStack installer that works:
> Confirmed
> Status in Fuel for OpenStack 6.0.x series:
> Confirmed
>
> Bug description:
> Cloud deployed with MOS 6.0 in HA mode with 3 controllers.
>
> It is not possible to use the keystone command from outside through
> the management VIP address. All other openstack CLI-commands works. It
> is possible to use the keystone command from inside the cloud on a
> controller. So it appears that something is misconfigured in keystone
> since it looks like the internal IP is exposed to the outside.
>
> This is what happens from the outside
> (OS_AUTH_URL=http://10.80.84.4:5000/v2.0):
>
> $ keystone --debug tenant-list
> DEBUG:keystoneclient.auth.identity.v2:Making authentication request to
> http://10.80.84.4:5000/v2.0/tokens
> INFO:requests.packages.urllib3.connectionpool:Starting new HTTP
> connection (1): 10.80.84.4
> DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens
> HTTP/1.1" 200 4366
> DEBUG:keystoneclient.session:REQ: curl -g -i -X GET
> http://192.168.0.2:35357/v2.0/tenants -H "User-Agent:
> python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token:
> {SHA1}1362926ee9e5652dcf8f19b14b70820bd6aee8ad"
> INFO:requests.packages.urllib3.connectionpool:Starting new HTTP
> connection (1): 192.168.0.2
> Unable to establish connection to http://192.168.0.2:35357/v2.0/tenants
>
> If I source the openrc file on a controller and run the same command
> it works (OS_AUTH_URL=http://192.168.0.2:5000/v2.0/):
>
> :~# keystone tenant-list
> +----------------------------------+-----------+---------+
> | id | name | enabled |
> +----------------------------------+-----------+---------+
> | 1ab2038ab4554253b6e188cad8fe3ba1 | admin | True |
> | df61f8c5b8a3455d83d6685e59fbf985 | tenant-1 | True |
> | a11522807d4c4c02a1e6e399816c9825 | tenant-2 | True |
> | 7cf7f21595534c96a72e19bbc3605f70 | services | True |
> +----------------------------------+-----------+---------+
>
> # fuel --fuel-version
> api: '1.0'
> astute_sha: 16b252d93be6aaa73030b8100cf8c5ca6a970a91
> auth_required: true
> build_id: 2014-12-26_14-25-46
> build_number: '58'
> feature_groups:
> - mirantis
> fuellib_sha: fde8ba5e11a1acaf819d402c645c731af450aff0
> fuelmain_sha: 81d38d6f2903b5a8b4bee79ca45a54b76c1361b8
> nailgun_sha: 5f91157daa6798ff522ca9f6d34e7e135f150a90
> ostf_sha: a9afb68710d809570460c29d6c3293219d3624d4
> production: docker
> release: '6.0'
> release_versions:
> 2014.2-6.0:
> VERSION:
> api: '1.0'
> astute_sha: 16b252d93be6aaa73030b8100cf8c5ca6a970a91
> build_id: 2014-12-26_14-25-46
> ...

[Bogdan Dobrelya (bogdando)]

I have an update. The command
keystone endpoint-get --service identity
works and returns a valid public endpoint for keystone:
"identity.publicURL | http://10.109.1.2:5000/v2.0"

I'm wondering, is this bug related to https://bugs.launchpad.net/python-openstackclient/+bug/1184012 ?

[Bogdan Dobrelya (bogdando)]

Ok, I can confirm what catalog, token-get commands work as well. This issue looks similar to that one https://bugs.launchpad.net/fuel/+bug/1362641 (was invalidated, as we can see - by mistake)

[Bogdan Dobrelya (bogdando)]

Here is the way how it works:

first get a token by your creds
 /usr/bin/keystone --debug token-get
second, get a public endpoint
 /usr/bin/keystone --debug endpoint-get --service identity
next use this endpoint and the given token to query keystone, for example
 /usr/bin/keystone --debug --os-token=7dfe9dc1914b4a8b91ba44ab47cae639 --os-endpoint=http://10.109.1.2:5000/v2.0/ tenant-list