heat.conf ec2authtoken/auth_uri points to localhost

[Igor Yozhikov]

Openstack Heat configuration option points to localhost by deafult instead of controller's ip address
...
[ec2authtoken]
auth_uri = http://127.0.0.1:5000/v2.0/ec2tokens
...
This configurations comes from puppet heat modules (fuel-library) and the same as in upstream (puppet-heat).

From the source code of OpenStack HEAT( https://github.com/openstack/heat/blob/stable/icehouse/heat/api/aws/ec2token.py#L76 ) was investigated next:

If auth_uri variable ends without ec2tokens, ec2tokens would be added automatically.

[Sergey Kraynev (skraynev)]

Priority was increased, because this bug based on request from support team.

[Sergey Kraynev (skraynev)]

Link on fix for this bug https://review.openstack.org/#/c/111354/

[Yaroslav Lobankov (ylobankov)]

Oops, accidentally verified the issue for MOS 6.1 while the issue has milestone 5.1.

[Shannon Mitchell (shannon-mitchell)]

# With default stable/rocky settings

root@infra01-utility-container-52890918:~# openstack stack list
ERROR: Internal Error

# Instead of going out of eth1 and hitting keystone over the management network, its
# dropping out of the lxc eth0 interface and being routed to the public haproxy
# ip.

root@infra01-heat-api-container-90b2b9a2:/# tcpdump -n -l -i eth0 port 5000
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
21:55:29.050097 IP 10.0.3.175.49400 > 172.20.41.14.5000: Flags [S], seq 3076311433, win 64240, options [mss 1460,sackOK,TS val 233649765 ecr 0,nop,wscale 7], length 0
21:55:29.050307 IP 172.20.41.14.5000 > 10.0.3.175.49400: Flags [S.], seq 2434666710, ack 3076311434, win 62636, options [mss 8960,sackOK,TS val 2932159875 ecr 233649765,nop,wscale 7], length 0
21:55:29.050340 IP 10.0.3.175.49400 > 172.20.41.14.5000: Flags [.], ack 1, win 502, options [nop,nop,TS val 233649765 ecr 2932159875], length 0
21:55:29.059212 IP 10.0.3.175.49400 > 172.20.41.14.5000: Flags [P.], seq 1:213, ack 1, win 502, options [nop,nop,TS val 233649774 ecr 2932159875], length 212
21:55:29.061917 IP 172.20.41.14.5000 > 10.0.3.175.49400: Flags [P.], seq 1:1357, ack 213, win 488, options [nop,nop,TS val 2932159886 ecr 233649774], length 1356
21:55:29.061947 IP 10.0.3.175.49400 > 172.20.41.14.5000: Flags [.], ack 1357, win 492, options [nop,nop,TS val 233649777 ecr 2932159886], length 0
21:55:29.062294 IP 10.0.3.175.49400 > 172.20.41.14.5000: Flags [P.], seq 213:220, ack 1357, win 492, options [nop,nop,TS val 233649777 ecr 2932159886], length 7
21:55:29.062565 IP 172.20.41.14.5000 > 10.0.3.175.49400: Flags [F.], seq 1357, ack 220, win 488, options [nop,nop,TS val 2932159887 ecr 233649777], length 0
21:55:29.062623 IP 10.0.3.175.49400 > 172.20.41.14.5000: Flags [R.], seq 220, ack 1358, win 501, options [nop,nop,TS val 233649777 ecr 2932159887], length 0

# I'm assuming like other services, this communication should be happening over
# the management network via eth1 on the container. After some testing, it looks
# like the [clients-keystone]/auth_uri setting handles this. Chaning this back
# to the internal endpoint fixed the issues.

# cat /etc/openstack_deploy/user_heat_fix.yml
heat_heat_conf_overrides:
    clients_keystone:
        auth_uri: '{{ keystone_service_internaluri }}'

root@infra01:/opt/openstack-ansible/playbooks# openstack-ansible os-heat-install.yml --tags heat-config

root@infra01-utility-container-52890918:~# openstack stack list; echo $?

0

[Shannon Mitchell (shannon-mitchell)]

Please ignore, posted in wrong bug.