Packstack behind proxy fails

[Don Weeks]

http://paste.openstack.org/show/720303/

10.1.32.8_controller.pp: [ ERROR ]
Applying Puppet manifests [ ERROR ]

ERROR : Error appeared during Puppet run: 10.1.32.8_controller.pp
Error: Failed to apply catalog: Execution of '/usr/bin/openstack domain list --quiet --format csv' returned 1: Service Unavailable (HTTP 503) (tried 32, for a total of 170 seconds)
You will find full trace in log /var/tmp/packstack/20180503-155505-mO7jQB/manifests/10.1.32.8_controller.pp.log
Please check log file /var/tmp/packstack/20180503-155505-mO7jQB/openstack-setup.log for more information

Paste above is from the command line.

I follow the documentation RDO website. Have completely deleted and reinstalled.

http://paste.openstack.org/show/720307/

manifest log above.

[Javier Peña (jpena-c)]

Hi,

The controller log got cut (length limitations in paste.openstack.org). Looking at the errors, it looks like something failed when setting up Keystone. Could you upload the following data?

- Keystone logs (under /var/log/keystone)
- Full controller log (same as http://paste.openstack.org/show/720307/)
- Apache logs (under /var/log/httpd)

Was this your first setup on a clean system, or did you run packstack several times?

[Don Weeks (dweeksjr)]

Will upload. This is on a clean system. New install of CentOS. Basically followed the RDO install instructions starting with new OS install. I have done this 3 times and each time, I hit this issue.

[Javier Peña (jpena-c)]

Wait a second, I think I have an idea.

You mentioned that your system is behind a proxy, is the "http_proxy" environment variable set? If so, openstackclient will be trying to contact the Keystone service via the proxy, which is very likely to fail since the proxy will only be used to access external resources.

If that is your case, I have found http://docs.platform9.com/support/providing-access-openstack-cli-proxy-server/, where they explain how to set up the no_proxy environment variable. That will probably help.

[Don Weeks (dweeksjr)]

https://www.dropbox.com/s/4rmhesyzmbow7ww/logs.tgz?dl=0

Let me know if this works.

[Don Weeks (dweeksjr)]

I do have a no_proxy set:
export http_proxy=http://10.1.20.57:80
export https_proxy=http://10.1.20.57:80
export no_proxy=10.1.32.8,localhost,10.1.20.16

10.1.32.8 is the server ip.

[Don Weeks (dweeksjr)]

Also, I have access to the squid logs and I do not see any requests from this machine towards keystone. Only towards the repos.

[Javier Peña (jpena-c)]

So looking at the logs, we see the following in the Keystone log:

2018-05-03 16:10:29.855 8740 INFO keystone.common.wsgi [req-73b1dd57-331d-474c-b2fb-63072269647b - - - - -] POST http://10.1.32.8:5000/v3/auth/tokens
2018-05-03 16:10:29.968 8740 WARNING keystone.auth.core [req-73b1dd57-331d-474c-b2fb-63072269647b - - - - -] Could not find project: services.: ProjectNotFound: Could not find project: services.

This is very strange, since the services project should be created by this snippet of puppet code: https://github.com/openstack/packstack/blob/master/packstack/puppet/modules/packstack/manifests/keystone.pp#L61-L67

From the previous comments, I see that /var/tmp/packstack/20180503-155505-mO7jQB/manifests/10.1.32.8_controller.pp.log is incomplete in the paste. Could you attach it to the bug? It should give us a hint on why this happened.

[Don Weeks (dweeksjr)]

https://www.dropbox.com/s/71xti8juhv8wci5/pplogs.tgz?dl=0

controller.pp.log attached.

[Javier Peña (jpena-c)]

I'm puzzled about this. Looking at the controller.log file, right after starting httpd we are trying to create a Keystone domain, but that fails because the services tenant is not present. This has not happened before in our CI environments or real-life tests :-/.

I have a couple questions:

- Was this run on a clean system, or did you try to run packstack several times before it failed like this?
- What is the puppet version installed in the system (puppet --version)?

[Alfredo Moralejo (amoralej)]

Could you please provide us with the answers file you used to deploy the environment and the full list of packages installed after the packstack command failed?

[Don Weeks (dweeksjr)]

Will gladly upload. As far as how it was deployed, I have tried several times using different scenarios. Clean install and no answer file and with the answer file. All have ended here at this error.

[Don Weeks (dweeksjr)]

https://www.dropbox.com/s/tte88tunapf06fk/answers.cfg?dl=0

[Don Weeks (dweeksjr)]

The only thing I am trying to turn on is magnum but as I said, the first try was without the answer file and failed at same place. Each time, I reload the OS completely for each new try.

[Javier Peña (jpena-c)]

I have tried to reproduce the issue on a CentOS 7 VM with RDO Queens, and there is no way I can do it. Having another look at the logs, I find the following at the beginning of the keystone log:

2018-05-03 16:01:03.563 1806 WARNING oslo_db.sqlalchemy.engines [-] URL mysql://keystone:***@localhost/keystone does not contain a '+drivername' portion, and will make use of a default driver. A full dbname+drivername:// protocol is recommended. For MySQL, it is strongly recommended that mysql+pymysql:// be specified for maximum service compatibility
2018-05-03 16:01:03.650 1806 CRITICAL keystone [-] Unhandled error: OperationalError: (_mysql_exceptions.OperationalError) (1045, "Access denied for user 'keystone'@'localhost' (using password: YES)") (Background on this error at: http://sqlalche.me/e/e3q8)

This is really strange. It looks like the Keystone service was started about 10 minutes before being properly configured. At that time, some wrong configuration was in effect at keystone.conf, as noted by the "mysql://keystone:***@localhost/keystone" database connection string. Looking at the Apache logs, httpd was started at 16:10:24.

Could you run the following command and paste its output:

$ rpm -qa|grep -e openstack-keystone -e puppet -e openstack-packstack

Also, could you try another installation from scratch, and just before executing Packstack, run the same command above and paste its output? I'm quite convinced there is some issue with packages being installed, since Keystone cannot start without httpd, and httpd was stopped until 10 minutes after we got logs from Keystone.

[Don Weeks (dweeksjr)]

Will do. Give me a bit of time

[Don Weeks (dweeksjr)]

[root@rdo manifests]# rpm -qa | grep -e openstack-keystone -e puppet -e openstack-packstack
puppet-sysctl-0.0.11-1.el7.noarch
puppet-ssh-3.0.1-4.fb2de75git.el7.noarch
puppet-ovn-12.4.0-1.el7.noarch
puppet-nova-12.4.0-1.el7.noarch
puppet-trove-12.4.0-1.el7.noarch
openstack-packstack-12.0.0-1.el7.noarch
puppet-inifile-2.2.0-1.d2c38b9git.el7.noarch
puppet-memcached-3.1.0-1.el7.noarch
puppet-rabbitmq-8.1.1-0.d4b06b7git.el7.noarch
puppet-keystone-12.4.0-1.el7.noarch
puppet-heat-12.4.0-1.el7.noarch
puppet-tempest-12.5.0-1.el7.noarch
puppet-concat-4.1.1-1.d4857dfgit.el7.noarch
puppet-vcsrepo-2.3.0-1.bb1209egit.el7.noarch
puppet-vswitch-8.4.0-1.el7.noarch
puppet-nssdb-1.0.1-1.el7.noarch
puppet-corosync-6.0.1-0.9940eb9git.el7.noarch
puppet-mysql-5.2.1-1.a5497b2git.el7.noarch
puppet-glance-12.5.0-1.el7.noarch
puppet-manila-12.4.0-1.el7.noarch
puppet-panko-12.4.0-1.el7.noarch
puppet-swift-12.4.0-1.el7.noarch
puppet-remote-0.0.1-3.7420908git.el7.noarch
openstack-keystone-13.0.0-1.el7.noarch
puppet-4.8.2-1.el7.noarch
puppet-apache-2.3.1-1.e587f2agit.el7.noarch
puppet-rsync-0.4.0-3.447685fgit.el7.noarch
puppet-horizon-12.4.0-1.el7.noarch
puppet-certmonger-2.3.0-1.el7.noarch
puppet-openstack_extras-12.4.0-1.el7.noarch
puppet-openstacklib-12.4.0-1.el7.noarch
puppet-cinder-12.4.0-1.el7.noarch
puppet-ceilometer-12.5.0-1.el7.noarch
puppet-sahara-12.4.0-1.el7.noarch
puppet-magnum-12.2.0-1.el7.noarch
openstack-packstack-puppet-12.0.0-1.el7.noarch
puppet-stdlib-4.24.0-1.b0d99adgit.el7.noarch
puppet-firewall-1.12.0-1.3dc1990git.el7.noarch
puppet-archive-2.2.1-0.10888dbgit.el7.noarch
puppet-oslo-12.4.0-1.el7.noarch
puppet-gnocchi-12.4.0-1.el7.noarch
puppet-aodh-12.4.0-1.el7.noarch
puppet-xinetd-3.0.0-1.b95c79cgit.el7.noarch
puppet-redis-3.2.0-2.bfcc212git.el7.noarch
puppet-staging-1.0.4-1.b466d93git.el7.noarch
puppet-neutron-12.4.0-1.el7.noarch
puppet-ironic-12.4.0-1.el7.noarch

[Fadil (fadilmohd007)]

Im trying to install openstack behind a proxy network I have gone through the comment above but none is working for me.Can anyone please say what all are the domain openstack packstack installation is trying to access so admin can free those domain.
Thanks in advance
fad