neutron

Bug #1811941
Comment #41

Comment 41 for bug 1811941

Revision history for this message

Amer Hwitat (amer.hwitat) wrote on 2019-01-24:

#41

Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'keystoneauth1.exceptions.connection.ConnectTimeout'> (HTTP 500) (Request-ID: req-a0fa98c2-4707-42e6-b31d-1021c8538428)

this was caused because of connection timeout default settings in the configuration files, and it's because all nodes (Glance, Swift, Nova, Neutron, ceilometer ..etc) have a default timeout for binding IPs , client connections, and connection timeout mainly most are set to 30 seconds or less 0.1 s in some cases ...

[DEFAULT]
functions get passed: conf, name, log_to_console, log_route, fmt, logger, adapted_logger
bind_port = 8080
workers = 4
user = swift
###################
bind_ip= 0.0.0.0
bind_timeout = 999
client_timeout = 999
conn_timeout = 999
node_timeout = 999
swift_dir = /etc/swift
############################
this is proxy-server.conf at /etc/swift/ added below hashes

/usr/lib/python2.7/site-packages/swift/common/wsgi.py

import ConfigParser, io

because I want to make sure that conf.get reads from conf files like proxy-server.conf and object-server.conf, well it's a mod_wsgi run on httpd, in which by the way I need to restart it each time I reboot, I have to check this out:
https://stackoverflow.com/questions/9327554/mod-wsgi-python-conf-parser

added some lines to object-server.conf at /etc/swift

bind_timeout = 999
client_timeout = 999
conn_timeout = 999

didn't really have to change anything to proxy-server.conf, even though I added the same above parameters to it, restarted swift services and servers: systemctl restart openstack*swift* bottom line the swift (volume node group), when is failed, it affects (controller nodes), (compute nodes), like physically a PC can not run with out a storage, but it can run with out network for example, but in our case here network sockets is what run everything inside OSP 14, OSP as a whole. so simple it is done once , configure python script, configure swift conf files, make sure that the br-ex is configured right from the beginning. and if you are running it on VM on a different platform, like windows 10, make sure that your firewall is disabled, and defender also, I don't know why but it keeps turning on real time protection automatically, check settings... add static routes on VM and on Windows, to ping your Virtual router outer and inner interfaces .. and that's it you're good to go just some remarks about Horizon in this slow case ... it keeps logging out and gives some other error on the screen, be patient and wait it will re-tune itself, but you have to logon each time there is a lag on the system..

my bad I have installed the full rich GUI interface of RHEL 7.6,and full Daemons (services), and not the infrastructure server which consumes really some more RAM and CPU ..

during the process I had some erros like watchdog Error on CPU on VM, and there is a parser server that needs also to be configured for timeout too

[root@localhost log]#
Message from syslogd@localhost at Jan 23 02:23:31 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ovsdb-server:10088]

so this error below really affected neutron, nova, glance, cinder, and swift

● openstack-swift-object.service - OpenStack Object Storage (swift) - Object Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-swift-object.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2019-01-23 05:25:19 EST; 5min ago
  Process: 32721 ExecStart=/usr/bin/swift-object-server /etc/swift/object-server.conf (code=exited, status=1/FAILURE)
Main PID: 32721 (code=exited, status=1/FAILURE)

Jan 23 05:25:19 localhost.localdomain swift-object-server[32721]: File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 1036, in run_wsgi
Jan 23 05:25:19 localhost.localdomain swift-object-server[32721]: error_msg = strategy.do_bind_ports()
Jan 23 05:25:19 localhost.localdomain swift-object-server[32721]: File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 615, in do_bind_ports
Jan 23 05:25:19 localhost.localdomain swift-object-server[32721]: self.sock = get_socket(self.conf)
Jan 23 05:25:19 localhost.localdomain swift-object-server[32721]: File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 212, in get_socket
Jan 23 05:25:19 localhost.localdomain swift-object-server[32721]: 'timeout': bind_timeout})
Jan 23 05:25:19 localhost.localdomain swift-object-server[32721]: Exception: Could not bind to 192.168.43.110:6000 after trying for 30 seconds
Jan 23 05:25:19 localhost.localdomain systemd[1]: openstack-swift-object.service: main process exited, code=exited, status=1/FAILURE
Jan 23 05:25:19 localhost.localdomain systemd[1]: Unit openstack-swift-object.service entered failed state.
Jan 23 05:25:19 localhost.localdomain systemd[1]: openstack-swift-object.service failed.

and the remedy was given earlier, after editing object-server.conf and proxy-server.conf and python wsgi.py script the Machine runs steady for quite a longer time than before , you just need to run through all conf files and set timeout settings to longer intervals ...

to this successful configuration on my slow VM:
[root@localhost swift]# systemctl status openstack-swift-object.service -l
● openstack-swift-object.service - OpenStack Object Storage (swift) - Object Server Loaded: loaded (/usr/lib/systemd/system/openstack-swift-object.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2019-01-23 10:00:07 EST; 7min ago Main PID: 63675 (swift-object-se) Tasks: 1 CGroup: /system.slice/openstack-swift-object.service
└─63675 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf
Jan 23 10:00:07 localhost.localdomain systemd[1]: Started OpenStack Object Storage (swift) - Object Server.
[root@localhost swift]#

[root@localhost swift]# systemctl status openstack-swift-object.service -l
● openstack-swift-object.service - OpenStack Object Storage (swift) - Object Server Loaded: loaded (/usr/lib/systemd/system/openstack-swift-object.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2019-01-23 10:00:07 EST; 2h 29min ago Main PID: 63675 (swift-object-se) Tasks: 1
CGroup: /system.slice/openstack-swift-object.service
└─63675 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf Jan 23 10:00:07 localhost.localdomain systemd[1]: Started OpenStack Object Storage (swift) - Object Server. [root@localhost swift]#

it ran for 3 hours .. where as it was running for 1 minute only before

this was caused because of connection timeout default settings in the configuration files, and it's because all nodes (Glance, Swift, Nova, Neutron, ceilometer ..etc) have a default timeout for binding IPs , client connections, and  connection timeout mainly most are set to 30 seconds or less 0.1 s in some cases ...

[DEFAULT] 
functions get passed: conf, name, log_to_console, log_route, fmt, logger, adapted_logger 
bind_port = 8080 
workers = 4 
user = swift 
################### 
bind_ip= 0.0.0.0
bind_timeout = 999 
client_timeout = 999 
conn_timeout = 999 
node_timeout = 999 
swift_dir = /etc/swift 
############################ 
this is proxy-server.conf at /etc/swift/ added below hashes

/usr/lib/python2.7/site-packages/swift/common/wsgi.py
 
import ConfigParser, io

because I want to make sure that conf.get reads from conf files like proxy-server.conf and object-server.conf, well it's a mod_wsgi run on httpd, in which by the way I need to restart it each time I reboot, I have to check this out: 
https://stackoverflow.com/questions/9327554/mod-wsgi-python-conf-parser

added some lines to object-server.conf at /etc/swift

bind_timeout = 999 
client_timeout = 999 
conn_timeout = 999

my bad I have installed the full rich GUI interface of RHEL 7.6,and full Daemons (services), and not the infrastructure server which consumes really some more RAM and CPU ..

during the process I had some erros like watchdog Error on CPU on VM, and there is a parser server that needs also to be configured for timeout too

[root@localhost log]# 
Message from syslogd@localhost at Jan 23 02:23:31 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ovsdb-server:10088]

so this error below really affected neutron, nova, glance, cinder, and swift

to this successful configuration on my slow VM: 
[root@localhost swift]# systemctl status openstack-swift-object.service -l 
● openstack-swift-object.service - OpenStack Object Storage (swift) - Object Server Loaded: loaded (/usr/lib/systemd/system/openstack-swift-object.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2019-01-23 10:00:07 EST; 7min ago Main PID: 63675 (swift-object-se) Tasks: 1 CGroup: /system.slice/openstack-swift-object.service
 └─63675 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf 
Jan 23 10:00:07 localhost.localdomain systemd[1]: Started OpenStack Object Storage (swift) - Object Server. 
[root@localhost swift]#

[root@localhost swift]# systemctl status openstack-swift-object.service -l 
● openstack-swift-object.service - OpenStack Object Storage (swift) - Object Server Loaded: loaded (/usr/lib/systemd/system/openstack-swift-object.service; enabled; vendor preset: disabled) 
Active: active (running) since Wed 2019-01-23 10:00:07 EST; 2h 29min ago Main PID: 63675 (swift-object-se) Tasks: 1 
CGroup: /system.slice/openstack-swift-object.service 
└─63675 /usr/bin/python2 /usr/bin/swift-object-server /etc/swift/object-server.conf Jan 23 10:00:07 localhost.localdomain systemd[1]: Started OpenStack Object Storage (swift) - Object Server. [root@localhost swift]#

it ran for 3 hours .. where as it was running for 1 minute only before