neutron

Bug #1818544
Comment #0

Comment 0 for bug 1818544

Revision history for this message

Sohny (sohnythomas) wrote on 2019-03-04:

I have a openstack setup which has newton version installed on CentOS 7.3. I am able to successfuly create VMs and associate floating Ips. Successfuly able to ssh into and out of VM from external network but I am not able to SCP any file out of the VM instance . SCP into the VM instance is fine. SCP btw VM instances are also fine

Below is a 70kb file for which transfer is still going on after 2 hours. At the destination only 10kb has been copied.

[root@test-server ~]# scp /var/cache/jenkins/war/WEB-INF/lib/remoting-3.29.jar dscadmin@1.20.28.146:/home/dscadmin/jenkins/remoting-3.29.jar 100% 771KB 35.0KB/s 00:22 [

System SPECs:

[root@newton-1 neutron]# openstack --version
openstack 3.2.1

[root@newton-1 neutron]# rpm -qa|grep neutron
openstack-neutron-common-9.4.1-1.el7.noarch
openstack-neutron-openvswitch-9.4.1-1.el7.noarch
puppet-neutron-9.5.0-1.el7.noarch
openstack-neutron-ml2-9.4.1-1.el7.noarch
python-neutron-9.4.1-1.el7.noarch
openstack-neutron-metering-agent-9.4.1-1.el7.noarch
python-neutron-lib-0.4.0-1.el7.noarch
openstack-neutron-9.4.1-1.el7.noarch
python2-neutronclient-6.0.0-2.el7.noarch

[root@newton-1 neutron]# uname -r
3.10.0-514.26.2.el7.x86_64

Following are some log files i tracked for this at /var/log/neutron

[root@newton-1 neutron]# tail -f server.log
2019-03-04 11:54:28.737 17152 INFO neutron.wsgi [req-de76d2a0-72c1-4d90-abf9-42e03aa9c76c bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:28] "GET /v2.0/ports.json?network_id=113e2ce7-475b-43ea-9765-7cce7565e639&device_owner=network%3Adhcp HTTP/1.1" 200 1119 0.035162
2019-03-04 11:54:28.749 17152 INFO neutron.wsgi [req-af4b081a-c900-4cd4-ad34-f64d19b57048 bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:28] "GET /v2.0/floatingips.json?fixed_ip_address=10.62.60.8&port_id=171c329c-e945-4bf4-bc05-82177a776c72 HTTP/1.1" 200 217 0.009857
2019-03-04 11:54:28.801 17152 INFO neutron.wsgi [req-44ab23c8-84dd-4176-865b-c47a1bc288c0 bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:28] "GET /v2.0/subnets.json?id=83a71783-dcc3-4d8a-8560-89972e03bab5 HTTP/1.1" 200 863 0.049811
2019-03-04 11:54:28.837 17152 INFO neutron.wsgi [req-bb98b71c-f5ca-4be1-bff9-bd7795c38ffa bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:28] "GET /v2.0/ports.json?network_id=bab474bb-88b7-490f-8c5c-19b08d758a02&device_owner=network%3Adhcp HTTP/1.1" 200 1119 0.034647
2019-03-04 11:54:28.849 17152 INFO neutron.wsgi [req-ce60d323-075b-4b30-9104-e708c22c44b8 bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:28] "GET /v2.0/floatingips.json?fixed_ip_address=10.62.61.9&port_id=5864f529-8cb6-4d30-b3a3-c76be8750d3a HTTP/1.1" 200 217 0.009498
2019-03-04 11:54:28.901 17152 INFO neutron.wsgi [req-384ae67c-c463-4815-be96-8b1710d88d21 bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:28] "GET /v2.0/subnets.json?id=95e5bc60-0309-401c-87a9-b6596957f8a6 HTTP/1.1" 200 863 0.050315
2019-03-04 11:54:28.940 17152 INFO neutron.wsgi [req-6fb418f2-1640-4daf-ad98-a9a8e870f6bd bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:28] "GET /v2.0/ports.json?network_id=3950ea82-c375-4fcc-bc19-9e01d92681fd&device_owner=network%3Adhcp HTTP/1.1" 200 1119 0.037263
2019-03-04 11:54:28.951 17152 INFO neutron.wsgi [req-5e7b3c25-3c32-4b97-a595-3b548daec6ae bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:28] "GET /v2.0/floatingips.json?fixed_ip_address=10.62.63.5&port_id=983c2442-ecab-49aa-988a-775fc70a9d10 HTTP/1.1" 200 217 0.009744
2019-03-04 11:54:29.000 17152 INFO neutron.wsgi [req-3e7fd50b-9fb1-4c72-9072-a0aa027f547c bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:29] "GET /v2.0/subnets.json?id=24c11aff-9785-4ed2-b33c-b98b037c24dc HTTP/1.1" 200 863 0.046999
2019-03-04 11:54:29.037 17152 INFO neutron.wsgi [req-2d71b123-b2aa-47d6-95db-bb254032fd7f bee9b87b7aa24677b3c536f7906fbf83 d1b8bebf20644e27b69e194b644d1154 - - -] 10.1.31.142 - - [04/Mar/2019 11:54:29] "GET /v2.0/ports.json?network_id=bcfd0431-aec1-42b0-98ad-b7155c836b97&device_owner=network%3Adhcp HTTP/1.1" 200 1118 0.035438

[root@newton-1 neutron]# tail -f dhcp-agent.log
2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent 'value': value}])
2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 912, in execute
2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent log_fail_as_error=log_fail_as_error, **kwargs)
2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 148, in execute
2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent raise ProcessExecutionError(msg, returncode=returncode)
2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent ProcessExecutionError: Exit code: 255; Stdin: ; Stdout: ; Stderr: sysctl: cannot stat /proc/sys/net/ipv6/conf/default/accept_ra: No such file or directory
2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent
2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent
2019-03-04 11:55:58.956 14992 INFO neutron.agent.dhcp.agent [-] Finished network d2b60324-af2b-458e-bc98-21026e00c953 dhcp configuration
2019-03-04 11:55:58.956 14992 INFO neutron.agent.dhcp.agent [req-c7d6f7cb-4229-4143-8b50-a1fb3af11b13 - - - - -] Synchronizing state complete

Let me know if anymore info is required. This is a really weird scenario which I am able to reproduce with any instance created on this openstack host.

Below is a 70kb file for which transfer is still going on after 2 hours. At the destination only 10kb has been copied.

[root@test-server ~]# scp  /var/cache/jenkins/war/WEB-INF/lib/remoting-3.29.jar dscadmin@1.20.28.146:/home/dscadmin/jenkins/remoting-3.29.jar                                                                                         100%  771KB  35.0KB/s   00:22    [

System SPECs:

[root@newton-1 neutron]# openstack --version
openstack 3.2.1

[root@newton-1 neutron]# uname -r
3.10.0-514.26.2.el7.x86_64

Following are some log files i tracked for this at /var/log/neutron

[root@newton-1 neutron]# tail -f dhcp-agent.log
2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent     'value': value}])
2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 912, in execute
2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent     log_fail_as_error=log_fail_as_error, **kwargs)
2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 148, in execute
2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent     raise ProcessExecutionError(msg, returncode=returncode)
2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent ProcessExecutionError: Exit code: 255; Stdin: ; Stdout: ; Stderr: sysctl: cannot stat /proc/sys/net/ipv6/conf/default/accept_ra: No such file or directory
2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent
2019-03-04 11:55:58.955 14992 ERROR neutron.agent.dhcp.agent
2019-03-04 11:55:58.956 14992 INFO neutron.agent.dhcp.agent [-] Finished network d2b60324-af2b-458e-bc98-21026e00c953 dhcp configuration
2019-03-04 11:55:58.956 14992 INFO neutron.agent.dhcp.agent [req-c7d6f7cb-4229-4143-8b50-a1fb3af11b13 - - - - -] Synchronizing state complete

Let me know if anymore info is required. This is a really weird scenario which I am able to reproduce with any instance created on this openstack host.