Bug #1803703 “[master][promotion] standalone job is failing duri...” : Bugs : tripleo

chandan kumar (chkumar246) on 2018-11-16

summary:

- [master] master standalone job is failing during undercloud install with
- Server Error for url: https://192.168.24.2:13696/v2.0/network
+ [master][promotion] standalone job is failing during undercloud install
+ with Server Error for url: https://192.168.24.2:13696/v2.0/network

Revision history for this message

Cédric Jeanneret (cjeanner) wrote on 2018-11-16:

#1

So the 504 is probably returned by haproxy directly, meaning the service behind it is down or not answering for some reason.

We can see in netstat haproxy is listening on that port (13696) at least:
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-master/0323c3f/logs/undercloud/var/log/extra/netstat.txt.gz

Revision history for this message

Harald Jensås (harald-jensas) wrote on 2018-11-16:

#2

Download full text (4.9 KiB)

Long story below.
Short story: Rabbit MQ is not running

## The first command in UndercloudCtlplaneNetworkDeployment would be [1], GET networks?name=ctlplane can be seen in the neutron log.

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-master/0323c3f/logs/undercloud/var/log/containers/neutron/server.log.txt.gz#_2018-11-16_12_01_34_564

2018-11-16 12:01:34.564 31 INFO neutron.wsgi [req-452824ab-2c8a-4a91-a627-0204a057c95b 4f57adfa456741acb057996b874bfd3b 1a495f2157094df0aed147f2115de58b - default default] 192.168.24.1 "GET /v2.0/networks?name=ctlplane HTTP/1.1" status: 200 len: 189 time: 1.0968869

Then not much happening in the neutron logs, and we the next thing in UndercloudCtlplaneNetworkDeployment POST command[2] gets the 504.

## The 504 ::

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-master/0323c3f/logs/undercloud/var/log/journal.txt.gz#_Nov_16_12_03_34

Nov 16 12:03:34 upstream-centos-7-rdo-cloud-tripleo-0000299802.tripleodomain.exa haproxy[42025]: 192.168.24.2:49982 [16/Nov/2018:12:01:34.564] neutron~ neutron/upstream-centos-7-rdo-cloud-tripleo-0000299802.internalapi.localdomain 3/0/0/-1/120004 504 194 - - sH-- 53/0/0/0/0 0/0 "POST /v2.0/networks HTTP/1.1"

At the same time Neutrons seems to be up, and responding

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-master/0323c3f/logs/undercloud/var/log/containers/neutron/server.log.txt.gz#_2018-11-16_12_03_35_231

What I find strange is what is not happening in neutron logs right after it starts:
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-master/0323c3f/logs/undercloud/var/log/containers/neutron/server.log.txt.gz

Ususally we get some RPC related stuff such as:

neutron.services.l3_router.l3_router_plugin [-] neutron.services.l3_router.l3_router_plugin.L3RouterPlugin method start_rpc_listeners
neutron.api.rpc.handlers.dhcp_rpc [req-49a9edd9-d581-4806-93e2-a6a872d50f4c - - - - -] get_active_networks_info from undercloud.localdomain

So, RPC ... Where is the rabbit mq logs?
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-master/0323c3f/logs/undercloud/var/log/containers/

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-master/0323c3f/logs/undercloud/var/log/extra/docker/containers/rabbitmq/stdout.log.txt.gz

[<0.32.0>,<0.31.0>]
[]
true
running
987
27
192
initial_call: pid: registered_name: error_info: ancestors: messages: links: dictionary: trap_exit: status: heap_size: stack_size: reductions: 2018-11-16 11:57:21 std_info
kernel
{{shutdown,{failed_to_start_child,inet_db,{'EXIT',{function_clause,[{inet_config,set_hostname,[{error,enametoolong}],[{file,"inet_config.erl"},{line,228}...

Long story below.
Short story: Rabbit MQ is not running

## The first command in UndercloudCtlplaneNetworkDeployment would be [1], GET networks?name=ctlplane can be seen in the neutron log.

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-master/0323c3f/logs/undercloud/var/log/containers/neutron/server.log.txt.gz#_2018-11-16_12_01_34_564

2018-11-16 12:01:34.564 31 INFO neutron.wsgi [req-452824ab-2c8a-4a91-a627-0204a057c95b 4f57adfa456741acb057996b874bfd3b 1a495f2157094df0aed147f2115de58b - default default] 192.168.24.1 "GET /v2.0/networks?name=ctlplane HTTP/1.1" status: 200  len: 189 time: 1.0968869

Then not much happening in the neutron logs, and we the next thing in UndercloudCtlplaneNetworkDeployment POST command[2] gets the 504.

## The 504 ::

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-master/0323c3f/logs/undercloud/var/log/journal.txt.gz#_Nov_16_12_03_34

Nov 16 12:03:34 upstream-centos-7-rdo-cloud-tripleo-0000299802.tripleodomain.exa haproxy[42025]: 192.168.24.2:49982 [16/Nov/2018:12:01:34.564] neutron~ neutron/upstream-centos-7-rdo-cloud-tripleo-0000299802.internalapi.localdomain 3/0/0/-1/120004 504 194 - - sH-- 53/0/0/0/0 0/0 "POST /v2.0/networks HTTP/1.1"

At the same time Neutrons seems to be up, and responding

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-master/0323c3f/logs/undercloud/var/log/containers/neutron/server.log.txt.gz#_2018-11-16_12_03_35_231

What I find strange is what is not happening in neutron logs right after it starts:
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-master/0323c3f/logs/undercloud/var/log/containers/neutron/server.log.txt.gz

Ususally we get some RPC related stuff such as:

neutron.services.l3_router.l3_router_plugin [-] neutron.services.l3_router.l3_router_plugin.L3RouterPlugin method start_rpc_listeners
  neutron.api.rpc.handlers.dhcp_rpc [req-49a9edd9-d581-4806-93e2-a6a872d50f4c - - - - -] get_active_networks_info from undercloud.localdomain

So, RPC ... Where is the rabbit mq logs?
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-master/0323c3f/logs/undercloud/var/log/containers/

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-master/0323c3f/logs/undercloud/var/log/extra/docker/containers/rabbitmq/stdout.log.txt.gz

[<0.32.0>,<0.31.0>]
[]
true
running
987
27
192
    initial_call:     pid:     registered_name:     error_info:     ancestors:     messages:     links:     dictionary:     trap_exit:     status:     heap_size:     stack_size:     reductions: 2018-11-16 11:57:21 std_info
kernel
{{shutdown,{failed_to_start_child,inet_db,{'EXIT',{function_clause,[{inet_config,set_hostname,[{error,enametoolong}],[{file,"inet_config.erl"},{line,228}]},{inet_config,init,0,[{file,"inet_config.erl"},{line,48}]},{inet_db,start_link,0,[{file,"inet_db.erl"},{line,113}]},{supervisor,do_start_child,2,[{file,"supervisor.erl"},{line,365}]},{supervisor,start_children,3,[{file,"supervisor.erl"},{line,348}]},{supervisor,init_children,2,[{file,"supervisor.erl"},{line,314}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}}},{kernel,start,[normal,[]]}}
permanent
    application:     exited:     type: Kernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,inet_db,{'EXIT',{function_clause,[{inet_config,set_hostname,[{error,enametoolong}],[

{"Kernel pid terminated",application_controller,"{application_start_failure,kernel,{{shutdown,{failed_to_start_child,inet_db,{'EXIT',{function_clause,[{inet_config,set_hostname,[{error,enametoolong}],[{file,\"inet_config.erl\"},{line,228}]},{inet_config,init,0,[{file,\"inet_config.erl\"},{line,48}]},{inet_db,start_link,0,[{file,\"inet_db.erl\"},{line,113}]},{supervisor,do_start_child,2,[{file,\"supervisor.erl\"},{line,365}]},{supervisor,start_children,3,[{file,\"supervisor.erl\"},{line,348}]},{supervisor,init_children,2,[{file,\"supervisor.erl\"},{line,314}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,247}]}]}}}},{kernel,start,[normal,[]]}}}"}
Crash dump is being written to: erl_crash.dump...

[1] https://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/post_deploy/undercloud_ctlplane_network.py#L47
[2] https://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/post_deploy/undercloud_ctlplane_network.py#L49-L53

Revision history for this message

Harald Jensås (harald-jensas) wrote on 2018-11-16:

#3

Looks like we need this change:

https://github.com/bugs-erlang-org/otp/commit/5da9e0ab5cb96f2cb37e534b2c1a55c0c2c9783c

Revision history for this message

Harald Jensås (harald-jensas) wrote on 2018-11-16:

#4

Or ... we make the hostname shorter:
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-master/0323c3f/logs/undercloud/etc/hostname.txt.gz

We are trying to use a hostname longer than what is supported by linux here?

GETHOSTNAME(2)

On Linux, HOST_NAME_MAX is defined with the value 64, which has been the limit since Linux 1.0 (earlier kernels imposed a limit of 8 bytes).

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-11-16: Fix proposed to tripleo-quickstart-extras (master)

#5

Fix proposed to branch: master
Review: https://review.openstack.org/618588

Changed in tripleo:
assignee:	nobody → Harald Jensås (harald-jensas)
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-11-19: Related fix proposed to tripleo-quickstart (master)

#6

Related fix proposed to branch: master
Review: https://review.openstack.org/618773

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-11-20: Related fix merged to tripleo-quickstart (master)

#7

Reviewed: https://review.openstack.org/618773
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=4453efa2f8925821fd65dfb45e2e3ac64bbd6665
Submitter: Zuul
Branch: master

commit 4453efa2f8925821fd65dfb45e2e3ac64bbd6665
Author: Harald Jensås <email address hidden>
Date: Mon Nov 19 17:00:17 2018 +0100

Use a shorter default domain name

    Some of the CI jobs use very long <hostname>'s,
    resulting in the FQDN to be more than 63 characters
    long which is the max for linux, see GETHOSTNAME(2).

Change the default to use: <hostname>.ooo.test instead.

Related-Bug: #1803703
Change-Id: Ia8f758a094d36c3f3bec383f1d3a8e1a5d4bd052

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-11-21: Fix merged to tripleo-quickstart-extras (master)

#8

Reviewed: https://review.openstack.org/618588
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=a31f99546aa512b50b82be646030512db4efa685
Submitter: Zuul
Branch: master

commit a31f99546aa512b50b82be646030512db4efa685
Author: Harald Jensås <email address hidden>
Date: Fri Nov 16 20:38:31 2018 +0100

Use a shorter default domain name

    Since https://review.openstack.org/615730 we default
    to <hostname>.tripleodomain.example.com if hostname is
    not set.

    Some of the CI jobs use very long <hostname>'s, resulting
    in the FQDN to be more than 63 characters long which is
    the max for linux, see GETHOSTNAME(2).

Change the default to use: <hostname>.ooo.test instead.

Also, yamllint complain about indentation.

Closes-Bug: #1803703
Change-Id: If041b1c6e1da1d89d66ffafbbc6ab6c33bd80801

Changed in tripleo:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-05-14: Fix included in openstack/tripleo-quickstart-extras 2.1.1

#9

This issue was fixed in the openstack/tripleo-quickstart-extras 2.1.1 release.

tripleo

[master][promotion] standalone job is failing during undercloud install with Server Error for url: https://192.168.24.2:13696/v2.0/network

Bug Description

Other bug subscribers

Remote bug watches