tripleo

periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-train job fails with pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'overcloud.ctlplane.ooo.test' ([Errno 113] EHOSTUNREACH)")

Bug #1887633 reported by Pooja Jadhav on 2020-07-15

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Fix Released	Critical	Unassigned	tripleo victoria-3 "tripleo victoria"

Bug Description

I am adding multinode-ipa job for stable/train [1] while testing, it fails having below trace back :

2020-07-16 06:53:51.652 24 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 10 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'overcloud.ctlplane.ooo.test' ([Errno 113] No route to host)")
2020-07-16 06:54:04.768 24 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 9 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'overcloud.ctlplane.ooo.test' ([Errno 113] No route to host)")
2020-07-16 06:54:17.888 24 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 8 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'overcloud.ctlplane.ooo.test' ([Errno 113] No route to host)")
2020-07-16 06:54:31.008 24 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 7 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'overcloud.ctlplane.ooo.test' ([Errno 113] No route to host)")
2020-07-16 06:54:44.128 24 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 6 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'overcloud.ctlplane.ooo.test' ([Errno 113] No route to host)")
2020-07-16 06:54:57.184 24 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 5 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'overcloud.ctlplane.ooo.test' ([Errno 113] No route to host)")
2020-07-16 06:55:10.304 24 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 4 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'overcloud.ctlplane.ooo.test' ([Errno 113] No route to host)")
2020-07-16 06:55:23.425 24 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 3 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'overcloud.ctlplane.ooo.test' ([Errno 113] No route to host)")
2020-07-16 06:55:36.544 24 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 2 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'overcloud.ctlplane.ooo.test' ([Errno 113] No route to host)")
2020-07-16 06:55:49.664 24 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 1 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'overcloud.ctlplane.ooo.test' ([Errno 113] No route to host)")
2020-07-16 06:55:59.666 24 CRITICAL glance [-] Unhandled error: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'overcloud.ctlplane.ooo.test' ([Errno 113] No route to host)")
(Background on this error at: http://sqlalche.me/e/e3q8)
2020-07-16 06:55:59.666 24 ERROR glance Traceback (most recent call last):
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 920, in connect
2020-07-16 06:55:59.666 24 ERROR glance **kwargs)
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib64/python3.6/socket.py", line 724, in create_connection
2020-07-16 06:55:59.666 24 ERROR glance raise err
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib64/python3.6/socket.py", line 713, in create_connection
2020-07-16 06:55:59.666 24 ERROR glance sock.connect(sa)
2020-07-16 06:55:59.666 24 ERROR glance OSError: [Errno 113] No route to host
2020-07-16 06:55:59.666 24 ERROR glance
2020-07-16 06:55:59.666 24 ERROR glance During handling of the above exception, another exception occurred:
2020-07-16 06:55:59.666 24 ERROR glance
2020-07-16 06:55:59.666 24 ERROR glance Traceback (most recent call last):
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect
2020-07-16 06:55:59.666 24 ERROR glance return fn()
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 294, in unique_connection
2020-07-16 06:55:59.666 24 ERROR glance return _ConnectionFairy._checkout(self)
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 751, in _checkout
2020-07-16 06:55:59.666 24 ERROR glance fairy = _ConnectionRecord.checkout(pool)
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 483, in checkout
2020-07-16 06:55:59.666 24 ERROR glance rec = pool._do_get()
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/impl.py", line 138, in _do_get
2020-07-16 06:55:59.666 24 ERROR glance self._dec_overflow()
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
2020-07-16 06:55:59.666 24 ERROR glance compat.reraise(exc_type, exc_value, exc_tb)
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 129, in reraise
2020-07-16 06:55:59.666 24 ERROR glance raise value
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/impl.py", line 135, in _do_get
2020-07-16 06:55:59.666 24 ERROR glance return self._create_connection()
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 299, in _create_connection
2020-07-16 06:55:59.666 24 ERROR glance return _ConnectionRecord(self)
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 428, in __init__
2020-07-16 06:55:59.666 24 ERROR glance self.__connect(first_connect_check=True)
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 630, in __connect
2020-07-16 06:55:59.666 24 ERROR glance connection = pool._invoke_creator(self)
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
2020-07-16 06:55:59.666 24 ERROR glance return dialect.connect(*cargs, **cparams)
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 453, in connect
2020-07-16 06:55:59.666 24 ERROR glance return self.dbapi.connect(*cargs, **cparams)
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib/python3.6/site-packages/pymysql/__init__.py", line 90, in Connect
2020-07-16 06:55:59.666 24 ERROR glance return Connection(*args, **kwargs)
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 699, in __init__
2020-07-16 06:55:59.666 24 ERROR glance self.connect()
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 967, in connect
2020-07-16 06:55:59.666 24 ERROR glance raise exc
2020-07-16 06:55:59.666 24 ERROR glance pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'overcloud.ctlplane.ooo.test' ([Errno 113] No route to host)")
2020-07-16 06:55:59.666 24 ERROR glance
2020-07-16 06:55:59.666 24 ERROR glance The above exception was the direct cause of the following exception:
2020-07-16 06:55:59.666 24 ERROR glance
2020-07-16 06:55:59.666 24 ERROR glance Traceback (most recent call last):
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/bin/glance-manage", line 10, in <module>
2020-07-16 06:55:59.666 24 ERROR glance sys.exit(main())
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib/python3.6/site-packages/glance/cmd/manage.py", line 563, in main
2020-07-16 06:55:59.666 24 ERROR glance return CONF.command.action_fn()
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib/python3.6/site-packages/glance/cmd/manage.py", line 395, in sync
2020-07-16 06:55:59.666 24 ERROR glance self.command_object.sync(CONF.command.version)
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib/python3.6/site-packages/glance/cmd/manage.py", line 154, in sync
2020-07-16 06:55:59.666 24 ERROR glance curr_heads = alembic_migrations.get_current_alembic_heads()
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib/python3.6/site-packages/glance/db/sqlalchemy/alembic_migrations/__init__.py", line 52, in get_current_alembic_heads
2020-07-16 06:55:59.666 24 ERROR glance engine = db_api.get_engine()
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib/python3.6/site-packages/glance/db/sqlalchemy/api.py", line 97, in get_engine
2020-07-16 06:55:59.666 24 ERROR glance facade = _create_facade_lazily()
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib/python3.6/site-packages/glance/db/sqlalchemy/api.py", line 87, in _create_facade_lazily
2020-07-16 06:55:59.666 24 ERROR glance _FACADE = session.EngineFacade.from_config(CONF)
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1374, in from_config
2020-07-16 06:55:59.666 24 ERROR glance expire_on_commit=expire_on_commit, _conf=conf)
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1295, in __init__
2020-07-16 06:55:59.666 24 ERROR glance slave_connection=slave_connection)
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 510, in _start
2020-07-16 06:55:59.666 24 ERROR glance engine_args, maker_args)
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 534, in _setup_for_connection
2020-07-16 06:55:59.666 24 ERROR glance sql_connection=sql_connection, **engine_kwargs)
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib/python3.6/site-packages/debtcollector/renames.py", line 43, in decorator
2020-07-16 06:55:59.666 24 ERROR glance return wrapped(*args, **kwargs)
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/engines.py", line 201, in create_engine
2020-07-16 06:55:59.666 24 ERROR glance test_conn = _test_connection(engine, max_retries, retry_interval)
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib/python3.6/site-packages/oslo_db/sqlalchemy/engines.py", line 387, in _test_connection
2020-07-16 06:55:59.666 24 ERROR glance six.reraise(type(de_ref), de_ref)
2020-07-16 06:55:59.666 24 ERROR glance File "/usr/lib/python3.6/site-packages/six.py", line 702, in reraise
2020-07-16 06:55:59.666 24 ERROR glance raise value.with_traceback(tb)
2020-07-16 06:55:59.666 24 ERROR glance oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'overcloud.ctlplane.ooo.test' ([Errno 113] No route to host)")
2020-07-16 06:55:59.666 24 ERROR glance (Background on this error at: http://sqlalche.me/e/e3q8)
2020-07-16 06:55:59.666 24 ERROR glance

Reference Links :

https://logserver.rdoproject.org/84/28284/10/check/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-train/df027a6/logs/undercloud/var/log/containers/glance/api.log.txt.gz

https://logserver.rdoproject.org/84/28284/10/check/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-train/df027a6/logs/undercloud/var/log/containers/mysql/mariadb.log.txt.gz

https://logserver.rdoproject.org/84/28284/10/check/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-train/df027a6/logs/undercloud/var/log/extra/failed_containers.log.txt.gz

https://logserver.rdoproject.org/84/28284/10/check/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-train/df027a6/logs/undercloud/var/log/extra/network.txt.gz

https://logserver.rdoproject.org/84/28284/10/check/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-train/df027a6/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

[1] https://review.rdoproject.org/r/#/c/28284/

See original description

Tags:

Pooja Jadhav (poojajadhav22) on 2020-07-16

description:

updated

Revision history for this message

wes hayutin (weshayutin) wrote on 2020-08-24:

Latest hit:
https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-tripleo-train/8de8403/logs/undercloud/var/log/paunch.log.txt.gz

Changed in tripleo:
importance:	Undecided → Critical
milestone:	none → victoria-3
tags:	added: promotion-blocker

Revision history for this message

Harald Jensås (harald-jensas) wrote on 2020-08-24:

The VIP does'nt get configured:

https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-tripleo-train/8de8403/logs/undercloud/home/zuul/tripleo_deploy.sh.txt.gz

DEPLOY_CONTROL_VIP=192.168.24.210

### IPv4 addresses
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000
    inet 192.168.101.9/22 brd 192.168.103.255 scope global dynamic noprefixroute eth0
       valid_lft 79641sec preferred_lft 79641sec
4: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 192.168.24.2/24 scope global br-ex
       valid_lft forever preferred_lft forever
6: br-ctlplane: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 192.168.24.1/24 brd 192.168.24.255 scope global br-ctlplane
       valid_lft forever preferred_lft forever

VIP is not in os-net-config ^

https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/net-config-standalone.j2.yaml#L133-L150

The VIP should be there ^

The hosts port map file is written correctly by tripleoclient/v1/tripleo_deploy.py ... and it should be in the environment - https://github.com/openstack/python-tripleoclient/blob/94344ae5b6a2174b4ece8173d9f6c6346f233cb1/tripleoclient/v1/tripleo_deploy.py#L710

It looks like, the condition here: https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/net-config-standalone.j2.yaml#L93 is not True, so the VIP is'nt in the network config? But afict the tripleoclient-hosts-portmaps.yaml which should be in the environment set's up the portmaps correctly.

The VIP does'nt get configured:

DEPLOY_CONTROL_VIP=192.168.24.210

VIP is not in os-net-config ^

https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/net-config-standalone.j2.yaml#L133-L150

The VIP should be there ^

wes hayutin (weshayutin) on 2020-10-26

Changed in tripleo:
status:	Triaged → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.