Seen in WIP ironic CI test:
TASK [ironic : Running Ironic Inspector bootstrap container] *******************
task path: /home/zuul/src/opendev.org/openstack/kolla-ansible/ansible/roles/ironic/tasks/bootstrap_service.yml:27
<23.253.160.154> ESTABLISH SSH CONNECTION FOR USER: kolla
<23.253.160.154> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/zuul/.ssh/id_rsa_kolla"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="kolla"' -o ConnectTimeout=10 -o ControlPath=/home/zuul/.ansible/cp/25772d2a5f 23.253.160.154 '/bin/sh -c '"'"'echo ~kolla && sleep 0'"'"''
<23.253.160.154> (0, '/home/kolla\n', '')
<23.253.160.154> ESTABLISH SSH CONNECTION FOR USER: kolla
<23.253.160.154> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/zuul/.ssh/id_rsa_kolla"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="kolla"' -o ConnectTimeout=10 -o ControlPath=/home/zuul/.ansible/cp/25772d2a5f 23.253.160.154 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/kolla/.ansible/tmp/ansible-tmp-1559570225.06-90034726882536 `" && echo ansible-tmp-1559570225.06-90034726882536="` echo /home/kolla/.ansible/tmp/ansible-tmp-1559570225.06-90034726882536 `" ) && sleep 0'"'"''
<23.253.160.154> (0, 'ansible-tmp-1559570225.06-90034726882536=/home/kolla/.ansible/tmp/ansible-tmp-1559570225.06-90034726882536\n', '')
Using module file /home/zuul/src/opendev.org/openstack/kolla-ansible/ansible/library/kolla_docker.py
<23.253.160.154> PUT /home/zuul/.ansible/tmp/ansible-local-13134L07LR2/tmpF4mwj5 TO /home/kolla/.ansible/tmp/ansible-tmp-1559570225.06-90034726882536/AnsiballZ_kolla_docker.py
<23.253.160.154> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/zuul/.ssh/id_rsa_kolla"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="kolla"' -o ConnectTimeout=10 -o ControlPath=/home/zuul/.ansible/cp/25772d2a5f '[23.253.160.154]'
<23.253.160.154> (0, 'sftp> put /home/zuul/.ansible/tmp/ansible-local-13134L07LR2/tmpF4mwj5 /home/kolla/.ansible/tmp/ansible-tmp-1559570225.06-90034726882536/AnsiballZ_kolla_docker.py\n', '')
<23.253.160.154> ESTABLISH SSH CONNECTION FOR USER: kolla
<23.253.160.154> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/zuul/.ssh/id_rsa_kolla"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="kolla"' -o ConnectTimeout=10 -o ControlPath=/home/zuul/.ansible/cp/25772d2a5f 23.253.160.154 '/bin/sh -c '"'"'chmod u+x /home/kolla/.ansible/tmp/ansible-tmp-1559570225.06-90034726882536/ /home/kolla/.ansible/tmp/ansible-tmp-1559570225.06-90034726882536/AnsiballZ_kolla_docker.py && sleep 0'"'"''
<23.253.160.154> (0, '', '')
<23.253.160.154> ESTABLISH SSH CONNECTION FOR USER: kolla
<23.253.160.154> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/zuul/.ssh/id_rsa_kolla"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="kolla"' -o ConnectTimeout=10 -o ControlPath=/home/zuul/.ansible/cp/25772d2a5f -tt 23.253.160.154 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-qmlqxafdugapubjysnotmbhypqewahxy ; /usr/bin/python /home/kolla/.ansible/tmp/ansible-tmp-1559570225.06-90034726882536/AnsiballZ_kolla_docker.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
<23.253.160.154> (1, 'BECOME-SUCCESS-qmlqxafdugapubjysnotmbhypqewahxy\r\n\r\n{"stdout": "", "changed": true, "failed": true, "stderr": "+ sudo -E kolla_set_configs\\nINFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\\nINFO:__main__:Validating config file\\nINFO:__main__:Kolla config strategy set to: COPY_ALWAYS\\nINFO:__main__:Copying service configuration files\\nINFO:__main__:Deleting /etc/ironic-inspector/inspector.conf\\nINFO:__main__:Copying /var/lib/kolla/config_files/inspector.conf to /etc/ironic-inspector/inspector.conf\\nINFO:__main__:Setting permission for /etc/ironic-inspector/inspector.conf\\nINFO:__main__:Writing out command to execute\\n++ cat /run_command\\n+ CMD=\'ironic-inspector --config-file /etc/ironic-inspector/inspector.conf\'\\n+ ARGS=\\n+ [[ ! -n \'\' ]]\\n+ . kolla_extend_start\\n++ LOG_PATH=/var/log/kolla/ironic-inspector\\n++ [[ ! -d /var/log/kolla/ironic-inspector ]]\\n++ mkdir -p /var/log/kolla/ironic-inspector\\n+++ stat -c %a /var/log/kolla/ironic-inspector\\n++ [[ 2755 != \\\\7\\\\5\\\\5 ]]\\n++ chmod 755 /var/log/kolla/ironic-inspector\\n++ [[ -n 0 ]]\\n++ ironic-inspector-dbsync --config-file /etc/ironic-inspector/inspector.conf upgrade\\nTraceback (most recent call last):\\n File \\"/usr/bin/ironic-inspector-dbsync\\", line 10, in <module>\\n sys.exit(main())\\n File \\"/usr/lib/python3/dist-packages/ironic_inspector/dbsync.py\\", line 92, in main\\n CONF.command.func(config, CONF.command.name)\\n File \\"/usr/lib/python3/dist-packages/ironic_inspector/dbsync.py\\", line 74, in with_revision\\n do_alembic_command(config, cmd, revision)\\n File \\"/usr/lib/python3/dist-packages/ironic_inspector/dbsync.py\\", line 79, in do_alembic_command\\n getattr(alembic_command, cmd)(config, *args, **kwargs)\\n File \\"/usr/lib/python3/dist-packages/alembic/command.py\\", line 254, in upgrade\\n script.run_env()\\n File \\"/usr/lib/python3/dist-packages/alembic/script/base.py\\", line 427, in run_env\\n util.load_python_file(self.dir, \'env.py\')\\n File \\"/usr/lib/python3/dist-packages/alembic/util/pyfiles.py\\", line 81, in load_python_file\\n module = load_module_py(module_id, path)\\n File \\"/usr/lib/python3/dist-packages/alembic/util/compat.py\\", line 82, in load_module_py\\n spec.loader.exec_module(module)\\n File \\"<frozen importlib._bootstrap_external>\\", line 678, in exec_module\\n File \\"<frozen importlib._bootstrap>\\", line 219, in _call_with_frames_removed\\n File \\"/usr/lib/python3/dist-packages/ironic_inspector/migrations/env.py\\", line 83, in <module>\\n run_migrations_online()\\n File \\"/usr/lib/python3/dist-packages/ironic_inspector/migrations/env.py\\", line 69, in run_migrations_online\\n session = db.get_writer_session()\\n File \\"/usr/lib/python3/dist-packages/ironic_inspector/db.py\\", line 206, in get_writer_session\\n return get_context_manager().writer.get_sessionmaker()()\\n File \\"/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py\\", line 848, in get_sessionmaker\\n return self._factory.get_writer_maker()\\n File \\"/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py\\", line 392, in get_writer_maker\\n self._start()\\n File \\"/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py\\", line 510, in _start\\n engine_args, maker_args)\\n File \\"/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py\\", line 534, in _setup_for_connection\\n sql_connection=sql_connection, **engine_kwargs)\\n File \\"/usr/lib/python3/dist-packages/debtcollector/renames.py\\", line 43, in decorator\\n return wrapped(*args, **kwargs)\\n File \\"/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/engines.py\\", line 177, in create_engine\\n engine = sqlalchemy.create_engine(url, **engine_args)\\n File \\"/usr/lib/python3/dist-packages/sqlalchemy/engine/__init__.py\\", line 425, in create_engine\\n return strategy.create(*args, **kwargs)\\n File \\"/usr/lib/python3/dist-packages/sqlalchemy/engine/strategies.py\\", line 81, in create\\n dbapi = dialect_cls.dbapi(**dbapi_args)\\n File \\"/usr/lib/python3/dist-packages/sqlalchemy/dialects/mysql/pymysql.py\\", line 62, in dbapi\\n return __import__(\'pymysql\')\\nModuleNotFoundError: No module named \'pymysql\'\\n", "rc": 1, "invocation": {"module_args": {"tty": false, "tls_key": null, "image": "kolla/ubuntu-binary-ironic-inspector:master", "labels": {"BOOTSTRAP": null}, "tls_verify": false, "tls_cacert": null, "auth_password": null, "graceful_timeout": 10, "environment": {"KOLLA_SERVICE_NAME": "bootstrap-ironic-inspector", "KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS", "KOLLA_BOOTSTRAP": null}, "auth_registry": null, "state": "running", "volumes_from": null, "security_opt": [], "privileged": false, "api_version": "auto", "remove_on_exit": true, "restart_retries": 10, "detach": false, "auth_username": null, "dimensions": {}, "name": "bootstrap_ironic_inspector", "tls_cert": null, "cap_add": [], "restart_policy": "never", "auth_email": null, "command": null, "volumes": ["/etc/kolla//ironic-inspector/:/var/lib/kolla/config_files/:ro", "/etc/localtime:/etc/localtime:ro", "kolla_logs:/var/log/kolla", "ironic_inspector_dhcp_hosts:/var/lib/ironic-inspector/dhcp-hostsdir"], "action": "start_container"}}, "msg": "Container exited with non-zero return code 1"}\r\n', 'Shared connection to 23.253.160.154 closed.\r\n')
<23.253.160.154> Failed to connect to the host via ssh: Shared connection to 23.253.160.154 closed.
<23.253.160.154> ESTABLISH SSH CONNECTION FOR USER: kolla
<23.253.160.154> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/zuul/.ssh/id_rsa_kolla"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="kolla"' -o ConnectTimeout=10 -o ControlPath=/home/zuul/.ansible/cp/25772d2a5f 23.253.160.154 '/bin/sh -c '"'"'rm -f -r /home/kolla/.ansible/tmp/ansible-tmp-1559570225.06-90034726882536/ > /dev/null 2>&1 && sleep 0'"'"''
<23.253.160.154> (0, '', '')
fatal: [primary]: FAILED! => {
"changed": true,
"invocation": {
"module_args": {
"action": "start_container",
"api_version": "auto",
"auth_email": null,
"auth_password": null,
"auth_registry": null,
"auth_username": null,
"cap_add": [],
"command": null,
"detach": false,
"dimensions": {},
"environment": {
"KOLLA_BOOTSTRAP": null,
"KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS",
"KOLLA_SERVICE_NAME": "bootstrap-ironic-inspector"
},
"graceful_timeout": 10,
"image": "kolla/ubuntu-binary-ironic-inspector:master",
"labels": {
"BOOTSTRAP": null
},
"name": "bootstrap_ironic_inspector",
"privileged": false,
"remove_on_exit": true,
"restart_policy": "never",
"restart_retries": 10,
"security_opt": [],
"state": "running",
"tls_cacert": null,
"tls_cert": null,
"tls_key": null,
"tls_verify": false,
"tty": false,
"volumes": [
"/etc/kolla//ironic-inspector/:/var/lib/kolla/config_files/:ro",
"/etc/localtime:/etc/localtime:ro",
"kolla_logs:/var/log/kolla",
"ironic_inspector_dhcp_hosts:/var/lib/ironic-inspector/dhcp-hostsdir"
],
"volumes_from": null
}
},
"msg": "Container exited with non-zero return code 1",
"rc": 1,
"stderr": "+ sudo -E kolla_set_configs\nINFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json\nINFO:__main__:Validating config file\nINFO:__main__:Kolla config strategy set to: COPY_ALWAYS\nINFO:__main__:Copying service configuration files\nINFO:__main__:Deleting /etc/ironic-inspector/inspector.conf\nINFO:__main__:Copying /var/lib/kolla/config_files/inspector.conf to /etc/ironic-inspector/inspector.conf\nINFO:__main__:Setting permission for /etc/ironic-inspector/inspector.conf\nINFO:__main__:Writing out command to execute\n++ cat /run_command\n+ CMD='ironic-inspector --config-file /etc/ironic-inspector/inspector.conf'\n+ ARGS=\n+ [[ ! -n '' ]]\n+ . kolla_extend_start\n++ LOG_PATH=/var/log/kolla/ironic-inspector\n++ [[ ! -d /var/log/kolla/ironic-inspector ]]\n++ mkdir -p /var/log/kolla/ironic-inspector\n+++ stat -c %a /var/log/kolla/ironic-inspector\n++ [[ 2755 != \\7\\5\\5 ]]\n++ chmod 755 /var/log/kolla/ironic-inspector\n++ [[ -n 0 ]]\n++ ironic-inspector-dbsync --config-file /etc/ironic-inspector/inspector.conf upgrade\nTraceback (most recent call last):\n File \"/usr/bin/ironic-inspector-dbsync\", line 10, in <module>\n sys.exit(main())\n File \"/usr/lib/python3/dist-packages/ironic_inspector/dbsync.py\", line 92, in main\n CONF.command.func(config, CONF.command.name)\n File \"/usr/lib/python3/dist-packages/ironic_inspector/dbsync.py\", line 74, in with_revision\n do_alembic_command(config, cmd, revision)\n File \"/usr/lib/python3/dist-packages/ironic_inspector/dbsync.py\", line 79, in do_alembic_command\n getattr(alembic_command, cmd)(config, *args, **kwargs)\n File \"/usr/lib/python3/dist-packages/alembic/command.py\", line 254, in upgrade\n script.run_env()\n File \"/usr/lib/python3/dist-packages/alembic/script/base.py\", line 427, in run_env\n util.load_python_file(self.dir, 'env.py')\n File \"/usr/lib/python3/dist-packages/alembic/util/pyfiles.py\", line 81, in load_python_file\n module = load_module_py(module_id, path)\n File \"/usr/lib/python3/dist-packages/alembic/util/compat.py\", line 82, in load_module_py\n spec.loader.exec_module(module)\n File \"<frozen importlib._bootstrap_external>\", line 678, in exec_module\n File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\n File \"/usr/lib/python3/dist-packages/ironic_inspector/migrations/env.py\", line 83, in <module>\n run_migrations_online()\n File \"/usr/lib/python3/dist-packages/ironic_inspector/migrations/env.py\", line 69, in run_migrations_online\n session = db.get_writer_session()\n File \"/usr/lib/python3/dist-packages/ironic_inspector/db.py\", line 206, in get_writer_session\n return get_context_manager().writer.get_sessionmaker()()\n File \"/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py\", line 848, in get_sessionmaker\n return self._factory.get_writer_maker()\n File \"/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py\", line 392, in get_writer_maker\n self._start()\n File \"/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py\", line 510, in _start\n engine_args, maker_args)\n File \"/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py\", line 534, in _setup_for_connection\n sql_connection=sql_connection, **engine_kwargs)\n File \"/usr/lib/python3/dist-packages/debtcollector/renames.py\", line 43, in decorator\n return wrapped(*args, **kwargs)\n File \"/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/engines.py\", line 177, in create_engine\n engine = sqlalchemy.create_engine(url, **engine_args)\n File \"/usr/lib/python3/dist-packages/sqlalchemy/engine/__init__.py\", line 425, in create_engine\n return strategy.create(*args, **kwargs)\n File \"/usr/lib/python3/dist-packages/sqlalchemy/engine/strategies.py\", line 81, in create\n dbapi = dialect_cls.dbapi(**dbapi_args)\n File \"/usr/lib/python3/dist-packages/sqlalchemy/dialects/mysql/pymysql.py\", line 62, in dbapi\n return __import__('pymysql')\nModuleNotFoundError: No module named 'pymysql'\n",
"stderr_lines": [
"+ sudo -E kolla_set_configs",
"INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json",
"INFO:__main__:Validating config file",
"INFO:__main__:Kolla config strategy set to: COPY_ALWAYS",
"INFO:__main__:Copying service configuration files",
"INFO:__main__:Deleting /etc/ironic-inspector/inspector.conf",
"INFO:__main__:Copying /var/lib/kolla/config_files/inspector.conf to /etc/ironic-inspector/inspector.conf",
"INFO:__main__:Setting permission for /etc/ironic-inspector/inspector.conf",
"INFO:__main__:Writing out command to execute",
"++ cat /run_command",
"+ CMD='ironic-inspector --config-file /etc/ironic-inspector/inspector.conf'",
"+ ARGS=",
"+ [[ ! -n '' ]]",
"+ . kolla_extend_start",
"++ LOG_PATH=/var/log/kolla/ironic-inspector",
"++ [[ ! -d /var/log/kolla/ironic-inspector ]]",
"++ mkdir -p /var/log/kolla/ironic-inspector",
"+++ stat -c %a /var/log/kolla/ironic-inspector",
"++ [[ 2755 != \\7\\5\\5 ]]",
"++ chmod 755 /var/log/kolla/ironic-inspector",
"++ [[ -n 0 ]]",
"++ ironic-inspector-dbsync --config-file /etc/ironic-inspector/inspector.conf upgrade",
"Traceback (most recent call last):",
" File \"/usr/bin/ironic-inspector-dbsync\", line 10, in <module>",
" sys.exit(main())",
" File \"/usr/lib/python3/dist-packages/ironic_inspector/dbsync.py\", line 92, in main",
" CONF.command.func(config, CONF.command.name)",
" File \"/usr/lib/python3/dist-packages/ironic_inspector/dbsync.py\", line 74, in with_revision",
" do_alembic_command(config, cmd, revision)",
" File \"/usr/lib/python3/dist-packages/ironic_inspector/dbsync.py\", line 79, in do_alembic_command",
" getattr(alembic_command, cmd)(config, *args, **kwargs)",
" File \"/usr/lib/python3/dist-packages/alembic/command.py\", line 254, in upgrade",
" script.run_env()",
" File \"/usr/lib/python3/dist-packages/alembic/script/base.py\", line 427, in run_env",
" util.load_python_file(self.dir, 'env.py')",
" File \"/usr/lib/python3/dist-packages/alembic/util/pyfiles.py\", line 81, in load_python_file",
" module = load_module_py(module_id, path)",
" File \"/usr/lib/python3/dist-packages/alembic/util/compat.py\", line 82, in load_module_py",
" spec.loader.exec_module(module)",
" File \"<frozen importlib._bootstrap_external>\", line 678, in exec_module",
" File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed",
" File \"/usr/lib/python3/dist-packages/ironic_inspector/migrations/env.py\", line 83, in <module>",
" run_migrations_online()",
" File \"/usr/lib/python3/dist-packages/ironic_inspector/migrations/env.py\", line 69, in run_migrations_online",
" session = db.get_writer_session()",
" File \"/usr/lib/python3/dist-packages/ironic_inspector/db.py\", line 206, in get_writer_session",
" return get_context_manager().writer.get_sessionmaker()()",
" File \"/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py\", line 848, in get_sessionmaker",
" return self._factory.get_writer_maker()",
" File \"/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py\", line 392, in get_writer_maker",
" self._start()",
" File \"/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py\", line 510, in _start",
" engine_args, maker_args)",
" File \"/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py\", line 534, in _setup_for_connection",
" sql_connection=sql_connection, **engine_kwargs)",
" File \"/usr/lib/python3/dist-packages/debtcollector/renames.py\", line 43, in decorator",
" return wrapped(*args, **kwargs)",
" File \"/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/engines.py\", line 177, in create_engine",
" engine = sqlalchemy.create_engine(url, **engine_args)",
" File \"/usr/lib/python3/dist-packages/sqlalchemy/engine/__init__.py\", line 425, in create_engine",
" return strategy.create(*args, **kwargs)",
" File \"/usr/lib/python3/dist-packages/sqlalchemy/engine/strategies.py\", line 81, in create",
" dbapi = dialect_cls.dbapi(**dbapi_args)",
" File \"/usr/lib/python3/dist-packages/sqlalchemy/dialects/mysql/pymysql.py\", line 62, in dbapi",
" return __import__('pymysql')",
"ModuleNotFoundError: No module named 'pymysql'"
],
"stdout": "",
"stdout_lines": []
}
Fix proposed to branch: master /review. opendev. org/662788
Review: https:/