logrotate-charm

Logrotated charm in error state with hook failed: "install"

Bug #2003232 reported by Konstantinos Kaskavelis
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Prometheus Openstack Exporter Charm
Fix Released
High
Sudeep Bhandari
Prometheus Openstack Exporter Charm 23.01
logrotate-charm
Fix Released
High
Sudeep Bhandari
logrotate-charm 23.01

Bug Description

Solutions QA team has a failed run (SKU: undertest_yoga_focal_lma_release) in lma-maas layer where
logrotated stays in error state with the message hook failed: "install"

From the logs we see:

unit-logrotated-7: 12:26:19 WARNING unit.logrotated/7.install _update_if_newer(pip, pre_install_pkgs)
unit-logrotated-7: 12:26:19 WARNING unit.logrotated/7.install File "lib/charms/layer/basic.py", line 367, in _update_if_newer
unit-logrotated-7: 12:26:19 WARNING unit.logrotated/7.install pkg], env=_get_subprocess_env())
unit-logrotated-7: 12:26:19 WARNING unit.logrotated/7.install File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
unit-logrotated-7: 12:26:19 WARNING unit.logrotated/7.install raise CalledProcessError(retcode, cmd)
unit-logrotated-7: 12:26:19 WARNING unit.logrotated/7.install subprocess.CalledProcessError: Command '['/var/lib/juju/agents/unit-logrotated-7/.venv/bin/pip', 'install', '-U', '--no-index', '-f', 'wheelhouse', 'pip']' returned non-zero exit status 1.
unit-logrotated-7: 12:26:19 ERROR juju.worker.uniter.operation hook "install" (via explicit, bespoke hook script) failed: exit status 1
unit-logrotated-7: 12:26:19 DEBUG juju.machinelock created rotating log file "/var/log/juju/machine-lock.log" with max size 10 MB and max backups 5
unit-logrotated-7: 12:26:19 DEBUG juju.machinelock machine lock released for logrotated/7 uniter (run install hook)
unit-logrotated-7: 12:26:19 DEBUG juju.worker.uniter.operation lock released for logrotated/7
unit-logrotated-7: 12:26:19 INFO juju.worker.uniter awaiting error resolution for "install" hook
unit-logrotated-7: 12:26:19 DEBUG juju.worker.uniter [AGENT-STATUS] error: hook failed: "install"
unit-logrotated-7: 12:27:04 DEBUG juju.worker.uniter.remotestate update status timer triggered for logrotated/7
unit-logrotated-7: 12:27:04 INFO juju.worker.uniter awaiting error resolution for "install" hook
unit-logrotated-7: 12:31:19 DEBUG juju.worker.uniter.remotestate retry hook timer triggered for logrotated/7
unit-logrotated-7: 12:31:19 INFO juju.worker.uniter awaiting error resolution for "install" hook
unit-logrotated-7: 12:31:19 DEBUG juju.worker.uniter.operation running operation run install hook for logrotated/7
unit-logrotated-7: 12:31:19 DEBUG juju.machinelock acquire machine lock for logrotated/7 uniter (run install hook)
unit-logrotated-7: 12:31:19 DEBUG juju.machinelock machine lock acquired for logrotated/7 uniter (run install hook)
unit-logrotated-7: 12:31:19 DEBUG juju.worker.uniter.operation preparing operation "run install hook" for logrotated/7
unit-logrotated-7: 12:31:19 DEBUG juju.worker.uniter.operation executing operation "run install hook" for logrotated/7
unit-logrotated-7: 12:31:19 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running install hook
unit-logrotated-7: 12:31:19 DEBUG juju.worker.uniter.runner starting jujuc server {unix @/var/lib/juju/agents/unit-logrotated-7/agent.socket <nil>}
unit-logrotated-7: 12:31:20 DEBUG unit.logrotated/7.install Reading package lists...
unit-logrotated-7: 12:31:20 DEBUG unit.logrotated/7.install Building dependency tree...
unit-logrotated-7: 12:31:20 DEBUG unit.logrotated/7.install Reading state information...
unit-logrotated-7: 12:31:21 DEBUG unit.logrotated/7.install python3-wheel is already the newest version (0.30.0-0.2).
unit-logrotated-7: 12:31:21 DEBUG unit.logrotated/7.install build-essential is already the newest version (12.4ubuntu1).
unit-logrotated-7: 12:31:21 DEBUG unit.logrotated/7.install python3-setuptools is already the newest version (39.0.1-2).
unit-logrotated-7: 12:31:21 DEBUG unit.logrotated/7.install python3-yaml is already the newest version (3.12-1build2).
unit-logrotated-7: 12:31:21 DEBUG unit.logrotated/7.install python3-pip is already the newest version (9.0.1-2.3~ubuntu1.18.04.5).
unit-logrotated-7: 12:31:21 DEBUG unit.logrotated/7.install python3-dev is already the newest version (3.6.7-1~18.04).
unit-logrotated-7: 12:31:21 DEBUG unit.logrotated/7.install 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
unit-logrotated-7: 12:31:21 DEBUG unit.logrotated/7.install Reading package lists...
unit-logrotated-7: 12:31:22 DEBUG unit.logrotated/7.install Building dependency tree...
unit-logrotated-7: 12:31:22 DEBUG unit.logrotated/7.install Reading state information...
unit-logrotated-7: 12:31:22 DEBUG unit.logrotated/7.install 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
unit-logrotated-7: 12:31:24 DEBUG unit.logrotated/7.install Collecting pip
unit-logrotated-7: 12:31:25 WARNING unit.logrotated/7.install pip requires Python '>=3.7' but the running Python is 3.6.9
unit-logrotated-7: 12:31:25 WARNING unit.logrotated/7.install Traceback (most recent call last):
unit-logrotated-7: 12:31:25 WARNING unit.logrotated/7.install File "/var/lib/juju/agents/unit-logrotated-7/charm/hooks/install", line 8, in <module>
unit-logrotated-7: 12:31:25 WARNING unit.logrotated/7.install basic.bootstrap_charm_deps()
unit-logrotated-7: 12:31:25 WARNING unit.logrotated/7.install File "lib/charms/layer/basic.py", line 202, in bootstrap_charm_deps
unit-logrotated-7: 12:31:25 WARNING unit.logrotated/7.install _update_if_newer(pip, pre_install_pkgs)
unit-logrotated-7: 12:31:25 WARNING unit.logrotated/7.install File "lib/charms/layer/basic.py", line 367, in _update_if_newer
unit-logrotated-7: 12:31:25 WARNING unit.logrotated/7.install pkg], env=_get_subprocess_env())
unit-logrotated-7: 12:31:25 WARNING unit.logrotated/7.install File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
unit-logrotated-7: 12:31:25 WARNING unit.logrotated/7.install raise CalledProcessError(retcode, cmd)
unit-logrotated-7: 12:31:25 WARNING unit.logrotated/7.install subprocess.CalledProcessError: Command '['/var/lib/juju/agents/unit-logrotated-7/.venv/bin/pip', 'install', '-U', '--no-index', '-f', 'wheelhouse', 'pip']' returned non-zero exit status 1.
unit-logrotated-7: 12:31:25 ERROR juju.worker.uniter.operation hook "install" (via explicit, bespoke hook script) failed: exit status 1
unit-logrotated-7: 12:31:25 DEBUG juju.machinelock created rotating log file "/var/log/juju/machine-lock.log" with max size 10 MB and max backups 5
unit-logrotated-7: 12:31:25 DEBUG juju.machinelock machine lock released for logrotated/7 uniter (run install hook)
unit-logrotated-7: 12:31:25 DEBUG juju.worker.uniter.operation lock released for logrotated/7
unit-logrotated-7: 12:31:25 INFO juju.worker.uniter awaiting error resolution for "install" hook
unit-logrotated-7: 12:31:25 DEBUG juju.worker.uniter [AGENT-STATUS] error: hook failed: "install"
unit-logrotated-7: 12:31:43 DEBUG juju.worker.uniter.remotestate update status timer triggered for logrotated/7
unit-logrotated-7: 12:31:43 INFO juju.worker.uniter awaiting error resolution for "install" hook
unit-logrotated-7: 12:35:53 DEBUG juju.worker.uniter.remotestate update status timer triggered for logrotated/7
unit-logrotated-7: 12:35:53 INFO juju.worker.uniter awaiting error resolution for "install" hook
unit-logrotated-7: 12:36:25 DEBUG juju.worker.uniter.remotestate retry hook timer triggered for logrotated/7
unit-logrotated-7: 12:36:25 INFO juju.worker.uniter awaiting error resolution for "install" hook
unit-logrotated-7: 12:36:25 DEBUG juju.worker.uniter.operation running operation run install hook for logrotated/7
unit-logrotated-7: 12:36:25 DEBUG juju.machinelock acquire machine lock for logrotated/7 uniter (run install hook)
unit-logrotated-7: 12:36:25 DEBUG juju.machinelock machine lock acquired for logrotated/7 uniter (run install hook)
unit-logrotated-7: 12:36:25 DEBUG juju.worker.uniter.operation preparing operation "run install hook" for logrotated/7
unit-logrotated-7: 12:36:25 DEBUG juju.worker.uniter.operation executing operation "run install hook" for logrotated/7
unit-logrotated-7: 12:36:25 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running install hook
unit-logrotated-7: 12:36:25 DEBUG juju.worker.uniter.runner starting jujuc server {unix @/var/lib/juju/agents/unit-logrotated-7/agent.socket <nil>}
unit-logrotated-7: 12:36:26 DEBUG unit.logrotated/7.install Reading package lists...
unit-logrotated-7: 12:36:26 DEBUG unit.logrotated/7.install Building dependency tree...
unit-logrotated-7: 12:36:26 DEBUG unit.logrotated/7.install Reading state information...
unit-logrotated-7: 12:36:27 DEBUG unit.logrotated/7.install python3-wheel is already the newest version (0.30.0-0.2).
unit-logrotated-7: 12:36:27 DEBUG unit.logrotated/7.install build-essential is already the newest version (12.4ubuntu1).
unit-logrotated-7: 12:36:27 DEBUG unit.logrotated/7.install python3-setuptools is already the newest version (39.0.1-2).
unit-logrotated-7: 12:36:27 DEBUG unit.logrotated/7.install python3-yaml is already the newest version (3.12-1build2).
unit-logrotated-7: 12:36:27 DEBUG unit.logrotated/7.install python3-pip is already the newest version (9.0.1-2.3~ubuntu1.18.04.5).
unit-logrotated-7: 12:36:27 DEBUG unit.logrotated/7.install python3-dev is already the newest version (3.6.7-1~18.04).
unit-logrotated-7: 12:36:27 DEBUG unit.logrotated/7.install 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
unit-logrotated-7: 12:36:27 DEBUG unit.logrotated/7.install Reading package lists...
unit-logrotated-7: 12:36:27 DEBUG unit.logrotated/7.install Building dependency tree...
unit-logrotated-7: 12:36:27 DEBUG unit.logrotated/7.install Reading state information...
unit-logrotated-7: 12:36:27 DEBUG unit.logrotated/7.install 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
unit-logrotated-7: 12:36:29 DEBUG unit.logrotated/7.install Collecting pip
unit-logrotated-7: 12:36:29 WARNING unit.logrotated/7.install pip requires Python '>=3.7' but the running Python is 3.6.9
unit-logrotated-7: 12:36:29 WARNING unit.logrotated/7.install Traceback (most recent call last):
unit-logrotated-7: 12:36:29 WARNING unit.logrotated/7.install File "/var/lib/juju/agents/unit-logrotated-7/charm/hooks/install", line 8, in <module>
unit-logrotated-7: 12:36:29 WARNING unit.logrotated/7.install basic.bootstrap_charm_deps()
unit-logrotated-7: 12:36:29 WARNING unit.logrotated/7.install File "lib/charms/layer/basic.py", line 202, in bootstrap_charm_deps
unit-logrotated-7: 12:36:29 WARNING unit.logrotated/7.install _update_if_newer(pip, pre_install_pkgs)
unit-logrotated-7: 12:36:29 WARNING unit.logrotated/7.install File "lib/charms/layer/basic.py", line 367, in _update_if_newer
unit-logrotated-7: 12:36:29 WARNING unit.logrotated/7.install pkg], env=_get_subprocess_env())
unit-logrotated-7: 12:36:29 WARNING unit.logrotated/7.install File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
unit-logrotated-7: 12:36:29 WARNING unit.logrotated/7.install raise CalledProcessError(retcode, cmd)
unit-logrotated-7: 12:36:29 WARNING unit.logrotated/7.install subprocess.CalledProcessError: Command '['/var/lib/juju/agents/unit-logrotated-7/.venv/bin/pip', 'install', '-U', '--no-index', '-f', 'wheelhouse', 'pip']' returned non-zero exit status 1.
unit-logrotated-7: 12:36:29 ERROR juju.worker.uniter.operation hook "install" (via explicit, bespoke hook script) failed: exit status 1
unit-logrotated-7: 12:36:29 DEBUG juju.machinelock created rotating log file "/var/log/juju/machine-lock.log" with max size 10 MB and max backups 5
unit-logrotated-7: 12:36:29 DEBUG juju.machinelock machine lock released for logrotated/7 uniter (run install hook)
unit-logrotated-7: 12:36:29 DEBUG juju.worker.uniter.operation lock released for logrotated/7
unit-logrotated-7: 12:36:29 INFO juju.worker.uniter awaiting error resolution for "install" hook
unit-logrotated-7: 12:36:29 DEBUG juju.worker.uniter [AGENT-STATUS] error: hook failed: "install"

Logrotated in this test scenario is a subordinate charm of nagios (runs on bionic) which supports up to python 3.6.

Test run:

https://solutions.qa.canonical.com/v2/testruns/3cdec558-afd1-4165-9158-eb118cedf4aa

(log artifacts found in the link at the bottom of the test run page)

Related branches

~sudeephb/charm-prometheus-openstack-exporter:bionic_fix
Tianqi Xiao (community): Approve
Robert Gildein: Approve
🤖 prod-jenkaas-bootstack (community): Approve (continuous-integration)
BootStack Reviewers: Pending requested
BootStack Reviewers: Pending requested
Diff: 34 lines (+9/-3)
2 files modified
charmcraft.yaml (+5/-1)
src/tox.ini (+4/-2)
~sudeephb/charm-logrotated:master
Ramesh Sattaru (community): Approve
Martin Kalcok (community): Approve
🤖 prod-jenkaas-bootstack (community): Approve (continuous-integration)
BootStack Reviewers: Pending requested
Diff: 38 lines (+6/-6)
2 files modified
charmcraft.yaml (+5/-5)
src/tests/functional/test_logrotate.py (+1/-1)
Eric Chen (eric-chen)
tags: added: bseng-758
Sudeep Bhandari (sudeephb)
Changed in charm-logrotated:
assignee: nobody → Sudeep Bhandari (sudeephb)
Revision history for this message
Alexander Balderson (asbalderson) wrote :

we also saw this on charm-prometheus-openstack-exporter on bionic:

unit-prometheus-openstack-exporter-0: 20:05:30 WARNING unit.prometheus-openstack-exporter/0.install pip requires Python '>=3.7' but the running Python is 3.6.9
unit-prometheus-openstack-exporter-0: 20:05:30 WARNING unit.prometheus-openstack-exporter/0.install Traceback (most recent call last):
unit-prometheus-openstack-exporter-0: 20:05:30 WARNING unit.prometheus-openstack-exporter/0.install File "/var/lib/juju/agents/unit-prometheus-openstack-exporter-0/charm/hooks/install", line 8, in <module>
unit-prometheus-openstack-exporter-0: 20:05:30 WARNING unit.prometheus-openstack-exporter/0.install basic.bootstrap_charm_deps()
unit-prometheus-openstack-exporter-0: 20:05:30 WARNING unit.prometheus-openstack-exporter/0.install File "lib/charms/layer/basic.py", line 202, in bootstrap_charm_deps
unit-prometheus-openstack-exporter-0: 20:05:30 WARNING unit.prometheus-openstack-exporter/0.install _update_if_newer(pip, pre_install_pkgs)
unit-prometheus-openstack-exporter-0: 20:05:30 WARNING unit.prometheus-openstack-exporter/0.install File "lib/charms/layer/basic.py", line 367, in _update_if_newer
unit-prometheus-openstack-exporter-0: 20:05:30 WARNING unit.prometheus-openstack-exporter/0.install pkg], env=_get_subprocess_env())
unit-prometheus-openstack-exporter-0: 20:05:30 WARNING unit.prometheus-openstack-exporter/0.install File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
unit-prometheus-openstack-exporter-0: 20:05:30 WARNING unit.prometheus-openstack-exporter/0.install raise CalledProcessError(retcode, cmd)
unit-prometheus-openstack-exporter-0: 20:05:30 WARNING unit.prometheus-openstack-exporter/0.install subprocess.CalledProcessError: Command '['/var/lib/juju/agents

Sudeep Bhandari (sudeephb)
Changed in charm-logrotated:
status: New → Fix Committed
Sudeep Bhandari (sudeephb)
Changed in charm-prometheus-openstack-exporter:
status: New → Fix Committed
assignee: nobody → Sudeep Bhandari (sudeephb)
Eric Chen (eric-chen)
Changed in charm-prometheus-openstack-exporter:
milestone: none → 23.01
Changed in charm-logrotated:
milestone: none → 23.01
Changed in charm-prometheus-openstack-exporter:
importance: Undecided → High
Changed in charm-logrotated:
importance: Undecided → High
Sudeep Bhandari (sudeephb)
Changed in charm-logrotated:
status: Fix Committed → Fix Released
Sudeep Bhandari (sudeephb)
Changed in charm-prometheus-openstack-exporter:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.