[bvt] [regression] Execution of '/bin/systemctl enable atop' returned 1: Failed to issue method call: No such file or directory

[Vladimir Khlyunev]

Snapshot 611 introduces new fail:

https://product-ci.infra.mirantis.net/job/9.x.main.ubuntu.smoke_neutron/612/testReport/(root)/deploy_neutron_tun/deploy_neutron_tun/

AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[tools/1], Task[tools/3], Task[tools/2] Stopping the deployment process!'

From puppet log from any node:

2016-12-09 07:11:15 +0000 Puppet (err): Execution of '/bin/systemctl enable atop' returned 1: Failed to issue method call: No such file or directory

Steps to repro - deploy any cluster

Diag snapshot - https://product-ci.infra.mirantis.net/job/9.x.main.ubuntu.smoke_neutron/612/artifact/logs/fail_error_deploy_neutron_tun-fuel-snapshot-2016-12-09_07-11-55.tar

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/409112

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/409112
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=faefa23b759b1fb3cc1ff5cb6c6c216179fcb4c0
Submitter: Jenkins
Branch: stable/mitaka

commit faefa23b759b1fb3cc1ff5cb6c6c216179fcb4c0
Author: Oleksiy Molchanov <email address hidden>
Date: Fri Dec 9 14:07:17 2016 +0200

    Add systemd-shim package to Ubuntu image

    Change-Id: I095b340ce1752f44b416da8dc5ecb27a820e3b01
    Closes-Bug: 1648732

[Serg Melikyan (smelikyan)]

Folks, does this fix resolve the issue with already existing & built ISOs? From what I see fix is in the fuel-web, meaning that fix is part of the ISO, right?

[Serg Melikyan (smelikyan)]

Oleksiy, please confirm that this fix is working for already existing ISOs?

[Serg Melikyan (smelikyan)]

Following workaround for this might be usefull in order to install MOS:

1. Go to Environment -> Settings -> General
2. Add new additional package systemd-shim to the list of additional packages to install
3. Remove content of the folder "var/www/nailgun/targetimages"
4. Reset & Deploy Changes

[Serg Melikyan (smelikyan)]

systemd package was recently added to trusty-updates repository [1] as a part of this bug fix [2], and now puppet detects init system as being systemd and is trying to use systemctl to start services.
Change #409112 adds package which is in conflict with systemd, although it can’t fix problems with existing ISOs since we use ubuntu upstream repositories during installation image builds and deployment.
It's now a broader problem: https://groups.google.com/forum/#!msg/ansible-devel/dCEYP4Ojs2Q/uICWPJp4CgAJ

[1] https://lists.ubuntu.com/archives/trusty-changes/2016-December/023368.html
[2] https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1616422

(c) Michal Skalski

[Oleksiy Molchanov (omolchanov)]

MOS-linux team, this file breaks service management, why does it differ from upstream?

/usr/lib/ruby/vendor_ruby/puppet/provider/service/debian.rb

[Dmitry Teselkin (teselkin-d)]

There should be no difference with 3.8.5 [1]

[1] https://github.com/puppetlabs/puppet/blob/3.8.5/lib/puppet/provider/service/debian.rb

[Dmitry Teselkin (teselkin-d)]

Ok, sorry, the package was taken from Debian and there are two patches for this file.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 9.0
Change author: Ivan Suzdal <email address hidden>
Review: https://review.fuel-infra.org/29303

[Denis Meltsaykin (dmeltsaykin)]

Invalid for 7.0, its puppet isn't affected.

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/29303
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0

Commit: 18db8dec4970fd16ef7a496c027b195a524f37dc
Author: Ivan Suzdal <email address hidden>
Date: Mon Dec 12 13:22:07 2016

Change runs_on_systemd detection algo

Change-Id: I9cffdfeaa508a9d05f8eb7f25650d72ca97dfc9e
Closes-Bug: #1648732

[Johnson (zhengmx)]

Hi
I intalled MOS9.0 one month ago successly， But got the same error above after installing another new MOS9.0 yesterday.
I wondered how could I fix this issue ?

Thanks !

[Timur Nurlygayanov (tnurlygayanov)]

It is a blocker issue for QA team, all automated CI jobs blocked because of the issue.

Dev team, could you please take a look?

Thank you!

[Sergii Turivnyi (sturivnyi)]

It is reproduced on SNAPSHOT_ID='#610'

[Nastya Urlapova (aurlapova)]

@Sergii, Timur, it works since https://product-ci.infra.mirantis.net/job/9.x.snapshot/627/

[TatyanaGladysheva (tgladysheva)]

Verified on 9.2 snapshot #627.

Test is passed since #627 snapshot:
https://product-ci.infra.mirantis.net/job/9.x.main.ubuntu.smoke_neutron/627/testReport/(root)/deploy_neutron_tun/deploy_neutron_tun/

[Serg Melikyan (smelikyan)]

Status: Confirmed,

this patch is not available at http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/dists/mos9.0-updates/ which leads to the failure of installing MOS 9 from ISO using default repos settings.

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: 8.0
Change author: Ivan Suzdal <email address hidden>
Review: https://review.fuel-infra.org/29343

[greg (loughmiller)]

When will patch be available for mos9.0-updates?

[Harish Surabhi (harishrao99)]

I struggled 3 days with this error, I applied above workaround and problem got fixed. Deployment finished without any issues.

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/29343
Submitter: Pkgs Jenkins <email address hidden>
Branch: 8.0

Commit: ddd6a251d7864854718cf1840a147bc229765f49
Author: Ivan Suzdal <email address hidden>
Date: Tue Dec 13 17:15:15 2016

Change runs_on_systemd detection algo

Change-Id: I2ec333291d7b171ddd65d7631c9e24a3abdfff30
Related-Bug: #1648732

[Dmitry Teselkin (teselkin-d)]

Packages with fixes were merged to 9.0 [1] and 8.0 [2] recently.
There is another bug [3] I've created to track process of delivering updated packages to customers out of order, so I'm moving to 'Fix Committed' this one.

[1] https://review.fuel-infra.org/29303
[2] https://review.fuel-infra.org/29343
[3] https://bugs.launchpad.net/fuel/+bug/1649809

[TatyanaGladysheva (tgladysheva)]

Test 'deploy_neutron_tun' is still passed since #627 snapshot:
https://product-ci.infra.mirantis.net/job/9.x.main.ubuntu.smoke_neutron/

root@node-4:~# dpkg -l | grep puppet
ii puppet 3.8.5-2~u14.04+mos2 all configuration management system, agent
ii puppet-common 3.8.5-2~u14.04+mos2 all configuration management system

These packages with fix are located in http://mirror.seed-cz1.fuel-infra.org/mos-repos/ubuntu/snapshots/9.0-latest/pool/main/p/puppet/.

According to comment #23, move bug to Fix Released status.

[Dmitry Teselkin (teselkin-d)]

Ubuntu 16.04 based deployments are not affected.

[Charles Hsu (charles0126)]

Hi all,

How could I get the patch in MOS 8.x?
Just do a update with this docs(https://docs.mirantis.com/openstack/fuel/fuel-8.0/maintenance-updates.html#mu8-0-how-to-update) ?

If so, I did that 5 mins ago and it still failed.

[Vitaly Sedelnik (vsedelnik)]

Charles, the fix was just committed but it's not tested and not released yet. We will publish the technical bulletin containing the fix and instruction next week. Please stay tuned.

[greg (loughmiller)]

FYI - the workaround pointed out to me by Harish where installing the systemd-shim "worked" on Ubuntu 14.04 and MOS 9.0. (Initially, it didn't due to typo..).

[kbaegis (kbaegis)]

Still broken on Fuel 9.1. The workaround doesn't work. Also, the nailgun image build doesn't log the real error:
http://hastebin.com/zeyaloguta.cs
http://hastebin.com/igisoyeyar.sql
"E: Package 'systemd-shim' has no installation candidate\n"

[ganesan (ganesan-jd)]

I could not move forward due to this issue, totally struck.

I tried to update fuel with "yum update", and its still with 9.0

Please provide an alternate way to get rid of this.

I am struck with the workaround specified by Serg Melikyan

Following workaround for this might be usefull in order to install MOS:

1. Go to Environment -> Settings -> General
2. Add new additional package systemd-shim to the list of additional packages to install

What is URI and priority for the systemd-shim package?

3. Remove content of the folder "var/www/nailgun/targetimages"
4. Reset & Deploy Changes

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: mos/mitaka-hotfix-1
Change author: Oleksiy Molchanov <email address hidden>
Review: https://review.fuel-infra.org/29373

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: mos/mitaka-hotfix-2
Change author: Oleksiy Molchanov <email address hidden>
Review: https://review.fuel-infra.org/29380

[Serg Melikyan (smelikyan)]

ganesan,

not a repository, but a package. On the same page where you can add URI and Priority for the repository there is another field where you can add another package.

[Denis Meltsaykin (dmeltsaykin)]

IIUC, you can always remove the "ubuntu-updates" repository from the environment settings and rebuild the IBP target image. Systemd only resides currently in ubuntu-updates, and disabling them should resolve the issue quickly.

[Harish Surabhi (harishrao99)]

Just sharing my experience, Hope it will help someone.

While I am trying to deploy OpenStack by using Fuel 9, provisioning of servers is successfully completed but deployment is failed with below errors.

At the time of deployment, I ssh to one of the controller and in $ tail -f /var/log/puppet.log i found below error message.

" Execution of '/bin/systemctl enable atop' returned 1: Failed to issue method call: No such file or directory "

In MOS9 dashboard, One of the controller puppet log is showing below error.

(/Stage[main]/Osnailyfacter::Atop/Service[atop]/enable) change from false to true failed: Execution of '/bin/systemctl enable atop' returned 1: Failed to issue method call: No such file or directory

With these error messages deployment fails, with in 15 mins.

I followed below steps to resolve this issue.
Step 1:

[root@fuel /]# find -name openstack.yaml

./usr/lib/python2.7/site-packages/nailgun/fixtures/openstack.yaml

./usr/share/fuel-openstack-metadata/openstack.yaml

[root@fuel /]# vim /usr/lib/python2.7/site-packages/nailgun/fixtures/openstack.yaml

For me in line number 2136 below "ruby-stomp" add "systemd-shim" package and save.

Do the same in /usr/share/fuel-openstack-metadata/openstack.yaml file also.

Step 2:

remove everything in /var/www/nailgun/targetimages location.
Step 3:

In MOS9 dashboard In "Settings→ General" Page, At he bottom of the page you will see a heading "Provision" below that you will see "Initial packages" subheading.

Here you will see a small box with bunch of packages added, here you can add your own packages ( First time I realized), so add "systemd-shim" package below "ruby-stomp"

so once you add the package, Start Provisioning and deployment, if environment is already Provisioned reset the environment.

so after fixing this issue deployment went very smoothly without single error.

[kbaegis (kbaegis)]

I think this is a duplicate of the "systemd" bug. The solution appears to have been to remove systemd on the controllers: sudo apt-get remove systemd

This worked for our solution.

[ganesan (ganesan-jd)]

Serg Melikyan- Thanks for the comment.

I followed the comment specified by "Harish Surabhi" and it worked for me

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/29380
Submitter: Roman Vyalov <email address hidden>
Branch: mos/mitaka-hotfix-2

Commit: 8258dab6db5cd14f310c4292314aeb74a64ff6f1
Author: Oleksiy Molchanov <email address hidden>
Date: Thu Dec 15 17:09:15 2016

Add systemd-shim package to Ubuntu image

Change-Id: I095b340ce1752f44b416da8dc5ecb27a820e3b01
Closes-Bug: 1648732

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Vitaly Sedelnik <email address hidden> on branch: mos/mitaka-hotfix-1
Review: https://review.fuel-infra.org/29373

[Navneet Grewal (yellowcirclenet)]

Quick note to everyone trying to use #36 directions.

It will only work if you are using default repos and have not created a local mirror using fuel-mirror / fuel-createmirror

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 9.0-hotfix-1
Change author: Vitaly Sedelnik <email address hidden>
Review: https://review.fuel-infra.org/29463

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/29463
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0-hotfix-1

Commit: f44e24142ca1492bf2a6d5b1a6dbeb87575cd432
Author: Vitaly Sedelnik <email address hidden>
Date: Tue Dec 20 10:39:03 2016

Remove systemctl call from postinst

Closes-Bug: #1648732

Change-Id: Iea15d6b987c71d860574645eab7849803b97c228

[Tim Powell (tdp252)]

How do we implement this workaround to use systemd-shim if we are in a corporate intranet and do not have the ability to use anything except a local mirror?

[Sergii Rizvan (srizvan)]

Also StackLight - InfluxDB/Grafana Fuel Plugin needs to be fixed. Post-install script in the influxdb package erroneously tries to use systemd to enable influxdb service. More info about this issue is in a separate bug: https://bugs.launchpad.net/lma-toolchain/+bug/1652640

[Purnima Surampudi (psurampudi)]

I reran the deployment skipping explicit bootstrap image creation and local repo creation and it solved the issue for me.

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: 9.0
Change author: Sergii Rizvan <email address hidden>
Review: https://review.fuel-infra.org/29615

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: 9.0
Change author: Sergii Rizvan <email address hidden>
Review: https://review.fuel-infra.org/29616

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: 9.0
Change author: Sergii Rizvan <email address hidden>
Review: https://review.fuel-infra.org/29617

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Sergii Rizvan <email address hidden> on branch: 9.0
Review: https://review.fuel-infra.org/29617

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/29616
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0

Commit: 65c1290edb77868086e8b521ced56d6ce805154e
Author: Sergii Rizvan <email address hidden>
Date: Thu Dec 29 14:58:12 2016

Remove systemd from list of debian helpers

Change-Id: Id77d2f8054f98649fd1967b1d4bf145d74bbd6c8
Related-Bug: #1648732
Closes-Bug: #1652795

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/29615
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0

Commit: 8c50a750ba292c5a6f855730874b7d75a6b1a796
Author: Sergii Rizvan <email address hidden>
Date: Thu Dec 29 15:20:08 2016

Remove systemd from list of debian helpers

Change-Id: I384175d0c46d9ea216d189d6b23e7a3db02d1529
Related-Bug: #1648732
Closes-Bug: #1652901

[Charles Hsu (charles0126)]

Hi All,

Does it fixed in MOS 8.0 and 9.0 both? should I do some extra patch or workaround for it?

[Chandra (reddydodda)]

Hi Charles,

Still it is not fixed in MOS 8.0. I am stuck with this bug . could you please help me with work around. I see the bug for MOS 8.0 is fixed but not released.

Thanks,

[Charles Hsu (charles0126)]

Hi Chandu,

I think I ran into the same issue half month ago, and still wait for the release for MOS 8.0 update. That's why I asked the question here, hope someone has an idea how to do a workaround for this issue.

[Chandra (reddydodda)]

Hi Charles,

I found work around for this

1. Edit /usr/share/fuel-mirror/ubuntu.yaml and add systemd-shim in packages section.

2. again rerun fuel-mirror apply -P ubuntu -G mos ubuntu ( i am using local repo )

3. now if u check for system-shim in your fuel node you will be able to find it ( find -name system-shim )

4. Install system-shim package on all target nodes ( apt-get install system-shim ).

5. Now redeploy. it will work

If someone has better approach please share. ( i am behind proxy so have to use local repo )

[Charles Hsu (charles0126)]

Hi Developers,

For MOS 8.0, did you consider to provide a release for this issue?

[Denis Meltsaykin (dmeltsaykin)]

Charles, the fix for MOS8.0 has been released on 21-Dec-2016 and is available in the mos8.0-holdback repository. If you are still facing the issue please make sure you've updated the local mirrors if any. Otherwise, please provide more details.

[Charles Hsu (charles0126)]

Denis,

Thanks, that's great!
Do you know where I can find the info for mos8.0-holdback repo? I follow the instructions (https://docs.mirantis.com/openstack/fuel/fuel-8.0/maintenance-updates.html#mu8-0-how-to-update) to update my fuel master but I don't see the repo is mentioned in the docs.
Should I add that manually to local repo?

[Denis Meltsaykin (dmeltsaykin)]

Charles, the mos8.0-holdback repository is for slave nodes (Ubuntu-based), so you can find it in the environment settings (please refer to https://docs.mirantis.com/openstack/fuel/fuel-8.0/operations.html#configuring-repositories). This repo is normally present in the setting if you didn't remove it. If you used the fuel-createmirror tool to download all repositories to use them locally you will have to re-run fuel-createmirror to refresh your local copy of repos.

Also, if you already have a running environment and want to add nodes you need to delete the provisioning image of this environment from the /var/www/nailgun/targetimages directory. It will be regenerated with fixed puppet once you start adding nodes.

[Charles Hsu (charles0126)]

Denis,

Thanks for the help, yes, I saw the repo in my openstack env and I just sync repo to my local two days ago. And it seems like I still run into the problem.

[root@fuel /]# fuel-createmirror -M
This script is DEPRECATED. Please use fuel-mirror utility!
Starting new HTTP connection (1): 10.20.0.2
Starting new HTTP connection (1): 10.20.0.2
Starting new HTTP connection (1): 10.20.0.2
Starting new HTTP connection (1): 10.20.0.2
loading packages from mos8.0-updates.restricted ...
loaded: 0 packages from mos8.0-updates.restricted.
loading packages from mos8.0.restricted ...
loaded: 0 packages from mos8.0.restricted.
loading packages from mos8.0.main ...
loaded: 1077 packages from mos8.0.main.
loading packages from mos8.0-security.main ...
loaded: 0 packages from mos8.0-security.main.
loading packages from mos8.0.main ...
loaded: 1077 packages from mos8.0.main.
loading packages from mos8.0-updates.main ...
loaded: 236 packages from mos8.0-updates.main.
loading packages from mos8.0-security.restricted ...
loaded: 0 packages from mos8.0-security.restricted.
loading packages from mos8.0.restricted ...
loaded: 0 packages from mos8.0.restricted.
clone repository mos8.0-updates.restricted to /var/www/nailgun/mirrors/mos-repos/ubuntu/8.0/
clone repository mos8.0.restricted to /var/www/nailgun/mirrors/mos-repos/ubuntu/8.0/
clone repository mos8.0.main to /var/www/nailgun/mirrors/mos-repos/ubuntu/8.0/
clone repository mos8.0-security.main to /var/www/nailgun/mirrors/mos-repos/ubuntu/8.0/
clone repository mos8.0.main to /var/www/nailgun/mirrors/mos-repos/ubuntu/8.0/
clone repository mos8.0-updates.main to /var/www/nailgun/mirrors/mos-repos/ubuntu/8.0/
clone repository mos8.0-security.restricted to /var/www/nailgun/mirrors/mos-repos/ubuntu/8.0/
clone repository mos8.0.restricted to /var/www/nailgun/mirrors/mos-repos/ubuntu/8.0/
copy packages from - mos8.0-updates.restricted
loading packages from mos8.0-updates.restricted ...
loaded: 0 packages from mos8.0-updates.restricted.
saved 0 packages in mos8.0-updates.restricted
added repository suite release file: /var/www/nailgun/mirrors/mos-repos/ubuntu/8.0/dists/mos8.0-updates/Release
copy packages from - mos8.0-security.restricted
loading packages from mos8.0-security.restricted ...
loaded: 0 packages from mos8.0-security.restricted.
saved 0 packages in mos8.0-security.restricted
added repository suite release file: /var/www/nailgun/mirrors/mos-repos/ubuntu/8.0/dists/mos8.0-security/Release
copy packages from - mos8.0.main
loading packages from mos8.0.main ...
loaded: 1077 packages from mos8.0.main.
saved 1077 packages in mos8.0.main
added repository suite release file: /var/www/nailgun/mirrors/mos-repos/ubuntu/8.0/dists/mos8.0/Release
copy packages from - mos8.0-security.main
loading packages from mos8.0-security.main ...
loaded: 0 packages from mos8.0-security.main.
saved 0 packages in mos8.0-security.main
added repository suite release file: /var/www/nailgun/mirrors/mos-repos/ubuntu/8.0/dists/mos8.0-security/Release
copy packages from - mos8.0-updates.main
loading packages from mos8.0-updates.main ...
loaded: 236 packages from mos8.0-updates.main.
saved 236 packages in mos8.0-updates.main
added reposi...

[Charles Hsu (charles0126)]

And I need to do a workaround (https://bugs.launchpad.net/fuel/+bug/1648732/comments/57) for this issue.

So I'm not sure which I can apply the commit, should I manually change repos? And which repo that I can use for this? And I don't see the 8.1 repo from http://mirror.fuel-infra.org/mos-repos/ubuntu/

8.0/ 21-Dec-2016 12:54 -
8.0-mu-1/ 12-May-2016 14:13 -
8.0-mu-2/ 20-Jul-2016 10:26 -
8.0-mu-3/ 21-Sep-2016 09:19 -

[Charles Hsu (charles0126)]

Sorry for my last message, it seems like I don't need to find the 8.1 repo because you mentioned that is updated in MOS8.0 repos. How can I check the patch already applied?

[Denis Meltsaykin (dmeltsaykin)]

Charles, I see 2 problems in your output:

1) the definition of the mos-holdback repo in your settings is wrong, please find the correct definition in https://content.mirantis.com/rs/451-RBY-185/images/Mirantis-Technical-Bulletin-33-systemd.pdf

2) from the provided log it seems that your local copy doesn't contain mos-holdback, this might be caused by the bug which is not fixed yet in 8.0 (https://review.openstack.org/#/c/290525/). If this is the case you have to patch the fuel-mirror manually with the mentioned patch.

If you fix these, you don't need systemd-shim to be installed, it should work as is w/o any workarounds.

[Charles Hsu (charles0126)]

Denis,

Thanks for the detail, I can move forward now. I think the problem is my repo (mos-holdback) is wrong and thanks for point that ((https://review.openstack.org/#/c/290525/).
After modify the code and do a repo sync to local, everything looks good.