Fedora: Incompatible version of run-parts script

[Kevin Brackney]

The version of the run-parts script installed on a Fedora system comes from the crontabs package. Although this script was originally excerpted from Debian, it is missing the "regex" command line swtich. Therefore, the --regex argument is interpreted as a directory name which does not exists causing the script to fail.

[root@domU-12-31-39-00-85-B2 cloud]# yum whatprovides */run-parts
crontabs-1.11-2.20101115git.fc15.noarch : Root crontab files used to schedule the execution of programs
Repo : fedora
Matched from:
Filename : /usr/bin/run-parts

It appears as though changing line 150 of the "/usr/lib/python2.7/site-packages/cloudinit/util.py" file as follows allows the successful execution of the runcmd and user defined scripts (on Fedora):

From:
cmd = [ 'run-parts', '--regex', '.*', dirp ]

To:
cmd = [ 'run-parts', dirp ]

[Scott Moser (smoser)]

The '--regex .*' was added in revision 99, and kept since then.
http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/99

The primary issue was that it does not run programs with a '.' in them. Ie, if a user added a script with name 'foo.sh' it gets skipped.

$ mkdir x
$ printf '#!/bin/sh\necho hi world\n' > x/go.sh
$ chmod 755 x/go.sh
$ ./x/go.sh
hi world
$ run-parts ./x
$ run-parts --regex='.*' x
hi world

I definitely want to support people putting scripts in the /var/lib/cloud/scripts/per-{once,instance,boot} that are named "myscript.sh".

[Pádraig (pdraig-redhat-bugs)]

Created attachment 564806
proposed workaround

From http://lists.fedoraproject.org/pipermail/cloud/2012-February/001258.html

This error is thrown:

CalledProcessError: Command '['run-parts', '--regex', '.*', '/var/lib/cloud/instance/scripts']

The workaround in the attached seems straightforward enough

[Garrett Holmstrom (gholms)]

Would you be amenable to a for loop that calls every executable in a given directory without invoking run-parts?

[Kevin Brackney (kbrackney)]

That would certainly work for my needs. I would say implement a method that would be the most portable across the different flavors of linux that you plan to support.

Kevin

On Feb 21, 2012, at 10:22 PM, Garrett Holmstrom wrote:

> Would you be amenable to a for loop that calls every executable in a
> given directory without invoking run-parts?
>
> ** Bug watch added: Red Hat Bugzilla #795998
> https://bugzilla.redhat.com/show_bug.cgi?id=795998
>
> ** Also affects: fedora via
> https://bugzilla.redhat.com/show_bug.cgi?id=795998
> Importance: Unknown
> Status: Unknown
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/933553
>
> Title:
> Fedora: Incompatible version of run-parts script
>
> Status in Init scripts for use on cloud images:
> Confirmed
> Status in Fedora:
> Unknown
>
> Bug description:
> The version of the run-parts script installed on a Fedora system comes
> from the crontabs package. Although this script was originally
> excerpted from Debian, it is missing the "regex" command line swtich.
> Therefore, the --regex argument is interpreted as a directory name
> which does not exists causing the script to fail.
>
> [root@domU-12-31-39-00-85-B2 cloud]# yum whatprovides */run-parts
> crontabs-1.11-2.20101115git.fc15.noarch : Root crontab files used to schedule the execution of programs
> Repo : fedora
> Matched from:
> Filename : /usr/bin/run-parts
>
> It appears as though changing line 150 of the "/usr/lib/python2.7
> /site-packages/cloudinit/util.py" file as follows allows the
> successful execution of the runcmd and user defined scripts (on
> Fedora):
>
> From:
> cmd = [ 'run-parts', '--regex', '.*', dirp ]
>
> To:
> cmd = [ 'run-parts', dirp ]
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/cloud-init/+bug/933553/+subscriptions

[Garrett Holmstrom (gholms)]

How does this look?

[Scott Moser (smoser)]

Garrentt,
  Some questions:
a.) what if dir does not exist? it seems like you're going to raise exception on os.listdir ENOENT
b.) I dont know (or quick google did'nt show) if sort() is affected by LC_COLLATE (we don't want it to be). My quick test did not seem to be, but I'd like to make sure that we're executing things in the same order that run-parts is independent of locale.
c.) do you have any reason for not raising the same thing as before on executable failures (ie, why not raise subprocess.CalledProcessError)

[Garrett Holmstrom (gholms)]

> a.) what if dir does not exist? it seems like you're going to raise exception on os.listdir ENOENT

And it should. If the caller passes skip_no_exist=False to the function then he or she is explicitly opting into getting an exception instead of a silent failure. But by default skip_no_exist is True, which will trigger the return statement at the top of the function if the supplied directory does not exist.

> b.) I dont know (or quick google did'nt show) if sort() is affected by LC_COLLATE (we don't want it to be). My quick test did not seem to be, but I'd like to make sure that we're executing things in the same order that run-parts is independent of locale.

I'm not completely sure about this one. Some quick googling seems to point toward sorted() ignoring locale settings unless one passes locale.strcoll to it as a comparator:

  import locale
  locale.setlocale(locale.LC_ALL, "")
  print sorted(stuff, cmp=locale.strcoll)

Shall I look into this question further?

> c.) do you have any reason for not raising the same thing as before on executable failures (ie, why not raise subprocess.CalledProcessError)

If we want to match run-parts's behavior then we can't simply raise a CalledProcessError as soon as a program fails because run-parts always runs everything regardless of success or failure. CalledProcessError.__init__ takes two parameters: returncode and cmd. Since we have now run more than one program, what values do we use?

That said, I chose RuntimeError simply because it was the first thing that came to my mind. If something else is more appropriate then go for it. :-)

That describes my thought process, at least. Does that help?

[Scott Moser (smoser)]

fixed in upstream revno 533.

[Garrett (garrett-redhat-bugs)]

After discussion in the upstream bug and on IRC, this will be fixed upstream by removing the call to run-parts altogether and simply executing every executable in /var/lib/cloud/instance/scripts. The next update should fix this.

[Pádraig (pdraig-redhat-bugs)]

Any ideas on when.
I was think of trying some cloud-init stuff for the fedora openstack test day on March 8th, so it might be sensible to push the fix in the meantime?

[Fedora (fedora-redhat-bugs)]

cloud-init-0.6.3-0.1.bzr532.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/cloud-init-0.6.3-0.1.bzr532.fc17

[Pádraig (pdraig-redhat-bugs)]

Actually I note one of the images we're thinking of testing has this patch manually applied, so don't worry about March 8th at least.

cheers

[Fedora (fedora-redhat-bugs)]

cloud-init-0.6.3-0.1.bzr532.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/cloud-init-0.6.3-0.1.bzr532.fc16

[Fedora (fedora-redhat-bugs)]

cloud-init-0.6.3-0.1.bzr532.fc17 has been pushed to the Fedora 17 stable repository. If problems still persist, please make note of it in this bug report.

[Fedora (fedora-redhat-bugs)]

cloud-init-0.6.3-0.1.bzr532.fc16 has been pushed to the Fedora 16 stable repository. If problems still persist, please make note of it in this bug report.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2240