rpm cannot change directory to symlink

[Jeff Johnson]

Reported in e-mail.

[Patrice (patrice-redhat-bugs)]

Description of problem:

when a directory is changed to a symlink rpm errors out.
For example
https://bugzilla.redhat.com/show_bug.cgi?id=433096

error: unpacking of archive failed on file
/usr/lib/xulrunner-1.9pre/dictionaries: cpio: rename

This is a very important and urgent item since it can block any
update.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:

Expected results:

Additional info:

[Jeff (jeff-redhat-bugs)]

Add a %pretrans script to rename the directory and create the symlink.

WORKSFORME (and for all versions of rpm back to when %pretrans was added.)

You will have to worry about dependencies if you wish to install into an empty chroot.
%pretreans -p <lua> rather than using /bin/sh is recommended.

[Patrice (patrice-redhat-bugs)]

That's not a solution. rpm should handle this situation right.

[devzero2000 (devzero2000-redhat-bugs)]

It is a solution that rpm can handle. What's the problem ? Other package management system, deb for example,have the same problem, but IMHO they can't handle it - almost in the past i am not sure today. Probably if it is difficult for most to implemt or know, well it is sufficent to write about it.

jmho

[Patrice (patrice-redhat-bugs)]

It is better to have a workaround than not to have one, but still
this is a bug. If it is hard to correct, it can be postponed, of
course, I haven't proposed a patch so I can wait an infinite time.

The fact that it is also a debian bug is not relevant, however.

[Bug (bug-redhat-bugs)]

This bug appears to have been reported against 'rawhide' during the Fedora 10 development cycle.
Changing version to '10'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

[Bug (bug-redhat-bugs)]

This bug appears to have been reported against 'rawhide' during the Fedora 11 development cycle.
Changing version to '11'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

[Christoph (christoph-redhat-bugs)]

Still a problem in F12. Raising priority and severity to high.

[Christoph (christoph-redhat-bugs)]

Florian, we talked about on IRC this back in December. Is there any progress? This bug prevents us from doing some important updates.

[Bug (bug-redhat-bugs)]

This bug appears to have been reported against 'rawhide' during the Fedora 13 development cycle.
Changing version to '13'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

[Christoph (christoph-redhat-bugs)]

This is getting more and more urgent as it will break updating to F13. Anyone with the Xfce Terminal package installed won't be able to update. Florian, please tell us what to do.

(Changing version back to 'rawhide' so this doesn't get closed without actually being fixed.)

[Bug (bug-redhat-bugs)]

This bug appears to have been reported against 'rawhide' during the Fedora 14 development cycle.
Changing version to '14'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

[Jeff (jeff-redhat-bugs)]

Two years, two months, and two weeks after being reported
... (the RPM issue with symlinks <-> dirs is much much older) ...
... I still suggest using "%pretrans -p <lua>" as a solution.

Honking urgent! urgent! urgent! has not solved the issue for a decade.

[Christoph (christoph-redhat-bugs)]

Jeef, if you can provide me a working scriptlet' to change %{_docdir}/%{name}/html/foo from a directory to a symlink, I'd be more than happy. Several people and me have been looking at it and whatever we tried resulted ether in missing files or changes that showed up in rpm -V.

Florian, can this please get fixed? What more do you want us to do? We have pinged you several times and set the bug to NEEDINFO. What more does it take to get a response?

Changing version back to rawhide and adding the FutureFeature to avoid it gets rebased against release again.

[Jeff Johnson (n3npq)]

An example using %post -p <lua> needs to Asked & Answered for FAQ fodder.

[devzero2000 (pinto-elia)]

Just some old reference on this argument, but always actual and with a clear explanation in my opinion.

http://<email address hidden>/msg00031.html (LUA and YAML)

https://lists.dulug.duke.edu/pipermail/rpm-devel/2007-April/002259.html (upgrade when directory changes to symlink
)

[devzero2000 (pinto-elia)]

As the reference in comment #2 can tell, this bug was solved year ago with the introduction in rpm of the embedded lua engine and the %pretrans scriptlet.

I have commited, in @rpm5.org CVS HEAD, two toy spec example for changing a directory with a symlink, that use, in upgrade, the following lua %pretrans

%pretrans -p <lua>
-- Lua block of code for removing a directory recursively
-- The Lua function remove_directory_deep should be called
-- with a directory name or, in a spec file, also with
-- a rpm macro defined to a directory name. This function
-- is a possible lua equivalent of the shell command "rm -rf"
-- using the lua posix extension embedded in rpm
local leaf_indent = '| '
local tail_leaf_indent = ' '
local leaf_prefix = '|-- '
local tail_leaf_prefix = '`-- '
local link_prefix = ' -> '

local function printf(...)
    io.write(string.format(unpack(arg)))
end

local function remove_directory(directory, level, prefix)
    local num_dirs = 0
    local num_files = 0
    if posix.access(directory,"rw") then
    local files = posix.dir(directory)
    local last_file_index = table.getn(files)
    table.sort(files)
    for i, name in ipairs(files) do
        if name ~= '.' and name ~= '..' then
            local full_name = string.format('%s/%s', directory, name)
            local info = assert(posix.stat(full_name))
            local is_tail = (i==last_file_index)
            local prefix2 = is_tail and tail_leaf_prefix or leaf_prefix
            local link = ''
            if info.type == 'link' then
                linked_name = assert(posix.readlink(full_name))
                link = string.format('%s%s', link_prefix, linked_name)
                posix.unlink(full_name)
            end

             -- printf('%s%s%s%s\n', prefix, prefix2, name, link)

            if info.type == 'directory' then
                local indent = is_tail and tail_leaf_indent or leaf_indent
                sub_dirs, sub_files = remove_directory(full_name, level+1,
                    prefix .. indent)
                num_dirs = num_dirs + sub_dirs + 1
                num_files = num_files + sub_files
                posix.rmdir(full_name)
            else
                posix.unlink(full_name)
                num_files = num_files + 1
            end
        end
    end
    end -- if access
    return num_dirs, num_files
end

local function remove_directory_deep(directory)

    -- print(directory)

    num_dirs, num_files = remove_directory(directory, 0, '')

    -- printf('\ndropped %d directories, %d files\n', num_dirs, num_files)

    posix.rmdir(directory)
end
remove_directory_deep("/tmp/%{name}")

The two toy rpm spec, previously cited, of a ipotetical package test-change-dir-to-symlink show how to solve, using lua in %pretrans with the above fragment of code, the problem of changing a symlink in the directory update in rpm. Tested with @rpm.org 4.4.2.xyz rpm (RHEL 5.5), in rpm 4.7.2 (FC12) in rpm 4.8.0 (RHEL6 Beta2)

The first version is

http://rpm5.org/cvs/fileview?f=rpm/tests/test-change-dir-to-symlink-1.1.spec&v=1.1

and the second version is

http://rpm5.org/cvs/fileview?f=rpm/tests/test-chan...

[devzero2000 (devzero2000-redhat-bugs)]

Probably <email address hidden>.

https://bugs.launchpad.net/rpm/+bug/633636/comments/3

Sure there is the possibility that exists some other minor flaw related to fix also.

[devzero2000 (pinto-elia)]

Fix committed is not right : only some docu more and an example

[Hicham (hicham-redhat-bugs)]

rpm can't change symlink to directory also, see bug 646523

[Jindrich (jindrich-redhat-bugs)]

*** Bug 647068 has been marked as a duplicate of this bug. ***

[Christoph (christoph-redhat-bugs)]

*** Bug 670210 has been marked as a duplicate of this bug. ***

[devzero2000 (pinto-elia)]

I apologize but now i am not sure anymore that rpm5 can hadle this. So i have dropped the borken toy test from CVS HEAD

[Jan (jan-redhat-bugs)]

What is the status of this bug? (If it is really a bug.)

I'm looking into RPM testsuite. There is a test "replace directory with symlink" with comment "this is expected to fail".

(I will fix bug #739318 using %pretrans workaround in the meantime.)

[David (david-redhat-bugs)]

As far as I can see, the needinfo flag was set to ask for a progress report. So clearing the flag to make the bug more visible...

[devzero2000 (pinto-elia)]

Really it is not so important ? I think no. But i have no idea anymore, really

[Jeff Johnson (n3npq)]

There are multiple contexts of interpretation here while ROSA
is participating with ROADMAP planning through January 20th.

From a purely @rpm5.org "forward" POV, the ability to replace
a directory with a symlink is absolutely ESSENTIAL.

There are several known good "workarounds" including %pretrans
with lua. Whether using
     %pretrans -p <lua>
handles all possible "upgrade" cases is a separate issue than whether
a symlink is created before file resolutions are pre-computed during
install is a different issue in need of further testing.

Whether ROSA deems the ability to replace a directory with a symlink
as important on a Mandriva/ROSA 2012 release deliverable is up to others
to decide than me.

MHO, is that this is essential functionality going "forward" @rpm5.org: the rate
of deployment is due to other factors.

[Fedora (fedora-redhat-bugs)]

This package has changed ownership in the Fedora Package Database. Reassigning to the new owner of this component.

[Panu (panu-redhat-bugs)]

Rpm >= 4.11 detects unsupported replace-attempts and reports them as file conflicts, instead of barfing up in middle of transaction. %pretrans hacks can still be used to work around it though.

That's the extent to which this is going to be "fixed" for the foreseeable future. Of course if rpm some day learns to truly deal with some of these situations, conflicts can be lifted accordingly.

[Adam (adam-redhat-bugs)]

This seems to break yum, as %pretrans hacks here are ignored by yum in the transaction test. rpm will successfully upgrade packages, but yum will not.

Can this fix be rolled back? Or another workaround proposed?

[Panu (panu-redhat-bugs)]

If you're seeing a regression, please file a new bug with the reproducer details. Whether such a regression is caused by changes explained in comment #22 or something else cannot be determined without closer study, but if rpm works and yum doesn't then its likely to be "something else" (and it wouldn't be the first time either)

[Adam (adam-redhat-bugs)]

Filed in bug 975909.

[Vít (vt-redhat-bugs)]

I think this issue should be kept open to remind that the matter is not resolved yet. I needed this functionality just recently (bug 988490) and also the bug 975909 seems to be issue for that case.

[Adam Williamson (awilliamson)]

"There are several known good "workarounds" including %pretrans with lua."

Are there really, though?

I seem to keeping bumping up against this problem lately, and I'm not sure I've *ever* seen a single known good snippet for converting a directory to a symlink. There are snippets that appear to achieve the opposite OK, as removing a symlink is relatively trivial, but I haven't seen a single pure lua snippet which looks like it successfully removes a populated directory.

This is a major problem for webapp unbundling, as given how idiotic webapps are, we're often going to have to use symlinks to unbundle stuff. I have at least two and I think three cases where I really need to turn a directory (bundled copy of some library or other) into a symlink (to the systemwide copy of same library) and I just don't know how to do it. This is kind of a major problem.

[Adam (adam-redhat-bugs)]

Do we have an official / known good snippet for converting a *directory* into a *symlink*? I can't seem to find one anywhere. Converting a *symlink* into a *directory* isn't so bad as it's relatively trivial to remove a symlink in pure lua, but recursively removing a populated directory seems to be a much bigger problem, and I'm damned if I can find a snippet that looks trustworthy enough to use. Does anyone have one?

I'm currently sitting on two or three packages where I *really need* to replace a directory with a symlink, and I'm pretty stuck. :(

[Panu (panu-redhat-bugs)]

Recursively removing a directory tree with pure Lua is hardly worth the trouble, shelling out to do 'rm -rf /some/path' with something like this is perfectly fine:

%pretrans -p <lua>
st = posix.stat("<path to dir>")
if st and st.type == "directory" then
  os.execute("rm -rf <path to dir>")
end

The directory you're wanting to remove can only exist on upgrades, in which case shell is going to be there too (or you're looking at one very ill system). The special case of initial install simply falls through there as there's no directory to remove.

[Adam (adam-redhat-bugs)]

I thought that might be the case, but I wasn't sure. Thanks.

Could you just state - here is fine, as I'll take it to FPC - a couple of recommended snippets for both main cases here (dir->symlink and symlink->dir) and I'll try and get them added to the packaging guidelines? I think it'd help people. thanks!

[Ľuboš (ubo-redhat-bugs)]

*** Bug 1124473 has been marked as a duplicate of this bug. ***

[Ľuboš (ubo-redhat-bugs)]

*** Bug 1205005 has been marked as a duplicate of this bug. ***

[Panu (panu-redhat-bugs)]

*** Bug 646523 has been marked as a duplicate of this bug. ***

[Hin-Tak (hin-tak-redhat-bugs)]

Just been bitten by this and spent a good part of a day trying to figure out why. The bug description above wasn't clear. My issue is that I'd like to
support xbuild targeting .net framework 2.0
( https://bugzilla.redhat.com/show_bug.cgi?id=1255204 )
and is happy to just use the binary-blob reference bits from upstream mono. Fedora has policy of not shipping such, so I am modifying the src rpm to revert that and rebuild.

However, the installed mono has symlinks for /usr/lib/mono/4.0-api and /usr/lib/mono/4.5-api both pointing to /usr/lib/mono/4.5 (which is a real directory), while I'd like to upgrade to my custom rpms with those as real directories populated with the upstream binary blobs.

Can we change the bug description a bit, to:

"rpm upgrade,reinstall,downgrade cannot change directory to symlink (or the other way around)"

please?

Anyway, the symptom is that there are piles of file conflicts; and I worked around it by doing "rpm -e --nodeps ..." then "rpm -ivh ..." in two steps, instead of "rpm -Uvh ..." I prefer to do.

[Jan (jan-redhat-bugs)]

Counting myself to the crowds unhappy about this essential lack.

It basically and ilogically push the burden of figuring all the bits of
the deployment layout beforehand to the packages maintainers, but that's
in steep contrast with a fact not everything can be anticipated in the
organic development of particular projects, whereas symlinks are basic
means of retaining the compatibility and/or maintaining non-redundancy.

One big issue I see with scriptlet-based workarounds is that such files
are not covered in metadata tracking for integrity checks (rpm -qV).

Still no chance to get this reconsidered?

In my case, I am going to stick with per-file granular symlinks, even
if mere directory symlinking would be a better fit. But I can imagine
situations where it would get painful, luckily I have just a few files
to symlink.

[Jan (jan-redhat-bugs)]

(First I stuck with symlinking whole directory as direct wildcard
matching in ln command seems impossible if one has to make do without
-r switch when the wildcard expression would be using artifical,
to be dropped, %{buildroot} prefix. Have to go an indirect way.)

[dkaspar (dkaspar-redhat-bugs)]

I also would like to see this getting implemented, sooner than later if possible. There were some valid points stated here why we need this functionality, and using the %pretrans workaround is just simply not viable...

[praiskup (praiskup-redhat-bugs)]

I would say that after the years the workaround can be called "viable".
Since it is not yet mentioned here in the bug, see:
https://fedoraproject.org/wiki/Packaging:Directory_Replacement

It's not easy problem to solve in RPM, but at least we could have shortcut
in some parametric macro as a replacement for the multi-line rpm snippets.

[pmatilai (pmatilai-redhat-bugs)]

(quoting comment #12)
> Honking urgent! urgent! urgent! has not solved the issue for a decade.

...nor two, and actually closing in on three decades. Just to give a bit perspective for that "sooner than later" wish.

[dkaspar (dkaspar-redhat-bugs)]

(In reply to Pavel Raiskup from comment #37)
> I would say that after the years the workaround can be called "viable".
> Since it is not yet mentioned here in the bug, see:
> https://fedoraproject.org/wiki/Packaging:Directory_Replacement
>
> It's not easy problem to solve in RPM, but at least we could have shortcut
> in some parametric macro as a replacement for the multi-line rpm snippets.

Well, but it would still be workaround, meaning some of the problems mentioned above could still cause a problem... For example:

(In reply to Jan Pokorný from comment #34)
> One big issue I see with scriptlet-based workarounds is that such files
> are not covered in metadata tracking for integrity checks (rpm -qV).

(In reply to Panu Matilainen from comment #38)
> (quoting comment #12)
> > Honking urgent! urgent! urgent! has not solved the issue for a decade.
>
> ...nor two, and actually closing in on three decades. Just to give a bit
> perspective for that "sooner than later" wish.

Well, it was more like a New Year's wish... :D You can still postpone it for the next New Year, if you know what I mean... :D