Empty directories cause rich history to fail to be adopted

[Robie Basak]

If a source package contains empty directories, a developer working on a related branch inadvertently drops them (due to lack support in git) and then git-ubuntu fails to adopt rich history when this branch is presented back to git-ubuntu for adoption.

See bug 1687057 for some history on this.

[Robie Basak (racb)]

I've written/proposed a workaround here: https://git.launchpad.net/~racb/git-ubuntu/tree/wip/emptydirfixup.py?h=emptydirfixup

[Paride Legovini (paride)]

Thanks Robie for adding such a clear description to the workaround script. Just one comment:

You wrote that adding a ".empty" placeholder file wouldn't work, but I think that a file named ".git-ubuntu-empty~" would work, as it's ignored by the dpkg-source diff check for unexpected changes *and* it's deleted by dh_clean before all the other builds targets, so the build process finds all the empty dirs it expects.

This said, I agree that modifying the source tree from the archive is not nice. What I like of your workaround is that it works at the git (index) level, not at the git-ubuntu/importer level.

[Robie Basak (racb)]

On Fri, Mar 05, 2021 at 12:06:43PM -0000, Paride Legovini wrote:
> You wrote that adding a ".empty" placeholder file wouldn't work, but I
> think that a file named ".git-ubuntu-empty~" would work, as it's ignored
> by the dpkg-source diff check for unexpected changes *and* it's deleted
> by dh_clean before all the other builds targets, so the build process
> finds all the empty dirs it expects.

I appreciate that this should make .git-ubuntu-empty~ work in the common
case. From experience, what tends to go wrong in git-ubuntu though is
when a package maintainer does something odd, or historically did
something odd. At that point it comes down to the spec[1], rather than
the behaviour of particular tooling normally used to create to create
source packages. For example, when a package maintainer bypasses tooling
and hacks something up as a workaround for something.

So even if dpkg-source would normally clean it up, there's probably some
maintainer somewhere who once packed up a source manually (or perhaps
using an older version of dpkg-source that behaved differently, etc).
And of course using debhelper isn't mandatory, so the dh_clean
mitigation case is less strong.

I'd therefore much prefer to keep the git-ubuntu imports "clean" if at
all possible, to avoid finding ourselves stuck if a future edge case
appears in practice. IOW, given that I believe I can construct a source
package which would work normally but breaks with .git-ubuntu-empty~, I
am reluctant to rely on such an edge case not existing in the archive
(in history, now or in the future) in practice.

I have violated this principle in one case - where a source package
contains ".git" - since git absolutely will not accept that. However I
arranged to ensure that an unmodified source tree is still possible to
derive from git-ubuntu's imports: I came up with a lossless escaping
mechanism, which "git ubuntu build" will be able to precisely reverse in
the future.

> This said, I agree that modifying the source tree from the archive is
> not nice. What I like of your workaround is that it works at the git
> (index) level, not at the git-ubuntu/importer level.

Thanks! I'm hoping this will get us by for now. One day, I'd like to
implement proper empty directory support in git.

[1] We've also hit issues where spec-violating uploads have been accepted
so have to somehow import those, too.

[Simon Chopin (schopin)]

I just tried this tool for the first time (never encountered the empty dir issue before), and one thing tripped me up: it doesn't create the directory on the filesystem.

I'm guessing it didn't pop up before because I guess most folks use debclean, but I've become accustomed to `git clean -xdf` for various reasons, and so I'm guessing that it made the directory go away.

While it will still technically work as intended wrt git history, dpkg-buildpackage will issue a warning about the missing directory, which will lead your unsuspecting user to wonder if something went wrong somewhere. I know I certainly did :). So could we have the tool also create the missing directories in the worktree?