git cannot represent empty directories by default

[Nish Aravamudan]

At least src:software-properties has at one time had empty directories
in the source package, e.g. in 0.96.20.5 in xenial on Ubuntu, and git
(and thus the Ubuntu git importer) fail to
properly represent the srcpkg's contents in the import.

Specifically, tests/aptroot/etc/apt/apt.conf.d/ is missing from the import.

Given that git does not represent empty directories, I'm not sure what
we should do here.

List of known affected packages:

apache2
apparmor
debootstrap
exim4
gettext
mongodb
nagios-nrpe
ssl-cert
supermin
qemu
nmap

[Workaround]

Use the --no-verify to ignore the error.

[Nish Aravamudan (nacc)]

Affects src:apparmor, src:gettext, src:ssl-cert, src:supermin,

[Robie Basak (racb)]

Also affects src:exim4.

[Christian Ehrhardt  (paelzer)]

I haven't looked at the others yet, but for the exim4 case that is a dir called "Local".
BTW this easily finds offenders:
 $ find . -depth -empty

What I find interesting is that they maintain it in git at https://anonscm.debian.org/git/pkg-exim4/exim4.git

There the clue for them is that they only maintain the Debian dir itself which is free of issues. The "Local" dir in this case is from exim4 tarball.

It would be a major change in tree structure if we would do so as well and keep content outside of debian/ as tarball only. But since we already restore tarballs as needed we could use that.

OTOH we would also loose a lot of ability like easily comparing old and new content between package versions. And also the permanent "dirtiness" of the Tree when unpacked would be something I'd not like.

The suggestions of searches lead to files like .gitkeep, and "usd build" could maybe build and restore those as needed.

Looking forward to discuss later today ....

[Christian Ehrhardt  (paelzer)]

well the empty also spots empty files :-/, but one can add "-type d" to fix that.

I wanted to comlete the circle for exim and since it is developed upstream in git as well there is no empty Dir there either.
When upstream creates their tarballs from git the dir is created.

[Christian Ehrhardt  (paelzer)]

also affects nmap

[Christian Ehrhardt  (paelzer)]

also affects apache2 (xenial at least)

WARNING: empty directories exist but are not tracked by git:

docs/manual/style/lang
docs/manual/style/xsl/util

These will silently disappear on commit, causing extraneous
unintended changes. See: LP: #1687057.

[Rafael David Tinoco (rafaeldtinoco)]

Also affects samba ?

(c)inaddy@ctdbdisco:~/work/sources/ubuntu/samba$ git cherry-pick --continue
ERROR: Empty directories exist and will disappear on commit, causing
extraneous unintended changes:

source4/ldap_server/devdocs

See LP: #1687057.
Use "git commit --no-verify ..." to ignore this problem.

pre-commit hook failed.

[Bryce Harrington (bryce)]

There may also be old branches affected as well

$ git checkout remotes/pkg/ubuntu/xenial-security -b xenial-security
Branch 'xenial-security' set up to track remote branch 'ubuntu/xenial-security' from 'pkg' by rebasing.
Switched to a new branch 'xenial-security'

WARNING: empty directories exist but are not tracked by git:

crypto/des/t
include

These will silently disappear on commit, causing extraneous
unintended changes. See: LP: #1687057.

[Bryce Harrington (bryce)]

Christian's right that 99% of suggestions are to add a .gitkeep or similar, but found a couple other kludgy workaround ideas:

1. Add the empty dir as a submodule
    https://stackoverflow.com/a/58543445

    find . -type d -empty -delete -exec git submodule add -f https://gitlab.com/empty-repo/empty.git \{\} \;

2. Add a broken symlink in the empty dir
    https://stackoverflow.com/a/37597601/1004236

    mkdir empty
    ln -s .this.directory empty/.keep
    git add -f empty/.keep
    git commit -m synthetic-dir

Not sure what side effects or ramifications these would have.

For a proper solution, sounds like git upstream might accept a well-coded fix, but that's probably too out of scope for us for now.

[Paride Legovini (paride)]

The official git FAQ [1] suggest using an en .gitignore file as a placeholder for empty directories. This is also what git-svn does, see the --placeholder-filename option in git-svn(1).

Paride

[1] https://git.wiki.kernel.org/index.php/GitFaq#Can_I_add_empty_directories.3F

[Paride Legovini (paride)]

typo fix: an *empty* .gitignore file

[Robie Basak (racb)]

On Mon, Jun 29, 2020 at 12:37:33PM -0000, Paride Legovini wrote:
> The official git FAQ [1] suggest using an en .gitignore file as a
> placeholder for empty directories. This is also what git-svn does, see
> the --placeholder-filename option in git-svn(1).

Unfortunately that's not an option for git-ubuntu since we can't change
the source that we're importing.

Or if we do, then we have to do it in a reversible way, such as how we
do the .git escaping (if .git is present then it gets imported into git
as ..git).

Otherwise we end up in a situation where a build based on git-ubuntu
sources would fail to build, or result in a different build, from a
build from the source package that git-ubuntu imported.

[Paride Legovini (paride)]

As I hit this with apache2 I tried to figure out a solution.

If we use an empty file as placeholder in empty directories then I think the build won't fail, as the diff with the orig source tree would still be empty. There's also a list of file patterns that dpkg-source ignores when checking the diff (see `dpkg-source --help`):

'(?:^|/).*~$|(?:^|/)\.#.*$|(?:^|/)\..*\.sw.$|(?:^|/),,.*(?:$|/.*$)|(?:^|/)(?:DEADJOE|\.arch-inventory|\.(?:bzr|cvs|hg|git|mtn-)ignore)$|(?:^|/)(?:CVS|RCS|\.deps|\{arch\}|\.arch-ids|\.svn|\.hg(?:tags|sigs)?|_darcs|\.git(?:attributes|modules|review)?|\.mailmap|\.shelf|_MTN|\.be|\.bzr(?:\.backup|tags)?)(?:$|/.*$)'

so an empty .git-ubuntu-empty-directory-placeholder~ file should work fine.

This file is ignored by dpkg-source (as it matches the first regex and also because it's empty), but it's *also* auto-removed by dh_clean, which runs before the actual build. (See dh_clean(1), where it's documented that *~ files are removed). This should ensure we get identical build results.