backslash in filename causes InvalidEntryName etc

[Jelmer Vernooij]

  affects /products/bzr-svn

At the moment, backslashes in filenames cause weird errors (bzr refuses
to store file ids with a backslash in them).

--
Jelmer Vernooij <email address hidden> - http://samba.org/~jelmer/

It blocks the bzr import of lintian at https://code.launchpad.net/~wgrant/lintian/master

[Jelmer Vernooij (jelmer)]

  affects /products/bzr

Bazaar should support storing file ids with backslashes in them. Perhaps
the file store can simply urlencode them like a lot of other characters?
--
Jelmer Vernooij <email address hidden> - http://samba.org/~jelmer/

[Jelmer Vernooij (jelmer)]

Robert points out that this would have performance implications. I can
live with Bazaar not supporting backslashes in file ids, though it would
be nice if there was some place where the characters that are and are
not allowed were specified. At the moment, what is and what is not
allowed appears to be rather implementation-specific.
--
Jelmer Vernooij <email address hidden> - http://samba.org/~jelmer/

[Jelmer Vernooij (jelmer)]

This is also blocking imports of http://svn.a-eskwadraat.nl/svn/domjudge/trunk

[Robert Collins (lifeless)]

So bzr-svn could encode these itself; Jelmer - what do you want changed on bzr for this?

[Martin Pool (mbp)]

I think it's reasonable to expect the importer to convert the file ids. For file names, we should probably allow \ within a single name component.

[Dragomir Minkovski (dejuren)]

Hi guys, here is one more confirmation of the bug:

bzr add
added \
bzr: ERROR: bzrlib.errors.InvalidEntryName: Invalid entry name: \

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 846, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 797, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 499, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 373, in run
    no_recurse, action=action, save=not dry_run)
  File "/usr/lib/python2.5/site-packages/bzrlib/mutabletree.py", line 52, in tree_write_locked
    return unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/mutabletree.py", line 406, in smart_add
    _add_one(self, inv, parent_ie, directory, kind, action)
  File "/usr/lib/python2.5/site-packages/bzrlib/mutabletree.py", line 574, in _add_one
    file_id=file_id)
  File "/usr/lib/python2.5/site-packages/bzrlib/inventory.py", line 985, in make_entry
    return make_entry(kind, name, parent_id, file_id)
  File "/usr/lib/python2.5/site-packages/bzrlib/inventory.py", line 1312, in make_entry
    return factory(file_id, name, parent_id)
  File "/usr/lib/python2.5/site-packages/bzrlib/inventory.py", line 545, in __init__
    super(InventoryFile, self).__init__(file_id, name, parent_id)
  File "/usr/lib/python2.5/site-packages/bzrlib/inventory.py", line 215, in __init__
    raise errors.InvalidEntryName(name=name)
InvalidEntryName: Invalid entry name: \

bzr 1.5 on python 2.5.1 (linux2)
arguments: ['/usr/bin/bzr', 'add']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_CA.UTF-8'
plugins:
  gtk /usr/lib/python2.5/site-packages/bzrlib/plugins/gtk [0.93.0]
  launchpad /usr/lib/python2.5/site-packages/bzrlib/plugins/launchpad [unknown]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

[Markus Lindenberg (markusl)]

I consider this the #1 showstopper which keeps me from using bazaar.
i'm mostly working with upstream svn repositories (a circumstance that's not going to change soon), some of which have or have had backslashes in filenames. this bug prevents me from using these repositories and makes all usage of bzr as client/frontend for existing svn repositories a matter of chance, preventing any policy decisions favoring bazaar in this use case.

[Daniel Clemente (n142857)]

A solution, as suggested in comment 6, consists in two steps:
- bzr has to allow \ in names: see bug 163858 for that
- the importer should convert \ in file ids (is there a bug for this?)

[Robert Collins (lifeless)]

So, we don't have plans to support this; as such I'm going to mark it wishlist. If someone decides to do the work to make this work, they are welcome to do so - but its non trivial and needs plenty of care.

[snd (dns)]

lp:~rbose-vcs-imports/phpgroupware/trunk is affected by this bug.

"Unable to convert Subversion path trunk/img/images/addressbook\.jpg because it contains characters invalid in Bazaar."

[mestre.adamastor (mestre.adamastor)]

Silly question: how can I define a pattern that simply ignores the offending files?
It seems that:

bzr ignore "\*"

does not do the trick. Any suggestions?

[Martin Pool (mbp)]

Try *\\*

[Jelmer Vernooij (jelmer)]

This doesn't need any changes in bzr-git, just in the bzr core.

[Stefan Monnier (monnier)]

> This doesn't need any changes in bzr-git, just in the bzr core.

That presumes that the core will change. Which doesn't seem likely at
this point.

        Stefan

[Jelmer Vernooij (jelmer)]

> > This doesn't need any changes in bzr-git, just in the bzr core.
> That presumes that the core will change. Which doesn't seem likely at
> this point.
This has to be fixed in bzr; there is no (safe) way to fix this by changing just bzr-git.

Cheers,

Jelmer

[Valerio Bozzolan (bozzy)]

As an additional note, sadly this is the killer-bug for PHP projects involved in core features like namespaces and class autoloading.

[Richard Wilbur (richard-wilbur)]

I see Jelmer has a branch that looks like it fixes this bug by adding
support in bzr core for backslashes in filenames. Could someone
explain to me the drawbacks of supporting backslashes in filenames? I
don't understand why we haven't already merged this branch into trunk
several years ago aside from the fact no one has yet proposed a merge.

[Jelmer Vernooij (jelmer)]

The reason to be careful with support for backslashes is that it's impossible to check out files with backslashes in their name on Windows, and that Bazaar treats backslashes as path separators.

On 14 January 2018 08:45:14 GMT+08:00, Richard Wilbur <email address hidden> wrote:
>I see Jelmer has a branch that looks like it fixes this bug by adding
>support in bzr core for backslashes in filenames. Could someone
>explain to me the drawbacks of supporting backslashes in filenames? I
>don't understand why we haven't already merged this branch into trunk
>several years ago aside from the fact no one has yet proposed a merge.
>
>--
>You received this bug notification because you are subscribed to the
>bug
>report.
>https://bugs.launchpad.net/bugs/81844
>
>Title:
> backslash in filename causes InvalidEntryName etc
>
>To manage notifications about this bug go to:
>https://bugs.launchpad.net/brz/+bug/81844/+subscriptions

--
Jelmer Vernooij <email address hidden>
https://www.jelmer.uk/

[Richard Wilbur (richard-wilbur)]

On Jan 13, 2018, at 18:31, Jelmer Vernooij <email address hidden> wrote:
>
> The reason to be careful with support for backslashes is that it's impossible to check out files with backslashes in their name on Windows, and that Bazaar treats backslashes as path separators.
>
>> On 14 January 2018 08:45:14 GMT+08:00, Richard Wilbur <email address hidden> wrote:
>> I see Jelmer has a branch that looks like it fixes this bug by adding
>> support in bzr core for backslashes in filenames. Could someone
>> explain to me the drawbacks of supporting backslashes in filenames? I
>> don't understand why we haven't already merged this branch into trunk
>> several years ago aside from the fact no one has yet proposed a merge.
>
> --
> Jelmer Vernooij <email address hidden>
> https://www.jelmer.uk/

[Richard Wilbur (richard-wilbur)]

On Jan 13, 2018, at 18:31, Jelmer Vernooij <email address hidden> wrote:
>
> The reason to be careful with support for backslashes is that it's impossible to check out files with backslashes in their name on Windows, and that Bazaar treats backslashes as path separators.

So I'm guessing it's an issue in Windows because Windows considers the backslash as a path separator so it naturally has no business being part of a filename in Windows.

Do you think we could get away with doing the complement of what we do with path separators between the two ecosystems? In *nix filesystems we use the forward slash as a path separator and thus it would be confusing in a filename. By the same token in DOS/Windows filesystems the backslash is a path separator and hence not allowed in filenames.

We adapt the way we encode/decode path names depending on the filesystem:
if filesystem is in set(DOS, Windows):
    separator = '\'
else:
    separator = '/'
If we adapted the filename based on the filesystem similarly:
if filesystem is in set(DOS, Windows):
    separator, escape = ('\', '/')
else:
    separator, escape = ('/', '\')

would that solve some problems? Would it cause other problems? I would vote for encoding the filename in the repository in the *nix fashion (like your branch allows) and translating the name to and from the DOS/Windows filesystems.

[Jelmer Vernooij (jelmer)]

Bazaar already encodes path separators in unix style (i.e. a forward slash), and will transform backslashes on Windows as necessary.

I don't think Windows allows forward slashes in filenames either.

Either way, replacing backslashes in filenames with forward slashes (or another character) is a bad idea IMO. It's confusing, and unlikely to be the behaviour that users actually want. Users that care about portability to Windows can just avoid using backslashes in their filenames.

[Richard Wilbur (richard-wilbur)]

On Sat, Jan 13, 2018 at 10:40 PM, Jelmer Vernooij
<email address hidden> wrote:
> Bazaar already encodes path separators in unix style (i.e. a forward
> slash), and will transform backslashes on Windows as necessary.

Yes, I'm glad that part already works.

> I don't think Windows allows forward slashes in filenames either.

It doesn't seem to like it in a filename. Depending on context it
either says it can't find the corresponding path (treats it as a path
separator) or invalid option.
C:\> echo "Test" >"test/1.txt"
The system cannot find the path specified.

C:\Users\User>dir Documents/Fax
Invalid switch - "Fax".

C:\Users\User>dir "Documents/Fax"
 Volume in drive C is Local Disk
 Volume Serial Number is ####-####

 Directory of C:\Users\User\Documents\Fax
[...]

> Either way, replacing backslashes in filenames with forward slashes (or
> another character) is a bad idea IMO. It's confusing, and unlikely to be
> the behaviour that users actually want. Users that care about
> portability to Windows can just avoid using backslashes in their
> filenames.

So, it seems we should give people a strong warning about the lack of
portability of such filenames to Windows but some people seem to want
to still use them. The warning should appear when the user is adding
(naming) or renaming a file to a name that exhibits this problem.
Should we allow them to use such broken names because their
application calls for it?

I guess it would be nice if there were some way to modify a branch
under Windows so it was usable if it had come from a branch created
with an offending filename.

[Richard Wilbur (richard-wilbur)]

On Sun, Jan 14, 2018 at 12:45 AM, Richard Wilbur
<email address hidden> wrote:
> I guess it would be nice if there were some way to modify a branch
> under Windows so it was usable if it had come from a branch created
> with an offending filename.

Maybe even better to warn them when they branch into a DOS/Windows
filesystem that the tree contains a problematic filename?

[Jelmer Vernooij (jelmer)]

On 14 January 2018 15:48:18 GMT+08:00, Richard Wilbur <email address hidden> wrote:
>On Sun, Jan 14, 2018 at 12:45 AM, Richard Wilbur
><email address hidden> wrote:
>> I guess it would be nice if there were some way to modify a branch
>> under Windows so it was usable if it had come from a branch created
>> with an offending filename.
>
>Maybe even better to warn them when they branch into a DOS/Windows
>filesystem that the tree contains a problematic filename?

At that point, it's an error - not a warning. We can't check out files with backslashes in the names on Windows.

--
Jelmer Vernooij <email address hidden>
https://www.jelmer.uk/

[Martin Pool (mbp)]

I agree it's a bug that files with backslashes in the name can't be stored
and retrieved on Unix, and that they're probably not normalized in the
right way.

The other issue here is that when somebody is committing on Windows, we do
want to make sure any backslashes get converted to canonical path
separators, i.e. forward slashes. However really this should be treated as
a Windows-specific adaption rather than the core knowing anything about
backslashes.

[Richard Wilbur (richard-wilbur)]

On Jan 14, 2018, at 04:46, Jelmer Vernooij <email address hidden> wrote:
> At that point, it's an error - not a warning. We can't check out files with backslashes in the names on Windows.

Ugh! You are right.
I would be in favour of either:
1. offering to bzr mv the offending files to substitute a user-specified legal character for the backslash (as part of the branch operation, so it can succeed!),
-or-
2. nag the user at nearly every opportunity to change the problematic names (with a helpful message showing proper bzr syntax to accomplish the transformation). Id est, at commit, cp, mv, add, status, info, et cetera.

Otherwise, I would be in favour of at least giving a detailed explanatory error message mentioning which file(s) at what path(s) are causing the problem--whether we refuse to add, cp, mv, et cetera or simply refuse to branch onto a filesystem that can't handle the filename(s). We could offer syntax to fix the problem (valid on a *nix platform).

[Saša Janiška (gour)]

Besides consideration of having backslash in the filename on Windows platform, let me say that I was hit by this bug while trying to do git --> bzr conversion with Breezy (see bug https://bugs.launchpad.net/brz/+bug/1743213) on Linux platform.

[Richard Wilbur (richard-wilbur)]

Jelmer, thank you for all the work to improve brz/bzr. I have a few remaining questions:
How does this fail on a DOS/Windows filesystem? We aren't flagging the backslash in filenames anymore with InvalidEntryName. Are we depending on the OS to throw an exception now? For the users' sakes I hope it is at least as explanatory as ours was.

With this change we no longer even complain about the backslash in a filename. What about warning the user about incompatibility with DOS/Windows filesystems?

[Jelmer Vernooij (jelmer)]

On Tue, Jan 16, 2018 at 02:12:53PM -0000, Richard Wilbur wrote:
> Jelmer, thank you for all the work to improve brz/bzr. I have a few remaining questions:
> How does this fail on a DOS/Windows filesystem? We aren't flagging the backslash in filenames anymore with InvalidEntryName. Are we depending on the OS to throw an exception now? For the users' sakes I hope it is at least as explanatory as ours was.
>
> With this change we no longer even complain about the backslash in a
> filename. What about warning the user about incompatibility with
> DOS/Windows filesystems?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/81844
>
> Title:
> backslash in filename causes InvalidEntryName etc
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/brz/+bug/81844/+subscriptions

--
Jelmer Vernooij <email address hidden>
PGP Key: https://www.jelmer.uk/D729A457.asc

[Drew Freiberger (afreiberger)]

I'm having trouble with this bug when running etckeeper with bzr repo when I add snaps to my trusty server. snaps with dashes in them create /etc/systemd/system/snap-charm\x2dname\x2dwith\x2ddashes-0.mount directory for a snap named charm-name-with-dashes version 0. This causes 'cd /etc; bzr add -q .' to fail with Invalid Entry.

[Jelmer Vernooij (jelmer)]

Hi Drew, with Breezy rather than Bazaar installed this should work.

[JP Vossen (jp-jpsdomain)]

The hack-around in https://bugs.launchpad.net/ubuntu/+source/bzr/+bug/1035045/comments/3 "worked" for me with `etckeeper` + BZR on Mint-18, in that it stopped the error messages. I don't actually know or care what omitting the affected snap files in `/etc/` might do later.

Note I needed 2 lines in `/etc/.bzrignore` and was confused when the first line failed to work. And you probably also need to:
```
# Be root
cd /etc
sudo vi .bzrignore
 systemd/system/snap*
 systemd/system/multi-user.target.wants/snap*
bzr rm --keep systemd/system/snap* systemd/system/multi-user.target.wants/snap*
etckeeper commit 'Ugly hack-around for bzr/etckeeper errors for snapd use of backslash in file names'
```