Importing revisions with submodules is not supported

Reported by Martin Pool on 2009-07-22
322
This bug affects 55 people
Affects Status Importance Assigned to Milestone
Bazaar
High
Unassigned
Bazaar Git Plugin
Medium
Jelmer Vernooij
Launchpad itself
High
Unassigned
cloudfoundry
High
Unassigned

Bug Description

bzr-git converts submodules into nested trees, but nested trees aren't very well supported by bzr yet.

This affects importing of commits from Git repositories that contains submodules anywhere in their history. It doesn't matter if the submodules have been removed in the last revision.

Jelmer Vernooij (jelmer) wrote :

This is blocked by by-reference nested tree support in bzrlib.

Changed in bzr-git:
status: New → Triaged
importance: Undecided → Wishlist
importance: Wishlist → Medium

I think I'm affected by the same problem. When trying to branch pitivi:

jeff@testbed:~$ bzr branch git://git.pitivi.org/git/pitivi.git
Escaped 1 XML-invalid characters in ce745af077aeda4f8d22cae84d5b1f0baf8eb1b7. Will be unable to regenerate the SHA map.
bzr: ERROR: exceptions.NotImplementedError: <function import_git_submodule at 0xa803f7c>

Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 835, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 1030, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 647, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.6/dist-packages/bzrlib/builtins.py", line 1245, in run
    source_branch=br_from)
  File "/usr/lib/python2.6/dist-packages/bzrlib/bzrdir.py", line 1177, in sprout
    result_repo.fetch(source_repository, fetch_spec=fetch_spec)
  File "/usr/lib/python2.6/dist-packages/bzrlib/repository.py", line 1544, in fetch
    find_ghosts=find_ghosts, fetch_spec=fetch_spec)
  File "/usr/lib/python2.6/dist-packages/bzrlib/plugins/git/fetch.py", line 378, in fetch
    mapping=mapping, fetch_spec=fetch_spec)
  File "/usr/lib/python2.6/dist-packages/bzrlib/plugins/git/fetch.py", line 403, in fetch_refs
    self.fetch_objects(determine_wants, mapping, pb)
  File "/usr/lib/python2.6/dist-packages/bzrlib/plugins/git/fetch.py", line 437, in fetch_objects
    store, recorded_wants, pb)
  File "/usr/lib/python2.6/dist-packages/bzrlib/plugins/git/fetch.py", line 335, in import_git_objects
    parent_invs, target_git_object_retriever._idmap, lookup_object)
  File "/usr/lib/python2.6/dist-packages/bzrlib/plugins/git/fetch.py", line 246, in import_git_tree
    file_id, revision_id, parent_invs, shagitmap, lookup_object)
  File "/usr/lib/python2.6/dist-packages/bzrlib/plugins/git/fetch.py", line 174, in import_git_submodule
    raise NotImplementedError(import_git_submodule)
NotImplementedError: <function import_git_submodule at 0xa803f7c>

bzr 1.17 on python 2.6.2 (linux2)
arguments: ['/usr/bin/bzr', 'branch', 'git://git.pitivi.org/git/pitivi.git', 'pitivi-bazaar']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'fr_CA.UTF-8'
plugins:
  bzrtools /usr/lib/python2.6/dist-packages/bzrlib/plugins/bzrtools [1.17]
  dbus /usr/lib/python2.6/dist-packages/bzrlib/plugins/dbus [0.1dev]
  git /usr/lib/python2.6/dist-packages/bzrlib/plugins/git [0.4]
  gtk /usr/lib/python2.6/dist-packages/bzrlib/plugins/gtk [0.96.2.final]
  launchpad /usr/lib/python2.6/dist-packages/bzrlib/plugins/launchpad [1.17]
  netrc_credential_store /usr/lib/python2.6/dist-packages/bzrlib/plugins/netrc_credential_store [1.17]
  svn /usr/lib/python2.6/dist-packages/bzrlib/plugins/svn [0.6.3]

This is probably what is affecting https://code.launchpad.net/~vcs-imports/pitivi/trunk-git too, I presume.

Jelmer Vernooij (jelmer) wrote :

A proper error message is now printed.

Launchpad could automatically recognize and perhaps better format this error by explicitly catching objects of the class bzrlib.plugins.git.fetch.SubmodulesNotSupported

Jelmer Vernooij (jelmer) wrote :

This is blocked by nested tree support in Bazaar. I've marked it as affecting Bazaar, but I'm not sure if that's the right thing to do. I couldn't find any existing bug reports in Bazaar about missing nested tree support.

Changed in bzr:
status: New → Triaged
status: Triaged → Confirmed
importance: Undecided → Wishlist
Jelmer Vernooij (jelmer) on 2009-11-19
summary: - NotImplementedError importing a branch with submodules
+ Importing revisions with submodules does not work
summary: - Importing revisions with submodules does not work
+ Importing revisions with submodules is not supported
description: updated
Jelmer Vernooij (jelmer) on 2009-11-20
Changed in bzr-git:
status: Triaged → Fix Released
Jelmer Vernooij (jelmer) wrote :

Of course, this doesn't mean you can use submodules now - bzr-git can import them and converts them to nested trees, but nested tree support in bzr itself is still incomplete.

Benjamin Drung (bdrung) wrote :

The git import of xmms2 [1-2] failed due to missing submodules support:

bzrlib.plugins.git.fetch.SubmodulesNotSupported: Submodules can not yet be imported (requires nested tree support in Bazaar).

[1] https://code.launchpad.net/~vcs-imports/xmms2/stable
[2] https://code.launchpad.net/~vcs-imports/xmms2/trunk

Jelmer Vernooij (jelmer) wrote :

A version of bzr-git that supports submodules should land on launchpad within the next two weeks.

L.Lopez (ldotlopez) wrote :

Launchpad import still fails:

2010-01-09 12:30:49 INFO 51 bytes transferred |
2010-01-09 12:30:50 INFO Counting objects: 5920, done. 0
2010-01-09 12:30:52 INFO finding revisions to fetch:generating index 0/5920
2010-01-09 12:30:53 INFO finding revisions to fetch 1/1808
2010-01-09 12:31:18 INFO
Traceback (most recent call last):
  File "/srv/importd.launchpad.net/production/launchpad-rev-8806/scripts/code-import-worker.py", line 63, in <module>
    script.main()
  File "/srv/importd.launchpad.net/production/launchpad-rev-8806/scripts/code-import-worker.py", line 58, in main
    import_worker.run()
  File "/srv/importd.launchpad.net/production/launchpad-rev-8806/lib/lp/codehosting/codeimport/worker.py", line 409, in run
    self._doImport()
  File "/srv/importd.launchpad.net/production/launchpad-rev-8806/lib/lp/codehosting/codeimport/worker.py", line 526, in _doImport
    bazaar_tree.branch.pull(foreign_branch, overwrite=True)
  File "<string>", line 4, in pull_write_locked
  File "/home/pqm/for_rollouts/production/eggs/bzr-2.1.0b3-py2.5-linux-x86_64.egg/bzrlib/branch.py", line 949, in pull
  File "/srv/importd.launchpad.net/production/launchpad-rev-8806/optionalbzrplugins/git/branch.py", line 416, in pull
    graph=graph)
  File "/srv/importd.launchpad.net/production/launchpad-rev-8806/optionalbzrplugins/git/branch.py", line 379, in update_revisions
    interrepo.fetch_objects(determine_wants, self.source.mapping)
  File "/srv/importd.launchpad.net/production/launchpad-rev-8806/optionalbzrplugins/git/fetch.py", line 493, in fetch_objects
    store, recorded_wants, pb)
  File "/srv/importd.launchpad.net/production/launchpad-rev-8806/optionalbzrplugins/git/fetch.py", line 363, in import_git_objects
    allow_submodules=getattr(repo._format, "supports_tree_reference", False))
  File "/srv/importd.launchpad.net/production/launchpad-rev-8806/optionalbzrplugins/git/fetch.py", line 261, in import_git_tree
    raise SubmodulesRequireSubtrees()
bzrlib.plugins.git.fetch.SubmodulesRequireSubtrees: The repository you are fetching from contains submodules. To continue, upgrade your Bazaar repository to a format that supports nested trees, such as 'development-subtree'.
Import failed:
Traceback (most recent call last):
Failure: twisted.internet.error.ProcessTerminated: A process has ended with a probable error condition: process ended with exit code 1.

Tim Su (tim-todoroo) wrote :

Launchpad fails for me too, same error. I no longer have submodules in the repository, but there was a time in the past when there were submodules. The branch that's failing is "lp:astrid"

Aaron Bentley (abentley) on 2010-06-08
tags: added: udd
Tim Penhey (thumper) on 2010-08-19
Changed in launchpad-code:
status: New → Triaged
importance: Undecided → Wishlist

This is fairly important to me - more important than 'Wishlist'. We
can't import GDB or QEMU into Launchpad without this support.

Martin Pool (mbp) on 2010-08-19
Changed in bzr:
importance: Wishlist → High
Benjamin Drung (bdrung) on 2010-09-03
description: updated
Sebastian Dröge (slomo) wrote :

Also affects pitivi and gstreamer.

Jorge O. Castro (jorge) wrote :

Any update on this? This is really holding us back enabling daily builds.

Jorge O. Castro (jorge) wrote :

Sorry, I should have been more clear, this is holding back our ability to evangelize daily builds to existing upstreams who would be interested in using dailies and PPAs.

Benjamin Drung (bdrung) on 2010-09-23
description: updated
Jelmer Vernooij (jelmer) wrote :

For a more complete list of failing imports because of this issue, see https://dev.launchpad.net/FailingBzrSvnImports

Benjamin Drung (bdrung) on 2010-09-23
description: updated
Sebastian Dröge (slomo) wrote :

And also affects banshee

Eli Zaretskii (eliz) wrote :

Are there any concrete plans to add this to bzr stable in some specific future version? If not, how about making this higher priority, as it seems to render a valuable bzr feature only half workable.

Riccardo 'c10ud' (c10ud) wrote :

affects emesene

Vsevolod Velichko (torkvemada) wrote :

Affects qutim

Stefan H. (stefan-h) wrote :

Has been affecting Mumble for a long time now. Even being able to import without the submodules would be better than being completely unable to do so.

Any ETA on releasing a fix for this bug?

Paulo Dias (paulo-miguel-dias) wrote :

Affects tomahawk too

jaromil (jaromil) wrote :

affects hasciicam and more software

Changed in launchpad:
importance: Wishlist → Low

I updated the title because it was misleading.

summary: - Importing revisions with submodules is not supported
+ Bazaar needs support for nested trees
Martin Pool (mbp) wrote :

@Geoff this bug is for the specific git issue.

summary: - Bazaar needs support for nested trees
+ Importing revisions with submodules is not supported
Vsevolod Velichko (torkvemada) wrote :

Added affected software to description :)

description: updated
xaav (xaav) wrote :

Bug #769629 is not a duplicate then. Can someone please un-mark it as a duplicate.

karen pulsifer (froggy1234) wrote :

how do you fix this problem E:: command not found
root@kpulaifer-desktop:~# Go to the repository dialog to correct the problem.
Go: command not found
root@kpulaifer-desktop:~# E: _cache->open() failed, please repo
bash: syntax error near unexpected token `('
root@kpulaifer-desktop:~# /etc/apt/sources.list.d/ferramroberto
bash: /etc/apt/sources.list.d/ferramroberto: No such file or directory
root@kpulaifer-desktop:~# sudo remove /etc/apt/sources.list.d/ferramroberto
sudo: remove: command not found
root@kpulaifer-desktop:~# /etc/apt/sources.list.d/ferramroberto
bash: /etc/apt/sources.list.d/ferramroberto: No such file or directory
root@kpulaifer-desktop:~# sudo/etc/apt/sources.list.d/ferramroberto
bash: sudo/etc/apt/sources.list.d/ferramroberto: No such file or directory
root@kpulaifer-desktop:~# /etc/apt/sources.list.d/ferramroberto-
bash: /etc/apt/sources.list.d/ferramroberto-: No such file or directory
root@kpulaifer-desktop:~# sudo 'b-src
> sudo deb-src
> sudo E:Type 'b-src' is not known on line 1 in source list /etc/apt/sources.list.d/ferramroberto-java-natty.list

Hi Karen,

That's a different problem; please go to
<http://askubuntu.com/questions/ask> and describe what it is you want
to accomplish, what you're doing, and what goes wrong, and someone
will help you soon.

Martin

Dustin Kirkland  (kirkland) wrote :

This bug is impacting our work with CloudFoundry. Any likelihood of it being worked-on or solved in the near future?

Thanks,
Dustin

Changed in cloudfoundry:
importance: Undecided → High
status: New → Confirmed
DNS (dns) wrote :
Benjamin Drung (bdrung) on 2011-10-10
description: updated
Jelmer Vernooij (jelmer) on 2011-10-10
description: updated
chrysn (chrysn) wrote :

as a workaround, i used `git filter-branch` to remove the submodules from the project -- of course, this only works if the submodules are purely optional or otherwise not relevant to the purposes intended (in my case, i need the branch to build daily packages; upstreams might keep library dependencies in a subtree to build and ship them, while in debian/ubuntu, this is patched out anyway to use package dependencies).

the very line i used for filtering was

  git filter-branch -f --index-filter 'git rm --cached --ignore-unmatch .gitmodules libraries/MCAD'

(libraries/MCAD is the only submodule i had). for reference, this affects the lp:openscad project.

Benjamin Drung (bdrung) wrote :

chrysn, this workaround doesn't help, because a machine running this command is needed.

DNS (dns) wrote :

These projects/branches are also affected by this bug:

lp:anarchnet
lp:anubis
lp:autoconf-archive
lp:bison
lp:cflow
lp:coreutils
lp:cpio (lp:~rbose-vcs-imports/cpio/master)
lp:findutils
lp:gnu-radius
lp:grep (lp:~rbose-vcs-imports/grep/master)
lp:mumble
lp:patch
lp:sed

The GDB project you can remove from the list, the branch is fixed, as you can see here: lp:gdb

Benjamin Drung (bdrung) on 2011-10-13
description: updated
Benjamin Drung (bdrung) on 2011-10-13
description: updated
DNS (dns) wrote :

Also affected by this bug:

lp:gzip
lp:parted

DNS (dns) wrote :

sorry, lp:parted does not exist, i mean: lp:~rbose-vcs-imports/parted/master

Martin Pool (mbp) wrote :

Hi DNS,

We appreciate this bug is important and is blocking an increasing
number of projects. We have enough test data, so adding a list of
every affected branch is not especially going to help.

DNS (dns) wrote :

Ok :-)

Vistaus (djmusic121) wrote :

Then can we expect some work to be done, Martin? It's more than a year now and many are affected. Something has to be done.

Prevents Qt5 base module from being imported from gitorious.. isn't that reason enough to fix this pronto.

Jelmer Vernooij (jelmer) on 2012-01-24
Changed in bzr:
status: Confirmed → In Progress
assignee: nobody → Jelmer Vernooij (jelmer)
milestone: none → 2.6b1
Vincent Ladeuil (vila) on 2012-03-15
Changed in bzr:
milestone: 2.6b1 → none
Eli Zaretskii (eliz) wrote :

What does the change of the milestone to "none" mean? Does it mean that the bug will never be fixed? That'd be a pity.

Vincent Ladeuil (vila) wrote :

@Eli: No, it means whatever was in progress was not finished when 2.6b1 was frozen, nothing more.

Jelmer Vernooij (jelmer) on 2012-04-08
Changed in bzr:
status: In Progress → Triaged
assignee: Jelmer Vernooij (jelmer) → nobody
Riccardo 'c10ud' (c10ud) wrote :

2012-05-19 11:17:49 INFO Unable to import branch because of limitations in Bazaar.
2012-05-19 11:17:49 INFO The repository you are fetching from contains submodules. To continue, upgrade your Bazaar repository to a format that supports nested trees, such as 'development-subtree'.

We're no longer using git submodules, is there a way to tell bzr/vcs-imports to just ignore history and focus on current code?
We'd like to setup a daily ppa for our software without using anything else than github and launchpad.

Thanks

Jelmer Vernooij (jelmer) wrote :

Hi c1oud,

There is no way to tell the import system to just ignore history. You could provide a custom repository that has the submodules filtered out of the history or filter them out of your history on github.

Jelmer Vernooij (jelmer) wrote :

I'm not sure why this is marked as affecting cloudfoundry? I don't see what changes would be needed in cloudfoundry for this to be fixed.

Juan L. Negron (negronjl) wrote :

CloudFoundry uses github. Getting sub modules supported would allow the
package to be built directly from their repository.

Thanks,

Juan

Sent from my mobile.
Apologies for the brevity of this message.
On Jun 28, 2012 6:05 AM, "Jelmer Vernooij" <email address hidden>
wrote:

> I'm not sure why this is marked as affecting cloudfoundry? I don't see
> what changes would be needed in cloudfoundry for this to be fixed.
>
> --
> You received this bug notification because you are a member of
> Canonical-SIG, which is subscribed to the bug report.
> https://bugs.launchpad.net/bugs/402814
>
> Title:
> Importing revisions with submodules is not supported
>
> Status in Bazaar Version Control System:
> Triaged
> Status in bzr git support plugin:
> Fix Released
> Status in CloudFoundry:
> Confirmed
> Status in Launchpad itself:
> Triaged
>
> Bug description:
> bzr-git converts submodules into nested trees, but nested trees aren't
> very well supported by bzr yet.
>
> Affected projects:
>
> lp:anarchnet
> lp:anubis
> astrid
> lp:autoconf-archive
> banshee
> lp:bison
> lp:cflow
> lp:coreutils
> lp:cpio (lp:~rbose-vcs-imports/cpio/master)
> lp:dico
> lp:diffutils
> emesene
> lp:findutils
> f-spot
> lp:gnu-cssc
> lp:gnu-radius
> lp:grep (lp:~rbose-vcs-imports/grep/master)
> gstreamer
> haasciicam
> lp:mailutils
> lp:mumble
> lp:patch
> pitivi
> QEMU
> qutim
> lp:sed
> tomahawk
> lp:vlc
> lp:xmms2
> lp:zile
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/bzr/+bug/402814/+subscriptions
>

Changed in launchpad:
importance: Low → Critical
importance: Critical → Low
importance: Low → High
Jelmer Vernooij (jelmer) on 2012-07-09
Changed in bzr:
importance: High → Low
importance: Low → Medium
Changed in launchpad:
importance: High → Low
importance: Low → High
Changed in bzr:
importance: Medium → High
Changed in bzr-git:
assignee: nobody → Jelmer Vernooij (jelmer)
Jarl (jarl-dk) wrote :

hmmm, many of the affected packages seems to be way old in recent ubuntu releases due to this bug, e.g. coreutils.

Jelmer Vernooij (jelmer) wrote :

On Thu, 2012-10-04 at 19:43 +0000, Jarl wrote:
> hmmm, many of the affected packages seems to be way old in recent ubuntu
> releases due to this bug, e.g. coreutils.
This shouldn't have any effect on what version of these packages is in
Ubuntu. New versions can be packaged without working VCS imports

Cheers,

Jelmer

Curtis Hovey (sinzui) on 2012-10-29
tags: added: feature git

How can we workaround this bug?

I see vlc team is workarounding it [1], but I don't see how

[1] https://code.launchpad.net/~videolan/vlc/master-manual

Vsevolod Velichko (torkvemada) wrote :

LocutusOfBorg, I myself created cron script (for lp:qutim) with the following contents:
#!/bin/bash
cd /path/to/qutim-replica
git pull origin master
git submodule update --init --recursive
bzr stat -S|grep ^?|sed 's/^? //'| while read LINE; do bzr add "$LINE"; done
bzr ci -m "Cron update"
bzr push

You also have to add ..git* to your .bzrignore.

Benjamin Drung (bdrung) wrote :

I wrote a small hacky script [1] and run it every 15 minutes via Cron.

[1] https://bazaar.launchpad.net/~videolan/vlc/manual-bzr-import/view/head:/manual-bzr-import

Artem Vorotnikov (skybon) wrote :

So... any ETA on the fix for the bug?

Jelmer Vernooij (jelmer) wrote :

On Tue, Apr 16, 2013 at 01:16:25AM -0000, ?????????? ???????????????????? wrote:
> So... any ETA on the fix for the bug?
Nobody is actively working on this bug at the moment, as far as I am aware.

no longer affects: tribler
description: updated
description: updated
Jelmer Vernooij (jelmer) wrote :

> CloudFoundry uses github. Getting sub modules supported would allow the
> package to be built directly from their repository.
Fixing this doesn't require any changes to cloudfoundry though, which is what that field is about (I agree the naming is a bit confusing).

Changed in cloudfoundry:
status: Confirmed → Invalid
pataquets (pataquets) wrote :

Also, with a github repo with no submodules at all, LP import fails complaining of submodules:
https://code.launchpad.net/~dnscrypt-proxy/dnscrypt-proxy/master

Martin Erik Werner (arand) wrote :

pataquets: That's because there are submodules at some point in the commit history, which will trigger the issue as well.

pataquets (pataquets) on 2013-11-06
description: updated
Jelmer Vernooij (jelmer) on 2013-11-09
description: updated
hexafraction (rarkenin) wrote :

arand: So you're stating that removing submodules isn't a workaround? Anyway it would be extremely beneficial to getting things running if submodules could be skipped as an option for imports, as in many cases any recipes may not even need to care about submodules.

Martin Erik Werner (arand) wrote :

hexafraction:
The only "workaround" (as far as I can tell, provided the way I guess the importer currently works) would be to go through the whole repository history and replace all commits that contained submodules with commits that didn't, and then run the importer on that.

hexafraction (rarkenin) wrote :

Thanks. Unfortunately for two repositories I use their design makes this untenable.

The only workaround I found was to install bazaar WITHOUT the git plugin, bzr init inside the existing repo, fix up ignores, and then just push that. The only pitfall is an extra modification to the git repo's state.

I read about workarounds here, but I don't know how to apply them to lp:openra The web-interface seems very limited in that regard. Is there another way or did some post a step-by-step guide somewhere. Not that the recent commits don't use git submodules anymore. It was many years ago, but still bzr complains because somewhere in the history git submodules have been detected. :(

Jelmer Vernooij (jelmer) wrote :

On Sun, Dec 08, 2013 at 10:27:11PM -0000, Matthias Mailänder wrote:
> I read about workarounds here, but I don't know how to apply them to
> lp:openra The web-interface seems very limited in that regard. Is there
> another way or did some post a step-by-step guide somewhere. Not that
> the recent commits don't use git submodules anymore. It was many years
> ago, but still bzr complains because somewhere in the history git
> submodules have been detected. :(

The workarounds require you to do the import yourself and then
push the result bzr branch up to Launchpad; Launchpad can't
do them for you.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Related questions

Related blueprints