ghc 6.8.2 has important performance bug, should be updated to 6.8.3

[Mark Stosberg]

Binary package hint: ghc6

GHC 6.8.2 has an important performance bug which can cause dramatic slow-downs in some cases. This was noticed by the darcs project:

http://bugs.darcs.net/issue973

The bug was fixed in 6.8.3.

Until this is addressed, darcs built with GHC 6.8.2 on Ubuntu may run significantly slower in some cases.

[StefanPotyra (sistpoty)]

Hi,

thanks for the bug report, confirming.

As far as I've seen it right now, it looks though that the fix would change the ghc6 abi, so all libraries would need to rebuilt. Having entered feature freeze for intrepid already, I doubt we should go for the risk of doing a haskell transition before the release.

Since there seems to be a possible workaround for darcs, do you think it would be sensible to use the workaround instead? (adding a darcs task hence).

Cheers,
    Stefan.

[StefanPotyra (sistpoty)]

Dear motu-release,

the problem seems to be for real in ghc6, using the wrong defines for structs passed to libc, cf. http://hackage.haskell.org/trac/ghc/ticket/2096 and http://hackage.haskell.org/trac/ghc/ticket/2093. This means, that everything dealing with the file system (as in e.g. calling stat) created from ghc6 would be buggy.

To fix this in ghc6 properly, we'd need to transition all haskell packages though, otherwise we'd have a screwed up stack of haskell packages. Do you think that's feasible/or if we could do it in time for intrepid?

Since we were late for hardy with a haskell transition, and geser mainly did the grunt work, I'm not 100% convinced we'd manage such a transition for intrepid.

What's your opinion about this?/any other hints?

Thanks,
    Stefan.

[Iain Lane (laney)]

I'm willing to help with any work that needs to be done for this, as I'd really like to see 6.8.3 in Intrepid.

[Scott Kitterman (kitterman)]

Is it just rebuilds and how many packages?

[StefanPotyra (sistpoty)]

It should be merely a "rebuild", but all source packages would need to be modified: Since the ABI of the libs is also (as is the case for this bug) dependant on the haskell compiler used, all (library) packages have quite strict dependencies on the particular ghc6 version. Furthermore, these "rebuilds" will need to be done in order, so libraries build-depending on other libraries need to get updated to reflect the new version of the build-dependency.

I don't imagine there should be any problems due to this change in ghc6 so that packages won't build again if they previously did. However haskell allows to mix native calls with haskell code, so it's in theory possible.

Binary packages, that would be affected:
$ apt-cache rdepends ghc6
  libghc6-xmonad-dev
  libghc6-xmonad-dev
  libghc6-xmonad-contrib-dev
  libghc6-xmonad-contrib-dev
  libghc6-xhtml-dev
  libghc6-xhtml-dev
  libghc6-x11-dev
  libghc6-x11-dev
  libghc6-wash-dev
  libghc6-wash-dev
  libghc6-vty-dev
  libghc6-vty-dev
  libghc6-uulib-dev
  libghc6-uulib-dev
  libghc6-utf8-string-dev
  libghc6-utf8-string-dev
  libghc6-time-dev
  libghc6-time-dev
  libghc6-tagsoup-dev
  libghc6-tagsoup-dev
  libghc6-stream-dev
  libghc6-stream-dev
  libghc6-stm-dev
  libghc6-stm-dev
  libghc6-src-exts-dev
  libghc6-src-exts-dev
  libghc6-sourceview-dev
  libghc6-sourceview-dev
  libghc6-soegtk-dev
  libghc6-soegtk-dev
  libghc6-regex-posix-dev
  libghc6-regex-posix-dev
  libghc6-regex-compat-dev
  libghc6-regex-compat-dev
  libghc6-regex-base-dev
  libghc6-regex-base-dev
  libghc6-quickcheck-dev
  libghc6-quickcheck-dev
  libghc6-plugins-dev
  libghc6-plugins-dev
  libghc6-pcre-light-dev
  libghc6-pcre-light-dev
  libghc6-parsec-dev
  libghc6-parsec-dev
  libghc6-parallel-dev
  libghc6-parallel-dev
  libghc6-pandoc-dev
  libghc6-pandoc-dev
  libghc6-opengl-dev
  libghc6-opengl-dev
  libghc6-openal-dev
  libghc6-openal-dev
  libghc6-network-dev
  libghc6-network-dev
  libghc6-mtl-dev
  libghc6-mtl-dev
  libghc6-missingpy-dev
  libghc6-missingpy-dev
  libghc6-missingh-dev
  libghc6-missingh-dev
  libghc6-magic-dev
  libghc6-magic-dev
  libghc6-listlike-dev
  libghc6-listlike-dev
  libghc6-ldap-dev
  libghc6-ldap-dev
  libghc6-irc-dev
  libghc6-irc-dev
  libghc6-hunit-dev
  libghc6-hunit-dev
  libghc6-http-dev
  libghc6-http-dev
  libghc6-html-dev
  libghc6-html-dev
  libghc6-hsql-sqlite3-dev
  libghc6-hsql-sqlite3-dev
  libghc6-hsql-postgresql-dev
  libghc6-hsql-postgresql-dev
  libghc6-hsql-odbc-dev
  libghc6-hsql-odbc-dev
  libghc6-hsql-mysql-dev
  libghc6-hsql-mysql-dev
  libghc6-hsql-dev
  libghc6-hsql-dev
  libghc6-hspre...

[Scott Kitterman (kitterman)]

That seems like a lot.

[StefanPotyra (sistpoty)]

On Tuesday 02 September 2008 01:44:06 Scott Kitterman wrote:
> That seems like a lot.

yes, it actually *is* a lot.

I mean I cannot say which of the two options, to either do a lot of work (and
risk the transition to not complete), or to have a buggy ghc6 is the
better/worse option, and which will lead to less problems for/after the
intrepid release, that's why I asked for guidance in the first place ;).

[Scott Kitterman (kitterman)]

If there are people who can do this and are willing to invest the time it
seems reasonable to go for it. If a few get left to -updates it's not the
end of the world.

Given the way the last one went and geser's announcements tha he's largely
absent for some time, I think it comes down to you. Do you think you can
do it or get close?

[StefanPotyra (sistpoty)]

well, due to vacation, I'd only be able to update (or patch) ghc6 after Monday next week.

So starting after that, I doubt that I can get anywhere close to complete a haskell transition for intrepid on my own. OTOH, the packaging changes are actually quite easy/straightforward tasks, so with some help from other MOTU's it might be doable... *shrug*.

[Scott Kitterman (kitterman)]

I suspect the sequence is the main thing. Can you write something up on that?

[Cesare Tirabassi (norsetto)]

Having gone through the gfortran transition during last freeze (which was easy compared with this) and so having some experience about the issues that this will encounter (lack of manpower, reliance on archive-admins to do syncs, possible interdependencies, secondary build depends, etc.) doing this now seems a hopeless task.
Furthermore, is the above list complete? Don't we have a number of packages that would need to be rebuilt as well (those that depends on the libghc6-*-dev stuff)?
The potential to screw up things more, in my opinion, is very high. Unless a team of people lead by Stefan can commit to this I would err more on the cautious side and say to not do this right now and postpone it to intrepid+1.

[Luca Falavigna (dktrkranz)]

If I remember correctly, there are packages which have hardcoded (build-)dependencies with some haskell-related packages, so package list could be longer than the actual. Debian didn't manage to move to 6.8.3 yet (in experimental neither), so this job would be all ours.

Suppose a package wouldn't be transitioned for some reasons, are there known issues (package no longer functional, data loss, and so on) that can be worse than performance issues?

[Scott Kitterman (kitterman)]

I'm convinced we should not try this as unfortunate as that may be.

[Zooko Wilcox-O'Hearn (zooko)]

This is actually a correctness bug for other applications that use the filesystem and use GHC, right? GHC is returning incorrect values for file size and other stats, is that right? And darcs somehow manages to turn this incorrect answer into a correct but slow answer?

What about the option of patching the GHC shipped by Ubuntu to not have this bug without changing the ABI?

[Mark Stosberg (markstos)]

> This is actually a correctness bug for other applications that use the filesystem and use GHC, right?

Right, with darcs perhaps being the most popular.

> GHC is returning incorrect values for file size and other stats, is that right?

I believe so.

> And darcs somehow manages to turn this incorrect answer into a correct but slow answer?

Yes, it seems to trigger fallback mechanism.

> What about the option of patching the GHC shipped by Ubuntu to not have this bug without changing the ABI?

Stefan said "the fix would change the ghc6 abi". That could be clarified was to whether *any* fix would do that, or fixing it via upgrading 6.8.3 would do that.

Let me restate Zooko's concern more plainly from a users perspective: Is there a way to ship a fixed darcs binary, regardless of what happens with GHC and the Haskell package chain?

[StefanPotyra (sistpoty)]

Hi,

On Friday 05 September 2008 21:11:41 Zooko O'Whielacronx wrote:
> This is actually a correctness bug for other applications that use the
> filesystem and use GHC, right? GHC is returning incorrect values for
> file size and other stats, is that right? And darcs somehow manages to
> turn this incorrect answer into a correct but slow answer?

correct.

>
> What about the option of patching the GHC shipped by Ubuntu to not have
> this bug without changing the ABI?

The problem is, that the fix itself actually changes the ABI.

Cheers,
   Stefan.

[StefanPotyra (sistpoty)]

Hi,

On Monday 08 September 2008 13:32:36 Mark Stosberg wrote:
[..]
>
> have this bug without changing the ABI?
>
> Stefan said "the fix would change the ghc6 abi". That could be clarified
> was to whether *any* fix would do that, or fixing it via upgrading 6.8.3
> would do that.

Any fix applied to ghc would do that... :/

>
> Let me restate Zooko's concern more plainly from a users perspective: Is
> there a way to ship a fixed darcs binary, regardless of what happens
> with GHC and the Haskell package chain?

that's a good question, and I only know about it what's presented at
http://bugs.darcs.net/issue973.

Any help in this regard would be highly appreciated!

Cheers,
    Stefan.

[Iain Lane (laney)]

What about applying the patch at: http://hackage.haskell.org/trac/ghc/ticket/2093 ?

[Mark Stosberg (markstos)]

There is a binary distribution of GHC 6.8.3 available for Linux:

http://www.haskell.org/ghc/download_ghc_683.html#x86linux

Could this be used to build an updated darcs binary until the official GHC packages can be updated to 6.8.3?

[Scott Kitterman (kitterman)]

Are there darcs dependencies that also build against ghc? If so they'd all
have to be built against the same version.

[Mark Stosberg (markstos)]

On Fri, 12 Sep 2008 05:30:13 -0000
Scott Kitterman <email address hidden> wrote:

> Are there darcs dependencies that also build against ghc? If so they'd all
> have to be built against the same version.

Wouldn't all the dependencies be listed as 'dependencies' through the package
management system?

 http://packages.ubuntu.com/intrepid/devel/darcs

I see no additional Haskell libraries listed there. If there are any, I think
it's very small number.

    Mark

[Michael Casadevall (mcasadevall)]

Given the ABI breaks, what about packaging GHC 6.8.3 as a separate package, then port the necessary packages next release, and then for intrepid+1 we can simply sync the updated packages from Debian once they finish their transition?

[StefanPotyra (sistpoty)]

@Mark: using a binary distribution to build darcs doesn't make sense, since haskell packages are not arch:all packages. Also haskell libraries are not like regular c shared objects, but rather get linked statically once a binary is created, so there are no depends on haskell libraries found in darcs.

@Michael: I'd not like to do this, this would mean to change all haskell libs resulting from ghc6 itself, and hence would need more toolchain to get duplicated.

@Ian: yes, this patch could potentially break the ABI, or rather the underlying C ABI that gets used by haskell libraries, and hence I could imaging situations, that would result in breakage.
However I've really been thinking at length at the effect of this patch. Imho any breakage should be limited to packages that really do quite nasty things. Hence I guess we could risk a shot in the dark and apply that. I'd like to do some tests however first (esp. ghc --show-iface against the resulting .hi files from that change).

[Mark Stosberg (markstos)]

> Imho any breakage should be limited to packages that really do quite nasty things.
> Hence I guess we could risk a shot in the dark and apply that. I'd like to do some tests
> however first (esp. ghc --show-iface against the resulting .hi files from that change).

Stefan,

That sounds great, Thanks! I really appreciate your help.

   Mark

[Iain Lane (laney)]

To be honest this feels very risky to me, and I'm having grave doubts that anything other than a full transition is good enough. I can quite easily see this introducing unforseen regressions that are only picked up once Intrepid is released and people start hammering on it seriously. We then get into SRUs which are a lot of hassle for many people. I don't want people to think that Ubuntu is a broken platform for Haskell (yes, I know that the presence of this bug introduces a degree of breakage but IMO it's not as severe as the risk of the fix).

So consider this my vote for no change without a full transition (and then we may as well go to 6.8.3). This is clearly unfortunate for Darcs but I believe that the risks outweigh the potential benefits.

[StefanPotyra (sistpoty)]

Hi Iain,

first off, what I think that change will result in is -DLARGE_FILE_OFFSET
being passed to ghc6 internal libraries when compiled as C-code. I'm not
entirely sure if this will match/won't match the compiler flags for cabalized
packages. So the possibility of a different ABI for packages would result if
these would call c-code dependendant on -DLARGE_FILE_OFFSET being present or
not (or more specific, if the (external) c-ABI would change if
DLARGE_FILE_OFFSET was present or not).
Imho that means that only haskell libraries calling external c-code dependant
on that compiler flag would be affected. One candidate that comes to my mind
would be gtk2hs, and I'll certainly check if its ABI will stay the same before
applying the patch.

Did I miss s.th.?

On Friday 19 September 2008 22:18:38 Iain Lane wrote:
> To be honest this feels very risky to me, and I'm having grave doubts
> that anything other than a full transition is good enough. I can quite
> easily see this introducing unforseen regressions that are only picked
> up once Intrepid is released and people start hammering on it seriously.
> We then get into SRUs which are a lot of hassle for many people. I don't
> want people to think that Ubuntu is a broken platform for Haskell (yes,
> I know that the presence of this bug introduces a degree of breakage but
> IMO it's not as severe as the risk of the fix).

Fair enough. There certainly is the possibility that haskell libraries
depending on other haskell libraries would fail to build after that change
(w.o. rebuilding the former libraries beforehand).

>
> So consider this my vote for no change without a full transition (and
> then we may as well go to 6.8.3). This is clearly unfortunate for Darcs
> but I believe that the risks outweigh the potential benefits.

What other options do we have to fix darcs? Is there any workaround we could
apply to darcs?

Cheers and thanks for your help,
    Stefan.

[Mark Stosberg (markstos)]

> What other options do we have to fix darcs? Is there any workaround
> we could apply to darcs?

Perhaps this is a solution that it outside of the normal package management workflow, but wouldn't this work:

1. Download and compile GHC 6.8.3 for Ubuntu from here:
    http://www.haskell.org/ghc/download_ghc_683.html#x86linux

2. Compile the small number of Haskell dependencies for darcs for 6.8.3.

3. Compile darcs for Ubuntu using the above.

4. Post the resulting binary somewhere, maybe as package managed outside of the official Ubuntu repos.

###

Am I missing something about the feasibility of this? Surely there are GHC contributors who are already running at least 6.8.3 on Ubuntu.

[StefanPotyra (sistpoty)]

Woohoo. Good news everyone: After some looking, fiddling and test-building, I've found that I was wrong:
The _FILE_OFFSET_BITS definition is local to hsunix of ghc6, so it cannot affect the ABI of other libraries. And even better: Comparing the interface (I used --show-iface and checked defined symbols in the object file as well) before and after the patch shows that the interface doesn't change at all.

So sorry for shouting ABI breakage in the first place, where there is actually none :). I'll upload ghc6 in a minute, but it might take some time before it's shoved into the archives (as we're currently in beta freeze).

With the patched version, the test-case in the upstream haskell bug works now but I'll attach an i386 rebuilt darcs soon. Could you then check if that solves the issue?

Thanks,
   Stefan.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ghc6 - 6.8.2-6ubuntu2

---------------
ghc6 (6.8.2-6ubuntu2) intrepid; urgency=low

  * libraries/unix/configure.ac: Use AC_SYS_LARGEFILE, taken from
    <http://hackage.haskell.org/trac/ghc/ticket/2093>. Fixes
    accessing getSymbolicLinkStatus with the wrong offsets,
    LP: #263773.
  * Run autoreconf in libraries/unix so that the configure.ac change
    takes effect.
  * debian/control: Set Maintainer from myself to MOTU.

 -- Stefan Potyra <email address hidden> Sat, 27 Sep 2008 22:19:03 +0200

[Mark Stosberg (markstos)]

Stefan,

This is truly great news. I've begun to work with Eric Kow, a darcs leader, to
confirm the fix.

From the original darcs bug report, it looks like the trigger case to test is
running "darcs whatnews --summary" on a largish repo that is in the 'hashed' or
'darcs-2' format.

Trying that with the fixed and unfixed binaries may show a difference.

    Mark

[Mark Stosberg (markstos)]

A darcs developer has confirmed that the updated darcs is fixed.

( reference: http://bugs.darcs.net/msg6171 ).

Please proceed with making the fixed darcs binary available.

Again, thanks so much for this!

I plan to release a blog post soon highlighting the ease of installing darcs 2 on Ubuntu, and I have been waiting on this fix to do that.

    Mark

[StefanPotyra (sistpoty)]

excellent, thanks!

I'll upload the rebuild once I'm home from work.

[Mark Stosberg (markstos)]

Iain,

Would you be able to update your 2.0.2 PPA binaries for Hardy as well?

https://launchpad.net/~laney/+archive

   Mark

[StefanPotyra (sistpoty)]

uploaded, but it's currently sitting in the queue (beta freeze).

[Launchpad Janitor (janitor)]

This bug was fixed in the package darcs - 2.0.2-2ubuntu2

---------------
darcs (2.0.2-2ubuntu2) intrepid; urgency=low

  * Rebuild with fixed ghc6 to fix an performance problem, LP: #263773.

 -- Stefan Potyra <email address hidden> Mon, 29 Sep 2008 20:30:21 +0200

[Iain Lane (laney)]

Mark,

I've updated them to 2.1.0pre2, and subsequently to pre3. The binaries for Intrepid should therefore have the fix.

Stefan, do you think it's reasonable to SRU this latest GHC upload and a Darcs rebuild for Hardy? It'd be a shame if the LTS release was permanently hit by this. I'll prepare the upload if you'd like.

Thanks,
Iain

[StefanPotyra (sistpoty)]

Hi Iain,

On Tuesday 07 October 2008 12:55:51 Iain Lane wrote:
[..]
>
> Stefan, do you think it's reasonable to SRU this latest GHC upload and a
> Darcs rebuild for Hardy? It'd be a shame if the LTS release was
> permanently hit by this.

ghc6: yes, I'd personally like to see it fixed in hardy. I only wanted to wait
a bit after the last two uploads to intrepid, just in case. But I think, we
should go ahead now. Imho the best step would be to just ship the same
version (only version number adjust) that we have in intrepid right now to
hardy via SRU. What do you think?

darcs: it's still version 1.something, is it affected as well? (oh, and once
we've got the fix of ghc6 in, we could request a backport of darcs).

> I'll prepare the upload if you'd like.
That would be excellent, thanks for the offer!

Cheers,
    Stefan.

[Iain Lane (laney)]

It depends how conservative we want to be I guess. All of the fixes between Hardy and Intrepid are bug/buildfixes so there should be no new regressions introduced, but this is the only fix we really want. Subscribing motu-sru for an opinion.

As for darcs, a simple grepping of the source shows that the version in Hardy calls getSymbolicLinkStatus so I guess it's still affected, if not as severely as 2.x. It's worth a rebuild in -updates IMHO.

[StefanPotyra (sistpoty)]

Hi,

On Tuesday 07 October 2008 14:27:40 Iain Lane wrote:
> It depends how conservative we want to be I guess. All of the fixes
> between Hardy and Intrepid are bug/buildfixes so there should be no new
> regressions introduced, but this is the only fix we really want.
> Subscribing motu-sru for an opinion.

well, I guess the fix for Debian #491909 would also be good to have.
I guess it makes sense to also have the watcher script that prints stuff
during build, as I don't want to pester our buildd admins again if the build
times out on sparc.

Oh, can you eventually open a separate bug for the SRU, so that it's not so
clattered with a lengthy history of comments? (and wrong assumptions from
myself *cough* ;))

>
> As for darcs, a simple grepping of the source shows that the version in
> Hardy calls getSymbolicLinkStatus so I guess it's still affected, if not
> as severely as 2.x. It's worth a rebuild in -updates IMHO.

Ok, thanks!

Cheers,
    Stefan.

[Iain Lane (laney)]

Alright, SRU bug is at bug #280129

[Zooko Wilcox-O'Hearn (zooko)]

It has been marked as resolved on the darcs-roundup. Presumably launchpad will notice at some point and mark it as resolved here.