multilib selects wrong start files

Bug #1767223 reported by Ed
176
This bug affects 32 people
Affects Status Importance Assigned to Milestone
newlib (Ubuntu)
Fix Released
Medium
Unassigned
Bionic
Fix Released
Medium
Simon Quigley

Bug Description

[Impact]

This bug prevents users to produce binaries for some of the targets supported by the compiler. In some case, binaries can get produced but will fail to execute properly.

To fix this issue the package needs to be rebuilt from the same source. Risk of regressions are thus limited to an issue in the arm-none-eabi toolchain used for the rebuild.

The bug stems from newlib having libraries for various targets (called multilib) in a different location than where the compiler expects them to be, thus making the wrong libraries get selected. The location of the newlib libraries is decided at build time by asking the compiler (provided in gcc-arm-none-eabi) where it will be looking for these. Unfortunately, the version of newlib in Ubuntu bionic was built using a different version of gcc-arm-none-eabi than the one currently in bionic and GCC changed where to look for library between those 2 versions. This is why rebuilding against the current gcc-arm-none-eabi will solve the issue.

[Test Case]

Compile the following hello world testcase (in hello.c) with arm-none-eabi-gcc -o hello.axf hello.c -mthumb -Wl,--gc-sections -ffast-math -march=armv7e-m -specs=rdimon.specs

#include <stdio.h>

int
main (void)
{
  puts ("Hello, World!");
  return 0;
}

[Regression Potential]

As explained in the impact, no source change is necessary so any regression would be due to bugs in the compiler which would be good to catch anyway.

Original bug report below:

The multilib setup is not working in the bionic version of gcc-arm-none-eabi (gcc v6.3.1). For example, if I build a project with "-mcpu=cortex-m4" or "-march=armv7e-m" and link with:

arm-none-eabi-gcc -o someoutput.elf object1.o object2.o -mthumb -Wl,--gc-sections -ffast-math -march=armv7-m -Tlinker_script.ld

I get as output:

/usr/lib/gcc/arm-none-eabi/6.3.1/../../../arm-none-eabi/bin/ld: error: /usr/lib/gcc/arm-none-eabi/6.3.1/../../../arm-none-eabi/lib/crt0.o: Conflicting CPU architectures 13/1
/usr/lib/gcc/arm-none-eabi/6.3.1/../../../arm-none-eabi/bin/ld: failed to merge target specific data of file /usr/lib/gcc/arm-none-eabi/6.3.1/../../../arm-none-eabi/lib/crt0.o

arm-none-eabi-readelf -A on any of my object files produces:

Attribute Section: aeabi
File Attributes
  Tag_CPU_name: "Cortex-M4"
  Tag_CPU_arch: v7E-M
  Tag_CPU_arch_profile: Microcontroller
  Tag_THUMB_ISA_use: Thumb-2
  Tag_ABI_PCS_wchar_t: 4
  Tag_ABI_FP_number_model: Finite
  Tag_ABI_align_needed: 8-byte
  Tag_ABI_align_preserved: 8-byte, except leaf SP
  Tag_ABI_enum_size: small
  Tag_ABI_optimization_goals: Prefer Speed
  Tag_CPU_unaligned_access: v6

But "arm-none-eabi-readelf -A /usr/lib/gcc/arm-none-eabi/6.3.1/../../../arm-none-eabi/lib/crt0.o" gives

Attribute Section: aeabi
File Attributes
  Tag_CPU_arch: v4
  Tag_ARM_ISA_use: Yes

If I define "void _init(void){}" and use -nostartfiles, compilation finishes but the resulting binary does not work. I have not examined the output file but it appears that it has somehow linked to code containing ARM as well as thumb code:

# readelf -A someoutput.elf
Attribute Section: aeabi
File Attributes
  Tag_CPU_name: "Cortex-M4"
  Tag_CPU_arch: v7E-M
  Tag_CPU_arch_profile: Microcontroller
  Tag_ARM_ISA_use: Yes
  Tag_THUMB_ISA_use: Thumb-2
  Tag_ABI_PCS_wchar_t: 4
  Tag_ABI_FP_denormal: Needed
  Tag_ABI_FP_exceptions: Needed
  Tag_ABI_FP_number_model: IEEE 754
  Tag_ABI_align_needed: 8-byte
  Tag_ABI_align_preserved: 8-byte, except leaf SP
  Tag_ABI_enum_size: small
  Tag_ABI_optimization_goals: Prefer Speed
  Tag_CPU_unaligned_access: v6

I can work around the problem by manually specifying compatible versions of libc and libgcc with -nostdlib:

-nostdlib /usr/lib/arm-none-eabi/newlib/thumb/libc.a /usr/lib/gcc/arm-none-eabi/6.3.1/thumb/v7e-m/libgcc.a

This produces a working binary. However, there is still something wrong because readelf -A still includes "Tag_ARM_ISA_use: Yes":

Attribute Section: aeabi
File Attributes
  Tag_CPU_name: "Cortex-M4"
  Tag_CPU_arch: v7E-M
  Tag_CPU_arch_profile: Microcontroller
  Tag_ARM_ISA_use: Yes
  Tag_THUMB_ISA_use: Thumb-2
  Tag_ABI_PCS_wchar_t: 4
  Tag_ABI_FP_denormal: Needed
  Tag_ABI_FP_exceptions: Needed
  Tag_ABI_FP_number_model: IEEE 754
  Tag_ABI_align_needed: 8-byte
  Tag_ABI_align_preserved: 8-byte, except leaf SP
  Tag_ABI_enum_size: small
  Tag_ABI_optimization_goals: Prefer Speed
  Tag_CPU_unaligned_access: v6

Previously, using xenial and gcc v4, readelf gives the following output on the output binary:
Attribute Section: aeabi
File Attributes
  Tag_CPU_name: "7E-M"
  Tag_CPU_arch: v7E-M
  Tag_CPU_arch_profile: Microcontroller
  Tag_THUMB_ISA_use: Thumb-2
  Tag_ABI_PCS_wchar_t: 4
  Tag_ABI_FP_denormal: Needed
  Tag_ABI_FP_exceptions: Needed
  Tag_ABI_FP_number_model: IEEE 754
  Tag_ABI_align_needed: 8-byte
  Tag_ABI_enum_size: small
  Tag_ABI_optimization_goals: Prefer Speed
  Tag_CPU_unaligned_access: v6

(Also, what happened to gdb-arm-none-eabi?)

Revision history for this message
Ed (edd-robbins) wrote :

Note: I do not think it is an upstream problem, since I downloaded gcc-arm-none-eabi-6-2017-q2-update from ARM which contains a very similar version (both 6.3.1 20170620) and it does not exhibit the issue.

Revision history for this message
Luca Capra (luca-capra) wrote :

I am having that issue also (using mbed-cli) after upgrading to 18.04. (Used to work on 17.10).

quote >> (Also, what happened to gdb-arm-none-eabi?)

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in gcc-arm-none-eabi (Ubuntu):
status: New → Confirmed
Revision history for this message
Hubert Miś (hubertmis) wrote :

The same issue here. Needed to uninstall from apt and download from ARM website.

Revision history for this message
fl0 (fl0) wrote :

Encountered the same issue. Is there a workaround/patch?

Revision history for this message
Jochen Hoenicke (hoenicke-0) wrote :

I'm also affected by this (Cortex M3).

My workaround:

    cd /usr/lib/arm-none-eabi/lib/thumb/v7-m
    sudo ln -s ../* .

(for cortex m4, you need to go to v7e-m).

The real fix would be to fix the default library paths for arm-none-eabi-gcc, to also include the thumb directory. I have no idea where this config is located, though.

Revision history for this message
Jochen Hoenicke (hoenicke-0) wrote :

There is more broken, thumb/libc.a contains strcmp for the wrong architecture. So unless you don't need strcmp, there is no workaround.

Revision history for this message
Ed (edd-robbins) wrote :

Yes, I think aside from building your own libc (and I'm not even certain that would fix it) the only alternative at present is to use a different toolchain or build your own. The binary toolchains from ARM (https://developer.arm.com/open-source/gnu-toolchain/gnu-rm/downloads) do however work fine.

Revision history for this message
Ed (edd-robbins) wrote :

(The work around mentioned in my opening post only partially helps, my code eventually crashes, presumably executing a libc or other stdlib function)

Revision history for this message
Jacob (jacob11) wrote :

Upstream Debian has fixed the issue and Cosmic already has the new packages. If you are on Bionic you can install the Cosmic packages to temporarily fix the issue, see https://github.com/bbcmicrobit/micropython/issues/514#issuecomment-404759614

Revision history for this message
Ed (edd-robbins) wrote :

That's good, but I don't understand why fixed packages are not available for bionic. It's been almost 3 months! For now I will continue using the cross compiler builds provided by ARM.

Revision history for this message
Jacob (jacob11) wrote :

I think they just don't care enough about the "universe" packages (or don't have the time/resources to care), they are just importing them from Debian's repos "as is" and very rarely update them, if ever.

Revision history for this message
David Lawrence (dlaw) wrote :

I can confirm that this bug is still present in Bionic. The libnewlib-arm-none-eabi package is unusable. Pulling in the fixed version from upstream, or switching to the gcc-arm-embedded ppa, fixes the issue.

Revision history for this message
Akkenoth (akkenoth) wrote :

The bug is still present (2018-12-17).
Package version: 15:6.3.1+svn253039-1build1

Revision history for this message
Matthias Klose (doko) wrote :

> but I don't understand why fixed packages are not available for bionic.

because nobody stepped up to do the work?
See https://wiki.ubuntu.com/StableReleaseUpdates
See https://wiki.ubuntu.com/UbuntuBackports

And no, it doesn't help to state "the bug is still there" without addressing it.

Revision history for this message
Ed (edd-robbins) wrote :

> And no, it doesn't help to state "the bug is still there" without addressing it.

And your reply to my 5 month old comment serves no purpose and in no way helps either.

I am not in a position to do anything about this, so suggesting that I should is not helpful. I am not familiar with the process, nor am I paid to develop Ubuntu. They have people for that and I do not have the time. I have done my part as a user by reporting the bug. Most active projects do appreciate and address bug reports. It does seem strange to me that the exception to that often seems to be commercially backed projects.

Source and pre-rolled packages are available to address the issue if they so wish. If they want to keep distributing broken packages, then it is frustrating for users but it is not my problem. If the package is not going to be fixed, and given that as it stands it is completely non-functional, it would actually be better from a user experience perspective for it to simply be removed than to keep it live, in my opinion.

In any case, why would I spend my time trying to get this fixed when I have been using the compiler distribution directly from ARM reasonably comfortably every day since I reported this bug? I need an arm-none compiler for work, and I will take the fastest route to get one, anything else is a waste of time when I have deadlines to meet. The fastest and thus most optimal route for any user is to find an alternative provider for the package, especially given that in this case the next OS release will likely have fixed the issue anyway.

And yes, expressing my surprise that the completely broken package has not been replaced with the fully working package is not particularly helpful, but it is certainly not damaging. On the other hand, successive experiences of broken packages and this type of interaction may move some users away from Ubuntu in the longer term.

Revision history for this message
Matthias Klose (doko) wrote :

the universe section of the archive is mostly maintained by the Ubuntu community. Non-paid, non-commercially. So you either work with them to fix the issue, or live with it. No, I'm not willing to spend my free time on this issue, and apparently you aren't either.

Revision history for this message
Thomas Preud'homme (thomas-preudhomme) wrote :

This bug is due to newlib built against an older version of gcc-arm-none-eabi. It will be fixed by rebuilding newlib using the same source.

affects: gcc-arm-none-eabi (Ubuntu) → newlib (Ubuntu)
Revision history for this message
Thomas Preud'homme (thomas-preudhomme) wrote :

@ubuntu-sru and @ubuntu-sponsors: would you mind doing a new upload of this package to bionic using the same source as in the last build? This bug will be solved by rebuilding from the same source using the gcc-arm-none-eabi package currently in bionic.

Best regards.

description: updated
Changed in newlib (Ubuntu Bionic):
status: New → Triaged
Changed in newlib (Ubuntu Disco):
status: Confirmed → New
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in newlib (Ubuntu Cosmic):
status: New → Confirmed
Changed in newlib (Ubuntu):
status: New → Confirmed
Mathew Hodson (mhodson)
Changed in newlib (Ubuntu Bionic):
importance: Undecided → Medium
Changed in newlib (Ubuntu Cosmic):
importance: Undecided → Medium
Changed in newlib (Ubuntu Disco):
importance: Undecided → Medium
Revision history for this message
Simon Quigley (tsimonq2) wrote :

Hello Thomas, thank you for your work on this and I apologize for the delay on this.

The Ubuntu Sponsors Team reviews packages which have a diff already attached to the bug. Even if the solution is to just upgrade to the package in Cosmic, in order to sponsor this, that's what we need.

Taking my Sponsors Team hat off, if you attach a diff here that doesn't include packaging and personally subscribe me, I will be happy to do the packaging bits and drive this to completion.

Thanks again.

Revision history for this message
David Lawrence (dlaw) wrote :

Hi Simon, thank you very much for your attention to this bug as it has a major impact for many of us users!

My understanding is that the diff required to fix this bug is the empty diff. As Thomas said -- "This bug is due to newlib built against an older version of gcc-arm-none-eabi. It will be fixed by rebuilding newlib using the same source."

In other words, this is a build error rather than a packaging error. All that is needed to fix this package is a rebuild (with the latest bionic release of gcc-arm-none-eabi, rather than an obsolete version, present in the build environment). To the best of my knowledge there are no changes that would require a diff.

Warm regards,
David Lawrence

Revision history for this message
Simon Quigley (tsimonq2) wrote :

Hi David!

Is this only an issue on Bionic, or Cosmic and Disco as well?

Thanks for the quick response.

Revision history for this message
David Lawrence (dlaw) wrote :

This issue is present in Bionic only. (Looking back at the discussion on this bug, the package was confirmed working on Cosmic as of 2018-07-16.)

Best,
David

Revision history for this message
Simon Quigley (tsimonq2) wrote :

Uploaded to Bionic, it is now awaiting review by the SRU team.

Changed in newlib (Ubuntu Cosmic):
status: Confirmed → Fix Released
Changed in newlib (Ubuntu Disco):
status: Confirmed → Fix Released
Changed in newlib (Ubuntu Bionic):
status: Triaged → In Progress
assignee: nobody → Simon Quigley (tsimonq2)
Revision history for this message
Adam Conrad (adconrad) wrote : Please test proposed package

Hello Ed, or anyone else affected,

Accepted newlib into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/newlib/2.4.0.20160527-3build1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in newlib (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-bionic
Revision history for this message
David Lawrence (dlaw) wrote :

Dear Adam and/or Simon,

I would like to test the new package. However, the page at
https://launchpad.net/ubuntu/+source/newlib/2.4.0.20160527-3build1
does not show any binaries available, and there is an icon
visible which is captioned "Failed to Build".

It's been 22 hours since the change was accepted into
bionic-proposed -- is it normal for a build to take this long?
Is the "Failed to Build" icon always displayed when there are
no binaries associated with a package, or does this indicate a
particular failure during compilation?

Cheers,
David

Revision history for this message
David Lawrence (dlaw) wrote :

I dug a little deeper and found that that the build failed with
the following error message:

"""
The following packages have unmet dependencies:
 sbuild-build-depends-newlib-dummy : Depends: libcloog-isl-dev but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
"""

I am standing by for testing whenever new binaries are available.

Thanks,
David

David Lawrence (dlaw)
tags: added: verification-failed-bionic
removed: verification-needed-bionic
Mathew Hodson (mhodson)
Changed in newlib (Ubuntu):
status: Confirmed → Fix Released
no longer affects: newlib (Ubuntu Cosmic)
no longer affects: newlib (Ubuntu Disco)
Changed in newlib (Ubuntu Bionic):
status: Fix Committed → Triaged
Mathew Hodson (mhodson)
tags: removed: verification-needed
Revision history for this message
Dan Streetman (ddstreet) wrote :

fixed newlib package uploaded to bionic unapproved queue.

Revision history for this message
Steve Langasek (vorlon) wrote :

Hello Ed, or anyone else affected,

Accepted newlib into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/newlib/2.4.0.20160527-3ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in newlib (Ubuntu Bionic):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-bionic
removed: verification-failed-bionic
Revision history for this message
David Lawrence (dlaw) wrote :

I tested with libnewlib-arm-none-eabi and libnewlib-dev from bionic-proposed:

bionic-proposed/universe amd64 libnewlib-arm-none-eabi all 2.4.0.20160527-3ubuntu0.1
bionic-proposed/universe amd64 libnewlib-dev all 2.4.0.20160527-3ubuntu0.1

I am pleased to report that the bug is fixed.

"readelf -A" output before upgrade included "Tag_ARM_ISA_use: yes";
the same ELF file after upgrade correctly includes Thumb ISA only.

Big thanks to everyone who helped with the fix.

tags: added: verification-done-bionic
removed: verification-needed verification-needed-bionic
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for newlib has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package newlib - 2.4.0.20160527-3ubuntu0.1

---------------
newlib (2.4.0.20160527-3ubuntu0.1) bionic; urgency=medium

  * Change build dep from libisl-dev to libisl-0.18-dev,
    as libcloog-isl-dev depends on that older version (LP: #1767223)

 -- Dan Streetman <email address hidden> Tue, 04 Jun 2019 08:06:13 -0400

Changed in newlib (Ubuntu Bionic):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.