KiCad 5.0.0-rc1 FTBFS on mips, mipsel, powerpc and ppc64el due undeclared initializer LIBCONTEXT_CALL_CONVENTION

[tijuca]

This report is a attempt to keep the report on the ML tracked.
https://lists.launchpad.net/kicad-developers/msg34840.html

KiCad 5 uses some own assembler blobs in include/system/libcontext.hto replace boost.context
partially. The header file is only taking care for i386, amd64 and arm (without dedicated further specialisation).

 25 #if defined(__GNUC__) || defined(__APPLE__) || defined(__FreeBSD__)
 26
 27 #define LIBCONTEXT_COMPILER_gcc
 28
 29 #if defined(__linux__) || defined(__FreeBSD__)
 30 #if defined(__x86_64__) || defined(__amd64__)
 31 #define LIBCONTEXT_PLATFORM_linux_x86_64
 32 #define LIBCONTEXT_CALL_CONVENTION
 33 #elif __i386__
 34 #define LIBCONTEXT_PLATFORM_linux_i386
 35 #define LIBCONTEXT_CALL_CONVENTION
 36 #elif __arm__
 37 #define LIBCONTEXT_PLATFORM_linux_arm32
 38 #define LIBCONTEXT_CALL_CONVENTION
 39 #elif __aarch64__
 40 #define LIBCONTEXT_PLATFORM_linux_arm64
 41 #define LIBCONTEXT_CALL_CONVENTION
 42 #endif

Thus is resulting that the initializer LIBCONTEXT_CALL_CONVENTION is only defined for the platforms given in the header file. And this provokes build issues on all other platform than i386, amd64 and arm.

As I've understand the answer from Simon correctly with Boost >= 1.61 the used assembler blobs aren't needed any longer.
https://lists.launchpad.net/kicad-developers/msg34854.html

The current solution with the dedicated assembler blobs is a bit unfortunate as the build for arm32 would need to get more adjustments due needed additional compiler flags and probably later also needed linker flags. Otherwise the build on armvt5 will fail with the following error message.

[ 58%] Building CXX object common/CMakeFiles/common.dir/system/libcontext.cpp.o
cd "/<<BUILDDIR>>/kicad-5.0.0~rc1+dfsg1/debian/build/common" && /usr/bin/c++ -DHAVE_STDINT_H -DKICAD_SCRIPTING -DKICAD_SCRIPTING_MODULES -DKICAD_SCRIPTING_WXPYTHON -DKICAD_USE_OCE -DUSE_OPENMP -DWXUSINGDLL -DWX_COMPATIBILITY -D_FILE_OFFSET_BITS=64 -D__WXGTK__ -I"/<<BUILDDIR>>/kicad-5.0.0~rc1+dfsg1/include" -I"/<<BUILDDIR>>/kicad-5.0.0~rc1+dfsg1/common/." -isystem /usr/lib/arm-linux-gnueabi/wx/include/gtk2-unicode-3.0 -isystem /usr/include/wx-3.0 -I"/<<BUILDDIR>>/kicad-5.0.0~rc1+dfsg1/common/./dialogs" -I"/<<BUILDDIR>>/kicad-5.0.0~rc1+dfsg1/common/./widgets" -I"/<<BUILDDIR>>/kicad-5.0.0~rc1+dfsg1/common/./dialog_about" -I/usr/include/cairo -I/usr/include/pixman-1 -I/usr/include/arm-linux-gnueabi -I"/<<BUILDDIR>>/kicad-5.0.0~rc1+dfsg1/common/../3d-viewer" -I"/<<BUILDDIR>>/kicad-5.0.0~rc1+dfsg1/common/../pcbnew" -I"/<<BUILDDIR>>/kicad-5.0.0~rc1+dfsg1/common/../polygon" -I"/<<BUILDDIR>>/kicad-5.0.0~rc1+dfsg1/debian/build" -I/usr/include/python2.7 -I"/<<BUILDDIR>>/kicad-5.0.0~rc1+dfsg1/scripting" -Wall -g -O2 -fdebug-prefix-map=/<<BUILDDIR>>/kicad-5.0.0~rc1+dfsg1=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wsuggest-override -Wno-unused-local-typedefs -Wno-strict-aliasing -fopenmp -pthread -Wshadow -fPIC -fvisibility=hidden -fvisibility-inlines-hidden -std=gnu++11 -o CMakeFiles/common.dir/system/libcontext.cpp.o -c "/<<BUILDDIR>>/kicad-5.0.0~rc1+dfsg1/common/system/libcontext.cpp"
/tmp/ccawy5kU.s: Assembler messages:
/tmp/ccawy5kU.s:31: Error: selected processor does not support `vstmia sp,{d8-d15}' in ARM mode
/tmp/ccawy5kU.s:41: Error: selected processor does not support `vldmia sp,{d8-d15}' in ARM mode
common/CMakeFiles/common.dir/build.make:3410: recipe for target 'common/CMakeFiles/common.dir/system/libcontext.cpp.o' failed

The build is failing later too while linking if the CXXFLAGS are expanded by '-mthumb -mfloat-abi=soft -mfpu=neon' before so the compiler is building the shared objects.

[100%] Linking CXX shared module _pcbnew.kiface
cd "/build/kicad-5.0.0~rc1+dfsg1/debian/build/pcbnew" && /usr/bin/cmake -E cmake_link_script CMakeFiles/pcbnew_kiface.dir/link.txt --verbose=1
/usr/lib/ccache/c++ -fPIC -Wall -g -O2 -fdebug-prefix-map=/build/kicad-5.0.0~rc1+dfsg1=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -mthumb -mfloat-abi=soft -mfpu=neon -Wdate-time -D_FORTIFY_SOURCE=2 -Wsuggest-override -Wno-unused-local-typedefs -Wno-strict-aliasing -fopenmp -pthread -Wl,--no-undefined -Wl,-Bsymbolic -shared -o _pcbnew.kiface CMakeFiles/pcbnew_kiface.dir/pcbnew.cpp.o CMakeFiles/pcbnew_kiface.dir/microwave/microwave_inductor.cpp.o CMakeFiles/pcbnew_kiface.dir/dialogs/dialog_SVG_print.cpp.o CMakeFiles/pcbnew_kiface.dir/dialogs/dialog_SVG_print_base.cpp.o CMakeFiles/pcbnew_kiface.dir/dialogs/dialog_block_options.cpp.o CMakeFiles/pcbnew_kiface.d
...
_tool.cpp.o CMakeFiles/pcbnew_kiface.dir/tools/footprint_editor_tools.cpp.o CMakeFiles/pcbnew_kiface.dir/tools/pad_tool.cpp.o CMakeFiles/pcbnew_kiface.dir/tools/pcb_actions.cpp.o CMakeFiles/pcbnew_kiface.dir/tools/pcb_bright_box.cpp.o CMakeFiles/pcbnew_kiface.dir/tools/pcb_editor_control.cpp.o CMakeFiles/pcbnew_kiface.dir/tools/pcb_selection_conditions.cpp.o CMakeFiles/pcbnew_kiface.dir/tools/pcb_tool.cpp.o CMakeFiles/pcbnew_kiface.dir/tools/pcbnew_control.cpp.o CMakeFiles/pcbnew_kiface.dir/tools/picker_tool.cpp.o CMakeFiles/pcbnew_kiface.dir/tools/placement_tool.cpp.o CMakeFiles/pcbnew_kiface.dir/tools/point_editor.cpp.o CMakeFiles/pcbnew_kiface.dir/tools/position_relative_tool.cpp.o CMakeFiles/pcbnew_kiface.dir/tools/selection.cpp.o CMakeFiles/pcbnew_kiface.dir/tools/selection_tool.cpp.o CMakeFiles/pcbnew_kiface.dir/tools/size_menu.cpp.o CMakeFiles/pcbnew_kiface.dir/tools/tool_event_utils.cpp.o CMakeFiles/pcbnew_kiface.dir/tools/tools_common.cpp.o CMakeFiles/pcbnew_kiface.dir/tools/zone_create_helper.cpp.o CMakeFiles/pcbnew_kiface.dir/tools/zone_filler_tool.cpp.o CMakeFiles/pcbnew_kiface.dir/footprint_preview_panel.cpp.o CMakeFiles/pcbnew_kiface.dir/__/common/dialogs/dialog_page_settings.cpp.o CMakeFiles/pcbnew_kiface.dir/__/common/base_units.cpp.o CMakeFiles/pcbnew_kiface.dir/dialogs/dialog_scripting.cpp.o CMakeFiles/pcbnew_kiface.dir/dialogs/dialog_scripting_base.cpp.o CMakeFiles/pcbnew_kiface.dir/pcbnew_wrap.cxx.o CMakeFiles/pcbnew_kiface.dir/__/common/swig/wx_python_helpers.cpp.o CMakeFiles/pcbnew_kiface.dir/swig/pcbnew_action_plugins.cpp.o CMakeFiles/pcbnew_kiface.dir/swig/pcbnew_footprint_wizards.cpp.o CMakeFiles/pcbnew_kiface.dir/swig/pcbnew_scripting_helpers.cpp.o CMakeFiles/pcbnew_kiface.dir/swig/python_scripting.cpp.o -Wl,-rpath,"/build/kicad-5.0.0~rc1+dfsg1/debian/build/3d-viewer/3d_cache/sg:" ../3d-viewer/lib3d-viewer.a ../common/libpcbcommon.a router/libpnsrouter.a pcad2kicadpcb_plugin/libpcad2kicadpcb.a ../common/libcommon.a ../polygon/libpolygon.a ../bitmaps_png/libbitmaps.a ../common/libgal.a ../lib_dxf/liblib_dxf.a ../utils/idftools/libidf3.a -L/usr/lib/arm-linux-gnueabi -pthread -lwx_gtk2u_gl-3.0 -lwx_gtk2u_aui-3.0 -lwx_gtk2u_adv-3.0 -lwx_gtk2u_html-3.0 -lwx_gtk2u_core-3.0 -lwx_baseu_net-3.0 -lwx_baseu-3.0 -lwx_baseu_xml-3.0 -lwx_gtk2u_stc-3.0 github/libgithub_plugin.a -lpython2.7 -lrt ../3d-viewer/3d_cache/sg/libkicad_3dsg.so.2.0.0 -L/usr/lib/arm-linux-gnueabi -pthread -lwx_gtk2u_gl-3.0 -lwx_gtk2u_aui-3.0 -lwx_gtk2u_adv-3.0 -lwx_gtk2u_html-3.0 -lwx_gtk2u_core-3.0 -lwx_baseu_net-3.0 -lwx_baseu-3.0 -lwx_baseu_xml-3.0 -lwx_gtk2u_stc-3.0 ../common/libcommon.a ../common/libgal.a -lGLEW -lcairo -lpixman-1 -lGL -lGLU -lcurl -lssl -lcrypto
CMakeFiles/pcbnew_kiface.dir/pcbnew.cpp.o: In function `_GLOBAL__sub_I_pcbnew.cpp':
./debian/build/pcbnew/./pcbnew/pcbnew.cpp:312:(.text.startup+0x1ae): relocation truncated to fit: R_ARM_THM_CALL against symbol `FP_LIB_TABLE::FP_LIB_TABLE(FP_LIB_TABLE*)' defined in .text section in ../common/libpcbcommon.a(fp_lib_table.cpp.o)
collect2: error: ld returned 1 exit status

So would it be possible and desirable that the build environment is doing a smart job and skip the usage of the embedded assembler stuff if Boost >= 1.61 is found?
If not, could the environment get improved so KiCad can be successfully built with own assembler blobs also on mips, mipsel, powerpc and ppc64el?

I see no hard sense to cut down the platforms there KiCad can be built and used to only the typical Intel and ARM based platforms. Not that Chinese people mostly use Chromebooks that are mips based. They would benefit from KiCad.

[Tomasz Wlostowski (twlostow)]

We currently support x86 and x86_64 only.

Tom

[Wayne Stambaugh (stambaughw)]

We really need to come up with a better plan forward for v6. The idea
that we are only going to support x86 and x86_64 is short sighted with
the ever increasing power and availability of arm and arm 64 platforms.

On 3/14/2018 9:49 AM, Tomasz Wlostowski wrote:
> We currently support x86 and x86_64 only.
>
> Tom
>
> ** Changed in: kicad
> Importance: Undecided => Low
>

[Seth Hillbrand (sethh)]

Can we bump the minimum Boost version to 1.61 and drop our internal assembly blobs?

That will prevent use on Debian Jessie but that is oldstable already.

On windows, it looks like the pre-built boost for 1.61 goes back to msvc8.0-32bit, so that goes all the way back to XP.

Macs have libboost > 1.61 back to 10.9 (at least)

[Tomasz Wlostowski (twlostow)]

On 14/03/18 18:17, Wayne Stambaugh wrote:
> We really need to come up with a better plan forward for v6. The idea
> that we are only going to support x86 and x86_64 is short sighted with
> the ever increasing power and availability of arm and arm 64 platforms.
>
Hi Wayne,

I tested libcontext on x86, x86_64, ARM and ARM64. Maybe someone changed
some compiler flags for ARM targets so that it doesn't build anymore - I
have no idea for the moment.

The bug report is mostly (as the title suggests) about supporting
PowerPC and MIPS. I don't think we have any developer who has a MIPS/PPC
machine. Supporting exotic or prehistoric platforms is IMHO very outside
of our business of making EDA software.

Tom

[Tomasz Wlostowski (twlostow)]

@Seth, no, unless Boost devs guarantee to not change the API of boost::context again. This was the main reason for developing libcontext.

Tom

[tijuca (c-schoenert)]

If the outcome of this report is that only i386, amd64 and ARM is gonna be supported I'm fine with this. The problem with this own libcontext solution is like any other "own" solution. In a long term it's always better so convince the affected upstream project to do the modification on the source, otherwise it will get PITA over the time. But that's just my minds about this.

As Wayne noticed, the next 64bit platform are already there. So I think some decisions need to be reconsidered after the final release of KiCad 5.

Anyway, what about the build issues then on arm32 armvt5? Yes, this is a bit a historical platform and you won't do anything about it?

[Seth Hillbrand (sethh)]

@Tom, from reading the list messages it sounds like this was an ABI incompatibility (not API?) resulting from boost wanting MSVC build chain.

If that's the case, could we keep libcontext for MSW only? That would resolve the platform issue as the *nix builds could use libboost context. We keep the API, so there's no KiCad changes.

Am I missing a critical element here?

[Tomasz Wlostowski (twlostow)]

@Seth

There were multiple critical elements:
- the issue you mentioned was about building boost::context under mingw without having to get proprietary binaries for MASM/MASM64 (Microsoft Macro Assembler). This was fixed by Boost at some point in 2015.
- boost::context API is unstable - that is the definitions of jump_fcontext and make_fcontext change from time to time. We had to put a number of #ifdefs in KiCad coroutine library to accommodate for different versions of Boost.
- In the latest versions of Boost (at least 1.61 and newer), the fcontext api is no longer public (detail namespace). Therefore there's no guarantee of its stability. The now-official boost coroutine library is very complex and incomprehensible. Linking against a non-public API is not a good idea IMHO.
- Last but not least, since we moved to C++11, most features that used to be provided by Boost are now in the C++ standard library. I'd rather use C++11-features only in the longer run than rely on Boost.

All of the above forced me to develop libcontext, which is merely a two-file version of boost::context, including all original copyrights. I made it only for platforms I could test the library with.

@tijuca: See above. We support x86, x86_64, ARM and ARM64 for now (although I've never met any KiCad user running it on an ARM box). I've never seen anyone trying to run Kicad on a PPC/Mips machine. As you correctly pointed out, armv5-t is a sort of historical instruction set by today's standards, so there's no need to worry about it. For other exotic platforms like PPC, SPARC, DEC Alpha, MIPS, PDP-10, old IBM mainframe or Commodore 64, I guess if someone still manages to use such a machine, writing a context switch function should be trivial to him/her.

Tom
Tom

[tijuca (c-schoenert)]

@Tom

Thanks for this information. It should be a good idea to write about the reasons for using a own libcontext implementation and the problems with the existing Boost versions at least on kicad-pcb.o and finally some there in the release notes of KiCad 5 to prevent future questions about this all.

One outcome of the Boost development was definitely the speed up of future C++ standards. It's some years ago I've followed this all more active. Now there is C++17 alive for some time and the evolving hasn't stop and people working on C++20. I hope one day the problems you have can be solved by using standard functions of an ISO/IEC standard.

I will modify the build rules for future KiCad uploads to the Debian build infrastructure so the buildd's will only take care for arm64, i386, amd64 and armhf (armvt6 or armvt7).

[Wayne Stambaugh (stambaughw)]

On 3/14/2018 2:22 PM, Tomasz Wlostowski wrote:
> @Seth, no, unless Boost devs guarantee to not change the API of
> boost::context again. This was the main reason for developing
> libcontext.
>
> Tom
>

Was there a technical reason why we didn't use boost::coroutine instead
of writing our own coroutine object based on the boost::context code?
It appears that boost::coroutine hid some of the underlying changes in
boost::context over time. This may have save us some grief.

Wayne

[Seth Hillbrand (sethh)]

The 4 platforms listed are now supported.