Buildbot having trouble building MXE Octave

@jwe’s buildbots seem to have trouble building some jobs for MXE Octave.

The mxe-native-all-on-debian builder fails on this command:

PKG_CONFIG_PATH='' LD_LIBRARY_PATH='/scratch/buildbot/workers/jwe-debian-x86_64-5/mxe-native-all-on-debian/src/usr/lib:/scratch/buildbot/workers/jwe-debian-x86_64-5/mxe-native-all-on-debian/src/usr/lib64' PATH='/var/lib/buildbot/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games' ../configure --prefix=/scratch/buildbot/workers/jwe-debian-x86_64-5/mxe-native-all-on-debian/src/usr && \

with this error:

checking whether the C compiler works... no
configure: error: in `/scratch/buildbot/workers/jwe-debian-x86_64-5/mxe-native-all-on-debian/src/octave-hg-repo/.build':
configure: error: C compiler cannot create executables
See `config.log' for more details
make: *** [/scratch/buildbot/workers/jwe-debian-x86_64-5/mxe-native-all-on-debian/src/hg-octave-dist.mk:34: hg-octave-dist] Error 77

IIUC, that builder should be using the compiler that was built as part of the build process. The PATH variable pointing to paths on the worker (instead of paths inside the MXE Octave build tree) looks fishy to me. Maybe the compiler installed on the build system is directed to use the standard libraries that were built for MXE’s compiler?
But I might be misunderstanding what is supposed to be happening.

The w*-release and w*-stable builders are failing when they try to compile qtbase. I’ve no idea why though. ISTR that we saw these kinds of errors with gcc 11 before. But we adopted a patch that is supposed to fix them (mxe-octave: 09d3533acacf src/qtbase-2-gcc11.patch).
Is that patch not used for some reason?

After looking a bit more closely, I realized that that patch was only on the default branch.
I grafted these changes to the release branch here:
mxe-octave: 6d67c9994aed
mxe-octave: 72f9d9dab23f

Were the workers updated recently and the installed gcc is now version 11?

Still not sure why native-all is failing.

I did upgrade recently. I’ll see about getting more info on the mxe-native-all failure.

1 Like

The config.log file for the failed build of Octave from the hg archive showed

configure:7510: checking whether the C compiler works
configure:7532: gcc    conftest.c  >&5
/usr/bin/ld: /scratch/buildbot/workers/jwe-debian-x86_64-5/mxe-native-all-on-debian/src/usr/lib/libctf.so.0: version `LIBCTF_1.1' not found (required by /usr/bin/ld)
collect2: error: ld returned 1 exit status
configure:7536: $? = 1
configure:7576: result: no
configure: failed program was:
| /* confdefs.h */
| #define PACKAGE_NAME "GNU Octave"
| #define PACKAGE_TARNAME "octave"
| #define PACKAGE_VERSION "7.0.0"
| #define PACKAGE_STRING "GNU Octave 7.0.0"
| #define PACKAGE_BUGREPORT "https://octave.org/bugs.html"
| #define PACKAGE_URL "https://www.gnu.org/software/octave/"
| #define PACKAGE "octave"
| #define VERSION "7.0.0"
| #define OCTAVE_SOURCE 1
| /* end confdefs.h.  */
| 
| int
| main (void)
| {
| 
|   ;
|   return 0;
| }
configure:7581: error: in `/scratch/buildbot/workers/jwe-debian-x86_64-5/mxe-native-all-on-debian/src/octave-hg-repo/.build':
configure:7583: error: C compiler cannot create executables
See `config.log' for more details

Info in the bug report

https://sourceware.org/bugzilla/show_bug.cgi?id=27250

gave me a clue that the problem might be fixed by a more recent version of binutils.

Updating the mxe-octave version of binutils to 2.37 seems to have fixed the problem.

I wonder why it is using /usr/bin/ld for linking. Shouldn’t it be using /scratch/buildbot/workers/jwe-debian-x86_64-5/mxe-native-all-on-debian/src/usr/bin/ld instead?

I am not sure if that is related, but I am trying to use lld linker (ld.lld) for clang builds.
So I set LD=ld.lld and LDFLAGS="-fuse-ld=lld -Wl,–threads=4". Configure passes OK,
yet during compile I see:

 CXXLD    libinterp/dldfcn/__glpk__.la
clang-13: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
clang-13: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]clang-13: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
clang-13: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]

clang-13: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
clang-13: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
config.status: creating build-aux/subst-script-vals.sh-tmp
/usr/bin/ld: unrecognized option '--threads=4'
/usr/bin/ld: use the --help option for usage information
clang-13: error: linker command failed with exit code 1 (use -v to see invocation)
/usr/bin/ld: unrecognized option '--threads=4'

Why /usr/bin/ld is invoked?

p.s. I can set /usr/bin/ld → /usr/bin/ld.lld and then it finishes compile

The log file http://buildbot.octave.org:8010/#/builders/20/builds/1045/steps/7/logs/stdio shows

PATH=/var/lib/buildbot/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

That doesn’t include the directory where we are installing build tools, so the version of GCC built earlier in this compile step won’t be found.

It makes sense to me to at least begin building the build tools with the compiler that is installed on the system since we can’t do otherwise. But after we build our own GCC, we should be using that (at least) for the remaining packages, right? So I guess we need some changes to the build rules for the mxe-native-all configuration. Maybe even more changes if we want to bootstrap using the system compiler and then rebuild everything including the build tools with the new compiler. Either way, fixing this problem is a low priority for me.

The cross-builds for Windows are failing now.
The updated binutils version might be incompatible with the toolchain built by MXE Octave.

I guess the underlying issue is that LD_LIBRARY_PATH points to the MXE environment but PATH points to the tools on the build system.
We should probably get that sorted out (either by consistently using tools and libraries from the build system or those that MXE Octave just built).

For the time being, we should probably revert to the older version of binutils.

I reverted to the previous binutils version here:
mxe-octave: 768e44dcd00c

And followed up with what was probably fixing a typo here:
mxe-octave: 00a24f9b9cae

Let’s see if this will make the buildbots happy again.

There is already a comment in hg-octave-dist.mk that explains why it is difficult to build with our own toolchain.
When I tried to use our toolchain instead, the build failed on creating the images for the manual. We’d need to build (at least) Qt as part of the build toolchain to fix that IIUC.

Not sure if it is related either.
IIUC, libtool will use the ld it found during configure (by calling $CC -print-progname=ld?).
You could check the libtool script that is created by configure which that is. It’s set in line 310 of that script for me:

# The linker used to build libraries.
LD="/usr/bin/ld -m elf_x86_64"

I’m not sure if I correctly follow what LT_PATH_LD is doing. But it might be possible to override whatever it would select by setting lt_cv_path_LD=/usr/bin/ld.lld when configuring…

I started another topic

(with some extra details).

I hope it is ok to resurrect this old topic.

The buildbots are having issues for the MXE builders, e.g.:
http://buildbot.octave.org:8010/#/builders/14/builds/1423/steps/7/logs/stdio

It looks like it’s having trouble creating the tarball. Is lzip installed on that worker?

Also, the native MXE builds fail to compile gcc:
http://buildbot.octave.org:8010/#/builders/20/builds/1186/steps/7/logs/stdio

I’m not sure if I correctly understand the portion of the error message shown in the log of the web interface. Maybe, the version of ICU that is found on the build system is incompatible with the GCC that we are trying to build?

(sorry if I unduly hijacked this thread)

Today, May 2 '22, I get this with a fresh mxe-octave clone:

[philip@localhost mxe_64b_20220205H]$ make nsis-installer
[build]    nsis

Failed to build package nsis!
------------------------------------------------------------
                 from Source/exehead/bgbg.c:23:
Source/exehead/api.h:68:3: warning: 'stdcall' attribute ignored [-Wattributes]
   68 |   int (NSISCALL *ExecuteCodeSegment)(int, HWND);
      |   ^~~
Source/exehead/api.h:69:3: warning: 'stdcall' attribute ignored [-Wattributes]
   69 |   void (NSISCALL *validate_filename)(LPTSTR);
      |   ^~~~
Source/exehead/api.h:70:3: warning: 'stdcall' attribute ignored [-Wattributes]
   70 |   int (NSISCALL *RegisterPluginCallback)(HMODULE, NSISPLUGINCALLBACK); // returns 0 on success, 1 if already registered and < 0 on errors
      |   ^~~
In file included from Source/exehead/bgbg.c:24:
Source/exehead/ui.h:25:1: warning: 'stdcall' attribute ignored [-Wattributes]
   25 | extern int NSISCALL ui_doinstall(void);
      | ^~~~~
Source/exehead/ui.h:26:1: warning: 'stdcall' attribute ignored [-Wattributes]
   26 | void NSISCALL update_status_text(int strtab, const TCHAR *text2);
      | ^~~~
In file included from Source/exehead/bgbg.c:25:
Source/exehead/util.h:24:10: fatal error: shlobj.h: No such file or directory
   24 | #include <shlobj.h>
      |          ^~~~~~~~~~
compilation terminated.
scons: *** [build/urelease/stub_bzip2-amd64-unicode/bgbg.o] Error 1
scons: building terminated because of errors.
make[1]: *** [/home/philip/devel/octdev/mxe/mxe_64b_20220205H/Makefile:981: build-only-nsis] Error 2
make[1]: Leaving directory '/home/philip/devel/octdev/mxe/mxe_64b_20220205H'
real    0m2.587s
user    0m2.154s
sys     0m0.507s
------------------------------------------------------------
[log]      /home/philip/devel/octdev/mxe/mxe_64b_20220205H/log/nsis

make: *** [Makefile:983: /home/philip/devel/octdev/mxe/mxe_64b_20220205H/installed-packages/nsis] Error 1

Mageia-8 64 bit.
An older mxe-octave build tree (kept up-to-date until last week) still works but yields a bus error at the end while generating the installer .exe. I suspected a zlib update from Mageia but reverting to an older zlib didn’t solve the issue.

Should I enter a bug report for this? Or does anyone ahve suggestions on how to investigate further?

I rearranged the disk layout on my buildbot systems and decided to reinstall the OS from scratch to try to make the two systems more nearly identical. I think the problems are just due to missing packages, but whatever the cause, I’ll try to fix within a few days.

1 Like

The build-gcc target for the mxe-native-all-on-debian build is failing with

Making all in po
make[7]: Entering directory '/scratch/buildbot/workers/jwe-debian-x86_64-5/mxe-native-all-on-debian/src/tmp-build-gcc/gcc-11.3.0.build/x86_64-pc-linux-gnu/libstdc++-v3/po'
msgfmt -o de.mo /scratch/buildbot/workers/jwe-debian-x86_64-5/mxe-native-all-on-debian/src/tmp-build-gcc/gcc-11.3.0/libstdc++-v3/po/de.po
msgfmt -o fr.mo /scratch/buildbot/workers/jwe-debian-x86_64-5/mxe-native-all-on-debian/src/tmp-build-gcc/gcc-11.3.0/libstdc++-v3/po/fr.po
msgfmt: /scratch/buildbot/workers/jwe-debian-x86_64-5/mxe-native-all-on-debian/src/tmp-build-gcc/gcc-11.3.0.build/x86_64-pc-linux-gnu/libstdc++-v3/src/.libs/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /lib/x86_64-linux-gnu/libicuuc.so.71)
make[7]: *** [Makefile:556: de.mo] Error 1
make[7]: *** Waiting for unfinished jobs....

I’m not sure why that is happening. Here’s a full build log:
build-gcc.gz (96.8 KB)

Trying to read the log is pretty confusing. Are multiple configure scripts running in parallel? Could that be the cause of the issue? Does invoking make with JOBS=1 lead to a different result?

Other than that: IIUC, msgfmt is a program from the gettext package. At that point (before the GCC compilers are built), the version of msgfmt that is installed on the build system is used. For some reason, it tries to load the just compiled (stage 1) libstdc++.so from GCC’s build tree. It should probably load that library in the version installed on the build system instead. Not sure why it doesn’t.
Are we setting some environment variables that confuse the build process?

The MXE buildbots that target Windows are now failing for some Python packages. I believe we saw that error before. IIRC, it was fixed by installing sudo apt-get install libffi-dev on the build system.

Edit: See bug #61024, [mxe-octave] build-markupsafe fails to build if build-python3 is built without libffi

When the msgfmt command fails, LD_LIBRARY_PATH is set to

${GCC_BUILD_DIR}/x86_64-pc-linux-gnu/libstdc++-v3/src/.libs:${GCC_BUILD_DIR}/x86_64-pc-linux-gnu/libvtv/.libs:${GCC_BUILD_DIR}/x86_64-pc-linux-gnu/libssp/.libs:${GCC_BUILD_DIR}/x86_64-pc-linux-gnu/libitm/.libs:${GCC_BUILD_DIR}/x86_64-pc-linux-gnu/libatomic/.libs:${GCC_BUILD_DIR}/./gcc:${GCC_BUILD_DIR}/./prev-gcc

I think that is done by the GCC build system. I’m not sure why this wasn’t a problem on my previous system installation. Hmm.

Thanks, I installed libffi-dev on my buildbot systems.

1 Like

Maybe, the msgfmt that it is using now was built with a newer version of libstdc++ than the one that is built at that point? Or these early stage libraries don’t contain all symbols yet? But then why did it work before?

Anyway, IIUC GCC’s build system never builds its own version of msgfmt. So, it should be safe to always load the libraries installed on the build system for those commands. Does the following patch make any difference?
build-gcc-1-msgfmt.diff (870 Bytes)
(Not sure if we’d need to change both files or whether one would be enough…)

Edit: Did you install the same version of the OS as the one that was installed on that worker before? If this is an updated version, maybe the older version packaged libraries that linked against a version of libstdc++.so with a different version suffix. Thus, the libraries might not have clashed like they do now…