Buildbot having trouble building MXE Octave

The cross-builds for Windows are failing now.
The updated binutils version might be incompatible with the toolchain built by MXE Octave.

I guess the underlying issue is that LD_LIBRARY_PATH points to the MXE environment but PATH points to the tools on the build system.
We should probably get that sorted out (either by consistently using tools and libraries from the build system or those that MXE Octave just built).

For the time being, we should probably revert to the older version of binutils.

I reverted to the previous binutils version here:
mxe-octave: 768e44dcd00c

And followed up with what was probably fixing a typo here:
mxe-octave: 00a24f9b9cae

Let’s see if this will make the buildbots happy again.

There is already a comment in hg-octave-dist.mk that explains why it is difficult to build with our own toolchain.
When I tried to use our toolchain instead, the build failed on creating the images for the manual. We’d need to build (at least) Qt as part of the build toolchain to fix that IIUC.

Not sure if it is related either.
IIUC, libtool will use the ld it found during configure (by calling $CC -print-progname=ld?).
You could check the libtool script that is created by configure which that is. It’s set in line 310 of that script for me:

# The linker used to build libraries.
LD="/usr/bin/ld -m elf_x86_64"

I’m not sure if I correctly follow what LT_PATH_LD is doing. But it might be possible to override whatever it would select by setting lt_cv_path_LD=/usr/bin/ld.lld when configuring…

I started another topic

(with some extra details).

I hope it is ok to resurrect this old topic.

The buildbots are having issues for the MXE builders, e.g.:
http://buildbot.octave.org:8010/#/builders/14/builds/1423/steps/7/logs/stdio

It looks like it’s having trouble creating the tarball. Is lzip installed on that worker?

Also, the native MXE builds fail to compile gcc:
http://buildbot.octave.org:8010/#/builders/20/builds/1186/steps/7/logs/stdio

I’m not sure if I correctly understand the portion of the error message shown in the log of the web interface. Maybe, the version of ICU that is found on the build system is incompatible with the GCC that we are trying to build?

(sorry if I unduly hijacked this thread)

Today, May 2 '22, I get this with a fresh mxe-octave clone:

[philip@localhost mxe_64b_20220205H]$ make nsis-installer
[build]    nsis

Failed to build package nsis!
------------------------------------------------------------
                 from Source/exehead/bgbg.c:23:
Source/exehead/api.h:68:3: warning: 'stdcall' attribute ignored [-Wattributes]
   68 |   int (NSISCALL *ExecuteCodeSegment)(int, HWND);
      |   ^~~
Source/exehead/api.h:69:3: warning: 'stdcall' attribute ignored [-Wattributes]
   69 |   void (NSISCALL *validate_filename)(LPTSTR);
      |   ^~~~
Source/exehead/api.h:70:3: warning: 'stdcall' attribute ignored [-Wattributes]
   70 |   int (NSISCALL *RegisterPluginCallback)(HMODULE, NSISPLUGINCALLBACK); // returns 0 on success, 1 if already registered and < 0 on errors
      |   ^~~
In file included from Source/exehead/bgbg.c:24:
Source/exehead/ui.h:25:1: warning: 'stdcall' attribute ignored [-Wattributes]
   25 | extern int NSISCALL ui_doinstall(void);
      | ^~~~~
Source/exehead/ui.h:26:1: warning: 'stdcall' attribute ignored [-Wattributes]
   26 | void NSISCALL update_status_text(int strtab, const TCHAR *text2);
      | ^~~~
In file included from Source/exehead/bgbg.c:25:
Source/exehead/util.h:24:10: fatal error: shlobj.h: No such file or directory
   24 | #include <shlobj.h>
      |          ^~~~~~~~~~
compilation terminated.
scons: *** [build/urelease/stub_bzip2-amd64-unicode/bgbg.o] Error 1
scons: building terminated because of errors.
make[1]: *** [/home/philip/devel/octdev/mxe/mxe_64b_20220205H/Makefile:981: build-only-nsis] Error 2
make[1]: Leaving directory '/home/philip/devel/octdev/mxe/mxe_64b_20220205H'
real    0m2.587s
user    0m2.154s
sys     0m0.507s
------------------------------------------------------------
[log]      /home/philip/devel/octdev/mxe/mxe_64b_20220205H/log/nsis

make: *** [Makefile:983: /home/philip/devel/octdev/mxe/mxe_64b_20220205H/installed-packages/nsis] Error 1

Mageia-8 64 bit.
An older mxe-octave build tree (kept up-to-date until last week) still works but yields a bus error at the end while generating the installer .exe. I suspected a zlib update from Mageia but reverting to an older zlib didn’t solve the issue.

Should I enter a bug report for this? Or does anyone ahve suggestions on how to investigate further?

I rearranged the disk layout on my buildbot systems and decided to reinstall the OS from scratch to try to make the two systems more nearly identical. I think the problems are just due to missing packages, but whatever the cause, I’ll try to fix within a few days.

1 Like

The build-gcc target for the mxe-native-all-on-debian build is failing with

Making all in po
make[7]: Entering directory '/scratch/buildbot/workers/jwe-debian-x86_64-5/mxe-native-all-on-debian/src/tmp-build-gcc/gcc-11.3.0.build/x86_64-pc-linux-gnu/libstdc++-v3/po'
msgfmt -o de.mo /scratch/buildbot/workers/jwe-debian-x86_64-5/mxe-native-all-on-debian/src/tmp-build-gcc/gcc-11.3.0/libstdc++-v3/po/de.po
msgfmt -o fr.mo /scratch/buildbot/workers/jwe-debian-x86_64-5/mxe-native-all-on-debian/src/tmp-build-gcc/gcc-11.3.0/libstdc++-v3/po/fr.po
msgfmt: /scratch/buildbot/workers/jwe-debian-x86_64-5/mxe-native-all-on-debian/src/tmp-build-gcc/gcc-11.3.0.build/x86_64-pc-linux-gnu/libstdc++-v3/src/.libs/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /lib/x86_64-linux-gnu/libicuuc.so.71)
make[7]: *** [Makefile:556: de.mo] Error 1
make[7]: *** Waiting for unfinished jobs....

I’m not sure why that is happening. Here’s a full build log:
build-gcc.gz (96.8 KB)

Trying to read the log is pretty confusing. Are multiple configure scripts running in parallel? Could that be the cause of the issue? Does invoking make with JOBS=1 lead to a different result?

Other than that: IIUC, msgfmt is a program from the gettext package. At that point (before the GCC compilers are built), the version of msgfmt that is installed on the build system is used. For some reason, it tries to load the just compiled (stage 1) libstdc++.so from GCC’s build tree. It should probably load that library in the version installed on the build system instead. Not sure why it doesn’t.
Are we setting some environment variables that confuse the build process?

The MXE buildbots that target Windows are now failing for some Python packages. I believe we saw that error before. IIRC, it was fixed by installing sudo apt-get install libffi-dev on the build system.

Edit: See bug #61024, [mxe-octave] build-markupsafe fails to build if build-python3 is built without libffi

When the msgfmt command fails, LD_LIBRARY_PATH is set to

${GCC_BUILD_DIR}/x86_64-pc-linux-gnu/libstdc++-v3/src/.libs:${GCC_BUILD_DIR}/x86_64-pc-linux-gnu/libvtv/.libs:${GCC_BUILD_DIR}/x86_64-pc-linux-gnu/libssp/.libs:${GCC_BUILD_DIR}/x86_64-pc-linux-gnu/libitm/.libs:${GCC_BUILD_DIR}/x86_64-pc-linux-gnu/libatomic/.libs:${GCC_BUILD_DIR}/./gcc:${GCC_BUILD_DIR}/./prev-gcc

I think that is done by the GCC build system. I’m not sure why this wasn’t a problem on my previous system installation. Hmm.

Thanks, I installed libffi-dev on my buildbot systems.

1 Like

Maybe, the msgfmt that it is using now was built with a newer version of libstdc++ than the one that is built at that point? Or these early stage libraries don’t contain all symbols yet? But then why did it work before?

Anyway, IIUC GCC’s build system never builds its own version of msgfmt. So, it should be safe to always load the libraries installed on the build system for those commands. Does the following patch make any difference?
build-gcc-1-msgfmt.diff (870 Bytes)
(Not sure if we’d need to change both files or whether one would be enough…)

Edit: Did you install the same version of the OS as the one that was installed on that worker before? If this is an updated version, maybe the older version packaged libraries that linked against a version of libstdc++.so with a different version suffix. Thus, the libraries might not have clashed like they do now…

The buildbots that target Windows went a few steps further now. But they failed for ICU4C:
Buildbot: builder w32-on-debian build 1425 (octave.org)

# build some native tools
mkdir '/scratch/buildbot/workers/jwe-debian-x86_64-5/w32-on-debian/src/tmp-icu4c/icu.native' && cd '/scratch/buildbot/workers/jwe-debian-x86_64-5/w32-on-debian/src/tmp-icu4c/icu.native' && '/scratch/buildbot/workers/jwe-debian-x86_64-5/w32-on-debian/src/tmp-icu4c/icu/source/configure'
checking for ICU version numbers... release 71.1, library 71.1, unicode version 14.0
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking whether to build debug libraries... no
checking whether to build release libraries... yes
checking for clang... clang
checking whether the C compiler works... no
configure: error: in `/scratch/buildbot/workers/jwe-debian-x86_64-5/w32-on-debian/src/tmp-icu4c/icu.native':
configure: error: C compiler cannot create executables
See `config.log' for more details

It looks like configure selects clang as the C compiler. But then proceeds to fail creating executables with it. I pushed the following change to make it use the GCC compilers instead:
mxe-octave: 8b23f76955c6

I hope that’ll fix this error.

It looks like it almost completed the build attempt this time. But p7zip was missing when creating the package.
Buildbot (octave.org)

generating 7z file...
generating installer script...
/bin/bash: line 1: p7zip: command not found
make: *** [/scratch/buildbot/workers/jwe-debian-x86_64-5/w32-stable-on-debian/src/binary-dist-rules.mk:306: 7z-dist] Error 127
make: *** Waiting for unfinished jobs....

Is there anywhere where we document which programs are required to be installed on the build system for MXE Octave?

Not yet, but we seem to be working on it now. :slight_smile:

Once we have it working again, I’ll compile a list and describe as accurately as possible what I’ve done to set up these systems. As always, thanks for your help.

2 Likes

The MXE Windows buildbots seem to be happy now.

The MXE Linux buildbot that uses the system GCC now stops while compiling SuiteSparse with this error:

In file included from ../Source/slip_internal.h:106,
                 from ../Source/SLIP_LU_solve.c:51:
../Include/SLIP_LU.h:154:10: fatal error: mpfr.h: No such file or directory
  154 | #include <mpfr.h>
      |          ^~~~~~~~
compilation terminated.

Currently, we compile GNU MPFR as a dependency of build-gcc. For that configuration (using the system GCC), that dependency isn’t built.
Should we add gcc-mpfr as a dependency to SuiteSparse? I’m not sure if that could cause issues if the version of MPFR in MXE differs from the version that was used for building the system GCC…
Alternatively, we could require that MPFR (and probably also GMP, ISL, and MPC) are installed on the build system if the system GCC should be used.

The latter is probably the safer option.

I pushed the patch from comment #20 to the release branch of the MXE Octave repository in the hope that it will help with the library clash when calling msgfmt:
mxe-octave: 5f64fb928091

Edit: GCC seems to be building now.
However, a similar error is happening when calling out to emacs while building build-gettext:
Buildbot (octave.org)

Failed to build package build-gettext!
------------------------------------------------------------
  esac; \
  test -d "$am__dir" || /usr/bin/mkdir -p "$am__dir" || exit 1; \
  emacs --batch \
      \
    $am__subdir_includes -L . -L /scratch/buildbot/workers/jwe-debian-x86_64-5/mxe-native-all-on-debian/src/tmp-build-gettext/gettext-0.21/gettext-tools/emacs \
    --eval '(if (boundp (quote byte-compile-dest-file-function)) (setq byte-compile-dest-file-function (lambda (_) "po-compat.elc")) (defun byte-compile-dest-file (_) "po-compat.elc") )' \
    -f batch-byte-compile '/scratch/buildbot/workers/jwe-debian-x86_64-5/mxe-native-all-on-debian/src/tmp-build-gettext/gettext-0.21/gettext-tools/emacs/po-compat.el'; \
else :; fi
emacs: /scratch/buildbot/workers/jwe-debian-x86_64-5/mxe-native-all-on-debian/src/usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /lib/x86_64-linux-gnu/libicuuc.so.71)
emacs: /scratch/buildbot/workers/jwe-debian-x86_64-5/mxe-native-all-on-debian/src/usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /lib/x86_64-linux-gnu/libicuuc.so.71)
emacs: /scratch/buildbot/workers/jwe-debian-x86_64-5/mxe-native-all-on-debian/src/usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /lib/x86_64-linux-gnu/libicuuc.so.71)

I guess we don’t need the emacs bindings for the gettext that we are using as build tool. Maybe, there are flags to deactivate building those…

Edit 2: Pushed a patch here:
mxe-octave: d18a5545df0d

Looking at my notes for the system reconfiguration, most of the items are specific to my local setup. The important parts for setting up a buildbot worker on a Debian system are:

Install Debian. I chose the testing distribution. At the point when the installation presents a menu of base system components to install, I chose ssh server but no desktop or any other options.

Install other packages

apt build-dep octave
apt install \
  buildbot-worker \
  ccache \
  git \
  libffi-dev \
  libmpfr-dev \
  libxi-dev \
  lzip \
  mercurial \
  p7zip \
  texlive-extra-utils \
  xvfb

Installing the build dependencies for the most recent Octave release will provide most of the necessary tools.

Some of the remaining packages are only needed if the buildbot worker will be building Octave for Windows or a “native” build of Octave and all dependencies. Maybe some of those tools should be built by mxe-octave itself, but requiring them to be installed on the build system works for now. (See also this discussion: Drop “native” builds from mxe-octave?)

Finally, the buildbot worker needs to be configured and started. Then to do anything, the actions for that worker need to be added to the master buildbot configuration.

1 Like

Thanks for summing this up. That will hopefully help in the future. Maybe when we set up the ARM builder? (Is this still planned?)

Is there some place where this can be documented other than in this forum? Maybe somewhere in the Wiki?
I’m afraid this will get lost with time if it is just in a “random comment in some thread on the forum”…