Building a minimal Octave CLI only from source on Windows

First, a bit of context: I’m about to start a new project for a new customer that is about operating and interpreting results from a specifically designed DNA analysis machine they build. A requirement is having Octave embedded in the final solution (which is a Windows application) so they can write their scripts and work with the numbers without having to issue a new release of the software.

I initially tried with the MXE Octave Windows installer and it works in the sense that I can go with that for the development phase, however I will sooner or later need a “tailor-made” Windows version of it so it does not contain all the unneeded stuff included in the default package.

Since I don’t need anything about GUI, here are the configure arguments I think I need to use to get the very minimal working version:

--disable-docs 
--disable-java 
--without-bz2 
--without-hdf5 
--without-curl 
--without-qhull_r 
--without-magick 
--without-z 
--without-portaudio 
--without-sndfile 
--without-qt 
--without-qscintilla 
--without-opengl 
--without-framework-opengl 
--without-framework-carbon

So at this point, I’m currently running this compilation process in a Windows 10 virtual machine with MSYS2 installed on it and it is running (with very useful information I found on another thread here about CI tests on Windows, it is sooooooo slow but still) and I’m waiting for the result.
Has anyone tried this before? Is it realistic? Am I on the right track?

@jwe has a no-extras builder on his buildbot farm. AFAICT, that builder disables all optional features. It is configured with the following flags:

../configure --disable-docs --disable-fftw-threads --disable-java --disable-jit --disable-openmp --disable-readline --without-amd --without-arpack --without-bz2 --without-camd --without-ccolamd --without-cholmod --without-colamd --without-curl --without-cxsparse --without-fftw3 --without-fftw3f --without-fltk --without-fontconfig --without-framework-opengl --without-freetype --without-glpk --without-hdf5 --without-klu --without-magick --without-opengl --without-openssl --without-osmesa --without-portaudio --without-qhull --without-qrupdate --without-qscintilla --without-qt --without-sndfile --without-sundials-ida --without-sundials_ida --without-sundials_nvecserial --without-umfpack --without-x --without-z

I’m not sure if you’d really like to disable all features (e.g. readline is quite handy if you need an interactive shell).

Be aware that your Octave will still depend on the MSYS2 environment. So it won’t be very lightweight (if you also count the entire MSYS2 environment). But the base environment with just a few extra packages might suffice for your needs IIUC.

Spawning processes seems to be notably slower in the MSYS2 environment than on e.g. Linux. That slows down building considerably.
Using make with quite a high number of parallel jobs (2-4 times the number of processor threads) seems to somewhat mitigate it. Linking is still painfully slow though.
If you’ll need to re-build repeatedly, ccache might help cutting on build time.

I’d be interested to read what you’ll end up doing.

Thanks for the input, I’m not really interrested into disabling ALL features, actually I want to keep everything that is math related but strip down what is GUI, audio, video and so on (hence my configure option list).
I’m already using ccache indeed, it has been running for hours and hours now and I feel like it hangs… I don’t know what I am missing at this point, I probably will let it run for the night and see tomorrow what’s up. Here is my configure command that keeps running (largely inspired by your CI thread I quoted earlier)

 MSYS ~/octave/.build
$ ../configure CC="ccache x86_64-w64-mingw32-gcc" CXX="ccache x86_64-w64-mingw32-g++" F77="ccache x86_64-w64-mingw32-gfortran" PORTAUDIO_LDFLAGS="-L/mingw64/lib -L/usr/lib" --disable-docs --disable-java --without-bz2 --without-hdf5 --without-curl --without-qhull_r --without-magick --without-z --without-portaudio --without-sndfile --without-qt --without-qscintilla --without-opengl --without-framework-opengl --without-framework-carbon ac_cv_search_tputs=-ltermcap                                                                         configure: loading site script /etc/config.site
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a race-free mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether UID '197608' is supported by ustar format... yes
checking whether GID '197608' is supported by ustar format... yes
checking how to create a ustar tar archive... gnutar
checking whether make supports nested variables... (cached) yes
checking build system type... x86_64-pc-msys
checking host system type... x86_64-pc-msys
checking whether make supports the include directive... yes (GNU style)
checking for gcc... ccache x86_64-w64-mingw32-gcc

If it hangs at that point, this is probably caused by a buggy(?) ccache in MSYS2.
In my testing the version of ccache installed with pacman -S ccache hung intermittently.
They also package an older version of ccache that can be installed with pacman -S mingw-w64-x86_64-ccache. I ended up using that instead because it worked more reliably. (And uninstalled the other version with pacman -R ccache.)
See also the build instructions in the Wiki:
Building on Microsoft Windows - Octave

1 Like

Indeed using mingw-w64-x86_64-ccache instead of ccache gives a different result, I had issues with GCC not being able to produce executables or something but it turned out just being a path issue (for some reason mingw64 wasn’t part of it as I thought it was) and the configure part is now working :slight_smile:

  Build Octave Qt GUI:                  no
  JIT compiler for loops:               no
  Build Java interface:                 no
  Build static libraries:               no
  Build shared libraries:               yes
  Dynamic Linking API:                  dlopen
  Include support for GNU readline:     yes
  Use push parser in command line REPL: yes
  64-bit array dims and indexing:       yes
  64-bit BLAS array dims and indexing:  no
  OpenMP SMP multithreading:            yes
  Truncate intermediate FP results:     yes
  Build cross tools:                    no
  Build docs:                           no

Let’s go with the compilation process now… If all goes well I’ll probably try to make it static after that.

I’m a little bit surprised it will be using the dlopen API for you. It was using the LoadLibrary API for me. But maybe it was a dependency that enforced that?
I guess we’ll see how it works out for you.

Well… It ends up in error and it might be related to the dlopen API thing you mentioned, here is the compilation log:

../liboctave/util/oct-shlib.cc: In member function 'virtual void* octave::octave_dlopen_shlib::search(const string&, const name_mangler&)':
../liboctave/util/oct-shlib.cc:254:25: error: 'RTLD_DEFAULT' was not declared in this scope
  254 |       function = dlsym (RTLD_DEFAULT, sym_name.c_str ());
      |                         ^~~~~~~~~~~~
make[2]: *** [Makefile:25246: liboctave/util/libutil_la-oct-shlib.lo] Error 1

Not sure how to tackle this one now…

I’m also not sure why it ended up in this state.
To make sure it is not caused by something left-over from a previous attempt, maybe try to start over with a fresh checkout from the repository. Then run the bootstrap script and configure again.

Will it still think it can use dlopen?

Probably unrelated: Since you disabled portaudio, don’t override the portaudio flags.

I started over anyway since I now have a much more beefy VM to run my tests on, still don’t know what happened but the configure result is now correct with dynamic linking API set to LoadLibrary.

  Build Octave Qt GUI:                  no
  JIT compiler for loops:               no
  Build Java interface:                 no
  Build static libraries:               no
  Build shared libraries:               yes
  Dynamic Linking API:                  LoadLibrary
  Include support for GNU readline:     yes
  Use push parser in command line REPL: yes
  64-bit array dims and indexing:       yes
  64-bit BLAS array dims and indexing:  no
  OpenMP SMP multithreading:            yes
  Truncate intermediate FP results:     yes
  Build cross tools:                    no
  Build docs:                           no

Fingers crossed now.

EDIT:

Octave successfully built.  Now choose from the following:

   ./run-octave    - to run in place to test before installing
   make check      - to run the tests
   make install    - to install (PREFIX=/mingw64)

   HG ID for this build is "c6f9ff84dc84"

:partying_face:

Now let’s see it works though…

And it does not work, ./run-octave seems to do things for a couple of seconds and the return without error but nothing happen. Does it look like a known behavior/issue?

Yeah. Please have a look at the Wiki:
Building on Microsoft Windows - Octave

Edit:
In your case (without GUI), the last steps are probably slightly different from what I wrote in the Wiki. But the main points still hold:

Octave doesn’t run correctly from the MSYS2 shell due to issues with readline and backslash as a file separator. To be able to work with it anyway, install it with the following commands:

make install

Start Octave from a CMD shell (or with a batch script) with the following content (assuming MSYS2 was installed in its default location):

set PATH=C:\msys64\mingw64\bin;C:\msys64\usr\bin;%PATH%
set MSYSTEM=MINGW64
set TERM=cygwin
set GNUTERM=wxt
set GS=gs.exe
set PERL5SHELL=bash -l -c
octave-cli

Does that work for you?

Yes it works! Indeed the readline/backslash seem to be the issue, launching it from a .bat file is working perfectly. Thank you so much for helping me with this, I would never have made it without your help.

Now I need to figure out how to somehow package/deploy this on another machine and I have the feeling that this is a whole different story.

OK so just to get this straight, if I compile the whole thing all over again with the same configuration but with --disable-shared --enable-static it should result in a kind of “standalone” giant executable (or a few of them) that I can move onto another system without the need to install anything else, right?

IIUC, liboctinterp and liboctave will be built as static libraries with these configure switches. I’m not sure if this means that all of their dependencies will be linked in statically.

Even though all the dependencies are not statically linked inside those, IMO it would still be easier to redistribute. But I’m getting build errors that I don’t understand, hopefully it makes more sense for someone else:

C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: src/octave_cli-main-cli.o:C:\msys64\home\sycomore\octave\.build/../src/main-cli.cc:105: undefined reference to `__imp__Z7strsavePKc'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: src/octave_cli-main-cli.o:C:\msys64\home\sycomore\octave\.build/../src/main-cli.cc:63: undefined reference to `__imp__Z15liboctave_hg_idB5cxx11v'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: src/octave_cli-main-cli.o: in function `check_hg_versions':
C:\msys64\home\sycomore\octave\.build/../src/main-cli.cc:64: undefined reference to `__imp__Z18liboctinterp_hg_idB5cxx11v'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: src/octave_cli-main-cli.o: in function `wmain':
C:\msys64\home\sycomore\octave\.build/../src/main-cli.cc:116: undefined reference to `__imp_octave_block_async_signals'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:\msys64\home\sycomore\octave\.build/../src/main-cli.cc:118: undefined reference to `__imp__ZN6octave3sys3env16set_program_nameERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: src/octave_cli-main-cli.o: in function `octave::cli_application::cli_application(int, char**)':
C:\msys64\home\sycomore\octave\.build/../libinterp/octave.h:380: undefined reference to `__imp__ZN6octave11applicationC2EiPPc'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:\msys64\home\sycomore\octave\.build/../libinterp/octave.h:380: undefined reference to `__imp__ZTVN6octave15cli_applicationE'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: src/octave_cli-main-cli.o: in function `wmain':
C:\msys64\home\sycomore\octave\.build/../src/main-cli.cc:122: undefined reference to `__imp__ZN6octave15cli_application7executeEv'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: src/octave_cli-main-cli.o: in function `octave::cli_application::~cli_application()':
C:\msys64\home\sycomore\octave\.build/../libinterp/octave.h:389: undefined reference to `__imp__ZN6octave11applicationD2Ev'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:\msys64\home\sycomore\octave\.build/../libinterp/octave.h:389: undefined reference to `__imp__ZN6octave11applicationD2Ev'
collect2.exe: error: ld returned 1 exit status
make[2]: *** [Makefile:16094: src/octave-cli.exe] Error 1
make[2]: *** Waiting for unfinished jobs....
  GEN      libinterp/dldfcn/__delaunayn__.oct
/usr/bin/install: omitting directory 'libinterp/dldfcn/.libs/'
make[2]: *** [Makefile:31243: libinterp/dldfcn/__delaunayn__.oct] Error 1
make[2]: Leaving directory '/home/sycomore/octave/.build'
make[1]: *** [Makefile:28007: all-recursive] Error 1
make[1]: Leaving directory '/home/sycomore/octave/.build'
make: *** [Makefile:11433: all] Error 2

There are probably still issues with visibility on the current default branch.
Do you still see those linker errors if you configure with --disable-lib-visibility-flags?

Since this change

http://hg.savannah.gnu.org/hgweb/octave/rev/22ee68edcf3a

It is not possible to build Octave without dynamic linking of .oct files. Those files are DLLs, so building the libraries that they depend on (liboctinterp, liboctave) as static libraries is probably going to cause some trouble. Even if you make it work, I expect that each of the .oct files will be quite large.

To distribute a functional copy of Octave will require distributing many files (M-files, configuration and startup files, help files, etc.) I don’t think you can reduce it to one single octave.exe file. Octave is already relocatable. The installer that mxe-octave builds contains everything you need to install Octave. So with that distributing is easy.

What is it you really want to avoid by not using the installer?

You mentioned embedding Octave in another application. Are you just running Octave from that application or are they linked together such that Octave is a required part in order for that other application to function? Will that other application be distributed under terms that are compatible with the GPL?

I moved this topic to the help category since it seems to be a request for help rather than a discussion about Octave development.

1 Like

I guess you’re right about the .oct files since my compilation process always end up with make[2]: *** [Makefile:31243: libinterp/dldfcn/__delaunayn__.oct] Error 1

My goal here is simply to have Octave CLI in a subfolder of the application and access it through the command line inside the application so it is simply used an interpreter. This is also to avoid having to install external programs that may already be there in another version (in a lab it wouldn’t be a surprise if Octave was already installed on the machine) and to ease the distribution process by having everything the application needs directly in it.

I know mxe-octave and I went that road first, but it contains A LOT of things I don’t need/want as it is compiled with full support for basically everything it can support as well as a set of pre-installed packages. Even though the required disk space for a modern app does not really matter nowadays compared to what it was a decade ago, it still matters for the customer hence the need of a working, almost naked, Octave CLI interpreter that we have.

Now about the elephant in the room, I am not trying to violate the license terms by any mean. The application itself will indeed be commercial so it won’t be FOSS but no modification or linking (even dynamic) whatsoever is planned. The interaction between the apps will be achieved through a CLI wrapper and only this way. In addition, an explicit mention will be made to GNU Octave during the installation process and in the software license and documentation. All links will be provided to the site of Octave as well as its license exactly as it should be done for any commercial software which uses GPL software. We’re definitely not trying to hide or steal anything here.

Using mxe-octave it should not be difficult to eliminate dependencies that you don’t need. You can modify the options used to configure Octave or the dependencies available by changing one of the src/{default,stable,release}-octave.mk files and you can eliminate extra packages by editing the OCTAVE_FORGE_PACKAGES list in the Makefile.in file or change the contents of the installer as you wish.