Supporting mexext?

moving a recent post from mailing list to this forum, as suggested
https://lists.gnu.org/archive/html/help-octave/2022-04/msg00014.html


Dear group,

I have been writing various toolboxes for octave and matlab - both as part of my research and hobby. I especially care about portability of my toolboxes, and often times spent quite a bit effort to ensure they can run across matlab and octave, as well as various OSes.

One challenge I am facing is to deploy mex-based toolboxes. For example, when deploying my ZMat toolbox (GitHub - fangq/zmat: ZMat: A portable C-library and MATLAB toolbox for zlib/gzip/lzma/lz4/lz4hc data compression), I have provided precompiled mex files for MATLAB users so that the toolbox can be ready-to-use across various OSes. These precompiled files are stored by OS-sensitive extensions (mexext) and can be conveniently placed under the same folder, see

unfortunately, I am not aware of a way to do the same for Octave, and had to require users to manually add path to the .mex files built for each platform

I am wondering if there is way to organize or name these files so that they can be easily aggregated to create easily to deploy packages?

Another issue related to deploying these precompiled binaries is the .so file. I recognize that the mex files created by MATLAB’s mex command links to a libmex.so library that is not version dependent, however, the mex file produced by mkoctfile -mex links to specific version of liboctinterp.so.x. When the octave version on the user’s system is different from the one at compile time, the mex file throws an error. Although I can create symbolic links to the compile-time liboctinterp.so.x, it requires sudo and can not be automated.

I wonder if there is any workaround to this?

thanks

Qianqian

Dear Qianqian,

Thank you for your detailed question. The distribution of pre-compiled mex-files is indeed more complicated in Octave, than in Matlab. For Matlab there are basically only three versions for each release (macOS, MS Windows, and Linux) to regard. Octave is more heterogeneous and due to its open-source nature integrates “better” to the existing platform. Basically this means, two Octave installations, even on macOS are not necessarily the same. On macOS you can install Octave via Homebrew, Fink, dmg-file, etc.

On the positive side, Octave has on (almost?) all platforms a compiler builtin and ready to use. And the Octave package format makes seamless use of this circumstance, to the benefit of both package maintainers (who get things simply compiled for the respective platform specifying a Makefile) and users (who in the absence of a bug, do not notice anything at all).

To get things agnostic of Octave and Matlab, I would recommend the approach from another project (SeDuMi)

The following approach:

  • For Octave provide a user friendly install.m script to invoke the mex (a.k.a mkoctfile)-compiler. For pure .c/.cc/.f sources this is pretty straigt forward, if external libraries have to be linked to your code, things get a little more complicated, but still not impossible.
  • For Matlab compiling on the user end is a pain, provide three pre-compiled mex-files.

I hope this helps,

Kai

@siko1056, thank you for your detailed comments - I understood that a developer can write a .m script to make the compilation less painful, which I also took such approach by providing this script

however, I found it nearly impossible to make such script foolproof across platforms - the level of pain is especially high on Windows, which I am sure if anyone had provided such support, you understand what I meant … to disseminate toolboxes to a large user community (with majority of them using Windows and Mac), letting users to take the task of compiling your mex file is one efficient path to scare away 90% of your users – or being bombarded by questions in mailing list …

providing precompiled binaries with all dependencies statically linked is so far the least painful way to disseminate matlab/octave toolboxes to a broader audience (I am not referring to zmat, but other grant-funded projects) from my past experience. I know from a Linux packaging perspective, this breaks many best-practice guides, but in reality, I still don’t see anything more effective.

essentially, to support this feature – allow storing multiple mex files in one directory – is not complicated for Octave - the mex/oct file loader just make one attempt to look for ['filename', '.', mexext] before loading filename.mex (given that mexext.m is added first - which I am happy to contribute).

Perusing the files and structure of your package, I saw that it already has a subfolder src with a Makefile that is used to compile those .mex file libraries. Octave’s pkg command already checks for that file in that location and runs make.

I haven’t checked it in any detail. But it might be that just adding some files (DESCRIPTION, PKG_ADD, PKG_DEL) would be enough to create an Octave package from your repository. After that, a user could install it in Octave with pkg install path_to_tarball. No need to mess around with packaging binaries for a multitude of operating systems, platforms, distributions and Octave versions. And still pretty easy for end users.

Does your package have any third party dependencies?

@mmuetzel, thanks for your comments - sorry, perhaps I did not make my question clear - I do not have issue creating platform-specific octave packages for this tool – I have already packaged it for Debian and Fedora

Perhaps, maybe let me rephrase my question to the following one:

is there any reason why octave does not support mexext? any downside of adding this support (at least make mex files compatible with matlab)?

You suggest to make mexext output the same as Matlab (mexa64, mexw64, mexmaci64) if run from Linux, MS Windows, or macOS?

I think fixing this cosmetic difference is not a big issue and you can submit a patch to Savannah. Maybe changes in mkoctfile might be required as well, if mexext is not regarded during the compilation. If this already solves your problem that would be too simple to not do it :slight_smile:

However, this does not fix the general problem, described before:

You will still produce mexmaci64-files, that might not work on any Octave installation for macOS, as there is no standard macOS Octave installation like with Matlab.

For reasons @siko1056 already described, I don’t think that having a multitude of different extensions for .mex libraries would be a step in the right direction. Given that Octave can be compiled on a multitude of platforms and in many different configurations, we’d probably need tens or hundreds of different extensions if we wanted to make sure that those files match the given platform/configuration. I don’t think that is practical (also from a packaging point of view).

You might be misunderstanding how Octave packages work. A user wouldn’t need to hassle with installing compilers or anything. The pkg command takes care of correctly compiling and installing Octave packages.

To install the package, a user would need to execute one single command. E.g. for the signal package:

pkg install "https://downloads.sourceforge.net/project/octave/Octave%20Forge%20Packages/Individual%20Package%20Releases/signal-1.4.2.tar.gz"

It would probably be easy to tell users that command even if they have no experience in compiling software.

1 Like

You suggest to make mexext output the same as Matlab (mexa64, mexw64, mexmaci64) if run from Linux, MS Windows, or macOS? I think fixing this cosmetic difference is not a big issue

that’s part of what I meant, but missing the key piece - what’s missing is octave does not even look for mexa64/mexw64/mexmaci64 in the search path on these platforms when typing the mex file name.

we’d probably need tens or hundreds of different extensions

that’s not a strong reason for not supporting mexa64/mexw64/mexmaci64/… that MATLAB do have support. You can let all other non-overlapping platforms fall back to .mex.

You might be misunderstanding how Octave packages work.

this may work if the code does not have a lot of dependencies (or 3rd party dependencies), but even octave can not ensure that needed libraries are correctly installed, thus compiling the code properly, on Windows/Mac. Several of my packages requires CUDA (mcx/mcxlab at master · fangq/mcx · GitHub) and OpenCL (mcxcl/mcxlabcl at master · fangq/mcxcl · GitHub). Relying on package installer to handle the compilation of these toolboxes is very difficult.

I don’t fully understand the above case - can you explain? my impression is that mexext is processor-specific, not installation instance specific.

Would it be possible to use the same file extensions that Matlab already uses? The .mex interfaces of Octave and Matlab are (hopefully) API compatible. I don’t know if they are also ABI compatible.
Is it possible for you to rename a .mexw64 file created by Matlab’s mex compiler and just use it as it is in Octave on Windows?

Apart from that, I don’t know if it would be possible to use just one single file extension per platform. Matlab is closed source, and they know exactly which compiler version, linkage, standard libraries, … they used when compiling their distributed version.
Octave on the other hand is open source. It can be built with different tools also on the same platform. E.g. on Windows, it could be built with gcc or clang (or MSVC?). It could be a cygwin or a native application. It could link to MSVCRT (different versions) or UCRT. It could link to GCC’s libstdc++, to LLVM’s libc++ or to MS’s implementation of the STL. It could be configured differently (e.g., with or without polymorphic allocators). And probably many other differences that would likely make the binaries (Octave itself and the mex file) incompatible.
Similar situation on other platforms (e.g. Linux or Mac).
All of this might need to be “encoded” in the file extension if we would decide to go that route. That is what I meant with “not practical”.

It makes much more sense for Octave to give up on encoding that in a file extension and facilitate compiling a version of the .mex files that is known to be compatible on the user side. That is one of the main motivations for Octave packages (the pkg command).

I agree that it might be more difficult to do that if the package depends on third party libraries.
Could you describe which exact issues you are facing for those packages? Maybe we could try and make this less painful somehow.

1 Like

@mmuetzel, again, I appreciate the comment. My goal here is to help flush out some ideas on solving this issue. I am familiar with C and MATLAB/Octave, so I am happy to contribute to this solution if we can find one.

I still don’t feel the above argument is strong enough - what you were saying is that the .mex-arch name scheme that matlab used is too coarse-grained for Octave because it also support different compilers etc. However, using .mex cross the board is almost the most coarse-grained solution to address that one could find. In my opinion, this is rather a step in the wrong direction.

I agree that Octave does not have to entirely follow the conventions that matlab uses (although I believe following matlab conventions have been a great strategy and and effective one - a guideline that the community has been following to develop most octave function interfaces). If the .mex-arch suffix approach is too coarse in your perspective, then perhaps the developers should consider a more systematic approach, for example, binaries packages for Python are named using this naming convention since 2016

Linux and android also have systematic binary API/ABI naming conventions to address the same problem.

all I am saying is that there is a need to distinguish binaries build for different platforms/architectures, and attaching a uniform .mex is not enough.

on the other side, addressing this issue (i.e. letting octave find the arch-appropriate binary mex file) does not require the prerequisite of distinguishing differences due to different compilers/libraries/settings on a given platform. Imagine that a .dll or .so can also be compiled by numerous compilers/settings, but they can also be effectively loaded/used on the same platform that it is compiled.

PS: I want to also add that MATLAB also supports a wide range of compilers (officially, and unofficially), and the mex extensions have not been limited to 3 platforms, it also supported mexw32, mexglx, mexmaci, mexarmhf, mexmac, some of these were undocumented.

I think there is a misunderstanding here, the approach currently taken by Octave is not to distribute only binaries with .mex extension but rather to not distribute binaries at all but to build the binaries locally instead.

Note that, if you just want to be able to load a function definition from a binary shared library that has a different extension this is probably already possible now using the autoload command and PKGADD / PKGDEL directives.

c.

1 Like

There might be licencing issues here, I think binary mex files linked against liboctave are derived work of Octave and therefore must be GPLed, while CUDA is non-free …

IANAL but you may want to check this with someone who has the legal knowledge, before publishing binary packages that combine liboctave and CUDA.

c.

@cdf, making every binary package to be able to build on every system with every compiler of choice would certainly sound Utopian, but in reality, I think it requires an insurmountable effort to make everything work - small things like a compiler flag change could easily break numerous packages. Even most linux distributions are taking the path of disseminating packages in the form of precompiled binaries - with armies of volunteers as packagers and package maintainers.

thanks for mentioning autoload, which I was not aware previously. I wonder if there is a hook for a path that is added rather than a package is being loaded? inserting autoload in the path-adding hook callback would likely solve the problem I am having.

IANAL but you may want to check this with someone who has the legal knowledge, before publishing binary packages that combine liboctave and CUDA.

I was just using CUDA as one of the examples of the diverse needs from toolbox makers and users.

On the other side, a user can and are allowed to build/link non-opensource libraries with GPL licensed software - as long as the user does not intend to release/redistribute the linked binary/software with a 3rd party. This is guaranteed by Freedom 0 – “The freedom to run the program as you wish, for any purpose (freedom 0).”

I also want to add that whether octave maintainers prefer to make all mex/oct-based packages build-from-source or provide arch-dependent binaries is a different issue than whether mexext should be implemented to mimic matlab’s behaviors. This is like saying octave respects # ... as the comment, but it does not mean respecting % ... to make it compatible with matlab is a bad strategy.

I’m not saying the approach taken by Octave is the best possible choice.
Just that this is how things stand at the moment.
Changing this state of things would require a huge effort, but you are welcome to work on this project if interested.

Indeed, I see other cases where your idea of binary packages is not problematic, I was just pointing out this possible issue because I recall it being discussed before on the Octave lists.

This is correct, but I thought the discussion here was about ways to distribute your package to 3rd parties in binary form, wasn’t it?

PKG_ADD and PKG_DEL are mainly used in Octave packages. But commands in those files are executed whenever their path is added or removed from the load path, e.g., with addpath or rmpath (independent of whether that is part of loading or unloading a package).

2 Likes

What I was trying to express was that a solution that might be working for Matlab might not be working for Octave for the reasons I already tried to outline.

Is that really the case? What you are writing is that it would be possible to take a library that was built and packaged for - let’s say - any version of Fedora and use it - let’s say - in any version of Ubuntu. I don’t think that this would work in general (and is one of the motivations for things like snap or flatpack). Also, I don’t understand how naming conventions would come into play in that scenario…

That is orthogonal to the topic at hand. It is still possible to package those Octave packages in different distributions as precompiled binaries. And that is done for many Octave packages by many distributions already. The Octave packages that are included in Octave for Windows are kind of the same (precompiled binaries for that distribution)…
In that use case, the Octave binary is already “fix” (tools, configuration, …). So it makes sense to distribute binaries for Octave packages that match that specific version of Octave. In general, an Octave package (or .mex file package) cannot rely on shipping some binaries and expect that they would be compatible with any Octave…

1 Like

The only difference is that if you don’t use the package structure you need to manually provide PKGADD/PKGDEL FILES while in a package you can use the PKGADD/PKGDEL MACROS inside the source code and the files will be generated by the package manager during installation.

2 Likes