Inclusion of non-traditionally packaged octave packages to the package extension index

tl;dr: I propose that the Octave project / community would benefit from the github package extension index allowing high-quality packages to be included, even if they technically do not follow octave’s official packaging format. They could be differentiated from ‘proper’ packages by e.g. ticking a checkbox (i.e. to show ‘proper’ packages only).


Thesis:

I have followed with interest the recent developments regarding ‘augmenting’ the octave-forge presence with the github-hosted package index that Kai proposed.

I also followed with interest the discussions around how Octave’s packages potentially affect Octave’s ‘brand’, when potentially unmaintained packages are perceived by (often angry) users as being the responsibility of the main project, as opposed to what they really are, which is user-contributed / community-maintained, i.e. more akin to what pypi is to python. This tends to get even more confused when these packages are intended to provide functionality that attempts to mirror functions provided in matlab toolboxes.

While I find the octave forge website to be a great resource, particularly with respect to having searchable documentation (which the github package index unfortunately does not support), I do agree that the octave forge webpage suffers from this ‘branding’ problem, whereas the github page makes this distinction much clearer.

One aspect which I always felt was a bit of a problem in the past in the octave-packages space, is that there is a wealth of high-quality octave ‘packages’ in the more general sense, which however do not receive much recognition from the octave community as ‘proper’, worthy packages, because they do not follow octave’s official packaging format, and are thus not considered “packages” per se in octave-lingo. E.g. ORCA, EEGLAB, Psychtoolbox, Dynare, etc. This seems to preclude them from being associated more closely with the project. Typically the only way octave users get to know about these projects is if they are compatible and already popular in matlab, and people are trying to switch from matlab to octave.

I believe that in part, the ‘branding’ part was also partly to blame for this as well, in that, these package creators would not think or attempt to try and get a package “into” octave forge, if it was not a matlab-toolbox implementation.

Given this move away from packages as ‘part of the octave brand’ and more towards ‘community-led’, I feel like the github package index is a great opportunity to help these projects gain visibility from within the octave ecosystem (which also helps octave), by allowing their inclusion to the github package index, even though they have their own installers, and deviate from octave’s canonical packaging format. Perhaps a distinction can be made on the github page by including a tickbox which allows filtering “non-properly packaged” packages on/off.

I think this would be a good compromise, and partly achieves some of the goals that the now defunct Agora website was also intended to serve back in the day (albeit for complete packages rather than for individual functions or snippets). And hopefully, if they are allowed to add their package to the package index, maintainers may be more tempted to package their packages properly, so as to make it to the ‘proper’ list. This also helps users like me to open an issue on a project’s page and encourage the authors to list their package formally in the package index.


PS. Along similar lines, but as an addendum to this post. I think it would also be worth having a way to differentiate between packages/functions that attempt to ‘mimic’ matlab toolboxes/functions, versus other packages, unrelated to the matlab ecosystem.

1 Like

Your idea I like very much. Similar thoughts drive my current work on the Octave package ecosystem (Octave Packages, pkg). You pointed out a goal which is easier to reach than I originally planned :slightly_smiling_face:

In my understanding of existing additional Octave/Matlab software, some package developers appreciate writing compatible code for both softwares. Deciding for either being compatible to the Octave package format or the Matlab toolbox format feels like an overkill or “vendor lock-in”. Maintaining two separate package versions is time consuming and I am not aware of a project that seriously maintains both “vanilla” package formats (are there examples?). In the end it is most likely simple-stupid “install.m” in the root directory or a small “README”-install-section can almost do the job with less programming/maintenance effort.

In all packages/toolboxes it boils down to extracting and adding the code to the load path. In several cases compilations (mex) are involved and in a few Octave Forge packages additional package configurations.

To realize your idea, Octave packages regards a “pkg” dependency in the first version of a package, for example, packages/pkg-example.yaml at master · gnu-octave/packages · GitHub This addition is sound with the architecture of Octave package and the new pkg-tool (as it will be a package as well :sunglasses: ). I’ll report back, when I finished this addition, as certain elements on the web package should not be shown, if the package cannot be installed by pkg-tool.

@tpapastylianou The door is open :slightly_smiling_face: Any package or toolbox is now permitted to be added there. See a first example “sqlp-sedumi” I added. Please add what you like to see there :wink:

Octave Packages now shows for such packages only a download link, without any “pkg” instructions using the approach explained before.

I had some thoughts about this and I don’t want a discrimination of “non-properly packaged” packages:

  • A good package is maintained and works with the lastest stable Octave release. Let alone this point is not true for all Octave Forge packages. Thus why advertising this as “quality”, if a well-maintained non-standard package after a slightly more complicated setup than “pkg install -forge …” does the job perfectly, too?
  • The (nearer) future version of pkg shall be able to handle some sort of “custom package format”. That is if the package does not follow Octaves format, it falls back to some custom setup routine that is called. For the customization to work is alone the responsibility of the package maintainer and I do not expect those packages will be bundled with mxe, for example. If the package setup is smooth and fast, who cares? :sweat_smile: Details are yet to come.

Agora was thought too big, without any budget or full-time employees like the commercial archetype the development and maintenance is too overwhelming. Similar for Octave Forge, while many volunteers worked on the project, it worked out.

Octave packages is more like a editable phone book. All the hosting of the package code and documentation is in the responsibility of the package maintainer, which in times of GitLab, GitHub is not a problem at all. In my perception, a great deal of (not only pure Octave) package development already happens there and has to be pushed with effort back into the SourceForge ecosystem.

The value Octave Packages delivers is a searchable phone book and a really simple stupid fast mechanism to resolve the full Octave Packages index with all versions and dependencies from within Octave with less than 10 lines of code in under a second. Compared to “pkg list -forge” 18 seconds for just a simple list of names and the latest version.

This is indeed the only true value of Octave Forge that is “lost” for now (of course it is still online and will be there in the future, but it is no longer maintained). I hope others or me will come up with a better idea for Octave itself how to get it back in a more scaling fashion (see for example Function reference documentation).

The Octave Forge documentation has to be paid with a high maintenance price (generating and deploying the website manually) and little flexibility for the package maintainers. Always in need of a grateful Octave Forge admin willing to devote his time for his package release. A simple change in the documentation (new CSS styles) would require the manual regeneration of the 70 packages documentations again :sweat_smile:

Furthermore the package documentation has never been fully developed by the “pkg”-tool. At the package setup the “doc” folder is simply “dumped” deeply into the destination folder “for future reference” (quoting the manual) and that is it.

Hi Kai!

Thank you for responding so promptly, and with a change to the website as well!

I’m very happy to see this change, and keen to start pointing developers of non-traditional octave packages to the page!

However, I would also like to reply to this point

I had some thoughts about this and I don’t want a discrimination of “non-properly packaged” packages:

as I partly disagree with this.

You make two points here, the first about “implications of quality”, and the second about subsuming custom-installer functionality into the pkg tool.

The first is easier to address: I don’t think distinguishing between “pkg-installable” packages, and “custom-installers” has implications for quality, anymore than “pypi” installable packages are necessarily reflecting better quality than github packages (if anything, pypi has the reverse reputation these days, i.e. that it is easy to get tricked by malicious packages). Similarly, I don’t consider a .deb package to be superior to an ‘extract and run’ binary .tar.gz package – it’s just a different method of delivery. For me the only issue is a) is the package indexed somewhere so that I can find it easily if I’m looking for that functionality, and b) is it clear how to install it once I download it.

So for me the distinction is less about ‘branding’ and more about UX: you will avoid angry users who try to install the package “the octave way” and fail. E.g. if it was a common belief that all linux packages are always installed via dpkg, and you listed .deb and .tar.gz packages side by side in a table, with little distinction between them, you could reasonably expect to get users who would try to install a .tar.gz extractable package via dpkg and get frustrated.

I agree that perhaps having a tickbox in the main page to filter non-traditional packages out might be a bit overkill. But I don’t think the current distinction is clear enough at the moment. Having an http:// link as opposed to a ‘pkg://’ (so to speak) link in the package’s page is not clear enough in my view, and the added ‘installation’ link is easy to miss as it stands. My suggestion would be to at least include a nice notice like in the image below:

Regarding the second point you make, that ‘pkg’ may have a way to incorporate custom installers in the future … sure, but even then there’s going to be packages that do not try to be compatible with that, but may still be worthy additions to the octave ecosystem. Maintaining a general way to index these packages while still having a clear UX making it clear that their installation process differs from traditionally installable packages, would still be beneficial in my view.

Thanks again for entertaining my thoughts on this :slight_smile:

1 Like

Thanks for your reply. I agree that this notice is very helpful for users who get stuck installing the package in the “pkg way”. Thus I added your suggestion :slightly_smiling_face:

https://gnu-octave.github.io/packages/sqlp-sedumi

Woo! Thanks. Looks great! :slight_smile:

1 Like