Online Developer Meeting (2020-11-10) - The future of Octave Packages

maintainers
#1

Tuesday, November 10 at 18:00 UTC. That is 10:00 Pacific / 13:00 Eastern in the US, 19:00 in Central Europe and 3:00 in Japan (Wednesday).

Octave Forge a blessing or a blocker? — Octave Forge started as a project to cope with slow Octave core development and was successful with it for many years. Today the situation seems inverted.

The last big push came in 2017 with an amazing new website and hard repository migration work done by many volunteer developers. Many of them are no longer active and necessary maintenance work does not happen. For example, there are 330 open bug reports only for Octave Forge packages. More than half of the packages are “abandoned” for more than a year.

str = urlread ("https://gnu-octave.github.io/pkg-index/");
t = regexp (str, '<td>(\d{4}-\d{2}-\d{2})</td>', "tokens");
t = cellfun (@datenum, t);

number_of_packages = length (t)                        # 77
one_year__or_older = sum (t < datenum (2020, 01, 01))  # 49
two_years_or_older = sum (t < datenum (2019, 01, 01))  # 31

hist (t, 80)
xt = datenum (2008:2022, 01, 01);
set (gca (), "xtick", xt);
datetick (gca (), "x", "yyyy", "keepticks");

image

Other programming languages like Python live from a huge community of package contributors. Octave has an easy to use package system, but not many well-maintained packages. Why?

I would like to setup an Online Developer Meeting to discuss how to improve the situation and make it more attractive to create Octave packages.

  • Who enforces the “high-quality” of Octave Forge packages?
  • How to deal with the package legacy code?
  • Merge the lightweight approach pkg-index with Octave Forge?

:point_down::point_down: Start discussing here :point_down::point_down:

1 Like
#2

Perhaps a definition of abandoned packages needs to be made.

I know of a few that I would be maintaining that haven’t been touched for many years, however they install ok, and don’t have any bugs reported and don’t show warnings on install - so a case of not updated as nothing to update except the date.

Others of course are obvious abandoned contenders that either have no one willing to maintain them, or in some cases the maintainer/author says they want to continue maintaining but hasn’t released anything.

#3

Is there any measure of how many Python packages are actively maintained?

#4

According to pypi, there are 269,415 packages.

According https://pypistats.org/packages/__all__, there were 168,527,315 downloads yesterday

How many python users are out there compared to matlab/octave users?

#5

Those are large numbers. But I’m still wondering what the popularity distribution looks like, and which of those ~270,000 packages are seeing active development.

#6

To make packages useful, having a central location with a nice interface to find and download them is important. But it seems like we have created a situation where there is some expectation that there is some centralized responsibility for maintaining the packages we host at the Octave Forge site.

For other systems like Python, do the users expect that the packages available using pypi are somehow centrally maintained?

#7

Ignoring warnings during install, and attempting to install all packages that come from the pkg list -forge, I get the following not installed successfully:

In octave 5.2:

  • fuzzy-logic-toolkit 0.4.5 - anonymous function bodies must be single expressions - repo has fixes
  • ocs 0.1.5 - needs odepkg
  • quaternion 2.4.0 - no member named is_bool_type - error fixed in mxe
  • secs2d 0.0.8 - Octave_map does not name a type
  • strings 1.2.0 - gripe_wrong_type_arg was not declared in this scope - fixed in repo

In Octave 6.0:

  • fem-fenics 0.0.5 - not installed as no dolfin
  • fits 1.0.7 - fails from old defines - repo has fixes # 55374
  • fuzzy-logic-toolkit 0.4.5 - anonymous function bodies must be single expressions - repo has fixes # 53549
  • level-set 0.3.0 - const class Array has no member named length - fix in repo - # 49294
  • ltfat 2.3.1 - parse error near line 153 of file ltfat-2.3.1/nonstatgab/nsdgt.m
  • nurbs 1.3.13 - const class octave_value has no member named is_real_type - fixed in repo
  • ocs 0.1.5 - not installed as needs odepkg
  • quaternion 2.4.0 - no member named is_bool_type - error fixed in mxe - # 59163
  • secs2d 0.0.8 - Octave_map does not name a type # 44803
  • sockets 1.2.0 - octave/toplev.h: No such file or directory # 57415
  • strings 1.2.0 - gripe_wrong_type_arg was not declared in this scope - fixed in repo - # 55385
  • vibes 0.2.0 - const class octave_value has no member named is_cellstr - # 59376

Many of these have fixes either in the repo or on a bug report.

I’m ok with helping to ‘maintain’ a few of these in order to keep them installing at least, but also fine with culling those that are unmaintained where effort to keep them going is too great.

That of course is ignoring the additional other efforts required for review and publish them etc. :slight_smile:

1 Like
#8

I haven’t done any python packaging. Looks like many but not all are hosted on github, but I dont think pypi does anything other than host the list and the released package.

I’ve attempted to install python packages in the past that fail due to old requirements or other incompatibilities much the same as some octave packages.

#9

Thanks for the vivid discussion. Maybe I should clarify my choice for comparison with Pypi. I think (sorry for not having reliable figures) that pypi and may other free and unmaintained package managers suffer from similar aging problems.

Aging is not a problem in itself, as @lostbard pointed out. A simple package written in pure Matlab compatible Octave language only might work from Octave 3 up to Octave 7 without any need for maintenance. But, except for about 20 well-maintained packages, aging packages is all what Octave (Forge) has, nothing new is coming (see below) :sweat: In general, not all packages are that simple and this legacy code maintenance does not happen, as @lostbard greatly summarized.

Another interesting aspect of legacy code maintenance in a too strict environment is “package taking over”. If the original maintainer A does not continue and another B is interested in further maintenance, can B just continue the development? Such case happend 2019 for the fits package last released in 2015. My idea to this is forking and letting the original code rest in peace :innocent: But Octave Forge is not suited for this approach, adding and releasing a package is “expensive”.

My point with pypi is, that this community is without doubt active (see alone the front page of pypi). If a python package becomes unuseful, or even before, it gets forked, alternatives are developed, sometimes better than the original contribution. Getting started in pypi is an automated process, with no human interaction or delays at all. The same idea shares Octave pkg-index.

For Octave Forge, I have seen very few attempts in the last two years to contribute a new package:

To my knowledge this list is comprehensive. And I wished to get more contributors involved to make Octave packages development fun and seeing many interesting projects pop up :slightly_smiling_face:

1 Like
#10

Also:

  • 2020-01-03 midi toolbox which took over the unmaintained audio toolbox

I do like the pkg-index idea as it could provide a lot more central visibility into what octave code is out there rather than having to search the internet.

#11

It’s actually 10:00 Pacific, 13:00 Eastern.

1 Like
#12

this soooo could have been the year to kill DST. but mike is correct.

1 Like
#13

As Jitsi was working nicely for most of the previous meetings, I suggest to use it again:

https://meet.jit.si/octave-dev-2020-11-10

and created a wiki page. Please write here or in the wiki what you wish to discuss :slightly_smiling_face:

#14

For the mxe inclusion of packages JWE had added some initial ones. For packages added since then, generally I’ve added a package when it requires additional libraries to be installed in order for the forge package to be installed.

For example the dicom packages requires the GDCM libraries installed and its far easier to do that as part of the build of mxe than to have the user do it prior to installing the package.

Of course at that point it means that there are additional libraries and packages installed that the user may not want to need.

1 Like
#15

Hello, I missed last meeting because it was not announced on the octave-maintainers mailing list: if this forum is the one to monitor for now on, shall the mailing list be closed? Thank you.

#16

Sorry for not properly reaching you with the announcement. From the Online Developer Meeting (2020-07-07) I concluded, that this Discourse is the primary source of communication to monitor.

The mailing-lists shall remain active, but unadvertised. Some users might not know about Discourse yet and cutting long-established communication channels too early might have other side effects. One day in the future, when the communication on the mailing-list converges to zero, it can be closed and archived.

If you want to share your ideas at another meeting about this topic, please feel free to announce it at a time convenient to you. If possible, I am happy to attend such meetings or code sprints :slightly_smiling_face:

#17

Thanks, Kai. I did attend the meeting where Discourse was discussed so I should have known better! As all of the previous meetings were still announced on the mailing list, I didn’t look further. I will be more careful next time!