Welcome Octave Packages!

It has been two years since the introduction of Octave Packages: the new Octave Forge. After a long test run in parallel to Octave Forge, the new platform Octave Packages is ready to handle a growing community of Octave package developers.

Starting from Monday, July 4th, the “pkg install -forge” command will be handled by Octave Packages.

How does this affect me?

As Octave user

You gain easy installation access to about 20 more packages (number growing) than you can obtain now.

As new Octave package developer

  • If you have been rejected or felt discouraged by the work hosting an Octave package, take a look at Octave Packages. Your work can be published within an hour.

As previous Octave Forge package developer

  • Octave Forge remains in general untouched, despite deprecation notices.

  • You are free to keep hosting your Octave package on the Octave Forge platform or move to a more convenient workflow or location and register your updates on Octave Packages.

  • If you decide to keep hosting your package on Octave Forge, please contact @siko1056 or any other Octave Forge admin to promote your account with admin rights. Those are necessary to upload the release tarballs and documentation (see Reviewing Octave Forge packages - Octave).

    Starting from Sunday, July 10th, @siko1056 stops actively releasing packages on Octave Forge.

  • If you move away from Octave Forge, you must consider two things:

    1. Where to host your tarballs (own webspace, GitHub releases, …), see Releases · gnu-octave/pkg-example · GitHub for inspiration.
    2. Where to host your package documentation (own webspace, https://readthedocs.org/, or simply in the repositories README file, …), see GitHub - gnu-octave/pkg-jupyter-notebook: A package to run and fill Jupyter Notebooks within GNU Octave. for inspiration.

    Octave Packages provides you with full flexibility regarding both decisions, please contact @siko1056 if you have any doubts.

  • What you gain with Octave Packages is control over your package again: shorter release cycles in your own development workflow with higher release quality control via GitHub Actions package checks.

Timeline and outlook

To summarize the important dates:

  • Monday, July 4th: Octave Packages resolves pkg install -forge commands
  • Sunday, July 10th: Octave Forge releases must be done by package maintainer

Again, Octave Forge as website will be remain available, however, it is no longer of any importance for the Octave package ecosystem. It remains as one of many hosting options.

I am looking forward to a revitalized Octave packages development on the new Octave Packages platform.

5 Likes

Thank you for your maintenance efforts and for your work on this!

3 Likes

To browse through what packages are available on Github that weren’t there on SourceForge, I’m currently viewing the list at GNU Octave - Packages using a web browser and choosing what I want manually.

I have a dim recollection that SourceForge provided a mechanism to list all possible packages hosted, though I can’t recall how to access it any more. Does Github have an automatic machine-readable listing of packages to your knowledge? (Not screen scraping)

The url https://gnu-octave.github.io/packages/packages/ returns back a page with octave commands that builds a structure of all the packages.

2 Likes

The Octave package command provided the function

tic
pkg list -forge
toc
...
Elapsed time is 7.84078 seconds.

Which took several minutes before Octave 6, and a few seconds since Octave 6 to compile a list of all Octave Forge package alongside with the latest version.

As @lostbard pointed out, Octave Packages provides a full Octave readable index which enables my package tool under development to be much faster with package dependency resolution.

@arungiridhar what you are seeking:

function __pkg__ = package_index_resolve ()
  data = urlread ("https://gnu-octave.github.io/packages/packages/")(6:end);
  data = strrep (data, ">",  ">");
  data = strrep (data, "&lt;",  "<");
  data = strrep (data, "&amp;", "&");
  data = strrep (data, "&#39;", "'");
  eval (data);
endfunction
tic;
pkg_index = package_index_resolve ();
toc  # Elapsed time is 0.675889 seconds.

## List all available packages by name
pkg_names = fieldnames (pkg_index)

## List all available packages by name and latest version
tic
pkg_cell = struct2cell (pkg_index);
pkg_name_versions = cellfun (@(x) [x.name, " ", x.versions(1).id], ...
                             pkg_cell, "UniformOutput", false)
toc  # Elapsed time is 0.0139558 seconds.

Could be implemented into the current pkg tool, until the development version of mine is ready to take over too :innocent:

1 Like

Yes! Was looking for exactly that functionality!

Two things that I would like to work on in the upcoming months, unless this functionality already exists somewhere:

  • Automatic parsing and resolution of package dependencies, at least to guide a user in first-time installation of all packages. I maintain my own personal list of “install Forge packages in this specific sequence for a new Linux installation” based on past manual recursion of dependencies, but now that there are many more packages on Github than on Forge, automating that has more value. Has this already been done?

  • Some sort of app store equivalent for Octave packages, especially for keyword search. More than once I found some math functions already implemented but hiding in a package I had not thought of (like something related to finite group manipulation was in the communications toolbox though my finite group work back then was not related to communications). Right now I’m doing a basic search on Sourceforge or Github for this but it’s not reliable.

So far I’ve done the above manually for Forge packages when I’ve needed to, but it’s at the point where that doesn’t scale even for just one user.

This is a great improvement. Curious, how ‘portable’ the current setup is. With microsoft having acquired Github, and people being concerned about it going in a similar philosophical direction as sourceforge did, is there a concern we could find ourselfs in a similar position 5-10 years from now? And would transition to another service be any easier or harder than this transition? Are any decisions made now able to make that easier/harder down the road?

your first comment has been the subject of a longstanding bug

there were some concepts in there for how to go about parsing dependencies, but i think a lot was waiting on better infrastructure, or a better version of pkg. it may be that it’s an easier task now?

Thanks for sharing your concerns. Octane Packages is a static website build with a Ruby tool called Jekyll. The same holds true since 2016 for the Octave website.

Building and hosting/deploying Octave Packages is independent of GitHub, even though deployment of such static websites on GitHub is more than easy: GitHub Pages.

The Octave website is manually deployed on Octave’s FossHost server, for example. The same could be done for Octave Packages as well, but it would loose some dynamic elements that had to be replaced:

  1. Automatic page deployment on each package release (not necessary for the not too often changing Octave website for example)
  2. Automatic package quality checks with GitHub Actions.

If GitHub went rouge on software freedom, there is a not too hard exit strategy of hosting on FossHost, but the Octave project would need a stronger server in that case.

2 Likes

A draft is inside the new package tool:

The latter function lookup project is merely a question of a webspace with shell access (not given on Octave Forge for example). Can FossHost sponsor a second server :speak_no_evil::innocent:

1 Like

I expect that Fosshost will be willing to give us resources if we need it. The difficult thing is getting them to respond! I realize they’re driven by unpaid volunteers too though, so a long response time is the price for a scalable server infrastructure.

True, I must admit that I become more and more scared of the first real server down time :sweat: One day offline okay, but their response regarding another issue with the existence of off-site backups is pending a few weeks by now, right? :exploding_head:

Yes, the request re backups and the request for two more hypervisor accounts. But their uptime has been good though, so let’s hope.

1 Like

I made a helper for reading and resolving prerequisites. It just reads the metadata, constructs a graph, and goes in topological sort order, more or less.

File:
packagehelper.m (2.7 KB)

Output:

Using information current at 08-Jul-2022 15:51:59, install the packages in this order.
pkg install -forge audio               
pkg install -forge bioinfo             
pkg install -forge biosig              
pkg install -forge bsltl               
pkg install -forge caosdb              
pkg install -forge cfitsio             
pkg install -forge cgi                 
pkg install -forge coder               
pkg install -forge control             
pkg install -forge dataframe           
pkg install -forge dicom               
pkg install -forge divand              
pkg install -forge doctest             
pkg install -forge eeglab              
pkg install -forge fda                 
pkg install -forge fem-fenics          
pkg install -forge femoctave           
pkg install -forge fits                
pkg install -forge fpl                 
pkg install -forge fuzzy-logic-toolkit 
pkg install -forge ga                  
pkg install -forge general             
pkg install -forge generate_html       
pkg install -forge geographiclib       
pkg install -forge gsl                 
pkg install -forge image               
pkg install -forge image-acquisition   
pkg install -forge instrument-control  
pkg install -forge interval            
pkg install -forge io                  
pkg install -forge joystick            
pkg install -forge json                
pkg install -forge jupyter-notebook    
pkg install -forge level-set           
pkg install -forge linear-algebra      
pkg install -forge lssa                
pkg install -forge ltfat               
pkg install -forge matgeom             
pkg install -forge miscellaneous       
pkg install -forge molsim              
pkg install -forge mpi                 
pkg install -forge mqtt                
pkg install -forge mvn                 
pkg install -forge nan                 
pkg install -forge netcdf              
pkg install -forge nurbs               
pkg install -forge ocl                 
pkg install -forge ocs                 
pkg install -forge octclip             
pkg install -forge octproj             
pkg install -forge onsas               
pkg install -forge optics              
pkg install -forge optiminterp         
pkg install -forge packajoozle         
pkg install -forge pde1dm              
pkg install -forge pkg                 
pkg install -forge pkg-example         
pkg install -forge pythonic            
pkg install -forge quaternion          
pkg install -forge queueing            
pkg install -forge raspi               
pkg install -forge secs2d              
pkg install -forge signal              
pkg install -forge sockets             
pkg install -forge sparsersb           
pkg install -forge splines             
pkg install -forge sqlp-sedumi         
pkg install -forge statistics          
pkg install -forge stk                 
pkg install -forge strings             
pkg install -forge struct              
pkg install -forge symbolic            
pkg install -forge tablicious          
pkg install -forge tisean              
pkg install -forge tsa                 
pkg install -forge vibes               
pkg install -forge video               
pkg install -forge vrml                
pkg install -forge websockets          
pkg install -forge windows             
pkg install -forge zeromq              
pkg install -forge arduino             
pkg install -forge communications      
pkg install -forge csg-toolkit         
pkg install -forge database            
pkg install -forge financial           
pkg install -forge geometry            
pkg install -forge mapping             
pkg install -forge msh                 
pkg install -forge ncarray             
pkg install -forge optim               
pkg install -forge parallel            
pkg install -forge statistics-bootstrap
pkg install -forge bim                 
pkg install -forge data-smoothing      
pkg install -forge econometrics        
pkg install -forge secs1d              
pkg install -forge secs3d              

While making the above, I found a problem in the document returned from https://gnu-octave.github.io/packages/packages/ : it should say miscellaneous but misspells it as micellaneous, which causes a check to break. @siko1056 and @lostbard : Could either of you change that pls?

I see on ref t that name in it:

pkg.(“vrml”).versions(1).depends(3).name = “micellaneous”;

1 Like

Thanks @lostbard for working on this:

2 Likes

Updated
packagehelper.m (2.8 KB)

New output:

Using information current at 09-Jul-2022 11:29:30, install the packages in this order.
pkg install -forge audio                # 1
pkg install -forge bioinfo              # 2
pkg install -forge biosig               # 3
pkg install -forge bsltl                # 4
pkg install -forge caosdb               # 5
pkg install -forge cfitsio              # 6
pkg install -forge cgi                  # 7
pkg install -forge coder                # 8
pkg install -forge control              # 9
pkg install -forge dataframe            # 10
pkg install -forge dicom                # 11
pkg install -forge divand               # 12
pkg install -forge doctest              # 13
pkg install -forge eeglab               # 14
pkg install -forge fda                  # 15
pkg install -forge fem-fenics           # 16
pkg install -forge femoctave            # 17
pkg install -forge fits                 # 18
pkg install -forge fpl                  # 19
pkg install -forge fuzzy-logic-toolkit  # 20
pkg install -forge ga                   # 21
pkg install -forge general              # 22
pkg install -forge generate_html        # 23
pkg install -forge geographiclib        # 24
pkg install -forge gsl                  # 25
pkg install -forge image                # 26
pkg install -forge image-acquisition    # 27
pkg install -forge instrument-control   # 28
pkg install -forge internal-fluid-flow  # 29
pkg install -forge interval             # 30
pkg install -forge io                   # 31
pkg install -forge joystick             # 32
pkg install -forge json                 # 33
pkg install -forge jupyter-notebook     # 34 # after: 33 json
pkg install -forge level-set            # 35
pkg install -forge linear-algebra       # 36
pkg install -forge lssa                 # 37
pkg install -forge ltfat                # 38
pkg install -forge matgeom              # 39
pkg install -forge miscellaneous        # 40
pkg install -forge molsim               # 41
pkg install -forge mpi                  # 42
pkg install -forge mqtt                 # 43
pkg install -forge mvn                  # 44
pkg install -forge nan                  # 45
pkg install -forge netcdf               # 46
pkg install -forge nurbs                # 47
pkg install -forge ocl                  # 48
pkg install -forge ocs                  # 49
pkg install -forge octclip              # 50
pkg install -forge octproj              # 51
pkg install -forge onsas                # 52
pkg install -forge optics               # 53
pkg install -forge optiminterp          # 54
pkg install -forge packajoozle          # 55
pkg install -forge pde1dm               # 56
pkg install -forge pkg                  # 57
pkg install -forge pkg-example          # 58
pkg install -forge pythonic             # 59
pkg install -forge quaternion           # 60
pkg install -forge queueing             # 61
pkg install -forge raspi                # 62 # after: 28 instrument-control
pkg install -forge secs2d               # 63
pkg install -forge signal               # 64 # after: 9 control
pkg install -forge sockets              # 65
pkg install -forge sparsersb            # 66
pkg install -forge splines              # 67
pkg install -forge sqlp-sedumi          # 68
pkg install -forge statistics           # 69 # after: 31 io
pkg install -forge stk                  # 70
pkg install -forge strings              # 71
pkg install -forge struct               # 72
pkg install -forge symbolic             # 73
pkg install -forge tablicious           # 74
pkg install -forge tisean               # 75 # after: 9 control 64 signal
pkg install -forge tsa                  # 76 # after: 45 nan
pkg install -forge vibes                # 77
pkg install -forge video                # 78
pkg install -forge vrml                 # 79 # after: 31 io 36 linear-algebra 40 miscellaneous 69 statistics 72 struct
pkg install -forge websockets           # 80 # after: 65 sockets
pkg install -forge windows              # 81
pkg install -forge zeromq               # 82
pkg install -forge arduino              # 83 # after: 28 instrument-control
pkg install -forge communications       # 84 # after: 9 control 64 signal
pkg install -forge csg-toolkit          # 85 # after: 31 io
pkg install -forge database             # 86 # after: 72 struct
pkg install -forge financial            # 87 # after: 31 io 69 statistics
pkg install -forge geometry             # 88 # after: 39 matgeom
pkg install -forge mapping              # 89 # after: 31 io 39 matgeom 88 geometry
pkg install -forge msh                  # 90 # after: 67 splines
pkg install -forge ncarray              # 91 # after: 31 io 46 netcdf 69 statistics
pkg install -forge optim                # 92 # after: 31 io 69 statistics 72 struct
pkg install -forge parallel             # 93 # after: 72 struct
pkg install -forge statistics-bootstrap # 94 # after: 31 io 69 statistics 72 struct 93 parallel
pkg install -forge bim                  # 95 # after: 19 fpl 67 splines 90 msh
pkg install -forge data-smoothing       # 96 # after: 31 io 69 statistics 72 struct 92 optim
pkg install -forge econometrics         # 97 # after: 31 io 69 statistics 72 struct 92 optim
pkg install -forge secs1d               # 98 # after: 19 fpl 67 splines 90 msh 95 bim
pkg install -forge secs3d               # 99 # after: 19 fpl 67 splines 90 msh 95 bim

1 Like

Improved script:
packagehelper.m (3.3 KB)

Improved output:

# To install the package "optim":
pkg install -forge io
pkg install -forge statistics
pkg install -forge struct
pkg install -forge optim

That’s done for each package in the correct sequence of prerequisites, then it ends with how to install all 100 in the correct order if that’s what the user wants.

@siko1056 : it would be useful to new users to post the output from the packagehelper.m script either on the Octave website or on Github itself. What are your thoughts on this? Ideally that script would be run each time any package is updated or a new package is added. Can Github do that automatically?

I like your script a lot, but pkg not being able to resolve dependent packages is a missing pkg feature in my opinion, GNU Octave - Bugs: bug #44376, Dependencies when installing... [Savannah]. As said before, Welcome Octave Packages! - #10 by siko1056, my package tool contains that resolver. Help merging the code to Octave or driving the pkg-package development is very welcome :slightly_smiling_face:

If you want to move the “resolver” to the Octave Packages, it is possible but probably increases the page build time exponentially (all dependencies have to be resolved recursively). I do not know if you can incorporate an Octave call into the Jekyll website build process. However, you can work on extending the Liquid / Jekyll section packages/package.html at 2ac7ae97fd1e81b87c67394f61d30a3062358ddd · gnu-octave/packages · GitHub , for example. The task is mostly about efficiently digging into YAML files, where all information is given.

From the latter effort a pure Octave user, who is not using the Octave Packages website, will not benefit at all :sweat: Therefore I prefer extending the pkg tool, which also needs lots of attention.

Good points both. I’ll switch attention to the pkg tool you developed.