Tests for plotting/graphics functions

recent bug fix discussion for bar saw a small error almost slip through due to lack of BISTs for most plotting functions (most only have demo blocks requiring human visual error checking). While coding the patch I was wondering whether such functions could have useful tests derived from the handles that most of these functions return. In the case of bar, would the properties be present to probe for correct rectangular patch positions/coordinates, etc.

How messy/fragile would that be? Would a innocuous change to an underlying draw function break all of the tests by, say, a vertex ordering change, even when figure is actually still correct. Etc. we wouldn’t want something like that to require manually rewriting all such BISTs. Would there be a way to make it less problematic? are the right useful property hooks even there to probe in the first place?

second, if this approach works, how big a task would it be to start going through “all” of the functions in core octave that it would apply to? is there a way to screen functions needing these kind of tests? (e.g., bar now has tests, but just for input error handling. so it wouldn’t show up in a set of functions without any tests)

Many graphics objects that have dedicated types in Matlab are implemented in Octave using more “low level” graphics objects (like patch, surface or line).
IIUC, we considered this fact a mere “implementation detail” until now.
Do you want to add BISTs that check if the “exact realization” of these graphics objects stays unchanged? If that is the case, I am not sure if that is a good idea. It would be nice if these implementation details could change without many BISTs breaking (erroneously imho).
See e.g. the scatter graphics objects that were implemented more or less recently replacing the previous implementation using patch objects.

What do you propose to test actually ?

The behavior of dependent properties in an object is something we could test, e.g. “if I change property X this way I expect property Y to be updated that way”.

Indeed, there are implementation details and I agree we should avoid testing those, but there are also ML documented behaviors that would benefit from being tested.

As for the end result, pixels on a screen, it seems hardly testable. There is an example of such test in getframe but it has to be very “tolerant” in order to avoid false positive, and as such cannot be used to test finer aspects of rendering.

simple example -

h = bar([1 2 3; 4 5 6]);
ans =
[1,1] = 0.80000
[2,1] = 0.80000
[3,1] = 0.80000

h = bar ([1 2 3;3 4 5],.4);
ans =
[1,1] = 0.40000
[2,1] = 0.40000
[3,1] = 0.40000

so, that’s a testable value. so is ‘ydata’. although that’s just segment height not ‘coordinates’ to verify stacking, etc.

granted those aren’t really graphics properties, just input data fed to the low level graphics element.

I’d really like something to make sure we don’t introduce regressions when coding. This is a general goal for me. Part of accelerating the design/build/test loop of engineering is being able to rely on a regression suite. We don’t have a full BIST set for all of Octave functions, but there is particularly low coverage, often none at all, for the graphics functions.

I agree with Pantxo that we can’t use pixel-for-pixel comparison as a general methodology. And it is important to have the tests not be brittle so they don’t crack when a small change is made to the rendering engine. As an example, several versions ago we switched the default colormap from “jet” to “viridis”. If there had been tests looking for exact colors they would have failed.

What I think this points to is a less general, less ambitious strategy. Instead of adding BIST tests for all graphics functions below scripts/plot we should add tests where we can and when they are feasible. The bar chart is a potentially good example in that it is possible to check through properties some of the rendering decisions.

this is what I was getting at. realized though that the ‘obvious’ properties for bar are just storing input values and not really checking any work done by the function. I’m not familiar enough with the graphics objects to know how to probe for other properties than what comes up from a get on the function returned handles.

after finally just graduating ‘yay me!’ i was looking for a m-file level project and this seemed to pop up when fixing bar. (the set up one of my windoze machines (or a vm) to compile so i can look at more is another day’s project)

1 Like