Parsing command-style function call syntax

I’m trying to improve Octave’s compatibility with Matlab for parsing command-style function call syntax. See the following bug reports about trouble caused after changing Octave to be more like (modern) Matlab and not check the status of symbols as variables when deciding whether to parse something like foo +bar as an expression or a command-style function call:

https://savannah.gnu.org/bugs/?60882
https://savannah.gnu.org/bugs/?60941

To better understand what Matlab behavior actually is, I wrote some test scripts. Could someone please help by downloading and unzipping the attached file and running the included main_test_script in a current version of Matlab and then uploading the results here?

I decided to open a new topic instead of posting this file under the “Need help understanding Matlab behavior” topic because I anticipate additional discussion about how best to handle the conflicts that arise due to Octave’s extended set of operators.

command_syntax_tests.zip (5.9 KB)

The script fails early on with a syntax error. This is because the bash operator ! means something different in Matlab and Octave. It sends a command to the system shell in Matlab.
After changing the ! to ~, I see the following in Matlab R2021a:

>> main_test_script
    @ : FAILED 3/4 tests
cmd_fcn @ arg1;: evaluation error: 'arg1' is not a valid base class.
cmd_fcn@arg1;: evaluation error: 'arg1' is not a valid base class.
cmd_fcn@ arg1;: evaluation error: 'arg1' is not a valid base class.
   && : OK
    & : OK
    * : OK
    + : OK
    - : OK
   .* : OK
   ./ : OK
   .\ : OK
   .^ : OK
    / : OK
    : : OK
    < : OK
   <= : OK
   == : OK
    > : OK
   >= : OK
    \ : OK
    ^ : OK
    | : OK
   || : OK
   ~= : OK
    ' : FAILED 2/4 tests
cmd_fcn ' arg1;: evaluation error: Error: Character vector is not terminated properly.
cmd_fcn 'arg1;: evaluation error: Error: Character vector is not terminated properly.
   .' : FAILED 2/4 tests
cmd_fcn .' arg1;: evaluation error: Invalid expression. Check for missing multiplication operator, missing or unbalanced delimiters, or other syntax error. To construct matrices, use brackets instead of parentheses.
cmd_fcn .'arg1;: evaluation error: Error: Character vector is not terminated properly.
    ~ : OK
   .+ : FAILED 2/4 tests
cmd_fcn.+arg1;: evaluation error: Invalid use of operator.
cmd_fcn.+ arg1;: evaluation error: Invalid use of operator.
   .- : FAILED 2/4 tests
cmd_fcn.-arg1;: evaluation error: Invalid use of operator.
cmd_fcn.- arg1;: evaluation error: Invalid use of operator.
  .** : FAILED 2/4 tests
cmd_fcn.**arg1;: evaluation error: Invalid use of operator.
cmd_fcn.** arg1;: evaluation error: Invalid use of operator.
   != : FAILED 3/4 tests
cmd_fcn != arg1;: evaluation error: Invalid use of operator.
cmd_fcn!=arg1;: evaluation error: Invalid use of operator.
cmd_fcn!= arg1;: evaluation error: Invalid use of operator.
   ** : FAILED 2/4 tests
cmd_fcn**arg1;: evaluation error: Invalid use of operator.
cmd_fcn** arg1;: evaluation error: Invalid use of operator.
    ! : OK
   ++ : OK
   -- : OK
    $ : OK
    ? : OK
    ` : OK
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 14)
In main_test_script (line 116) 
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 37)
In main_test_script (line 116) 
   &= : FAILED 2/4 tests
cmd_fcn&=arg1;: evaluation error: Incorrect use of '=' operator. Assign a value to a variable using '=' and compare values for equality using '=='.
cmd_fcn&= arg1;: evaluation error: Incorrect use of '=' operator. Assign a value to a variable using '=' and compare values for equality using '=='.
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 14)
In main_test_script (line 116) 
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 37)
In main_test_script (line 116) 
  **= : FAILED 2/4 tests
cmd_fcn**=arg1;: evaluation error: Invalid use of operator.
cmd_fcn**= arg1;: evaluation error: Invalid use of operator.
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 14)
In main_test_script (line 116) 
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 37)
In main_test_script (line 116) 
   *= : FAILED 2/4 tests
cmd_fcn*=arg1;: evaluation error: Incorrect use of '=' operator. Assign a value to a variable using '=' and compare values for equality using '=='.
cmd_fcn*= arg1;: evaluation error: Incorrect use of '=' operator. Assign a value to a variable using '=' and compare values for equality using '=='.
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 14)
In main_test_script (line 116) 
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 37)
In main_test_script (line 116) 
   += : FAILED 2/4 tests
cmd_fcn+=arg1;: evaluation error: Incorrect use of '=' operator. Assign a value to a variable using '=' and compare values for equality using '=='.
cmd_fcn+= arg1;: evaluation error: Incorrect use of '=' operator. Assign a value to a variable using '=' and compare values for equality using '=='.
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 14)
In main_test_script (line 116) 
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 37)
In main_test_script (line 116) 
   -= : FAILED 2/4 tests
cmd_fcn-=arg1;: evaluation error: Incorrect use of '=' operator. Assign a value to a variable using '=' and compare values for equality using '=='.
cmd_fcn-= arg1;: evaluation error: Incorrect use of '=' operator. Assign a value to a variable using '=' and compare values for equality using '=='.
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 14)
In main_test_script (line 116) 
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 37)
In main_test_script (line 116) 
 .**= : FAILED 2/4 tests
cmd_fcn.**=arg1;: evaluation error: Invalid use of operator.
cmd_fcn.**= arg1;: evaluation error: Invalid use of operator.
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 14)
In main_test_script (line 116) 
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 37)
In main_test_script (line 116) 
  .*= : FAILED 2/4 tests
cmd_fcn.*=arg1;: evaluation error: Incorrect use of '=' operator. Assign a value to a variable using '=' and compare values for equality using '=='.
cmd_fcn.*= arg1;: evaluation error: Incorrect use of '=' operator. Assign a value to a variable using '=' and compare values for equality using '=='.
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 14)
In main_test_script (line 116) 
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 37)
In main_test_script (line 116) 
  .+= : FAILED 2/4 tests
cmd_fcn.+=arg1;: evaluation error: Invalid use of operator.
cmd_fcn.+= arg1;: evaluation error: Invalid use of operator.
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 14)
In main_test_script (line 116) 
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 37)
In main_test_script (line 116) 
  .-= : FAILED 2/4 tests
cmd_fcn.-=arg1;: evaluation error: Invalid use of operator.
cmd_fcn.-= arg1;: evaluation error: Invalid use of operator.
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 14)
In main_test_script (line 116) 
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 37)
In main_test_script (line 116) 
  ./= : FAILED 2/4 tests
cmd_fcn./=arg1;: evaluation error: Incorrect use of '=' operator. Assign a value to a variable using '=' and compare values for equality using '=='.
cmd_fcn./= arg1;: evaluation error: Incorrect use of '=' operator. Assign a value to a variable using '=' and compare values for equality using '=='.
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 14)
In main_test_script (line 116) 
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 37)
In main_test_script (line 116) 
  .\= : FAILED 2/4 tests
cmd_fcn.\=arg1;: evaluation error: Incorrect use of '=' operator. Assign a value to a variable using '=' and compare values for equality using '=='.
cmd_fcn.\= arg1;: evaluation error: Incorrect use of '=' operator. Assign a value to a variable using '=' and compare values for equality using '=='.
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 14)
In main_test_script (line 116) 
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 37)
In main_test_script (line 116) 
  .^= : FAILED 2/4 tests
cmd_fcn.^=arg1;: evaluation error: Incorrect use of '=' operator. Assign a value to a variable using '=' and compare values for equality using '=='.
cmd_fcn.^= arg1;: evaluation error: Incorrect use of '=' operator. Assign a value to a variable using '=' and compare values for equality using '=='.
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 14)
In main_test_script (line 116) 
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 37)
In main_test_script (line 116) 
   /= : FAILED 2/4 tests
cmd_fcn/=arg1;: evaluation error: Incorrect use of '=' operator. Assign a value to a variable using '=' and compare values for equality using '=='.
cmd_fcn/= arg1;: evaluation error: Incorrect use of '=' operator. Assign a value to a variable using '=' and compare values for equality using '=='.
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 14)
In main_test_script (line 116) 
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 37)
In main_test_script (line 116) 
   \= : FAILED 2/4 tests
cmd_fcn\=arg1;: evaluation error: Incorrect use of '=' operator. Assign a value to a variable using '=' and compare values for equality using '=='.
cmd_fcn\= arg1;: evaluation error: Incorrect use of '=' operator. Assign a value to a variable using '=' and compare values for equality using '=='.
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 14)
In main_test_script (line 116) 
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 37)
In main_test_script (line 116) 
   ^= : FAILED 2/4 tests
cmd_fcn^=arg1;: evaluation error: Incorrect use of '=' operator. Assign a value to a variable using '=' and compare values for equality using '=='.
cmd_fcn^= arg1;: evaluation error: Incorrect use of '=' operator. Assign a value to a variable using '=' and compare values for equality using '=='.
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 14)
In main_test_script (line 116) 
Warning: Identifier 'cmd_fcn' in the evaluated statement does not refer to the variable. In a future release, using an identifier different to how it is used in the file will not be supported. 
> In test_computed_assign_op>test_code (line 77)
In test_computed_assign_op (line 37)
In main_test_script (line 116) 
   |= : FAILED 2/4 tests
cmd_fcn|=arg1;: evaluation error: Incorrect use of '=' operator. Assign a value to a variable using '=' and compare values for equality using '=='.
cmd_fcn|= arg1;: evaluation error: Incorrect use of '=' operator. Assign a value to a variable using '=' and compare values for equality using '=='.

Changed files:
command_syntax_tests.zip (5.8 KB)

Oops. I guess I would have expected a syntax error for those uses but not a shell escape unless ! was at the beginning of a statement. Just out of curiosity, what is picked up as the shell escape arguments? Everything until the end of the line?

No. There were still open brackets that haven’t been closed before the bash operator. So it failed because of a syntax error.

OK, here is an updated test function. It’s all in one file now. I hope with this version we can at least capture the current behavior for both Octave and Matlab. I think I have all the “expected” (i.e., current behavior) results for Octave correct now.

main_test_fun.m (6.3 KB)

Some endif made it into the test function. After replacing them by end, I see the following in Matlab R2021a:

>> main_test_fun
   && : OK
    & : OK
    * : OK
    + : OK
    - : OK
   .* : OK
   ./ : OK
   .\ : OK
   .^ : OK
    / : OK
    : : OK
    < : OK
   <= : OK
   == : OK
    > : OK
   >= : OK
    ^ : OK
    | : OK
   || : OK
   ~= : OK
    ' : OK
   .' : OK
    ! : OK
    $ : OK
    ? : OK
    ` : OK
    @ : OK
    \ : OK
    ~ : OK
   .+ : OK
   .- : OK
  .** : OK
   != : OK
   ** : OK
   ++ : OK
   -- : OK
   &= : OK
  **= : OK
   *= : OK
   += : OK
   -= : OK
 .**= : OK
  .*= : OK
  .+= : OK
  .-= : OK
  ./= : OK
  .\= : OK
  .^= : OK
   /= : OK
   \= : OK
   ^= : OK
   |= : OK

Thanks.

The following text is copied from comment #39 of the bug report GNU Octave - Bugs: bug #60882, error parsing command syntax [Savannah] :

I think the only question left is whether to handle our extended set of operators (++, +=, etc.) specially or in a fully Matlab-compatible way when parsing command syntax. Since Matlab doesn’t have += (for example), I believe that an expression like

foo +=

is parsed as a command-style function call regardless of what follows the +=. If we do the same for Octave, then it is impossible to use += as an operator unless the expression is written with no space after the initial identifier:

foo+=something    ## OP=
foo+= something   ## OP=
foo +=something   ## command-style function call
foo += something  ## command-style function call

That seems bad to me, so I left the special treatment of these operators in Octave so that the only one that causes command-style function parsing is

foo +=something

similar to the way that both Octave and Matlab handle other binary operators (i.e., only the foo +something is handled as command-style function syntax).

Since ++ is a unary operator and + by itself is both a binary and prefix unary operator, the rules are a little different. In Matlab, the parsing of expressions involving repeated + characters is a bit weird:

foo ++            %% command-style function call
foo ++ something  %% command-style function call
foo++something    %% expression: foo + (+something)
foo++ something   %% expression: foo + (+something)
foo ++something   %% command-style function call

See test function to verify current behavior of Matlab and Octave that was posted earlier in this discussion. Note that in the results that Markus posted for Matlab, the “OK” doesn’t mean that Octave and Matlab behavior is the same, just that we know what the behaviors are. Looking at the function itself shows that there are currently differences between Octave and Matlab for the following operators:

@
\
~
.+
.-
.**
!=
**
++
--
all the OP= operators

It would probably be fairly painless to phase out and eventually remove

.+
.-
.**
**

We are already eliminating the problem with backslash as a continuation character outside of character strings. I think we can make the behavior of @ and ~ compatible with Matlab. Changing != (and !) to be compatible would be very unpleasant.

Since ++ and -- are not binary operators anyway, we can get closer to the Matlab behavior if we reject them as two-character tokens in some cases depending on the surrounding context, return just + (or -) from the lexer, and continue parsing beginning with the next + (or -) character from the input stream. That should provide compatibility with Matlab by parsing expressions like

foo++something

as

foo + (+something)

Sounds good to me :slightly_smiling_face:

Agree to remove them as soon as possible. Already in Octave 7 without deprication period?

Agree, please keep them :slightly_smiling_face: One case were I wished Matlab would finally follow Octave :sweat_smile:

Thanks for taking so much time to get the balance between Matlab compatiblity and Octave extensions done :slightly_smiling_face: :+1:

I deprecated **, .**, .+, and .- for Octave 7 but won’t remove them completely until Octave 9.

2 Likes