Help with looping

I am analyzing some weather data by plotting graphs and looking at trends. The data is being collected by a Campbell Scientific Datalogger CR1000X. Various datapoints are being collected such as air temperature, humidity, leaf temperature, net radiation etc. I still have to do some simple equations to get other answers; like:
Air temp - Leaf Temp = Leaf Temperature Depression

I have to perform these calculations several times and would like a way of automating the process. I just don’t know how to make a proper loop. This is what I have. DATA.LTD is a structure that has all of my values saved in a 48X1 format. The same is true for DATA.HMPlo… and DATA.Canopy…
I have 24 files that I need to perform this calculation on. I also have other calculations I need to perform on these structures.
`DATA.LTD = (DATA.HMPlo_T_Avg - DATA.Canopy_T_Avg);
DATA1.LTD = (DATA1.HMPlo_T_Avg - DATA1.Canopy_T_Avg);
DATA2.LTD = (DATA2.HMPlo_T_Avg - DATA2.Canopy_T_Avg);
DATA3.LTD = (DATA3.HMPlo_T_Avg - DATA3.Canopy_T_Avg);

DATA21.LTD = (DATA21.HMPlo_T_Avg - DATA21.Canopy_T_Avg);
DATA22.LTD = (DATA22.HMPlo_T_Avg - DATA22.Canopy_T_Avg);
DATA23.LTD = (DATA23.HMPlo_T_Avg - DATA23.Canopy_T_Avg);`

How do I create a loop that can perform the math, advance the structure number, and continue until it reaches the end?

For the code you have, maybe like this

for i = 1:23
  eval (sprintf ("DATA%d.LTD = (DATA%d.HMPlo_T_Avg - DATA%d.Canopy_T_Avg);", n, n, n));
end

You’ll have to special-case the first line, or renumber it to use 0.

I have not tested the above code though. Please check if it works. Maybe print it without eval and only add eval after examining the output from sprintf.

I used the code that you have provided. I think i used it in the right way…

DATA1.LTD = (DATA1.HMPlo_T_Avg - DATA1.Canopy_T_Avg);

for i = 1:23
  eval (sprintf ("DATA%d.LTD = (DATA%d.HMPlo_T_Avg - DATA%d.Canopy_T_Avg);", n, n, n));
end

I think I still have to have a bit of the code I want to change in the beginning, or the code that you wrote won’t do anything. Still it seems octave did not recognize the character “n” at the end of your code, as this is the message I received upon trying it:

error: 'n' undefined near line 4, column 78
error: called from
    AdvNumLoop at line 4 column 3

Sorry, that is what happens when I type code without being able to test it. Change the loop variable to n instead of i and it should at least give you results.

The sprintf call creates a string, in this case an Octave assignment statement. The eval function evaluates that string like an Octave command.

Also, the loop is already running from 1 to 23. Your special case statement should not use those numbers, or adjust the loop accordingly.

I changed the loop variable to n as you instructed, but then this happened:

error: 'DATA2' undefined near line 1, column 14
error: called from
    AdvNumLoop at line 4 column 3

Now I can see that it already changed DATA1 to DATA2, but now it doesn’t recognize it as a structure.

You have this line in your original code:

 DATA2.LTD = (DATA2.HMPlo_T_Avg - DATA2.Canopy_T_Avg);

Where is DATA2 defined for that line? If you’re loading from file, then the eval sprintf line should happen after it’s been loaded.

Use the loop as an example to build your own. The general idea is to use sprintf to construct a valid Octave statement (meaning, all variables on the RHS of an assignment should already exist by that point), then eval will execute that statement.

Also: Octave can make arrays of structures, so all your DATA variables could be subscripted like DATA(1).x = DATA(1).y + DATA(1).z where x,y,z are the member field names like your temperatures. Is this of use to you?

Yes, and I would strongly recommend using structure arrays or cell arrays of structures instead of eval-ing new numbered variables into existence.

1 Like

OK, I see what happened now. I reloaded all of the structures and renamed DATA to DATA0. Now when I run the code it works perfectly. I was even able to adapt it to some other equations I need to perform on these 24 files.

DATA0.CWSI = (DATA0.LTD - 0.154)/(-3.33 - 0.154);
for n = 0:23
  eval (sprintf ("DATA%d.CWSI = (DATA%d.LTD - 0.154)/(-3.33 - 0.154);", n, n));
end

and

DATA0.esat= 0.6108.*(exp(17.27.*DATA0.HMPlo_T_Avg ./(DATA0.HMPlo_T_Avg +237.3)));  % Saturation Vapor Pressure
DATA0.ea= DATA0.esat.*(DATA0.HMPlo_RH_Avg/100);                      % Actual Vapor Pressure
DATA0.VPD= DATA0.esat-DATA0.ea;                             % Vapor Pressure Deficit

for n = 0:23
  eval (sprintf ("DATA%d.esat = 0.6108 .*(exp(17.27 .*DATA%d.HMPlo_T_Avg ./(DATA%d.HMPlo_T_Avg +237.3)));", n, n, n));
  eval (sprintf ("DATA%d.ea = DATA%d.esat .*(DATA%d.HMPlo_RH_Avg/100);", n, n, n));
  eval (sprintf ("DATA%d.VPD = DATA%d.esat - DATA%d.ea;", n, n, n));
  end

Can I adapt this code to plot multiple pieces of data into a single figure?
I’m currently using code that calls each piece individually:

plot(DATA0.TS(dd),DATA0.LTD(dd),'b.', 'markersize', 10);
hold
plot(DATA1.TS(dd),DATA1.LTD(dd),'b.', 'markersize', 10);
plot(DATA2.TS(dd),DATA2.LTD(dd),'b.', 'markersize', 10);
plot(DATA3.TS(dd),DATA3.LTD(dd),'b.', 'markersize', 10);
plot(DATA4.TS(dd),DATA4.LTD(dd),'b.', 'markersize', 10);
plot(DATA5.TS(dd),DATA5.LTD(dd),'b.', 'markersize', 10);
plot(DATA6.TS(dd),DATA6.LTD(dd),'b.', 'markersize', 10);
plot(DATA7.TS(dd),DATA7.LTD(dd),'b.', 'markersize', 10);
plot(DATA8.TS(dd),DATA8.LTD(dd),'b.', 'markersize', 10);
plot(DATA9.TS(dd),DATA9.LTD(dd),'b.', 'markersize', 10);
plot(DATA10.TS(dd),DATA10.LTD(dd),'b.', 'markersize', 10);
plot(DATA11.TS(dd),DATA11.LTD(dd),'b.', 'markersize', 10);
plot(DATA12.TS(dd),DATA12.LTD(dd),'b.', 'markersize', 10);
plot(DATA13.TS(dd),DATA13.LTD(dd),'b.', 'markersize', 10);
plot(DATA14.TS(dd),DATA14.LTD(dd),'b.', 'markersize', 10);
plot(DATA15.TS(dd),DATA15.LTD(dd),'b.', 'markersize', 10);
plot(DATA16.TS(dd),DATA16.LTD(dd),'b.', 'markersize', 10);
plot(DATA17.TS(dd),DATA17.LTD(dd),'b.', 'markersize', 10);
plot(DATA18.TS(dd),DATA18.LTD(dd),'b.', 'markersize', 10);
plot(DATA19.TS(dd),DATA19.LTD(dd),'b.', 'markersize', 10);
plot(DATA20.TS(dd),DATA20.LTD(dd),'b.', 'markersize', 10);
plot(DATA21.TS(dd),DATA21.LTD(dd),'b.', 'markersize', 10);
plot(DATA22.TS(dd),DATA22.LTD(dd),'b.', 'markersize', 10);
plot(DATA23.TS(dd),DATA23.LTD(dd),'b.', 'markersize', 10);

I think that is what I am already doing. I am very new to Octave as a program, and just as new to coding/programing. I have only been using Octave since July so I know I don’t have the vocabulary. I have been learning as I go and getting pieces of code from my boss. Unfortunately, I will only have him as a resource until dec31, as he is retiring.

I see what you’re trying to do, but there’s a big difference between this:

x12 = y12 + z12
x13 = y13 + z13

and this

x(12) = y(12) + z(12)
x(13) = y(13) + z(13)

The first will set up many independent variables named x-something The second will set up only one big array variable x.

To see the difference, type whos. The parentheses aren’t just a notation thing but make a big difference.

The second is a lot safer all round.

Also convenient. Say you need to increment all the x-values. You could write 24 different x-something += 1 statements or you could write just one x += 1 statement and have it magically apply to all 24.

My original idea to use eval was taking your original nomenclature too literally. Using subscripts inside parentheses is the better way to organize your code.

PS: https://docs.octave.org/ has several resources available for you. Use any of the octave-version.pdf files for the full reference. Use quickref.pdf for a 2-page crash course.

I think I see what you are telling me. My boss gave me a bit of code that I have been using to retrieve the data from the CS datalogger. It only works on one file at a time though and gives me the structure DATA0. So, I have been changing the structure number 24 times to read all of the files, then I have 24 structures I have to deal with.

fclose('all')

fname = fopen('C:\Users\ARSTech\Desktop\CNL_supp\TOA5_SEB0.dat');         

L1 = fgetl(fname);                                %% Read 1st line and asign L1
L2 = fgetl(fname);                                %% Read 2nd line and assign L2
fgetl(fname);                                     %% Advance 1 line
fgetl(fname);                                     %% Advance 1 line
clear ans L1                                      %% Clear variables L1 and ans
%% program is still held at line 5 of file
vars = strsplit(L2,',')                           %% parse out fields in Line 2
vars = strrep(vars,'"','')                        %% replace " with nothing
vars = strrep(vars,'(','')                        %% replace ( with nothing
vars = strrep(vars,')','')                        %% replace ) with nothing

di = 1;
while ~feof(fname)                                %% While not at end of file
  timei = fscanf(fname,'"%u-%u-%u %u:%u:%f",',6); %% Parse out 6 char of timestamp
  TS(di) = datenum(timei');
  fgetl(fname);                                   %% Advance 1 line
  di = di+1;
endwhile
clear di timeifclose(fname)

dataMat = dlmread('C:\Users\ARSTech\Desktop\CNL_supp\TOA5_SEB0.dat');     %% Should use entire directory
DATA0.TS = TS';                                     %% Transpose of Timestamp
for f = 3:16
  DATA0.(vars{f}) = dataMat(5:end,f);
end

fclose('all')

On a single file, this essentially decodes the 5 lines of header information written by the proprietary CS software and gives me a workable format for manipulation in Octave.

Right. Here is an attempt at creating and calling a function for that sort of thing so you don’t have to do manual copy-paste. Pls go over this code with your boss and it may help you get more comfortable with managing your data files.

This is the function definition based on what you have:

function DATA_OUT = readfromfile (filename)

  fclose('all')

  fname = fopen(filename);         

  L1 = fgetl(fname);                                %% Read 1st line and asign L1
  L2 = fgetl(fname);                                %% Read 2nd line and assign L2
  fgetl(fname);                                     %% Advance 1 line
  fgetl(fname);                                     %% Advance 1 line
  clear ans L1                                      %% Clear variables L1 and ans
  %% program is still held at line 5 of file
  vars = strsplit(L2,',')                           %% parse out fields in Line 2
  vars = strrep(vars,'"','')                        %% replace " with nothing
  vars = strrep(vars,'(','')                        %% replace ( with nothing
  vars = strrep(vars,')','')                        %% replace ) with nothing

  di = 1;
  while ~feof(fname)                                %% While not at end of file
    timei = fscanf(fname,'"%u-%u-%u %u:%u:%f",',6); %% Parse out 6 char of timestamp
    TS(di) = datenum(timei');
    fgetl(fname);                                   %% Advance 1 line
    di = di+1;
  endwhile
  clear di timei
  fclose(fname)
  
  %% FIXME: Why are we reading this data file twice? Can we read it just once
  %% with textscan(), to simultaneously pull both timestamp and data?

  dataMat = dlmread(filename);     %% Should use entire directory
  DATA_OUT.TS = TS';                                     %% Transpose of Timestamp
  for f = 3:16
    DATA_OUT.(vars{f}) = dataMat(5:end,f);
  end

  fclose('all')
endfunction

And here is the intended use case:

datfiles = ls ('C:\Users\ARSTech\Desktop\CNL_supp\*.dat')  ## Please check that this lists your 24 data files. 
for i = 1:rows(datfiles)  # for each file
  filename = deblank (datfiles(i,:));  # throw out extra trailing spaces if any
  DATA(i) = readfromfile (filename);  # this *Should* do the right thing.
endfor

Then you would access the data with, e.g.:

plot(DATA(1).TS(dd),DATA(1).LTD(dd),'b.', 'markersize', 10);
hold on

for i = 2:numel (DATA)
  plot(DATA(i).TS(dd),DATA(i).LTD(dd),'b.', 'markersize', 10);
end

The aim is that tomorrow if you get 200 data files instead of 24, your copy-pasting won’t be the rate-determining step.

The only way you can learn this is by trying it out. Let us know how it goes. Make a backup copy of your dat files so you don’t worry about accidentally deleting them with Octave, then try Octave code.

When I try to use your first block of code as written I am given an error message:

>> ImportCSCC2

error: parse error near line 1 of file C:\Users\ARSTech\Desktop\Cody-ARS\Higher Lea
rning\Octave\ImportCSCC2.m

  invalid parameter list

>>> function DATA_OUT = readfromfile ('C:\Users\ARSTech\Desktop\CNL_supp\TOA5_SEB0.
dat');

   ^

However, if I run the code with the top line commented out along with the endfunction on line 38, it runs like it should. This then gives me the structure DATA_OUT.

When I run your second block of code, I get the error message:

̝ls: cannot access 'C:/Users/ARSTech/Desktop/CNL_supp\*.dat': No such file or direct
ory
error: ls: command exited abnormally with status 2

So I remove the single quotes on the file name, and then:

>> datfiles = ls (C:\Users\ARSTech\Desktop\CNL_supp\*.dat)  ## Please check that th
is lists your 24 data files.
for i = 1:rows(datfiles)  # for each file
  filename = deblank (datfiles(i,:));  # throw out extra trailing spaces if any
  DATA(i) = readfromfile (filename);  # this *Should* do the right thing.
endfor
error: error sourcing selected code
>>

I’m not sure if I should also be changing the ‘filename’ at the DATA(i) = readfromfile (filename); or not. I have tried changing it to the same *.dat that we use at the beginning of block 2, and I get the same error message.
BUT, if I change it to the original filename at the start of block 1(which is the first file in the sequence of 24 files), then I get the error message:

error: 'datfiles' undefined

That is not the right fix at all. The code I gave you turned your sequence of operations into a function that can return a data block like you want given an arbitrary filename like you have. If you remove the first and last line, you’ve turned it back into a sequence like you had, which you had difficulty with. For the second block, if you already have a list of dat files you should be able to feed it to the function to read from. Also avoid directory names with spaces, because that can cause difficult-to-debug errors. Use my code as a starting point but do discuss it with a colleague who knows Octave.

Are you familiar with other languages ? Are you familiar with the concept of functions?
You can’t change

function DATA_OUT = readfromfile (filename)

to

function DATA_OUT = readfromfile ('C:\Users\ARSTech\Desktop\CNL_supp\TOA5_SEB0.
dat');

The token filename is what one would call formal parameter and 'C:\Users\ARSTech\... is an actual parameter.
One of them is a function definition other is a function call.

1 Like

No, I am not familiar with other programing languages. Using octave is my first foray into the programing world. So, no, I’m not really familiar with functions or how they work. I am still learning, and I greatly appreciate the help from the community here; Even if that means I make dumb assumptions like thinking filename means insert filename here…

1 Like

I recommend:

After that you should be able to navigate octave manual for more details etc…

2 Likes

I also recommend:
https://en.wikibooks.org/wiki/Octave_Programming_Tutorial#Mathematical_View

follow the “suggested roadmap” for absolute beginners. it will walk through usage, syntax, scripts, functions, etc.

1 Like

Thank you both! I have been trying to learn as best as I can, but I find the textbooks I have found so far are quite hard for me to understand. I will give these a shot.

1 Like

I also like this one a lot these notes: https://web.sha1.bfh.science/Labs/PWF/Documentation/OctaveAtBFH.pdf

and the related printed book : Introduction to Octave/MATLAB | SpringerLink

c.

1 Like