this is a question related to what might look like an unexpected result of inferior efficiency of the setdiff function in finding complimentry set of indices of a vector given one set, compared to using find function twice.
While setdiff seems to be improving efficiency compared to “2 * find()” for relatively small sets of data up to ~1000, it quickly looses comparative efficiency as data sets get larger.
It seems counterintuitive. Manipulating ordered sets compared to searching random vector - granted find function most likely got quite a bit more attention from developers.
Got an impression from a recent python refresh class that set operations were highly efficient granted memory penalty. Well Octave is not python, but algorythms should be similar ?
In a recent excercise for an ML classification problem (result of the model is either 0 = failure or 1 = success) I came up with an idea of using setdiff instead of using find function twice. The results were discouraging. Scalability of ML models is a serious consideration, so I hope this question would not be considered to be irrelevant.
Comments / clarifications / explanations of errors / misunderstanding on my part would be greatly appreciated. Formal explanation of comparative algorythm complexity of O(find()) vs O(setdiff) woud be awesome as well.
Demo code: ===== (edit: for binary selection complimentary index is not required - initialize answer with all 0, and find index for all 1’s and replace … what about multi-classifiers ? thinking …
% compare two finds and find and setdifff
order_size = 8; % 8 max size of the vector to search to finish in resonable time
size_diff = [order_size,2]; % storage of difference in performance for two methods
for i = 1:order_size
tmp = rand(10^i,1); % generate set of random numbers between 0 and 1
full_index = 1:length(tmp); % to avoid creation inside setdiff and associated time penalty
tic(); index = find(tmp < 0.5); compl_index = find(tmp >=0.5); find_twice = toc(); tic(); index = find(tmp < 0.5); compl_index = setdiff(full_index, index); % find complimentary index find_n_diff = toc(); % how much slower find_twice_is_faster = find_n_diff / find_twice; size_diff(i,1) = 10^i; % size of the vector to search size_diff(i,2) = find_twice_is_faster; % how many times is setdiff slower