formating csv file for quick access of list of integers

Jyrki Majamaki

I want to read in lists of integers in OML. I figured that I might use a csv-file for it. 

The data is organized in set ids and contains a varying number of integers, for exampe a set id 100 could be

set 100 = 1 20 30 44 500 1234

and maybe set 556 could be

set 556 = 12000 33456 120000 140000 160000

 I am still rather free how the data is written into csv-file as I am writing from a Tcl-routine running in Hyperview/Hypermesh, so I could be like:

100       556

1           12000

20         33456

30         120000

44         140000

500       160000

1234

first row would be defining the set id and the rows below in the same column define then the ids belonging into the set. However this is a little bit tedious to write from tcl, it would be easier if I transposed it, but the problem might be then the line length as the number of ids can be rather high. I could write one set

over multiple lines:

100	1	2	3	4	5	6	7	8
100	9	10	11	12	13
200	22	25	26	27
300	30	31	32	33	34	35	36	37	38
300	39	40	41	42	43	44	45	46
300	47	48	49	50

Here set id 100 would be over two lines, set 200 needs only one line and 300 would be over 3 lines. I would then only need to concate the lists together, but I am not sure how to accomplish that in oml.

Can somebody give me a tip for a handy one-liner, I tend to think the last csv-format would be easiest to write from tcl but how do I combine the id lists from successive lines, in the end I might prefer to have something like:

set(100) = [1;2;3;4;5;6;7;8;9;10;11;12;13]  

 

Jyrki Majamaki

Ok, two answer at least part of my own question - it is not so simple to find examples for oml.

Anyway if I went for the last format, repeating the set id always on column 1. I would get there 

with:

setdata=csvread('sets.csv');  tmpsetdata=setdata(setdata(:,1) == 100,:); %gets matrix part where the first column is 100 (set id = 100) tmpsetdata(:,1) = 0; %sets the first column to 0 tmpsetdata=reshape(tmpsetdata,[],1); %reshape to 1 x [] matrix finalset=tmpsetdata(tmpsetdata(:,1) > 0,:); %omit the 0 values  

so, at least it is a five liner for starters.

João Marabisa

Jyrki Majamaki said:

Ok, two answer at least part of my own question - it is not so simple to find examples for oml.

Anyway if I went for the last format, repeating the set id always on column 1. I would get there 

with:

setdata=csvread('sets.csv');  tmpsetdata=setdata(setdata(:,1) == 100,:); %gets matrix part where the first column is 100 (set id = 100) tmpsetdata(:,1) = 0; %sets the first column to 0 tmpsetdata=reshape(tmpsetdata,[],1); %reshape to 1 x [] matrix finalset=tmpsetdata(tmpsetdata(:,1) > 0,:); %omit the 0 values  

so, at least it is a five liner for starters.

Hi @Jyrki Majamaki,

Let me understand: Will you generate one CSV using TCL and want to use Compose to read this CSV and have the values of each set into arrays?

The OML provided by you already work for the format having the sets in multiple lines

What is your question now?

RSGarciarivas

Hi Jyrki,

As Joao said, the commands you're using are correct, so I suppose you want to generalize it for sets that span multiple lines without explicitly declaring the set ID on each line, as we discussed previously?

In that case your csv file contains each set ID once, like this:

100,1,2,3,4,5,6,7,8
,9,10,11,12,13
200,22,25,26,27
300,30,31,32,33,34,35,36,37,38
,39,40,41,42,43,44,45,46
,47,48,49,50

And you could use something like this:

% Read data from CSV
filename = '.\sets.csv';
data = csvread(filename, 'emptyvalue', NaN);

% Get all sets (remove NaN)
setIDs = unique(data(:,1));
setIDs = setIDs(~isnan(setIDs));

% Get matrix of set definitions' locations
setDefs = data(:,1) == setIDs';

% Get number of sets and initialize cell for storing contents
nSets = size(setDefs,2);
sets = cell(nSets,1);

% Get elements for each set
for i = 1:nSets
    
    % Get current set definition start
    startIdx = find(setDefs(:,i));
    % Get current set definition end
    if i < nSets
        endIdx = find(setDefs(:,i+1))-1;
    else
        endIdx = size(setDefs,1);
    end
    
    % Reshape set definition into single row and remove NaN
    thisSet = reshape(data(startIdx:endIdx,:), 1, []);
    sets{i} = thisSet(~isnan(thisSet));
    
end

Where sets.csv is the file you created in tcl.

Hope this works as a good indexing example as well.

Rafael

Jyrki Majamaki

I realized that I have quite a bit of problem with my approach. Using the batchprocess of Hyperview, takes quite a bit of time and the problem is that the oml bit moves on. If I start the batch process, is there a way to make oml wait for the result? I could delete that transfer csvfile and make a while loop in oml to check against the existence of that file, but that is also a bit uncertain and has a taste of infinite loop.

RSGarciarivas

Jyrki Majamaki said:

I realized that I have quite a bit of problem with my approach. Using the batchprocess of Hyperview, takes quite a bit of time and the problem is that the oml bit moves on. If I start the batch process, is there a way to make oml wait for the result? I could delete that transfer csvfile and make a while loop in oml to check against the existence of that file, but that is also a bit uncertain and has a taste of infinite loop.

Hi Jyrki,

Yes, you could go on a loop and constantly check the file's existence. Not the most elegant approach but it would work.

A more elegant approach would be to use command system() and run your tcl script in HyperWorks from there. Execution mode can be set to synchronous using the keyword 'sync'. This command will stop the script from further execution until process ends and the file has been created.

When I automate workflows that involve other Altair products, I always try to use callaltairbatch as a first option. I can't recall right now if 'sync' can be given as a keyword there just the same, but it's worth a try.

Rafael