Reorganization of experimental data

Question

Daniele Sonaglioni 2021-7-30

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/889302-reorganization-of-experimental-data

评论： Star Strider 2021-8-2

采纳的回答： Star Strider

在 MATLAB Online 中打开

Hello everybody,

I have a problem with the reorganization process of my experimental data.

Attached there is an example of a typical data set

              0              0        -93.964    -0.00976171
              1          0.001        -93.964    -0.00976627
              2          0.002        -93.965    -0.00991459
              3          0.003        -93.965    -0.00997991
              4          0.004        -93.963     -0.0100311
              5          0.005        -93.962     -0.0101206
              6          0.006        -93.964     -0.0101955
              7          0.007        -93.961     -0.0098519
              8          0.008        -93.958    -0.00946246
              9          0.009        -93.945    -0.00970383
             10           0.01        -93.921     -0.0105877
             11          0.011        -93.884     -0.0125595
             12          0.012        -93.829      -0.015714
             13          0.013        -93.755     -0.0194494
Results:
Sample:
  2021_07_29_cool_variab_heat_fix_59603. 100.0000e-06 mg
Curve Name:
  ]61[&2021_07_29_cool_variab_heat_fix_59603
Curve Values:
          Index              t             Ts          Value
                           [s]           [°C]           [mW]
              0              0        -93.205   -0.000370132
              1           0.01        -93.365    -0.00237034
              2           0.02        -93.578    -0.00514436
              3           0.03        -93.713    -0.00678095
              4           0.04        -93.800    -0.00771421
              5           0.05        -93.857    -0.00846012
              6           0.06        -93.893    -0.00883113
              7           0.07        -93.917    -0.00909422
              8           0.08        -93.932    -0.00928991
              9           0.09        -93.944    -0.00943325
             10            0.1        -93.949    -0.00949218
             11           0.11        -93.954    -0.00950072
             12           0.12        -93.957    -0.00963767
             13           0.13        -93.958     -0.0096308
             14           0.14        -93.960    -0.00962155
             15           0.15        -93.961    -0.00964608
Results:
Sample:
  2021_07_29_cool_variab_heat_fix_59603. 100.0000e-06 mg
________________________________________________________________________________
Not signed                                                        STARe SW 16.10

I want to remove all the written texts and empty lines and I want to reorganize the blocks of data in a way to invert their order.

Do you have any suggestion?

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Star Strider 2021-7-30

Attach a ‘typical data set’ as a .txt or .mat file.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Star Strider 2021-7-30

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/889302-reorganization-of-experimental-data#answer_757612

编辑：Star Strider 2021-8-1

在 MATLAB Online 中打开

The data file you attached does not appear to bear a strong resemblance to the data file you quoted.

I chose to use textscan here because I have experience using it with files with interrupted formats, however this file appears to have no interruptions:

fidi = fopen('typical_data_set.txt','rt');
C = textscan(fidi, '%f%f%f%f %*f%*f%*f%*f%*f%*f', 'HeaderLines',1, 'CollectOutput',1);
fclose(fidi);
M = cell2mat(C);

Here ‘M’ is a (44006x4) double array.

A file with interruptions would simply require a while loop to loop through it, saving the individual sections. For an example, see: How to skip lines of Data in the middle of a text file . I believe that file was similar to the one you quoted.

EDIT — (1 Aug 2021 at 00:15)

Removed the generic code example. Nothing else changed.

.

14 个评论
显示 12更早的评论隐藏 12更早的评论

Star Strider 2021-7-31

在 MATLAB Online 中打开

I can only detect one block of data using:

fidi = fopen('typical_data_set.txt','rt');
k1 = 1;
while ~feof(fidi)
    C = textscan(fidi, '%f%f%f%f %*f%*f%*f%*f%*f%*f', 'HeaderLines',1, 'CollectOutput',1);
    M = cell2mat(C);
    if isempty(M)                                       % Empty Matrix Indicates End-Of-File
        fprintf(1,'Reading finished, %d Sections\n',k1-1)
        break
    end
    D{k1,:} = M;
    fseek(fidi, 0, 0);
    k1 = k1 + 1
end
fclose(fidi);
Out = cell2mat(D);

This code would detect a line of text that did not match the numeric format descriptors, stop, and re-start with the next block. (It then optionally combines the blocks into a single matrix if desired.) I have tested it on different files over the several years since I wrote it, and it consistently works. There is only one block of data in the posted file, and only one header line.

I have not gone through the file in detail by hand (it has 44006 rows), so I have no idea where the ‘two lines of text’ are. I would appreciate your displaying them and the rows and row numbers where they occur. I cannot detect their presence.

.

Star Strider 2021-7-31

在 MATLAB Online 中打开

This turned out to be something of an adventure!

My code works, however it needs to be modified a bit for your file:

fidi = fopen('typical_data_set_2ndversion.txt','rt');
k1 = 1;
while ~feof(fidi)
    readline = fgetl(fidi);
    if strcmp(readline,'Results:')
        C = textscan(fidi, '%f%f%f%f %*f%*f%*f%*f%*f%*f', 'HeaderLines',9, 'CollectOutput',1);
    else
        C = textscan(fidi, '%f%f%f%f %*f%*f%*f%*f%*f%*f', 'HeaderLines',0, 'CollectOutput',1);
    end
    M = cell2mat(C);
    fprintf(1,'\tSection %2d: (%4d x %d)\n', k1, size(M))
    if isempty(M)                                           % Empty Matrix Indicates End-Of-File
        fprintf(1,'Reading finished, %d Sections\n',k1-1)
        break
    end
    D{k1,:} = M;
    fseek(fidi, 0, 0);
    k1 = k1 + 1;
end
fclose(fidi);
Dflip = flip(D);                                            % Reverse Section Order
% Out = cell2mat(D);                                          % Optional Concatenation

The end result is 56 segments of varying row length, all with 4 columns. (The section information is not retained, only the numeric information.) I did not go through all 56 results, however the few I sampled appear to be correct.

.

Star Strider 2021-8-1

在 MATLAB Online 中打开

Thank you.

If you check, that is exactly what my code does. The individual block contents aree not reordered in any way. Only their order in the ‘D’ cell array are changed, so that the last block becomes the first (for example ‘D{1}’ becomes ‘Dflip{56}’, ‘D{56}’ becomes ‘Dflip{1}’) and so for the rest. I checked that as well as selected individual blocks before I posted my code.

Adding these two lines after the ‘Dflip’ assignment:

Check1 = [D{1}(1:5,:); NaN(1,4); Dflip{end}(1:5,:)]
Check2 = [D{end}(1:5,:); NaN(1,4); Dflip{1}(1:5,:)]

produces:

Check1 =
0000    0.0001   25.1940    0.0006
0000    0.0002   25.1910    0.0006
0000    0.0003   25.1900    0.0005
0000    0.0004   25.1910    0.0004
0000    0.0005   25.1820    0.0004
       NaN       NaN       NaN       NaN
0000    0.0001   25.1940    0.0006
0000    0.0002   25.1910    0.0006
0000    0.0003   25.1900    0.0005
0000    0.0004   25.1910    0.0004
0000    0.0005   25.1820    0.0004

Check2 =
         0         0  -93.9600   -0.0104
    1.0000    0.0100  -93.9590   -0.0103
    2.0000    0.0200  -93.9580   -0.0104
    3.0000    0.0300  -93.9590   -0.0103
    4.0000    0.0400  -93.9590   -0.0103
       NaN       NaN       NaN       NaN
         0         0  -93.9600   -0.0104
    1.0000    0.0100  -93.9590   -0.0103
    2.0000    0.0200  -93.9580   -0.0104
    3.0000    0.0300  -93.9590   -0.0103
    4.0000    0.0400  -93.9590   -0.0103

The NaN lines are simply used to separate the matrix samples in order to make it easier to read.

Note that ‘D{end}’ is ‘D{56}’ in this file, and similarly for the ‘Dflip’ elements.

So, the order of each block is flipped as requested, not the contents. This additional bit of code (not necessary for the rest of the code) demonstrates that.

.

Daniele Sonaglioni 2021-8-2

I have just earned my master degree in condensed matter physics and I am now focusing in soft matter physics, especially in the physics of proteins. In particular, I am specialized in differential scanning calorimetry.

Your field is very interesting: I also want to make some research in the medical field although you are more interested on "large scale" phenomena while I prefer smaller scales.

I think that coding is very useful in every field of study, not only physics and I am trying to lear as much as possible to make deeper data analysis to better understand what I am studying.