- With the -v7.3 flag you can store variables > 2GB with compression
- Without the -v7.3 flag (e.g. if the default version is set to -v7 or lower) you have no (or less) compression, but cannot store large arrays.
MATLAB function "save" and "-V7.3"
568 次查看(过去 30 天)
显示 更早的评论
Running R2010a, 64bit(win64). I have a large structure. Up to this point save has worked fine. Efficient, fast, great... I have recently added new data to this structure. not so cool... Now I am getting the message:
Warning: Variable 'iceChart' cannot be saved to a MAT-file whose version is older than 7.3. To save this variable, use the -v7.3 switch. Skipping...
Unfortunately “-v7.3” appears to have some form of compression algorithm embedded, saving and loading these “-v7.3” mat files istaking significant time. Apparently there was a “-nocompression” flag that worked in previous versions of save. But this is no longer supported??? My options are at present seem to be just regenerating the entire data set during each procession session OR I can hack this by having multiple mat files using the old ‘–v6’ flag... I don't particularly like either of these options...
Appreciate any advice on this matter...
Regards Carl
0 个评论
采纳的回答
Jan
2011-9-9
Your observation is correct:
It is surprising, that the compression cannot be disabled manually, because it can be very time-consuming. But this was not the case when the v7.3 format has been developped, because the hard disks have been slower and more expensive. Therefore a compression has been a good idea.
2 个评论
Malcolm Lidierth
2011-10-9
@Jan
See within. Compression appears to be applied only for the first variable in a v7.3 file. The HDF Groups HDFView program shows no compression on other variables. So
dumvar=0;
save(filename,'dumvar','-v7.3');
seems to act as a switch.
更多回答(9 个)
Yair Altman
2013-4-10
Since R2008b, v7.3 compression seems to be done only (and always) for [large?] numeric data, but never for non-numeric data, regardless of their relative placement within the file. The release notes state that this change took place in R2008a, but in practice I still see uncompressed data in R2008a. Compression is only used extensively starting in R2008b.
The internal implementation of v7.3 seems to have changed over the years, making the compression more efficient. Still, the end result is not significantly different than the one suggested above (read on).
For improved performance you can save an HDF5 without compression (Deflate=0) using either the low-level HDF5 primitives, or using the savefast utility (which does it internally). This provides nearly the same performance as v6, while preserving the HDF5 benefits (ability to save objects, 2GB+).
Note that the resulting file size may be much larger compared to v6, especially if the data contains cell or struct arrays. So for simple data types (non-objects, <2GB) v6 might still be preferable due to the reduced I/O and cross-platform compatibility. The only exception to this rule might be if you're constantly accessing parts of data in the file, where v7.3's ability might be more efficient (R2011b and newer).
As a side-note, I find it highly ironic that the proprietary v6/v7 MAT format is more cross-platform-compatible (due to its documentation), compared to the v7.3 format that is supposed to rely on the standard HDF5 format and yet is sufficiently different (in a non-documented manner) to make this format unusable on any platform outside Matlab.
2 个评论
Malcolm Lidierth
2011-10-8
TMW do seem to struggle with file formats
I agree with @Jan above that not having a switch to disable compression in both v7 and 7.3 is problematic.
As @Walter points out, v7 can be fast - but this is context-dependent. Problems arise because information about the variables disc class, size etc AND NAME is compressed along with the data. The v6 and v7 file format uses a linked list of data entries. If you have 26 variables named A...Z stored in that order, you will need to access each of A to Y to find Z. In v6, that is less of an issue but in v7 you will need to decompress each preceding variable to find the one you are after. This means
[1] data stored earlier in the file can be accessed more quickly and
[2] for v7, large data entries should be stored last.
For both v6 and v7, there is a problem with using -append. Both are conservative about using disc space and will squeeze the file to recover any space that is freed. So if you have a uint16 scalar called 'A' and replace its value using
save(filename, 'A', '-append);
up to 2Gb of data will be shifted upwards in the file before A is appended to the result. This squeeze seems to be done always - even when all conditions are met for the new entry just to overwrite the old one.
Note that v6 and v7 formats are both "Level 5". I would not recommend it, but you can mix save -appends with -v6 and -v7 to the same file without getting an immediate error. Whether or not you might eventually is another matter.
Two functions in my MAT-file Utilities http://www.mathworks.com/matlabcentral/fileexchange/12250 on the FEX may help:
where
is a whos like function but provides information about the class of data on disc and byte offset in the file. That info can be used to do low-level i/o on a -v6 file or to memory map it.
VarRename
can be used to rename a variable in the file to prevent its data area being reclaimed with a subsequent save -append.
I am presently updating these utilities to provide some easier to use functions e.g. getMap which simply returns a standard memmapfile object e.g.
map=getMap(filename, '/mydata/data/xdata');
There is, however, no good way to support v7 files - their design makes them useful only when the file is always to be loaded in its entirety or is small.
I am also adding v7.3 and HDF5 support. As v7.3 is HDF5-based you can use standard HDF5 tools such as HDFView http://www.hdfgroup.org/products/hdf5_tools/ to examine the content. The problems reported above might arise from the "chunk" size being inappropriate rather than gzip compression per se. I had hoped v7.3 would improve on v7 but from what is posted above it looks as though that may not be the case.
Rather disappointingly, TMW support tell me that the v7.3 format "...is not pure HDF5 however and we never claimed that these files will be readable by any HDF5 libraries".
So it looks as though the pre-2004 v6 will often be the best option as it's
- the fastest
- documented and supported outside of MATLAB - as is v7 - e.g. in Octave, Python and R amongst others.
3 个评论
Malcolm Lidierth
2011-10-9
Correction:
Compression is sometimes used with v7.3. It looks though, as only the first variable written to each file is compressed and all others are not. So, writing a small dummy variable to the file switches off compression for all subsequent variables.
Jurek Dziewierz
2012-2-15
Correction 2: it seems to me that compression depends on how big is the variable. For small ones there is no compression, for big ones (eg. 2MB) there is compression no matter where it is in the file.
As i am using matlab on a cluser with superfast storage system, compression is a serious hinderance to me!
Walter Roberson
2018-7-11
As of R2017a, the save() command supports the -nocompression flag for -v7.3 files (only)
0 个评论
Fangjun Jiang
2011-9-9
Are you using the '-append' option of save? If you look at the help of save, '-v7.3' means Version 7.0 features plus support for data items greater than or equal to 2GB on 64-bit systems, which should be in your advantage.
Your MATLAB is R2010a which should be later than v7.3. That is why the warnning message sounds strange. It complaints that you try to save to a MAT-file older than 7.3. Can you try to just run save?
2 个评论
Fangjun Jiang
2011-9-9
移动:DGM
2023-6-11
Can you check the default version for MAT-files?
File > Preferences > General > MAT-Files
Walter Roberson
2011-9-9
To cross-check: is it happening just because of variable sizes, or is do the variable perhaps contain "new style" matlab OO objects? Those new objects were not supported by -v7 .
When -v7.3 was introduced, users did some tests and reported results that would be consistent with only the header information being compressed, with the bulk of the (large) data being saved in space very nearly as large as if they were uncompressed. As that could mean gigabytes of disk I/O, the -v7.3 option was often avoided when practical by users: the reduced disk space usage of the compressed -v7 files made reading much faster, even taking in to account the time required for decompression.
I believe a couple of people from TMW have indicated that -v7.3 files are much faster than they initially were; I have not seen any comparison timings.
1 个评论
Jan
2011-9-9
I'm writing a struct with 50 fields and subfields, which needs 5 to 20 MB memory to the local harddisk. When I use the v6 MAT-file format, the reading takes the half time compared to v7 and v7.3.
Wenjuan Gong
2012-4-23
Do I need to use certain parameter when I load this file? coz I had a problem loading the file, maybe it is not saved correctly.
1 个评论
Walter Roberson
2012-4-23
Please clarify which file you are referring to, and how you saved it. What MATLAB release are you using, and what problem are you seeing? Are you trying to save any object classes, or any handle graphics ?
mursyidi azaki
2015-10-19
how to check our matlab version is it R2010 =v7.3?
1 个评论
Walter Roberson
2015-10-19
At the MATLAB prompt give the command
version
Note that since version 7.3, there has been a Preference as to which save file format to use by default. If you need a particular save file format then it is best to specify it in the save() command.
Sammy
2023-6-11
What is the difference of a compressed data or a saved data with no compression except for the storage space taken? Will the quality of data be compromised?
2 个评论
Walter Roberson
2023-6-11
The compression is lossless. The difference will be in the amount of storage and the time to write and the time to read.
Walter Roberson
2023-6-11
Notes:
- the compression used for -v7 files is different than what is used for -v7.3 files
- -v7.3 files are almost always notably larger and slower than -v7 files, with the possible exception of mat files that contain pure double precision arrays. If you have mixed mixed data types such as struct or cell or table or objects, then -v7.3 can end up being much larger and slower.
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Text Files 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!