Loading binary data into custom array

Question

1 个投票

Hi,

I'm trying to load in a binary data file with a unique data structure {uint32, uint32, double, double, double,.., double, uint32}. After doing a little bit of research, folks on the forums have recommended using the function "memmapfile()" to read in binary data files with custom arrays. The problem with this is my data is all in big-endian format and memmapfile() outputs the data in little endian format.

I'm looking for help as to whether or not memmapfile() can be produce a big-endian output without me using functions like swapbytes() or typecast() as it increases the loading/processing of the data significantly.

Any help is deeply appreciated,

MP

3 个评论
显示 1更早的评论隐藏 1更早的评论

MPan 2020-12-23

Thanks Walter, I'm going to give this a shot. This seems promising.

MPan 2021-1-5

Walter,

I had the chance to review this code today and it appears that the MappedTensor does not provide the ability to read data into custom data structures based on the issue that was opened in Github. The author suggested loading the data into different binary files to then be stored in the various fields of the struct but that doesn't really seem to be any more efficient than using fread().

https://github.com/DylanMuir/MappedTensor/issues/16

Regards,

Mithul

请先登录，再进行评论。

请先登录，再回答此问题。

Follow Question

Answer 1

weikang zhao 2020-12-23

0 个投票

The “fopen” function may be more flexible. It allows you to access the data in the binary file in any way, of course, including any data type and big-endian. The disadvantage is that it cannot be implemented with a simple function, and a script needs to be written.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

MPan 2020-12-23

Hi Weikang,

That was also a function I used during my first attempt; however, reading in binary files and assigning them to my custom struct one field via a "for-loop" was very time consuming. I also have several different structs that have variations in what order the fields and data types are in so I'm looking to create a generic script that can handle this for me.

Thanks for the suggestion.

请先登录，再进行评论。

Loading binary data into custom array

3 个评论
显示 1更早的评论隐藏 1更早的评论

回答（1 个）

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

类别

标签

Community Treasure Hunt

Loading binary data into custom array

3 个评论 显示 1更早的评论 隐藏 1更早的评论

回答（1 个）

1 个评论 显示 -1更早的评论 隐藏 -1更早的评论

类别

标签

另请参阅

Community Treasure Hunt

3 个评论
显示 1更早的评论隐藏 1更早的评论

1 个评论
显示 -1更早的评论隐藏 -1更早的评论