Import data tool speed

3 次查看（过去 30 天）
Brad_EE
Brad_EE 2020-12-11
评论： Yair Altman 2020-12-13
Platform: W10-64bit, 4 core, 500GB SSD, 32GB Ram
File to import description: 80MB, contains comma delimited numbers and short character arrays. Overall format 27 columns x 400K lines.
Clicking the import data tool icon prompts for a file name, shows a message "Opening a large text file ..." and then displays the file contents in a table format on the GUI after ~5 seconds. The table displayed matches the file contents.
On the same GUI I set the output type to cell array and range to A2:AA395201 (entire file minus header line). Clicking the Import Selection button displays message "Importing Data..." and a status bar that stays gray for 35 minutes before it suddenly disappears. At that point the import is complete and the variable name appears in the workspace.
Why does the initial opening large text file finish in 5 seconds but the import take 35 minutes? It seems for the opening large text file step to complete and display the data in the GUI table, it has essentially imported the data but 500X faster!

请先登录，再回答此问题。

采纳的回答

Yair Altman
Yair Altman 2020-12-13
The import tool GUI only shows you a preview of the data, based on the top N lines in the file, it does not read and process the entire file. Only when you click the <Import Selection> button is the entire file processed based on the selected range that you specified and the file format detected by the preview. This naturally takes much longer than the preview processing.
  4 个评论
显示 2更早的评论
Brad_EE
Brad_EE 2020-12-13
Perhaps it is reading 35 lines at a time based on the relative position of the scroll bar. So for instance if the scroll bar is pulled down 3/4 of its length, only 35 lines from that location in the file is actually read and displayed.
Yair Altman
Yair Altman 2020-12-13
Yes, this is exactly what I meant

请先登录，再进行评论。

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Workspace Variables and MAT-Files 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by