Urlread accessing millions of records

2 次查看（过去 30 天）

显示更早的评论

Muneer 2013-11-19

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/106816-urlread-accessing-millions-of-records

关闭： MATLAB Answer Bot 2021-8-20

在 MATLAB Online 中打开

Hello,

I am accessing a password protected database using urlread. Here's what the beginning of my code looks like.

clc;
clear;
tic
url = 'http://assa104/eurofot/csv';
block = urlread(url, 'Authentication', 'Basic', 'Post', {'login', '****', 'passwd', '*****', 'trips', '1332416521', 'signals', '1.AVL_BRTORQ_SUM.AVL_BRTORQ_SUM_DVCH','content_type', 'text'}); 
%, 'limit', '20000' after signals
toc

The problem is that I am accessing so much data this way that if I don't put a limit on how much I'm accessing, I just get an error and I can't read any data into my block. I removed a few signals from my code (one is shown, originally bringing in ~130) above so it wouldn't clutter this post but I'm accessing millions of data records. I need to bring the full set however. Can anyone offer a suggestion for working around this problem?

The other thing I was thinking about is instead of bringing in all 130 signals at once, bring in one at a time using urlread and save them into a matrix. The problem is the data set uses Last Observation Carried Forward so calling each signal individually may mean that each matrix has a different number of elements and then I won't be able to combine them into one matrix.

Any help would be great, thanks in advance!