check if url exists for large files without downloading the file

25 次查看（过去 30 天）
Diana
Diana 2016-10-11
回答： Fernando Bello 2019-11-28
I've searched this forum for a while and need some help with checking URLs. I have a set of files to download from a site and loop through the formatting options for the files basically the files are zip files and are extremely large (nearly 100MB). If I use the following statement:
raw_file = websave([p_raw,'\',fname_raw],[URL,MO_name,num2str(YR),'/raw/',fname_raw]);
Then for files that do not actually exist a dummy file with 0KB of data in it is made in the saved location. I want to avoid this. I've tried using:
[str status] = urlread([URL,MO_name,num2str(YR),'/raw/',fname_raw]);
but MATLAB still insists on trying to read the actual file instead of just telling me if it's there or not - wasting time in the process.
The function isurl does not apply to named URLs. How do I check merely the existence of a url without actually downloading it?
Thanks!

请先登录，再回答此问题。

回答（4 个）

Steven Lord
Steven Lord 2016-10-11
Try using Java.
url = 'http://www.mathworks.com';
J = java.net.URL(url);
conn = openConnection(J);
status = getResponseCode(conn)
When I ran this code I received status 200 which is OK. I tried it for a URL to a file that didn't exist and received code 404. I don't think creating an object using openConnection actually retrieves the data; I think you'd have to call that object's getContent method or something similar to obtain the data.
Image Analyst
Image Analyst 2016-10-11
Did you try exist()?
itExists = exist(filename, 'file');
filename will the the url of the file you want to check on.
  1 个评论
Diana
Diana 2016-10-11
编辑：Walter Roberson 2016-10-11
If you mean in this manner:
itexists = exist('http://amisr.com/database/tmp/loucks/Apr2016/raw/20160406.005.tar.gz','file')
nope - gives me a 0 return and I'm staring at the file on the URL.

请先登录，再进行评论。

Matthew Eicholtz
Matthew Eicholtz 2016-10-11
It seems I stumbled upon a similar approach to Steven, but a few minutes too late!
I think this function would do the trick:
function tf = urlexist(url)
URL = java.net.URL(url); %create the URL object
% Get the proxy information using the MATLAB proxy API.
proxy = com.mathworks.webproxy.WebproxyFactory.findProxyForURL(URL);
% Open a connection to the URL.
if isempty(proxy)
urlConnection = URL.openConnection;
else
urlConnection = URL.openConnection(proxy);
end
% Try to start the input stream
try
inputStream = urlConnection.getInputStream;
tf = true;
catch
tf = false;
end
end
Fernando Bello
Fernando Bello 2019-11-28
Maybe it is a little bit late, but I'm just reading this post because I had the same trouble.
I solved it this way:
url = ('https://matlab/file.nc'); % the URL where data is, it can be what ever you want, png, doc, mat, etc.
filename = 'filename_test'; % the name you want to save the file
options = weboptions('Username','username','Password','passwd','CertificateFilename','');
try
websave(filename,url,options);
disp('OK') %if the file exist it is downloades
catch
disp('URL read of link was unsuccessful') % if file do not exist
end

类别

在 Help Center 和 File Exchange 中查找有关 Downloads 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by