How to read a website and download pdf files

Question

Joy Shen 2023-4-3

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1940299-how-to-read-a-website-and-download-pdf-files

回答： Voss 2023-8-23

Is there a way in matlab to essentially batch read a website and download the pdfs without having the specific pdf url?

For example, on the websites I'm looking through, each page has the following type of link where you download the pdf: https://restservice.epri.com/publicdownload/000000000001013457/0/Product

I just have an excel spreadsheet of all the URLs I want to look through, not the PDF file URLs.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Saffan 2023-4-5

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1940299-how-to-read-a-website-and-download-pdf-files#answer_1209294

在 MATLAB Online 中打开

It can be done using “webread” and “websave” functions.

Here is an example code snippet to extract all the URLs present in a particular webpage:

%extract entire source code of the page
html_text = webread(url); 
%extracts URLs present in the source code
all_urls = regexp(html_text,'https?://[^"]+','match'); 

Once you have obtained the URLs of the downloadable PDFs, you can use the "websave" function to download them. Here is an example code snippet to demonstrate this:

websave('filename.pdf',pdf_url);

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Joy Shen 2023-5-1

编辑：Joy Shen 2023-5-1

在 MATLAB Online 中打开

@Saffan

How do I do a batch download though? For example, I have an excel spreadsheet of all the links and the names of the pdf title. I sorted through and stored all the links that match my string. How do I get webread and websave to open each of the links in my excel spreadsheet (which is a 64x1 table) and download the pdf? Especially when the link to the pdf doesn't seem to lead to a .pdf, it leads to a link like this: https://restservice.epri.com/publicdownload/000000000001013457/0/Product

URL = readtable('EPRI NMAC Repository.xlsx','Range','I2:K638'); % Load excel from specified sheet
substr = 'Nuclear Maintenance Applications Center';
name = table2array(URL(:,1));
selectedcol = contains(name,substr);
links = URL(:,3);
selectedlinks= links(selectedcol,:)
%extract entire source code of the page
html_text = webread(selectedlinks); 

请先登录，再进行评论。

Answer 2

Voss 2023-8-23

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1940299-how-to-read-a-website-and-download-pdf-files#answer_1292637

在 MATLAB Online 中打开

URL = readtable('EPRI NMAC Repository.xlsx','Range','I2:K638'); % Load excel from specified sheet
substr = 'Nuclear Maintenance Applications Center';
selectedrow = contains(URL{:,1},substr);
pdf_url = URL{selectedrow,3};
% make valid file names from the urls:
pdf_fn = regexprep(pdf_url,'[/\\.*<>|?"]','_');
pdf_fn = strcat(pdf_fn,'.pdf');
% download the pdf files:
for ii = 1:numel(pdf_url)
    websave(pdf_fn{ii},pdf_url{ii});
end