How to make sure a web page is entirely loaded before using webread

9 次查看（过去 30 天）

tom3w 2023-1-1

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1886982-how-to-make-sure-a-web-page-is-entirely-loaded-before-using-webread

编辑： tom3w 2023-1-2

Hello,

While scraping data from a website, I don't have any issue for the main part of it, except for a data table; the latter is made available as its elements can be observed while inspecting the html/webpage.

My assumption is that the webpage requires some time in order to be fully loaded, before using using webread properly.

A timeout parameter or a simple for loop until success do not allow to fix the issue. Similarly, I don't mind loading text data instead of a table, as this is not the real problem here.

So all in all, I'm trying to find a way to open/fully load a web page and use this latter output as an argument/input to webread in a second step. Unless a specific parameter related to webread or weboptions exists and allows to address the issue.

Thanks for your help!

11 个评论
显示 9更早的评论隐藏 9更早的评论

tom3w 2023-1-2

编辑：tom3w 2023-1-2

在 MATLAB Online 中打开

Thanks for your views.

I was also thinking about a dynamic content, but it seems the table contents (each element of it) is available when right clicking on them and inspecting the elements. Would the table be a dynamic content I assume a link to an external web page would be displayed instead (if I'm not mistaken).

I'm now working on another assumption. It seems the table in question can only be accessed after login with password on the webpage, which would require using weboptions.Though, after using the following statements, the issue can't be fixed either, which means I might need to examine authentification/authorization aspects.

options=weboptions('Username','xxx','Password','yyy');
details_tmp=webread(url_name,options);

Therefore, if the tentative here above (the pause(10) statement) is confirmed to be sufficient (to make sure the webpage can be entirely loaded before loading the data) I might publish another post on authentification/authorization issues when using webread if needed, after reviewing the existing posts on this topic.

tom3w 2023-1-2

在 MATLAB Online 中打开

@Rik Yes this is how I understood it. In your example (the statistics of a Matlab user), three dynamic links are being used for the chart:

<script src="/matlabcentral/profile/assets/core-01ec908d852dd668e7ae3c25324b00002997ef998387f5dffd4d996b7fde242d.js"
<script src="/matlabcentral/profile/assets/charts-70abe752108b9b28083412ff1a9c89ef24c92337c7c011e8bf0018430a231b92.js" cache="true"></script>
<script src="/matlabcentral/profile/assets/animated-c33dad0fafaa02c86b4f422dbf8280866a6029167dd5c6211af74be83cc26ed6.js" cache="true"></script>

whereas the right hand side contents can be easily read, i.e. the Rank, Reputation, Contributions, etc...

In my case, when logged in to the web page, inspecting the elements doesn't show dynamic links for the table, but contents close to the format related to the Rank, Reputation, Contributions in your example. After having imposed a pause(10) in webread.m I could conclude that the issue was at another level, probably the login.

After several trials I could fix the login issue, even if it's not as clean as I wanted, based on the post:

https://www.mathworks.com/matlabcentral/answers/569052-pull-webpage-from-matlab-site-using-matlab-but-with-login

So basically:

[a, h] = web(url_name);
manual login on Matlab's internal browser; the latter needs to stay open at any time
details_tmp = get(h, 'HtmlText');

All in all, it seems webread has some strong limitations, since the following statements don't work.

options=weboptions('Username','xxx','Password','yyy')
details_tmp=webread(url_name,options);

It would hence be interesting to know under which conditions they are assumed to work.

Rik 2023-1-2

编辑：Rik 2023-1-2

I would venture the guess that it has to do with cookies, which are tricky to deal with in the context of a simple webread.

tom3w 2023-1-2

编辑：tom3w 2023-1-2

@Rik Yes it's a very likely possibility. Let's hope Matlab's tech/dev team will review different use cases in the perspective of improving or better documenting webread/weboptions.

Thanks all for your questions and suggestions, I could eventually fix my main issue.

请先登录，再进行评论。

请先登录，再回答此问题。