Iteratively search in a website (for dummies)

Question

Luca D'Angelo 2024-5-3

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2114826-iteratively-search-in-a-website-for-dummies

回答： Luca D'Angelo 2024-5-9

Hi all,

I have a list of thousands of chemical formula (or potentially formula). What I'd like to do is to iteratively get one of this formula (for i=1:size(FormulaList,1)....end), insert the formula into the search bar of the website (that is: https://pubchem.ncbi.nlm.nih.gov/ ), and check if I have a possible matches or I get something like this ("0 results found"):

I've tried to apply the method described here ( https://it.mathworks.com/matlabcentral/answers/400522-retrieving-data-from-a-web-page ) but I was not able to understand how to get the "curl" (sorry: I'm completely ignorant in this!).

Cheers,

Luca

[SL: removed the parenthesis from the end of one of the hyperlinks]

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Luca D'Angelo 2024-5-9

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2114826-iteratively-search-in-a-website-for-dummies#answer_1455166

在 MATLAB Online 中打开

I've found the solution.

% MassList: column-vector with molecular formula
tic
for mass=1:size(MassList,1)
    url=strcat('https://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/fastformula/',MassList(mass,1),'/cids/JSON?list_return=cachekey');
    try
        jsonData = webread(url);
        ResNum(mass,1)=jsonData.IdentifierList.Size;
    catch
        ResNum(mass,1)=0;
    end
    pause(0.205) % the website asks for max 5 requests / second
end
toc

The resulting column-array provides the number of compounds with the same molecular formula found in PubChem.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Answer 2

Steven Lord 2024-5-3

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2114826-iteratively-search-in-a-website-for-dummies#answer_1452111

Your best bet is probably to use one of the access methods that PubChem provides, as described on this page. Note the usage policy. If you have thousands of requests it's likely going to take minutes or longer, or the bulk data downloads functionality linked in the usage policy may be a better fit for your needs.

From the MATLAB side of things, the functions in this documentation category likely will be of use to you as may be the functions on this documentation page. [Before you ask no, I don't have any examples specific to using those functions to access that database.]

3 个评论
显示 1更早的评论隐藏 1更早的评论

Luca D'Angelo 2024-5-6

在 MATLAB Online 中打开

Thanks for you kind reply. The PubChem policy defines that "we ask that any script or application not make more than 5 requests per second, in order to avoid overloading these servers".

I've tried with a simple test (also with the help of ChatGPT):

opt=weboptions("Timeout",5);
molecularFormula = 'C9H8O4'; % Example molecular formula
apiUrl = sprintf('https://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/formula/%s/cids/JSON', molecularFormula);
attempt = 1;
% Loop to check the status of the request
while attempt <= maxAttempts
    % Fetch JSON response from PubChem API
    jsonData = webread(apiUrl);
    
    % Check if the request is still processing
    if ~isfield(jsonData, 'Waiting') || isempty(jsonData.Waiting) %|| ~strcmpi(jsonData.Waiting, 'true')
        break; % Exit loop if request is not waiting anymore
    end
    
    % Increment attempt counter
    attempt = attempt + 1;
    
    % Wait for some time before making the next attempt
    pause(waitTime);
end
% Check if the request is still processing after the loop
if isfield(jsonData, 'Waiting') && ~isempty(jsonData.Waiting) && strcmpi(jsonData.Waiting, 'true')
    disp('Your request is still processing. Please wait and try again later.');
    return;
end
% Check if the request was successful
if isfield(jsonData, 'Fault')
    disp(['Error: ', jsonData.Fault.Message]);
    return;
end
% Step 3: Parse JSON response to extract the number of search results
numResults = 0; % Initialize number of results
if isfield(jsonData, 'IdentifierList') && isfield(jsonData.IdentifierList, 'CID')
    numResults = numel(jsonData.IdentifierList.CID); % Number of search results
end
% Display the number of results
disp(['Number of results for molecular formula "', molecularFormula, '": ', num2str(numResults)]);

This doesn't work because it always reports:

>> jsonData
jsonData = 
  struct with fields:
    Waiting: [1×1 struct]
 
>> jsonData.Waiting
ans = 
  struct with fields:
    ListKey: '4044371352122785656'
    Message: 'Your request is running'

Do you have any clue how to solve this?

Cheers

Steven Lord 2024-5-6

You haven't shown us what values you're using for the maxAttempts and waitTime variables in your code.

Luca D'Angelo 2024-5-6

在 MATLAB Online 中打开

opt=weboptions("Timeout",5);
molecularFormula = 'C9H8O4'; % Example molecular formula
apiUrl = sprintf('https://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/formula/%s/cids/JSON', molecularFormula);
maxAttempts = 10; % Maximum number of attempts
waitTime = 5; % Time to wait between attempts (in seconds)
attempt = 1;
while attempt <= maxAttempts
    jsonData = webread(apiUrl);
    if ~isfield(jsonData, 'Waiting') || isempty(jsonData.Waiting) %|| ~strcmpi(jsonData.Waiting, 'true')
        break; % Exit loop if request is not waiting anymore
    end
    attempt = attempt + 1;
    pause(waitTime);
end
% Check if the request is still processing after the loop
if isfield(jsonData, 'Waiting') && ~isempty(jsonData.Waiting) && strcmpi(jsonData.Waiting, 'true')
    disp('Your request is still processing. Please wait and try again later.');
    return;
end
if isfield(jsonData, 'Fault')
    disp(['Error: ', jsonData.Fault.Message]);
    return;
end
numResults = 0; % Initialize number of results
if isfield(jsonData, 'IdentifierList') && isfield(jsonData.IdentifierList, 'CID')
    numResults = numel(jsonData.IdentifierList.CID); % Number of search results
end
disp(['Number of results for molecular formula "', molecularFormula, '": ', num2str(numResults)]);

It doesn't really matter, actually. Most of the previous code was written by chatgpt but it's useless. The main lines are:

opt=weboptions("Timeout",5);
molecularFormula = 'C9H8O4'; % Example molecular formula
apiUrl = sprintf('https://pubchem.ncbi.nlm.nih.gov/rest/pug/compound/formula/%s/cids/JSON', molecularFormula);
jsonData = webread(apiUrl);

if webread worked, maybe I would be able to find the information I am looking for. The problem is that I think the function launches the search but then doesn't wait for the website to ‘load’ the result, so it shows ‘Your request is still running’. Maybe I should find a way to launch the command, wait and then check if the webpage 'loaded' the results. What do you think?

请先登录，再进行评论。

Iteratively search in a website (for dummies)

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

采纳的回答

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

更多回答（1 个）

3 个评论
显示 1更早的评论隐藏 1更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

Iteratively search in a website (for dummies)

0 个评论 显示 -2更早的评论隐藏 -2更早的评论

采纳的回答

0 个评论 显示 -2更早的评论隐藏 -2更早的评论

更多回答（1 个）

3 个评论 显示 1更早的评论隐藏 1更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

3 个评论
显示 1更早的评论隐藏 1更早的评论