How to avoid or mitigate url changing special character to hexadecimal when interpreted by web() or webread() functions?

Question

James Anderson 2023-1-15

1
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1894580-how-to-avoid-or-mitigate-url-changing-special-character-to-hexadecimal-when-interpreted-by-web-or

评论： Nathan Kalish 2024-6-20

cargurusScraper.m

Attempting to use web or webread functions to access a url which features multiple pages that use "#resultPage=[PAGE NUMBER]" to denote which page the user is on. When using either of the web-accessing commands, if a # character is included, all = characters following the # become %3D. Checking with the ASCII value for the = character in hexadecimal, = is represented as 3D. So, it seems that when the url is passed to web or webread, all special characters following it become their hexadecimal values. This leads to the end of the url becoming "#resultPage%3D1" which does not open the desired page.

Testing with other special characters ($, %, ^, &) results in the same issue, they all are converted to a % sign followed by their hexadecimal values. I looked on many pages concerning web and webread, looked at special characters for MatLab, and through forums and bug reports and was unable to find anything about this particular issue. Attempted to use both default browser and the MatLab built in browser, but both have the same issue.

I assume this is an intended interaction, as # is a special special character, and the rest of the special characters are converted to %hexadecimal because websites are supposed to still work with this. However, the site I'm trying to access doesn't seem to like that.

I am able to replace the # in the original link with a %23 (the hexadecimal code for #), and on Chrome the website opens properly, however when fed through either web or webread it does not with either the default browser or built in.

Would greatly appreciate whatever advice could be given towards getting this to work as intended. For full transparency, I'm trying to scrape CarGurus.com for car listings so I can save them in an Excel sheet. I've attached my code for attempting to do this, as well as the link I'm attempting to reach below. Thank you!

(To use the function, call it with the link provided as the only input as a character array)

Base URL: https://www.cargurus.com/Cars/inventorylisting/viewDetailsFilterViewInventoryListing.action?sourceContext=carGurusHomePageBody&bodyTypeGroup=bg6&distance=250&maxMileage=90000&minYear=2016&maxYear=2022&minPrice=0&maxPrice=23000&zip=30324

Page 2 URL: https://www.cargurus.com/Cars/inventorylisting/viewDetailsFilterViewInventoryListing.action?sourceContext=carGurusHomePageBody&bodyTypeGroup=bg6&distance=250&maxMileage=90000&minYear=2016&maxYear=2022&minPrice=0&maxPrice=23000&zip=30324#resultsPage=2

Web/Webread URL: https://www.cargurus.com/Cars/inventorylisting/viewDetailsFilterViewInventoryListing.action?sourceContext=carGurusHomePageBody&bodyTypeGroup=bg6&distance=250&maxMileage=90000&minYear=2016&maxYear=2022&minPrice=0&maxPrice=23000&zip=30324#resultsPage%3D2

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Fabian Schuette 2023-7-17

2
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1894580-how-to-avoid-or-mitigate-url-changing-special-character-to-hexadecimal-when-interpreted-by-web-or#answer_1274578

在 MATLAB Online 中打开

The example above results with Matlab R2023a and Firefox into this:

https://www.google.com/search?q=matlab#search_results_mode%3Dinline

The symbol = was replace into %3D.

How can I avoid this?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Answer 2

Sarthak 2023-3-24

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1894580-how-to-avoid-or-mitigate-url-changing-special-character-to-hexadecimal-when-interpreted-by-web-or#answer_1200144

在 MATLAB Online 中打开

Hi James,

I tried to use the web function to access url’s with special characters such as ‘#’ and ‘=’ however the function is behaving as intended. It would be great if you could attach some screenshots of what exactly is the error and what you are trying to achieve.

url='https://www.google.com/search?q=matlab#search_results_mode=inline';
web(url)

You may also try to encode your url’s before passing them to the web and webread functions and see if the issue still persists.

Open web page or file in browser - MATLAB web (mathworks.com)

2 个评论
显示无隐藏无

Aditya 2024-2-8

I am using R2023b and have a URL with = and & and the web command is not handling this correctly.

Nathan Kalish 2024-6-20

在 MATLAB Online 中打开

There is an internal matlab function being called inside of the web function which is mangling the urls.

myUrl='https://www.google.com/search?q=matlab#search_results_mode=inline';
matlab.internal.web.resolveLocation(myUrl).EncodedURI
ans = "https://www.google.com/search?q=matlab#search_results_mode%3Dinline"

Result:

ans = "https://www.google.com/search?q=matlab#search_results_mode%3Dinline"

Is there a way to avoid this behavior being called within web(url)? I don't see a way to avoid it without calling several internal matlab methods (which I assume will change in the future).

请先登录，再进行评论。