'How to enable cookies while using HTMLunit for webscraping?
I am using HTMLunit to retrieve a html page. link: https://racing.hkjc.com/racing/information/english/Racing/LocalResults.aspx?RaceDate=2021/12/08&Racecourse=HV&RaceNo=1 following is my code:
WebClient webClient = new WebClient();
webClient.getOptions().setJavaScriptEnabled(false);
webClient.getOptions().setCssEnabled(false);
HtmlPage page = webClient.getPage("https://racing.hkjc.com/racing/information/english/Racing/LocalResults.aspx?RaceDate=2021/12/08&Racecourse=HV&RaceNo=1");
System.out.println(page.asXml());
However, in the console I see the following: GenericErrorMessageCookies="Cookies must be enabled in order to view this page.";
I tried scraping a few Wikipedia links and it works alright for that. Just for this specific website it doesn't seem to work.
Techniques for using Jsoup are also welcome!
Can someone please help solve this probelm? Thanks!
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|