I am using VS C# Express
Ok, so I'm trying to scrape some HTML from a website. It requires a login as well as re-captcha upon login. As I assume there is now way of breaking the re-captcha, I have a browser where I login to the site I'm trying to scrape. To scrape the data I will be using httpwebrequest and httpwebresponse. So, my question is how or where can I find the cookie that I get after logging into the site? Once I have it, I understand how start using it but I'm not sure where to start looking. Can it be obtained from the HTML document? or do I have to look in the IE explorer program files for it? Not sure what the best way to approach this.
-Jose
Loading
Suthish NairPosted Feb 5, 2011, 3:10 AM
I am not sure if CookieContainer will help you, but give a try.
The above listed links have some samples, refer it.
Sam HobbsPosted Feb 4, 2011, 11:13 PM
I have gone down many paths in the past that appear to go where I need to go, but then they don't. I think that you won't be successful getting the cookies you need unless you use something that navigates to the web site that the cookie is for.
JosePosted Feb 4, 2011, 9:29 PM
Create a web browser
AxSHDocVw.AxWebBrowser axWebBrowser1;
mshtml.HTMLDocument doc;
doc = (mshtml.HTMLDocument)axWebBrowser1.Document;
Login to the site in the web browse once. Once logged in, I want to be able to use the cookie in subsequent HTTPWebequest to obtain data accessible only when logged in, thus requiring the cookies in the request. Now, I understand that HTMLDocument has a cookie property, but how to use that property in the CookieContainer property of HTTPWebequest?
Sam HobbsPosted Feb 4, 2011, 6:29 PM
However I think the best solution is to obtain the cookies using the HTML document as you say. I assume the document won't be in a WebBrowser object. You describe how you will get the HTML but you don't specify how you will use it after that. In order to get the cookie from the HTML, you actually will need to get it from the web site in the sense that IE must navigate to it. So I think that httpwebrequest and httpwebresponse won't help with that. You don't need to use the WebBrowser control; I think you can use an InternetExplorer object. I think I can help with that; do you want to do that?
Suthish NairPosted Feb 4, 2011, 3:23 PM
I dont remember exact coding and logic.
Do a web search, might got some solutions.
refer these listed links
ShinurajPosted Feb 4, 2011, 4:52 AM