Web Scrapping in c#

navneet kaur

-I am using selenium and PhantomJSDriver for web scrapping.I am facing the issue for scrapping list of items for wbsite.The result is not consistent,I get different number of records each time.To overcome this,I use explicit and implicit wait for webdriver.But it through timeout error for that case.

driver.FindElementsByClassName("Tile").ToList()

retrive different number of records.and if I use debugger and wait for some time then it retrive all records.

driver.Url = uri.ToString() ;
driver.Navigate();
if (uri.PathAndQuery == "/")
{
//This is synchronous, it will not go to the next line until the crawl has completed
driver.FindElementsByClassName("Tile").ToList().ForEach(x =>
{
objUPCProcess = new UPCProcessingModel();
try
{
objUPCProcess.Description = x.FindElement(By.ClassName("Tile-heading")).Text;
objUPCProcess.Price = x.FindElement(By.ClassName("Price-display")).Text;
listOfProductDetails.Add(objUPCProcess);
if (i == 0)
{
log.Item = objUPCProcess.Description;
i++;
}
}
catch (OpenQA.Selenium.NoSuchElementException ex)
{
log.Message = ex.Message;
log.Status = "Error";
}
});
}

4 Replies

Know the answer? Post it — somebody with the same question will find it here.

Ravi Sangtani9y ago
@navneet kaur

If the data is loading by javascript backhand or ajex then it is difficult to scrape the data.

If their is any loading gif or any loder image is their when data is still loading then we can easly manage that data has properly loaded and after we make scrape code when onces successfully data loaded.
+1
Dharmraj Thakur9y ago
You can use like below...
driver.manage().timeouts().pageLoadTimeout(10, TimeUnit.SECONDS);
Suggestion from my side:
Casperjs better than these all... they are providing like AtElementLoad() and OnPageCompleteLoad() events so every trigger execute when you want...

Its based on only js no need c#.
+1
navneet kaur9y ago
Hello Ravi Sangtani,

Thanks for your response. But by using Thread.Sleep(10000); ,it is not ensure that every time it take all the record.I need some solution that sure me that the outcome after scrapping must be equal to the actually number of records.
+1
Ravi Sangtani9y ago
Hi Navneet,

Put below line after navigate your Url...

Thread.Sleep(10000);

It will wait 10 sec for load whole the data then after it may give you correct result i think
Please try onces.
0