Matching a URL

Sarah Reynolds

Hi Everyone,

I'm trying to locate a websites position in google using c# and regex. I can match where the domain to check is a simple domain ie: 'mywebsite.com' but it doesn't work when the domain to be checked is 'mywebsite.com/a-product-name-p-23.html'

I assume this is the regex I am using but for the life of me I cannot work out what this should be. Basically I want to run the script to check my website pages position in serps but the url's could all be very different.

My code at the moment is:

public int GetPosition(Uri url, string searchTerm)
{
string raw = "http://www.google.co.uk/search?q={0}&num=100&hl=en&lr=&ie=UTF-8&safe=off&output=search#q={0}&hl=en&lr=&safe=off&prmd=imvns&ei=avgiT6HOGILPsgaRwfHpCA&start=0&sa=N&bav=on.2,or.r_gc.r_pw.,cf.osb&fp=c7b3e04f0f892e66&biw=1366&bih=624";
string search = string.Format(raw, HttpUtility.UrlEncode(searchTerm));

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(search);
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (StreamReader reader = new StreamReader(response.GetResponseStream(), Encoding.ASCII))
{
string html = reader.ReadToEnd();
return FindPosition(html, url);
}
}
}

private static int FindPosition(string html, Uri url)
{

string lookup = "(

MatchCollection matches = Regex.Matches(html, lookup);

for (int i = 0; i < matches.Count; i++)
{
string match = matches[i].Groups[2].Value;
if (match.Contains(url.AbsoluteUri))
return i + 1;
}

return 0;
}

An example of the data passed into 'string html' is:

BBC News - Home

If I use Keyword 'World News' and URL 'bbc.co.uk' then a serp position is retrieved but if I use the URL 'http://news.sky.com/home/world-news' OR 'www.msnbc.msn.com/id/3032507/' no serp position is retrieved despite both urls being on page1 of the serps.

I guess the regex cant match the more complex urls?

Any regex experts here who might be able to point me in the right direction or is there an easier way to do this in c#?

Many thanks, Sarah

13 Replies

Know the answer? Post it — somebody with the same question will find it here.

Sam Hobbs14y ago
No problem. The forum software messes up sometime so we will assume it is the culprit for making the bold text and such.

Please do let me know if you need help with the code.
0
Sarah Reynolds14y ago
Mmm, not sure what happened with the bold in the last statement either? Apologies for that.
0
Sarah Reynolds14y ago
Hi Sam, Sorry, the links are preceded by
(not sure what happened with c&p before) I'll have another look at your example in more details. Like I said I couldnt actually get it to run earlier so I'll look into the errors and try and resolve them 1st. Thanks Sarah
0
Sam Hobbs14y ago
Note that if you do not want to show the page then say so. Just remove the line that sets the Visible property to true. Then the browser window will not show. Just be sure that you always call the Quit method for the page. The problem is that I am not sure how to determine that the page is truly complete. Doing that with reasonable certainty will require some more analysis. The simple solution of course is to use a timer to wait a specified amount of time. If it were me, I would prefer to use that as a last resort.

Those web pages (at least the one I used in my sample for you) are complex; they are not simple pages. I am nearly certain that the only way you can get all the data that normally shows in the page is to navigate to it somehow. They create content dynamically. When you get the HTML from the web site, the scripts have not executed that generate additional HTML. You might be able to analyze the scripts to determine where the data is coming from and develop a more direct way to get the data but I really think the data is not in the HTML that exists in the server that you get with a HttpWebRequest.

Thank you for mentioning that the links you need have the attribute class="r". I will try adding that to my sample.
0
Sarah Reynolds14y ago
No, I'm saying I didn't want to open a browser window to perform the tasks and your example, from what I could see, only loaded the page in a browser and counted the links on the page. I need to loop through each relevant link (ie: each link preceded by class="r") and find a match to one input by the user. Anyway, I'm going to put this on the backburner for now as I have other parts to accomplish first. Thanks Sarah
0
Sam Hobbs14y ago
Are you saying "this isn't how I would like to perform this" because I said "I have never even tried to use regular expressions for HTML."?

What do you mean by "the console kept throwing mshtml.HTMLDocumentClass errors"?

I hope Vulpes understands what I am saying about the problem of downloading the HTML in the manner you are doing it.
0
Sarah Reynolds14y ago
Sorry for the delay in getting back. Vulpes, I agree that the issue seems to be with the string passed to the findposition method. I checked this out again and it looks like google inserts the news items using a different format to the regular links so the expression does not match.

Sky News - World News: First for breaking global and international ...

I guess I need to look at this a bit differently as the results seen in the browser often differ. (ie:bbc news actually returns 5 results but in the browser they count for 1 position in the serps)

The regex definitely matches more urls on testing so I will definitely use that when I get round to this again so my thanks for posting and the time taken to help.

Sam, Unfortunately your project didn't work for me as the console kept throwing mshtml.HTMLDocumentClass errors but I could see enough to see this isn't how I would like to perform this.

Basically, the idea was to pass a keyword to google search and then inspect the results for the position of a given url. (All without opening a browser window) The original method with Vulpes' regex works to a point but does not take into account news/shopping results mixed into the page so the returned serps position is slightly different to the results viewed in a browser.

Thanks for your help guys.
Sarah
0
Sam Hobbs14y ago
It is impossible to use regular expressions by reading those pages in the manner you are. The reason is that the data is not there. You suspect that to be true but you are not sure why. I know why. The page starts as incomplete and then later the page is filled in; or at least that is the terminology I am using and I hope you understand what I mean. You can see this if you try doing the way I suggested. The DocumentComplete event fires many times, whereas normally it happens only once after navigation (once per page).

The attached project is a console application that shows all the links in the page. It writes the links to a file, because I am not sure what you are looking for. Note that when it executes it waits for a key to press. This is done for two reasons; to keep the window open, but also to give time for the document to completely download. The page downloads quickly, but not quickly enough for you to access the page immediately. You can look at the Links.txt file in the project's bin\Debug folder without building the project to see the links my program found.

I am not sure what you need to do but I hope this points you in a useful direction.
0
Vulpes14y ago
OK, Sarah, after spending some time with this, I've now managed to figure out a few things.

First of all, the MSDN regular expression does produce some results though it needs to be incorporated into the program as follows:

Replace this line:

string lookup = "(

with this one:

string lookup = "(

This produces a result of 1 for 'http://www.bbc.co.uk' and a result of 13 for 'http://www.msnbc.msn.com/id/3032507/'.

It was able to return a result of 48 for 'http://news.bbc.co.uk/2/hi/video_and_audio/news_summary/8070822.stm' which your own expression couldn't find as it doesn't allow for underscores.

However, it was still unable to find 'http://news.sky.com/home/world-news'.

So I then examined the HTML source for the google search page and discovered that the (apparent) reason why it's not showing up is because the URL is preceded by class=r rather than class="r". When I allowed for this in the regular expression, it still didn't produce a result and when I subsequently examined the MatchCollection of 97 items there was no sign of it.

I then looked at the html string that is being passed to the FindPosition method and found that it simply isn't there to be found!

It therefore looks like it's being suppressed in the HttpWebResponse for some reason, possibly because there's an associated video link when the page is displayed in the browser (I use Chrome). The same applies to www.guardian.co.uk/world.
0
Sam Hobbs14y ago
Developers use regular expressions to parse HTML because it seems to be the easiest solution. Then they enounter unexpected complications and spend a lot of time trying to get it to work. I have never even tried to use regular expressions for HTML.

I have used the DOM to parse HTML, such as in my Introduction to Web Site Scraping. The DOM is designed to do that. The mshtml component has the classes for using the DOM. Reead my article for a description of it.

I am not sure I understand what you are trying to do, but if I understand then you can use an InernetExplorer object in the shdocvw component to navigage to Google. You will want to add a handler for the DocumentComplete event. Then (in or after the DocumentComplete) you can use the Document property of the InernetExplorer object to access the HTML document (a mshtml.HTMLDocument object). I am not sure what you need to do from there, but note that there is a links property that is a collection of all the links in the page.
0
Vulpes14y ago
All I actually did was to test that the regular expression in MSDN could identify the 2 sample urls you gave plus a few more of my own. FWIW here's the code:

using System;
using System.Text.RegularExpressions;

class Test
{
static void Main()
{
string regExp = @"((ht|f)tp(s?)\:\/\/)?[0-9a-zA-Z]([-.\w]*[0-9a-zA-Z])*(:(0-9)*)*(\/?)([a-zA-Z0-9\-\.\?\,\'\/\\\+%\$#_]*)?";
string[] urls =
{
"http://news.sky.com/home/world-news",
"www.msnbc.msn.com/id/3032507/",
"http://www.c-sharpcorner.com/Forums/",
"http://msdn.microsoft.com/en-us/library/ff650303.aspx",
"www.youtube.com/"
};

foreach(string url in urls)
{
Console.WriteLine(Regex.IsMatch(url, regExp)); // all True
}

Console.ReadKey();
}
}

However, the problem is clearly more complicated than that and, as I don't have much time just now, I'll try and have another look at it tomorrow when I can look at the problem as a whole.
0
Sarah Reynolds14y ago
Hi Vulpes,

Thanks for that. Unfortunately its not returning any results for me (even the 2 examples in the OP) Did this actually return a result for you? Did you hardcode your keyword/url or pass them from a textbox?

Thanks
Sarah
0
Vulpes14y ago
There's a more complex regular expression for matching a URL here in this old MSDN article:

http://msdn.microsoft.com/en-us/library/ff650303.aspx

which if you make the http(s):// optional, remove the ^ and $ anchors and remove the & leaves us with:

string regExp = @"((ht|f)tp(s?)\:\/\/)?[0-9a-zA-Z]([-.\w]*[0-9a-zA-Z])*(:(0-9)*)*(\/?)([a-zA-Z0-9\-\.\?\,\'\/\\\+%\$#_]*)?";

That was able to identify the two samples you gave, plus a few more I fed it, as valid URLS so I'd see if you can integrate it into your validation expression.
0

Matching a URL

(not sure what happened with c&p before) I'll have another look at your example in more details. Like I said I couldnt actually get it to run earlier so I'll look into the errors and try and resolve them 1st. Thanks Sarah