1. How to scrape a website.
2. How to use special software to make writing Regular Expressions easier.
3. How to write Regular Expressions in C#.
This tutorial is intended for people who know how to do some basic Regular Expressions and know the syntax of the language.
Since there is a lot of material to cover I won’t go over everything. If you want to see the program in action or the bits I left out download the fully commented source code.
Till a little while ago I never really thought much of Regular expressions. I used them a couple of times just to validate an email address. It was not till one day I was approached to make a web scraping program that would deal with pages from amazon.com and then store all the grabbed information in an xml file.
At first I did not even know how to do this. The first thing I had to tackle was getting the ASIN number (A unique number Amazon uses for its products). The only solution I could think of was read each line of code looking for the product links and then using a string builder to break up that link and using a for-loop putting the ASIN back together once found.
I got to this work but quickly realized that doing something similar will not work for all the fields I needed to get. On top of that the program would probably be slow seeing that I would have to check almost 2200 lines of code one line at a time.
So I thought to myself there must be a better way to do this. Later on someone suggested why not use Regular expressions. My initial thought was I am not going to be validating an email address here I am trying to grab content of this amazon.com page.
It was not till I picked up this gem of a book called Mastering Regular Expressions by Jeffrey Friedl that I finally realized the power of Regular Expressions.
How to scrape a website
The first thing we have to do is make the scraper program. A scraper is a program that basically just downloads all the source code of the webpage.
1. Fire up Visual Studios
2. File –> New Website -> Asp.net Website -> Ok
3. Make a new .aspx file(right click on your solution and click on “add a new item”)
4. Name this page “Scraper.aspx”
5. Go to the code behind file(Scrapper.aspx.cs)
Now it’s time to start writing the C# to download the page.
The first thing you will have to do is add to packages to your Using statements.
using System.Net;
using System.IO;
For the shake of this tutorial we will just write all the code in the Page_Load method.
The comments should explain what each line is doing. The big picture though is on page load where going to download all the source code of the given page that has been hardcoded into the HttpWebRequest.Create() method.
Now we have this giant string with all the source code. Now before we go diving in on how to extract the information we want with Regular Expressions we will first learn how to use a program called Espresso that will make our life so much easier.
Expresso
When I started the scraper program for the client I quickly found that it was very hard to write long regular expressions. Every time I wrote something it always came back with no results. I thought to myself this will take forever till I make a Regular expression that works. I then set out to find if someone had a program that would make it easier to write regular expressions. After much searching I found a program called Expresso, a free program that makes writing Regular expressions easier.
First grab a copy of Expresso from their page
So we are going to make a scraping program that will grab the Title of this book, book authors, list price, price and how much you can save.
Regular expressions are just basically patterns so in order to grab information from this area we will have to make some kind of pattern to grab the information. So we have to learn how they displayed this information to us.
If you would look at the source code you would see that their code for this page is about 14,828 lines of code(this include blank lines). That’s a lot of code to go through to find the information that we need.
If you’re using in firefox I highly recommend you grab firebug. With firebug you will be able to hover over the area you want and it will show you the source code of that area.
With using firebugs inspect feature then hovering over all the parts of I wanted (everything in the above picture). I found that this is the code where interested in. With using the inspect feature it took me only a couple minutes to find the 30 lines of code where interested it in.
The above code is copied out of firebug, however when you do this only use firebug to find the starting point of the data you want. So in this case the starting point is
. Once you found this go to the source code and search for this till you find the right area.
The reason why you don’t want to take the firebug code is that it takes away all the spaces and fixes errors. For example look at the line<span> by <a href="”…..”">. If you would use this information to make your regular expression it would never work. The reason for this is if you look in the source code you will see this instead <span > by ‘< a href=”…..”>’.
The span tag has some extra spaces in it. So if you don’t account for them your pattern would fail. So do not use that above code. I have included the good code in the sample code that you can download.
So the above code would display everything in this picture provided if all the tags closed.
So now we can see that this is contained in a div container called buying. In this container you got the book title that uses a header 1 tag. The authors are contained in a span tag with an anchor tag.
This code refers to the stars. In this tutorial we are not concerned about the stars so we will just ignore it.
This is just the horizontal line that goes across the page. We also don’t care about this.
This table stores the heart of the information what we want. It has the list price, the price and how much you save.
All of this information is contained in a table and in their own cells. The 3 lines where interested in our
<td class="listprice">$49.99 </td>, <td><b class="priceLarge">$31.49</b> and <td class="price">$18.50 (37%) </td> .
We don’t care about the tags like “You Save” or “Price” since we can just hardcode these labels in later.
Writing regular expressions.
The first one we will work on is the title.
The source code where interested in is this:
<div class="buying"><h1 class="parseasinTitle"><span id="btAsinTitle" style="">Head First C# (Brain-Friendly Guides) [ILLUSTRATED] (Paperback)</span></h1>
Regular Expression:
<div \s class="buying"><h1 \s class="parseasinTitle"><span \s id="btAsinTitle" \s style="">(?<BookTitle>.*)</span></h1>
This is the regular expression I came up with. It looks for the div with a class called “buying” . You will notice \s in between div and class. This tells the regular expression that there is a space between these words.
This alone would result in 3 found matches.
Next the regular expression looks for a header1 tag with a class name “parseasinTitle” followed by a span tag.
After that we have this weird brace with a question mark and a name in it. This is called name capture. This will help us out later in our C# code by giving us an easy way to call the found code. In this name capture it captures any character until it hits the closing tag.
Authors Source Code
Authors Regular Expression
[\s+a-zA-Z]+(?=</a>\s+\(Author\))
When looking at books on Amazon.com you will notice that after each of the Authors Name they have (Author) right behind it. This seems to be a convention they used. We can use this convention to our advantage.
(?=</a>\s+\(Author\)).
This says go keep going through the code till you find a match. Once you find this match ( \s+(Author)) don’t consume it. Instead go backwards till you met the other condition ([\s+a-zA-Z]+).
First author:
This regular expression would find (Author) and then go backwards and grab Andrew Stellman but once it hits “>” it will stop since that is not one of the characters that should be captured.
Prices Regular Expression
I won’t go through this one since it’s kind of long. I am just going through the whole table and putting Name Captures on the areas I want.
Writing the C# Code
Now it’s time to write the C# code.
The first thing I like to do is put all the Regular Expressions in their own strings that way I can easily change them if I need to later on.
So for the author one I have this:
When you write the expression in C# you have to remember to escape the characters that need to be escaped like \s needs to be \\s. Expresso can do this for you if you go to Tools->View Code.
Next I like to make a variable to hold all the Regex Options to make things look cleaner.
So these ones I like to use especially IgnoreCase since it makes life just easier if you don’t care about case. So if you want to capture all “bob’s” reguadless of case just have that option on and your all set to go.
Now lets look on how you actually would write the code to grab the results from the authors Expression.
Knowing your data is really important when dealing with Regular Expression. In the Case of Authors we know that there can be more then one author per book. So we have to make sure we grab all results and not just one result.
So we will use a MatchCollection to store all the Matches.
Regex.Matches returns all matches and takes in 3 paramters (string input, string pattern, RegexOptions)
Now that we have all the matches we have to get them out somehow.
To do this we will make a while loop.
First we get the count of how many matches where found. Once in the for-loop we grab the first match in the matchCollection and store it in a Match Variable. We then store that Value into printAuthors string.
So that was pretty easy. Now we will try one that made use of name captures.
BookTitleExpression
So this time we know that there should only be one match. Instead of using a MatchCollection we will just use Match and instead of Matches() we will use Match().
Now this will return the whole match meaning that it will return this:
<div class="buying"><h1 class="parseasinTitle"><span id="btAsinTitle" style="">Head First C# (Brain-Friendly Guides) [ILLUSTRATED] (Paperback)</span></h1>
We of course only want “Head First C# (Brain-Friendly Guides) [ILLUSTRATED] (Paperback)”
This is where our name capture comes in handly. With an extra line we wil be able to grab out that value.
So this looks for the Group called “BookTitle” and stores it into a Group Variable. Once it is stored into the title Variable it will only have what we want.
The full program will display this at the end:
Title: Head First C# (Brain-Friendly Guides) [ILLUSTRATED] (Paperback)
Authors: Andrew Stellman, Jennifer Greene
list price: $49.99
Price: $31.49
save: $18.50 (37%)
As you can see it has extracted all the right values.
With a bit of tweaking you could make the program scrape more than one page at a time and store then print out all the page results out.
Source code
The reason why you don’t want to take the firebug code is that it takes away all the spaces and fixes errors. For example look at the line
The span tag has some extra spaces in it. So if you don’t account for them your pattern would fail. So do not use that above code. I have included the good code in the sample code that you can download.
So the above code would display everything in this picture provided if all the tags closed.
So now we can see that this is contained in a div container called buying. In this container you got the book title that uses a header 1 tag. The authors are contained in a span tag with an anchor tag.
This code refers to the stars. In this tutorial we are not concerned about the stars so we will just ignore it.
This is just the horizontal line that goes across the page. We also don’t care about this.
This table stores the heart of the information what we want. It has the list price, the price and how much you save.
All of this information is contained in a table and in their own cells. The 3 lines where interested in our
We don’t care about the tags like “You Save” or “Price” since we can just hardcode these labels in later.
Writing regular expressions.
The first one we will work on is the title.
The source code where interested in is this:
Regular Expression:
This is the regular expression I came up with. It looks for the div with a class called “buying” . You will notice \s in between div and class. This tells the regular expression that there is a space between these words.
This alone would result in 3 found matches.
Next the regular expression looks for a header1 tag with a class name “parseasinTitle” followed by a span tag.
After that we have this weird brace with a question mark and a name in it. This is called name capture. This will help us out later in our C# code by giving us an easy way to call the found code. In this name capture it captures any character until it hits the closing tag.
Authors Source Code
Authors Regular Expression
When looking at books on Amazon.com you will notice that after each of the Authors Name they have (Author) right behind it. This seems to be a convention they used. We can use this convention to our advantage.
This says go keep going through the code till you find a match. Once you find this match ( \s+(Author)) don’t consume it. Instead go backwards till you met the other condition ([\s+a-zA-Z]+).
First author:
This regular expression would find (Author) and then go backwards and grab Andrew Stellman but once it hits “>” it will stop since that is not one of the characters that should be captured.
Prices Regular Expression
I won’t go through this one since it’s kind of long. I am just going through the whole table and putting Name Captures on the areas I want.
Writing the C# Code
Now it’s time to write the C# code.
The first thing I like to do is put all the Regular Expressions in their own strings that way I can easily change them if I need to later on.
So for the author one I have this:
When you write the expression in C# you have to remember to escape the characters that need to be escaped like \s needs to be \\s. Expresso can do this for you if you go to Tools->View Code.
Next I like to make a variable to hold all the Regex Options to make things look cleaner.
So these ones I like to use especially IgnoreCase since it makes life just easier if you don’t care about case. So if you want to capture all “bob’s” reguadless of case just have that option on and your all set to go.
Now lets look on how you actually would write the code to grab the results from the authors Expression.
Knowing your data is really important when dealing with Regular Expression. In the Case of Authors we know that there can be more then one author per book. So we have to make sure we grab all results and not just one result.
So we will use a MatchCollection to store all the Matches.
Regex.Matches returns all matches and takes in 3 paramters (string input, string pattern, RegexOptions)
Now that we have all the matches we have to get them out somehow.
To do this we will make a while loop.
First we get the count of how many matches where found. Once in the for-loop we grab the first match in the matchCollection and store it in a Match Variable. We then store that Value into printAuthors string.
So that was pretty easy. Now we will try one that made use of name captures.
BookTitleExpression
So this time we know that there should only be one match. Instead of using a MatchCollection we will just use Match and instead of Matches() we will use Match().
Now this will return the whole match meaning that it will return this:
We of course only want “Head First C# (Brain-Friendly Guides) [ILLUSTRATED] (Paperback)”
This is where our name capture comes in handly. With an extra line we wil be able to grab out that value.
So this looks for the Group called “BookTitle” and stores it into a Group Variable. Once it is stored into the title Variable it will only have what we want.
The full program will display this at the end:
Title: Head First C# (Brain-Friendly Guides) [ILLUSTRATED] (Paperback)
Authors: Andrew Stellman, Jennifer Greene
list price: $49.99
Price: $31.49
save: $18.50 (37%)
As you can see it has extracted all the right values.
With a bit of tweaking you could make the program scrape more than one page at a time and store then print out all the page results out.
Source code
