Automation with Grab Them All and Notepad++

Introduction

Much of SEO work is very repetitive and I like most developers will always try to automate tedious tasks! As Larry Wall famously commented, laziness is one of the 3 virtues of a good programmer.

So, I was faced with checking results from a piece of software that I had built using Google’s Custom Search API. My test sample was about 1/250 of the whole data set size, however I did not intend on manually copying and pasting each URL into Google! I wanted to find a way to automate this, and Firefox extensions seemed the way to go.

Now this task was effectively to put a search term into Google and see whether it returned any results, pretty simple. If it did, the test passed. These terms were massively long and my program was returning a 100% fail rate, which seemed a bit suspicious considering in the test before every one passed! (Albeit with much shorter search terms)

Now all I really wanted to do was remove the boring copy and pasting, and just watch the results come up on screen. As I was supervising it I didn’t want complex reporting and result storage; that could be done with my app or iMacros. I wanted a quick way to finish the manual checking so I could get back to the interesting development stuff!

Installing Grab Them All

Grab them all is a Firefox extension, and is therefore very easy to set up.

  1. Go to https://addons.mozilla.org/en-US/firefox/addon/grab-them-all/
  2. Click “Add to Firefox”
  3. When the popup comes up click install.
  4. Restart Firefox
  5. Et voila, you’re all set!

Want to get stuck in?

Well hold on, Grab Them All needs a text file of URLs to work from. First decide how complex your search queries are going to be? If you are testing a single word, for example to check your sites position on Google’s front page, then just open notepad and copy and paste in all the URLs you want to test (separated by new lines).

If you want to search Google for a URL or any string with complex characters in then you may need to bear in mind URL encoding. This is why Notepad++ is mentioned in the title; even though lightweight, Notepad++ has got some great time saving features.

What’s that I hear? You don’t have Notepad++? Well download it here and make sure to include the TextFX package when installing!

Some useful Notepad++ stuff

I chose NP++ because it has the cool TextFX package as standard, which here we’ll be using to URL Encode the addresses we’ll put in the Google search query string.

  1. Firstly copy and paste in all the URLs you want to search for in Google.
  2. Next click TextFx > TextFx Convert > Encode Uri Component
  3. Now you see everything is on one line, and probably thinking I’m leading you up a dead end, but stick with me.
    What NP++ has done has encoded the newline characters, now we’ll fix that.
  4. The URL encoding of a new line is %0D%0A, confused? Well it’s actually very simple.
    1. %0D => is a carriage return or \r.
    2. %0A => is a line feed or \n

    These are the characters automatically entered in by text editors when you press the enter key.

    So go to Search > Replace or ctrl H and find any instance of %0D%0A and replace with \r\n. Done.

  5. Now we don’t want to visit the site we just encoded, we want to search for it in Google. I’ll keep this very basic for now, but we now need to suffix this encoded address onto the Google search address.

    So again go to Search > Replace or ctrl H and find any instance of http and replace with http://google.co.uk/search?q=http.
  6. If you have a list that looks vaguely like this, click File > Save or ctrl s and you are good to go!

Using Grab Them All

Now you have a file to input all the URLs, the rest will be easy.

  1. Open tools > Grab Them All
  2. Click “Load file with URLs to grab” and find the text file you have just made.
  3. Click “Select destination dir” and select the folder where you want the generated images to be saved to.
  4. Click “Let’s go!”, sit back and relax.

Summing it up

This is a great technique to quickly automate some simple testing, there is barely any setup involved after the first time. Plus as all the pages rendered are saved as images you don’t even need to sit and watch it! You can go grab a drink and come back later to flick through the images of the worlds most boring photo album.

What I have used the add-on for is just one example; you could also use it for:

  1. Checking rendering similarities on separate pages of a site.
  2. Automatically generate some screenshots of your sites for your portfolio.
  3. and many, many more.

If I want more specific results, with more economy on disk space I like to use an add-on called iMacro. This is more in depth and really flexible but it requires more scripting experience and takes longer to set up these macros. I’ll also be writing a tutorial on these iMacros for the power users out there.

Oh and one final note, my results from this and my application mentioned at the top matched perfectly! I love it when something works.

Comments

comments powered by Disqus