multithreaded IE automation

Multithreaded browser automation (VBA Web Scraping)

1 Star2 Stars3 Stars4 Stars5 Stars (2 votes, average: 4.00 out of 5)
Loading...

Web Scraping is very useful for getting the information you need directly off websites. Sometimes however simple browser automation is not enough in terms of performance. Having created both the IE and Parallel classes I decided to jump an opportunity of creating a simple example of how mulithreaded browser automation can be achieved. Daniel Ferry achieved the same here. However, he used VBscript and a lot of coding was required to manage the swarm, as he called it. What I wanted to show you is rather an example how you can combine the Parallel class and the IE class to achieve the same functionality but in a much more concise and easy to manage way. So let’s jump straight in.

What does it do?

The browser automation procedure queries the Google page and copies the first resulting link text to the Excel workbook. In the mulithreaded example a “swarm” of 4 threads (or more if needed) is maintained to carry out simultaneous Google queries. This way the overall execution time is significantly reduced as more IE browser objects are created when some of them are waiting for a callback.

A multithreaded browser automation example
A multithreaded browser automation example

Multithreaded browser automation: Video

Instead of going into the details I encourage you to watch this short video example of a single and mulithreaded IE automation example:

Download

Feel free to download the workbook here:

Next steps

Check out the deterministic IE automation class here:
EXCEL: Simple class for using IE automation in VBA

Check out the Parallel class mulithreading tool here:
EXCEL: VBA Multithreading Tool

Related Posts

3 thoughts on “Multithreaded browser automation (VBA Web Scraping)”

  1. Nice tool !
    Piloting EXCEL/Chrome from a web page could help me much.
    I would like when browsing in Google Chrome
    1. to select a text , input some keywords of my own,
    2. add in an existing Excel file the keywords with the URL of the current page.
    3 insert the selected text in a Word file.
    It seems that those tasks cannot not be processed by calling, on a shortcut in Chrome, an EXCEL VBA routine ? It may be easier in IE, using VBA. I was looking for a Chrome Extension, as I am not allowed to install new sofware (I have goot EXCEL VBA and OFFICE 2007 2013 soon). Thank you very much for your hints. Congratulations for your work that i will test with pleasure.

    1. Thanks again! The IE Automation class can easily be adapted to input data to other forms.

      It seems that those tasks cannot not be processed by calling, on a shortcut in Chrome, an EXCEL VBA routine ?

      No, not that I am aware of. Although, I prefer my own automation solutions (Excel VBA or JS, C#, Python etc.), there are some Google Chrome Add-In to consider:

      Haven’t tested them but seem promising.

Leave a Reply