Today’s post will touch on something most analysts and VBA developers might find useful: downloading files using vba from the Web (VBA Download). Excel VBA again proves to be a versatile tool for Analytics – in this case for extracting and storing data. Downloading files can be a useful way to save data extracted from the web and to build your own data repository, or simply to make a backup of any data downloaded from the Web.
Excel can be a great tool to harness the data of the Internet. If you are more into the subject of Web Scraping I encourage you to take a look at the Excel Scrape HTML Add-In which let’s you easily download HTML content from most Web Pages without resorting to VBA. In case browser simulation is needed read on my Simple class for using IE automation in VBA.
VBA download file macro
In some cases you will need to download large files (not text/HTML) and will want to be able to control the process of downloading the data e.g. might want to interrupt the process, enable the user to interact with Excel (DoEvent) etc. In these cases the above procedure won’t do. The procedure below may however prove more efficient as it will download the file in 128 byte chunks of data instead of a single stream.
Private Const INTERNET_FLAG_NO_CACHE_WRITE = &H4000000 Private Declare Function InternetOpen Lib "wininet.dll" Alias "InternetOpenA" (ByVal lpszAgent As String, ByVal dwAccessType As Long, ByVal lpszProxyName As String, ByVal lpszProxyBypass As String, ByVal dwFlags As Long) As Long Private Declare Function InternetReadBinaryFile Lib "wininet.dll" Alias "InternetReadFile" (ByVal hfile As Long, ByRef bytearray_firstelement As Byte, ByVal lNumBytesToRead As Long, ByRef lNumberOfBytesRead As Long) As Integer Private Declare Function InternetOpenUrl Lib "wininet.dll" Alias "InternetOpenUrlA" (ByVal hInternetSession As Long, ByVal sUrl As String, ByVal sHeaders As String, ByVal lHeadersLength As Long, ByVal lFlags As Long, ByVal lContext As Long) As Long Private Declare Function InternetCloseHandle Lib "wininet.dll" (ByVal hInet As Long) As Integer Sub DownloadFile(sUrl As String, filePath As String, Optional overWriteFile As Boolean) Dim hInternet, hSession, lngDataReturned As Long, sBuffer() As Byte, totalRead As Long Const bufSize = 128 ReDim sBuffer(bufSize) hSession = InternetOpen("", 0, vbNullString, vbNullString, 0) If hSession Then hInternet = InternetOpenUrl(hSession, sUrl, vbNullString, 0, INTERNET_FLAG_NO_CACHE_WRITE, 0) Set oStream = CreateObject("ADODB.Stream") oStream.Open oStream.Type = 1 If hInternet Then iReadFileResult = InternetReadBinaryFile(hInternet, sBuffer(0), UBound(sBuffer) - LBound(sBuffer), lngDataReturned) ReDim Preserve sBuffer(lngDataReturned - 1) oStream.Write sBuffer ReDim sBuffer(bufSize) totalRead = totalRead + lngDataReturned Application.StatusBar = "Downloading file. " & CLng(totalRead / 1024) & " KB downloaded" DoEvents Do While lngDataReturned <> 0 iReadFileResult = InternetReadBinaryFile(hInternet, sBuffer(0), UBound(sBuffer) - LBound(sBuffer), lngDataReturned) If lngDataReturned = 0 Then Exit Do ReDim Preserve sBuffer(lngDataReturned - 1) oStream.Write sBuffer ReDim sBuffer(bufSize) totalRead = totalRead + lngDataReturned Application.StatusBar = "Downloading file. " & CLng(totalRead / 1024) & " KB downloaded" DoEvents Loop Application.StatusBar = "Download complete" oStream.SaveToFile filePath, IIf(overWriteFile, 2, 1) oStream.Close End If Call InternetCloseHandle(hInternet) End Sub
See effect below when executing macro:
How it works
The procedure will download the binary file in 128 byte chunks while saving the contents to the data stream and flushing it into the file once completed. In between the chunks you can call “DoEvents” to enable user interaction, inform the user of the download progress “Application.StatusBar” or do other thing including interrupting the process and closing the connection. In case you want to do a proper Progress Bar and inform the user of the % progress you may want to leverage this solution.
Let us use the procedure above to download a simple text file from AnalystCave.com:
Sub TestDownload() DownloadFile "http://analystcave.com/junk.txt", ThisWorkbook.Path & "\junk.txt", True End Sub
Download the example
Provide your email to get the link to a file with an example usage of this script:
Reading / writing files in VBA
So you know how to download files using VBA. The next step is learning how to read files using VBA.