Today’s post will touch on something most analysts and VBA developers might find useful: downloading files using vba from the Web (VBA Download). Excel VBA again proves to be a versatile tool for Analytics – in this case for extracting and storing data. Downloading files can be a useful way to save data extracted from the web and to build your own data repository, or simply to make a backup of any data downloaded from the Web.

Excel can be a great tool to harness the data of the Internet. If you are more into the subject of Web Scraping I encourage you to take a look at the Excel Scrape HTML Add-In which let’s you easily download HTML content from most Web Pages without resorting to VBA. In case browser simulation is needed read on my Simple class for using IE automation in VBA.

VBA download file macro

In some cases you will need to download large files (not text/HTML) and will want to be able to control the process of downloading the data e.g. might want to interrupt the process, enable the user to interact with Excel (DoEvent) etc. In these cases the above procedure won’t do. The procedure below may however prove more efficient as it will download the file in 128 byte chunks of data instead of a single stream.

Private Const INTERNET_FLAG_NO_CACHE_WRITE = &H4000000
Private Declare Function InternetOpen Lib "wininet.dll" Alias "InternetOpenA" (ByVal lpszAgent As String, ByVal dwAccessType As Long, ByVal lpszProxyName As String, ByVal lpszProxyBypass As String, ByVal dwFlags As Long) As Long
Private Declare Function InternetReadBinaryFile Lib "wininet.dll" Alias "InternetReadFile" (ByVal hfile As Long, ByRef bytearray_firstelement As Byte, ByVal lNumBytesToRead As Long, ByRef lNumberOfBytesRead As Long) As Integer
Private Declare Function InternetOpenUrl Lib "wininet.dll" Alias "InternetOpenUrlA" (ByVal hInternetSession As Long, ByVal sUrl As String, ByVal sHeaders As String, ByVal lHeadersLength As Long, ByVal lFlags As Long, ByVal lContext As Long) As Long
Private Declare Function InternetCloseHandle Lib "wininet.dll" (ByVal hInet As Long) As Integer

Sub DownloadFile(sUrl As String, filePath As String, Optional overWriteFile As Boolean)
  Dim hInternet, hSession, lngDataReturned As Long, sBuffer() As Byte, totalRead As Long
  Const bufSize = 128
  ReDim sBuffer(bufSize)
  hSession = InternetOpen("", 0, vbNullString, vbNullString, 0)
  If hSession Then hInternet = InternetOpenUrl(hSession, sUrl, vbNullString, 0, INTERNET_FLAG_NO_CACHE_WRITE, 0)
  Set oStream = CreateObject("ADODB.Stream")
  oStream.Open
  oStream.Type = 1

  If hInternet Then
    iReadFileResult = InternetReadBinaryFile(hInternet, sBuffer(0), UBound(sBuffer) - LBound(sBuffer), lngDataReturned)
    ReDim Preserve sBuffer(lngDataReturned - 1)
    oStream.Write sBuffer
    ReDim sBuffer(bufSize)
    totalRead = totalRead + lngDataReturned
    Application.StatusBar = "Downloading file. " & CLng(totalRead / 1024) & " KB downloaded"
    DoEvents

    Do While lngDataReturned <> 0
      iReadFileResult = InternetReadBinaryFile(hInternet, sBuffer(0), UBound(sBuffer) - LBound(sBuffer), lngDataReturned)
      If lngDataReturned = 0 Then Exit Do

      ReDim Preserve sBuffer(lngDataReturned - 1)
      oStream.Write sBuffer
      ReDim sBuffer(bufSize)
      totalRead = totalRead + lngDataReturned
      Application.StatusBar = "Downloading file. " & CLng(totalRead / 1024) & " KB downloaded"
      DoEvents
    Loop

    Application.StatusBar = "Download complete"
    oStream.SaveToFile filePath, IIf(overWriteFile, 2, 1)
    oStream.Close
  End If
  Call InternetCloseHandle(hInternet)
End Sub

See effect below when executing macro:

How it works

The procedure will download the binary file in 128 byte chunks while saving the contents to the data stream and flushing it into the file once completed. In between the chunks you can call “DoEvents” to enable user interaction, inform the user of the download progress Application.StatusBar or do other thing including interrupting the process and closing the connection. In case you want to do a proper Progress Bar and inform the user of the % progress you may want to leverage this solution.

Example

Let us use the procedure above to download a simple text file from AnalystCave.com:

Sub TestDownload()
    DownloadFile "https://analystcave.com/junk.txt", ThisWorkbook.Path & "\junk.txt", True
End Sub

Download the example

Download

Reading / writing files in VBA

So you know how to download files using VBA. The next step is learning how to read files using VBA.