How to Download Large files in Python using Python Requests Library

Python Requests Library

How to Download Large files in Python using Python Requests Library
xresch / Pixabay

Python requests Library is a great library that allows you to send HTTP/1.1 request to Apache2 servers to serve data requested by clients. With request one can add form data, headers, multi part files etc. To know more about amazing things you can do with Python Request library, read here too.

However at times you may face problem while downloading a Big file which may be over a GB. It may become impossible to keep whole file in memory and read it in small chunks.

By default in Python whenever you make request to server for downloading any file, the body of the response is also downloaded immediately. By setting the parameter Stream as true (Stream = true) in our code you can deffer downloading the body of the response and consume it later or call Response.close to release the connection back to pool.

Python Requests Library
def download_file(url):

    fileURL = url.split('/')[-1]

    # setting parameter stream=True

    r = requests.get(url, stream=True)

    with open(fileURL, 'wb') as f:

        for chunk in r.iter_content(chunk_size=1024):

            if chunk:

                f.write(chunk)              

    return local_filename

To learn how to download files from internet using Python Programming Language, see here.