Get Started With the Requests Library in Python – Tutorial

The requests library is the de facto standard for making HTTP requests in Python. It abstracts the complexities of making requests behind a beautiful, simple API so that you can focus on interacting with services and consuming data in your application.
In this post on Requests Tutorial, I will explain to you all the basics of Requests Module and how you can send HTTP/1.1 requests using Python. By the end of this blog, you will be able to perform web scraping using Python.I will be covering the following topics in this post:

What is Requests module?
Installing Requests module
Making a GET Requests
Downloading Image with Requests
Making POST Requests
Sending Cookies and Headers

What Is Requests Module?

Requests is a Python module that you can use to send all kinds of HTTP requests. It is an easy-to-use library with a lot of features ranging from passing parameters in URLs to sending custom headers and SSL Verification. In this tutorial, you will learn how to use this library to send simple HTTP requests in Python.

Requests allow you to send HTTP/1.1 requests. You can add headers, form data, multi-part files, and parameters with simple Python dictionaries, and access the response data in the same way.

Installing The Requests module

Before we can do anything, we need to install the library. So let’s go ahead and install requests using pip. It’s a good idea to create a virtual environment first if you don’t already have one.

$ pip install requests

Making a GET Request

HTTP methods such as GET and POST, determine which action you’re trying to perform when making an HTTP request. Besides GET and POST, there are several other common methods.

One of the most common HTTP methods is GET. The GET method indicates that you’re trying to get or retrieve data from a specified resource. To make a GET request, invoke requests.get().

To test this out, you can make a GET request to GitHub’s Root REST API by calling get() with the following URL:

>>> requests.get(‘https://api.github.com’)

Congratulations! You’ve made your first request. Let’s dive a little deeper into the response of that request.

The Response

A Response is a powerful object for inspecting the results of the request. Let’s make that same request again, but this time store the return value in a variable so that you can get a closer look at its attributes and behaviors:

>>> response = requests.get(‘https://api.github.com’)

In this example, you’ve captured the return value of get(), which is an instance of Response, and stored it in a variable called response. You can now use response to see a lot of information about the results of your GET request.

Status Codes

The first bit of information that you can gather from Response is the status code. A status code informs you of the status of the request.

For example, a 200 OK status means that your request was successful, whereas a 404 NOT FOUND status means that the resource you were looking for was not found. There are many other possible status codes as well to give you specific insights into what happened with your request.

By accessing .status_code, you can see the status code that the server returned:

>>> response.status_code 200

.status_code returned a 200, which means your request was successful and the server responded with the data you were requesting.
Now, you know a lot about how to deal with the status code of the response you got back from the server. However, when you make a GET request, you rarely only care about the status code of the response. Usually, you want to see more. Next, you’ll see how to view the actual data that the server sent back in the body of the response.

Content

The response of a GET request often has some valuable information, known as a payload, in the message body. Using the attributes and methods of Response, you can view the payload in a variety of different formats.

To see the response’s content in bytes, you use .content:

>>> response.content
b'{“—“}

While .content gives you access to the raw bytes of the response payload, you will often want to convert them into a string using a character encoding such as UTF-8. response will do that for you when you access .text:

>>> >>> response.text
{“—“}

Because the decoding of bytes to a str requires an encoding scheme, requests will try to guess the encoding based on the response’s headers if you do not specify one. You can provide an explicit encoding by setting .encoding before accessing .text:

>>> response.encoding = ‘utf-8’
>>> response.text
{“—“}

Headers

The response headers can give you useful information, such as the content type of the response payload and a time limit on how long to cache the response. To view these headers, access .headers:

>>> response.headers

.headers returns a dictionary-like object, allowing you to access header values by key. For example, to see the content type of the response payload, you can access Content-Type:

>>> response.headers[‘Content-Type’]
‘application/json; charset=utf-8

There is something special about this dictionary-like headers object, though. The HTTP spec defines headers to be case-insensitive, which means we are able to access these headers without worrying about their capitalization. Whether you use the key ‘content-type’ or ‘Content-Type’, you’ll get the same value

Downloading An Image Using Requests Module

So let’s download the following image of a forest on Pixabay using the Requests module we learned about. Here is the actual image

This is the code that you will need to download the image:

import requests
req = requests.get(‘path/to/forest.jpg’, stream=True)
req.raise_for_status()
with open(‘forest.jpg’, ‘wb’) as fd:
for chunk in req.iter_content(chunk_size=10000):
print(‘Received a Chunk’)
fd.write(chunk)

Note that the ‘path/to/forest.jpg’ is the actual image URL. You can put the URL of any other image here to download something else as well. This is just an example showed here and the given image file is about 547kb in size and you have set chunk_size to 10,000 bytes.
Requests also allow you to pass parameters in a URL. This is particularly helpful when you are searching for a webpage for some results like a tutorial or a specific image. You can provide these query strings as a dictionary of strings using the params keyword in the GET request. Check out this easy example:

import requests

query = ‘q’: ‘Forest’, ‘order’: ‘popular’, ‘min_width’: ‘800’, ‘min_height’: ‘600’
req = requests.get(‘https://pixabay.com/en/photos/’, params=query)

req.url
# returns ‘https://pixabay.com/en/photos/ ?order=popular&min_ height=600&q=Forest&min_ width=800

Making a POST Request

Making a POST request is just as easy as making GET requests. You just use the post() function instead of get().

This can be useful when you are automatically submitting forms. For example, the following code will download the whole Wikipedia page on Python and save it on your PC

import requests
req = requests.post(‘https://en.wikipedia.org/w/index.php’, data = {‘search’:’Python’})
req.raise_for_status()
with open(‘Python.html’, ‘wb’) as fd:
for chunk in req.iter_content(chunk_size=10000):
fd.write(chunk)

Sending Cookies and Headers

As previously mentioned, you can access the cookies and headers that the server sends back to you using req.cookies and req.headers. Requests also allow you to send your own custom cookies and headers with a request. This can be helpful when you want to, let’s say, set a custom user agent for your request.

To add HTTP headers to a request, you can simply pass them in a dict to the headers parameter. Similarly, you can also send your own cookies to a server using a dict passed to the cookies parameter

import requests url = ‘http://some-domain.com/set/cookies/headers’

headers = {‘user-agent’: ‘your-own-user-agent/0.0.1’}
cookies = {‘location’: ‘noida’}

req = requests.get(url, headers=headers, cookies=cookies

Next up on this “Requests Tutorial” blog, let us look at session objects!

The Session Object

Until now, you’ve been dealing with high level requests APIs such as get() and post(). These functions are abstractions of what’s going on when you make your requests. They hide implementation details such as how connections are managed so that you don’t have to worry about them.

Underneath those abstractions is a class called Session. If you need to fine-tune your control over how requests are being made or improve the performance of your requests, you may need to use a Session instance directly.

The Session object uses urllib3’s connection pooling. This means that the underlying TCP connection will be reused for all the requests made to the same host.

This can significantly boost the performance. You can also use methods of the Requests object with the Session object.

Sessions are used to persist parameters across requests. For example, if you want to use the same authentication across multiple requests, you could use a session

import requests
from getpass import getpass

# By using a context manager, you can ensure the resources used by
# the session will be released after use
with requests.Session() as session:
session.auth = (‘username’, getpass())

# Instead of requests.get(), you’ll use session.get()
response = session.get(‘https://api.github.com/user’)

# You can inspect the response just like you did before
print(response.headers)
print(response.json()

Each time you make a request with session, once it has been initialized with authentication credentials, the credentials will be persisted.

The primary performance optimization of sessions comes in the form of persistent connections. When your app makes a connection to a server using a Session, it keeps that connection around in a connection pool. When your app wants to connect to the same server again, it will reuse a connection from the pool rather than establishing a new one.

Conclusion

The concepts discussed in this tutorial should help you make basic requests to a server by passing specific headers, cookies, or query strings.

This will be very handy when you are trying to scrape some webpages for information. Now, you should also be able to automatically download music files and wallpapers from different websites once you have figured out a pattern in the URLs.

After reading this blog on Requests tutorial using Python, I am pretty sure you want to know more about Python. To know more about Python you can refer course offered by Mildaintrainings. Learn Python programming from scratch through Python programming certification course. This Python Course will help you master important Python programming concepts such as data & file operations in Python, object-oriented concepts in Python & various Python libraries such as Pandas, Numpy, Matplotlib, and so on. This Python certification course is also a gateway towards your Data Science career. The course is curated by industry experts which includes real-time case studies. Enroll & Get Certified now!

I hope you have enjoyed this post on Requests Tutorial. To know more about Python you can refer the following blogs: