Python requests Library: Sending HTTP Requests with Python

The Python requests library allows you to send HTTP requests, such as GET and POST, and handle responses from web servers. requests is a user-friendly and powerful module for interacting with web resources. This is where Python 3 makes working with web protocols especially accessible for beginners.

Quick Answer: How to Use the requests Module in Python

The requests library is the standard way to make HTTP requests in Python. First, you must install it (pip install requests), then import it into your script. The most common use is making a GET request to fetch data from a URL.

Basic Steps & Example:

1. Install the library: pip install requests
2. Import it: import requests
3. Make a request: Use requests.get() for GET requests or requests.post() for POST requests.
4. Check the response: The function returns a Response object. Check response.status_code to see if it was successful (200 means OK).
5. Access the content: Use response.text for HTML/text or response.json() for JSON data.

import requests

try:
    # Make a GET request to an API endpoint
    response = requests.get('https://api.github.com/users/mimo-org')

    # Check if the request was successful
    if response.status_code == 200:
        # Parse the JSON response into a dictionary
        data = response.json()
        print(f"Successfully fetched data for user: {data['login']}")
        print(f"Public repos: {data['public_repos']}")
    else:
        print(f"Error: Received status code {response.status_code}")

except requests.exceptions.RequestException as e:
    print(f"An error occurred: {e}")

How to Use the requests Module in Python

To work with the requests library, you first need to install it, for example, using pip. This step ensures proper encoding support for request and response data.

pip install requests

Once installed, you can import the library and send requests to URLs.

The requests library supports multiple HTTP methods, including GET, POST, PUT, and DELETE. Usually, you pick the method based on your desired operation (e.g., fetching data, submitting forms, or updating resources). These operations often rely on http headers to properly communicate between client and server:

• requests.get(): Sends a GET request to the specified URL.
• requests.post(): Creates a POST request, often to submit data to a server.
• requests.put(): Sends a PUT request, updating a resource on the server.
• requests.delete(): The DELETE request deletes a resource on the server.

How to Send a Python Request (GET)

In Python, GET requests using requests.get() retrieve information from a server without modifying any data. GET is the most commonly used HTTP method for accessing web pages, APIs, and other resources. Unlike other methods, the GET request parameters are part of the URL, making it ideal for read-only operations. This includes working with query strings for filtering data.

import requests

# Send a GET request to a webpage
response = requests.get('<https://example.com>')

# Print the status code of the response
print(response.status_code)

In this example, requests.get() sends an HTTP GET request to a URL while response.status_code provides the HTTP status code of the request (e.g., 200 for success). The returned response object contains useful metadata like headers, cookies, and text.

How to Send a POST Request in Python

While GET retrieves data, the Python request POST sends data to a server to create or update resources. Typically, POST submits information, such as form data, user input, or API payloads, as part of the request's body. This often includes a content-type header to specify the data format.

import requests

url = '<https://api.example.com/data>'
data = {'name': 'John', 'age': 30}

response = requests.post(url, json=data)

# Print the response content
print(response.json())

In this example, requests.post() sends an HTTP POST request to a URL and passes data in JSON format. The response.json() method parses the JSON-formatted response from the server.

When to Use the Python Module requests

The requests library in Python is well-suited whenever you need your Python program to interact with web resources. Many docs recommend requests for working with APIs due to its simplicity and reliability.

Fetching Web Page Content

You can use requests.get() to retrieve and process content from a webpage, also known as web scraping.

import requests

response = requests.get('<https://example.com>')
print(response.text)

This code sends a GET request to fetch the webpage content and prints the HTML response.

Interacting with APIs

The requests Python library is ideal for working with RESTful APIs. The library allows you to send and receive JSON data from APIs without much effort.

import requests

api_url = '<https://api.example.com/users/123>'
response = requests.get(api_url)
data = response.json()

print(data)

This example retrieves data from a user profile on an API and prints the parsed JSON response.

Sending Form Data

You can use requests.post() to send form data to a server, which is helpful for login forms or submitting search queries.

import requests

url = '<https://example.com/login>'
form_data = {'username': 'user123', 'password': 'mypassword'}

response = requests.post(url, data=form_data)
print(response.status_code)

In this example, the data parameter is used to submit form fields to the server.

Examples of Using Python Requests

Web Scraping for Price Comparison

A price comparison platform might import Python requests to scrape prices from different e-commerce websites and compare them. You might encounter sites hosted on github.com that expose structured JSON data for public scraping.

import requests

url = '<https://example.com/product-page>'
response = requests.get(url)

if response.status_code == 200:
    page_content = response.text
    # Logic to extract price from the HTML content
    print("Page loaded successfully")
else:
    print("Failed to load page")

User Data Automation

Automation scripts can use the requests library to interact with third-party APIs. For example, a company might automate updating user data.

import requests

url = '<https://api.example.com/update_user>'
data = {'user_id': 123, 'new_email': 'new@example.com'}

response = requests.post(url, json=data)
print(response.json())

Website Availability Monitors

Website administrators might use Python requests to create scripts that monitor the availability of the websites and services they manage.

import requests

url = '<https://example.com>'
response = requests.get(url)

if response.status_code == 200:
    print("Website is up!")
else:
    print("Website is down!")

Learn More About Python Requests

Adding Python requests Headers

You can customize the headers by providing a header parameter with the request. A custom header parameter can help you with authorization or other customization purposes.

import requests

url = '<https://api.example.com/data>'
headers = {'Authorization': 'Bearer token123'}

response = requests.get(url, headers=headers)
print(response.status_code)

Handling Python requests Timeouts

You can set a timeout with the timeout parameter to avoid long waits for requests that take too long to complete.

import requests

try:
    response = requests.get('<https://example.com>', timeout=5)
    print(response.status_code)
except requests.Timeout:
    print("Request timed out")

This example sets a timeout of 5 seconds. If the server doesn’t respond within that time, the request raises a Timeout exception.

Using URL HTTP Parameters

You can add query parameters to your requests using the params parameter.

import requests

url = '<https://example.com/search>'
params = {'q': 'python', 'page': 2}

response = requests.get(url, params=params)
print(response.url)

This example sends a GET request with URL parameters (q for a search query and page for pagination).

Sending Files with Python Requests (POST)

Within a POST request, you can use the files parameter to upload files and other file-like objects.

import requests

url = '<https://example.com/upload>'
files = {'file': open('example.txt', 'rb')}

response = requests.post(url, files=files)
print(response.status_code)

Handling JSON Responses

If a server returns JSON data, you can use response.json() to parse the response directly into Python objects.

import requests

url = '<https://api.example.com/data>'
response = requests.get(url)

data = response.json()
print(data)

Authentication with Usernames and Passwords or Tokens

APIs or websites often require authentication through usernames and passwords or, in more modern applications, tokens. You can add credentials to your request using the auth parameter in any requests method.

import requests
from requests.auth import HTTPBasicAuth

url = '<https://example.com/api>'
response = requests.get(url, auth=HTTPBasicAuth('username', 'password'))

print(response.status_code)

Many APIs use tokens instead of passwords, which you can include in the request headers as well.

import requests

url = '<https://api.example.com/data>'
headers = {'Authorization': 'Bearer your_token_here'}

response = requests.get(url, headers=headers)

print(response.status_code)

In this case, the token is sent as part of the Authorization header using the Bearer token scheme, which is standard with API and OAuth tokens.

Handling HTTP Status Codes

HTTP status codes indicate the result of a request. The requests library makes it easy to check these status codes and handle responses based on them.

import requests

response = requests.get('<https://example.com>')

# Check if the request was successful
if response.status_code == 200:
    print("Success!")
elif response.status_code == 404:
    print("Not Found!")
else:
    print(f"Status Code: {response.status_code}")

• 200 OK: The request was successful.
• 201 Created: The request successfully created a new resource.
• 204 No Content: The request was successful, but there’s no content to return.
• 400 Bad Request: The request was invalid or otherwise impossible to process.
• 401 Unauthorized: To access the resource, a username/password, a token, or similar credentials is necessary.
• 403 Forbidden: You're authenticated but lack permission to access the resource.
• 404 Not Found: The requested resource doesn't exist or is no longer available.
• 500 Internal Server Error: The server encountered an error processing the request.

Checking the status code allows you to evaluate whether your HTTP request was successful. In case of an error, the status code hints at what went wrong.

Using urllib for Requests in Python

In Python, HTTP requests can also use the built-in library urllib.

import urllib.request

url = '<https://example.com>'
response = urllib.request.urlopen(url)
html = response.read()

print(html)

In this example, urllib.request.urlopen() sends a GET request to the URL while response.read() reads the response's content.

While urllib can be enough for basic tasks, the requests library is much more straightforward to use. However, you might still encounter urllib in legacy code or lightweight projects that want to avoid external dependencies.

Key Takeaways for the Python requests Library

• It's a Third-Party Library: requests is not part of Python's standard library, so you must install it first with pip install requests.
• Use requests.get() for Fetching Data: This is the most common method, used to retrieve information from a URL.
• Use requests.post() for Sending Data: Use this method to send data (like a form or JSON payload) to a server. Pass your data to the data= or json= parameter.
• Always Check the Status Code: After making a request, check the response.status_code attribute to ensure the request was successful (a code of 200 means "OK").
• Use .text or .json() for Content: Access the raw text/HTML content of the response with response.text. If the response is in JSON format, use the response.json() method to automatically parse it into a Python dictionary or list.