Mocking External APIs in Python

First steps

Begin by setting up a new development environment to hold your project code. Create a new virtual environment and then install the following libraries:

$ pip install nose requests

Here is a quick rundown of each library you are installing, in case you have never encountered them:

• The mock library is used for testing Python code by replacing parts of your system with mock objects. NOTE: The mock library is part of unittest if you are using Python 3.3 or greater. If you are using an older version, please install the backport mock library.
• The nose library extends the built-in Python unittest module to make testing easier. You can use unittest or other third-party libraries such as pytest to achieve the same results, but I prefer nose‘s assertion methods.
• The requests library greatly simplifies HTTP calls in Python.

For this tutorial, you will be communicating with a fake online API that was built for testing - JSON Placeholder. Before you write any tests, you need to know what to expect from the API.

First, you should expect that the API you are targeting actually returns a response when you send it a request. Confirm this assumption by calling the endpoint with cURL:

$ curl -X GET 'http://jsonplaceholder.typicode.com/todos'

This call should return a JSON-serialized list of todo items. Pay attention to the structure of the todo data in the response. You should see a list of objects with the keys userId, id, title, and completed. You are now prepared to make your second assumption–you know what to expect the data to look like. The API endpoint is alive and functioning. You proved that by calling it from the command line. Now, write a nose test so that you can confirm the life of the server in the future. Keep it simple. You should only be concerned with whether the server returns an OK response.

project/tests/test_todos.py

# Third-party imports...
from nose.tools import assert_true
import requests


def test_request_response():
    # Send a request to the API server and store the response.
    response = requests.get('http://jsonplaceholder.typicode.com/todos')

    # Confirm that the request-response cycle completed successfully.
    assert_true(response.ok)

Run the test and watch it pass:

$ nosetests --verbosity=2 project
test_todos.test_request_response ... ok

----------------------------------------------------------------------
Ran 1 test in 9.270s

OK

Refactoring your code into a service

Chances are good that you will call an external API many times throughout your application. Also, those API calls will likely involve more logic than simply making an HTTP request, such as data processing, error handling, and filtering. You should pull the code out of your test and refactor it into a service function that encapsulates all of that expected logic.

Rewrite your test to reference the service function and to test the new logic.

project/tests/test_todos.py

# Third-party imports...
from nose.tools import assert_is_not_none

# Local imports...
from project.services import get_todos


def test_request_response():
    # Call the service, which will send a request to the server.
    response = get_todos()

    # If the request is sent successfully, then I expect a response to be returned.
    assert_is_not_none(response)

Run the test and watch it fail, and then write the minimum amount of code to make it pass:

project/services.py

# Standard library imports...
try:
    from urllib.parse import urljoin
except ImportError:
    from urlparse import urljoin

# Third-party imports...
import requests

# Local imports...
from project.constants import BASE_URL

TODOS_URL = urljoin(BASE_URL, 'todos')


def get_todos():
    response = requests.get(TODOS_URL)
    if response.ok:
        return response
    else:
        return None

project/constants.py

BASE_URL = 'http://jsonplaceholder.typicode.com'

The first test that you wrote expected a response to be returned with an OK status. You refactored your programming logic into a service function that returns the response itself when the request to the server is successful. A None value is returned if the request fails. The test now includes an assertion to confirm that the function does not return None.

Notice how I instructed you to create a constants.py file and then I populated it with a BASE_URL. The service function extends the BASE_URL to create the TODOS_URL, and since all of the API endpoints use the same base, you can continue to create new ones without having to rewrite that bit of code. Putting the BASE_URL in a separate file allows you to edit it in one place, which will come in handy if multiple modules reference that code.

Run the test and watch it pass.

$ nosetests --verbosity=2 project
test_todos.test_request_response ... ok

----------------------------------------------------------------------
Ran 1 test in 1.475s

OK

Your first mock

The code is working as expected. You know this because you have a passing test. Unfortunately, you have a problem–your service function is still accessing the external server directly. When you call get_todos(), your code is making a request to the API endpoint and returning a result that depends on that server being live. Here, I will demonstrate how to detach your programming logic from the actual external library by swapping the real request with a fake one that returns the same data.

project/tests/test_todos.py

# Standard library imports...
from unittest.mock import Mock, patch

# Third-party imports...
from nose.tools import assert_is_not_none

# Local imports...
from project.services import get_todos


@patch('project.services.requests.get')
def test_getting_todos(mock_get):
    # Configure the mock to return a response with an OK status code.
    mock_get.return_value.ok = True

    # Call the service, which will send a request to the server.
    response = get_todos()

    # If the request is sent successfully, then I expect a response to be returned.
    assert_is_not_none(response)

Notice that I did not change the service function at all. The only part of the code that I edited was the test itself. First, I imported the patch() function from the mock library. Next, I modified the test function with the patch() function as a decorator, passing in a reference to project.services.requests.get. In the function itself, I passed in a parameter mock_get, and then in the body of the test function, I added a line to set mock_get.return_value.ok = True.

Great. So what actually happens now when the test is run? Before I dive into that, you need to understand something about the way the requests library works. When you call the requests.get() function, it makes an HTTP request behind the scenes and then returns an HTTP response in the form of a Response object. The get() function itself communicates with the external server, which is why you need to target it. Remember the image of the hero swapping places with the enemy while wearing his uniform? You need to dress the mock to look and act like the requests.get() function.