Mocking External APIs in Python
Integrating with a third-party application is a great way to extend the functionality of your product.
However, the added value also comes with obstacles. You do not own the external library, which means that you cannot control the servers that host it, the code that comprises its logic, or the data that gets transferred between it and your app. On top of those issues, users are constantly manipulating the data through their interactions with the library.
If you want to enhance the utility of your application with a third-party API, then you need to be confident that the two systems will play nice. You need to test that the two applications interface in predictable ways, and you need your tests to execute in a controlled environment.
At first glance, it might seem like you do not have any control over a third-party application. Many of them do not offer testing servers. You cannot test live data, and even if you could, the tests would return unreliable results as the data was updated through use. Also, you never want your automated tests to connect to an external server. An error on their side could bring a halt to your development if releasing your code depends on whether your tests pass. Luckily, there is a way to test the implementation of a third-party API in a controlled environment without needing to actually connect to an outside data source. The solution is to fake the functionality of the external code using something known as mocks.
A mock is a fake object that you construct to look and act like real data. You swap it with the actual object and trick the system into thinking that the mock is the real deal. Using a mock reminds me of a classic movie trope where the hero grabs a henchman, puts on his uniform, and steps into a line of marching enemies. Nobody notices the impostor and everybody keeps moving—business as usual.
Third-party authentication, such as OAuth, is a good candidate for mocking within your application. OAuth requires your application to communicate with an external server, it involves real user data, and your application relies on its success in order to gain access to its APIs. Mocking authentication allows you to test your system as an authorized user without having to go through the actual process of exchanging credentials. In this case you do not want to test whether your system successfully authenticates a user; you want to test how your application’s functions behave after you have been authenticated.
NOTE: This tutorial uses Python v3.5.1.
Begin by setting up a new development environment to hold your project code. Create a new virtual environment and then install the following libraries:
$ pip install nose requests
Here is a quick rundown of each library you are installing, in case you have never encountered them:
- The mock library is used for testing Python code by replacing parts of your system with mock objects. NOTE: The
mocklibrary is part of
unittestif you are using Python 3.3 or greater. If you are using an older version, please install the backport mock library.
- The nose library extends the built-in Python
unittestmodule to make testing easier. You can use
unittestor other third-party libraries such as pytest to achieve the same results, but I prefer nose‘s assertion methods.
- The requests library greatly simplifies HTTP calls in Python.
For this tutorial, you will be communicating with a fake online API that was built for testing - JSON Placeholder. Before you write any tests, you need to know what to expect from the API.
First, you should expect that the API you are targeting actually returns a response when you send it a request. Confirm this assumption by calling the endpoint with cURL:
$ curl -X GET 'http://jsonplaceholder.typicode.com/todos'
This call should return a JSON-serialized list of todo items. Pay attention to the structure of the todo data in the response. You should see a list of objects with the keys
completed. You are now prepared to make your second assumption–you know what to expect the data to look like. The API endpoint is alive and functioning. You proved that by calling it from the command line. Now, write a nose test so that you can confirm the life of the server in the future. Keep it simple. You should only be concerned with whether the server returns an OK response.
# Third-party imports... from nose.tools import assert_true import requests def test_request_response(): # Send a request to the API server and store the response. response = requests.get('http://jsonplaceholder.typicode.com/todos') # Confirm that the request-response cycle completed successfully. assert_true(response.ok)
Run the test and watch it pass:
$ nosetests --verbosity=2 project test_todos.test_request_response ... ok ---------------------------------------------------------------------- Ran 1 test in 9.270s OK
Refactoring your code into a service
Chances are good that you will call an external API many times throughout your application. Also, those API calls will likely involve more logic than simply making an HTTP request, such as data processing, error handling, and filtering. You should pull the code out of your test and refactor it into a service function that encapsulates all of that expected logic.
Rewrite your test to reference the service function and to test the new logic.
# Third-party imports... from nose.tools import assert_is_not_none # Local imports... from project.services import get_todos def test_request_response(): # Call the service, which will send a request to the server. response = get_todos() # If the request is sent successfully, then I expect a response to be returned. assert_is_not_none(response)
Run the test and watch it fail, and then write the minimum amount of code to make it pass:
# Standard library imports... try: from urllib.parse import urljoin except ImportError: from urlparse import urljoin # Third-party imports... import requests # Local imports... from project.constants import BASE_URL TODOS_URL = urljoin(BASE_URL, 'todos') def get_todos(): response = requests.get(TODOS_URL) if response.ok: return response else: return None
BASE_URL = 'http://jsonplaceholder.typicode.com'
The first test that you wrote expected a response to be returned with an OK status. You refactored your programming logic into a service function that returns the response itself when the request to the server is successful. A
None value is returned if the request fails. The test now includes an assertion to confirm that the function does not return
Notice how I instructed you to create a
constants.py file and then I populated it with a
BASE_URL. The service function extends the
BASE_URL to create the
TODOS_URL, and since all of the API endpoints use the same base, you can continue to create new ones without having to rewrite that bit of code. Putting the
BASE_URL in a separate file allows you to edit it in one place, which will come in handy if multiple modules reference that code.
Run the test and watch it pass.
$ nosetests --verbosity=2 project test_todos.test_request_response ... ok ---------------------------------------------------------------------- Ran 1 test in 1.475s OK
Your first mock
The code is working as expected. You know this because you have a passing test. Unfortunately, you have a problem–your service function is still accessing the external server directly. When you call
get_todos(), your code is making a request to the API endpoint and returning a result that depends on that server being live. Here, I will demonstrate how to detach your programming logic from the actual external library by swapping the real request with a fake one that returns the same data.
# Standard library imports... from unittest.mock import Mock, patch # Third-party imports... from nose.tools import assert_is_not_none # Local imports... from project.services import get_todos @patch('project.services.requests.get') def test_getting_todos(mock_get): # Configure the mock to return a response with an OK status code. mock_get.return_value.ok = True # Call the service, which will send a request to the server. response = get_todos() # If the request is sent successfully, then I expect a response to be returned. assert_is_not_none(response)
Notice that I did not change the service function at all. The only part of the code that I edited was the test itself. First, I imported the
patch() function from the
mock library. Next, I modified the test function with the
patch() function as a decorator, passing in a reference to
project.services.requests.get. In the function itself, I passed in a parameter
mock_get, and then in the body of the test function, I added a line to set
mock_get.return_value.ok = True.
Great. So what actually happens now when the test is run? Before I dive into that, you need to understand something about the way the
requests library works. When you call the
requests.get() function, it makes an HTTP request behind the scenes and then returns an HTTP response in the form of a
Response object. The
get() function itself communicates with the external server, which is why you need to target it. Remember the image of the hero swapping places with the enemy while wearing his uniform? You need to dress the mock to look and act like the