Haki Benita - Django

Reliable Django Signals

2025-10-30T00:00:00+02:00

Django signals are extremely useful for decoupling modules and implementing complicated workflows. However, the underlying transport for signals makes them unreliable and subject to unexpected failures.

In this article, I present an alternative transport implementation for Django signals using background tasks which makes them reliable and safer to use in mission critical workflows.

Table of Contents

A Common Workflow

Say you have an application that accept payments from users. Usually, you don't go and implement your own payment solution. Instead, you integrate with some 3rd-party provider.

Creating a Payment Process

This is a common workflow for integrating with a 3rd-party payment provider:

You create some payment process in the provider's system
You redirect the user to some URL, or you get something to pass to the provider's client SDK
Sometime in the future you get notified about the status of the payment, usually by webhook or redirect

A simple state machine for a payment process can look like this:

payment process state machine

To keep track of payments you create a simple Django module:

# payment/models.py

from typing import Literal, Self
from django.db import models, transaction


class Error(Exception):
    # Abstract.
    pass

class StateError(Error):
    pass


class PaymentProcess(models.Model):
    id = models.BigAutoField(primary_key=True)
    amount = models.BigIntegerField()
    status: Literal['initiated', 'succeeded', 'failed'] = models.CharField(max_length=20)

    @classmethod
    def create(cls, *, amount: int) -> Self:
        assert amount > 0
        return cls.objects.create(amount=amount, status='initiated')

    @classmethod
    def set_status(cls, id: int, *, succeeded: bool) -> Self:
        with transaction.atomic():
            payment_process = cls.objects.select_for_update(of=('self', ), no_key=True).get(id=id)
            if payment_process.status not in {'initiated'}:
                raise StateError()
            if succeeded:
                payment_process.status = 'succeeded'
            else:
                payment_process.status = 'failed'
            payment_process.save()

        return payment_process

To create a new payment process you provide an amount:

>>> from payment.models import PaymentProcess
>>> pp = PaymentProcess.create(amount=100_00)
>>> print(vars(pp))
{'id': 1, 'amount': 10000, 'status': 'initiated'}

The initial status is "initiated". At some point you'll make an API call to your payment provider, get some ID and pass it over to the client - this is outside the scope of this article.

Next, the user interacts with the 3rd-party to provide their payment details. When the user is done, you get an update on the outcome of the payment, usually a webhook or a redirect, and you set the status of the payment process in your local database:

>>> pp = PaymentProcess.set_status(1, succeeded=True)
>>> print(vars(pp))
{'id': 1, 'amount': 10000, 'status': 'succeeded'}

The payment succeeded and the status is set to "succeeded". So far so good!

Now that you can process payment, you can move on to handling orders.

Placing an Order

In your website, the user browse around until they find something they want, and proceed to checkout. At this point, you calculate the amount to be paid and create an order with a payment process. The user then interacts with the payment process to complete the payment. Based on the outcome of the payment, you decide if the order should be filled or cancelled.

A state machine for an order can look like this:

order state machine

To keep track of orders you create a new "orders" module:

from django.db import models, transaction
from payment.models import PaymentProcess
from typing import Literal, Self

class Order(models.Model):
    id = models.BigAutoField(primary_key=True)
    payment_process = models.ForeignKey(PaymentProcess, on_delete=models.PROTECT)
    amount = models.BigIntegerField()
    status: Literal['pending_payment', 'completed', 'cancelled'] = models.CharField(max_length=20)

    @classmethod
    def create(cls, *, amount: int) -> Self:
        assert amount > 0
        with transaction.atomic():
            payment_process = PaymentProcess.create(amount=amount)
            order = cls.objects.create(
                payment_process=payment_process,
                amount=amount,
                status='pending_payment',
            )
        return order

To create the order you provide an amount to charge. The module then goes and create a payment process for the same amount and associates it with your order via a foreign key:

>>> o = Order.create(amount=120_00)
>>> print(vars(o))
{ 'id': 1, 'payment_process_id': 2, 'amount': 12000, 'status': 'pending_payment'}
>>> print(vars(o.payment_process))
{ 'id': 2, 'amount': 12000, 'status': 'initiated'}

In real life, when you create an order you keep a lot more information such as the user who placed the order, the items, shipping information and so on. All of this is not important for this article, so we ignore it.

Decoupling Modules

The initial state of an order is "pending_payment" and the current state of the payment is "initiated". The next step is for the user to complete the payment.

When a payment is updated, we need to update the state of the order. Here is a function that given a payment process, sets the status of the order:

# order/models.py
class Order(models.Model):
    # ...

    @classmethod
    def on_payment_completed(cls, *, payment_process_id: int) -> Self:
        """Update the order status based on the payment process status."""
        with transaction.atomic():
            order = (
                cls.objects
                .select_related('payment_process')
                .select_for_update(of=('self', ), no_key=True)
                .get(payment_process_id=payment_process_id)
            )
            if order.status not in {'pending_payment'}:
                return order

            match order.payment_process.status:
                case 'succeeded':
                    order.status = 'completed'
                case 'failed':
                    order.status = 'cancelled'
                case 'initiated':
                    assert False, f'Unexpected payment process status "{order.payment_process.status}"'
                case ever:
                    assert_never(ever)

            order.save()

        return order

The function first looks for the order that references the payment process. If the status of the order is not "pending_payment", we assume this function was already called, and we return the order. This provides some level of idempotency. In real life, you probably should verify that the current state of the order matches the state of the provided payment process.

Next, update the status of the order based on the status of the payment process, save to the database, and return the updated order.

This is where it gets hairy...

Who's in charge of calling this function? The order is not aware of changes to the payment process, so what's triggering this function?

Circular Dependency

When you create an order, the order creates a payment process. The order module is referencing the payment module using a foreign key, therefore, the order module depends on the payment module:

module dependencies

In our workflow, after the user completes the payment, the payment module receives a webhook with the outcome of the payment, and the status of the payment process is updated. Our order is not aware of changes to the payment process, so at what point do we trigger a change in the order?

A naive way of doing this is to simply update the order directly from the payment process using the reverse relation:

diff --git i/payment/models.py w/payment/models.py
 from typing import Literal, Self
 from django.db import models, transaction
+from order.models import Order

@@ -13,23 +13,25 @@  class PaymentProcess(models.Model):
     @classmethod
     def set_status(cls, id: int, *, succeeded: bool) -> Self:
         with transaction.atomic():
             payment_process = cls.objects.select_for_update(of=('self', ), no_key=True).get(id=id)
             if payment_process.status not in {'initiated'}:
                 raise StateError()
             if succeeded:
                 payment_process.status = 'succeeded'
             else:
                 payment_process.status = 'failed'
             payment_process.save()

+            Order.on_payment_completed(payment_process_id=payment_process.id)

         return payment_process

Now, when the payment process is updated, it explicitly goes to the order and attempts to update it as well. However, if you try to execute this, you'll get an exception:

$./manage.py check
Traceback (most recent call last):
    ... snipped ...
ImportError: cannot import name 'PaymentProcess' from partially initialized
module 'payment.models' (most likely due to a circular import) (payment/models.py)

Python is warning us about a circular dependency! An order currently references a payment process. With this change, the payment process is referencing the order back - this creates a circular dependency:

circular dependency

There is a way to make this work - you can import the order inside the function - but you should really avoid that. A circular dependency is usually a symptom of bad design!

Another reason this is a bad approach is that payment process is a low level module - it can potentially be used by other modules other than order. Should a low level module like payment be aware of all the modules that are using it? This won't scale well and will cause a web of dependencies within the application.

Polling Changes

To avoid circular dependency we can't have the payment process reference the order directly. Another approach, is for the order to periodically check for changes in relevant payment process:

from django.core.management.base import BaseCommand
from ...models import Order

class Command(BaseCommand):
    help = 'Check all orders with pending payment and update their status if needed.'

    def handle(self, *args, **options):
        for payment_process_id in Order.objects.filter(
            status='pending_payment',
            payment_process__status__in=['succeeded', 'failed'],
        ).values_list('payment_process_id', flat=True):
            order = Order.on_payment_completed(payment_process_id=payment_process_id)
            self.stdout.write(self.style.SUCCESS(f'order {order.id} status changed to "{order.status}"'))

This Django management command is looking for orders pending payment with a payment process that reached either "failed" or "succeeded" state, and triggers a status update for the order.

To demonstrate, create an order and mark the payment as successful:

>>> o = Order.create(amount=120_00)
>>> o.payment_process_id
3
>>> pp = PaymentProcess.set_status(3, succeeded=True)
>>> print(vars(pp))
{ 'id': 2, 'payment_process_id': 3, 'amount': 12000, 'status': 'pending_payment'}
>>> o.refresh_from_db()
>>> print(vars(o))
{ 'id': 3, 'amount': 12000, 'status': 'initiated'}

Notice that the payment completed successfully, but the order is pending payment. Let's use the management command to sync the state:

$ ./manage.py sync_orders_pending_payment
order 2 status changed to "completed"

Great! You can now execute this task on a schedule and your orders will eventually reach the correct state.

This approach has several advantages and disadvantages:

Extremely reliable: As long as you can keep the command running on a schedule, your orders are going to get synced. This approach is extremely reliable and resilient to a wide variety of unexpected failures.
Duplicated business logic: To identify the orders that needs to be synced you had to duplicate some of the business logic. This means that as you add statuses and make the workflow more complicated, you'll have to constantly maintain this logic in more than one place. There are ways to keep things DRY, such as using a custom QuerySet or an access method, but that requires a certain level of discipline which is not always easy to maintain.
Delays the process: If for example you run the task every hour, it means an order can take at most one hour to reach completed state. In some situations this is acceptable, but in other cases, such as in user-facing workflows, this is unacceptable. You can reduce the interval, but this also comes at a cost.
Load on the system: Running a task on a schedule means it's running even if it has nothing to do. With a long interval you introduce longer delays in the process, but potentially less unnecessary load on the system. With a short interval you reduce the delay on the process, but add potentially more unnecessary load on the system.
Cumbersome: The implementation is fairly straightforward and running tasks on a schedule is something most systems already have. However, as the number of workflows increase, it becomes hard to track and debug.

Using scheduled tasks is a good and reliable solution. However, for user-facing workflows that require quick response it's often not a good fit.

Django Signals

So far we tried to trigger a change in the order from the payment it references. This caused circular dependencies so we decided it's a bad idea. We then tried polling for changes which proved to be reliable and simple, but introduced unacceptable delays in the workflow.

To address these challenges, Django provides signals dispatcher as a way to communicate between modules in the system:

Django includes a “signal dispatcher” which helps decoupled applications get notified when actions occur elsewhere in the framework. In a nutshell, signals allow certain senders to notify a set of receivers that some action has taken place.

Using signals dispatcher, we can dispatch a signal and have one or more receivers subscribe to it. In our case, the payment process can send a signal when it completes, and the order can subscribe to it and update its status. Using signals the payment module can communicate with other modules in the system without explicitly depending on them!

decouple modules using signals

First, define the signal:

# payment/signals.py.
from django.dispatch import Signal

payment_process_completed = Signal()

Next, send the signal when the payment completes:

diff --git i/payment/models.py w/payment/models.py
 from typing import Literal, Self
 from django.db import models, transaction

+from . import signals

@@ -13,23 +15,28 @@ class PaymentProcess(models.Model):
     @classmethod
     def set_status(cls, id: int, *, succeeded: bool) -> Self:
         with transaction.atomic():
             payment_process = cls.objects.select_for_update(of=('self', ), no_key=True).get(id=id)
             if payment_process.status not in {'initiated'}:
                 raise StateError()
             if succeeded:
                 payment_process.status = 'succeeded'
             else:
                 payment_process.status = 'failed'
             payment_process.save()

+            signals.payment_process_completed.send(
+                sender=None,
+                payment_process_id=payment_process.id,
+            )
+
         return payment_process

The order can now register a receiver that will be executed when the "payment_process_completed" signal is sent. This requires some minor adjustments on the receiving end:

diff --git a/order/models.py b/order/models.py
+from __future__ import annotations
 from django.db import models, transaction
 from payment.models import PaymentProcess
 from typing import Literal, Self, assert_never
+from django.dispatch import receiver

+import payment.signals

@@ -21,16 +24,22 @@ class Order(models.Model):
-    @classmethod
-    def on_payment_completed(cls, *, payment_process_id: int) -> Self:
+    @staticmethod
+    @receiver(payment.signals.payment_process_completed, dispatch_uid='1da6190f-0cf1-45e1-8481-0d1e27bf6e6f')
+    def on_payment_completed(payment_process_id: int, *args, **kwargs) -> Order | None:
         """Update the order status based on the payment process status."""
         with transaction.atomic():
-            order = (
-                cls.objects
-                .select_related('payment_process')
-                .select_for_update(of=('self', ), no_key=True)
-                .get(payment_process_id=payment_process_id)
-            )
+            try:
+                order = (
+                    Order.objects
+                    .select_related('payment_process')
+                    .select_for_update(of=('self', ), no_key=True)
+                    .get(payment_process_id=payment_process_id)
+                )
+            except Order.DoesNotExist:
+                # Not related to order.
+                return None
+
             if order.status not in {'pending_payment'}:
                 return order

Let's break it down:

Use the @receiver decorator the register on_payment_completed for execution when the payment_process_completed signal is sent.
Receiver functions require that you accept args and kwargs, so add that.
Setting a unique dispatch_uid to prevent duplicate signals.
Signal receiver functions cannot be class or instance methods. To keep the function namespaced to the Order model, we switch to using a @staticmethod instead.
Static methods don't operate in the context of an enclosed class, so we can't use the Self type to indicate that the function is returning an Order instance. Instead, we explicitly use Order as the return type.
Payment processes can potentially be used by modules other than order, and a signal can have many receivers. If we don't find a matching order for a payment process, it probably means it was intended for another model. Since this is now a valid scenario, we don't fail in this case.

We can now see it in action:

>>> o = Order.create(amount=150_00)
>>> print(vars(o))
{'id': 3, 'payment_process_id': 4, 'amount': 15000, 'status': 'pending_payment'}
>>> PaymentProcess.set_status(4, succeeded=True)
>>> o.refresh_from_db()
>>> print(vars(o))
{'id': 3, 'payment_process_id': 4, 'amount': 15000, 'status': 'completed'}

Notice how the state of the order changes to "completed" even though we did not explicitly call Order.on_payment_completed. This function was invoked implicitly when PaymentProcess.set_status dispatched the signal.

Using signals we can trigger changes in other modules without creating direct dependencies between them. As the documentation promised, signals allow us to keep modules decoupled. In our scenario, using signals, payment processes can trigger changes in orders without explicitly depending on them - problem solved!

This principle of keeping modules decoupled should also extend to how we name signals. It's tempting to name our signal something like "complete_order", but that creates an implicit dependency between the modules because this name implies intent - the payment process should not be aware of how its signal is being used. Instead, we name signals in a way that only reflects what happened, in our case "payment completed". Each receiver can then make whatever they want from that!

Another advantage of signals is that they can have many receivers. If for example we have an "analytics" module and we want to keep track of how many payment processes succeeded or failed, we can simply register another receiver for the same signal and increment some counter.

In the next sections we are going to challenge the signals approach and demonstrate when it falls short of its promise!

Robust Django Signals

In the previous section we used signals as a way to communicate between two modules without creating an explicit dependency between them. But, did we really achieve that?

Consider what happens when a receiver encounters an error and raises an exception:

>>> o = Order.create(amount=160_00)
>>> PaymentProcess.set_status(o.payment_process_id, succeeded=True)
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
Cell In[2], line 1
----> 1 PaymentProcess.set_status(o.payment_process_id, succeeded=True)

File payment/models.py:37, in PaymentProcess.set_status(cls, id, succeeded)
File .venv/lib/python3.13/site-packages/django/dispatch/dispatcher.py:209, in Signal.send(self, sender, **named)
File order/models.py:31, in Order.on_payment_completed(payment_process_id, *args, **kwargs)
     27 @staticmethod
     28 @receiver(payment.signals.payment_process_completed, dispatch_uid='1da6190f-0cf1-45e1-8481-0d1e27bf6e6f')
     29 def on_payment_completed(payment_process_id: int, *args, **kwargs) -> Order | None:
     30     """Update the order status based on the payment process status."""
---> 31     raise Exception("on_payment_completed FAILED!!")
     32     with transaction.atomic():
     33         try:

Exception: on_payment_completed FAILED!!

Oh no! An error in the order caused the payment process to fail. We thought payment process has nothing to do with orders any more, but we were wrong! To keep modules truly decoupled we can't have exceptions from signal receivers propagate to the signal sender.

Django provides another way of sending a signal, in a way that does not propagate errors to the sender:

--- a/payment/models.py
+++ b/payment/models.py
@@ -34,7 +34,7 @@ class PaymentProcess(models.Model):
                 payment_process.status = 'failed'
             payment_process.save()

-            signals.payment_process_completed.send(
+            signals.payment_process_completed.send_robust(
                 sender=None,
                 payment_process_id=payment_process.id,
             )

The documentation for Signal.send_robust explain the difference very well:

send() differs from send_robust() in how exceptions raised by receiver functions are handled. send() does not catch any exceptions raised by receivers; it simply allows errors to propagate. Thus not all receivers may be notified of a signal in the face of an error.

send_robust() catches all errors derived from Python’s Exception class, and ensures all receivers are notified of the signal. [...]

Using send_robust() we can make sure that our modules remain decoupled even when errors happen.

So, are we finally truly decoupled?

Django Signals and Database Transactions

In the previous section we found that we weren't decoupled as we thought when the receiver raises an exception. We switched from Signal.send to Signal.send_robust which doesn't propagate errors. So now we are no longer affected by anything the receiver is doing, right? Not really!

Imagine we have another module, "analytics", to keep track of metrics in our system. To keep count of how many successful and failed payment processes we set up this simple receiver:

# analytics/handlers.py
import urllib.request
from django.dispatch import receiver

import payment.models import PaymentProcess
import payment.signals

@receiver(payment.signals.payment_process_completed, dispatch_uid='a4e3cd9c-1314-40c1-8251-955c20dd5d93')
def on_payment_process_completed(payment_process_id: int, *args, **kwargs) -> None:
    status = PaymentProcess.objects.values_list('status', flat=True).get(id=payment_process_id)
    response = urllib.request.urlopen('https://myanalytics.com/metric/inc', data={'key': 'payment_process:{status}'})
    if response.status != 200:
        raise Exception('Failed to increase metric')

The function fetches the status of the payment process and reports to some 3rd-party analytics service. We already know that if this fails the sender will not be affected. But what will happen if this request takes a very long time?

Receiver functions are called immediately by the signals framework when the signal is sent. This means where and when we send the signal is significant. This is where we send the payment_completed signal:

class PaymentProcess(models.Model):
    # ...
    @classmethod
    def set_status(cls, id: int, *, succeeded: bool) -> Self:
        with transaction.atomic():
            payment_process = cls.objects.select_for_update(of=('self', ), no_key=True).get(id=id)
            if payment_process.status not in {'initiated'}:
                raise StateError()
            if succeeded:
                payment_process.status = 'succeeded'
            else:
                payment_process.status = 'failed'
            payment_process.save()

            signals.payment_process_completed.send_robust(
                sender=None,
                payment_process_id=payment_process.id,
            )

        return payment_process

The signal is sent inside a database transaction. This can cause some problems:

Prolonged database transaction: the receiver function is making a network call to an external API. We don't have any control over the 3rd-party service and it's possible for a request to take a long time. Since receiver functions are executed immediately when the signal is sent, this can affect the overall execution time of the caller and make the transaction longer. Long running transactions can cause contention and locking issues, and increase the likelihood of failure or rollback. Ideally, we want to keep database transactions as short as possible.
Unhandled side-effects: what happens if the receiver was executed and the caller ended up rolling back? In this case, the payment process is not marked as completed, yet the call to the external API has already been made. This can cause inconsistent data between the system and the remote service.
Unintentionally affecting receiver transactions: if the receivers are using transactions of their own, they are being executed as sub-transactions of the sender transactions. If the caller rollback, they are also being rolled back. Some may consider this a feature, but in fact, this is implicit and arguably unexpected.

The most straight forward solution here is to simply send the signal outside of the transaction:

--- a/payment/models.py
@@ -15,28 +15,28 @@ class PaymentProcess(models.Model):
     @classmethod
     def set_status(cls, id: int, *, succeeded: bool) -> Self:
         with transaction.atomic():
             payment_process = cls.objects.select_for_update(of=('self', ), no_key=True).get(id=id)
             if payment_process.status not in {'initiated'}:
                 raise StateError()
             if succeeded:
                 payment_process.status = 'succeeded'
             else:
                 payment_process.status = 'failed'
             payment_process.save()

-            signals.payment_process_completed.send_robust(
-                sender=None,
-                payment_process_id=payment_process.id,
-            )
+        signals.payment_process_completed.send_robust(
+            sender=None,
+            payment_process_id=payment_process.id,
+        )

         return payment_process

Sending the signal outside of the database transaction prevents prolonged transactions and issues that can be caused by unexpected side-effects, however, the solution is still not 100% reliable!

transaction.on_commit

Django provides a nice way of executing something only after the database transaction completed successfully, without having to move the call down. Using on_commit we can trust that the signal is only being sent after the transaction was successfully committed. If the transaction rolls-back, the callable in on_commit will not be executed, and the signal will not be sent.

Fault Tolerance

To understand how reliable our approach really is, we need to evaluate what happens when it fails at different points in the process. The easiest way of thinking about it is to imagine the server crashing while your process is running.

Simulating Failures

Consider the following places where the server might crash during the execution of the function:

@classmethod
def set_status(cls, id: int, *, succeeded: bool) -> Self:
    # 💥 Before the transaction
    with transaction.atomic():
        payment_process = cls.objects.select_for_update(of=('self', ), no_key=True).get(id=id)
        if payment_process.status not in {'initiated'}:
            raise StateError()
        if succeeded:
            payment_process.status = 'succeeded'
        else:
            payment_process.status = 'failed'

        payment_process.save()
        # 💥 Inside the transaction

    # 💥 After the transaction, before the signal is sent

    signals.payment_process_completed.send_robust(
        sender=None,
        payment_process_id=payment_process.id,
    )

    return payment_process

Let's analyze what happens if the server crashes in each of these points:

✅ Before the transaction: At this point nothing really happened yet so no changes were made to any objects. Whoever called this function will have to try again, but the process is in the same state as it were - payment process is "initiated" and order is "pending payment".
✅ Inside the transaction: At this point some changes were made to the payment but the transaction is not commited yet. If the server crash the database transaction will rollback and the process is at the same state as it were before any changes were made - payment process is "initiated" and order is "pending payment".
❌ After the transaction, before the signal is sent: At this point the transaction was commited to the database so the payment is either "succeeded" or "failed". However, since we crashed before we sent the signal, the state of the order is not updated and remains "pending_payment". This is very bad! unless we somehow sync the states, the order will never complete even though the payment succeeded.

Our approach is not resilient to server crash at any point in the process so we have to consider it unreliable!

Atomicity

Database transactions provide atomicity - changes to multiple rows inside a single transaction are commited all at once or not at all. Ideally, we want the change to the payment and the following change to the order to be executed "all or nothing", otherwise, we risk leaving the process in an inconsistent state.

In our case, the change to the order is triggered by the signal which is sent outside the database transaction so we cannot guarantee atomic execution of both these changes. As a result, if the payment is updated and we crash before we sent the signal, the system will charge the user but the order will never be marked as completed! You'll end up with very angry users, and for a good reason.

Reliable Execution

We started by using Django signals to decouple modules. We then refined our implementation to minimize the impact of receivers on callers by sending signals outside the database transaction. As a result, we introduced scenarios that can leave the process in an inconsistent state.

Despite our best efforts so far, we are still left with a few significant problems:

Receivers may have significant effect on the caller execution time: receivers are executed serially by the signals framework when a signal is sent. The sender has no control over what receivers are doing, and no control over how many receives are registered to it. This makes it very hard to maintain consistent performance.
No guarantee of at-least-once delivery: a signal sender has no way of making sure all receivers will execute eventually at least once. This breaks the contract between the sender and the receiver. Imagine the server crashed while receivers were being sent - these receivers are not going to be delivered, ever! In our case, this means the order would never complete, even though the payment succeeded.
No built-in mechanism for handling failed receivers: when using Signal.send_robust, if a receiver fails to execute, the sender is not aware of that and there is no built-in process for retry. It's up to the developer to track failed receivers and put in-place a process for retrying. This makes signals unreliable for mission critical workflows.

All of these problems are not new, but they are rooted in the way Django signals work. An ideal solution should provide the following guarantees:

Receivers should be executed at-least-once if the sender committed
Receivers should not execute if the sender's transaction rolled back (due to failure or otherwise)
Receivers should have minimal effect on the sender

A lot of work and careful thought went into the Django signals framework, and the API is actually quite nice! So with that in mind, we'll try to adjust the execution mechanism for Django signals so that it's reliable and compatible as possible with the existing framework.

Django Tasks

Django 6.0 introduces a new "Tasks Framework":

Background Tasks can offload work to be run outside the request-response cycle, to be run elsewhere, potentially at a later date. This keeps requests fast, reduces latency, and improves the user experience.

Django tasks in its initial release is mostly an interface - it comes with two built-in backends, dummy and immediate, which are mostly intended for debug and development. The idea behind this approach is that developers can implement their own backends, and have seamless integration with other applications using Django tasks.

One prominent backend that has been developed in parallel with the tasks framework is the DatabaseBackend of django-tasks. The database backend maintains a queue in a database table, and provides a worker implementation to dequeue and execute tasks. It also comes with a built-in retry mechanism and a nice admin panel.

Using a database queue we can make changes to database objects and enqueue tasks atomically.

This is the idea:

We send a signal in the database transaction of the sender.
When a signal is sent, we enqueue a task in the database queue for each receiver.
The receiver tasks are not visible to workers until the sender transaction is commited.
Once the sender transaction commits, worker processes can start executing receiver tasks.
If the sender transaction rollback, the enqueued receiver tasks are also rolled-back and therefor won't execute.
If a receiver happens to fail, the worker can retry it.
Depending on the number of workers, we can execute multiple receiver tasks at the same time, minimizing delays in the overall workflow.

This approach checks all of our requirements!

Using Django Tasks

First, since we're using the new tasks framework and the database backend, we need to install Django version 6 and django-tasks:

$ uv add "Django>=6"
$ uv add django-tasks

Next, configure django-tasks and set the default backend to be the DatabaseBackend:

+++ settings.py
@@ -38,6 +38,8 @@ INSTALLED_APPS = [
     'order.apps.OrderConfig',
     'payment.apps.PaymentConfig',
+    'django_tasks',
+    'django_tasks.backends.database',
 ]

@@ -117,3 +119,10 @@ USE_TZ = True
+
+TASKS = {
+    "default": {
+        "BACKEND": "django_tasks.backends.database.DatabaseBackend",
+        "ENQUEUE_ON_COMMIT": False,
+    }
+}

Make sure you are using a PostgreSQL database backend:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'djangoreliablesignals',
        'USER': 'djangoreliablesignals',
    }
}

Run the migrations to create the queue tables:

$ ./manage.py migrate

Tasks are executed by a worker process. This means in addition to the processes that run Django itself, you also need a worker process running in the background. In another shell:

$ ./manage.py db_worker

Make sure to check the options for the worker if this ever makes it to production!

Great, on to the actual implementation...

Execute Receivers as Django Tasks

A Django signal is essentially a registry of receiver functions. When you use the receiver decorator, the wrapped function is added to a list of receivers on the signal instance. When you send the signal, the signal is iterating over the internal list of receivers and executes them.

One of the limitations of a database queue is that to enqueue a task, you must save all of the necessary information for executing it to the database. This means you need to serialize all the information to JSON - this includes the arguments as well.

In our case, to execute a receiver function we need to be able to tell the worker what function to execute. Since we can't persist a function object to the database, we need to find another way of referencing it. One way to reference a function is to generate a string with the name of the module and the fully qualified name of the function:

from typing import Callable, Any

def callable_to_qualname(f: Callable[..., Any]) -> str:
    """Return the <module>::<qualname> identifier of a function."""
    return f'{f.__module__}::{f.__qualname__}'

The function produces a string that includes the module name and the fully qualified name of the function we want to reference:

>>> callable_to_qualname(Order.on_payment_completed)
'order.models::Order.on_payment_completed'

To get the function from the string, we implement the opposite function:

import importlib

def qualname_to_callable(qualname: str) -> Callable[..., Any]:
    """Get a callable from its <module>::<qualname> identifier."""
    module_name, func_qualname = qualname.split('::', 1)
    module = importlib.import_module(module_name)

    # Handle nested attributes (e.g., 'ClassName.method_name')
    obj = module
    for attr in func_qualname.split('.'):
        obj = getattr(obj, attr)

    return obj  # type: ignore[return-value]

Given the fully qualified name we generated, the function returns the callable:

>>> receiver = qualname_to_callable('order.models::Order.on_payment_completed')
>>> receiver
<function order.models.Order.on_payment_completed(payment_process_id: 'int', *args, **kwargs) -> 'Order | None'>

Now that we are able to persist a reference to our receiver function, we can create a task to execute an arbitrary receiver:

from collections.abc import Mapping
from django_tasks import task

@task()
def execute_task_signal_receiver(
    *,
    receiver_qualname: str,
    named: Mapping[str, object],
) -> None:
    receiver = qualname_to_callable(receiver_qualname)
    receiver(signal=None, sender=None, **named)

This registers a new django-tasks task that accepts a receiver qualified name and arguments, and executes it. Simple as that!

To change the way signals are sent, we provide an alternative implementation of a Django Signal that instead of executing receivers immediately, enqueues a task for each one:

# reliable_signal/__init__.py
from django.dispatch import Signal as DjangoSignal
from django.dispatch.dispatcher import NO_RECEIVERS

class Signal(DjangoSignal):
    """A django-workers-capable signal."""

    def send_reliable(self, sender: None, **named) -> None:
        """Like send_robust(), but enqueues a task for each registered receiver."""
        if not self.receivers:
            return
        if self.sender_receivers_cache.get(sender) is NO_RECEIVERS:
            return
        sync_receivers, async_receivers = self._live_receivers(sender)
        assert not async_receivers, 'Async receivers not supported by task'
        for receiver in sync_receivers:
            execute_task_signal_receiver.enqueue(
                receiver_qualname=callable_to_qualname(receiver),
                named=named,
            )

Our reliable signal is extending Django's built-in Signal class and adds a function called send_reliable. The function works like send_robust, but instead of executing the receiver functions immediately, it enqueues a task for each receiver instead. We discuss this approach further later on.

Finally, to adjust our code to use the reliable signal, all we need to do is to use our new reliable signal, and replace send_robust with send_reliable:

diff --git i/payment/signals.py w/payment/signals.py
@@ -1,3 +1,3 @@
-from django.dispatch import Signal
+from reliable_signal import Signal

 payment_process_completed = Signal()

diff --git i/payment/models.py w/payment/models.py
@@ -34,9 +34,9 @@ class PaymentProcess(models.Model):
                 payment_process.status = 'failed'
             payment_process.save()

-        signals.payment_process_completed.send_robust(
-            sender=None,
-            payment_process_id=payment_process.id,
-        )
+            signals.payment_process_completed.send_reliable(
+                sender=None,
+                payment_process_id=payment_process.id,
+            )

         return payment_process

Notice that we enqueue the task inside the database transaction. Previously, we said this might cause some issues, but using a database queue, you actually do want to enqueue the task inside the sender transaction. This way, the task will be executed only after the sender commits. If the sender rollback, the task will not be enqueued and will not be executed.

Now we are ready to test this out. Make sure you have a worker running and execute this in Django shell:

>>> o = Order.create(amount=170_00)
>>> PaymentProcess.set_status(o.payment_process_id, succeeded=True)
<PaymentProcess: PaymentProcess object (7)>
>>> o.refresh_from_db()
>>> o.status
'completed'

Amazing! Quickly after we set the status for payment process, the worker picked up the task and updated the status of the order. You can see it in the worker logs as well:

$ ./manage.py db_worker
Watching for file changes with StatReloader
Starting worker worker_id=4tLA6TEzAdIZ7W620DrVnHuC342wiDfs queues=default
Task id=c34fa024-db4d-41b4-b875-723b2436a346 path=reliable_signal.execute_task_signal_receiver_simple state=RUNNING
Task id=c34fa024-db4d-41b4-b875-723b2436a346 path=reliable_signal.execute_task_signal_receiver_simple state=SUCCEEDED

We now have a reliable execution engine for Django signals:

Receivers are enqueued inside the sender database transaction
Failed receivers can be retried by the tasks framework
Enqueuing receivers is quick, with minimal impact on sender function

Testing Reliable Django Signals

When we test workflows we usually don't care much about the execution engine, but rather with the business logic. We also want to keep our test suite simple and deterministic as possible - this means we don't want to execute receivers in another worker, we want to execute them immediately.

If you recall, we mentioned that Django comes with two built-in backends for testing and development. One of them is the ImmediateBackend. This backend will execute tasks immediately when they are enqueued - exactly what we need in tests.

from django.test import TestCase, override_settings
from order.models import Order
from payment.models import PaymentProcess

@override_settings(TASKS={'default': {'BACKEND': 'django.tasks.backends.immediate.ImmediateBackend'}})
class OrderTestCase(TestCase):
    def test_order_happy_path(self):
        order = Order.create(amount=100_00)
        self.assertEqual(order.status, 'pending_payment')
        self.assertEqual(order.amount, 100_00)
        self.assertEqual(order.payment_process.status, 'initiated')
        self.assertEqual(order.payment_process.amount, 100_00)

        # This will trigger the signal, which should execute the receiver immediately
        PaymentProcess.set_status(order.payment_process_id, succeeded=True)
        order.refresh_from_db()
        self.assertEqual(order.status, 'completed')
        self.assertEqual(order.payment_process.status, 'succeeded')

This tests the common "happy path" for an order - create an order, payment is successful, order status is updated. To provide an alternative backend for tasks during the test, we use the @override_settings with the path to the built-in ImmediateBackend.

Running the test:

$ ./manage.py test
Found 1 test(s).
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
.
----------------------------------------------------------------------
Ran 1 test in 0.058s

OK
Destroying test database for alias 'default'...

Great! We now have reliable execution that we can easily test.

Reliable Signals Limitations

Reliable signals provide great benefits, but they also come with some restrictions and limitations:

Receiver arguments must be serializable: Django signals that execute immediately can accept any python object, such as Django model instances, and pass them immediately to the receiver function. When using a background task, the arguments are written to the database first, so they must be serializable. This means that instead of sending a Django model instance for example, you need to pass the pk instead. This limitation is not unique to Django tasks or even database queues in general. Any task engine needs to serialize arguments in some way.
Partial compatibility with existing Django signals: When executing a receiver function, the django signal framework providers two additional arguments - sender - usually the model that sent the signal, and signal - the signal that executes the receiver. These arguments are usually useful when you have a single receiver that handles many different signals. Common use cases include analytics and audit. In these use-cases, it's useful to know which signal was triggered (the action) and on what object (the sender). In our simple implementation we do not provide sender and signal because they are objects, and we can't easily serialize them. A more complex implementation can provide them.
Not a good fit for all type of signals: In this article we discussed custom signals that we use for our own workflows. However, Django has its own built-in signals. For example, pre- and post-save signals on models. Receivers of these signals often rely on the sender and model arguments, which we are not sending (see previous note). Also, some signals are actually intended to be executed inside the process that ran them, for example management and app registry signals.

Due to these limitations, we think it is crucial that reliable signals co-exist with the built-in signal system:

Reliable signals don't fit all types of signals so we don't attempt to monkey-patch Django. Instead, reliable signals should be explicitly defined using reliable_signal.Signal.
Due to the different behavior, offloading execution to a background task should be explicitly invoked using send_reliable rather than attempting to patch send_robust or send.

Future Work

The current implementation is mostly offered as a reference. While operational under the restrictions mentioned above, there are a few bits we did not address:

Support async tasks: both Django signals and django-tasks support async, but we haven't in our implementation. It should be fairly easy to add.
Utilize tasks features: the tasks framework defines additional capabilities such as priority, custom queues and retries. For additional reliability and observability, it might be useful to add dedicated queues for signal receivers and potentially set a higher priority for mission critical signals so they'll execute sooner.
Increase compatibility with Django signals: to provide full compatibility receivers should accept the arguments sender and signal. They are rarely needed in our custom workflow signals and they are harder to implement, so we didn't bother, but it's possible! Signals can reference Django models using their content type ID or qualified name. Identifying the sending signal is bit harder because there is no unique identifier we can reference - to do that you'll have to add a unique name for each signal and manage some registry. We've done it, it just wasn't that useful.
Integrate into Django itself: Since we're stitching together two built-in components of Django itself - tasks and signals - an additional Signal.send_reliable / Signal.send_durable / Signal.send_task might be a good fit for Django itself!

Final Thoughts

We have a lot of custom workflow in our systems - this is really most of what we do! We are using Django signals extensively in situations where two decoupled modules needs to communicate with each other. As our system grew, we experienced first-hand the issues that can come from having un-reliable signals. Eventually, we developed our own database task queue and integrated it with Django signals. So far its been working pretty well with moderate traffic.

This article was motivated by our pains and learning from implementing reliable signals in our systems. The release of the the Django tasks framework (and the backends) is surely a welcome addition to an increasing number of large systems built with Django, that needs to have reliable and durable workflows.

How to Get Foreign Keys Horribly Wrong

2025-07-15T00:00:00+03:00

Constraints keep the integrity of your system and prevent you from shooting yourself in the foot. Foreign keys are a special type of constraint because, unlike unique, check, and primary keys, they span more than one relation. This makes foreign keys harder to enforce and harder to get right.

In this article, I demonstrate common pitfalls, potential optimizations, and implicit behavior related to foreign keys.

Table of Contents

Naive Implementation
Enhanced Implementation
Takeaways
- The Mandatory AI Angle

Watch

📺 This article is inspired by a talk I gave at DjangoCon EU. Watch it here.

Naive Implementation

Imagine a simple application to manage a product catalog:

Generated using dbdiagram.io

The first table, or model, in the catalog is the Category model:

class Category(models.Model):
    id: int = models.BigAutoField(primary_key=True)
    name: str = models.CharField(max_length=50)

Categories can be "household items", "fruit", "apparel", and so on.

Next, a model to store products:

class Product(models.Model):
    class Meta:
        unique_together = (
            ('category', 'category_sort_order'),
        )

    id = models.BigAutoField(primary_key=True)
    name = models.CharField(max_length=50)
    description = models.TextField()

    category = models.ForeignKey(to=Category, on_delete=models.PROTECT, related_name='products')
    category_sort_order = models.IntegerField()

    created_by = models.ForeignKey(to=User, on_delete=models.PROTECT, related_name='+')
    last_edited_by = models.ForeignKey(to=User, on_delete=models.PROTECT, related_name='+', null=True)

Products have a name and description, and they are associated with a category by foreign key.

To make sure we present products in the right order in the UI, we add a sort order within the category. To make sure the order is deterministic, we add a unique constraint to prevent products in the same category from having the same sort order.

Finally, we added two columns to keep track of the user who created the product and the user who last edited the product. This is mostly for auditing purposes.

Since we are on the subject of foreign keys, here are the 3 foreign keys we have in this very simple product table:

category: foreign key to the catalog model we created.
created_by: foreign key to the user model.
last_edited_by: foreign key to the user model. Can be null.

This naive implementation, as the name suggests, is very naive! There is a lot more than meets the eye. In the next sections, we'll make adjustments to improve this model.

Enhanced Implementation

It's been said that developers spend more time reading code than writing code. As someone who has spent a fair amount of time giving code reviews and going through code to understand how it works, I believe this to be true. We started with a naive implementation and now we'll review the code and make it better.

Replacing `unique_together`

To control the order we present products in the UI we added a category_sort_order to each product. To make sure two products in the same category don't have the same value, we added a unique constraint on the combination of category and category_sort_order:

class Product(models.Model):
    class Meta:
        unique_together = (
            ('category', 'category_sort_order'),
        )

The Django documentation on unique_together includes a special note we should consider:

Use UniqueConstraint with the constraints option instead. UniqueConstraint provides more functionality than unique_together. unique_together may be deprecated in the future.

Unique together is deprecated and discouraged, so let's replace it with a unique constraint:

@@ -13,9 +13,15 @@ class Category(models.Model):

 class Product(models.Model):
     class Meta:
-        unique_together = (
-            ('category', 'category_sort_order', ),
-        )
+        constraints = [
+            models.UniqueConstraint(
+                name='product_category_sort_order_uk',
+                fields=(
+                    'category',
+                    'category_sort_order',
+                ),
+            ),
+        ]

Unique constraint is easier to modify, allows to use advanced B-Tree index features (we'll use some later) and is recommended by the documentation, so let's always use that!

💡 Takeaway

Don't use unique_together, use UniqueConstraint instead.

Identifying Duplicate Indexes

Now that we got rid of unique_together, let's have a look at the schema:

catalog=# \d catalog_product
       Column        │         Type
─────────────────────┼───────────────────────
 id                  │ bigint
 name                │ character varying(50)
 category_sort_order │ integer
 category_id         │ bigint
 created_by_id       │ integer
 last_edited_by_id   │ integer
Indexes:
    "catalog_product_pkey" PRIMARY KEY, btree (id)
    "catalog_product_category_id_35bf920b" btree (category_id)
    "catalog_product_category_id_category_sort_order_b8206596_uniq" UNIQUE CONSTRAINT, btree (category_id, category_sort_order)
    "catalog_product_created_by_id_4e458b98" btree (created_by_id)
    "catalog_product_last_edited_by_id_05484fb6" btree (last_edited_by_id)
    -- ...

It's easy to get lost in all the information here, but notice that we have two indexes that are prefixed by category.

The first index is the unique constraint we just created:

class Product(models.Model):
    class Meta:
        constraints = [
            models.UniqueConstraint(
                name='product_category_sort_order_uk',
                fields=('category', 'category_sort_order'),
            ),
        ]

The second index is not that obvious:

class Product(models.Model):
    # ...
    category = models.ForeignKey(
        to=Category,
        on_delete=models.PROTECT,
        related_name='products',
    )

Where is the index, you ask? The answer lies in the official documentation of the ForeignKey field:

A database index is automatically created on the ForeignKey.

When you define a foreign key, Django implicitly creates an index behind the scenes. In most cases, this is a good idea, but in this case, this field is already (sufficiently) indexed. Reading further in the documentation:

A database index is automatically created on the ForeignKey. You can disable this by setting db_index to False

If we believe that we don't need the index, we can instruct Django not to create it by setting db_index to False:

@@ -35,6 +35,8 @@ class Product(models.Model):
     category = models.ForeignKey(
         to=Category,
         on_delete=models.PROTECT,
         related_name='products',
+        # Indexed in unique constraint.
+        db_index=False,
     )

Queries using category can use the unique index instead.

⚠️ Implicit Behavior

ForeignKey field implicitly creates an index unless explicitly setting db_index=False.

To actually remove the index, we first need to generate a migration:

$ ./manage.py makemigrations
Migrations for 'catalog':
  demo/catalog/migrations/0003_alter_product_category.py
    ~ Alter field category on product

But before we move on to apply this migration, there is another small thing we need to take care of.

Identifying Blocking Migrations

Let's have a look at the migration we just generated to remove the index from the foreign key:

class Migration(migrations.Migration):
    dependencies = [
        ('catalog', '0002_alter_product_unique_together_and_more'),
    ]

    operations = [
        migrations.AlterField(
            model_name='product',
            name='category',
            field=models.ForeignKey(
                db_index=False,
                on_delete=django.db.models.deletion.PROTECT,
                related_name='products',
                to='catalog.category',
            ),
        ),
    ]

The migration looks harmless. Let's dig deeper and review the actual SQL generated by the migration:

$ ./manage.py sqlmigrate catalog 0003
BEGIN;
--
-- Alter field category on product
--
SET CONSTRAINTS "catalog_product_category_id_35bf920b_fk_catalog_category_id" IMMEDIATE;
ALTER TABLE "catalog_product" DROP CONSTRAINT "catalog_product_category_id_35bf920b_fk_catalog_category_id";
ALTER TABLE "catalog_product" ADD CONSTRAINT "catalog_product_category_id_35bf920b_fk_catalog_category_id"
    FOREIGN KEY ("category_id") REFERENCES "catalog_category" ("id") DEFERRABLE INITIALLY DEFERRED;
COMMIT;

For the inexperienced eye, this might look OK, but if you really pay attention you'll notice that something very dangerous is going on here.

💡 Takeaway

Always check the SQL generated by migrations.

When we set db_index=False, what we wanted to do is to drop the index but keep the constraint. Unfortunately, Django is unable to detect this nuanced change, so instead, Django is re-creating the entire foreign key constraint without the index!

Re-creating the constraint works in two steps:

1. Dropping the existing constraint:

ALTER TABLE "catalog_product" DROP CONSTRAINT "catalog_product_category_id_35bf920b_fk_catalog_category_id";

The database will drop the index on the field and also stop validating the constraint. Dropping the index requires a lock on the table while the index is dropped. This can block some operations on the table.

2. Creating the constraint:

ALTER TABLE "catalog_product" ADD CONSTRAINT "catalog_product_category_id_35bf920b_fk_catalog_category_id"
    FOREIGN KEY ("category_id") REFERENCES "catalog_category" ("id") DEFERRABLE INITIALLY DEFERRED;

The database first obtains a lock on the table. Then, to make sure the constraint is valid, it needs to make sure all the values in the column have a matching field in the referenced table. In this case, it checks that the product's category exists in the category table. Depending on the size of the tables, this can take some time and interfere with ongoing operations against the table.

⚠️ Implicit Behavior

When making changes to ForeignKey Django may implicitly re-create the constraint. This can require an extended locks and block certain operations on the table.

All of this is really not necessary though. We don't have anything against the constraint, what we really want is to keep the constraint as-is and only drop the index.

Safely Migrating Foreign Key

There are some types of changes to ForeignKey fields Django is (currently?) unable to detect. As a result, Django may end up re-creating the constraint in the database. This can lead to extended locking and may impact live systems. To avoid that, we need to change the way Django applies the migration. To understand how we can do that, we need to understand how Django generates migrations in the first place:

"Django migrations"

Generate migrations using makemigrations:

Render a state from the existing migrations
Compare to the desired state of the models in models.py
Generate migration operations from the difference

Applying migrations using migrate:

Generate SQL from the migration operations (sqlmigrate)
Apply to SQL to the database

Let's break down the case of changing a foreign key:

Django identify that a ForeignKey has changed: db_index changed from True -> False
Django generates an operation to "sync" the new state: a migrations.AlterField operation with the new definition of the ForeignKey field.
Generate SQL from the migration operation: To "sync" the foreign key Django drops the constraint and re-creates it.

Django identified the change to the field, but it's unable to generate a migration operation to just drop the index.

If the migration and the SQL generated by Django is not exactly what we want, there is a special operation called SeparateDatabaseAndState we can use:

A highly specialized operation that lets you mix and match the database (schema-changing) and state (autodetector-powering) aspects of operations.

Using SeparateDatabaseAndState we can provide one set of operations to execute against Django's internal state, and another set of operations to execute against the database:

@@ -11,9 +11,18 @@ class Migration(migrations.Migration):
     ]

     operations = [
-        migrations.AlterField(
-            model_name='product',
-            name='category',
-            field=models.ForeignKey(db_index=False, on_delete=django.db.models.deletion.PROTECT, related_name='products', to='catalog.category'),
-        ),
+        migrations.operations.SeparateDatabaseAndState(
+            state_operations=[
+                migrations.AlterField(
+                    model_name='product',
+                    name='category',
+                    field=models.ForeignKey(db_index=False, on_delete=django.db.models.deletion.PROTECT, related_name='products', to='catalog.category'),
+                ),
+            ],
+            database_operations=[
+                migrations.RunSQL(
+                    'DROP INDEX catalog_product_category_id_35bf920b',
+                ),
+            ],
+        )
     ]

The SeparateDatabaseAndState migration operation accepts two arguments:

state_operations: operations to execute against the internal state when aggregating the changes from the migrations. In most cases, this will be the operations Django generated automatically from makemigrations.
database_operations: operations to execute in the database. This is where we control exactly what to execute in the database when the migration is applied using migrate. In our case, we use RunSQL to execute a DROP INDEX command.

To demonstrate, this is what will be executed when we apply this migration:

$ ./manage.py sqlmigrate catalog 0003
BEGIN;
--
-- Custom state/database change combination
--
DROP INDEX catalog_product_category_id_35bf920b;
COMMIT;

Exactly what we want!

Applying the migration:

$ ./manage.py migrate
Operations to perform:
  Apply all migrations: catalog
Running migrations:
  Applying catalog.0003_alter_product_category... OK

So are we done...?

Reversible Migration Operations

Say you applied the migration to drop the index and it went OK. A few minutes go by and then you realize you made a horrible mistake. You rush back to your laptop to reverse the migration:

$ ./manage.py migrate catalog 0002
Rendering model states... DONE
Unapplying catalog.0003_alter_product_category...
django.db.migrations.exceptions.IrreversibleError:
Operation <RunSQL ''> in catalog.0003_alter_product_category is not reversible

Oh no! You now have some explaining to do...

The reason you can't un-apply this migration is that the RunSQL command to drop the index did not include an opposite operation. This is an easy fix:

index 962c756..7ad45e0 100644
--- a/demo/catalog/migrations/0003_alter_product_category.py
+++ b/demo/catalog/migrations/0003_alter_product_category.py
@@ -22,6 +22,7 @@ class Migration(migrations.Migration):
             database_operations=[
                 migrations.RunSQL(
                     'DROP INDEX catalog_product_category_id_35bf920b',
+                    'CREATE INDEX "catalog_product_category_id_35bf920b" ON "catalog_product" ("category_id")',
                 ),
             ],
         )

The second argument to RunSQL is the reverse operation - what to execute to undo the migration. In this case, to reverse dropping an index is to create an index!

But, where do you get this SQL from? Usually from the migration that added it. In this case, the initial migration:

$ ./manage.py sqlmigrate catalog 0001
...
BEGIN;
...
--
-- Create model Product
--
...
CREATE INDEX "catalog_product_category_id_35bf920b" ON "catalog_product" ("category_id");
...

Now if you found out you made a horrible mistake and you want to reverse the migration:

$ ./manage.py migrate catalog 0002
Rendering model states... DONE
Unapplying catalog.0003_alter_product_category... OK

Great!

💡 Takeaway

Provide reverse operations whenever possible - you don't know when you're going to need it.

Concurrent Index Operations

You now made sure to only drop the index without recreating the constraint and you provided a reverse operation in case you made a mistake and want to "undo". That's all great, but just one more thing... from PostgreSQL documentation on DROP INDEX:

A normal DROP INDEX acquires an ACCESS EXCLUSIVE lock on the table, blocking other accesses until the index drop can be completed.

To drop the index, PostgreSQL acquires a lock on the table which blocks other operations. If the index is tiny, that's probably fine, but what if it's a very big index? Dropping a big index can take some time and we can't lock a live table for very long.

PostgreSQL provides an option to create an index without acquiring restrictive locks on the index:

@@ -5,6 +5,7 @@ from django.db import migrations, models
 class Migration(migrations.Migration):
+    atomic = False

     dependencies = [
         ('catalog', '0002_alter_product_unique_together_and_more'),
@@ -21,7 +22,7 @@ class Migration(migrations.Migration):
             ],
             database_operations=[
                 migrations.RunSQL(
-                    'DROP INDEX catalog_product_category_id_35bf920b',
+                    'DROP INDEX CONCURRENTLY catalog_product_category_id_35bf920b',
-                    'CREATE INDEX "catalog_product_category_id_35bf920b" ON "catalog_product" ("category_id")',
+                    'CREATE INDEX CONCURRENTLY "catalog_product_category_id_35bf920b" ON "catalog_product" ("category_id")',
                 ),
             ],

Dropping an index concurrently works in two phases. First the index is marked as "deleted" in the dictionary table. During this time, ongoing transactions can still use it. Next, the index is actually dropped. This way of dropping indexes requires minimal locking but can take a bit more time.

💡 Takeaway

Use concurrent index operations in busy systems. Concurrent operations can take a bit more time but they don't block operations to the table while they execute.

Databases such as PostgreSQL that support transactional DDL, can execute CREATE, DROP and ALTER commands inside a database transaction. This is a very useful feature because it allows you to execute schema changes atomically (it also allows you to do some wild things like making indexes "invisible"). Unfortunately, concurrent operations cannot be executed inside a database transaction. This means we need to set the entire migration to be non-atomic by setting atomic=False.

In an atomic migration, if something fails in the middle, the entire transaction is rolled-back and it's like the migration never ran. In a non-atomic migration however, if something fails in the middle, you can end up with an incomplete execution and an inconsistent state. To reduce the risk of getting stuck with a half-applied migration, if you have operations such as drop/create index concurrently in the migration, it's best to split the migration and move these operations to a separate migration.

💡 Takeaway

To avoid incomplete migrations, move operations that can't be executed atomically to a separate migration.

This is the final migration:

class Migration(migrations.Migration):
    atomic = False

    dependencies = [
        ('catalog', '0002_alter_product_unique_together_and_more'),
    ]

    operations = [
        migrations.operations.SeparateDatabaseAndState(
            state_operations=[
                migrations.AlterField(
                    model_name='product',
                    name='category',
                    field=models.ForeignKey(db_index=False, on_delete=django.db.models.deletion.PROTECT, related_name='products', to='catalog.category'),
                ),
            ],
            database_operations=[
                migrations.RunSQL(
                    'DROP INDEX CONCURRENTLY catalog_product_category_id_35bf920b',
                    'CREATE INDEX CONCURRENTLY "catalog_product_category_id_35bf920b" ON "catalog_product" ("category_id")',
                ),
            ],
        )
    ]

In this migration we prevented Django from re-creating the entire foreign key and instead only drop the index. This migration also uses concurrent index operations so it's safe to execute on a live system, and if you make a mistake it is also reversible.

Indexes on Foreign Keys

So far we handled the foreign key on category, so let's move on to the next foreign key in the model:

class Product(models.Model):
    #...
    created_by = models.ForeignKey(
        to=User,
        on_delete=models.PROTECT,
        related_name='+',
    )

Django will implicitly add an index on a ForeignKey field unless explicitly stated otherwise. Since we didn't define db_index=False on this field, Django created an index on created_by. But, do we really need it?

To answer this question, we need first to ask how this field is being used:

Used primarily for audit purposes
Products are rarely queried by the user who created them

From these two use-cases, it seems like no one is actually going to query the table by created_by, so this index is most likely unnecessary. However, there is another use for this index, which is not as obvious.

To demonstrate, let's create a user:

>>> haki = User.objects.create_user(
...    username='haki',
...    first_name='haki',
...    last_name='benita',
... )
<User: haki>

Now turn on SQL logging on and delete the user we just created a second ago:

>>> haki.delete()
(0.438) SELECT * FROM "catalog_product" WHERE "catalog_product"."created_by_id" IN (102); args=(102,)
(0.002) SELECT * FROM "catalog_product" WHERE "catalog_product"."last_edited_by_id" IN (102); args=(102,)
(0.000) BEGIN;
(0.002) DELETE FROM "django_admin_log" WHERE "django_admin_log"."user_id" IN (102); args=(102,)
(0.001) DELETE FROM "auth_user_groups" WHERE "auth_user_groups"."user_id" IN (102); args=(102,)
(0.001) DELETE FROM "auth_user_user_permissions" WHERE "auth_user_user_permissions"."user_id" IN (102); args=(102,)
(0.001) DELETE FROM "auth_user" WHERE "auth_user"."id" IN (102); args=(102,)
(0.368) COMMIT;
(1, {'auth.User': 1})

A lot of things are happening here! Let's break it down:

Django checks if there are products that were created by or last edited by this user.
Django deletes any admin logs, group memberships and permissions associated with this user.
Django actually deletes the user from the user tables.
Django commits the transaction and then the database does all of these checks too!

This brings us to the next, less obvious way, indexes on foreign keys are being used - to validate the foreign key constraint. If the foreign key is defined with on_delete=PROTECT, the index is used to make sure there are no related objects referencing a specific object. In our case, products that reference the user we are about to delete. If the foreign key is defined with on_delete=CASCADE, the index is used to delete the related objects. In our case, deleting the user may also delete products referencing the user.

You may have noticed that Django delete() function returns a counter-like structure that keeps how many objects were deleted for each type of model. This is why despite the fact the database also does all of these checks, Django is also doing them. The indexes on the foreign key are working extra-hard here.

💡 Takeaway

Indexes on foreign keys are used indirectly when deleting related objects. Removing these indexes may cause unexpected and hard to debug performance issues with deletes.

Now that we know this index is in-fact necessary, we explicitly set db_index on the field and add an appropriate comment so the next developer understands why we decided to keep it:

@@ -44,7 +44,8 @@
 class Product(models.Model):
     created_by = models.ForeignKey(
         to=User,
         on_delete=models.PROTECT,
         related_name='+',
+        # Used to speed up user deletion.
+        db_index=True,
     )

When someone else (or you in a couple of months) encounters this comment, they won't have to go through the entire process again.

💡 Takeaway

Always explicitly set db_index on ForeignKey and add a comment on how it's being used.

Partial Foreign Key Indexes

So far we covered two of three foreign keys - category and created_by. Here is the last one:

class Product(models.Model):
    [...]
    last_edited_by = models.ForeignKey(
        to=User,
        on_delete=models.PROTECT,
        related_name='+',
        null=True,
    )

Just like the index on created_by, this index is used mostly for audit purposes. Nobody wants to query for products last edited by some user. However, we've been down this road before and we know this index is used when users are deleted, so we'll keep it. But, before we call it a day, there is something we can still do with this index.

To demonstrate we first need to add some data.

Create 100 users:

from django.contrib.auth.models import User

users = [
    User.objects.create_user(
        username=f'user{i}',
        email=f'user{i}@email.com',
        first_name=f'User {i}',
    ) for i in range(100)
]

Create 50 categories:

from catalog.models import Category

categories = [
    Category.objects.create(
        name=f'Category {i}',
    ) for i in range(50)
]

Create 1,000,000 products:

import random
from django.utils import lorem_ipsum
from catalog.models import Product

random.seed(8080)

Product.objects.bulk_create((
    Product(
        name=f'Product {i}',
        description=lorem_ipsum.words(100),
        category=random.choice(categories),
        category_sort_order=i,
        created_by=random.choice(users),
        last_edited_by=random.choice(users) if i % 1000 == 0 else None,
    ) for i in range(1_000_000)),
    batch_size=100_000,
)

Notice that only one in every 1,000 products has been edited.

Now that we have some data, let's have a look at the indexes:

catalog=# \di+ *product*
 Schema │                  Name                        │  Size
────────┼──────────────────────────────────────────────┼─────────
 public │ catalog_product_created_by_id_4e458b98       │ 6440 kB
 public │ catalog_product_last_edited_by_id_05484fb6   │ 6320 kB
 public │ catalog_product_pkey                         │ 21 MB
 public │ product_category_sort_order_uk               │ 32 MB

There is one strange thing going on. If you don't spot this right away that's fine, most people don't.

Consider this query to check how many users we have in created_by and last_edited_by:

catalog=# SELECT
    COUNT(created_by_id) AS created_by,
    COUNT(last_edited_by_id) AS last_edited_by
FROM catalog_product;

 created_by │ last_edited_by
────────────┼────────────────
    1000000 │           1000

Out of 1M rows, all products have a value in created_by, but only 1,000 rows have a value for last_edited_by - that's ~99.9% empty values! If that's the case, how come indexes on both these fields are the same size:

catalog=# \di+ *product*
 Schema │                  Name                        │  Size
────────┼──────────────────────────────────────────────┼─────────
 public │ catalog_product_created_by_id_4e458b98       │ 6440 kB
 public │ catalog_product_last_edited_by_id_05484fb6   │ 6320 kB
 public │ catalog_product_pkey                         │ 21 MB
 public │ product_category_sort_order_uk               │ 32 MB

Both indexes are ~6MB, but one has 1M values and the other only 1K values. The reason both indexes are the same size is that in PostgreSQL, null values are indexed!

💡 Takeaway

Null values are indexed (In all major databases except Oracle).

I started my career as an Oracle DBA, where null values are not indexed. It took me some time (and a lot of expensive storage) until I realized that in PostgreSQL null values are indexed.

We have a foreign key column which is mostly used to validate the constraint and it is 99.9% empty. What if we could only index the rows which are not null? From PostgreSQL documentation on "partial index":

A partial index is an index built over a subset of a table; the subset is defined by a conditional expression [...]. The index contains entries only for those table rows that satisfy the predicate.

Exactly what we need! Let's replace the index on the ForeignKey with a partial B-Tree index:

@@ -22,6 +22,13 @@ class Product(models.Model):
    class Meta:
+        indexes = (
+            models.Index(
+                name='product_last_edited_by_part_ix',
+                fields=('last_edited_by',),
+                condition=models.Q(last_edited_by__isnull=False),
+            ),
+        )
@@ -53,5 +60,7 @@ class Product(models.Model):
     last_edited_by: User | None = models.ForeignKey(
         on_delete=models.PROTECT,
         related_name='+',
         null=True,
+        # Indexed in Meta.
+        db_index=False,
     )

We start by setting db_index=False to instruct Django we don't want the default index. We also make sure to add a comment saying that the field is indexed in Meta.

Next, we add a new index in Meta. What makes this index partial is the condition we added in the index definition:

models.Index(
    name='product_last_edited_by_part_ix',
    fields=('last_edited_by',),
    condition=models.Q(last_edited_by__isnull=False),
)

This will make the index include only values which are not null. If we generate and apply the migration, these are the sizes of the indexes:

catalog=# \di+ *product*
 Schema │                  Name                  │  Size
────────┼────────────────────────────────────────┼─────────
 public │ catalog_product_created_by_id_4e458b98 │ 6440 kB
 public │ product_last_edited_by_part_ix         │ 32 kB
 public │ catalog_product_pkey                   │ 21 MB
 public │ product_category_sort_order_uk         │ 32 MB

The full index is ~6.4MB, the partial index only 32 kB. That's ~99.5% smaller.

💡 Takeaway

Use partial indexes when you don't need to index all the values in a column. Nullable foreign key columns are usually great candidates. If you are not convinced check out the story about The Unexpected Find That Freed 20GB of Unused Index Space.

Using Built-in Concurrent Index Operations

In the previous section we managed to save some $$$ by switching to a partial index. But with all the excitement we forgot something very very important - always check the generated SQL before applying migrations!

This is the migration:

class Migration(migrations.Migration):
    dependencies = [
        ('catalog', '0005_alter_product_created_by'),
        migrations.swappable_dependency(settings.AUTH_USER_MODEL),
    ]

    operations = [
        migrations.AlterField(
            model_name='product',
            name='last_edited_by',
            field=models.ForeignKey(
                db_index=False,
                null=True,
                on_delete=django.db.models.deletion.PROTECT,
                related_name='+',
                to=settings.AUTH_USER_MODEL
            ),
        ),
        migrations.AddIndex(
            model_name='product',
            index=models.Index(
                condition=models.Q(('last_edited_by__isnull', False)),
                fields=['last_edited_by'],
                name='product_last_edited_by_part_ix',
            ),
        ),
    ]

Looks about right. The SQL:

$ ./manage.py sqlmigrate catalog 0006
BEGIN;
--
-- Alter field last_edited_by on product
--
SET CONSTRAINTS "catalog_product_last_edited_by_id_05484fb6_fk_auth_user_id" IMMEDIATE;
ALTER TABLE "catalog_product" DROP CONSTRAINT "catalog_product_last_edited_by_id_05484fb6_fk_auth_user_id";
ALTER TABLE "catalog_product" ADD CONSTRAINT "catalog_product_last_edited_by_id_05484fb6_fk_auth_user_id"
    FOREIGN KEY ("last_edited_by_id") REFERENCES "auth_user" ("id") DEFERRABLE INITIALLY DEFERRED;
--
-- Create index product_last_edited_by_part_ix on field(s) last_edited_by of model product
--
DROP INDEX IF EXISTS "product_last_edited_by_part_ix";
CREATE INDEX "product_last_edited_by_part_ix" ON "catalog_product" ("last_edited_by_id")
    WHERE "last_edited_by_id" IS NOT NULL;
COMMIT;

Oh no! It's recreating the foreign key from scratch again. We already know what to do - use SeparateDatabaseAndState to only drop the index:

@@ -13,10 +13,20 @@ class Migration(migrations.Migration):
operations = [
-     migrations.AlterField(
-         model_name='product',
-         name='last_edited_by',
-         field=models.ForeignKey(db_index=False, null=True, on_delete=django.db.models.deletion.PROTECT, # ...
+     migrations.operations.SeparateDatabaseAndState(
+         state_operations=[
+             migrations.AlterField(
+                 model_name='product',
+                 name='last_edited_by',
+                 field=models.ForeignKey(db_index=False, null=True, on_delete=django.db.models.deletion.PROTECT, # ...
+             ),
+         ],
+         database_operations=[
+             migrations.RunSQL(
+                 'DROP INDEX CONCURRENTLY catalog_product_last_edited_by_id_05484fb6;',
+                 'CREATE INDEX CONCURRENTLY catalog_product_last_edited_by_id_05484fb6 ON public.catalog_product USING btree (last_edited_by_id)',
+             ),
+         ],

The migration will now drop the index instead of re-creating the constraint.

Next, we want to create the partial index concurrently to not interrupt any live system. So far we used RunSQL for concurrent operations, but this time, we can use one of the built-in Django concurrent operations instead:

@@ -3,9 +3,11 @@
+from django.contrib.postgres.operations import AddIndexConcurrently

 class Migration(migrations.Migration):
+    atomic = False

     dependencies = [
         ('catalog', '0005_alter_product_created_by'),
@@ -28,7 +30,7 @@ class Migration(migrations.Migration):
                 ),
             ],
         ),
-        migrations.AddIndex(
+        AddIndexConcurrently(
             model_name='product',
             index=models.Index(
                condition=models.Q(('last_edited_by__isnull', False)),
                fields=['last_edited_by'],
                name='product_last_edited_by_part_ix',
            ),
         ),

Django offers two special drop-in concurrent migration index operations for PostgreSQL:

AddIndex -> AddIndexConcurrently
RemoveIndex -> RemoveIndexConcurrently

These operations are only available for PostgreSQL. For other databases you still need to use RunSQL.

The reason we couldn't use these operations for the implicit index on ForeignKey is that these operations can only be used for indexes defined in Meta.indexes, and not for implicitly created indexes or indexes created outside Django's context.

Order Migration Operations

So far we prevented the migration from re-creating the foreign key and made sure all index operations are concurrent. This is the current migration:

class Migration(migrations.Migration):
    atomic = False

    dependencies = [
        ('catalog', '0005_alter_product_created_by'),
        migrations.swappable_dependency(settings.AUTH_USER_MODEL),
    ]

    operations = [
        migrations.operations.SeparateDatabaseAndState(
            state_operations=[
                migrations.AlterField(model_name='product', name='last_edited_by', field=models.ForeignKey(db_index=False, null=True, on_delete=django.db.models.deletion. #...
            ],
            database_operations=[
                migrations.RunSQL(
                    'DROP INDEX catalog_product_last_edited_by_id_05484fb6;',
                    'CREATE INDEX catalog_product_last_edited_by_id_05484fb6 ON public.catalog_product USING btree (last_edited_by_id)',
                ),
            ],
        ),
        AddIndexConcurrently(
            model_name='product',
            index=models.Index(condition=models.Q(('last_edited_by__isnull', False)), fields=['last_edited_by'], # ...
        ),
    ]

If you pay close attention, this migration can still impact a live system. Consider the current order of migration operations:

Drop full index
Create partial index

Between the first and second steps, the system is left with no index. Keep in mind that we used concurrent operations, so we had to make the migration non-atomic. This means that all changes take effect immediately, and not at the end of the migration. This order of operations can have two consequences:

From the time the full index is dropped until the partial index is created, the system doesn't have an index. This can cause queries using the index to become slower.
If the migration fails between the first and second steps, the system will be left without an index until the migration is attempted again.

This solution to this headache is to simply switch the order of operations:

@@ -15,6 +15,10 @@ class Migration(migrations.Migration):
     operations = [
+        AddIndexConcurrently(
+            model_name='product',
+            index=models.Index(condition=models.Q(('last_edited_by__isnull', False)), fields=['last_edited_by'], #...
+        ),
         migrations.operations.SeparateDatabaseAndState(
             state_operations=[
                 migrations.AlterField(
@@ -30,8 +34,4 @@ class Migration(migrations.Migration):
                 ),
             ],
         ),
-        AddIndexConcurrently(
-            model_name='product',
-            index=models.Index(condition=models.Q(('last_edited_by__isnull', False)), fields=['last_edited_by'], #...
-        ),
     ]

First create the partial index and then drop the full index. This way, the system is never left without an index, and if the migration fails at any point, we still have at least one index.

💡 Takeaway

Adjust the order of migration operations to reduce the impact on a running application during the migration. It is usually better to create first, and drop after.

This wraps up the migrations part, on to actual business logic!

Locking Across Relations

Say we want to add a the ability to edit a product with the following requirements:

Let a user update the name and description of a product
Keep track of the user who last edited the product
Only superusers can edit products created by other superusers

A naive implementation can look like this:

class Product(models.Model):
    # ...
    def edit(self, *, name: str, description: str, edited_by: User) -> None:
        if self.created_by.is_superuser and not edited_by.is_superuser:
            # Only superusers can edit products created by other superusers.
            raise errors.NotAllowed()

        self.name = name
        self.description = description
        self.last_edited_by = edited_by
        self.save()

The edit function is implemented as an instance method on the Product model. It accepts the name and description to update, and the editing user. It performs the necessary permission check and moves on to set the required fields and save.

In some cases this is fine, and you can stop here. However, blindly operating on an instance like this is not always ideal:

The copy you have in-memory may be stale
Not safe against concurrent updates

If you are not convinced that concurrency is a real concern, check out How to Get or Create in PostgreSQL and Handling Concurrency Without Locks.

A solution more resilient to concurrent edits using a pessimistic approach can look like this:

from django.db import transaction

@classmethod
def edit(cls, id: int, *, name: str, description: str, edited_by: User) -> Self:
    with transaction.atomic():
        product = (
            cls.objects
            .select_for_update()
            .get(id=id)
        )

        if product.created_by.is_superuser and not edited_by.is_superuser:
            # Only superusers can edit products created by other superusers.
            raise errors.NotAllowed()

        product.name = name
        product.description = description
        product.last_edited_by = edited_by
        product.save()

    return product

These are the main differences:

Using a class method instead of an instance method: with an instance method we are operating on an already fetched object. To obtain a lock on the row we need to control how it's fetched from the database, so switching to classmethod.
Lock the row: to acquire a lock we must operate inside a database transaction. Then, to actually lock the row, we use select_for_update which resolves to a SELECT ... FOR UPDATE SQL clause.

All this locking business can be pretty distracting. So distracting in-fact, that you can end up missing the most obvious optimization in Django, the optimization that appears in every "top 10 Django optimizations" list. I'm taking of course, about the all mighty select_related!

To implement the permission check we access the user that created the product:

if product.created_by.is_superuser and not edited_by.is_superuser:
    # Only superusers can edit products created by other superusers.
    raise errors.NotAllowed()

The user is not fetched in advance, so when we access it, Django will go the database and fetch it. Since we know in advance that we are going to access the user, we can tell Django to fetch it along with the product using select_related:

product = (
    cls.objects
    .select_for_update()
    .select_related('created_by')
    .get(id=id)
)

This will reduce the number of queries by one. However, there is a little gotcha here that can easily go unnoticed.

Consider the following scenario with two session operating at the same time:

>>> # Session 1
>>> with transaction.atomic():
...     product = (
...          Product.objects
...         .select_for_update()
...         .select_related('created_by')
...         .get(id=1)
...     )
...
...
...     # transaction is still ongoing...

>>> # Session 2
>>>
>>>
>>>
>>>
>>>
>>> u = User.objects.get(id=97)
>>> u.is_active = False
>>> u.save()
# Blocked!

In this scenario, session 1 is in the process of editing a product. At the same time, session 2 is attempting to update the user and gets blocked. Why did session 2, which is not updating a product, gets blocked?

The reason session 2 got blocked is that SELECT ... FOR UPDATE locks rows from all the tables referenced by the query! In this case, when we added select_related we added a join to the query, to fetch both the product and the user who created it. As a result, the rows of both the product and the user who created it are locked! If someone happens to try to update a user while we update a product they created, they can get blocked.

⚠️ Implicit Behavior

By default, select_for_update locks the rows from all the referenced tables.

To avoid locking all the rows you can explicitly state which tables to lock using select_for_update:

@@ -80,6 +80,7 @@ class Product(models.Model):
    product: Self = (
        cls.objects
        .select_related('created_by')
+       .select_for_update(of=('self', ))
        .get(id=id)
    )

self is a special keyword that evaluates to the queryset's model, in this case, product.

💡 Takeaway

Always explicitly state which tables to lock when using select_for_update. Even if you don't have select_related right now, you might have in the future.

Permissive No Key Locks

Previously we used select_for_update(of=('self', )) to avoid locking the user when we only want to lock and update the product. Now let's look at another scenario. This time, we want to insert a new product:

>>> Product.objects.create(
...     name='my product',
...     description='a lovely product',
...     category_id=1,
...     category_sort_order=999997,
...     created_by_id=1,
...     last_edited_by_id=None,
... )
<Product 1000001>

This works.

Now let's do it again, but this time, while another session is trying to update user 1:

>>> # Session 1
>>> with transaction.atomic():
...   user = (
...     User.objects
...     .select_for_update(of=('self',))
...     .get(id=1)
...   )
...   # transaction is still ongoing...
...
...
...
...
...
...
...
...

>>> # Session 2
>>>
>>>
>>>
>>>
>>>
>>>
>>> Product.objects.create(
...   name='my product',
...   description='a lovely product',
...   category_id=1,
...   category_sort_order=999998,
...   created_by_id=1,
...   last_edited_by_id=None,
... )
... # Blocked!

Session 1 locks user 1 for update. At the same time, Session 2 attempts to insert a new product which is created by the same user and it gets blocked! Why did it get blocked?

Imagine you are the database. You have one session locking the user and another trying to reference it. How can you be sure the first session is not going to change the primary key for this user? How can you be sure it's not going to delete this user? If Session 1 is going to do any of these things, Session 2 should fail. This is why the database has to lock Session 2 until Session 1 completes.

The two scenarios the database is concerned about are valid, but very rare in most cases. How often do you actually update the primary key of an object or a unique constraint that may be referenced by a foreign key? probably never. In PostgreSQL, there is a way to communicate that to the database, and obtain a more permissive lock when selecting for update, using FOR NO KEY UPDATE:

>>> # Session 1
>>> with transaction.atomic():
...   user = (
...     User.objects
...     .select_for_update(no_key=True, of=('self',))
...     .get(id=1)
...   )
...   # transaction is still ongoing...
...
...
...
...
...
...
...

>>> # Session 2
>>>
>>>
>>>
>>>
>>>
>>>
>>> Product.objects.create(
...   name='my product',
...   description='a lovely product',
...   category_id=1,
...   category_sort_order=999998,
...   created_by_id=1,
...   last_edited_by_id=None,
... )
... <Product 1000002>

By setting no_key=True, we tell the database we are not going to update the primary key of the user. The database can then acquire a more permissive lock, and the second session can now safely create the product.

💡 Takeaway

Use select_for_update(no_key=True) to select a row for update when not updating primary keys or unique constraints that are referenced by foreign keys. This will require a more permissive lock and prevent unnecessary locks when operating on referencing objects.

Going back to our Product model, to edit a product we used select_for_update to prevent concurrent updates from updating the product at the same time. By locking the row, we also accidentally prevent other referencing models from creating objects while we have the lock. In the case of model like Product, this can have a huge impact on the system.

Imagine you run an e-commerce website using this catalog. When a user makes a purchase you create an Order instance that references the product. Now imagine that every time someone updates a product, your system can't create orders. This is unacceptable!

To allow the system to create orders while a product is being updated, add no_key=True:

--- a/demo/catalog/models.py
+++ b/demo/catalog/models.py
@@ -80,7 +80,7 @@ class Product(models.Model):
             product: Self = (
                 cls.objects
                 .select_related('created_by')
-                .select_for_update(of=('self', ))
+                .select_for_update(of=('self', ), no_key=True)
                 .get(id=id)
             )

Now we can safely edit a product without interfering with a live system.

The Final Model

It's been quite a ride from the naive model we started with, but here is the final model:

class Product(models.Model):
    class Meta:
        constraints = [
            models.UniqueConstraint(
                name='product_category_sort_order_uk',
                fields=(
                    'category',
                    'category_sort_order',
                ),
            ),
        ]
        indexes = (
            models.Index(
                name='product_last_edited_by_part_ix',
                fields=('last_edited_by',),
                condition=models.Q(last_edited_by__isnull=False),
            ),
        )

    id = models.BigAutoField(
        primary_key=True,
    )
    name = models.CharField(
        max_length=50,
    )
    description = models.TextField()

    category = models.ForeignKey(
        to=Category,
        on_delete=models.PROTECT,
        related_name='products',
        # Indexed in unique constraint.
        db_index=False,
    )
    category_sort_order = models.IntegerField()

    created_by = models.ForeignKey(
        to=User,
        on_delete=models.PROTECT,
        related_name='+',
        # Used to speed up user deletion.
        db_index=True,
    )

    last_edited_by = models.ForeignKey(
        to=User,
        on_delete=models.PROTECT,
        related_name='+',
        null=True,
        # Indexed in Meta.
        db_index=False,
    )

    @classmethod
    def edit(
        cls,
        id: int,
        *,
        name: str,
        description: str,
        edited_by: User,
    ) -> Self:
        with db_transaction.atomic():
            product: Self = (
                cls.objects
                .select_related('created_by')
                .select_for_update(of=('self', ), no_key=True)
                .get(id=id)
            )

            if product.created_by.is_superuser and not edited_by.is_superuser:
                # Only superusers can edit products created by other superusers.
                raise errors.NotAllowed()

            product.name = name
            product.description = description
            product.last_edited_by = edited_by
            product.save()

        return product

This model and the accompanying migrations are safe, resilient, and most importantly, production ready!

Takeaways

Here is a recap of the takeaways from this article:

Don't use unique_together, use UniqueConstraint instead
Always explicitly set db_index and add a comment
Always check the SQL generated by migrations before applying them
Provide reverse migration operations whenever possible
Use concurrent index operations in busy systems
Indexes on foreign keys are used indirectly by deletes
Use partial indexes when possible
Adjust the order of migration operations to reduce impact on live systems
Explicitly state tables to lock when using select_for_update
Use no_key=True when selecting a row for update to allow referencing objects to be created

The Mandatory AI Angle

Since it's 2025 you can't really have a serious article about technology that doesn't mention AI. So here is a funny story about AI and this exact article.

When I first presented this article in a talk at DjangoCon EU, I showed the final slide with all the takeaways and made a joke along the lines of "show me the LLM that can do that". Most people seemed to appreciate this little joke.

After the talk, a friend came to me and said "come man, let me show you something". He pulled his laptop and opened Cursor. He took my takeaways from the slide and put them in a rules file. He then had Cursor refactor one of his models.py files with my takeaways as guidelines... I can only say I had mixed feelings about the result.

Handling Concurrency Without Locks

2022-06-09T00:00:00+03:00

Concurrency is not very intuitive. You need to train your brain to consider what happens when multiple processes execute a certain code block at the same time. There are several issues I often encounter:

Failing to recognize potential concurrency issues: It's not uncommon for both beginner and seasoned developers to completely miss a potential concurrency problem. When this happens, and the concurrency issue end up causing bugs, it's usually very hard to trace and debug.
Dismiss concurrency issues due to low likelihood: If you recognized a potential concurrency issue, at some point you probably thought to yourself "what are the chances if this happening...". It's very tempting to dismiss concurrency issues when the likelihood is low. However, I personally found that concurrency issues tend to creep up at the worst time - when your system is under significant load and you have very little time (and grace) to come up with a solution.
Abusing locks: If you recognized a potential issue and decided to handle it properly, your next step will usually involve some kind of lock. Sometimes locks are necessary, but more often than not they can be avoided, or replaced by more permissive locks.

In this article I present common concurrency challenges and how to overcome them with minimal locking.

Other type of race...

Table of Contents

A Simple Web Application
Naive Implementation
Handling Possible Collisions
Time-of-check to Time-of-use
Locking
Lock in the Database
Asking for Forgiveness
Asking for Forgiveness in PostgreSQL
Identifying Race Conditions
Select for Update
Increment in the Database
Update and Immediately Return
Take Away

A Simple Web Application

A URL shortener provides short URLs that redirects to other URLs. There are several reasons why you would want that:

Include links in space-constrained places: Links in SMS messages, Tweets etc.
Track the number of clicks: Ads, campaigns, newsletter links, promotional emails etc.

To most developers a URL shortener sounds like a straight forward project. This is why it makes a great example to demonstrate common concurrency issues, and how easy they are to miss and get wrong. I'm using Python, Django and PostgreSQL, but the concepts apply to any programming language and RDBMs.

⚙️ Django Project setup

To build your URL shortener start by creating a new Django project:

$ python -m venv venv
$ source venv/bin/activate
$ pip install django
$ django-admin startproject project
$ cd project

This will create a Python virtual environment, install the latest Django version and create a Django project named project.

For the URL shortener, add a new app called shorturl in the project:

$ python manage.py startapp shorturl

Next, register the new app in settings.py:

# settings.py
INSTALLED_APPS = [
    # ...
    "shorturl.apps.ShortUrlConfig",
]

Django uses SQLlite by default. If you want to configure Django to use PostgreSQL instead, install psycopg and create a database:

$ pip install psycopg2
$ createdb -O yourdbuser shorturl

Then edit the following parameters in settings.py:

# settings.py
DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql",
        "NAME": "shorturl",
        "USER": "yourdbuser",
    }
}

Finally, run the initial migrations:

$ python manage.py migrate

You are now ready for the interesting part!

Naive Implementation

A short URL is composed of a short unique identifier that points to some target URL, and a counter to keep track of the number of hits. A Django model for a short URL can look like this:

from django.db import models

class ShortUrl(models.Model):
    key: str = models.CharField(max_length=20, unique=True)
    target_url: str = models.URLField()
    hits: int = models.PositiveIntegerField()

A simple function to create a new short URL can look like this:

import secrets, string

CHARACTERS = string.ascii_letters + string.digits

class ShortUrl(models.Model):

    # ...

    @classmethod
    def create(cls, target_url: str) -> ShortUrl:
        key = ''.join((secrets.choice(CHARACTERS) for _ in range(5)))
        return cls.objects.create(key=key, target_url=target_url, hits=0)

The function accepts a target URL, generates a random key from a list of possible characters and saves a new ShortUrl to the database. You can now use this function to create a new short URL:

>>> shorturl = ShortURL.create('https://hakibenita.com')
>>> vars(shorturl)
{'_state': <django.db.models.base.ModelState at 0x7fd5e05558a0>,
 'id': 1,
 'created_at': datetime.datetime(2022, 4, 29, 8, 2, 18, 615165, tzinfo=datetime.timezone.utc),
 'key': 'c6UFG',
 'target_url': 'https://hakibenita.com',
 'hits': 0}

This seems innocent enough, so what can possibly go wrong?

Handling Possible Collisions

Say your URL shortener becomes a wild success and you have millions of new short URLs created every day. At some point, the function that generates the random key may produce a key that already exist:

>>> shorturl = ShortUrl.create('https://hakibenita.com/tag/django')
IntegrityError: duplicate key value violates unique constraint "shorturl_shorturl_key_uk"
DETAIL:  Key (key)=(c6UFG) already exists.

Notice that the function generated the random key c6UFG which is similar to a short URL you previously created. The key column has a unique constraint defined on it, so you got a database error.

In order to avoid a unique constraint violation, you might try to check the key in advance, like this:

class ShortUrl(models.Model):

    @classmethod
    def create(cls, target_url: str) -> ShortUrl:
        while True:
            key = ''.join((secrets.choice(CHARACTERS) for _ in range(5)))
            if not cls.objects.filter(key=key).exists():
                break
        return cls.objects.create(key=key, target_url=target_url, hits=0)

Instead of creating the short URL immediately, you first check that the random key you generated does not already exist. If you find that is does, you keep iterating on random keys until you find one that doesn't.

Aside from the fact that this function can now potentially go into an infinite loop, there is another problem!

Time-of-check to Time-of-use

The purpose of checking that the key does not exist in advance was to avoid a database error, but is it really impossible to end up with a unique constraint violation this way? Consider the following scenario:

Process 1	Process 2
Check key `c6UFG` -> ✅
	Check key `c6UFG` -> ✅
	Use Key `c6UFG`
Use key `c6UFG` -> 💥 Already exists

Let's break it down:

Process #1 generates the random key c6UFG and checks that it does not already exist
Before Process #1 has a chance to write the new shorturl to the database, process #2 generates the same key c6UFG and checks that it does not already exist. At this point, it doesn't!
Process #2 writes the short URL to the database and completes successfully
Process #1 now tries to save a short URL with the same key c6UFG, which it checked in advance, and fails with a unique constraint violation

This is a very common concurrency issue commonly referred to as "time-of-check to time-of-use", or "TOCTOU". The name describes the issue pretty well - the problem is caused when another process changes the data between the time a process checked a value until the time it used it. In this case, Process #2 added a short URL with the same key after Process #1 had checked it, but before it used it.

Locking

Whenever you have a problem with multiple processes accessing the same resource at the same time, the most intuitive solutions is a lock. But, what exactly should you lock?

A simple architecture for a web application usually contains a web application process and a database:

Single process

If that's your setup, you might be able to obtain a lock at the process level and make sure you are the only one accessing a certain resource at a time.

However, a common setup for Django (as well as other web applications) is to run with multiple worker processes:

Multiple worker processes

With multiple worker processes on the same server it's no longer enough to lock a resource within a single process. However, since the two processes are running on the same server, you might be tempted to try and find a solution at the operating system level. But, is it enough?

Multiple servers

If your system is running on multiple servers, with multiple worker processes on each one, even the OS can't save you. You now might think the lowest common denominator is the application itself. You can spend days trying to come up with an original way to coordinate a lock between the servers, but would that solve your problem?

Multiple applications

Modern systems can run multiple applications on top of the same database. We for example, do it with Django admin.

At this point it becomes clearer that the lowest common denominator, the resource that all servers, processes and applications share, is the database. If you want to "lock" a resource, you better do it in the database.

Lock in the Database

Now that you know where to lock, there is another challenge. If you were updating an existing row in the database you could have locked that specific row, but this is not the case. You want to create a new row, for a new short URL, so what can you possibly lock?

One option is to lock the entire table:

from django.db import connection, transaction

class ShortUrl(models.Model):

    @classmethod
    def create(target_url: str) -> ShortUrl:
        with transaction.atomic(), connection.cursor() as cursor:
            cursor.execute('LOCK TABLE shorturl_shorturl IN EXCLUSIVE MODE;')
            while True:
                key = ''.join((secrets.choice(CHARACTERS) for _ in range(5)))
                if not cls.objects.filter(key=key).exists():
                    break
            return cls.objects.create(key=key, target_url=target_url, hits=0)

To prevent other processes from creating short URLs and potentially causing a unique constraint violation, you locked the entire shorturl table.

First, to obtain a lock in the database you need to operate inside a database transaction:

with transaction.atomic():

In most cases, Django operates in "autocommit" mode, meaning it implicitly opens a transaction for every command, and commits immediately after. In this case you want to execute multiple commands in the same database transaction, so you explicitly control the transaction using the transaction.atomic context.

The Django ORM does not provide functions for locking tables. To lock a table in the database you need to use a cursor and execute raw SQL:

with transaction.atomic(), connection.cursor() as cursor:
    cursor.execute('LOCK TABLE shorturl_shorturl IN EXCLUSIVE MODE;')

After you obtained an exclusive lock on the table, no other transaction can obtain the same lock until you release it. This guarantees that the data cannot change between the time you checked the values and the time you used them.

So, problem solved?

Asking for Forgiveness

Imagine your app makes it to the top page of a very popular news site. In a matter of minutes you start getting thousands of hits, and users are creating thousands of new short URLs. Now imagine your system can only create a single short URL from a single user at a time, and all other users need to wait in line. Is that acceptable?

When you locked the entire table you made sure no other transaction can make changes to the table until you are done. This means adding new short URLs is now safe, but it can also get pretty slow when you have many concurrent requests.

What if you could ditch the lock? What if you could make your function safe without preventing multiple users from creating short URLs at the same time? Have another look at the exception you got in the beginning:

>>> shorturl = ShortUrl.create('https://hakibenita.com/tag/django')
IntegrityError: duplicate key value violates unique constraint "shorturl_shorturl_key_uk"
DETAIL:  Key (key)=(c6UFG) already exists.

When the function attempted to create a short URL with a key that already existed, the database returned an error. This is because the key column has a unique constraint defined on it. So, if the database is already making sure that there are no duplicates, why not rely on it?

With this in mind, you can now change your function to handle the unique constraint violation:

from django.db import IntegrityError

class ShortUrl(models.Model):

    @classmethod
    def create(cls, target_url: str) -> ShortUrl:
        while True:
            key = ''.join((secrets.choice(CHARACTERS) for _ in range(5)))
            try:
                return ShortUrl.objects.create(key=key, target_url=target_url, hits=0)
            except IntegrityError:
                # Key exists, try try again!
                continue

The function now generates a random key and attempts to create a new short URL. If the command fails with an IntegrityError it means a short URL with the same key already exists. In this case, the function will generate another random key and try again until it succeeds.

The first thing you might notice is that there is no explicit lock, and there is no explicit database transaction. Multiple processes can now safely create short URLs without getting an IntegrityError.

According to this quote from the Python glossary, this approach is also much more "pythonic":

EAFP
Common Python coding style [ ... ] This clean and fast style is characterized by the presence of many try and except statements.

The term EAFP stands for "Easier to ask for forgiveness than permission", and this is exactly what you did. Instead of checking in advance, you tried to write the row to the database and handled the exception. The opposite of EAFP is LBYL, "Look Before You Leap", which is what you did when you checked the values in advance.

Python as a language encourages "asking or forgiveness" (EAFP), as opposed to other languages such as JavaScript or Java that encourage you to check values in advance, the LBYL style.

Asking for Forgiveness in PostgreSQL

Sometimes it's necessary to generate short URLs as part of another transaction. For example, if you include the short URL in a notification you save to the database, your code can look like this:

>>> with transaction.atomic():
...     shorturl = ShortUrl.create('https://hakibenita.com')
...     notification = Notification.create(to='...', message='...')

To make sure both the notification and the short URL are created together, you execute the code inside a database transaction. This is a perfectly valid use of database transactions - this is exactly what they are for!

If you use PostgreSQL, under some circumstances you might encounter this error:

InternalError: current transaction is aborted, commands ignored until end of transaction block

In PostgreSQL, when there is an error during a transaction, all other commands are aborted until the end of the transaction. Now recall how ShortUrl.create is implemented - you iterate over random keys and attempt to create until you don't get an IntegrityError. This means that if you execute your function inside a transaction and it did trigger an IntegrityError, PostgreSQL will abort the transaction, and you won't be able to proceed with the rest of the code.

To accommodate possible exceptions within transaction in PostgreSQL, you can use another transaction:

from django.db import transaction, IntegrityError

class ShortUrl(models.Model):

    @classmethod
    def create(cls, target_url: str) -> ShortUrl:
        while True:
            key = ''.join((secrets.choice(CHARACTERS) for _ in range(5)))
            try:
                # In PostgreSQL, when an SQL command fails (usually due to
                # IntegrityError) it is not possible to execute other commands
                # until the end of the atomic block. To be able to retry different
                # keys multiple times after, we execute the command in its own
                # atomic block.
                # https://docs.djangoproject.com/en/4.0/topics/db/transactions/#handling-exceptions-within-postgresql-transactions
                with transaction.atomic():
                    return ShortUrl.objects.create(key=key, target_url=target_url, hits=0)
            except IntegrityError:
                # Key exists, try try again!
                continue

The additional transaction introduces very little overhead, it simply restricts the effect a possible exception can have on any outer transactions.

Identifying Race Conditions

Now that you have all of these short URLs in your system, it's time to actually use them. The URL shortener system includes a view that redirects a short URL to its target URL, and increments the counter. The view can look like this:

# views.py
from django.http import HttpRequest, HttpResponse, HttpResponseRedirect, Http404
from django.views.decorators.http import require_http_methods

from .models import ShortUrl

@require_http_methods(["GET"])
def resolve_short_url(request: HttpRequest, key: str) -> HttpResponse:
    try:
        shorturl = ShortUrl.resolve(key)
    except ShortUrl.DoesNotExist:
        raise Http404()
    return HttpResponseRedirect(shorturl.target_url)

The view attempts to "resolve" a key to a ShortURL instance, and if it finds one, redirects to it.

A naive implementation of the function ShortUrl.resolve can look like this:

class ShortUrl(models.Model):

    @classmethod
    def resolve(cls, key: str) -> ShortUrl:
        shorturl = cls.objects.get(key=key)
        shorturl.hits += 1
        shorturl.save(update_fields=['hits'])
        return shorturl

The function accepts the key as argument and attempts to find a short URL with that key. If the key is not found, .get(key=key) throws a ShortUrl.DoesNotExist exception and the view will return a 404 response. If a short URL is found, the hit counter is incremented and saved, the object is returned to the view and the user is redirected to the target URL.

So, where is the problem?

As you already experienced when you created the short URL, concurrency issues often require special attention. Imagine what will happen if multiple users are trying to resolve the same key at the same time. Consider the following scenario:

Process 1	Process 2	Hits
Select hits -> 0		0
	Select hits -> 0	0
Update hits -> 1		1
	💥 Update hits -> 1	1

In this scenario, two processes are resolving the same URL at the same time. The short URL is resolved twice, but the hits counter is 1. This is incorrect!

Select for Update

The problem with the naive implementation is that when multiple concurrent processes resolve the same short URL at the same time, the counter can get out of sync. Each of the processes is incrementing the counter based on the value it received when it fetched the row from the database, and that's how the counter gets out of sync.

What if you could lock the row to prevent multiple processes from selecting and updating it at the same time? Consider the following implementation:

from django.db import transaction

class ShortUrl(models.Model):

    @classmethod
    def resolve(cls, key: str) -> ShortUrl:
        with transaction.atomic():
            shorturl = (
                cls.objects
                .select_for_update()
                .get(key=key)
            )
            shorturl.hits = shorturl.hits + 1
            shorturl.save(update_fields=['hits'])
        return shorturl

The function now opens a database transaction, and uses select_for_update to lock the row. If the lock is obtained, other processes cannot obtain the same lock on the row until the transaction is finished. This means the counter can no longer get out of sync, because only a single process can fetch and update it at the same time. But it also means that any concurrent processes must either wait or fail.

Imagine you launch a big campaign to hundreds of thousands of users. To check how effective your campaign is, you use your URL shortener to keep track of how many users clicked the links. Immediately when you send out the campaign it lands in your users' emails and thousands of them click the links. Now imagine each user needs to wait in line until the previous one fetched the row and updated the hit counter in the database. Sounds like it might be a problem...

Increment in the Database

Using select_for_update you solved the problem of the hit counter going out of sync, but your system is now doing a very poor job at the one thing it should be doing very well - redirect short URLs!

The main issue with the previous approach is that you incremented the counter based on what you fetched. With many concurrent processes operating at the same time, it is very much possible that since you fetched the row, the counter was incremented multiple times by other processes and you just don't know about it.

What if instead of incrementing the counter based on what you have stored in memory, you instruct the database to update based on what it currently has stored?

from django.db.models import F

class ShortUrl(models.Model):

    @classmethod
    def resolve(cls, key: str) -> ShortUrl:
        shorturl = cls.objects.get(key=key)
        shorturl.hits = F('hits') + 1
        shorturl.save(update_fields=['hits'])
        return shorturl

The function now uses an F expression to update the counter relative to what is in the database.

The difference seems very mild, so the best way to understand it is to look at the SQL generated by the update commands. The naive approach will execute the following command:

UPDATE shorturl_shorturl
SET hits = 1
WHERE id = 154;

This will update hits to 1, regardless of the current value of hits in the database.

The function using the F expression will execute the following command:

UPDATE shorturl_shorturl
SET hits = hits + 1
WHERE id = 154;

The hit counter is now incremented by one and not set to a fixed value. This is how using an F expression solves the problem without obtaining an explicit lock on the row.

Update and Immediately Return

Using F expressions solved the problem without an exclusive lock, but there are still two minor downsides to this approach:

There are two round trips to the database: the function first access the database to fetch the short URL by key, and then access it again to update the row.
The hit counter on the returned instance is not updated: the function is incrementing the value in the database, but the short URL object it returns stores the hits counter prior to when the hits counter was incremented.

Normally, the F expression solution with two round trips to the database is good enough, no reason to go through the trouble of trying to optimize it. However in this case, the system should be able to accommodate sudden bursts and redirect very quickly, so it might be worth the trouble.

To solve both issues, you can extend the solution beyond Django's ORM built-in capabilities with some SQL magic:

from django.db import transaction

class ShortUrl(models.Model):

    @classmethod
    def resolve(cls, key: str) -> ShortUrl:
        short_url = cls.objects.raw('''
            UPDATE shorturl_shorturl
            SET hits = hits + 1
            WHERE key = %s
            RETURNING *
        ''', [key])
        if not short_url:
            raise cls.DoesNotExist()
    return short_url[0]

Let's break it down:

Use RETURNING to get updated results: In PostgreSQL and SQLite you can return the rows affected by an UPDATE statement. The rows returned by the RETURNING clause are the updated rows. Even though you only update the column hits, you can use * to return the entire row with the updated values.
Use .raw to construct a ShortUrl instance: Django ORM allows you to construct ORM objects from raw SQL.
If no rows were affected, raise DoesNotExist: if no rows were affected it means there is no short URL for the provided key. To mimic Django's behavior in this case, raise a DoesNotExist exception, otherwise return the object.

Binding the table name

Under some circumstances you might want to avoid explicitly using the name of the table. Mainly if you have a reuseable model and you cannot be sure what the name of the database table is.

Using string concatenation to set the name of the table is not acceptable as it puts the query at risk of SQL Injection. Using psycopg, the Python driver for PostgreSQL, there is a way to safely bind identifiers such as table names:

from psycopg2.sql import SQL, Identifier

@classmethod
def resolve(cls, key: str) -> ShortUrl:
    short_url = cls.objects.raw(
        raw_query=SQL("""
            UPDATE {}
            SET hits = hits + 1
            WHERE key = %s
            RETURNING *
        """).format(Identifier(cls._meta.db_table)),
        params=(key,),
    )
    if not short_url:
        raise cls.DoesNotExist()
    return short_url[0]

The function now uses the name of the table as parameter.

Check out Preventing SQL Injection Attacks With Python for more about how to safely compose SQL for PostgreSQL in Python using Psycopg2.

Take Away

Throughout this article you used several different approaches to solve very common concurrency issues in two seemingly simple tasks:

Create short URL
- ⛔ #1: Failing to recognize possible collisions
- ⛔ #2: Time-of-check time-of-use (TOCTOU)
- ✅ #1: Lock!
- ✅ #2: Ask for forgiveness
Increment hit counter
- ⛔ #3: Ignore race conditions
- ✅ #3: Select for update
- ✅ #4: Increment in the database
- ✅ #5: Update and immediately return

Some of these approaches were fragile, and broke under even minor loads, and others has their advantages and disadvantages.

The main take away is this:

Keep concurrency in mind
Concurrency issues are very hard to miss, and even harder to debug. Recognizing concurrency issues requires special attention and awareness. This article covers the awareness part, so now you just need to pay attention.
Don't let probabilities distract you
It's very tempting to dismiss concurrency issues when the likelihood is low. Hopefully now that you know a few more ways to handle concurrency you are in a better position to resist the temptation.
Avoid locks if possible
Locks slow things down and cause contention so it's best to avoid them when possible. Now you know how.

For more advanced approaches for managing concurrency in Django, check out How to Manage Concurrency in Django Models.

One Database Transaction Too Many

2021-06-07T00:00:00+03:00

Have you ever wondered how bugs are born? I'm not talking about the trivial kind you can catch with simple unit testing. I'm talking about bugs that may not be apparent on first sight, but are so obvious in retrospect.

This is a story about how I accidentally sent hundreds of users messages they got paid when they didn't!

What it feels like when you realized you made a mistake
Illustration by Milica Vezmar Basara

Table of Contents

The Story
Remedies
Testing
- Testing with Django
- Testing with Pytest
Thoughts on Django Signals

The Story

We have a process in the system where we pay out money to merchants and other types of users. The payout process is a pretty big deal for most users because this is how they get paid.

Creating a Payout

To facilitate the payout process we have a Django model called PayoutProcess. To create a new payout we use a function that looks roughly like this:

from __future__ import annotations
from django.db import model, transaction as db_transaction

class PayoutProcess(models.Model):

    #... fields

    @classmethod
    def create(cls, to: User, amount: int) -> PayoutProcess:
        # ... Validate input ...

        with db_transaction.atomic():
            payout = cls.objects.create(
                to=user,
                amount=amount,
                status='pending',
            )

            # Create related objects etc...
        return payout

The simplified version of this function creates a new instance of a payout process and returns it. In real life this function validates the input and creates several related objects. To make sure all of the related objects are created along with the payout process instance, we use a database transaction.

The new instance now represents a payout process in the system, and the payout module is responsible for completing the payout. A payout can be fulfilled in many different ways, such as by bank transfer, credit card and other methods. Not all payout methods are immediate, so a payout is an asynchronous process that can take some time to complete.

When the payout is eventually paid, the module updates the instance status:

class PayoutProcess(models.Model):

    @classmethod
    def mark_paid(cls, pk: int) -> PayoutProcess:
        with db_transaction.atomic():
            payout = cls.objects.select_for_update().get(pk=pk)
            if payout.status != 'pending':
                raise StateError()
            payout.status = 'paid'
            payout.save()
        return payout

The function fetches the payout, checks its state and marks it as paid. So far so good!

Sending Notifications

At some point our staffers came to us with an idea. They said it would be nice if the system would notify users that they got paid. We thought that this is a great idea! Who doesn't like to get a message saying they got some $$$?

The payout module is a core module in our system. We have payouts going out to different types of users and top level apps use it to create payouts in different contexts. For example, one app sends out commission payments to merchants, and another issues payments to business partners.

To keep the payout module independent and decoupled from the apps that use it, the top level apps are the ones sending the notification to the users. The problem is that after the top level app created the payout process, the payout module handles the actual payment internally, and the top level app has no way of knowing about it unless it constantly monitors its status.

Top level app creates a payout

To have the top level app respond to changes in the payout module, we need to have a mechanism to let the top level app know that something had changed. The tricky part is that the top level apps depend on the payout module, so the payout module cannot depend back on them. It will cause a circular dependency.

Top level app gets notified when the payout is paid

One way to avoid circular dependencies and keep modules decoupled in Django is to use signals:

# payouts/signals.py
from django.dispatch import Signal

payout_paid = Signal()

After declaring the signal we can send it when a payout is paid. This is done by the model:

from . import signals

class PayoutProcess(models.Model):

    @classmethod
    def mark_paid(cls, pk: int) -> PayoutProcess:
        with db_transaction.atomic():
            payout = cls.objects.select_for_update().get(pk=pk)
            if payout.status != 'pending':
                raise StateError()
            payout.status = 'paid'
            payout.save()

        signals.payout_paid.send_robust(sender=PayoutProcess, payout=payout)
        return payout

Now a top level app can listen to this signal, and send a notification to the user:

from django.dispatch import receiver

import payout.signals
import payout.models
from .models import MerchantPayoutProcess

@receiver(payout.signals.payout_paid)
def on_merchant_was_paid(sender, payout: payout.models.PayoutProcess) -> None:
    try:
        p = MerchantPayoutProcess.objects.get(payout_id=payout.id)
    except MerchantPayoutProcess.DoesNotExist:
        # Not a merchant payout
        return

    p.user.email_user(f'Dear merchant, you got paid {payout.amount}$!')

When the signal receiver is triggered, it first checks to see if it's one of its own payouts. If it is, it fetches the related object, in this case a payout to a merchant, and sends a notification to the user.

N receivers

With this scheme, if you have N receivers, then every dispatch causes N-1 useless queries. This can be avoided by adding a bit of context to the signal.

dispatch_uid

It's usually a good idea to set dispatch_uid on signal receivers. The documentation explains it well.

The benefit of using signals this way is that the low level payout module can communicate with apps that depend on it, without forming a dependency back on them. This pattern eliminates the circular dependency and keeps low level modules independent and decoupled.

Working in Bulk

This design was working pretty well - payouts were flying out and the users were happy.

At some point the staffers came back with another idea. They said that work is picking up and they now want to automate and streamline some of it. They asked if it was possible to mark payouts as paid in bulk. After a quick discussion we decided it's best to have the bulk process "all or nothing", meaning, if the operation fails for even one of the payouts in the bulk, it should not be applied to any of them.

We figured this would be a straightforward task, all we have to do is execute the command on all of the given payouts inside of a database transaction:

from django.db import transaction as db_transaction

def mark_paid_in_bulk(payout_ids: list[int]) -> None:
    with db_transaction.atomic():
        for pk in payout_ids:
            PayoutProcess.mark_paid(pk)

The bulk process simply iterates over payout IDs and marks each one as paid. To make sure the process is atomic, or "all or nothing", we wrap the loop in a database transaction.

Easy, right? This is where it gets hairy.

The Bug

This bulk process was also working great for a while. Staffers would upload Excel files (what else), and the system would go over the payouts and mark them all as paid.

One day, the person that usually does this was on holiday, and asked someone else to do it instead. The other person prepared the Excel file and uploaded it to the system. This new person was not familiar with the process so they made some mistakes with the amounts. As a result, the system rejected some of the payouts.

Now what does a normal person do when a system reports an error? They try again and again...

At some point we started getting complaints from users saying they are getting a ton of messages that they got paid. Some were happy, but others opened the app to check the details and saw that they were in fact not paid, and realized it must be a mistake.

At this point hundreds of users got these messages, but none of them did get paid! So what's causing the issue? How are notifications being sent when all of the payouts are still marked as pending? A closer look at the implementation of the bulk process revealed the problem.

Nested Transactions

The function that marks a payout as paid is executed inside of a database transaction. To make sure that the signal is sent only when the payout status is committed to the database, the signal is sent after the transaction is completed:

@classmethod
def mark_paid(cls, pk: int) -> PayoutProcess:
    with db_transaction.atomic():
        payout = cls.objects.select_for_update().get(pk=pk)
        if payout.status != 'pending':
            raise StateError()
        payout.status = 'paid'
        payout.save()

    signals.payout_paid.send_robust(sender=PayoutProcess, payout=payout)
    return payout

When this function is executed on a single payout it works as expected. But what happens if we add the bulk process:

with db_transaction.atomic():
    for pk in payout_ids:

        # inline `mark_paid()`
        with db_transaction.atomic():
            payout = cls.objects.select_for_update().get(pk=pk)
            if payout.status != 'pending':
                raise StateError()
            payout.status = 'paid'
            payout.save()

        signals.payout_paid.send_robust(sender=PayoutProcess, payout=payout)

Ha! The bulk process is using its own database transaction! When the signal is sent the payout can still be rolled back if a later payout in the bulk fails.

Just to illustrate, if we mark three payouts in bulk and we fail to mark the third one, all three payouts are rolled back, but notifications for the first two had already been sent:

>>> from django.db import transaction as db_transaction
>>> with db_transaction.atomic():
...     for fail in [False, False, True]:
...         with db_transaction.atomic():
...             if fail:
...                 raise Exception('Failed!')
...         print('Message sent!')
...
Message sent!
Message sent!
Exception: Failed!

Notice how the first two messages were sent even though the third failure caused the outer transaction to rollback all three.

Remedies

The single mark_paid function assumes that it is not executed inside of a database transaction, but it is not checking or enforcing it in any way. This is a problem.

Assert Atomic Block

Before Django 3.2 we had some cases where we wanted to make sure a function is executed, or not executed, inside a database transaction. We ended up implementing two functions:

# common/db.py
from django.db import connection

def assert_is_in_atomic_block() -> None:
    assert connection.in_atomic_block, (
        'This function must be run inside of a DB transaction.'
    )

def assert_is_not_in_atomic_block() -> None:
    assert not connection.in_atomic_block, (
        'This function must not be run inside of a DB transaction.'
    )

Using these utility functions we could prevent some code from being executed inside a database transaction:

import common.db

def do_not_run_inside_a_db_transaction():
    common.db.assert_is_not_in_atomic_block()
    # Rest of function goes here...

Running this code block inside of an atomic block will now trigger an assertion error at runtime:

>>> from django.db import transaction as db_transaction
>>> with db_transaction.atomic():
...     do_not_run_inside_a_db_transaction()
AssertionError: This function must not be run inside of a DB transaction.

The main downside to this approach is that unless explicitly stated otherwise, tests will run inside a database transaction. This will cause any test which uses a transaction to fail. To overcome that we ended up patching these functions in tests:

@pytest.fixture(scope='session', autouse=True)
def patch_is_in_db_transaction():
    # Patch atomic transaction check in tests.
    # The checks can't be run in tests because tests are always wrapped in a transaction.
    patch_in = mock.patch('common.db.assert_is_in_atomic_block')
    patch_not_in = mock.patch('common.db.assert_is_not_in_atomic_block')
    with patch_in, patch_not_in:
        yield

This function creates a fixture which is automatically applied for the entire test session. The fixture mocks the two functions and disables their functionality.

Durable Transaction

Starting with Django 3.2, there is another way to prevent a transaction from being executed inside of another transaction, by marking a transaction as "durable":

with db_transaction.atomic(durable=True):
    payout = cls.objects.select_for_update().get(pk=pk)
    if payout.status != 'pending':
        raise StateError()
    payout.status = 'paid'
    payout.save()

signals.payout_paid.send_robust(sender=PayoutProcess, payout=payout)

If you try to open a durable transaction inside of another transaction, a RuntimeError is raised:

>>> from django.db import transaction as db_transaction
>>> with db_transaction.atomic():
...     with db_transaction.atomic(durable=True):
...             pass
...
RuntimeError: A durable atomic block cannot be nested within another atomic block.

Using a durable transaction may have prevented this issue from happening, but it would have also made the bulk feature impossible, or at least very complicated to implement!

Sending Signal on Commit

Another way to tackle this issue is to instead try to make sure the signal is only sent when the overall transaction is successfully committed. One way of doing this is using on_commit.

Using on_commit we can register a function to be executed only when the transaction is actually committed. To illustrate how using on_commit can solve the issue, consider the following example:

>>> from django.db import transaction as db_transaction
... with db_transaction.atomic():
...     for i, fail in enumerate([False, False, True], 1):
...         with db_transaction.atomic():
...             print(f'processing {i}...')
...             if fail:
...                 raise Exception('Failed!')
...             db_transaction.on_commit(lambda: print(f'Message sent!'))
processing 1...
processing 2...
processing 3...
Exception: Failed!

In the example we loop over three values where the third one is expected to fail. To print the message only when the transaction is committed successfully we use on_commit. Notice in the output that three items were processed, but since the third one failed, the entire process failed and none of the messages were sent.

To illustrate what happens when all items succeed, consider the following example:

>>> from django.db import transaction as db_transaction
... with db_transaction.atomic():
...     for i, fail in enumerate([False, False, False], 1):
...         with db_transaction.atomic():
...             print(f'processing {i}...')
...             if fail:
...                 raise Exception('Failed!')
...             db_transaction.on_commit(lambda: print(f'Message sent!'))
processing 1...
processing 2...
processing 3...
Message sent!
Message sent!
Message sent!

Amazing! Three items were processed and three messages were sent. We can now apply a similar fix to our payout module:

from . import signals

class PayoutProcess(models.Model):

    @classmethod
    def mark_paid(cls, pk: int) -> PayoutProcess:
        with db_transaction.atomic():
            payout = cls.objects.select_for_update().get(pk=pk)
            if payout.status != 'pending':
                raise StateError()
            payout.status = 'paid'
            payout.save()

        db_transaction.on_commit(lambda signals.payout_paid.send_robust(PayoutProcess, payout))
        return payout

When a payout is marked as paid the function now sends the signal only when the transaction is committed. This makes the function safe to execute inside of another transaction!

Using a Queue

When dealing with such problems it's not uncommon to immediately think about queues. As a thought exercise, let's examine two common patterns often referred to as "queues".

Async Tasks

Async task runners such as Celery are very popular. They let you execute tasks asynchronously, now, at a later time or at a predetermined time. Using async tasks in this case would not solve the issue:

Fire an async task in on_commit
If we set aside the fact that the payout module is not the one sending the messages, the outcome in this case is exactly the same as sending a signal in on_commit and firing an async task from the receiver (which is what we do).
Fire an async task instead of sending a signal
This will suffer from the same issues as the signal. If the bulk process fails, the task was already fired and the message will be sent.
Schedule an async task for a later time and check the status before sending
This may work in some cases, but there are other problems:
- We have a race: How long after the payout was processed should the task be executed? 1s? 10s? 1m? What if the bulk process takes 2m to run? When the task will be fired the transaction will not be commited yet and the message will not be sent. What do you do then?
- We do extra work: You now have to fetch the payout again before sending the message.
- We send the message later: If we wait, users can receive the message a few minutes or even hours after they were paid. This may not be a big deal in some cases, but in others sending the message close to when the event occurred may be crucial.

Another downside to using an async task runner, is that now you need to have an async task runner. If you already have one it may no be so bad, but if you don't, it can be a pain to set up and operate.

Transactional Queue

If you decided to implement a queue in the database you are likely one step closer to a proper solution. Instead of using signals, you can stage tasks to a database table that acts as a queue.

The main benefit of using a queue table in the database is that the task will be added only when the transaction is committed. This plays very nicely with the overall transaction management of the process, and guarantees that the task is only added when it should.

The challenging part is how to make sure the tasks are being picked up shortly after they were added to the queue. If you are using a cron job to process tasks, the sending may be delayed up to the cron job's repeat interval. If you use database triggers, LISTEN/NOTIFY or similar to trigger processing of tasks then the delay can be shorter.

Testing

We ended up implementing the on_commit solution because it required very little changes to existing code. However, after we finished making the changes to the code, we faced yet another challenge - the tests!

Testing with Django

Our tests included scenarios to make sure a notification is sent when a payout is paid:

def test_should_send_notification(db, mailoutbox, merchant_user: User) -> None:
    comm = MerchantCommission.create_payout(merchant_user, amount=100_00)
    PayoutProcess.mark_paid(comm.payout_process_id)
    assert len(mailoutbox) == 1

After we made the change to send the signal in on_commit, all of these tests failed. After some debugging we found that the receiver function we registered for the signals was not executed, but only in the tests!

The fact that the on_commit handler is not fired is not surprising if you know how tests are executed. To speed things up, Django starts a database transaction at the beginning of every test and then rolls it back immediately after. Executing tests in this manner is a fast way to prevent tests that change data in the database from affecting each other.

To make it possible to test things that are triggered in on_commit without using slow transactional tests, Django 3.2 added a new context manager called captureOnCommitCallbacks (Ticket #30457):

from django.core import mail
from django.test import TestCase

class TestPayoutProcess(TestCase):

    def test_should_send_notification(self, merchant_user: User) -> None:
        comm = MerchantCommission.create_payout(merchant_user, amount=100_00)
        with self.captureOnCommitCallbacks(execute=True):
            PayoutProcess.mark_paid(comm.payout_process_id)
        assert len(mail.outbox) == 1

The context manager is available on instances of TestCase, and when execute=True any on_commit handlers will also be executed, not just captured.

Testing with Pytest

Unfortunately we are not using Django's TestCase directly anymore, we are using pytest, and we were not in a position to start rewriting stuff. Lucky for us, pytest-django implemented equivalent functionality. A quick upgrade to pytest-django version 4.4 and we were ready to go:

def test_should_send_notification(
    db,
    mailoutbox,
    django_capture_on_commit_callbacks,
    merchant_user: User,
) -> None:
    comm = MerchantCommission.create_payout(merchant_user, amount=100_00)
    with django_capture_on_commit_callbacks(execute=True):
        PayoutProcess.mark_paid(comm.payout_process_id)
    assert len(mailoutbox) == 1

The fixture django_capture_on_commit_callbacks is based on the Django function. Once you inject it you can use it just like you would the Django one.

The "bug" caused by this nested transaction ended up causing some users to get multiple messages saying they got paid, but all of these users were eventually paid.

Thoughts on Django Signals

As described in this story, Django signals are useful for implementing interactions between modules without creating explicit dependencies between them. The official documentation about Signals also provide this as the main reason for using signals:

Django includes a “signal dispatcher” which allows decoupled applications get notified when actions occur elsewhere in the framework. In a nutshell, signals allow certain senders to notify a set of receivers that some action has taken place. They’re especially useful when many pieces of code may be interested in the same events.

If you look at how signals are implemented in Django you'll find that there is not a lot of magic under the hood. The function connect adds a function to a list of receivers, and when a signal is send (or send_robust) the signal object iterates over the list of receiver functions, and executes them one by one.

This is very similar to a pub-sub pattern, but it lacks some of the guarantees of more advanced implementations. One of the main disadvantages of Django signals is that there is no guarantee that a "message" ever reaches the destination. If for example, the server crashes while a signal is being broadcast, some receivers may not be executed and they will not be attempted when the service starts up again. This can become a problem if you rely on signals exclusively to trigger certain actions in the system.

Exciting New Features in Django 3.2

2021-03-03T00:00:00+02:00

Django 3.2 is just around the corner and it's packed with new features. Django versions are usually not that exciting (it's a good thing!), but this time many features were added to the ORM, so I find it especially interesting!

This is a list of my favorite features in Django 3.2

Image from the Django welcome page

A lot of great people worked on this release and none of them is me. I included links to the tickets of each new feature to show my appreciation to the people behind it.

Table of Contents

Covering Indexes
Provide Timezone to TruncDate
Building JSON Objects
Loud Signal Receiver
QuerySet Alias
New Admin Decorators
Value Expression Detects Type
More Mentionable Features
Wishlist

⚙ Setup local environment with the latest version of Django

To setup an environment with the latest version of Django start by creating a new directory and a virtual environment:

$ mkdir django32
$ cd django32
$ python3.9 -m venv venv
$ source venv/bin/activate

To install the latest version of Django you can either install using pip, or if it hasn't been released yet, install directly from git:

(venv) $ pip install git+https://github.com/django/django@3.2a1

Start a new project and app:

(venv) $ django-admin startproject project
(venv) $ ./manage.py startapp store

Add the new app to the list of INSTALLED_APPS, and configure a PostgreSQL database:

# settings.py

INSTALLED_APPS = [
    # ...
    'store',
]

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'django32',
        'USER': 'postgres',
    }
}

To try out some of the new features, create a Customer model:

# store/models.py
import datetime

from django.db import models

class Customer(models.Model):
    joined_at: datetime.datetime = models.DateTimeField()
    name: str = models.CharField(max_length=1000)

Finally, create the DB, generate and apply the migrations:

(venv) $ createdb django32 -O postrges
(venv) $ ./manage.py makemigrations
(venv) $ ./manage.py migrate

Great! Now add some random customer data:

import string
import datetime
import random
import pytz

starting_at = pytz.UTC.localize(datetime.datetime(2021, 1, 1))
DAY = datetime.timedelta(days=1)

Customer.objects.bulk_create((Customer(
    joined_at = starting_at + (DAY * random.random() * 365),
    name = ''.join(random.choices(string.ascii_letters + ' ' * 10, k=random.randint(10, 20))),
) for __ in range(10_000)))

Congratulations! You now have 10K new customers, you are ready to go!

Covering Indexes

Ticket #30913

Covering indexes let you store additional columns in an index. The main benefit of a covering index is that when a query only uses fields that are present in the index, the database can use an index-only scan, meaning the actual table is not accessed at all. This can make queries faster.

Django 3.2 added support for PostgreSQL covering indexes:

The new Index.include and UniqueConstraint.include attributes allow creating covering indexes and covering unique constraints on PostgreSQL 11+.

If for example you are searching for names of customers that have joined during a certain period of time, you can create an index on joined_at, and include the field name in the index:

class Customer(models.Model):
    class Meta:
        indexes = (
            models.Index(
                name='%(app_label)s_%(class)s_joined_at_ix',
                fields=('joined_at',),
                include=('name',),
            ),
        )

The include arguments makes this a covering index.

For queries that only use the fields joined_at and name, the database will be able to satisfy queries using just the index:

>>> print(
...   Customer.objects
...   .filter(joined_at__lt=pytz.UTC.localize(datetime.datetime(2021, 2, 1)))
...   .values_list('name')
...   .explain()
... )

Index Only Scan using store_customer_joined_at_ix on store_customer
  Index Cond: (joined_at < '2021-02-01 00:00:00+00'::timestamp with time zone)

The query above finds names of customers that joined before February 2021. According to the execution plan, the database was able to satisfy the query using just the index, without even accessing the table. This is called an "index only scan".

Index only scans can be a bit confusing at first. As described in the official documentation of PostgreSQL, it might take some time before PostgreSQL can actually use just the index:

But there is an additional requirement for any table scan in PostgreSQL: it must verify that each retrieved row be “visible” to the query's MVCC snapshot [...]. Visibility information is not stored in index entries, only in heap entries; so at first glance it would seem that every row retrieval would require a heap access anyway.

Another way to check if a table page can be viewed by the current transaction is to check the table's visibility map, which is significantly smaller and faster to access than the table itself. It may take some time for PostgreSQL to update the visibility map, so until then you might see an execution plan like this one:

Bitmap Heap Scan on store_customer  (cost=27.07..117.02 rows=876 width=16)
  Recheck Cond: (joined_at < '2021-02-01 00:00:00+00'::timestamp with time zone)
  ->  Bitmap Index Scan on store_customer_joined_at_ix  (cost=0.00..26.86 rows=876 width=0)
        Index Cond: (joined_at < '2021-02-01 00:00:00+00'::timestamp with time zone)

To check if your index can really be used for index only scans, you can speed up the process by manually issuing vacuum analyze on the table:

VACUUM ANALYZE store_customer;

Executing VACUUM will also reclaim some unused space and make it available for re-use.

UPDATE 2020-03-04: I originally suggested using VACUUM FULL instead of plain VACUUM. A commenter on twitter mentioned that using VACUUM will be sufficient for this purpose, and will be much less intrusive and disruptive, so use that instead!

It is also important to keep in mind that inclusive indexes are not free. Additional fields in the index will make the index bigger.

Provide Timezone to TruncDate

Ticket #31948

I write a lot about mistakes in SQL and timezones are usually at the top of the list. One of the most dangerous mistakes when working with timestamps is truncating without explicitly specifying a timezone, which can lead to incorrect and inconsistent results.

In Django 3.2 it becomes easier to avoid this mistake:

The new tzinfo parameter of the TruncDate and TruncTime database functions allows truncating datetimes in a specific timezone.

In previous Django versions, the timezone was set internally according to the current timezone:

# django/db/models/functions/datetime.py
tzname = timezone.get_current_timezone_name() if settings.USE_TZ else None

As of Django 3.2, you can explicitly provide a timezone to the TruncDate functions family:

import pytz
from django.db.models.functions import TruncDay

Customer.objects
.annotate(joined_at_day=TruncDay('joined_at', tzinfo=pytz.UTC))
.values('joined_at_day')

# SELECT DATE_TRUNC('day', "store_customer"."joined_at" AT TIME ZONE 'UTC') AS "joined_at_day"
# FROM "store_customer"

Customer.objects
.annotate(joined_at_day=TruncDay('joined_at', tzinfo=pytz.timezone('America/New_York')))
.values('joined_at_day')

# SELECT DATE_TRUNC(
#     'day',
#     "store_customer"."joined_at" AT TIME ZONE 'America/New_York'
# ) AS "joined_at_day"
# FROM "store_customer"

A step in the right direction!

Building JSON Objects

Ticket #32179

Building JSON objects in PostgreSQL is very handy, especially if you are working with unstructured data.

As of Django 3.2, the function json_build_object from PostgreSQL that accepts arbitrary key-value pairs was added to the ORM:

Added the JSONObject database function.

One interesting use case is serializing objects directly in the DB, bypassing the need to create ORM objects:

>>> from django.db.models import F
>>> from django.db.models.functions import JSONObject
>>> Customer.objects.annotate(obj=JSONObject(
    id=F('id'),
    name=F('name'),
    joined_at_day=TruncDay('joined_at', tzinfo=pytz.UTC),
).values_list('obj').first()

({
    'id': 1,
    'name': 'Haki Benita',
    'joined_at_day': '2021-04-25T00:00:00',
},)

We already showed how crucial serialization performance can be, so this is something to consider.

Loud Signal Receiver

Ticket #32261

A while back I twitted about a mysterious bug I had that went unnoticed for a long time because it happened inside a signal receiver.

When you use send_robust to broadcast signals, if the signal fails, Django keeps the error and moves on to the next receiver. After all of the receivers processed the signal, Django returns a list with the receivers' return values and exceptions. To check if any of the receivers failed, you need to go over the list and check for instances of Exception. Signals are often used to decouple modules, and handing exceptions from receivers this way defeats that purpose.

To make sure I don't miss exceptions in signal receivers again, I created a "loud signal receiver" that logs exceptions:

from django.dispatch import receiver

def loud_receiver(signal, logger, **kwargs):
    """Subscribe to Django signal and log errors from the receiver function.

    When using `send_robust` to send Django Signals, errors happening in the
    receivers are kept and returned. Because signals are mostly used for decoupling
    modules, the return value from `send_robust` is often dismissed.
    To make it easier not to miss errors from Django signal receivers, use this decorator
    instead to log the exceptions to a specific logger.

    NOTE: Not necessary as of Django 3.2

    Example:
        logger = logging.getLogger('some.logger')
        @loud_receiver(signals.SomeSignal, logger=logger, dispatch_uid='uid')
        def receiver_func():
            pass
    """
    def _decorator(func):
        def loud_func(*func_args, **func_kwargs):
            try:
                func(*func_args, **func_kwargs)
            except Exception:
                logger.exception('exception from signal receiver')
                raise
        return receiver(signal, **kwargs)(loud_func)
    return _decorator

As of Django 3.2 this is no longer necessary:

Signal.send_robust() now logs exceptions.

Great!

QuerySet Alias

Ticket #27719

The alias function is an entirely new feature in Django 3.2:

The new QuerySet.alias() method allows creating reusable aliases for expressions that don’t need to be selected but are used for filtering, ordering, or as a part of complex expressions.

I often use SubQuery and OuterRef to write complex queries, and there is a little gotcha when combined with annotate:

from django.db.models import Subquery, OuterRef

Customer.objects.annotate(
    id_of_previous_customer=Subquery(
        Customer.objects
        .filter(joined_at__lt=OuterRef('joined_at'))
        .order_by('-joined_at')
        .values('id')[:1],
    )
).filter(id_of_previous_customer__isnull=True)

The query above is a complicated way to find the first customer that joined. The queryset is using SubQuery to find the previous customer for every customer by joined_at, and then looks for the customer which no other customer has joined before. This is very inefficient, but I'm using it to illustrate my point.

To understand the problem, inspect the query this queryset is producing:

SELECT
    "store_customer"."id",
    "store_customer"."joined_at",
    "store_customer"."name",
    (
        SELECT U0."id"
        FROM "store_customer" U0
        WHERE U0."joined_at" < "store_customer"."joined_at"
        ORDER BY U0."joined_at"
        DESC LIMIT 1
    ) AS "id_of_previous_customer"
FROM
    "store_customer"
WHERE
    (
        SELECT U0."id"
        FROM "store_customer" U0
        WHERE U0."joined_at" < "store_customer"."joined_at"
        ORDER BY U0."joined_at" DESC
        LIMIT 1
    ) IS NULL

The annotated subquery appears in both the SELECT and the WHERE clauses. This affects the execution plan:

Seq Scan on store_customer  (cost=0.00..4401.05 rows=50 width=32)
  Filter: ((SubPlan 2) IS NULL)
  SubPlan 1
    ->  Limit  (cost=0.29..0.42 rows=1 width=12)
        ->  Index Scan Backward using store_customer_joined_at_ix on store_customer u0
            (cost=0.29..450.59 rows=3333 width=12)
            Index Cond: (joined_at < store_customer.joined_at)
  SubPlan 2
    ->  Limit  (cost=0.29..0.42 rows=1 width=12)
        ->  Index Scan Backward using store_customer_joined_at_ix on store_customer u0_1
            (cost=0.29..450.59 rows=3333 width=12)
            Index Cond: (joined_at < store_customer.joined_at)

The subquery is executed twice!

To solve this in Django versions prior to 3.2, you can provide a values_list that excludes the annotated subquery from the SELECT clause:

# Django 3.1
from django.db.models import Subquery, OuterRef

Customer.objects.annotate(
    id_of_previous_customer=Subquery(
        Customer.objects
        .filter(joined_at__lt=OuterRef('joined_at'))
        .order_by('-joined_at')
        .values('id')[:1],
    )
).filter(id_of_previous_customer__isnull=True)
.values('id')

Side note: You might think that instead of using values_list in this case you can omit the annotated field using .defer('id_of_previous_customer'). This won't work. Django will throw a KeyError: 'id_of_previous_customer' at you!

Starting with Django 3.2, you can replace annotate with alias and the field will not be added to the select clause:

# Django 3.2
from django.db.models import Subquery, OuterRef

Customer.objects.alias(
    id_of_previous_customer=Subquery(
        Customer.objects
        .filter(joined_at__lt=OuterRef('joined_at'))
        .order_by('-joined_at')
        .values('id')[:1],
    )
).filter(id_of_previous_customer__isnull=True)

The generated SQL now uses the subquery only once:

SELECT "store_customer"."id", "store_customer"."joined_at", "store_customer"."name"
FROM "store_customer"
WHERE (
    SELECT U0."id"
    FROM "store_customer" U0
    WHERE U0."joined_at" < "store_customer"."joined_at"
    ORDER BY U0."joined_at" DESC
    LIMIT 1
) IS NULL

The execution plan is simpler:

Seq Scan on store_customer  (cost=0.00..4380.04 rows=50 width=28)
  Filter: ((SubPlan 1) IS NULL)
  SubPlan 1
    ->  Limit  (cost=0.29..0.42 rows=1 width=12)
        ->  Index Scan Backward using store_customer_joined_at_ix on store_customer u0
            (cost=0.29..450.59 rows=3333 width=12)
            Index Cond: (joined_at < store_customer.joined_at)

One less way to shoot yourself in the foot!

New Admin Decorators

Ticket #16117

Before Django 3.2, to customize a calculated field in Django admin you first added a function, and then assigned some attributes to it:

# Django 3.1
from django.contrib import admin

from .models import Customer

@admin.register(Customer)
class CustomerAdmin(admin.ModelAdmin):
    list_display = (
        'id',
        'joined_at',
        'joined_at_year',
        'name',
    )

    def joined_at_year(self, obj: Customer) -> str:
        return obj.joined_at.year
    joined_at_year.admin_order_field = 'joined_at__year'
    joined_at_year.short_description = 'Year joined'

This is the kind of weird APIs that are mostly only possible in dynamic languages such as Python.

If you are using Mypy (and you should), this code will trigger an annoying warning, and the only way to silence it is to add a type: ignore comment:

...
joined_at_year.admin_order_field = 'joined_at__year'  # type: ignore[attr-defined]
joined_at_year.short_description = 'Year joined'  # type: ignore[attr-defined]

If you are using Django Admin and Mypy as much as I do, this can be pretty annoying.

The new display decorator solves this problem:

The new display() decorator allows for easily adding options to custom display functions that can be used with list_display or readonly_fields. Likewise, the new action() decorator allows for easily adding options to action functions that can be used with actions.

Adjusting the code to use the new display decorator:

@admin.display(ordering='joined_at__year', description='Year joined')
def joined_at_year(self, obj: Customer) -> str:
    return obj.joined_at.year

No type errors!

Another useful decorator is action which uses a similar approach to customize custom admin actions.

Value Expression Detects Type

Ticket #30446

This is a small feature that addresses a small nuisance in the ORM:

Value() expression now automatically resolves its output_field to the appropriate Field subclass based on the type of its provided value for bool, bytes, float, int, str, datetime.date, datetime.datetime, datetime.time, datetime.timedelta, decimal.Decimal, and uuid.UUID instances. As a consequence, resolving an output_field for database functions and combined expressions may now crash with mixed types when using Value(). You will need to explicitly set the output_field in such cases.

In previous Django versions if you wanted to use some constant value in a query, you had to explicitly set an output_field, otherwise it will fail:

>>> # Django 3.1
>>> from django.db.models import Value
>>> Customer.objects.annotate(
    number=Value(1),
    text=Value('text'),
    boolean=Value(True),
    date_=Value(datetime.date(2020, 1, 1)),
    datetime_=Value(pytz.UTC.localize(datetime.datetime(2020, 1, 1))),
).values_list('number', 'text', 'boolean', 'date_', 'datetime_').first()

FieldError: Cannot resolve expression type, unknown output_field

In Django 3.2, the ORM figures it out on its own:

>>> # Django 3.2
>>> from django.db.models import Value
>>> Customer.objects.annotate(
    number=Value(1),
    text=Value('text'),
    boolean=Value(True),
    date_=Value(datetime.date(2020, 1, 1)),
    datetime_=Value(pytz.UTC.localize(datetime.datetime(2020, 1, 1))),
).values_list('number', 'text', 'boolean', 'date_', 'datetime_').first()

(1, 'text', True, datetime.date(2020, 1, 1), datetime.datetime(2020, 1, 1, 0, 0, tzinfo=<UTC>))

Very cool!

More Mentionable Features

There are plenty more features in Django 3.2 that the documentation explains better than me. To name a few:

Navigable links in the admin (Ticket #31181): Read-only related fields are now rendered as navigable links if target models are registered in the admin. I'm still using a decorator to add links to Django Admin, guess now i'll have less use for it.
Durable argument for atomic() (Ticket #32220): When you execute code inside a database transaction, when the transaction finishes without any errors you expect it to be committed to the database. However, if the caller executed your code inside a database transaction of his own, if the parent transaction is rolled back, so is yours. To prevent this from happening, you can now mark your transaction as durable. When there is an attempt to open a durable transaction inside another transaction, a RuntimeError error is raised.
Cached templates are reloaded on Django's development server (Ticket #25791): If you are using Django's runserver command to develop locally, you probably got used to it reloading when a python file changes. However, if you are using Django's django.template.loaders.cached.Loader loader, when an HTML file is changing the dev server will not reload it, and you will have to restart the devserver to see the changes. This is pretty annoying, and so far I had to disable the cached loader in dev. Starting at Django 3.2 this is no longer necessary because cached templates are correctly reloaded in development.
Support for function based indexes (Ticket #26167): FBIs are useful when you query an expression often and you want to index it. A classic example is indexing lower case texts.

Wishlist

Django ORM is comprehensive and feature rich, but there are still some things on my wishlist for future versions:

Custom joins: Django is currently able to perform joins only between tables that are connected via ForeignKey. There are situations where you want to join tables that are not necessarily connected with a foreign key, or using more complex conditions. One common example is slowly changing dimensions, where a join condition require a BETWEEN operator.
Update returning: When updating many rows it's sometimes useful to fetch them immediately. This is a well known (and a very useful) feature in SQL. Django currently has no support for it, but I hear it might soon.
Database views: There are many hacks for getting database views to work with the ORM. These hacks usually involves creating a view directly in the database or in a manual migration, and then setting managed=False on the model. These hacks get the job done, but not in a very graceful way. I wish there was a way to define database views so that migrations can detect changes. Maybe even an option to create views using a Django queryset.
Database partitions: Database partitions are extremely useful in data modeling. When used correctly they can make queries much faster and maintenance a lot easier. Some database engines such as Oracle already provide very mature implementations for database partitioning, and other engines such as PostgreSQL are getting there. At the moment, there is no native support for database partitioning in Django, and most implementations I've seen resort to manually managing tables. As a result, I often avoid partitions all together and that's unfortunate.
Require authentication by default: Django currently permits access to any view unless explicitly marked otherwise, usually using the require_login decorator. This makes it easier to get started with Django, but it can potentially cause security issues down the road if you are not careful. I know there are solutions for this, usually using custom middleware and decorators. I really wish Django had an option to flip the condition so that access is restricted by default unless marked otherwise.
Typing: If you are following this blog you know I'm a big fan of type hinting in Python. At the moment, Django does not come with type hinting or official stubs. Shiny new frameworks such as Starlette and FastAPI advertise themselves as being 100% type annotated, but Django is still lagging behind. There is a project called django-stubs that is making some progress in this regard.
Database connection pooling Django currently supports two modes for managing database connections - creating a new connection per request, or a new connection per thread (persistent connections). Creating database connections in common deployments is a relatively heavy operation. It requires setting up a TCP connection, often a TLS connection, and initializing the connection, which adds significant latency. In PostgreSQL in particular, it also consumes many database server resources, so creating a new connection per request is a really bad idea.

Persistent connections are much better. They work well with the way Django is usually deployed, small amount of worker processes and/or threads. But such deployments tend to breakdown under real world conditions. Whenever your database or one of your upstreams starts taking longer to process requests for some reason, the workers get tied up, requests back up, and the entire system chokes. Even with strict timeouts, this will still happen.

To improve upon this catastrophic failure mode, a common solution is to use async workers such as gevent greenlets, or in the future, asycnio tasks. But now, each request gets its own lightweight thread, hence its own connection, which renders Django's persistent connections feature useless.

It would be great if Django included a high-quality connection pool, which maintains a certain number of connections and hands them out to requests as needed. External solutions like PgBouncer exist, but they add operational overhead. A built-in solution would often be sufficient.

The Unexpected Find That Freed 20GB of Unused Index Space

2021-02-01T00:00:00+02:00

Every few months we get an alert from our database monitoring to warn us that we are about to run out of space. Usually we just provision more storage and forget about it, but this time we were under quarantine, and the system in question was under less load than usual. We thought this is a good opportunity to do some cleanups that would otherwise be much more challenging.

To start from the end, we ended up freeing more than 70GB of un-optimized and un-utilized space without dropping a single index or deleting any data!

Using conventional techniques such as rebuilding indexes and tables we cleared up a lot of space, but then one surprising find helped us clear an additional ~20GB of unused indexed values!

This is what the free storage chart of one of our databases looked like in the process:

Free space over time (higher means more free space)

Table of Contents

The Usual Suspects
- Unused Indexes
- Index and Table Bloat
The "Find"
- The "Aha Moment"
- Utilizing Partial Indexes
Bonus: Migrating with Django ORM
- Prevent Implicit Creation of Indexes on Foreign Keys
- Migrate Exiting Full Indexes to Partial Indexes
Conclusion

The Usual Suspects

Provisioning storage is something we do from time to time, but before we throw money at the problem we like to make sure we make good use of the storage we already have. To do that, we start with the usual suspects.

Unused Indexes

Unused indexes are double-edged swords; you create them to make things faster, but they end up taking space and slow inserts and updates. Unused indexes are the first thing we always check when we need to clear up storage.

To find unused indexes we use the following query:

SELECT
    relname,
    indexrelname,
    idx_scan,
    idx_tup_read,
    idx_tup_fetch,
    pg_size_pretty(pg_relation_size(indexrelname::regclass)) as size
FROM
    pg_stat_all_indexes
WHERE
    schemaname = 'public'
    AND indexrelname NOT LIKE 'pg_toast_%'
    AND idx_scan = 0
    AND idx_tup_read = 0
    AND idx_tup_fetch = 0
ORDER BY
    pg_relation_size(indexrelname::regclass) DESC;

The query is looking for indexes that were not scanned or fetched since the last time the statistics were reset.

Some indexes may seem like they were not used but they were in-fact used:

The documentation lists a few scenarios when this is possible. For example, when the optimizer uses meta data from the index, but not the index itself.
Indexes used to enforce unique or primary key constraints for tables that were not updated in a while. The indexes will look like they were not used, but it doesn't mean we can dispose of them.

To find the unused indexes you can actually drop, you usually have to go over the list one by one and make a decision. This can be time consuming in the first couple of times, but after you get rid of most unused indexes it becomes easier.

It's also a good idea to reset the statistics counters from time to time, usually right after you finished inspecting the list. PostgreSQL provides a few functions to reset statistics at different levels. When we find an index we suspect is not being used, or when we add new indexes in place of old ones, we usually reset the counters for the table and wait for a while:

-- Find table oid by name
SELECT oid FROM pg_class c WHERE relname = 'table_name';
-- Reset counts for all indexes of table
SELECT pg_stat_reset_single_table_counters(14662536);

We do this every once in a while, so in our case there were no unused indexes to drop.

Index and Table Bloat

The next suspect is bloat. When you update rows in a table, PostgreSQL marks the tuple as dead and adds the updated tuple in the next available space. This process creates what's called "bloat", which can cause tables to consume more space than they really need. Bloat also affects indexes, so to free up space, bloat is a good place to look.

Estimating bloat in tables and indexes is apparently not a simple task. Lucky for us, some good people on the world wide web already did the hard work and wrote queries to estimate table bloat and index bloat. After running these queries you will most likely find some bloat, so the next thing to do it clear up that space.

Clearing Bloat in Indexes

To clear bloat in an index, you need to rebuild it. There are several ways to rebuild an index:

Re-create the index: If you re-create the index, it will be built in an optimal way.
Rebuild the index: Instead of dropping and creating the index yourself, PostgreSQL provides a way to re-build an existing index in-place using the REINDEX command:

REINDEX INDEX index_name;

Rebuild the index concurrently: The previous methods will obtain a lock on the table and prevent it from being changed while the operation is in progress, which is usually unacceptable. To rebuild the index without locking it for updates, you can rebuilt the index concurrently:

REINDEX INDEX CONCURRENTLY index_name;

When using REINDEX CONCURRENTLY, PostgreSQL creates a new index with a name suffixed with _ccnew, and syncs any changes made to the table in the meantime. When the rebuild is done, it will switch the old index with the new index, and drop the old one.

Clearing bloat in Indexes

If for some reason you had to stop the rebuild in the middle, the new index will not be dropped. Instead, it will be left in an invalid state and consume space. To identify invalid indexes that were created during REINDEX, we use the following query:

-- Identify invalid indexes that were created during index rebuild
SELECT
    c.relname as index_name,
    pg_size_pretty(pg_relation_size(c.oid))
FROM
    pg_index i
    JOIN pg_class c ON i.indexrelid = c.oid
WHERE
    -- New index built using REINDEX CONCURRENTLY
    c.relname LIKE  '%_ccnew'
    -- In INVALID state
    AND NOT indisvalid
LIMIT 10;

Once the rebuild process is no longer active, it should be safe to drop any remaining invalid indexes.

Activating B-Tree Index Deduplication

PostgreSQL 13 introduced a new efficient way of storing duplicate values in B-Tree indexes called "B-Tree Deduplication".

For each indexed value, a B-Tree index will hold in its leaf both the value and a pointer to the row (TID). The larger the indexed values, the larger the index. Up until PostgreSQL 12, when the index contained many duplicate values, all of these duplicate values would be stored in the index leaves. This is not very efficient and can take up a lot of space.

B-Tree Index Deduplication

Starting at PostgreSQL 13, when B-Tree deduplication is activated, duplicate values are only stored once. This can make a huge impact on the size of indexes with many duplicate values.

In PostgreSQL 13 index deduplication in enabled by default, unless you deactivate it:

-- Activating de-deduplication for a B-Tree index, this is the default:
CREATE INDEX index_name ON table_name(column_name) WITH (deduplicate_items = ON)

If you are migrating from PostgreSQL versions prior to 13, you need to rebuild the indexes using the REINDEX command in order to get the full benefits of index de-deduplication.

To illustrate the effect of B-Tree deduplication on the size of the index, create a table with a unique column and a non unique column, and populate it with 1M rows. On each column create two B-Tree indexes, one with deduplication enabled and another with deduplication disabled:

db=# CREATE test_btree_dedup (n_unique serial, n_not_unique integer);
CREATE TABLE

db=# INSERT INTO test_btree_dedup (n_not_unique)
SELECT (random() * 100)::int FROM generate_series(1, 1000000);
INSERT 0 1000000

db=# CREATE INDEX ix1 ON test_btree_dedup (n_unique)     WITH (deduplicate_items = OFF);
CREATE INDEX

db=# CREATE INDEX ix2 ON test_btree_dedup (n_unique)     WITH (deduplicate_items = ON);
CREATE INDEX

db=# CREATE INDEX ix3 ON test_btree_dedup (n_not_unique) WITH (deduplicate_items = OFF);
CREATE INDEX

db=# CREATE INDEX ix4 ON test_btree_dedup (n_not_unique) WITH (deduplicate_items = ON);
CREATE INDEX

Next, compare the sizes of the four indexes:

Column	Deduplication	Size
Not unique	Yes	6840 kB
Not unique	No	21 MB
Unique	Yes	21 MB
Unique	No	21 MB

As expected, deduplication had no effect on the unique index, but it had a significant effect on the index that had many duplicate values.

Unfortunately for us, PostgreSQL 13 was still fresh at the time, and our cloud provider did not have support for it yet, so we were unable to use deduplication to clear space.

Clearing Bloat in Tables

Just like in indexes, tables can also contain dead tuples that cause bloat and fragmentation. However, unlike indexes that contain data from an associated table, a table can not just simply be re-created. To re-create a table you would have to create a new table, migrate the data over while keeping it synced with new data, create all the indexes, constraints and any referential constraints in other tables. Only after all of this is done, you can switch the old table with the new one.

Clearing bloat in Tables

There are several ways to rebuild a table and reduce bloat:

Re-create the table: Using this method as described above often requires a lot of development, especially if the table is actively being used as it's being rebuilt.
Vacuum the table: PostgreSQL provides a way to reclaim space occupied by bloat and dead tuples in a table using the VACUUM FULL command. Vacuum full requires a lock on the table, and is not an ideal solution for tables that need to be available while being vacuumed:

-- Will lock the table
VACUUM FULL table_name;

The two options above require either a significant effort, or some down time.

Using pg_repack

Both built-in options for rebuilding tables are not ideal unless you can afford downtime. One popular solution for rebuilding tables and indexes without downtime is the pg_repack extension.

Being a popular extension, pg_repack is likely available from your package manager or already installed by your cloud provider. To use pg_repack, you first need to create the extension:

CREATE EXTENSION pg_repack;

To "repack" a table along with its indexes, issue the following command from the console:

$ pg_repack -k --table table_name db_name

To rebuild a table with no downtime, the extension creates a new table, loads the data from the original table into it while keeping it up to date with new data, and then also rebuilds the indexes. When the process is finished, the two tables are switched and the original table is dropped. See here for full details.

pg_repack on RDS

pg_repack is a supported extensions for PostgreSQL on Amazon RDS. You can find more details on how to use pg_repack on AWS RDS.

There are two caveats to be aware of when using pg_repack to rebuild tables:

Requires amount of storage roughly the amount of the table to rebuild: the extension creates another table to copy the data to, so it requires additional storage roughly the size of the table and its indexes.
May require some manual cleanup: if the "repack" process failed or stopped manually, it may leave intermediate objects laying around, so you may need to do some manual cleanup.

Despite these caveats, pg_repack is a great option for rebuilding tables and indexes with no downtime. However, because it requires some additional storage to operate, it's not a good option when you are already out of storage. It's a good idea to monitor the free storage space and plan rebuilds in advance.

The "Find"

At this point we already used all the conventional techniques we could think of and cleared up a lot of space. We dropped unused indexes and cleared bloat from tables and indexes, but... there was still more space to shave off!

The "Aha Moment"

While we were looking at the sizes of the indexes after we finished rebuilding them, an interesting thing caught our eye.

One of our largest tables stores transaction data. In our system, after a payment is made, the user can choose to cancel and get a refund. This is not happening very often, and only a fraction of the transactions end up being cancelled.

In our transactions table, there are foreign keys to both the purchasing user and the cancelling user, and each field has a B-Tree index defined on it. The purchasing user has a NOT NULL constraint on it so all the rows hold a value. The cancelling user on the other hand, is nullable, and only a fraction of the rows hold any data. Most of the values in the cancelling user field are NULL.

Transactions table with FK to purchasing and cancelling user

We expected the index on the cancelling user to be significantly smaller than the index on the purchasing user, but they were exactly the same. Coming from Oracle, I was always taught that NULLs are not indexed, but in PostgreSQL they are! This "Aha" moment led us to the realization that we were indexing a lot of unnecessary values for no reason.

This was the original index we had for the cancelling user:

CREATE INDEX transaction_cancelled_by_ix ON transactions(cancelled_by_user_id);

To check our thesis, we replaced the index with a partial index that excludes null values:

DROP INDEX transaction_cancelled_by_ix;

CREATE INDEX transaction_cancelled_by_part_ix ON transactions(cancelled_by_user_id)
WHERE cancelled_by_user_id IS NOT NULL;

The full index after we reindexed it was 769MB in size, with more than 99% null values. The partial index that excluded null values was less than 5MB. That's more than 99% percent of dead weight shaved off the index!

Index	Size
Full index	769MB
Partial Index	5MB

Difference	-99%

To make sure those NULL values were indeed unnecessary, we reset the stats on the table and waited a while. Not long after, we observed that the index is being used just like the old one! We just shaved off more than 760MB of unused indexed tuples without compromising performance!

Clearing space, play by play

Utilizing Partial Indexes

Once we had a good experience with one partial index, we figured we might have more indexes like that. To find good candidates for partial index we wrote a query to search for indexes on fields with high null_frac, the percent of values of the column that PostgreSQL estimates are NULL:

-- Find indexed columns with high null_frac
SELECT
    c.oid,
    c.relname AS index,
    pg_size_pretty(pg_relation_size(c.oid)) AS index_size,
    i.indisunique AS unique,
    a.attname AS indexed_column,
    CASE s.null_frac
        WHEN 0 THEN ''
        ELSE to_char(s.null_frac * 100, '999.00%')
    END AS null_frac,
    pg_size_pretty((pg_relation_size(c.oid) * s.null_frac)::bigint) AS expected_saving
    -- Uncomment to include the index definition
    --, ixs.indexdef

FROM
    pg_class c
    JOIN pg_index i ON i.indexrelid = c.oid
    JOIN pg_attribute a ON a.attrelid = c.oid
    JOIN pg_class c_table ON c_table.oid = i.indrelid
    JOIN pg_indexes ixs ON c.relname = ixs.indexname
    LEFT JOIN pg_stats s ON s.tablename = c_table.relname AND a.attname = s.attname

WHERE
    -- Primary key cannot be partial
    NOT i.indisprimary

    -- Exclude already partial indexes
    AND i.indpred IS NULL

    -- Exclude composite indexes
    AND array_length(i.indkey, 1) = 1

    -- Larger than 10MB
    AND pg_relation_size(c.oid) > 10 * 1024 ^ 2

ORDER BY
    pg_relation_size(c.oid) * s.null_frac DESC;

The results of this query can look like this:

   oid   |         index      | index_size | unique | indexed_column | null_frac | expected_saving
---------+--------------------+------------+--------+----------------+-----------+-----------------
  138247 | tx_cancelled_by_ix | 1418 MB    | f      | cancelled_by   |   96.15%  | 1363 MB
   16988 | tx_op1_ix          | 1651 MB    | t      | op1            |    6.11%  | 101 MB
 1473377 | tx_token_ix        | 22 MB      | t      | token          |   11.21%  | 2494 kB
  138529 | tx_op_name_ix      | 1160 MB    | t      | name           |           | 0 bytes

In the table above we can identify several types of results:

tx_cancelled_by_ix is a large index with many null values: great potential here!
tx_op_1_ix is a large index with few null values: there's not much potential
tx_token_ix is a small index with few null values: I wouldn't bother with this index
tx_op_name_ix is a large index with no null values: nothing to do here

The results show that by turning tx_cancelled_by_ix into a partial index that excludes null we can potentially save ~1.3GB.

Is it always beneficial to exclude nulls from indexes?

No. NULL is as meaningful as any other value. If your queries are searching for null values using IS NULL, these queries might benefit from an index on NULL.

So is this method beneficial only for null values?

Using partial indexes to exclude values that are not queried very often or not at all can be beneficial for any value, not just null values. NULL usually indicate a lack of value, and in our case not many queries were searching for null values, so it made sense to exclude them from the index.

So how did you end up clearing more than 20GB?

You may have noticed that the title mentions more than 20GB of free space but the charts only show half, well... indexes are also dropped from replications! When you release 10GB from your primary database, you also release roughly the same amount of storage from each replica.

Bonus: Migrating with Django ORM

This story is taken from a large application built with Django. To put the above techniques to practice with Django, there are several things to note.

Prevent Implicit Creation of Indexes on Foreign Keys

Unless you explicitly set db_index=False, Django will implicitly create a B-Tree index on a models.ForeignKey field. Consider the following example:

from django.db import models
from django.contrib.auth.models import User

class Transaction(models.Model):
    # ...
    cancelled_by_user = models.ForeignKey(
        to=User,
        null=True,
        on_delete=models.CASCADE,
    )

The model is used to keep track of transaction data. If a transaction is cancelled, we keep a reference to user that cancelled it. As previously described, most transactions don't end up being cancelled, so we set null=True on the field.

In the ForeignKey definition above we did not explicitly set db_index, so Django will implicitly create a full index on the field. To create a partial index instead, make the following changes:

from django.db import models
from django.contrib.auth.models import User

class Transaction(models.Model):
    # ...
    cancelled_by_user = models.ForeignKey(
        to=User,
        null=True,
        on_delete=models.CASCADE,
        db_index=False,
    )

    class Meta:
        indexes = (
            models.Index(
                fields=('cancelled_by_user_id', ),
                name='%(class_name)s_cancelled_by_part_ix',
                condition=Q(cancelled_by_user_id__isnull=False),
            ),
        )

We first tell Django not to create the index on the FK field, and then add a partial index using models.Index.

take away

Nullable foreign keys are good candidates for a partial index!

To prevent implicit features such as this one from sneaking indexes without us noticing, we create Django checks to force ourselves to always explicitly set db_index in foreign keys.

Migrate Exiting Full Indexes to Partial Indexes

One of the challenges we were facing during this migration is to replace the existing full indexes with partial indexes without causing downtime or degraded performance during the migration. After we identified the full indexes we want to replace, we took the following steps:

Replace full indexes with partial indexes: Adjust the relevant Django models and replace full indexes with partial indexes, as demonstrated above. The migration Django generates will first disable the FK constraint (if the field is a foreign key), drop the existing full index and create the new partial index. Executing this migration may cause both downtime and degraded performance, so we won't actually run it.
Create the partial indexes manually: Use Django's ./manage.py sqlmigrate utility to produce a script for the migration, extract only the CREATE INDEX statements and adjust them to create the indexes CONCURRENTLY. Then, create the indexes manually and concurrently in the database. Since the full indexes are not dropped yet, they can still be used by queries so performance should not be impacted in the process. It is possible to create indexes concurrently in Django migrations, but this time we decided it's best to do it manually.
Reset full index statistics counters: To make sure it's safe to drop the full indexes, we wanted to first make sure the new partial indexes are being used. To keep track of their use we reset the counters for the full indexes using pg_stat_reset_single_table_counters(<full index oid>).
Monitor use of partial indexes: After resetting the stats we monitored both overall query performance and the partial index usage by observing the values of idx_scan, idx_tup_read and idx_tup_fetch in the pg_stat_all_indexes tables, for both the partial and the full indexes.
Drop the full indexes: Once we were convinced the partial indexes are being used, we dropped the full indexes. This is a good point to check the sizes of both partial and full indexes to find out exactly how much storage you are about to free.
Fake the Django migration: Once the database state was effectively in-sync with the model state, we fake the migration using ./manage.py migrate --fake. When faking a migration, Django will register the migration as executed, but it won't actually execute anything. This is useful for situations like this when you need better control over a migration process. Note that on other environments such as dev, QA or staging where there is not downtime considerations, the Django migrations will execute normally and the full indexes will be replaced with the partial ones. For more advanced Django migration operations such as "fake", check out How to Move a Django Model to Another App.

Conclusion

Optimizing disks, storage parameters and configuration can only affect performance so much. At some point, to squeeze that final drop of performance you need to make changes to the underlying objects. In this case, it was the index definition.

To sum up the process we took to clear an much storage as we could:

Remove unused indexes
Repack tables and indexes (and activate B-Tree deduplication when possible)
Utilize partial indexes to index only what's necessary

Hopefully, after applying these techniques you can gain a few more days before you need to reach into your pocket and provision more storage.

Exhaustiveness Checking with Mypy

2020-12-08T00:00:00+02:00

Mypy is an optional static type checker for Python. It's been around since 2012 and is gaining traction even since. One of the main benefits of using a type checker is getting errors at "compile time" rather than at run time.

Exhaustiveness checking is a common feature of type checkers, and a very useful one! In this article I'm going to show you how you can get mypy to perform exhaustiveness checking!

Playing cards are also useful for explaining enumeration types...
Photo by Daniel Rykhev

Table of Contents

Exhaustiveness Checking
Enumeration types
Type Narrowing in Mypy
The Future
Bonus: Exhaustiveness Checking in Django
Updates

Exhaustiveness Checking

Say you have a system to manage orders. To represent the status of an order, you have the following enum:

import enum

class OrderStatus(enum.Enum):
    Ready = 'ready'
    Shipped = 'shipped'

You also have the following code to process an Order:

def handle_order(status: OrderStatus) -> None:
    if status is OrderStatus.Ready:
        print('ship order')

    elif status is OrderStatus.Shipped:
        print('charge order')

When the order is ready, you ship it; and when it's shipped, you charge it.

A few months go by and your system becomes big. So big in fact, that you can no longer ship orders immediately, and you add a new status:

import enum

class OrderStatus(enum.Enum):
    Ready = 'ready'
    Scheduled = 'scheduled'
    Shipped = 'shipped'

Before you push this change to production, you run a quick check with mypy to make sure everything is OK:

$ mypy main.py
Success: no issues found in 1 source file

Mypy does not see anything wrong in this code, Do you? The problem is that you forgot to handle the new status in your function.

One way to make sure you always handle all possible order statuses is to add an assert, or throw an exception:

def handle_order(status: OrderStatus) -> None:
    if status is OrderStatus.Ready:
        print('ship order')

    elif status is OrderStatus.Shipped:
        print('charge order')

    assert False, f'Unhandled status "{status}"'

Now, when you execute the function with the new status OrderStatus.Scheduled, you will get a runtime error:

>>> handle_order(OrderStatus.Scheduled)
AssertionError: Unhandled status "OrderStatus.Scheduled"

Another way to with deal with cases like this is to go over your test suite and add scenarios in all the places that use order status. But... if you forgot to change the function when you added the status, what are the chances you'll remember to update the tests? That's not a good solution...

Exhaustiveness Checking in Mypy

What if mypy could warn you at "compile time" about such cases? Well... it can, using this little magic function:

from typing import NoReturn
import enum

def assert_never(value: NoReturn) -> NoReturn:
    assert False, f'Unhandled value: {value} ({type(value).__name__})'

Before you dig into the implementation, try to use it to see how it works. In the function above, place assert_never after you handled all the possible order statuses, where you previously used assert or raises an exception:

def handle_order(status: OrderStatus) -> None:
    if status is OrderStatus.Ready:
        print('ship order')

    elif status is OrderStatus.Shipped:
        print('charge order')

    else:
        assert_never(status)

Now, check the code with Mypy:

$ mypy main.py
error: Argument 1 to "assert_never" has incompatible type "Literal[OrderStatus.Scheduled]";
expected "NoReturn"
Found 1 error in 1 file (checked 1 source file)

Amazing! Mypy warns you about a status you forgot to handle! The message also includes the value, OrderStatus.Scheduled. If you use a modern editor such as VSCode you can get these warnings immediately as you type:

mypy Error in VSCode

You can now go ahead and fix your function to handle the missing status:

def handle_order(status: OrderStatus) -> None:
    if status is OrderStatus.Pending:
        print('schedule order')

    elif status is OrderStatus.Scheduled:
        print('ship order')

    elif status is OrderStatus.Shipped:
        print('charge order')

    else:
        assert_never(status)

Check with mypy again:

$ mypy main.py
Success: no issues found in 1 source file

Great! You can now rest assure you handled all order statuses. The best part is that you did that with no unit tests, and there were no runtime errors. If you include mypy in your CI, the bad code will never make it into production.

Enumeration types

In the previous section you used mypy to perform exhaustiveness check on an Enum. You can use mypy, and assert_never to perform exhaustiveness check on other enumeration types as well.

Exhaustiveness Checking of a Union

A Union type represents several possible types. For example, a function that casts an argument to float can look like this:

from typing import Union

def get_float(num: Union[str, float]) -> float:
    if isinstance(num, str):
        return float(num)

    else:
        assert_never(num)

Check the function with mypy:

$ mypy main.py
error: Argument 1 to "assert_never" has incompatible type "float"; expected "NoReturn"

Whoops... you forgot to handle the float type in the code:

from typing import Union

def get_float(num: Union[str, float]) -> float:
    if isinstance(num, str):
        return float(num)

    elif isinstance(num, float):
        return num

    else:
        assert_never(num)

Check again:

$ mypy main.py
Success: no issues found in 1 source file

Great! mypy is happy...

Exhaustiveness Checking of a Literal

Another useful type is Literal. It is included in the built-in typing module since Python3.8, and prior to that it is part of the complementary typing_extensions package.

A Literal is used to type primitive values such as strings and numbers. Literal is also an enumeration type, so you can use exhaustiveness checking on it as well:

from typing_extensions import Literal

Color = Literal['R', 'G', 'B']

def get_color_name(color: Color) -> str:
    if color == 'R':
        return 'Red'
    elif color == 'G':
        return 'Green'
    # elif color == 'B':
    #     return 'Blue'
    else:
        assert_never(color)

Checking the code without the commented part will produce the following error:

$ mypy main.py
error: Argument 1 to "assert_never" has incompatible type "Literal['B']"; expected "NoReturn"

Very handy indeed!

Type Narrowing in Mypy

Now that you've seen what assert_never can do, you can try and understand how it works. assert_never works alongside "type narrowing", which is a mypy feature where the type of a variable is narrowed based on the control flow of the program. In other words, mypy is gradually eliminating possible types for a variable.

First, it's important to understand how various things translate to a Union type in mypy:

Optional[int]
# Equivalent to Union[int, None]

Literal['string', 42, True]
# Equivalent to Union[Literal['string'], Literal[42], Literal[True]]

class Suit(Enum):
    Clubs = "♣"
    Diamonds = "♦"
    Hearts = "♥"
    Spades = "♠"

Suit
# ~Equivalent to Union[
#   Literal[Suit.Clubs],
#   Literal[Suit.Diamonds],
#   Literal[Suit.Hearts],
#   Literal[Suit.Spades]
# ]

To display the type of an expression, mypy provides a useful utility called reveal_type. Using reveal_type you can ask mypy to show you the inferred type for a variable at the point it's called:

def describe_suit(suit: Optional[Suit]) -> str:
    # Revealed type is Union[Suit, None]
    reveal_type(suit)

In the function above, the reveled type of suit is Union[Suit, None], which is the type of the argument suit.

At this point you haven't done anything in the function, so mypy is unable to narrow down the type. Next, add some logic and see how mypy narrows down the type of the variable suit:

def describe_suit(suit: Optional[Suit]) -> str:
    assert suit is not None
    # Revealed type is Suit
    reveal_type(suit)

After eliminating the option of suit being None, the revealed type is Suit. Mypy used your program's logic to narrow the type of the variable.

Keep in mind, the type Suit is equivalent to the type Union[Literal[Suit.Clubs], Literal[Suit.Diamonds], Literal[Suit.Hearts], Literal[Suit.Spades]], so next, try to narrow down the type even more:

def describe_suit(suit: Optional[Suit]) -> str:
    assert suit is not None

    if suit is Suit.Clubs:
        # Revealed type is Literal[Suit.Clubs]
        reveal_type(suit)
        return "Clubs"

    # Revealed type is Literal[Suit.Diamonds, Suit.Hearts, Suit.Spades]
    reveal_type(suit)

After checking if suit is Suit.Clubs, mypy is able to narrow down the type to Suit.Clubs. Mypy is also smart enough to understand that if the condition does not hold, the variable is definitely not Clubs, and narrows down the type to Diamonds, Hearts or Spades.

Mypy can also use other conditional statements to further narrow the type, for example:

def describe_suit(suit: Optional[Suit]) -> str:
    assert suit is not None

    if suit is Suit.Clubs:
        # Revealed type is Literal[Suit.Clubs]
        reveal_type(suit)
        return "Clubs"

    # Revealed type is Literal[Suit.Diamonds, Suit.Hearts, Suit.Spades]
    reveal_type(suit)

    # `and`, `or` and `not` also work.
    if suit is Suit.Diamonds or suit is Suit.Spades:
        # Revealed type is Literal[Suit.Diamonds, Suit.Spades]
        reveal_type(suit)
        return "Diamonds or Spades"

    # Revealed type is Literal[Suit.Hearts]
    reveal_type(suit)

By the end of the function, mypy narrowed down the type of suit to Suit.Hearts. If, for example, you add a condition that imply a different type for suit, mypy will issue an error:

def describe_suit(suit: Optional[Suit]) -> str:
    assert suit is not None

    if suit is Suit.Clubs:
        # Revealed type is Literal[Suit.Clubs]
        reveal_type(suit)
        return "Clubs"

    # Revealed type is Literal[Suit.Diamonds, Suit.Hearts, Suit.Spades]
    reveal_type(suit)

    # `and`, `or` and `not` also work.
    if suit is Suit.Diamonds or suit is Suit.Spades:
        # Revealed type is Literal[Suit.Diamonds, Suit.Spades]
        reveal_type(suit)
        return "Diamonds or Spades"

    # Revealed type is Literal[Suit.Hearts]
    reveal_type(suit)

    # mypy error [comparison-overlap]: Non-overlapping identity check
    # left operand type: "Literal[Suit.Hearts]"
    # right operand type: "Literal[Suit.Diamonds]"
    if suit is Suit.Diamonds:
        # mypy error [unreachable]: Statement is unreachable
        return "Diamonds"

After mypy narrowed down the type of suit to Literal[Suit.Hearts], it knows the next condition suit is Suit.Diamonds will always evaluate to False, and issues an error.

Once all the possibilities have been narrowed-out, the rest of the function becomes unreachable:

def describe_suit(suit: Optional[Suit]) -> str:
    assert suit is not None

    if suit is Suit.Clubs:
        return "Clubs"

    if suit is Suit.Diamonds or suit is Suit.Spades:
        return "Diamonds or Spades"

    if suit == Suit.Hearts:
        return 'Hearts'

    # This is currently unreachable
    assert_never(suit)

assert_never works by taking an argument of type NoReturn, which is only possible when the argument type is "empty". That is, when all possibilities have been narrowed-out and the statement is unreachable. If the statement does become reachable, then the NoReturn is not allowed and mypy issues an error. To illustrate, remove the last condition and check the code with mypy:

def describe_suit(suit: Optional[Suit]) -> str:
    assert suit is not None

    if suit is Suit.Clubs:
        return "Clubs"

    if suit is Suit.Diamonds or suit is Suit.Spades:
        return "Diamonds or Spades"

    # if suit == Suit.Hearts:
    #     return 'Hearts'

    # mypy error: Argument 1 to "assert_never" has
    # incompatible type "Literal[Suit.Hearts]"; expected "NoReturn"
    assert_never(suit)

Mypy narrowed down the type of suit to Suit.Hearts, but assert_never expects NoReturn. This mismatch triggers the error, which effectively performs exhaustiveness checking for suit.

The Future

In 2018 Guido though assert_never is a pretty clever trick, but it never made it into mypy. Instead, exhaustiveness checking will become officially available as part of mypy if/when PEP 622 - Structural Pattern Matching is implemented. Until then, you can use assert_never instead.

UPDATE

Starting at Python 3.11 assert_never was added to the built-in typing module. Python 3.11 was released on Oct 2022, this article was originally published in Dec 2020.

Bonus: Exhaustiveness Checking in Django

Django provides a very useful attribute to most model field types called choices:

from django.db import models
from django.utils.translation import gettext_lazy as _

class Order(models.Model):

    status: str = models.CharField(
        max_length = 20,
        choices = (
            ('ready', _('Ready')),
            ('scheduled', _('Scheduled')),
            ('shipped', _('Shipped')),
        ),
    )

When you provide choices to a field, Django adds all sorts of nice things to it:

Add a validation check to ModelForm (which are used by Django admin, among others)
Render the field as a <select> html element in forms
Add a get_{field}_display_name method to get the description

However, mypy can't know that a Django field with choices has a limited set of values, so it cannot perform exhaustiveness checking on it. To adapt our example from before:

# Will not perform exhaustiveness checking!
def handle_order(order: Order) -> None:
    if order.status == 'ready':
        print('ship order')

    elif order.status == 'shipped':
        print('charge order')

    else:
        assert_never(status)

The function is not handling the status "scheduled", but mypy can't know that.

One way to overcome this is to use an enum to generate the choices:

import enum
from django.db import models

class OrderStatus(enum.Enum):
    Ready = 'ready'
    Scheduled = 'scheduled'
    Shipped = 'shipped'

class Order(models.Model):
    status: str = models.CharField(
        max_length = 20,
        choices = ((e.value, e.name) for e in OrderStatus),
    )

Now, you can achieve exhaustiveness checking with a slight change to the code:

def handle_order(order: Order) -> None:
    status = OrderStatus(order.status)

    if status is OrderStatus.Pending:
        print('ship order')

    elif status is OrderStatus.Shipped:
        print('charge order')

    else:
        assert_never(status)

The tricky part here is that the model field status is actually a string, so to achieve exhaustiveness checking you have to turn the value into an instance of the OrderStatus enum. There are two downsides to this approach:

You have to cast the value every time: This is not very convenient. This can possibly be solved by implementing a custom "Enum field" in Django.
The status descriptions are not translated: Previously you used gettext (_) to translate the enum values, but now you just used the description of the enum.

While the first is still a pain, the second issue was addressed in Django 3.1 with the addition of Django enumeration types:

from django.db import models

class OrderStatus(models.TextChoices):
    Ready = 'ready', _('Ready')
    Scheduled = 'scheduled', _('Scheduled')
    Shipped = 'shipped', _('Shipped')

class Order(models.Model):
    status: str = models.CharField(
        max_length = 20,
        choices = OrderStatus.choices,
    )

Notice how you replaced the enum with a TextChoices. The new enumeration type looks a lot like an Enum (it actually extends Enum under the hood), but it let's you provide a tuple with a value and a description instead of just the value.

Updates

After publishing this article a few readers suggested ways to improve the implementation, so I made the following edits:

2020-12-09: The initial version of the article had assert_never take a value of type NoReturn. A commenter on Lobsters made an excellent suggestion to use the more intuitive Union[()] type instead. This also results in a better error message.
2020-12-09: The initial version of the article used assert False, ... in assert_never instead of raise AssertionError(...). A commenter on Lobsters mentioned that assert statements are removed when python is run with the -O flag. Since the assert in assert_never should not be removed, I changed it to raise AssertionError instead.
2020-12-10: After looking some more, tmcb found that Union[()] is not currently accepted by Python at runtime, so I reverted the argument to NoReturn again.

Stop Using datetime.now!

2020-06-01T00:00:00+03:00

One of my favorite job interview questions is this:

Write a function that returns tomorrow's date

This looks innocent enough for someone to suggest this as a solution:

import datetime

def tomorrow() -> datetime.date:
    return datetime.date.today() + datetime.timedelta(days=1)

This will work, but there is a followup question:

How would you test this function?

Before you move on.... take a second to think about your answer.

One of these pigeons is a mock
Photo by Pedro Figueras

Table of Contents

Naive Approach
Dependency Injection
Dependency Injection in Practice
Conclusion

Naive Approach

The most naive approach to test a function that returns tomorrow's date is this:

# Bad
assert tomorrow() == datetime.date(2020, 4, 16)

This test will pass today, but it will fail on any other day.

Another way to test the function is this:

# Bad
assert tomorrow() == datetime.date.today() + datetime.timedelta(days=1)

This will also work, but there is an inherent problem with this approach. The same way you can't define a word in the dictionary using itself, you should not test a function by repeating its implementation.

Another problem with this approach is that it's only testing one scenario, for the day it is executed. What about getting the next day across a month or a year? What about the day after 2020-02-28?

The problem with both implementations is that today is set inside the function, and to simulate different test scenarios you need to control this value. One solution that comes to mind is to mock datetime.date, and try to set the value returned by today():

>>> from unittest import mock
>>> with mock.patch('datetime.date.today', return_value=datetime.date(2020, 1, 1)):
...     assert tomorrow() == datetime.date(2020, 1, 2)
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.7/unittest/mock.py", line 1410, in __enter__
    setattr(self.target, self.attribute, new_attr)
TypeError: can't set attributes of built-in/extension type 'datetime.date'

As the exception suggests, built-in modules written in C cannot be mocked. The unittest.mock documentation specifically addresses this attempt to mock the datetime module. Apparently, this is a very common issue and the writers of the official documentation felt it's worth mentioning. They even go the extra mile and link to a blog post on this exact problem. The article is worth a read, and we are going to address the solution it presents later on.

Like every other problem in Python, there are libraries that provide a solution. Two libraries that stand out are freezegun and libfaketime. Both provide the ability to mock time at different levels. However, resorting to external libraries is a luxury only developers of legacy system can afford. For new projects, or projects that are small enough to change, there are other alternatives that can keep the project free of these dependencies.

Dependency Injection

The problem we were trying to solve with mock, can also be solved by changing the function's API:

import datetime

def tomorrow(asof: datetime.date) -> datetime.date:
    return asof + datetime.timedelta(days=1)

To control the reference time of the function, the time can be provided as an argument. This makes it easier to test the function in different scenarios:

import datetime
assert tomorrow(asof=datetime.date(2020, 5, 1))   == datetime.date(2020, 5, 2)
assert tomorrow(asof=datetime.date(2019, 12, 31)) == datetime.date(2020, 1, 1)
assert tomorrow(asof=datetime.date(2020, 2, 28))  == datetime.date(2020, 2, 29)
assert tomorrow(asof=datetime.date(2021, 2, 28))  == datetime.date(2021, 3, 1)

To remove the function's dependency on datetime.date.today, we provide today's date as an argument. This pattern of providing, or "injecting" dependencies into functions and objects is often called "dependency injection", or in short "DI".

Dependency Injection in The Wild

Dependency injection is a way to decouple modules from each other. As our previous example shows, the function tomorrow no longer depends on today.

Using dependency injection is very common and often very intuitive. It's very likely that you already use it without even knowing. For example, this article suggests that providing an open file to json.load is a form of dependency injection:

import json

with open('path/to/file.json', 'r') as f:
  data = json.load(f)

The popular test framework pytest builds its entire fixture infrastructure around the concept of dependency injection:

import pytest

@pytest.fixture
def one() -> int:
  return 1

@pytest.fixture
def two() -> int:
  return 2

def test_one_is_less_than_two(one: int, two: int) -> None:
  assert one < two

The functions one and two are declared as fixtures. When pytest executes the test function test_one_is_less_than_two, it will provide it with the values returned by the fixture functions matching the attribute names. In pytest, the injection is magically happening simply by using the name of a known fixture as an argument.

Dependency injection is not limited just to Python. The popular JavaScript framework Angular is also built around dependency injection:

@Component({
  selector: 'order-list',
  template: `...`
})
export class OrderListComponent {
  orders: Order[];

  constructor(orderService: OrderService) {
    this.orders = orderService.getOrders();
  }
}

Notice how the orderService is provided, or injected, to the constructor. The component is using the order service, but is not instantiating it.

Injecting Functions

Sometimes injecting a value is not enough. For example, what if we need to get the current date before and after some operation:

from typing import Tuple
import datetime

def go() -> Tuple[datetime.datetime, datetime.datetime]:
    started_at = datetime.datetime.now()
    # Do something ...
    ended_at = datetime.datetime.now()
    return started_at, ended_at

To test this function, we can provide the start time like we did before, but we can't provide the end time. One way to approach this is to make the calls to start and end outside the function. This is a valid solution, but for the sake of discussion we'll assume they need to be called inside.

Since we can't mock datetime.datetime itself, one way to make this function testable is to create a separate function that returns the current date:

from typing import Tuple
import datetime

def now() -> datetime.datetime:
  return datetime.datetime.now()

def go() -> Tuple[datetime.datetime, datetime.datetime]:
    started_at = now()
    # Do something ...
    ended_at = now()
    return started_at, ended_at

To control the values returned by the function now in tests, we can use a mock:

>>> from unittest import mock
>>> fake_start = datetime.datetime(2020, 1, 1, 15, 0, 0)
>>> fake_end = datetime.datetime(2020, 1, 1, 15, 1, 30)
>>> with mock('__main__.now', side_effect=[fake_start, fake_end]):
...    go()
(datetime.datetime(2020, 1, 1, 15, 0),
 datetime.datetime(2020, 1, 1, 15, 1, 30))

Another way to approach this without mocking, is to rewrite the function once again:

from typing import Callable, Tuple
import datetime

def go(
    now: Callable[[], datetime.datetime],
) -> Tuple[datetime.datetime, datetime.datetime]:
    started_at = now()
    # Do something ...
    ended_at = now()
    return started_at, ended_at

This time we provide the function with another function that returns a datetime. This is very similar to the first solution we suggested, when we injected the datetime itself to the function.

The function can now be used like this:

>>> go(datetime.datetime.now)
(datetime.datetime(2020, 4, 18, 14, 14, 5, 687471),
 datetime.datetime(2020, 4, 18, 14, 14, 5, 687475))

To test it, we provide a different function that returns known datetimes:

>>> fake_start = datetime.datetime(2020, 1, 1, 15, 0, 0)
>>> fake_end = datetime.datetime(2020, 1, 1, 15, 1, 30)
>>> gen = iter([fake_start, fake_end])
>>> go(lambda: next(gen))
(datetime.datetime(2020, 1, 1, 15, 0),
 datetime.datetime(2020, 1, 1, 15, 1, 30))

This pattern can be generalized even more using a utility object:

from typing import Iterator
import datetime

def ticker(
    start: datetime.datetime,
    interval: datetime.timedelta,
) -> Iterator[datetime.datetime]:
    """Generate an unending stream of datetimes in fixed intervals.

    Useful to test processes which require datetime for each step.
    """
    current = start
    while True:
        yield current
        current += interval

Using ticker, the test will now look like this:

>>> gen = ticker(datetime.datetime(2020, 1, 1, 15, 0, 0), datetime.timedelta(seconds=90))
>>> go(lambda: next(gen)))
(datetime.datetime(2020, 1, 1, 15, 0),
 datetime.datetime(2020, 1, 1, 15, 1, 30))

Fun fact: the name "ticker" was stolen from Go.

Injecting Values

The previous sections demonstrate injection of both values and functions. It's clear from the examples that injecting values is much simpler. This is why it's usually favorable to inject values rather than functions.

Another reason is consistency. Take this common pattern that is often used in Django models:

from django.db import models

class Order(models.Model):
    created = models.DateTimeField(auto_now_add=True)
    modified = models.DateTimeField(auto_now=True)

The model Order includes two datetime fields, created and modified. It uses Django's auto_now_add attribute to automatically set created when the object is saved for the first time, and auto_now to set modified every time the object is saved.

Say we create a new order and save it to the database:

>>> o = Order.objects.create()

Would you expect this test to fail:

>>> assert o.created == o.modified
False

This is very unexpected. How can an object that was just created have two different values for created and modified? Can you imagine what would happen if you rely on modified and created to be equal when an object was never changed, and actually use it to identify unchanged objects:

from django.db.models import F

# Wrong!
def get_unchanged_objects():
  return Order.objects.filter(created=F('modified'))

For the Order model above, this function will always return an empty queryset.

The reason for this unexpected behavior is that each individual DateTimeField is using django.timezone.now internally during save() to get the current time. The time between when the two fields are populated by Django causes the values to end up slightly different:

>>> o.created
datetime.datetime(2020, 4, 18, 11, 41, 35, 740909, tzinfo=<UTC>)

>>> o.modified
datetime.datetime(2020, 4, 18, 11, 41, 35, 741015, tzinfo=<UTC>)

If we treat timezone.now like an injected function, we understand the inconsistencies it may cause.

So, can this be avoided? Can created and modified be equal when the object is first created? I'm sure there are a lot of hacks, libraries and other such exotic solutions but the truth is much simpler. If you want to make sure these two fields are equal when the object is first created, you better avoid auto_now and auto_now_add:

from django.db import models

class Order(models.Model):
    created = models.DateTimeField()
    modified = models.DateTimeField()

Then, when you create a new instance, explicitly provide the values for both fields:

>>> from django.utils import timezone
>>> asof = timezone.now()
>>> o = Order.objects.create(created=asof, modified=asof)
>>> assert o.created == o.modified
>>> Order.objects.filter(created=F('modified'))
<QuerySet [<Order: Order object (2)>]>

To quote the "Zen of Python", explicit is better than implicit. Explicitly providing the values for the fields requires a bit more work, but this is a small price to pay for reliable and predictable data.

using auto_now and auto_now_add

When is it OK to use auto_now and auto_now_add? Usually when a date is used for audit purposes and not for business logic, it's fine to make this shortcut and use auto_now or auto_now_add.

When to Instantiate Injected Values

Injecting values poses another interesting question, at what point should the value be set? The answer to this is "it depends", but there is a rule of thumb that is usually correct: values should be instantiated at the topmost level.

For example, if asof represents when an order is created, a website backend serving a store front may set this value when the request is received. In a normal Django setup, this means that the value should be set by the view. Another common example is a scheduled job. If you have jobs that use management commands, asof should be set by the management command.

Setting the values at the topmost level guarantees that the lower levels remain decoupled and easier to test. The level at which injected values are set, is the level that you will usually need to use mock to test. In the example above, setting asof in the view will make the models easier to test.

Other than testing and correctness, another benefit of setting values explicitly rather than implicitly, is that it gives you more control over your data. For example, in the website scenario, an order's creation date is set by the view immediately when the request is received. However, if you process a batch file from a large customer, the time in which the order was created may well be in the past, when the customer first created the files. By avoiding "auto-magically" generated dates, we can implement this by passing the past date as an argument.

Dependency Injection in Practice

The best way to understand the benefits of DI and the motivation for it is using a real life example.

IP Lookup

Say we want to try and guess where visitors to our Django site are coming from, and we decide to try an use the IP address from the request to do that. An initial implementation can look like this:

from typing import Optional
from django.http import HttpRequest
import requests

def get_country_from_request(request: HttpRequest) -> Optional[str]:
    ip = request.META.get('REMOTE_ADDR', request.META.get('HTTP_X_FORWARDED_FOR'))
    if ip is None or ip == '':
        return None

    response = requests.get(f'https://ip-api.com/json/{ip}')
    if not response.ok:
        return None

    data = response.json()
    if data['status'] != 'success':
        return None

    return data['countryCode']

This single function accepts an HttpRequest, tries to extract an IP address from the request headers, and then uses the requests library to call an external service to get the country code.

ip lookup

I'm using the free service https://ip-api.com to lookup a country from an IP. I'm using this service just for demonstration purposes. I'm not familiar with it, so don't see this as a recommendation to use it.

Let's try to use this function:

>>> from django.test import RequestFactory
>>> rf = RequestFactory()
>>> request = rf.get('/', REMOTE_ADDR='216.58.210.46')
>>> get_country_from_request(request)
'US'

OK, so it works. Notice that to use it we created an HttpRequest object using Django's RequestFactory

Let's try to write a test for a scenario when a country code is found:

import re
import json
import responses

from django.test import RequestFactory

rf = RequestFactory()

with responses.RequestsMock() as rsps:
    url_pattern = re.compile(r'http://ip-api.com/json/[0-9\.]+')
    rsps.add(responses.GET, url_pattern, status=200, content_type='application/json', body=json.dumps({
        'status': 'success',
        'countryCode': 'US'
    }))
    request = rf.get('/', REMOTE_ADDR='216.58.210.46')
    countryCode = get_country_from_request(request)
    assert countryCode == 'US'

The function is using the requests library internally to make a request to the external API. To mock the response, we used the responses library.

If you look at this test and feel like it's very complicated than you are right. To test the function we had to do the following:

Generate a Django request using a RequestFactory.
Mock a requests response using responses.
Have knowledge of the inner works of the function (what url it uses).

That last point is where it gets hairy. To test the function we used our knowledge of how the function is implemented: what endpoint it uses, how the URL is structured, what method it uses and what the response looks like. This creates an implicit dependency between the test and the implementation. In other words, the implementation of the function cannot change without changing the test as well. This type of unhealthy dependency is both unexpected, and prevents us from treating the function as a "black box".

Also, notice that that we only tested one scenario. If you look at the coverage of this test you'll find that it's very low. So next, we try and simplify this function.

Assigning Responsibility

One of the techniques to make functions easier to test is to remove dependencies. Our IP function currently depends on Django's HttpRequest, the requests library and implicitly on the external service. Let's start by moving the part of the function that handles the external service to a separate function:

def get_country_from_ip(ip: str) -> Optional[str]:
    response = requests.get(f'http://ip-api.com/json/{ip}')
    if not response.ok:
        return None

    data = response.json()
    if data['status'] != 'success':
        return None

    return data['countryCode']

def get_country_from_request(request: HttpRequest) -> Optional[str]:
    ip = request.META.get('REMOTE_ADDR', request.META.get('HTTP_X_FORWARDED_FOR'))
    if ip is None or ip == '':
        return None

    return get_country_from_ip(ip)

We now have two functions:

get_country_from_ip: receives an IP address and returns the country code.
get_country_from_request: accepts a Django HttpRequest, extract the IP from the header, and then uses the first function to find the country code.

After splitting the function we can now search an IP directly, without crating a request:

>>> get_country_from_ip('216.58.210.46')
'US'
>>> from django.test import RequestFactory
>>> request = RequestFactory().get('/', REMOTE_ADDR='216.58.210.46')
>>> get_country_from_request(request)
'US'

Now, let's write a test for this function:

import re
import json
import responses

with responses.RequestsMock() as rsps:
    url_pattern = re.compile(r'http://ip-api.com/json/[0-9\.]+')
    rsps.add(responses.GET, url_pattern, status=200, content_type='application/json', body=json.dumps({
        'status': 'success',
        'countryCode': 'US'
    }))
    country_code = get_country_from_ip('216.58.210.46')
    assert country_code == 'US'

This test looks similar to the previous one, but we no longer need to use RequestFactory. Because we have a separate function that retrieves the country code for an IP directly, we don't need to "fake" a Django HttpRequest.

Having said that, we still want to make sure the top level function works, and that the IP is being extracted from the request correctly:

# BAD EXAMPLE!
import re
import json
import responses

from django.test import RequestFactory

rf = RequestFactory()
request_with_no_ip = rf.get('/')
country_code = get_country_from_request(request_with_no_ip)
assert country_code is None

We created a request with no IP and the function returned None. With this outcome, can we really say for sure that the function works as expected? Can we tell that the function returned None because it couldn't extract the IP from the request, or because the country lookup returned nothing?

Someone once told me that if to describe what a function does you need to use the words "and" or "or", you can probably benefit from splitting it. This is the layman's version of the Single-responsibility principle that dictates that every class or function should have just one reason to change.

The function get_country_from_request extracts the IP from a request and tries to find the country code for it. So, if the rule is correct, we need to split it up:

def get_ip_from_request(request: HttpRequest) -> Optional[str]:
    ip = request.META.get('REMOTE_ADDR', request.META.get('HTTP_X_FORWARDED_FOR'))
    if ip is None or ip == '':
        return None
    return ip


# Maintain backward compatibility
def get_country_from_request(request: HttpRequest) -> Optional[str]:
    ip = get_ip_from_request(request)
    if ip is None:
        return None
    return get_country_from_ip(ip)

To be able to test if we extract an IP from a request correctly, we yanked this part to a separate function. We can now test this function separately:

rf = RequestFactory()
assert get_ip_from_request(rf.get('/')) is None
assert get_ip_from_request(rf.get('/', REMOTE_ADDR='0.0.0.0')) == '0.0.0.0'
assert get_ip_from_request(rf.get('/', HTTP_X_FORWARDED_FOR='0.0.0.0')) == '0.0.0.0'
assert get_ip_from_request(rf.get('/', REMOTE_ADDR='0.0.0.0', HTTP_X_FORWARDED_FOR='1.1.1.1')) =='0.0.0.0'

With just these 5 lines of code we covered a lot more possible scenarios.

Using a Service

So far we've implemented unit tests for the function that extracts the IP from the request, and made it possible to do a country lookup using just an IP address. The tests for the top level function are still very messy. Because we use requests inside the function, we were forced to use responses as well to test it. There is nothing wrong with responses, but the less dependencies the better.

Invoking a request inside the function creates an implicit dependency between this function and the requests library. One way to eliminate this dependency is to extract the part making the request to a separate service:

import requests

class IpLookupService:

    def __init__(self, base_url: str) -> None:
        self.base_url = base_url

    def get_country_from_ip(self, ip: str) -> Optional[str]:
        response = requests.get(f'{self.base_url}/json/{ip}')
        if not response.ok:
            return None

        data = response.json()
        if data['status'] != 'success':
            return None

        return data['countryCode']

The new IpLookupService is instantiated with the base url for the service, and provides a single function to get a country from an IP:

>>> ip_lookup_service = IpLookupService('http://ip-api.com')
>>> ip_lookup_service.get_country_from_ip('216.58.210.46')
'US'

Constructing services this way has many benefits:

Encapsulate all the logic related to IP lookup
Provides a single interface with type annotations
Can be reused
Can be tested separately
Can be developed separately (as long as the API it provides remains unchanged)
Can be adjusted for different environments (for example, use a different URL for test and production)

The top level function should also change. Instead of making requests on its own, it uses the service:

def get_country_from_request(
    request: HttpRequest,
    ip_lookup_service: IpLookupService,
) -> Optional[str]:
    ip = get_ip_from_request(request)
    if ip is None:
        return None
    return ip_lookup_service.get_country_from_ip(ip)

To use the function, we pass an instance of the service to it:

>>> ip_lookup_service = IpLookupService('http://ip-api.com')
>>> request = RequestFactory().get('/', REMOTE_ADDR='216.58.210.46')
>>> get_country_from_request(request, ip_lookup_service)
'US'

Now that we have full control of the service, we can test the top level function without using responses:

from unittest import mock
from django.test import RequestFactory

fake_ip_lookup_service = mock.create_autospec(IpLookupService)
fake_ip_lookup_service.get_country_from_ip.return_value = 'US'

request = RequestFactory().get('/', REMOTE_ADDR='216.58.210.46')

country_code = get_country_from_request(request, fake_ip_lookup_service)
assert country_code == 'US'

To test the function without actually making http requests we created a mock of the service. We then set the return value of get_country_from_ip, and passed the mock service to the function.

Changing Implementations

Another benefit of DI which is often mentioned, is the ability to completely change the underlying implementation of an injected service. For example, one day you discover that you don't have to use a remote service to lookup an IP. Instead, you can use a local IP database.

Because our IpLookupService does not leak its internal implementation, it's an easy switch:

from typing import Optional
import GeoIP

class LocalIpLookupService:
    def __init__(self, path_to_db_file: str) -> None:
        self.db = GeoIP.open(path_to_db_file, GeoIP.GEOIP_STANDARD)

    def get_country_from_ip(self, ip: str) -> Optional[str]:
        return self.db.country_code_by_addr(ip)

The service API remained unchanged, so you can use it the same way as the old service:

>>> ip_lookup_service = LocalIpLookupService('/usr/share/GeoIP/GeoIP.dat')
>>> ip_lookup_service.get_country_from_ip('216.58.210.46')
'US'
>>> from django.test import RequestFactory
>>> request = RequestFactory().get('/', REMOTE_ADDR='216.58.210.46')
>>> get_country_from_request(request, ip_lookup_service)
'US'

The best part here is that the tests are unaffected. All the tests should pass without making any changes.

GeoIP

In the example I use the MaxMind GeoIP Legacy Python Extension API because it uses files I already have in my OS as part of geoiplookup. If you really need to lookup IP addresses check out GeoIP2 and make sure to check the license and usage restrictions.

Also, Django users might be delighted to know that Django provides a wrapper around geoip2.

Typing Services

In the last section we cheated a bit. We injected the new service LocalIpLookupService into a function that expects an instance of IpLookupService. We made sure that these two are the same, but the type annotations are now wrong. We also used a mock to test the function which is also not of type IpLookupService. So, how can we use type annotations and still be able to inject different services?

from abc import ABCMeta
import GeoIP
import requests

class IpLookupService(metaclass=ABCMeta):
    def get_country_from_ip(self, ip: str) -> Optional[str]:
        raise NotImplementedError()


class RemoteIpLookupService(IpLookupService):
    def __init__(self, base_url: str) -> None:
        self.base_url = base_url

    def get_country_from_ip(self, ip: str) -> Optional[str]:
        response = requests.get(f'{self.base_url}/json/{ip}')
        if not response.ok:
            return None

        data = response.json()
        if data['status'] != 'success':
            return None

        return data['countryCode']


class LocalIpLookupService(IpLookupService):
    def __init__(self, path_to_db_file: str) -> None:
        self.db = GeoIP.open(path_to_db_file, GeoIP.GEOIP_STANDARD)

    def get_country_from_ip(self, ip: str) -> Optional[str]:
        return self.db.country_code_by_addr(ip)

We defined a base class called IpLookupService that acts as an interface. The base class defines the public API for users of IpLookupService. Using the base class, we can provide two implementations:

RemoteIpLookupService: uses the requests library to lookup the IP at an external.
LocalIpLookupService: uses the local GeoIP database.

Now, any function that needs an instance of IpLookupService can use this type, and the function will be able to accept any subclass of it.

Before we wrap things up, we still need to handle the tests. Previously we removed the test's dependency on responses, now we can ditch mock as well. Instead, we subclass IpLookupService with a simple implementation for testing:

from typing import Iterable

class FakeIpLookupService(IpLookupService):
    def __init__(self, results: Iterable[Optional[str]]):
        self.results = iter(results)

    def get_country_from_ip(self, ip: str) -> Optional[str]:
        return next(self.results)

The FakeIpLookupService implements IpLookupService, and is producing results from a list of predefined results we provide to it:

from django.test import RequestFactory

fake_ip_lookup_service = FakeIpLookupService(results=['US'])
request = RequestFactory().get('/', REMOTE_ADDR='216.58.210.46')

country_code = get_country_from_request(request, fake_ip_lookup_service)
assert country_code == 'US'

The test no longer uses mock.

Using a Protocol

The form of class hierarchy demonstrated in the previous section is called "nominal subtyping". There is another way to utilize typing without classes, using Protocols:

from typing import Iterable, Optional
from typing_extensions import Protocol
import GeoIP
import requests


class IpLookupService(Protocol):
    def get_country_from_ip(self, ip: str) -> Optional[str]:
        pass


class RemoteIpLookupService:
    def __init__(self, base_url: str) -> None:
        self.base_url = base_url

    def get_country_from_ip(self, ip: str) -> Optional[str]:
        response = requests.get(f'{self.base_url}/json/{ip}')
        if not response.ok:
            return None

        data = response.json()
        if data['status'] != 'success':
            return None

        return data['countryCode']


class LocalIpLookupService:
    def __init__(self, path_to_db_file: str) -> None:
        self.db = GeoIP.open(path_to_db_file, GeoIP.GEOIP_STANDARD)

    def get_country_from_ip(self, ip: str) -> Optional[str]:
        return self.db.country_code_by_addr(ip)


class FakeIpLookupService:
    def __init__(self, results: Iterable[Optional[str]]):
        self.results = iter(results)

    def get_country_from_ip(self, ip: str) -> Optional[str]:
        yield from self.results

The switch from classes to protocols is mild. Instead of creating IpLookupService as a base class, we declare it a Protocol. A protocol is used to define an interface and cannot be instantiated. Instead, a protocol is used only for typing purposes. When a class implements the interface defined by the protocol, is means "Structural Subtyping" exits and the type check will validate.

In our case, we use a protocol to make sure an argument of type IpLookupService implements the functions we expect an IP service to provide.

structural and nominal subtyping

I've written about protocols, structural and nominal subtyping to in the past. Check out Modeling Polymorphism in Django With Python.

So which to use? Some languages, like Java, use nominal typing exclusively, while other languages, like Go, use structural typing for interfaces. There are advantages and disadvantages to both ways, but we won't get into that here. In Python, nominal typing is easier to use and understand, so my recommendation is to stick to it, unless you need the flexibility afforded by protocols.

Nondeterminism and Side-Effects

If you ever had a test that one day just started to fail, unprovoked, or a test that fails once every blue moon for no apparent reason, it's possible your code is relying on something that is not deterministic. In the datetime.date.today example, the result of datetime.date.today relies on the current time which is always changing, hence it's not deterministic.

There are many sources of nondeterminism. Common examples include:

Randomness
Network access
Filesystem access
Database access
Environment variables
Mutable global variables

Dependency injection provides a good way to control nondeterminism in tests. The basic recipe is this:

Identify the source of nondeterminism and encapsulate it in a service: For example, TimeService, RandomnessService, HttpService, FilesystemService and DatabaseService.
Use dependency injection to access these services: Never bypass them by using datetime.now() and similar directly.
Provide deterministic implementations of these services in tests: Use a mock, or a custom implementation suited for tests instead.

If you follow the recipe diligently, your tests will not be affected by external circumstances and you will not have flaky tests!

Conclusion

Dependency injection is a design pattern just like any other. Developers can decide to what degree they want to take advantage of it. The main benefits of DI are:

Decouple modules, functions and objects.
Switch implementations, or support several different implementations.
Eliminate nondeterminism from tests.

In the use-case above we took several twists and turns to illustrate a point, which might have caused the implementation to seem more complicated than it really is. In addition to that, searching for information about dependency injection in Python often result in libraries and packages than seem to completely change the way you structure your application. This can be very intimidating.

In reality, DI can be used sparingly and in appropriate places to achieve the benefits listed above. When implemented correctly, DI can make your code easier to maintain and to test.

How to Move a Django Model to Another App

2020-05-06T00:00:00+03:00

In my latest article for RealPython I share three ways to tackle one of the most challenging tasks involving Django migrations: moving a model from one Django app to another.

The article covers some exotic migration operations and many of the built-in migration CLI commands such sqlmigrate, showmigrations and sqlsequencereset. In the article I also demonstrate important migrations concepts such as reversible migrations, migration plans and introspection.

Read "How to Move a Django Model to Another App" on RealPython ≫

How to Move a Django Model to Another App

Testing an Interactive Voice Response System With Python and Pytest

2020-05-01T00:00:00+03:00

Following my previous article on how to build an Interactive Voice Response (IVR) system with Twilio, Python and Django, in this follow-up tutorial I show how to write automated tests for this system.

It can be very challenging to test a system that rely heavily on a third party service such as Twilio. In this article, I show how to organize your code in a way that would isolate your business logic and make it easier to test separately. The article demonstrate useful testing patterns using Django's RequestFactory, unittest.mock, Pytest fixtures, build-in django-pytest and many more.

Source Code

The source code for this article and the previous one can be found here.

Read "Testing a Twilio Interactive Voice Response System With Python and Pytest" on the Twilio blog ≫

Building an IVR System with Django and Twilio

How to Provide Test Fixtures for Django Models in Pytest

2020-04-08T00:00:00+03:00

One of the most challenging aspects of writing good tests is maintaining test fixtures. Good test fixtures motivate developers to write better tests, and bad fixtures can cripple a system to a point where developers fear and avoid them all together. The key to maintaining good fixtures is to find a good balance between flexibility and usability. Good fixtures are ones that are easy to use and easy to modify.

In my latest article for RealPython I share some insights on how to maintain good test fixtures for Django models using Pytest. The article covers everything from setting up Pytest for a Django project, creating test fixtures and how to create dependency between fixtures.

The article focuses on a pattern called "factory as a service". Using this pattern, you can create fixture for Django models that depend on other fixtures. This makes it easier to set up data for tests and focus on the the scenario at hand rather than setting up the data.

Read "How to Provide Test Fixtures for Django Models in Pytest" on RealPython ≫

How to Provide Test Fixtures for Django Models in Pytest

Using Markdown in Django

2020-03-30T00:00:00+03:00

As developers, we rely on static analysis tools to check, lint and transform our code. We use these tools to help us be more productive and produce better code. However, when we write content using markdown the tools at our disposal are scarce.

In this article we describe how we developed a Markdown extension to address challenges in managing content using Markdown in Django sites.

Do you think they had a linter?
Photo by mali maeder from Pexels

Table of Contents

The Problem
- Prior Work
Using Markdown
Validate and Transform Django Links
Conclusion
- Taking it Further

The Problem

Like every website, we have different types of (mostly) static content in places like our home page, FAQ section and "About" page. For a very long time, we managed all of this content directly in Django templates.

When we finally decided it's time to move this content out of templates and into the database, we thought it's best to use Markdown. It's safer to produce HTML from Markdown, it provides a certain level of control and uniformity, and is easier for non-technical users to handle. As we progressed with the move, we noticed we are missing a few things:

Internal Links

Links to internal pages can get broken when the URL changes. In Django templates and views we use reverse and {% url %}, but this is not available in plain Markdown.

Copy Between Environments

Absolute internal links cannot be copied between environments. This can be resolved using relative links, but there is no way to enforce this out of the box.

Invalid Links

Invalid links can harm user experience and cause the user to question the reliability of the entire content. This is not something that is unique to Markdown, but HTML templates are maintained by developers who know a thing or two about URLs. Markdown documents on the other hand, are intended for non-technical writers.

Prior Work

When I was researching this issue I searched for Python linters, Markdown preprocessor and extensions to help produce better Markdown. I found very few results. One approach that stood out was to use Django templates to produce Markdown documents.

Preprocess Markdown using Django Template

Using Django templates, you can use template tags such as url to reverse URL names, as well as conditions, variables, date formats and all the other Django template features. This approach essentially uses Django template as a preprocessor for Markdown documents.

I personally felt like this may no be the best solution for non-technical writers. In addition, I was worried that providing access to Django template tags might be dangerous.

Using Markdown

With a better understanding of the problem, we were ready to dig a bit deeper into Markdown in Python.

Converting Markdown to HTML

To start using Markdown in Python, install the markdown package:

$ pip install markdown
Collecting markdown
Installing collected packages: markdown
Successfully installed markdown-3.2.1

Next, create a Markdown object and use the function convert to turn some Markdown into HTML:

>>> import markdown
>>> md = markdown.Markdown()
>>> md.convert("My name is **Haki**")
<p>My name is <strong>Haki</strong></p>

You can now use this HTML snippet in your template.

Using Markdown Extensions

The basic Markdown processor provides the essentials for producing HTML content. For more "exotic" options, the Python markdown package includes some built-in extensions. A popular extension is the "extra" extension that adds, among other things, support for fenced code blocks:

>>> import markdown
>>> md = markdown.Markdown(extensions=['extra'])
>>> md.convert("""```python
... print('this is Python code!')
... ```""")
<pre><code class="python">print(\'this is Python code!\')\n</code></pre>

To extend Markdown with our unique Django capabilities, we are going to develop an extension of our own.

Creating a Markdown Extension to Process Inline Links

If you look at the source, you'll see that to convert markdown to HTML, Markdown uses different processors. One type of processor is an inline processor. Inline processors match specific inline patterns such as links, backticks, bold text and underlined text, and converts them to HTML.

The main purpose of our Markdown extension is to validate and transform links. So, the inline processor we are most interested in is the LinkInlineProcessor. This processor takes markdown in the form of [Haki's website](https://hakibenita.com), parses it and returns a tuple containing the link and the text.

To extend the functionality, we extend LinkInlineProcessor and create a Markdown.Extension that uses it to handle links:

import markdown
from markdown.inlinepatterns import LinkInlineProcessor, LINK_RE


def get_site_domain() -> str:
    # TODO: Get your site domain here
    return 'example.com'


def clean_link(href: str, site_domain: str) -> str:
    # TODO: This is where the magic happens!
    return href


class DjangoLinkInlineProcessor(LinkInlineProcessor):
    def getLink(self, data, index):
        href, title, index, handled = super().getLink(data, index)
        site_domain = get_site_domain()
        href = clean_link(href, site_domain)
        return href, title, index, handled


class DjangoUrlExtension(markdown.Extension):
    def extendMarkdown(self, md, *args, **kwrags):
        md.inlinePatterns.register(DjangoLinkInlineProcessor(LINK_RE, md), 'link', 160)

Let's break it down:

The extension DjangoUrlExtension registers an inline link processor called DjangoLinkInlineProcessor. This processor will replace any other existing link processor.
The inline processor DjangoLinkInlineProcessor extends the built-in LinkInlineProcessor, and calls the function clean_link on every link it processes.
The function clean_link receives a link and a domain, and returns a transformed link. This is where we are going to plug in our implementation.

How to get the site domain

To identify links to your own site you must know the domain of your site. If you are using Django's sites framework you can use it to get the current domain.

I did not include this in my implementation because we don't use the sites framework. Instead, we set a variable in Django settings.

Another way to get the current domain is from an HttpRequest object. If content is only edited in your own site, you can try to plug the site domain from the request object. This may require some changes to the implementation.

To use the extension, add it when you initialize a new Markdown instance:

>>> md = markdown.Markdown(extensions=[DjangoUrlExtension()])
>>> md.convert("[haki's site](https://hakibenita.com)")
<p><a href="https://hakibenita.com">haki\'s site</a></p>

Great, the extension is being used and we are ready for the interesting part!

Validate and Transform Django Links

Now that we got the extension to call clean_link on all links, we can implement our validation and transformation logic.

Validating `mailto` Links

To get the ball rolling, we'll start with a simple validation. mailto links are useful for opening the user's email client with a predefined recipient address, subject and even message body.

A common mailto link can look like this:

<a href="mailto:support@service.com?subject=I need help!">Help!</a>

This link will open your email client set to compose a new email to "support@service.com" with subject line "I need help!".

mailto links do not have to include an email address. If you look at the "share" buttons at the bottom of this article, you'll find a mailto link that looks like this:

<a
  href="mailto:?subject=Django Markdown by Haki Benita&body=http://hakibenita.com/django-markdown"
  title="Email">
  Share via Email
</a>

This mailto link does not include a recipient, just a subject line and message body.

Now that we have a good understanding of what mailto links look like, we can add the first validation to the clean_link function:

from typing import Optional
import re

from django.core.exceptions import ValidationError
from django.core.validators import EmailValidator


class Error(Exception):
    pass


class InvalidMarkdown(Error):
    def __init__(self, error: str, value: Optional[str] = None) -> None:
        self.error = error
        self.value = value

    def __str__(self) -> str:
        if self.value is None:
            return self.error
        return f'{self.error} "{self.value}"';


def clean_link(href: str, site_domain: str) -> str:
    if href.startswith('mailto:'):
        email_match = re.match('^(mailto:)?([^?]*)', href)
        if not email_match:
            raise InvalidMarkdown('Invalid mailto link', value=href)

        email = email_match.group(2)
        if email:
            try:
                EmailValidator()(email)
            except ValidationError:
                raise InvalidMarkdown('Invalid email address', value=email)

        return href

    # More validations to come...

    return href

To validate a mailto link we added the following code to clean_link:

Check if the link starts with mailto: to identify relevant links.
Split the link to its components using a regular expression.
Yank the actual email address from the mailto link, and validate it using Django's EmailValidator.

Notice that we also added a new type of exception called InvalidMarkdown. We defined our own custom Exception type to distinguish it from other errors raised by markdown itself.

Custom error class

I wrote about custom error classes in the past, why they are useful and when you should use them.

Before we move on, let's add some tests and see this in action:

>>> md = markdown.Markdown(extensions=[DjangoUrlExtension()])
>>> md.convert("[Help](mailto:support@service.com?subject=I need help!)")
'<p><a href="mailto:support@service.com?subject=I need help!">Help</a></p>'

>>> md.convert("[Help](mailto:?subject=I need help!)")
<p><a href="mailto:?subject=I need help!">Help</a></p>

>>> md.convert("[Help](mailto:invalidemail?subject=I need help!)")
InvalidMarkdown: Invalid email address "invalidemail"

Great! Worked as expected.

Handling Internal and External Links

Now that we got our toes wet with mailto links, we can handle other types of links:

External Links

Links outside our Django app.
Must contains a scheme: either http or https.
Ideally, we also want to make sure these links are not broken, but we won't do that now.

Internal Links

Links to pages inside our Django app.
Link must be relative: this will allow us to move content between environments.
Use Django's URL names instead of a URL path: this will allow us to safely move views around without worrying about broken links in markdown content.
Links may contain query parameters (?) and a fragment (#).

SEO

From an SEO standpoint, public URL's should not change. When they do, you should handle it properly with redirects, otherwise you might get penalized by search engines.

With this list of requirements we can start working.

Resolving URL Names

To link to internal pages we want writers to provide a URL name, not a URL path. For example, say we have this view:

from django.urls import path
from app.views import home

urlpatterns = [
    path('', home, name='home'),
]

The URL path to this page is https://example.com/, the URL name is home. We want to use the URL name home in our markdown links, like this:

Go back to [homepage](home)

This should render to:

<p>Go back to <a href="/">homepage</a></p>

We also want to support query params and hash:

Go back to [homepage](home#top)
Go back to [homepage](home?utm_source=faq)

This should render to the following HTML:

<p>Go back to <a href="/#top">homepage</a></p>
<p>Go back to <a href="/?utm_source=faq">homepage</a></p>

Using URL names, if we change the URL path, the links in the content will not be broken. To check if the href provided by the writer is a valid url_name, we can try to reverse it:

>>> from django.urls import reverse
>>> reverse('home')
'/'

The URL name "home" points to the url path "/". When there is no match, an exception is raised:

>>> from django.urls import reverse
>>> reverse('foo')
NoReverseMatch: Reverse for 'foo' not found.
'foo' is not a valid view function or pattern name.

Before we move forward, what happens when the URL name include query params or a hash:

>>> from django.urls import reverse
>>> reverse('home#top')
NoReverseMatch: Reverse for 'home#top' not found.
'home#top' is not a valid view function or pattern name.

>>> reverse('home?utm_source=faq')
NoReverseMatch: Reverse for 'home?utm_source=faq' not found.
'home?utm_source=faq' is not a valid view function or pattern name.

This makes sense because query parameters and hash are not part of the URL name.

To use reverse and support query params and hashes, we first need to clean the value. Then, check that it is a valid URL name and return the URL path including the query params and hash, if provided:

import re
from django.urls import reverse

def clean_link(href: str, site_domain: str) -> str:
    # ... Same as before ...

    # Remove fragments or query params before trying to match the URL name.
    href_parts = re.search(r'#|\?', href)
    if href_parts:
        start_ix = href_parts.start()
        url_name, url_extra = href[:start_ix], href[start_ix:]
    else:
        url_name, url_extra = href, ''

    try:
        url = reverse(url_name)
    except NoReverseMatch:
        pass
    else:
        return url + url_extra

    return href

This snippet uses a regular expression to split href in the occurrence of either ? or #, and return the parts.

Make sure that it works:

>>> md = markdown.Markdown(extensions=[DjangoUrlExtension()])
>>> md.convert("Go back to [homepage](home)")
<p>Go back to <a href="/">homepage</a></p>

>>> md.convert("Go back to [homepage](home#top)")
<p>Go back to <a href="/#top">homepage</a></p>

>>> md.convert("Go back to [homepage](home?utm_source=faq)")
<p>Go back to <a href="/?utm_source=faq">homepage</a></p>

>>> md.convert("Go back to [homepage](home?utm_source=faq#top)")
<p>Go back to <a href="/?utm_source=faq#top">homepage</a></p>

Amazing! Writers can now use URL names in Markdown. They can also include query parameters and fragment to be added to the URL.

Handling External Links

To handle external links properly we want to check two things:

External links always provide a scheme, either http: or https:.
Prevent absolute links to our own site. Internal links should use URL names.

So far, we handled URL names and mailto links. If we passed these two checks it means href is a URL. Let's start by checking if the link is to our own site:

from urllib.parse import urlparse

def clean_link(href: str, site_domain: str) -> str:
    parsed_url = urlparse(href)
    if parsed_url.netloc == site_domain:
        # TODO: URL is internal.

The function urlparse returns a named tuple that contains the different parts of the URL. If the netloc property equals the site_domain, the link is really an internal link.

If the URL is in fact internal, we need to fail. But, keep in mind that writers are not necessarily technical people, so we want to help them out a bit and provide a useful error message. We require that internal links use a URL name and not a URL path, so it's best to let writers know what is the URL name for the path they provided.

To get the URL name of a URL path, Django provides a function called resolve:

>>> from django.utils import resolve
>>> resolve('/')
ResolverMatch(
    func=app.views.home,
    args=(),
    kwargs={},
    url_name=home,
    app_names=[],
    namespaces=[],
    route=,
)
>>> resolve('/').url_name
'home'

When a match is found, resolve returns a ResolverMatch object that contains, among other information, the URL name. When a match is not found, it raises an error:

>>> resolve('/foo')
Resolver404: {'tried': [[<URLPattern '' [name='home']>]], 'path': 'foo'}

This is actually what Django does under the hood to determine which view function to execute when a new request comes in.

To provide writers with better error messages we can use the URL name from the ResolverMatch object:

from urllib.parse import urlparse

def clean_link(href: str, site_domain: str) -> str:
    # ...

    parsed_url = urlparse(href)
    if parsed_url.netloc == site_domain:
        try:
            resolver_match = resolve(parsed_url.path)
        except Resolver404:
            raise InvalidMarkdown(
                "Should not use absolute links to the current site.\n"
                "We couldn't find a match to this URL. Are you sure it exists?",
                value=href,
            )
        else:
            raise InvalidMarkdown(
                "Should not use absolute links to the current site.\n"
                'Try using the url name "{}".'.format(resolver_match.url_name),
                value=href,
            )

    return href

When we identify that the link in internal, we handle two cases:

We don't recognize the URL: The url is most likely incorrect. Ask the writer to check the URL for mistakes.
We recognize the URL: The url is correct so tell the writer what URL name to use instead.

Let's see it in action:

>>> clean_link('https://example.com/', 'example.com')
InvalidMarkdown: Should not use absolute links to the current site.
Try using the url name "home". "https://example.com/"

>>> clean_link('https://example.com/foo', 'example.com')
InvalidMarkdown: Should not use absolute links to the current site.
We couldn't find a match to this URL.
Are you sure it exists? "https://example.com/foo"

>>> clean_link('https://external.com', 'example.com')
'https://external.com'

Nice! External links are accepted and internal links are rejected with a helpful message.

Requiring Scheme

The last thing we want to do is to make sure external links include a scheme, either http: or https:. Let's add that last piece to the function clean_link:

def clean_link(href: str, site_domain: str) -> str:
    # ...
    parsed_url = urlparse(href)

    #...
    if parsed_url.scheme not in ('http', 'https'):
        raise InvalidMarkdown(
            'Must provide an absolute URL '
            '(be sure to include https:// or http://)',
            href,
        )

    return href

Using the parsed URL we can easily check the scheme. Let's make sure it's working:

>>> clean_link('external.com', 'example.com')
InvalidMarkdown: Must provide an absolute URL (be sure to include https:// or http://) "external.com"

We provided the function with a link that has no scheme, and it failed with a helpful message. Cool!

Putting it All Together

This is the complete code for the clean_link function:

def clean_link(href: str, site_domain: str) -> str:
    if href.startswith('mailto:'):
        email_match = re.match(r'^(mailto:)?([^?]*)', href)
        if not email_match:
            raise InvalidMarkdown('Invalid mailto link', value=href)

        email = email_match.groups()[-1]
        if email:
            try:
                EmailValidator()(email)
            except ValidationError:
                raise InvalidMarkdown('Invalid email address', value=email)

        return href

    # Remove fragments or query params before trying to match the url name
    href_parts = re.search(r'#|\?', href)
    if href_parts:
        start_ix = href_parts.start()
        url_name, url_extra = href[:start_ix], href[start_ix:]
    else:
        url_name, url_extra = href, ''

    try:
        url = reverse(url_name)
    except NoReverseMatch:
        pass
    else:
        return url + url_extra

    parsed_url = urlparse(href)

    if parsed_url.netloc == site_domain:
        try:
            resolver_match = resolve(parsed_url.path)
        except Resolver404:
            raise InvalidMarkdown(
                "Should not use absolute links to the current site.\n"
                "We couldn't find a match to this URL. Are you sure it exists?",
                value=href,
            )
        else:
            raise InvalidMarkdown(
                "Should not use absolute links to the current site.\n"
                'Try using the url name "{}".'.format(resolver_match.url_name),
                value=href,
            )

    if parsed_url.scheme not in ('http', 'https'):
        raise InvalidMarkdown(
            'Must provide an absolute URL '
            '(be sure to include https:// or http://)',
            href,
        )

    return href

To get a sense of what a real use case for all of these features look like, take a look at the following content:

# How to Get Started?

Download the [mobile app](https://some-app-store.com/our-app) and log in to your account.
If you don't have an account yet, [sign up now](signup?utm_source=getting_started).
For more information about pricing, check our [pricing plans](home#pricing-plans)

This will produce the following HTML:

<h1>How to Get Started?</h1>
<p>Download the <a href="https://some-app-store.com/our-app">mobile app</a> and log in to your account.
If you don't have an account yet, <a href="signup/?utm_source=getting_started">sign up now</a>.
For more information about pricing, check our <a href="/#pricing-plans">pricing plans</a></p>

Nice!

Conclusion

We now have a pretty sweet extension that can validate and transform links in Markdown documents! It is now much easier to move documents between environments and keep our content tidy and most importantly, correct and up to date!

Source

The full source code can be found in this gist.

Taking it Further

The capabilities described in this article worked well for us, but you might want to adjust it to fit your own needs.

If you need some ideas, then in addition to this extension we also created a markdown Preprocessor that lets writers use constants in Markdown. For example, we defined a constant called SUPPORT_EMAIL, and we use it like this:

Contact our support at [$SUPPORT_EMAIL](mailto:$SUPPORT_EMAIL)

The preprocessor will replace the string $SUPPORT_EMAIL with the text we defined, and only then render the Markdown.

Building an IVR System with Python, Django and Twilio

2020-02-12T00:00:00+02:00

Last year my team and I worked on a very challenging IVR system. After almost a year in production and thousands of processed transactions, I teamed up with the great people over at the Twilio blog to write an introductory tutorial for developing IVR systems using Django and Twilio IVR.

Aside from "making your server talk" and diving into the cool speech features, I found the most challenging part working on IVR is designing the views. Unlike APIs and Forms, IVR is very limited in the type of input it takes (DTMF tones, transcribed speech), and the amount of data it can communicate and process is limited.

Read "Building an Interactive Voice Response (IVR) System with Python, Django and Twilio" on the Twilio blog ≫

Building an IVR System with Django and Twilio

Understand Group by in Django with SQL

2020-02-11T00:00:00+02:00

Aggregation is a source of confusion in any type of ORM and Django is no different. The documentation provides a variety of examples and cheat-sheets that demonstrate how to group and aggregate data using the ORM, but I decided to approach this from a different angle.

In this article I put QuerySets and SQL side by side. If SQL is where you are most comfortable, this is the Django GROUP BY cheat-sheet for you.

Image by Jason Leung

Table of Contents

How to Group By in Django
Going Further

How to Group By in Django

To demonstrate different GROUP BY queries, I will use models from Django's built-in django.contrib.auth app.

>>> from django.contrib.auth.models import User

Django ORM produces SQL statements with long aliases. For brevity, I will show a cleaned-up, but equivalent, version of what Django executes.

SQL Logging

To see the SQL actually executed by Django, you can turn on SQL logging in the Django settings.

How to Count Rows

Let's count how many users we have:

SELECT
    COUNT(*)
FROM
    auth_user;

User.objects.count()

Counting rows is so common that Django includes a function for it right on the QuerySet. Unlike other QuerySets we'll see next, count returns a number.

How to Use Aggregate Functions

Django offers two more ways to count rows in a table.

We'll start with aggregate:

SELECT
    COUNT(id) AS id__count
FROM
    auth_user;

from django.db.models import Count

User.objects.aggregate(Count('id'))

To use aggregate we imported the aggregate function Count. The function accepts an expression to count. In this case, we used the name of the primary key column id to count all the rows in the table.

Aggregate NULL

Aggregations ignore NULL values. For more on how aggregations handle NULL, see 12 Common Mistakes and Missed Optimization Opportunities in SQL.

The result of aggregate is a dict:

>>> from django.db.models import Count
>>> User.objects.aggregate(Count('id'))
{"id__count": 891}

The name of the key is derived from the name of the field and the name of the aggregate. In this case, it's id__count. It's a good idea not to rely on this naming convention, and instead provide your own name:

SELECT
    COUNT(id) as total
FROM
    auth_user;

>>> from django.db.models import Count
>>> User.objects.aggregate(total=Count('id'))
{"total": 891}

The name of the argument to aggregate is also the name of the key in the resulting dictionary.

How to Group By

Using aggregate we got the result of applying the aggregate function on the entire table. This is useful, but usually we want to apply the aggregation on groups of rows.

Let's count users by their active status:

SELECT
    is_active,
    COUNT(id) AS total
FROM
    auth_user
GROUP BY
    is_active

(User.objects
.values('is_active')
.annotate(total=Count('id')))

This time we used the function annotate. To produce a GROUP BY we use a combination of values and annotate:

values('is_active'): what to group by
annotate(total=Count('id')): what to aggregate

The order is important: failing to call values before annotate will not produce aggregate results.

Just like aggregate, the name of the argument to annotate is the key in the result of the evaluated QuerySet. In this case it's total.

How to Filter a QuerySet With Group By

To apply aggregation on a filtered query you can use filter anywhere in the query. For example, count only staff users by their active status:

SELECT
    is_active,
    COUNT(id) AS total
FROM
    auth_user
WHERE
    is_staff = True
GROUP BY
    is_active

(User.objects
.values('is_active')
.filter(is_staff=True)
.annotate(total=Count('id')))

How to Sort a QuerySet With Group By

Like filter, to sort a queryset use order_by anywhere in the query:

SELECT
    is_active,
    COUNT(id) AS total
FROM
    auth_user
GROUP BY
    is_active
ORDER BY
    is_active,
    total

(User.objects
.values('is_active')
.annotate(total=Count('id'))
.order_by('is_active', 'total'))

Notice that you can sort by both the GROUP BY key and the aggregate field.

How to Combine Multiple Aggregations

To produce multiple aggregations of the same group, add multiple annotations:

SELECT
    is_active,
    COUNT(id) AS total,
    MAX(date_joined) AS last_joined
FROM
    auth_user
GROUP BY
    is_active

from django.db.models import Max

(User.objects
.values('is_active')
.annotate(
    total=Count('id'),
    last_joined=Max('date_joined'),
))

The query will produce the number of active and inactive users, and the last date a user joined in each group.

How to Group by Multiple Fields

Just like performing multiple aggregations, we might also want to group by multiple fields. For example, group by active status and staff status:

SELECT
    is_active,
    is_staff,
    COUNT(id) AS total
FROM
    auth_user
GROUP BY
    is_active,
    is_staff

(User.objects
.values('is_active', 'is_staff')
.annotate(total=Count('id')))

The result of this query includes is_active, is_staff and the number of users in each group.

How to Group by an Expression

Another common use case for GROUP BY is to group by an expression. For example, count the number of users that joined each year:

SELECT
    EXTRACT('year' FROM date_joined),
    COUNT(id) AS total
FROM
    auth_user
GROUP BY
    EXTRACT('year' FROM date_joined)

(User.objects
.values('date_joined__year')
.annotate(total=Count('id')))

Notice that to get the year from the date we used the special expression <field>__year in the first call to values(). The result of the query is a dict, and the name of the key will be date_joined__year.

Sometimes, the built-in expressions are not enough, and you need to aggregate on a more complicated expression. For example, group by users that have logged in since they signed-up:

SELECT
  last_login > date_joined AS logged_since_joined,
  COUNT(id) AS total
FROM
  auth_user
GROUP BY
  last_login > date_joined

from django.db.models import (
    ExpressionWrapper,
    Q, F, BooleanField,
)

(User.objects
.annotate(
    logged_since_joined=ExpressionWrapper(
        Q(last_login__gt=F('date_joined')),
        output_field=BooleanField(),
    )
)
.values('logged_since_joined')
.annotate(total=Count('id'))
.values('logged_since_joined', 'total')

The expression here is fairly complicated. We first use annotate to built the expression, and we mark it as a GROUP BY key by referencing the expression in the following call to values(). From here on, it's exactly the same.

How to Use Conditional Aggregation

Using conditional aggregation, you can aggregate only a part of the group. Conditions come in handy when you have multiple aggregates. For example, count the number of staff and non-staff users by the year they signed-up:

SELECT
    EXTRACT('year' FROM date_joined),

    COUNT(id) FILTER (
        WHERE is_staff = True
    ) AS staff_users,

    COUNT(id) FILTER (
        WHERE is_staff = False
    ) AS non_staff_users

FROM
    auth_user
GROUP BY
    EXTRACT('year' FROM date_joined)

from django.db.models import F, Q

(User.objects
.values('date_joined__year')
.annotate(
    staff_users=(
        Count('id', filter=Q(is_staff=True))
    ),
    non_staff_users=(
        Count('id', filter=Q(is_staff=False))
    ),
))

The SQL above is from PostgreSQL, which along with SQLite is currently the only database backend that supports the FILTER syntax shortcut (formally called "selective aggregates"). For other database backends, the ORM will use CASE ... WHEN instead.

tip

I previously wrote about aggregations with filters. Check out my 9 Django tips for working with databases.

How to Use Having

The HAVING clause is used to filter on the result of an aggregate function. For example, find the years in which more than a 100 users joined:

SELECT
    is_active,
    COUNT(id) AS total
FROM
    auth_user
GROUP BY
    is_active
HAVING
    COUNT(id) > 100

(User.objects
.annotate(year_joined=F('date_joined__year'))
.values('is_active')
.annotate(total=Count('id'))
.filter(total__gt=100))

The filter on the annotated field total added an HAVING clause in the generated SQL.

How to Group by Distinct

For some aggregate functions such as COUNT, it is sometimes desirable to only count distinct occurrences. For example, how many different last names are there per user active status:

SELECT
    is_active,
    COUNT(id) AS total,
    COUNT(DISTINCT last_name) AS unique_names
FROM
    auth_user
GROUP BY
    is_active

(User.objects
.values('is_active')
.annotate(
    total=Count('id'),
    unique_names=Count('last_name', distinct=True),
))

Notice the use of distinct=True in the call to Count.

How to Create Expressions Using Aggregate Fields

Aggregate fields are often just the first step to a greater question. For example, what is the percent of unique last names by user active status:

SELECT
    is_active,
    COUNT(id) AS total,
    COUNT(DISTINCT last_name) AS unique_names,
    (COUNT(DISTINCT last_name)::float
        / COUNT(id)::float) AS pct_unique_names
FROM
    auth_user
GROUP BY
    is_active

from django.db.models import FloatField
from django.db.models.functions import Cast

(User.objects
.values('is_active')
.annotate(
    total=Count('id'),
    unique_names=Count('last_name', distinct=True),
)
.annotate(pct_unique_names=(
    Cast('unique_names', FloatField())
    / Cast('total', FloatField())
))

The first annotate() defines the aggregate fields. The second annotate() uses the aggregate function to construct an expression.

How to Group By Across Relations

So far we've used only data in a single model, but aggregates are often used across relations. The simpler scenario is of a one-to-one or a foreign key relation. For example, say we have a UserProfile with a one-to-one relationship to the User, and we want to count users by the type of profile:

SELECT
    p.type,
    COUNT(u.id) AS total
FROM
    auth_user u
    JOIN user_profile p ON u.id = p.user_id
GROUP BY
    p.type

(User.objects
.values('user_profile__type')
.annotate(total=Count('id')))

Just like GROUP BY expressions, using relations in values will group by that field. Note that the name of the user profile type in the result will be 'user_profile__type'.

How to Group By a Many to Many Relationship

A more complicated type of relation is the many to many relationship. For example, count in how many groups each user is a member:

SELECT
    u.id,
    COUNT(ug.group_id) AS memberships
FROM
    auth_user
    LEFT OUTER JOIN auth_user_groups ug ON (
        u.id = ug.user_id
    )
GROUP BY
    u.id

(User.objects
.annotate(memberships=Count('groups'))
.values('id', 'memberships'))

A user can be a member of more than one group. To count the number of groups the user is member of we used the related name "groups" in the User model. If the related name is not explicitly set (and not explicitly disabled), Django will automatically generate a name in the format {related model model}_set. For example, group_set.

Going Further

To dig deeper into the ORM and GROUP BY in particular, check out these links:

How to use grouping sets in Django: An article about advanced group by techniques such as group by cube, group by rollup and group by grouping sets.
How to Get the First or Last Value in a Group Using Group By in SQL: A neat little trick using arrays in PostgreSQL.
12 Common Mistakes and Missed Optimization Opportunities in SQL: Some SQL do's and dont's you need to know if you are working with data and writing SQL.
Django Aggregation cheat-sheet page: How to do common aggregate queries.

How "Export to Excel" Almost Killed Our System

2019-09-17T00:00:00+03:00

A few weeks ago we had some trouble with an "Export to Excel" functionality in one of our systems. In the process of resolving this issue, we made some interesting discoveries and came up with original solutions.

This article is inspired by the actual issue we used to track this incident over a period of two days. We go through the process of identifying the problem, experimenting and benchmarking different solutions until eventually deploying to production.

These are the main takeaways described in this article:

Generating xlsx files can consume significant amount of resources.
Under some circumstances better performance can be gained by not using prefetch_related.
pyexcelerate is a fast package for creating simple Excel files.
tablib (and django-import-export) can be patched to use pyexcelerate and produce excel files faster.

Table of Contents

Exporting a QuerySet to Excel
- Using django-import-export
- Finding the Best File Format
Improving the Query
Using a Different Excel Writer
- A Faster Excel Writer in Python
- Patching tablib
Results Summary
Seifa
Comments From Readers

How a server must feel when asked to produce an Excel file

A few weeks ago we started getting complaints from users about slow response time from one of our systems. A quick glance at the server metrics showed higher than normal CPU usage. This system is mostly IO intensive, so high CPU usage is not something we experience regularly.

The first thing we did was to identify the worker process that is consuming high CPU using htop. After getting the process identifier (PID) of the process, we used py-spy to get a glance at what it's doing:

$ py-spy -p 8187 -r 1000

This command samples the process a 1000 times per second and provides a top-like view of the results:

Total Samples 17974
GIL: 0.00%, Active: 0.00%, Threads: 1

OwnTime  TotalTime  Function (filename:line)
0.000s    173.7s   get_response (django/core/handlers/base.py:75)
0.000s    173.7s   inner (django/core/handlers/exception.py:34)
0.000s    173.7s   __call__ (django/utils/deprecation.py:94)
0.000s    173.7s   __call__ (django/core/handlers/wsgi.py:141)
0.000s    173.6s   view (django/views/generic/base.py:71)
0.000s    173.6s   _get_response (django/core/handlers/base.py:113)
0.000s    173.6s   dispatch (django/contrib/auth/mixins.py:52)
0.000s    173.6s   dispatch (django/contrib/auth/mixins.py:109)
0.000s    173.6s   dispatch (django/views/generic/base.py:97)
0.050s    173.6s   get (dashboard/views/list_views.py:100)
0.000s    94.69s   get_resource_to_export (dashboard/views/list_views.py:70)
0.000s    94.69s   export (dashboard/views/list_views.py:73)
0.000s    94.68s   export (dashboard/resources.py:215)
0.000s    83.81s   __iter__ (django/db/models/query.py:274)
0.040s    82.73s   _fetch_all (django/db/models/query.py:1242)
0.000s    78.84s   export (dashboard/views/list_views.py:74)
0.000s    70.58s   __iter__ (django/db/models/query.py:55)
0.000s    68.98s   execute_sql (django/db/models/sql/compiler.py:1100)
68.81s    68.81s   _execute (django/db/backends/utils.py:84)
0.000s    68.81s   _execute_with_wrappers (django/db/backends/utils.py:76)
0.000s    68.81s   execute (django/db/backends/utils.py:67)
0.000s    50.11s   save (tablib/packages/openpyxl3/workbook.py:186)
0.000s    50.11s   export_set (tablib/formats/_xlsx.py:46)
0.000s    46.41s   save (tablib/packages/openpyxl3/writer/excel.py:124)
0.000s    46.41s   save_workbook (tablib/packages/openpyxl3/writer/excel.py:141)
0.000s    42.40s   _fetch_all (django/db/models/query.py:1244)
0.000s    42.40s   _prefetch_related_objects (django/db/models/query.py:771)
0.000s    42.38s   prefetch_related_objects (django/db/models/query.py:1625)
0.000s    41.94s   prefetch_one_level (django/db/models/query.py:1738)
0.000s    41.25s   get_prefetch_queryset (django/db/models/fields/related_descriptors.py:627)
0.000s    32.30s   _write_worksheets (tablib/packages/openpyxl3/writer/excel.py:91)

After monitoring this view for a minute or two, we had a few insights:

A lot of time is spent fetching data.
A lot of time is spent on some call to prefetch_related.
The problem is in the dashboard, and more specifically in the view that exports data.

With these insights, we wanted to moved on to identify the exact view. We then turned to the nginx access log:

$ journalctl -u nginx -r | grep dashboard

We managed to identify several endpoints that were taking very long to execute. Some of them finished in just under 60 seconds, others were killed by PostgreSQL after hitting the statement_timeout limit and returned a 500 status code.

At this point we had a pretty good idea where the problem is, but we were still clueless as to why. The next step was to inspect the problematic code, and try to reproduce.

Exporting a QuerySet to Excel

The system is used to report and track violations in public transportation. During an inspection, the inspector documents different types of violations such as dirty bus, bus running late etc. The models for this system look roughly like this:

class ViolationType(models.Model):
    name = model.TextField()


class Inspection(models.Model):
    id = models.AutoField(primary_key=True)


class Violation(models.Model):
    inspection = models.ForeignKey(Inspection, on_delete=models.PROTECT)
    violation_type = models.ForeignKey(ViolationType, on_delete=models.PROTECT)
    comments = models.TextField()

Every once in a while, a back office user would download the inspection information to Excel for further analysis.

Users + Excel = Love

The report includes a lot of information about the inspection, but most importantly, it includes a list of the violation types for each inspection:

inspection, violations
1, dirty floors | full trash can
2, full trash can | no light | missing signs

Using `django-import-export`

To produce the Excel report we use a package called django-import-export. Using the package, we define a ModelResource that can produce an Excel file from a queryset:

from import_export import resources, fields, widgets

from .models import Inspection, Violation


class InspectionResource(resources.ModelResource):
    violations = fields.Field(
        widget=widgets.ManyToManyWidget(Violation, field='violation_type')
    )

    class Meta:
        model = Inspection
        fields = (
            'id',
            'violations',
        )

The query produced by this ModelResource causes an N+1 queries issue, so before we ever deployed it to production we patched it and added prefetch_related:

from django.db.models import Prefetch
from import_export import resources, fields, widgets

from .models import Inspection, Violation


class InspectionResource(resources.ModelResource):
    violations = fields.Field(
        widget=widgets.ManyToManyWidget(Violation, field='violation_type')
    )

    class Meta:
        model = Inspection
        fields = (
            'id',
            'violations',
        )

    def export(self, queryset=None):
        queryset = (
            queryset
            .prefetch_related(Prefetch(
                'violations',
                queryset=Violation.objects.select_related('violation_type'),
                to_attr='prefetched_violations',
            ))
        )
        return super().export(list(queryset))

    def dehydrate_violations(self, inspection: Inspection) -> str:
        return '\n'.join(
            v.violation_type.name
            for v in inspection.prefetched_violations
        )

To use prefetch_related in a ModelResource we had to make the following changes:

Override export and adjust the query to prefetch the violations using prefetch_related. We use the Prefetch object because we needed to customize the prefetch query, and add the violation type name from a related table.
Evaluate the query and have the export function return a list instead of a queryset. django-import-export uses iterator to speed up the query. Using iterator(), the ORM uses a cursor to iterate over the data in chunks and reduce memory. While this is usually useful, Django is unable to use iterator() with prefetch_related.
Add a custom dehydrate_ function for the violations field that will render a comma-delimited list of violation type names.

Prefetch Related

This is all you need to know about prefetching in Django

The resource was used by the view to produce the Excel report:

from django.http import HttpRequest, HttpResponse
from .models import Inspection
from .resources import InspectionResource

LIMIT = 10000

def export_to_excel(request: HttpRequest) -> HttpResponse:
    inspections = Inspection.objects.all()
    # Apply some filter on the queryset based on request
    data = InspectionResource().export(inspections[:LIMIT]).xlsx
    response = HttpResponse(data, content_type='text/xlsx')
    response['Content-Disposition'] = 'attachment; filename=export.xlsx'
    return response

The view takes a request, apply some filter on the inspections and produces the xlsx file using the ModelResource.

Finding the Best File Format

Before we can start improving the export process, we need to establish a baseline. To get the timings and identify the hot spots in the call stack we used cProfile. To identify and time query execution we turned SQL logging on in the Django settings:

# settings.py

LOGGING = {
    # ...
    'loggers': {
        'django.db.backends': {
            'level': 'DEBUG',
        },
        # ...
    },
}

The benchmark looked like this:

import cProfile

from .models import Inspection
from .resources import InspectionResource

qs = VehicleInspection.objects.all()[:10000]
cProfile.run('resources.VehicleInspectionResource().export(qs).xlsx')

These were the results of exporting 10,000 rows in xlsx format using prefetch_related:

56830808 function calls (47142920 primitive calls) in 41.574 seconds
select    5.009
prefetch  8.009

56660555 function calls (47149065 primitive calls) in 39.927 seconds
select    2.356
prefetch  7.991

We ran the benchmark twice to make sure the results were not effected by caches. The function took 40s to complete, and only 10s of it (25%) were spent in the database.

At this point, we suspected that the problem might be in the file format. This assumption was supported by the application server's high CPU usage.

Next, we wanted to try the same benchmark, only instead of xlsx we produced a csv:

cProfile.run('resources.VehicleInspectionResource().export(qs).csv')

These were the results of exporting 10,000 rows in csv format using prefetch_related:

9179705 function calls (9107672 primitive calls) in 17.429 seconds
select    1.970
prefetch  8.343

Wow! That's a big improvement. This confirmed our suspicion that the actual production of the xlsx was the problem.

Before we moved on, we wanted to check another file format that might be more useful to our users, the old xls format:

cProfile.run('resources.VehicleInspectionResource().export(qs).xls')

These were the results of exporting 10,000 rows in xls format using prefetch_related:

16317592 function calls (15745704 primitive calls) in 20.694 seconds
select    1.922
prefetch  7.976

OK, so that's surprising. I'm not familiar with the internals of the Microsoft Office file formats, but it seems like the old format is only a little bit slower than the csv format, and much faster than the new xlsx format.

This benchmark results brought up an old dilemma. In the past we used to serve users with only csv files, but they complained a lot about troubles opening the files, and encoding and formatting issues. For this reason we decided to produce xlsx in the first place, so at that time, producing xls files seemed like the best solution.

I should already tell you, using the old xls format was a bad decision, but we didn't know that yet.

Improving the Query

After reducing the overall execution time by half, our next targets were the queries. Two queries are executed to produce the dataset for the export. Before any change is made, it took the "main" query ~2s and the prefetch ~8s to complete.

The "main" query looked like this:

SELECT
    "inspection"."id",
    -- around 50 more fields from joined tables
FROM
    "inspection"
    INNER JOIN "auth_user" ON ("inspection"."user_id" = "auth_user"."id")
    -- around 11 more joined tables

The resource used a lot of data from related tables, and the query joined ~12 tables and had many fields listed in the SELECT clause. The table is one of the main tables in the database so it is heavily indexed, and the lookup tables were relatively small so the query didn't take long to complete.

The prefetch query looked like this:

SELECT
    "violation"."id",
    "violation"."inspection_id",
    "violation"."violation_type_id",
    "violation_type"."id",
    "violation_type"."name",
FROM "violation"
    INNER JOIN "violation_type" ON (
        "violation"."violation_type_id" = "violation_type"."id"
    )
WHERE
    "violation"."inspection_id" = ANY([
        2814, 9330, 8848, 8971, 9372, 9084, 78, 3896, 2609, 5177, 2866
        -- another 10,000 inspection IDs
        1399, 9348, 914, 8884, 9082, 3356, 2896, 742, 9432, 8926, 9153
    ])

This query seems innocent, but in fact, it took ~8s to complete. The execution plan of this query looked like this:

Nested Loop (cost=1000.28..2040346.39 rows=26741 width=181)
  ->  Gather (cost=1000.00..2032378.29 rows=26741 width=115)
        Workers Planned: 2
        ->  Parallel Seq Scan on violation  (cost=0.00..2028704.19 rows=11142 width=115)
              Filter: (vehicle_inspection_id = ANY ('{2814,9330,....,8926,9153}'::integer[]))
  ->  Index Scan using violationtype_pkey on violationtype  (cost=0.28..0.30 rows=1 width=66)
        Index Cond: (id = violation.violation_type_id)

I trimmed the execution plan for brevity, but the Filter line was three or four pages long, filled with IDs. This got us thinking, is it possible that this huge ANY filter is what's causing us trouble?

Replacing `prefetch_related` with `Subquery` and `OuterRef`

To answer this question we decided to try and implement the query without prefetch_related. Instead, we decided to use the new Subquery expression.

Using Subquery the query using the ORM looked like that:

from django.db.models import OuterRef, Subquery, Value
from django.contrib.postgres.aggregates import ArrayAgg

inspections = inspections.annotate(violations_csv=Subquery(
    Violation.objects
    # Reference the inspection ID of the outer table, inspection.
    .filter(inspection_id=OuterRef('id'))
    # Prevent Django from adding a group by column.
    .annotate(dummy=Value('1')).values('dummy')
    # Construct an array of violation names.
    .annotate(violations=ArrayAgg('violation_type__name', distinct=True))
    .values('violations')
))

If you never experimented with Subquery there is a lot to take in here. Before we break it down, this is what the query looks like:

SELECT
    "inspection"."id",
    (
        SELECT
            ARRAY_AGG(DISTINCT U2."name") AS "violations"
        FROM
            "violation" U0
            INNER JOIN "violationtype" U2 ON (U0."violation_type_id" = U2."id")
        WHERE
            U0."inspection_id" = ("inspection"."id")

    ) AS "violations_csv",
    -- around 50 more fields from joined tables
FROM
    "inspection"
    INNER JOIN "auth_user" ON ("inspection"."user_id" = "auth_user"."id")
    -- around 11 more joined tables

Now, let's break it down:

Subquery is a query expression that can only exist inside another query. In this case, the outer query is inspection.
Subquery in used in annotate so the result of the subquery is stored in a another column for each row.
We added a dummy annotation to prevent Django from grouping the results. The subquery is executed for each inspection, this is what the filter using OuterRef does. For this reason, we don't need to group by any other column.
The subquery must return at most one row, so we group the names into an array using ARRAY_AGG.

After all this hard work, we were keen to see if this is the silver bullet we were waiting for, but in fact, when we executed this on 10,000 rows it choked. To see it through, we executed the export function with only 1,000 rows.

These were the results of exporting 1,000 rows in xls format using subquery:

1571053 function calls (1514505 primitive calls) in 60.962 seconds
select 59.917

The query is now crazy slow. I won't paste the execution plan because there were so many other tables, but PostgreSQL used a nested loop join on the top level of the query to produce the value for this field. Surprisingly, the database did a significantly worse job than the ORM did in this case.

Using an Iterator

Before we completely abandoned this solution, we wanted to check one last thing. We previously mentioned that django-import-export is using iterator() to create a cursor over the results. We also mentioned that using prefetch_related prevents us from using iterator(). Well, we no longer use prefetch_related so we might as well check if using iterator() makes any difference.

These were the results of exporting 1,000 rows in xls format using subquery and iterator:

1571580 function calls (1514788 primitive calls) in 62.130 seconds
select 60.618

The iterator made no difference.

Simplifying the Query

In a final attempt to get something out of this expedition, we wanted to see if the complexity of the query prevented PostgreSQL from finding an optimal execution plan. To do that, we could have adjusted the database parameters from_collapse_limit and join_collapse_limit and let PostgreSQL take all the time and resources it needs to find an optimal execution plan, but instead, we decided to strip all other fields from the resources besides id and violations.

These were the results of exporting 1,000 rows containing only the id and violations fields in xls format using subquery and iterator:

6937 function calls (6350 primitive calls) in 57.280 seconds
select  57.255

No change, this is officially a dead end!

Manual Prefetch

After a quick lunch break we decided it's time pull out the big guns. If Django's prefetch implementation wasn't working for us, and PostgreSQL was unable to produce a decent execution plan, we will just have to do it ourselves.

To implement our own "prefetch" we needed to adjust some of the other functions in the resource:

from import_export import resources, fields

from .models import Inspection, Violation


class InspectionResource(resources.ModelResource):
    violations = fields.Field()

    class Meta:
        model = Inspection
        fields = (
            'id',
            'violations',
        )

    def export(self, queryset=None):
        # Manually prefetch the violations.
        self.prefetched_violations = dict(
            Violation.objects
            .filter(inspection_id__in=(
                queryset
                # Clean all joins.
                .select_related(None)
                .values_list('pk')
            )
            .annotate(
                violations_csv=ArrayAgg('violation_type__name'),
            )
            .values_list(
                'vehicle_inspection_id',
                'violations_csv',
            )
        )

        return super().export(queryset)

    def dehydrate_violations(self, inspection: Inspection) -> str:
        return '\n'.join(self.prefetched_violations.get(inspection.id, []))

This looks like a lot, but it's actually not:

We create our own "prefetch related" dict prefetched_violations:
- The key is the violation ID, and the value is an array containing the violation names (violations_csv).
- To fetch only relevant violations, we use filter using queryset to filter only the necessary inspections.
- We executed select_related(None) to remove all previously set select_related tables, and make the ORM remove any unnecessary joins.
We return the original queryset to the export function which produces the Excel file.
To construct the value for the violations field, we use the prefetched_violations we populated during export. This is the "lookup" part of the prefetch. While using Django's prefetch_related we have access to this value on the instance, when we do it manually we have to look it up ourselves.
Once again, since we no longer use Django's prefetch_related we were able to use an iterator. So, instead of evaluating the query we return a queryset.

We already got disappointed after putting in a lot of effort the last time, let's see if this time the hard work paid off.

These were the results of exporting 10,000 rows in xls format using manual prefetch and iterator:

15281887 function calls (14721333 primitive calls) in 11.411 seconds
select  0.833
manual prefetch 0.107

Compared to the 40 seconds we started with, this is an overall 75% improvement. 20s were reduced by switching to xls format, another 10s were from manually doing the prefetch.

We are ready for production!

Trouble in Paradise

Quickly after rolling out the new version to production we started getting complaints from users not being able to open the file.

Remember I told you using xls was a bad idea? Well, when users started downloading the xls files they got a nasty message saying the file is corrupt, and excel, thank god, managed to salvage some of the data (which is way worse!).

One might ask, "but how come you didn't catch this in QA?". Well, that's just another reason we hate working with Excel. When we tested it locally on our Linux desktops using LibreOffice, it worked just fine.

"But it works on my machine!"

So let's recap:

xlsx is slow and consumes a lot of CPU.
xls is not supported by the excel version used by our users.
csv has many encoding and formatting issues, and proved to be unusable in the past.

Using a Different Excel Writer

As always, when all options suck and the future is looking bleak, we turned to Google.

A quick search of "python excel performance" brought up this gist which compares 4 different Excel writers in Python (gotta love the internet!).

These are the benchmark results:

# Source: https://gist.github.com/jmcnamara/ba25c2bf4ba0777065eb

Versions:
    python      : 2.7.2
    openpyxl    : 2.2.1
    pyexcelerate: 0.6.6
    xlsxwriter  : 0.7.2
    xlwt        : 1.0.0

Dimensions:
    Rows = 10000
    Cols = 50

Times:
    pyexcelerate          :  10.63
    xlwt                  :  16.93
    xlsxwriter (optimised):  20.37
    xlsxwriter            :  24.24
    openpyxl   (optimised):  26.63
    openpyxl              :  35.75

According to the results, there is a big difference between the xlsx libraries.

As mentioned before, we use django-import-export to produce excel files from Django models and querysets. Under the hood, django-import-export is using the popular tablib package to do the actual export.

Tablib offers export and import capabilities to and from many formats, but it doesn't do any of the heavy lifting itself. To produce xlsx files, tablib is using the package openpyxl.

A Faster Excel Writer in Python

Looking back at the benchmark results, openpyxl is the slowest among all packages. It looks like by switching to the fastest implementation, pyexcelerate we might be able to gain some significant improvement for this export process.

The package pyexcelerate looked great from start. The tag line is just what we needed:

PyExcelerate is a Python for writing Excel-compatible XLSX spreadsheet files, with an emphasis on speed.

Even the snarky subtitles on the "Usage" section in the README were just what we wanted: fast, faster and fastest!

With such promising benchmarks and README, we had to try it out!

Patching `tablib`

We already have an entire system built on top of django-import-export and tablib, and we didn't want to start making changes everywhere. So instead, we looked for a way to patch tablib, and make it use pyexcelerate instead of openpyxl.

After some digging, we found that tablib uses an internal function called _register_formats to add export and import formats such as csv, xls and xlsx. To get a list of available formats, tablib imports a collection called available from the module formats. The contents of the file formats/__init__.py where the collection is defined, look like this:

# -*- coding: utf-8 -*-

""" Tablib - formats
"""

from . import _csv as csv
from . import _json as json
from . import _xls as xls
from . import _yaml as yaml
from . import _tsv as tsv
from . import _html as html
from . import _xlsx as xlsx
from . import _ods as ods
from . import _dbf as dbf
from . import _latex as latex
from . import _df as df
from . import _rst as rst
from . import _jira as jira

available = (json, xls, yaml, csv, dbf, tsv, html, jira, latex, xlsx, ods, df, rst)

The interesting part is the contents of the file _xlsx.py. The file defines some functions to export and import from Excel using openpyxl.

To patch tablib, we first need to implement a similar interface to the one in _xlsx.py using pyexcelerate, and then register it in tablib.

Let's start with implementing _xlsx.py using pyexcelerate:

# fast_xlsx.py
import itertools
from io import BytesIO

from tablib.formats._xlsx import *  # noqa
from pyexcelerate import Workbook


# Override the default xlsx implementation
title = 'xlsx'


def export_set(dataset, freeze_panes=True):
    """Returns XLSX representation of Dataset."""
    title = dataset.title or 'Sheet1'
    wb = Workbook()
    wb.new_sheet(title, data=itertools.chain([dataset.headers], dataset))
    stream = BytesIO()
    wb.save(stream)
    return stream.getvalue()


def export_book(databook, freeze_panes=True):
    """Returns XLSX representation of DataBook."""
    assert len(databook._datasets) == 1
    return export_set(databook._datasets[0], freeze_panes)


def dset_sheet(*args, **kwargs):
    assert False, 'How did you get here?'

This is a simple implementation of the main functions. It lacks some functionalities such a multiple sheets, but it was fine for our needs.

Next, we need to make tablib register this file instead of the existing xlsx format. To do that, we created a new file called monkeypatches.py:

# monkeypatches.py
import tablib

from . import fast_xlsx

# Override default xlsx format with a faster implementation
# using `pyexcelerate` (export only).
tablib.formats.available += (fast_xlsx, )

To apply the patch to tablib, we import our implementation and add it to the available formats list. We then import this file in the module's __init__.py so every time the system starts up, tablib is patched.

Now for the moment of truth, did all this hard work finally paid off?

These were the results of exporting 10,000 rows in xlsx format with pyexcelerate using manual prefetch and iterator:

13627507 function calls (13566956 primitive calls) in 10.944 seconds
select 0.137
manual prefetch 2.219

The hard work definitely paid off! Just so we have an honest comparison, these are the results of exporting 10,000 rows in xlsx format without patching tablib using manual prefetch and iterator:

55982358 function calls (46155371 primitive calls) in 29.965 seconds
select 0.137
manual prefetch 1.724

That's a 64% improvement compared to the default implementation provided by tablib, and a 75% improvements compared to the 40s we started with.

Results Summary

This a summary of all the results mentioned in the article:

Time	Rows	Format	Method
39.927s	10,000	xlsx	`prefetch_related` (Django)
17.429s	10,000	csv	`prefetch_related` (Django)
20.694s	10,000	xls	`prefetch_related` (Django)
60.962	1,000	xls	subquery
62.130	1,000	xls	subquery and iterator
57.280s	1,000	xls	simplified query, subquery and iterator
29.965s	10,000	xlsx	default `tablib` implementation, manual prefetch and iterator
11.411s	10,000	xls	using manual prefetch and iterator
10.944s	10,000	xlsx	using `pyexcelerate`, manual prefetch and iterator

Seifa

We try to study every incident and take actions to prevent similar incidents from happening in the future. During this incident, some of our users did experience slowness for a short period of time, however, the "Export to Excel" functionality did not really killed our app.

Following this incident, there are a few open questions we haven't had the chance to fully explore yet:

Why was the prefetch query so slow? The difference boils down to executing Model.objects.filter(fk__in = [1,2,3,4....9,999, 10,000]) vs executing Model.objects.filter(fk__in = OtherModel.objects.filter( ... ).values_list('pk')). When we tried to compare the two in the database, we found no difference, but the built-in prefetch_related was significantly slower. Is it possible that time is being spent generating the query in Python?
Can openpyxl3 performance be improved? When I talked to John, the author of the Excel writers benchmark, he mentioned that openpyxl3 can be faster if lxml is installed.
Is xlsx really the best format? Can we eliminate some of the problems we had with csv by switching to a different textual format such as tsv?

If you have the answer to any of these questions feel free to share them with me and i'll be happy to post the response.

UPDATED: Aug 19, 2019

Comments From Readers

A reader from lobste.rs ran a quick benchmark to check how faster openpyxl can get using lxml. These were his results:

Versions:
python: 3.6.8
Dimensions:
    Cols = 50
    Sheets = 1
    Proportion text = 0.10
optimised = True

Rows = 10000
Times:
openpyxl: 2.6.3 using LXML True:   3.70
openpyxl: 2.6.3 using LXML False:   6.03

Rows = 1000
Times:
openpyxl: 2.6.3 using LXML True:   0.37
openpyxl: 2.6.3 using LXML False:   0.57

This benchmark shows that openpyxl can be made almost twice as fast just by installing lxml. However, pyexcelerate improved the speed by a factor of 3.

Many reader on Reddit and Lobsters suggested that a better approach would be to generate the Excel file on the client side using Javascript. This is definitely something worth considering when designing a new system, even thought I think this approach might be problematic for very large files.

What You Need to Know to Manage Users in Django Admin

2019-08-05T00:00:00+03:00

Have you ever stopped to think what your staff users can do in Django admin? Did you know staff users with misconfigured permissions on the user model can make themselves superusers?

Permissive permissions to staff users can cause disastrous human errors at best, and lead to major data leaks at worst. With the great staff of RealPython, I wrote about ways to protect your Django admin and make it safer for users, and staff users.

Read "What You Need to Know to Manage Users in Django Admin" on RealPython ≫

What You Need to Know to Manage Users in Django Admin

Improve Serialization Performance in Django Rest Framework

2019-06-08T00:00:00+03:00

When a developer chooses Python, Django, or Django Rest Framework, it's usually not because of its blazing fast performance. Python has always been the "comfortable" choice, the language you choose when you care more about ergonomics than skimming a few microseconds of some process.

There is nothing wrong with ergonomics. Most projects don't really need that micro second performance boost, but they do need to ship quality code fast.

All of this doesn't mean performance is not important. As this story taught us, major performance boosts can be gained with just a little attention, and a few small changes.

"mip mip"

Table of Contents

Model Serializer Performance
Why is This Happening?
Takeaway
Bonus: Forcing Good Habits

Model Serializer Performance

A while back we noticed very poor performance from one of our main API endpoints. The endpoint fetched data from a very large table, so we naturally assumed that the problem must be in the database.

When we noticed that even small data sets get poor performance, we started looking into other parts of the app. This journey eventually led us to Django Rest Framework (DRF) serializers.

versions

In the benchmark we use Python 3.7, Django 2.1.1 and Django Rest Framework 3.9.4.

Simple Function

Serializers are used for transforming data into objects, and objects into data. This is a simple function, so let's write one that accepts a User instance, and returns a dict:

from typing import Dict, Any

from django.contrib.auth.models import User


def serialize_user(user: User) -> Dict[str, Any]:
    return {
        'id': user.id,
        'last_login': user.last_login.isoformat() if user.last_login is not None else None,
        'is_superuser': user.is_superuser,
        'username': user.username,
        'first_name': user.first_name,
        'last_name': user.last_name,
        'email': user.email,
        'is_staff': user.is_staff,
        'is_active': user.is_active,
        'date_joined': user.date_joined.isoformat(),
    }

Create a user to use in the benchmark:

>>> from django.contrib.auth.models import User
>>> u = User.objects.create_user(
>>>     username='hakib',
>>>     first_name='haki',
>>>     last_name='benita',
>>>     email='me@hakibenita.com',
>>> )

For our benchmark we are using cProfile. To eliminate external influences such as the database, we fetch a user in advance and serialize it 5,000 times:

>>> import cProfile
>>> cProfile.run('for i in range(5000): serialize_user(u)', sort='tottime')
15003 function calls in 0.034 seconds

Ordered by: internal time
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  5000    0.020    0.000    0.021    0.000 {method 'isoformat' of 'datetime.datetime' objects}
  5000    0.010    0.000    0.030    0.000 drf_test.py:150(serialize_user)
     1    0.003    0.003    0.034    0.034 <string>:1(<module>)
  5000    0.001    0.000    0.001    0.000 __init__.py:208(utcoffset)
     1    0.000    0.000    0.034    0.034 {built-in method builtins.exec}
     1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

The simple function took 0.034 seconds to serialize a User object 5,000 times.

`ModelSerializer`

Django Rest Framework (DRF) comes with a few utility classes, namely the ModelSerializer.

A ModelSerializer for the built-in User model might look like this:

from rest_framework import serializers

class UserModelSerializer(serializers.ModelSerializer):
    class Meta:
        model = User
        fields = [
            'id',
            'last_login',
            'is_superuser',
            'username',
            'first_name',
            'last_name',
            'email',
            'is_staff',
            'is_active',
            'date_joined',
        ]

Running the same benchmark as before:

>>> cProfile.run('for i in range(5000): UserModelSerializer(u).data', sort='tottime')
18845053 function calls (18735053 primitive calls) in 12.818 seconds

Ordered by: internal time
  ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   85000    2.162    0.000    4.706    0.000 functional.py:82(__prepare_class__)
 7955000    1.565    0.000    1.565    0.000 {built-in method builtins.hasattr}
 1080000    0.701    0.000    0.701    0.000 functional.py:102(__promise__)
   50000    0.594    0.000    4.886    0.000 field_mapping.py:66(get_field_kwargs)
 1140000    0.563    0.000    0.581    0.000 {built-in method builtins.getattr}
   55000    0.489    0.000    0.634    0.000 fields.py:319(__init__)
 1240000    0.389    0.000    0.389    0.000 {built-in method builtins.setattr}
    5000    0.342    0.000   11.773    0.002 serializers.py:992(get_fields)
   20000    0.338    0.000    0.446    0.000 {built-in method builtins.__build_class__}
  210000    0.333    0.000    0.792    0.000 trans_real.py:275(gettext)
   75000    0.312    0.000    2.285    0.000 functional.py:191(wrapper)
   20000    0.248    0.000    4.817    0.000 fields.py:762(__init__)
 1300000    0.230    0.000    0.264    0.000 {built-in method builtins.isinstance}
   50000    0.224    0.000    5.311    0.000 serializers.py:1197(build_standard_field)

It took DRF 12.8 seconds to serialize a user 5,000 times, or 2.56ms to serialize just a single user. That is 377 times slower than the plain function.

We can see that a significant amount of time is spent in functional.py. ModelSerializer uses the lazy function from django.utils.functional to evaluate validations. It is also used by Django verbose names and so on, which are also being evaluated by DRF. This function seem to be weighing down the serializer.

Read Only `ModelSerializer`

Field validations are added by ModelSerializer only for writable fields. To measure the effect of validation, we create a ModelSerializer and mark all fields as read only:

from rest_framework import serializers

class UserReadOnlyModelSerializer(serializers.ModelSerializer):
    class Meta:
        model = User
        fields = [
            'id',
            'last_login',
            'is_superuser',
            'username',
            'first_name',
            'last_name',
            'email',
            'is_staff',
            'is_active',
            'date_joined',
        ]
        read_only_fields = fields

When all fields are read only, the serializer cannot be used to create new instances.

Let's run our benchmark with the read only serializer:

>>> cProfile.run('for i in range(5000): UserReadOnlyModelSerializer(u).data', sort='tottime')
14540060 function calls (14450060 primitive calls) in 7.407 seconds

 Ordered by: internal time
 ncalls  tottime  percall  cumtime  percall filename:lineno(function)
6090000    0.809    0.000    0.809    0.000 {built-in method builtins.hasattr}
  65000    0.725    0.000    1.516    0.000 functional.py:82(__prepare_class__)
  50000    0.561    0.000    4.182    0.000 field_mapping.py:66(get_field_kwargs)
  55000    0.435    0.000    0.558    0.000 fields.py:319(__init__)
 840000    0.330    0.000    0.346    0.000 {built-in method builtins.getattr}
 210000    0.294    0.000    0.688    0.000 trans_real.py:275(gettext)
   5000    0.282    0.000    6.510    0.001 serializers.py:992(get_fields)
  75000    0.220    0.000    1.989    0.000 functional.py:191(wrapper)
1305000    0.200    0.000    0.228    0.000 {built-in method builtins.isinstance}
  50000    0.182    0.000    4.531    0.000 serializers.py:1197(build_standard_field)
  50000    0.145    0.000    0.259    0.000 serializers.py:1310(include_extra_kwargs)
  55000    0.133    0.000    0.696    0.000 text.py:14(capfirst)
  50000    0.127    0.000    2.377    0.000 field_mapping.py:46(needs_label)
 210000    0.119    0.000    0.145    0.000 gettext.py:451(gettext)

Only 7.4 seconds. A 40% improvement compared to the writable ModelSerializer.

In the benchmark's output we can see a lot of time is being spent in field_mapping.py and fields.py. These are related to the inner workings of the ModelSerializer. In the serialization and initialization process the ModelSerializer is using a lot of metadata to construct and validate the serializer fields, and it comes at a cost.

"Regular" `Serializer`

In the next benchmark, we wanted to measure exactly how much the ModelSerializer "costs" us. Let's create a "regular" Serializer for the User model:

from rest_framework import serializers

class UserSerializer(serializers.Serializer):
    id = serializers.IntegerField()
    last_login = serializers.DateTimeField()
    is_superuser = serializers.BooleanField()
    username = serializers.CharField()
    first_name = serializers.CharField()
    last_name = serializers.CharField()
    email = serializers.EmailField()
    is_staff = serializers.BooleanField()
    is_active = serializers.BooleanField()
    date_joined = serializers.DateTimeField()

Running the same benchmark using the "regular" serializer:

>>> cProfile.run('for i in range(5000): UserSerializer(u).data', sort='tottime')
3110007 function calls (3010007 primitive calls) in 2.101 seconds

Ordered by: internal time
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    55000    0.329    0.000    0.430    0.000 fields.py:319(__init__)
105000/5000    0.188    0.000    1.247    0.000 copy.py:132(deepcopy)
    50000    0.145    0.000    0.863    0.000 fields.py:626(__deepcopy__)
    20000    0.093    0.000    0.320    0.000 fields.py:762(__init__)
   310000    0.092    0.000    0.092    0.000 {built-in method builtins.getattr}
    50000    0.087    0.000    0.125    0.000 fields.py:365(bind)
     5000    0.072    0.000    1.934    0.000 serializers.py:508(to_representation)
    55000    0.055    0.000    0.066    0.000 fields.py:616(__new__)
     5000    0.053    0.000    1.204    0.000 copy.py:268(_reconstruct)
   235000    0.052    0.000    0.052    0.000 {method 'update' of 'dict' objects}
    50000    0.048    0.000    0.097    0.000 fields.py:55(is_simple_callable)
   260000    0.048    0.000    0.075    0.000 {built-in method builtins.isinstance}
    25000    0.047    0.000    0.051    0.000 deconstruct.py:14(__new__)
    55000    0.042    0.000    0.057    0.000 copy.py:252(_keep_alive)
    50000    0.041    0.000    0.197    0.000 fields.py:89(get_attribute)
     5000    0.037    0.000    1.459    0.000 serializers.py:353(fields)

Here is the leap we were waiting for!

The "regular" serializer took only 2.1 seconds. That's 60% faster than the read only ModelSerializer, and a whooping 85% faster than the writable ModelSerializer.

At this point it become obvious that the ModelSerializer does not come cheap!

Read Only "regular" `Serializer`

In the writable ModelSerializer a lot of time was spent on validations. We were able to make it faster by marking all fields as read only. The "regular" serializer does not define any validation, so marking fields as read only is not expected to be faster. Let's make sure:

from rest_framework import serializers

class UserReadOnlySerializer(serializers.Serializer):
    id = serializers.IntegerField(read_only=True)
    last_login = serializers.DateTimeField(read_only=True)
    is_superuser = serializers.BooleanField(read_only=True)
    username = serializers.CharField(read_only=True)
    first_name = serializers.CharField(read_only=True)
    last_name = serializers.CharField(read_only=True)
    email = serializers.EmailField(read_only=True)
    is_staff = serializers.BooleanField(read_only=True)
    is_active = serializers.BooleanField(read_only=True)
    date_joined = serializers.DateTimeField(read_only=True)

And running the benchmark for a user instance:

>>> cProfile.run('for i in range(5000): UserReadOnlySerializer(u).data', sort='tottime')
3360009 function calls (3210009 primitive calls) in 2.254 seconds

Ordered by: internal time
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    55000    0.329    0.000    0.433    0.000 fields.py:319(__init__)
155000/5000    0.241    0.000    1.385    0.000 copy.py:132(deepcopy)
    50000    0.161    0.000    1.000    0.000 fields.py:626(__deepcopy__)
   310000    0.095    0.000    0.095    0.000 {built-in method builtins.getattr}
    20000    0.088    0.000    0.319    0.000 fields.py:762(__init__)
    50000    0.087    0.000    0.129    0.000 fields.py:365(bind)
     5000    0.073    0.000    2.086    0.000 serializers.py:508(to_representation)
    55000    0.055    0.000    0.067    0.000 fields.py:616(__new__)
     5000    0.054    0.000    1.342    0.000 copy.py:268(_reconstruct)
   235000    0.053    0.000    0.053    0.000 {method 'update' of 'dict' objects}
    25000    0.052    0.000    0.057    0.000 deconstruct.py:14(__new__)
   260000    0.049    0.000    0.076    0.000 {built-in method builtins.isinstance}

As expected, marking the fields as readonly didn't make a significant difference compared to the "regular" serializer. This reaffirms that the time was spent on validations derived from the model's field definitions.

Results Summary

Here is a summary of the results so far:

serializer	seconds
`UserModelSerializer`	12.818
`UserReadOnlyModelSerializer`	7.407
`UserSerializer`	2.101
`UserReadOnlySerializer`	2.254
`serialize_user`	0.034

Why is This Happening?

A lot of articles were written about serialization performance in Python. As expected, most articles focus on improving DB access using techniques like select_related and prefetch_related. While both are valid ways to improve the overall response time of an API request, they don't address the serialization itself. I suspect this is because nobody expects serialization to be slow.

Prior Work

Other articles that do focus solely on serialization usually avoid fixing DRF, and instead motivate new serialization frameworks such as marshmallow and serpy. There is even a site devoted to comparing serialization formats in Python. To save you a click, DRF always comes last.

In late 2013, Tom Christie, the creator of Django Rest Framework, wrote an article discussing some of DRF's drawbacks. In his benchmarks, serialization accounted for 12% of the total time spend on processing a single request. In the summary, Tom recommends to not always resort to serialization:

4. You don't always need to use serializers.

For performance critical views you might consider dropping the serializers entirely and simply use .values() in your database queries.

As we see in a bit, this is solid advice.

Fixing Django's `lazy`

In the first benchmark using ModelSerializer we saw a significant amount of time being spent in functional.py, and more specifically in the function lazy.

The function lazy is used internally by Django for many things such as verbose names, templates etc. The source describes lazy as follows:

Encapsulate a function call and act as a proxy for methods that are called on the result of that function. The function is not evaluated until one of the methods on the result is called.

The lazy function does its magic by creating a proxy of the result class. To create the proxy, lazy iterates over all attributes and functions of the result class (and its super-classes), and creates a wrapper class which evaluates the function only when its result is actually used.

For large result classes, it can take some time to create the proxy. So, to speed things up, lazy caches the proxy. But as it turns out, a small oversight in the code completely broke the cache mechanism, making the lazy function very very slow.

To get a sense of just how slow lazy is without proper caching, let's use a simple function which returns an str (the result class), such as upper. We choose str because it has a lot of methods, so it should take a while to set up a proxy for it.

To establish a baseline, we benchmark using str.upper directly, without lazy:

>>> import cProfile
>>> from django.utils.functional import lazy
>>> upper = str.upper
>>> cProfile.run('''for i in range(50000): upper('hello') + ""''', sort='cumtime')

 50003 function calls in 0.034 seconds

 Ordered by: cumulative time

 ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1    0.000    0.000    0.034    0.034 {built-in method builtins.exec}
     1    0.024    0.024    0.034    0.034 <string>:1(<module>)
 50000    0.011    0.000    0.011    0.000 {method 'upper' of 'str' objects}
     1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

Now for the scary part, the exact same function but this time wrapped with lazy:

>>> lazy_upper = lazy(upper, str)
>>> cProfile.run('''for i in range(50000): lazy_upper('hello') + ""''', sort='cumtime')

 4900111 function calls in 1.139 seconds

 Ordered by: cumulative time

 ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      1    0.000    0.000    1.139    1.139 {built-in method builtins.exec}
      1    0.037    0.037    1.139    1.139 <string>:1(<module>)
  50000    0.018    0.000    1.071    0.000 functional.py:160(__wrapper__)
  50000    0.028    0.000    1.053    0.000 functional.py:66(__init__)
  50000    0.500    0.000    1.025    0.000 functional.py:83(__prepare_class__)
4600000    0.519    0.000    0.519    0.000 {built-in method builtins.hasattr}
  50000    0.024    0.000    0.031    0.000 functional.py:106(__wrapper__)
  50000    0.006    0.000    0.006    0.000 {method 'mro' of 'type' objects}
  50000    0.006    0.000    0.006    0.000 {built-in method builtins.getattr}
     54    0.000    0.000    0.000    0.000 {built-in method builtins.setattr}
     54    0.000    0.000    0.000    0.000 functional.py:103(__promise__)
      1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

No mistake! Using lazy it took 1.139 seconds to turn 5,000 strings uppercase. The same exact function used directly took only 0.034 seconds. That is 33.5 faster.

This was obviously an oversight. The developers were clearly aware of the importance of caching the proxy. A PR was issued, and merged shortly after (diff here). Once released, this patch is supposed to make Django overall performance a bit better.

Fixing Django Rest Framework

DRF uses lazy for validations and fields verbose names. When all of these lazy evaluations are put together, you get a noticeable slowdown.

The fix to lazy in Django would have solved this issue for DRF as well after a minor fix, but nonetheless, a separate fix to DRF was made to replace lazy with something more efficient.

To see the effect of the changes, install the latest of both Django and DRF:

(venv) $ pip install git+https://github.com/encode/django-rest-framework
(venv) $ pip install git+https://github.com/django/django

After applying both patches, we ran the same benchmark again. These are the results side by side:

serializer	before	after	% change
`UserModelSerializer`	12.818	5.674	-55%
`UserReadOnlyModelSerializer`	7.407	5.323	-28%
`UserSerializer`	2.101	2.146	+2%
`UserReadOnlySerializer`	2.254	2.125	-5%
`serialize_user`	0.034	0.034	0%

To sum up the results of the changes to both Django and DRF:

Serialization time for writable ModelSerializer was cut by half.
Serialization time for a read only ModelSerializer was cut by almost a third.
As expected, there is no noticeable difference in the other serialization methods.

Takeaway

Our takeaways from this experiment were:

Take away

Upgrade DRF and Django once these patches make their way into a formal release.

Both PR's were merged but not yet released.

Take away

In performance critical endpoints, use a "regular" serializer, or none at all.

We had several places where clients were fetching large amounts or data using an API. The API was used only for reading data from the server, so we decided to not use a Serializer at all, and inline the serialization instead.

Take away

Serializer fields that are not used for writing or validation, should be read only.

As we've seen in the benchmarks, the way validations are implemented makes them expensive. Marking fields as read only eliminate unnecessary additional cost.

Bonus: Forcing Good Habits

To make sure developers don't forget to set read only fields, we added a Django check to make sure all ModelSerializers set read_only_fields:

# common/checks.py

import django.core.checks

@django.core.checks.register('rest_framework.serializers')
def check_serializers(app_configs, **kwargs):
    import inspect
    from rest_framework.serializers import ModelSerializer
    import conf.urls  # noqa, force import of all serializers.

    for serializer in ModelSerializer.__subclasses__():

        # Skip third-party apps.
        path = inspect.getfile(serializer)
        if path.find('site-packages') > -1:
            continue

        if hasattr(serializer.Meta, 'read_only_fields'):
            continue

        yield django.core.checks.Warning(
            'ModelSerializer must define read_only_fields.',
            hint='Set read_only_fields in ModelSerializer.Meta',
            obj=serializer,
            id='H300',
        )

With this check in place, when a developer adds a serializer she must also set read_only_fields. If the serializer is writable, read_only_fields can be set to an empty tuple. If a developer forgets to set read_only_fields, she gets the following error:

$ python manage.py check
System check identified some issues:

WARNINGS:
<class 'serializers.UserSerializer'>: (H300) ModelSerializer must define read_only_fields.
    HINT: Set read_only_fields in ModelSerializer.Meta

System check identified 1 issue (4 silenced).

We use Django checks a lot to make sure nothing falls through the cracks. You can find many other useful checks in this article about how we use the Django system check framework.

How to Let Google Know of Other Languages in Your Django Site

2019-05-07T00:00:00+03:00

If you have a public facing Django site in multiple languages, you probably want to let Google and other search engines know about it.

Linguistic map of the world (source)

Multi-Language Django Site

Django has a very extensive framework to serve sites in multiple languages. The least amount of setup necessary to add additional languages to a Django site are these.

Activate the i18n framework in settings.py:

# settings.py

USE_I18N = True

Define the supported languages:

# settings.py

from django.utils.translation import gettext_lazy as _

LANGUAGES = [
    ('en', _('English')),
    ('he', _('Hebrew')),
]

Set the default language:

# settings.py

LANGUAGE_CODE = 'en'

Add LocaleMiddleware:

# settings.py

MIDDLEWARE = [
    # ...
    'django.middleware.locale.LocaleMiddleware',
    # ...
]

Use gettext to mark texts for translation:

# app/views.py

from django.utils.translation import gettext_lazy as _
from django.http import HttpResponse

def about(request) -> HttpResponse:
    return HttpResponse(_('Hello!'))

Generate the translation files:

$ python manage.py makemessages

Translate the text:

msgid "Hello!"
msgstr "שלום!"

Compile the translation files:

$ python manage.py compilemessages

Serve views in multiple languages using i18n_patterns:

# urls.py

from django.conf.urls.i18n import i18n_patterns
from django.conf.urls import url

from . import views


urlpatterns = i18n_patterns(
    url(r'^about$', views.about, name='about'),
)

Make sure that it works:

$ curl http://localhost:8000/en/about
Hello!

$ curl http://localhost:8000/he/about
שלום!

This is it!

There are a few additional steps like adding a view to switch the language, but all in all, your multi-language Django site is ready to go!

Link to Other Languages Using `hreflang`

To let search engines know a page is available in a different language, you can use a special link tag:

<link rel="alternate" hreflang="en" href="https://example.com/en" />

The tag has the following attributes:

hreflang: Language code of the linked page.
href: Link to the page in the specified language.

According to Google's guidelines, and the information in Wikipedia, these are the rules we need to follow:

Use absolute URLs, including the schema.
Link must be valid, and the linked page should be in the specified language.
List all languages, including the current one.
If language X links to language Y, language Y should link back to language X.

To implement the following in Django, start by listing the available languages in a template, and set the language code in the hreflang attribute:

{% load i18n %}

{% get_available_languages as LANGUAGES %}
{% for language_code, language_name in LANGUAGES %}
<link
    rel="alternate"
    hreflang="{{ language_code }}"
    href="TODO" />
{% endfor %}

The next step is to add localized links for each language. It took some digging, but it turns out Django already has a function called translate_url we can use:

>>> from django import urls
>>> from django.utils import translation
>>> translation.activate('en')
>>> reverse('about')
'/en/about'
>>> urls.translate_url('/en/about', 'he')
'/he/about'

The function translate_url accepts a URL and a language, and returns the URL in that language. In the example above, we activated the English language and got the a URL prefixed with /en.

The guidelines require absolute URLs.

Let's make sure translate_url can handle absolute URLs as well:

>>> urls.translate_url('https://example.com/en/about', 'he')
'https://example.com/he/about'

Great! translate_url can "translate" absolute URLs.

How about URLs with query parameters or hash?

>>> urls.translate_url('https://example.com/en/about?utm_source=search#top', 'he')
'https://example.com/he/about?utm_source=search#top'

Cool, it worked too!

NOTE: It doesn't make much sense to have a page URL with query params and hashes in a place like a link tag (or canonical for that matter). The reason I mention it is because it might be useful for deep linking into other pages.

This is basically all that we need. But, translate_url has some limitations that are worth knowing.

Translate a non-localized URL:

>>> urls.translate_url('/about', 'en')
'/about'

If you use the built-in LocaleMiddleware and try to navigate to /about, Django will redirect you to the page in the current language. translate_url is unable to do the same.

good to know

translate_url cannot "translate" a non-localized URL (even though it might exist).

What about translating a URL already in a language which is not the current language?

>>> translation.activate('en')
>>> urls.translate_url('/he/about', 'en')
'/he/about'

Nope, can't do that either.

good to know

translate_url can only translate localized urls in the current language.

If you look at the implementation of translate_url this restriction becomes clear:

# django/urls/base.py

def translate_url(url, lang_code):
    """
    Given a URL (absolute or relative), try to get its translated version in
    the `lang_code` language (either by i18n_patterns or by translated regex).
    Return the original URL if no translated version is found.
    """
    parsed = urlsplit(url)
    try:
        match = resolve(parsed.path)
    except Resolver404:
        pass
    else:
        if match.namespace
            to_be_reversed = "%s:%s" % (match.namespace, match.url_name)
        else
            to_be_reversed = match.url_name
        with override(lang_code):
            try:
                url = reverse(to_be_reversed, args=match.args, kwargs=match.kwargs)
            except NoReverseMatch:
                pass
            else:
                url = urlunsplit((parsed.scheme, parsed.netloc, url, parsed.query, parsed.fragment))
    return url

Django first tries to resolve the URL path. This is Django's way of checking if the URL is valid. Only if the URL is valid, it is split into parts, and reversed in the desired language.

`translate_url` template tag

Now that we know how to "translate" URLs to different languages, we need to be able to use it in a template. Django provides us with a way to define custom template tags and filters.

Let's add a custom template tag for translate_url :

# app/templatetags/urls.py

from typing import Optional, Any

from django import urls


register = template.Library()


@register.simple_tag(takes_context=True)
def translate_url(context: Dict[str, Any], language: Optional[str]) -> str:
    """Get the absolute URL of the current page for the specified language.

    Usage:
        {% translate_url 'en' %}
    """
    url = context['request'].build_absolute_uri()
    return urls.translate_url(url, language)

Our translate_url template tag takes context. This is necessary if we want to provide an absolute URL. We use build_absolute_uri to grab the absolute URL from the request.

The tag also accepts the target language code to translate the URL to, and uses translate_url to generate the translated URL.

With our new template tag, we can fill in the blanks in the previous implementation:

{% load i18n urls %}

{% get_available_languages as LANGUAGES %}
{% for language_code, language_name in LANGUAGES %}
<link
    rel="alternate"
    hreflang="{{ language_code }}"
    href="{% translate_url language_code %}" />
{% endfor %}

Using `x-default` for the default language

The guidelines include another recommendation:

The reserved value hreflang="x-default" is used when no other language/region matches the user's browser setting. This value is optional, but recommended, as a way for you to control the page when no languages match. A good use is to target your site's homepage where there is a clickable map that enables the user to select their country.

So it's also a good idea to add a link to some default language. If, for example, we want to make our default language English, we can add the following to the snippet above:

{% load i18n urls %}

{% get_available_languages as LANGUAGES %}
{% for language_code, language_name in LANGUAGES %}
<link
    rel="alternate"
    hreflang="{{ language_code }}"
    href="{% translate_url language_code %}" />
{% endfor %}
<link
    rel="alternate"
    hreflang="x-default"
    href="{% translate_url en %}" />

When we setup our Django project, we already defined a default language. Instead of hard-coding English (or any other language for that matter), we want to use the LANGUAGE_CODE defined in settings.py.

To use values from settings.py in templates, we can use an old trick we used in the past to visually distinguish between environments in Django admin. It's a simple context processor that exposes specific values from settings.py to templates through the request context:

# app/context_processor.py

from typing import Dict, Any

from django.conf import settings

def from_settings(request) -> Dict[str, Any]:
    return {
        attr: getattr(settings, attr, None)
        for attr in (
            'LANGUAGE_CODE',
        )
    }

To register the context processor, add the following in settings.py:

# settings.py

TEMPLATES = [{
    # ...
    'OPTIONS': {
        'context_processors': [
            #...
            'app.context_processors.from_settings',
        ],
        #...
    }
]}

Now that we have access to LANGUAGE_CODE in the template, we can really complete our snippet:

{% load i18n urls %}

{% get_available_languages as LANGUAGES %}
{% for language_code, language_name in LANGUAGES %}
<link
    rel="alternate"
    hreflang="{{ language_code }}"
    href="{% translate_url language_code %}" />
{% endfor %}
<link
    rel="alternate"
    hreflang="x-default"
    href="{% translate_url LANGUAGE_CODE %}" />

The rendered markup for the about page looks like this:

<link
    rel="alternate"
    hreflang="en"
    href="https://example.com/en/about" />
<link
    rel="alternate"
    hreflang="he"
    href="https://example.com/he/about" />
<link
    rel="alternate"
    hreflang="x-default"
    href="https://example.com/en/about" />

Final Words

Hopefully this short article helped you gain a better understanding of how search engines can identify different languages in your Django site. In the process, you might have also picked up on some little tricks to manipulate localized URLs in Django.

How to Create an Index in Django Without Downtime

2019-04-10T00:00:00+03:00

If you maintain a Django site with a decent traffic, you probably need to deal with graceful migrations. With the help of the RealPython team, I wrote an article about one the most common problems in Django migrations: How to create an index without causing downtime.

In the article I explain what atomic and reversible migrations are, how to execute "raw" SQL in migrations the right way, and how using a little known migration command called SeparateDatabaseAndState, you can completely alter the Django migrations built-in behavior.

Read "How to Create an Index in Django Without Downtime" on RealPython ≫

How to Create an Index in Django Without Downtime

How to Use Grouping Sets in Django

2019-03-10T00:00:00+02:00

I recently had the pleasure of optimizing an old dashboard. The solution we came up with required some advanced SQL that Django does not support out of the box. In this article I present the solution, how we got to it, and a word of caution.

advanced SQL

This article covers advanced topics in SQL aggregation. If you need to perform GROUP BY in Django ORM check out Understand Group by in Django with SQL.

The Dashboard

The dashboard is of a sales model. It includes a simple table with metrics grouped by merchants and their devices, and a summary line.

The code to produce the table looks roughly like this:

from django.db.models import Sum, Count, Avg, Q

metrics =  {
    'total': Count('id'),
    'avg_charged_amount': Avg('charged_amount'),
    'unique_users': Count('user', distinct=True),
}

results = (
    Sale.objects
    .values('merchant', 'device')
    .annotate(**metrics)
)

The code to produce the summary line uses the same metrics, and looks likes this:

summary =  Sale.objects.annotate(**metrics)

Our admin gets a nice dashboard that looks roughly like this:

A summary line in Django Admin page

see also

On how to create the dashboard above see How to Turn Django Admin Into a Lightweight Dashboard

The Problem

The dashboard was working well for about three years. We got good response times and accurate information. However, as data piled up, performance has degraded to a point where the page became unusable.

To analyze the problem, we inspected the SQL, and timed it. The query to produce the table looks like this:

SELECT
    merchant,
    device,
    COUNT(id) AS total,
    AVG(charged_amount) AS avg_charged_amount,
    COUNT(DISTINCT user_id) AS unique_users
FROM
    sale
GROUP BY
    merchant,
    device

At worst, this query took about 30s to complete.

The next query executed by the dashboard was used to produce the summary line:

SELECT
    COUNT(id) AS total,
    AVG(charged_amount) AS avg_charged_amount,
    COUNT(DISTINCT user_id) AS unique_users
FROM
    sale

This query took roughly the same time, ~30s. Together, at their worst, the two queries took more than a minute to complete.

Aggregate in Memory

Both queries process the exact same data, the only difference is the GROUP BY key. The first query produces results at the merchant and device level, the second produces the same aggregates for the entire dataset.

The first thought that came to mind was to calculate the summary by aggregating the results in-memory.

The first metric, total, is easy to calculate:

summary_total = sum(r['total'] for r in results)

The second metric is the average charged amount. We can't just sum the average for each device and merchant, we need additional information.

To calculate the average charged amount for all merchants and devices, we need to divide the total charged amount by the number of sales. We already have the number of sales, so we need to add a metric for the total charged amount:

metrics =  {
    'total': Count('id'),
    'avg_charged_amount': Avg('charged_amount'),
    'total_charged_amount': Sum('charged_amount'),
    'unique_users': Count('user', distinct=True),
}

Now that we have both total and total_charged_amount, we can compute avg_charged_amount:

summary_total = sum(r['total'] for r in results)
summary_total_charged_amount = sum(r['total_charged_amount'] for r in results)
summary_avg_charged_amount = summary_total_charged_amount / summary_total

We have one metric left, unique_users. This metric counts the unique number of users that visited each device at each merchant. The same user can visit several devices at different merchants. If we sum unique_users we won't get the correct metric for the entire set.

It's impossible to accurately compute distinct values from aggregated results, so the solution must be in the database.

Aggregating in the Database

Most SQL implementations provide several useful functions to aggregate data at different levels.

Database Support

Throughout this article I use PostgreSQL. Similar functions are available in Oracle, MySQL and MSSQL. As far as I know, SQLite has no support for the functions I'm about to use.

Let's start with some data:

CREATE TABLE sale (
    id              integer,
    merchant        varchar,
    device          varchar,
    user_id         integer,
    charged_amount  integer,
    sold_at         timestamp
);

INSERT INTO sale
    (id, merchant,  device, user_id, charged_amount, sold_at)
VALUES
    (1,  'Walmart', 'D1',   1,       54,             NOW() + INTERVAL '1 days'),
    (2,  'Walmart', 'D1',   1,       100,            NOW() + INTERVAL '2 days'),
    (3,  'Walmart', 'D2',   1,       22,             NOW() + INTERVAL '1 days'),
    (4,  'Walmart', 'D2',   3,       14,             NOW() + INTERVAL '3 days'),
    (5,  'Walmart', 'D2',   4,       15,             NOW() + INTERVAL '4 days'),
    (6,  'Walmart', 'D3',   2,       29,             NOW() + INTERVAL '1 days'),
    (7,  'Walmart', 'D3',   2,       45,             NOW() + INTERVAL '3 days'),
    (8,  'Costco',  'D1',   1,       47,             NOW() + INTERVAL '5 days'),
    (9,  'Costco',  'D1',   4,       223,            NOW() + INTERVAL '1 days'),
    (10, 'Costco',  'D1',   5,       67,             NOW() + INTERVAL '2 days'),
    (11, 'Costco',  'D4',   1,       25,             NOW() + INTERVAL '5 days'),
    (12, 'Costco',  'D5',   3,       125,            NOW() + INTERVAL '4 days'),
    (13, 'Costco',  'D5',   4,       150,            NOW() + INTERVAL '1 days');

The query we used in our dashboard produces the following results:

db=# SELECT
    merchant,
    device,
    COUNT(id) AS total,
    AVG(charged_amount) AS avg_charged_amount,
    COUNT(DISTINCT user_id) AS unique_users
FROM
    sale
GROUP BY
    merchant,
    device;

 merchant | device | total |  avg_charged_amount  | unique_users
----------+--------+-------+----------------------+--------------
 Costco   | D1     |     3 | 112.3333333333333333 |            3
 Costco   | D4     |     1 |  25.0000000000000000 |            1
 Costco   | D5     |     2 | 137.5000000000000000 |            2
 Walmart  | D1     |     2 |  77.0000000000000000 |            1
 Walmart  | D2     |     3 |  17.0000000000000000 |            3
 Walmart  | D3     |     2 |  37.0000000000000000 |            1
(6 rows)

The query to produce the summary line:

db=# SELECT
    COUNT(id) AS total,
    AVG(charged_amount) AS avg_charged_amount,
    COUNT(DISTINCT user_id) AS unique_users
FROM
    sale;

 total | avg_charged_amount  | unique_users
-------+---------------------+--------------
    13 | 70.4615384615384615 |            5
(1 row)

Rollup

The first special GROUP BY expression is ROLLUP. As the name suggest, ROLLUP aggregate at the lowest level and up:

db=# SELECT
    device,
    merchant,
    COUNT(*)
FROM
    sale
GROUP BY
    ROLLUP(device, merchant);

 device | merchant | count
--------+----------+-------
        |          |    13
 D3     | Walmart  |     2
 D5     | Costco   |     2
 D1     | Walmart  |     2
 D4     | Costco   |     1
 D1     | Costco   |     3
 D2     | Walmart  |     3
 D2     |          |     3
 D4     |          |     1
 D1     |          |     5
 D5     |          |     2
 D3     |          |     2
(12 rows)

We grouped by two fields, device and merchant, and we got three groups of aggregation:

() all
(device, merchant)
(device)

ROLLUP aggregates "up", so the order of the fields is significant. Let's flip the order of the fields:

db=# SELECT
    device,
    merchant,
    count(*)
FROM
    sale
GROUP BY
    ROLLUP(merchant, device);

 device | merchant | count
--------+----------+-------
        |          |    13
 D4     | Costco   |     1
 D3     | Walmart  |     2
 D1     | Walmart  |     2
 D1     | Costco   |     3
 D5     | Costco   |     2
 D2     | Walmart  |     3
        | Costco   |     6
        | Walmart  |     7

This time we got the following groups:

() all
(merchant, device)
(merchant)

Cube

The next group by expression is most likely borrowed from OLAP, which often mention cubes. The CUBE expression aggregates all possible combinations:

db=# SELECT
    device,
    merchant,
    count(*)
FROM
    sale
GROUP BY
    CUBE(merchant, device);

 device | merchant | count
--------+----------+-------
        |          |    13
 D4     | Costco   |     1
 D3     | Walmart  |     2
 D1     | Walmart  |     2
 D1     | Costco   |     3
 D5     | Costco   |     2
 D2     | Walmart  |     3
        | Costco   |     6
        | Walmart  |     7
 D2     |          |     3
 D4     |          |     1
 D1     |          |     5
 D5     |          |     2
 D3     |          |     2
(14 rows)

The results contain the following groups:

() all
(device, merchant)
(merchant)
(device)

Grouping Sets

Grouping sets allows us to provide the exact groups of aggregation we want. For example, to recreate the results of the ROLLUP above, we can provide the following grouping sets:

db=# SELECT
    device,
    merchant,
    count(*)
FROM
    sale
GROUP BY
    GROUPING SETS (
        (merchant),
        (merchant, device),
        ()
    )

 device | merchant | count
--------+----------+-------
        |          |    13
 D4     | Costco   |     1
 D3     | Walmart  |     2
 D1     | Walmart  |     2
 D1     | Costco   |     3
 D5     | Costco   |     2
 D2     | Walmart  |     3
        | Costco   |     6
        | Walmart  |     7
(9 rows)

Each list of fields inside parentheses in the GROUPING SETS is a group in the result.

Both CUBE and ROLLUP can be implemented using GROUPING SETS. The following table shows the equivalent GROUPING SETS expression for both ROLLUP and CUBE, on two fields a and b:

expression	equivalent grouping sets
`ROLLUP(a, b)`	`GROUPING SETS ((a, b), (a), ())`
`CUBE(a, b)`	`GROUPING SETS ((a, b), (a), (b) ())`

In our original query we had metrics at the merchant and device level, and we wanted to get a summary line. Using GROUPING SETS, this query will look like this:

db=# SELECT
    merchant,
    device,
    COUNT(id) AS total,
    AVG(charged_amount) AS avg_charged_amount,
    COUNT(DISTINCT user_id) AS unique_users
FROM
    sale
GROUP BY
    GROUPING SETS (
        (merchant, device),
        ()
    )

 merchant | device | total |  avg_charged_amount  | unique_users
----------+--------+-------+----------------------+--------------
 Costco   | D1     |     3 | 112.3333333333333333 |            3
 Costco   | D4     |     1 |  25.0000000000000000 |            1
 Costco   | D5     |     2 | 137.5000000000000000 |            2
 Walmart  | D1     |     2 |  77.0000000000000000 |            1
 Walmart  | D2     |     3 |  17.0000000000000000 |            3
 Walmart  | D3     |     2 |  37.0000000000000000 |            1
          |        |    13 |  70.4615384615384615 |            5
(7 rows)

The first 6 lines are similar to the original query. The last line is similar to the results from the summary query we used.

Using GROUPING SETS we get the results we need in just one query instead of two.

Using Grouping Sets in Django

Now that we have the query, we need to find a way to use it with Django. Unfortunately, Django still has no support for grouping sets. On top of that, the query is generated by Django Admin, and it includes predicates from list filters and date hierarchy. We couldn't just use raw SQL.

We need to find a way to modify a given Django QuerySet, and add grouping sets to it.

Since Django has no built-in support for grouping sets, we are forced to manipulate the query. The base query we need to manipulate is the query that Django generates, along with any predicates and annotations added by Django Admin. Eventually, we want to execute the query in the database the same way Django does.

Getting the Query

A nice feature of Django QuerySet is that it provides the generated SQL:

>>> qs = (
>>> ... Sale.objects
>>> ... .values('merchant', 'device')
>>> ... .annotate(**metrics)
>>> )
>>> print(qs.query)
SELECT "sale"."merchant", "sale"."device", COUNT("sale"."id") AS "total",
AVG("sale"."charged_amount") AS "avg_charged_amount",
COUNT(DISTINCT "sale"."user_id") AS "unique_users" FROM "sale"
GROUP BY "sale"."merchant", "sale"."device"

This is a simple query, can we execute it directly in the database?

>>> from django.db import connection
>>>
>>> with connection.cursor() as cursor:
>>> ... cursor.execute(str(qs.query))
>>> ... results = cursor.fetchall()
>>> results
[('Costco', 'D1', 3, 112.3333333333333333, 3),
('Costco', 'D4', 1, 25.0000000000000000, 1),
...
('Walmart', 'D3', 2, 37.0000000000000000, 1)]

This looks like something we can work with, let's dig deeper...

As mentioned before, the QuerySet is generated by Django Admin, and it might include predicates for list filters and date hierarchy. Let's try to execute a query with a predicate on the sold_at date field:

>>> import datetime
>>> from django.db import connection
>>> from django.utils import timezone
>>>
>>> qs = qs.filter(sold_at__lt=timezone.now())
>>> with connection.cursor() as cursor:
>>> ... cursor.execute(str(qs.query))
*** django.db.utils.ProgrammingError: syntax error at or near "00"
LINE 1: ..."sale"."sold_at" >= 2019-03-02 00:00:00+0...

Looks like Django is unable to execute the query as is. The reason for that is that the text generated by str(qs.query) is just a text representation of the query. Under the hood, Django uses proper bind variables (might also be known as substitution variables) to avoid SQL injection.

Much of the Django ORM QuerySet logic is carried out by an internal class called Query. The class is not documented. The only place to learn about it is in the source. One promising function of Query is sql_with_params. Let's use it on the query above, and see what we get:

>>> qs = qs.filter(sold_at__lt=timezone.now())
>>> sql, params = qs.query.sql_with_params()
>>> sql
SELECT "sale"."merchant", "sale"."device", COUNT("sale"."id") AS "total",
AVG("sale"."charged_amount") AS "avg_charged_amount",
COUNT(DISTINCT "sale"."user_id") AS "unique_users" FROM "sale"
WHERE "sale"."sold_at" < %s
GROUP BY "sale"."merchant", "sale"."device" ORDER BY "sale"."merchant", "sale"."device"
>>> params
(datetime.datetime(2019, 3, 2, 0, 0, tzinfo=<UTC>))

The function sql_with_params returns a tuple. The first argument is the SQL query. The second, is a list of parameters to that query.

The keen-eyed might have spotted the placeholder %s in the query text:

WHERE "sale"."sold_at" < %s

This placeholder corresponds to the parameter we got in the second argument. Let's try to execute the query with the placeholder, and the params:

>>> import datetime
>>> from django.db import connection
>>> from django.utils import timezone
>>>
>>> qs = qs.filter(sold_at__lt=timezone.now())
>>> sql, params = qs.query.sql_with_params()
>>>
>>> with connection.cursor() as cursor:
>>> ... cursor.execute(sql, params)
>>> ... results = cursor.fetchall()
>>> results
[('Costco', 'D1', 3, 112.3333333333333333, 3),
('Costco', 'D4', 1, 25.0000000000000000, 1),
...
('Walmart', 'D3', 2, 37.0000000000000000, 1)]

Great! We are now able to execute a query as Django does. We are ready to manipulate the query.

Manipulating the Query

The query generated by Django includes a simple GROUP BY clause:

GROUP BY "sale"."merchant", "sale"."device" ORDER BY "sale"."merchant", "sale"."device"

We want to replace that with the following group by clause:

GROUP BY GROUPING SETS (("sale"."merchant", "sale"."device"), ()) ORDER BY "sale"."merchant", "sale"."device"

This looks like a job for re.

We want catch the grouped by fields between GROUP BY and ORDER BY, and make them the first group in the GROUPING SET expression. Then, we want add the group () for the summary:

import re

sql_with_grouping_sets = re.sub(
    r'GROUP BY (.*) ORDER',
    r'GROUP BY GROUPING SETS (( \1 ), ()) ORDER',
    sql,
)

Now we can take the modified query, and execute it with the params:

>>> with connection.cursor() as cursor:
>>> ... cursor.execute(sql_with_grouping_sets, params)
>>> ... results = cursor.fetchall()
>>> results
[('Costco', 'D1', 3, 112.3333333333333333, 3),
('Costco', 'D4', 1, 25.0000000000000000, 1),
...
('Walmart', 'D3', 2, 37.0000000000000000, 1)]
(None, None, 13, 70.4615384615384615, 5)]

Lo and behold... We got both the results and the summary line in a single query.

Conclusion

Several important issues to consider about this approach:

Don't do this!: This is as bad as it gets. This approach is a nice exercise, and a great opportunity to explore the ORM internals, but the implementation is too fragile. When using an internal, undocumented API, there is no guaranty it wont change unexpectedly in the future. Having said that, we decided to use this approach in one of our internal admin pages. It's a very specific scenario involving a queryset that's not used for any user facing features. It helped us cut the page response time exactly in half and we are pleased with the result.
Make the sort order deterministic: When using GROUPING SETS (and ROLLUP or CUBE for that matter), you mix more than one level of aggregation in a single query. To be able to fetch the results in a predictable way, it's important to explicitly sort the results. For example, in the query above, to make sure the summary row is the first row, add the following sort order qs.order_by( F('merchant').desc(nulls_last=False) ).

Modeling Polymorphism in Django

2019-01-02T00:00:00+02:00

If you ever added a type, kind or a mode field to a Django model, you probably had to deal with polymorphism at some level. With the great people over at RealPython, I wrote about 5 ways to model polymorphism in Django.

Read "Modeling Polymorphism in Django With Python" on RealPython ≫

Modeling Polymorphism in Django With Python

Optimizing the Django Admin Paginator

2018-11-06T00:00:00+02:00

In almost every project we work on, we use Django admin for support and operations. Over time we experienced an influx of new users and the amount of data we had stored grew rapidly. With a large dataset we started to experience the real cost of some Django admin features.

I often talk about making Django scale but what does it actually mean? It means getting consistent performance regardless of the amount of data. Over the past few years I've written about different approaches to scale and optimize Django admin for large tables:

Things You Must Know About Django admin As Your App Gets Bigger: Included an overview of built-in options to optimize for larger tables.
How to Add Custom Action Buttons to Django Admin: Explained how to make Django admin more suitable for operations.
How to turn Django Admin into a lightweight dashboard: Showed how to add custom pages to the admin with fancy graphs and chart. This was our first dashboard (and we still use it!).
5 ways to make Django Admin safer: Described different measures we took to make our admin more secure (this one also inspired a package).
Scaling Django Admin Date Hierarchy: Identified issues with the date hierarchy and suggested ways to alter the admin behavior to better handle large tables (we also turned it into a package!).
Django Admin Range-Based Date Hierarchy : Looked at the queries generated by the date hierarchy and made them much faster. I'm especially proud of this one because I managed to get it into Django 2.1.
How to add a text filter to Django Admin: We added a new type of filter to save some time when generating list filters.

After all of this, only one problem remained...

The Paginator

The last nail in Django admin's scalability coffin is the paginator:

Count is taking so much time!

Django spent 781ms out of 786ms just to count the rows in the table. That's ~99.4% of the time, just for the paginator!

For reference, the actual query used to fetch the data took only 2.10ms. The reason it's much quicker is that it only needs to fetch one page (notice the LIMIT 100).

What Can We Do?

Making something scale is all about eliminating operations that work on the entire dataset. The paginator has to count the rows to determine how many pages there are. This forces every single page in the admin to scan the entire table.

As the table grows in size this query takes longer and longer to execute. The size of the table has a direct impact on the load time of a single page and this is exactly what we want to avoid.

Django ModelAdmin provides a way to override the paginator. Our initial thought was to implement an entirely different type of paginator - one that doesn't calculate the number of pages, but only shows links to the previous and next pages.

Unfortunately, the paginator is embedded deep into Django admin - Django uses partial templates to render the paginator. This makes it difficult to create a paginator that doesn't count pages.

To provide a different paginator to a ModelAdminwe need to implement a Paginator. The interesting function in the paginator implementation is count:

# django/core/paginator.py

@cached_property
def count(self):
    try:
        return self.object_list.count()
    except (AttributeError, TypeError):
        return len(self.object_list)

This is where the horror happens. Django counts the rows in the queryset (or the object list) and caches the result.

If we want to eliminate the count this is where we need to start.

A Dumb Paginator

Let's take the simplest approach and create a paginator that returns a very large number without actually counting the rows:

# common/paginator.py

from django.core.paginator import Paginator

class DumbPaginator(Paginator):
    """
    Paginator that does not count the rows in the table.
    """
    @cached_property
    def count(self):
        return 9999999999

Let's add it to a ModelAdmin of a large table and see what happens:

# app/admin.py

from common.paginator import DumbPaginator

@admin.register(models.LargeTable)
class LargeTableAdmin(admin.ModelAdmin):
    paginator = DumbPaginator

And the admin page:

Admin page did not count rows

Wow! With the dumb paginator the page now loads in only 4ms. This is impressive but how does the UI looks like?

Pagination UI

We made the paginator think there are 9999999999 results so this is what it shows. Clicking on a page that doesn't exist will open an empty list view.

Getting Creative

We saw that we can provide a custom pagination and eliminate the count. We also know that Django is very attached to this specific paginator template so we can't easily replace it.

At this point it's obvious that if we don't want to work too hard and alter Django's templates, we need to make a compromise. A compromise can be made either in the UI or in accuracy.

I personally don't like to sacrifice accuracy so fast so I tend to make compromises in the UI. Especially in internal projects such as ones implemented with Django admin.

Instead of always eliminating the pagination, what if we just limit the execution time of the count query:

# common/paginator.py

from django.core.paginator import Paginator
from django.db import connection, transaction, OperationalError

class TimeLimitedPaginator(Paginator):
    """
    Paginator that enforces a timeout on the count operation.
    If the operations times out, a fake bogus value is
    returned instead.
    """
    @cached_property
    def count(self):
        # We set the timeout in a db transaction to prevent it from
        # affecting other transactions.
        with transaction.atomic(), connection.cursor() as cursor:
            cursor.execute('SET LOCAL statement_timeout TO 200;')
            try:
                return super().count
            except OperationalError:
                return 9999999999

We let the original paginator count the rows. If the count takes longer than 200ms, the database will kill the query, an OperationError will be raised and we return the fake value. If the count takes less than 200ms it will work as usual.

To check the new approach we set the paginator in the ModelAdmin:

# app/admin.py

from common.paginator import TimeLimitedPaginator

@admin.register(models.LargeTable)
class LargeTableAdmin(admin.ModelAdmin):
    paginator = TimeLimitedPaginator

First let's try it with a big result set. For example, all the rows from year 2018:

Query took longer than 200ms and was killed

Query took longer than 200ms and was killed.

Let's try a shorter period. For example, only records from a single day in 2018:

Query finished in under 200ms and was not killed.

Count took only 2.41ms. It was not killed and we got the regular "paginator" - exactly what we wanted!

source code

The complete code for TimeLimitedPaginator can be found in this gist.

Other Approaches

As discussed in the previous section, compromises can be made in different ways. Some ideas I encountered when I researched this issue were:

Estimate the number of rows based on the database execution plan - in Django 2.1 there is even a new explain function on QuerySet. This approach is very inaccurate, especially with complicated predicates, and might yield unexpected results.
Cache the number of rows - the boring and unimaginative solution. As always with a solution that involves caching, you need to decide when to invalidate the cache which is a problem of it's own.
Completely replace Django's paginator - replacing templates and possibly making adjustments to the change list itself.
I went down this road initially but gave up when I realized it will be difficult to make this "plug and play". I do believe that a proper solution should let users replace the pagination implementation to something like cursor pagination (pagination that does not need to evaluate the entire dataset).

Automating the Boring Stuff in Django Using the Check Framework

2018-06-05T00:00:00+03:00

Every team has a unique development style. Some teams implement localization and require translations. Some teams are more sensitive to database issues and require more careful handling of indexes and constraints.

Existing tools can not always address these specific issues out of the box, so we came up with a way to enforce our own development style using the Django check framework, the inspect and the ast modules from the Python standard library.

Image by Wright Design

Table of Contents

Django checks
Custom Checks in the Real World

Django checks

Django checks are part of the Django System Check framework. To quote the docs:

The system check framework is a set of static checks for validating Django projects. It detects common problems and provides hints for how to fix them. The framework is extensible so you can easily add your own checks.

One check you might be familiar with is this one from Django admin:

SystemCheckError: System check identified some issues:

ERRORS:
<class 'app.admin.BarAdmin>
(admin.E108) The value of 'list_display[3]' refers to 'foo',
which is not a callable, an attribute of 'Bar', or an attribute
or method on 'app.Bar'.

The Django admin developers added a system check to warn developers about fields in the model admin that does not exist in the actual model. In this case the field 'foo' do not exist in model Bar.

Checks are executed by some management commands such as makemigrations and migrate. It's also possible to explicitly run check using manage.py:

$ ./manage.py check

It's a good idea to incorporate check in your CI. If you want to fail the CI on warnings you can do that by setting a flag:

$ ./manage.py check --fail-level=WARNING

A simple example of how Django uses checks can be found in the source code of the model Field checks.

Our first check

Most of our apps are not designated for English speakers so we use translations extensively. We put a lot of focus during code review to make sure everything is translated properly.

One of the main issues that come up during code reviews is that developers often forget to set verbose_name on model fields.

Checking that a field has a verbose name is a pretty straightforward task and we wanted to automate the process of making sure it was set.

To get us started we are going to define a simple customer profile model:

class CustomerProfile(models.Model):

    id = models.PositiveSmallIntegerField(
        primary_key=True,
        verbose_name=_('id'),
    )

    name = models.CharField(
        max_length=100,
    )

    created_by = models.ForeignKey(
        User,
        on_delete=models.PROTECT,
    )

The "name" field does not have verbose_name. Let's see if we can identify that using only the model's _meta:

>>> name_field = CustomerProfile._meta.get_field('name')
>>> name_field.verbose_name
name

It looks like Django did something under to hood to set the verbose_name. Looking at the Field class, there is a function called set_attributes_from_name that populates verbose_name by transforming the name of the field - this is where the verbose_name "name" came from.

Because Django is setting the verbose_name on its own the string "name" will not be picked up by makemessages and will not be added to the po file automatically. This will probably cause the string "name" to go unnoticed. We don't want that.

Also, because Django is populating the field automatically we can't use the model _meta to check if verbose_name was originally set. To do that we need to inspect the actual source code.

Inspecting the code

I didn't use the word inspect for no reason - Python has a module called inspect that we can use to, well, inspect code:

The inspect module provides several useful functions to help get information about live objects such as modules, classes, methods, functions, tracebacks, frame objects, and code objects. For example, it can help you examine the contents of a class, retrieve the source code of a method, extract and format the argument list for a function, or get all the information you need to display a detailed traceback.

Let's see what we can get from inspect:

>>> import inspect
>>> inspect.getsource(CustomerProfile)

"class CustomerProfile(models.Model):\n id = models.PositiveSmallIntegerField(\n
primary_key=True,\n verbose_name=_('Name'),\n )\n name = models.CharField(\n
max_length=100,\n )\n created_by = models.ForeignKey(\n User,\n on_delete=models.PROTECT,\n
)\n\n def __str__(self):\n return self.name\n"

That's pretty exciting. We gave inspect the class and got the source code for that class as text.

Given the source code we could have used some fancy RegExp to parse the code but once again, Python already has us covered.

Parsing the code

Parsing code in Python is done by the ast module:

The ast module helps Python applications to process trees of the Python abstract syntax grammar.

Great! A tree is much easier to work with than text.

Let's use ast to parse the source code of our model:

>>> import inspect
>>> import ast
>>> model_source = inspect.getsource(CustomerProfile)
>>> model_node = ast.parse(model_source)
>>> ast.dump(model_node, False)

Module([
    ClassDef('CustomerProfile',
    [Attribute(Name('models', Load()), 'Model', Load())],
    [],
    [

        Assign(
            [Name('id', Store())],
            Call(
                Attribute(Name('models', Load()), 'PositiveSmallIntegerField', Load()),
                [],
                [
                keyword('primary_key', NameConstant(True)),
                keyword('verbose_name', Call(Name('_', Load()), [Str('Name')], []))
                ]
            )
        ),

        Assign(
            [Name('name', Store())],
            Call(
                Attribute(Name('models', Load()), 'CharField', Load()),
                [],
                [keyword('max_length', Num(100))]
            )
        ),

        Assign(
            [Name('created_by', Store())],
            Call(
                Attribute(Name('models', Load()), 'ForeignKey', Load()),
                [Name('User', Load())],
                [keyword('on_delete', Attribute(Name('models', Load()), 'PROTECT', Load()))]
            )
        ),

        FunctionDef(
            '__str__',
            arguments([arg('self', None)],None,[],[],None,[]),
            [Return(Attribute(Name('self', Load()), 'name', Load()))], [], None
        )
    ],
    []
    )
])

If we look closely at the dump we can identify that our model fields are all Assign nodes.

Let's zoom-in on the "name" field:

Assign(
    [Name('name', Store())],
    Call(
        Attribute(Name('models', Load()), 'CharField', Load()),
        [],
        [keyword('max_length', Num(100))]
    )
)

The model field is an assignment of a Call node (CharField) to a Name node ("name"). The Call node has a list of arguments. In this case we only have one argument "max_length" with the numeric value 100.

Our id field looks like this:

Assign(
    [Name('id', Store())],
    Call(
        Attribute(Name('models', Load()), 'PositiveSmallIntegerField', Load()), [], [
            keyword('primary_key', NameConstant(True)),
            keyword('verbose_name', Call(
               Name('_', Load()), [Str('Name')], []
            )
        )
    ])
)

The id field is also an Assign node with a Name node and a Call node. The id field has two keywords - primary_key and verbose_name, which is the one we are looking for.

Evaluating a Model Field

To evaluate the fields we first need to identify them. We already saw that model fields are Assign nodes but we can't rely on them being the only Assign nodes in the class.

The only thing we can rely on is that at the top level of the class the attribute names are unique. Meaning, if we know there is a field called "name" we can assume the attribute "name" of the class is the field.

Let's join forces with Django model _meta to find the nodes of the model fields:

from django.db.models import FieldDoesNotExist


for node in model_node.body[0].body:
    if not isinstance(node, ast.Assign):
        continue

    if len(node.targets) != 1:
        continue

    if not isinstance(node.targets[0], ast.Name):
        continue

    field_name = node.targets[0].id
    try:
        field = model._meta.get_field(field_name)
    except FieldDoesNotExist:
        continue

   # node is field!

Let's break it down:

Model fields are defined at the top level of the class - we only need to check attributes defined at the top level (no need to "visit" nodes recursively).
Model fields will have a Name target - the name of the field.
Finally, the field we assign will be registered in the Django model as a field.

Now we have the field node and we can check if there is a verbose_name attribute defined.

Let's iterate the keywords and search for verbose_name:

for kw in node.value.keywords:
    if kw.arg == 'verbose_name':
       verbose_name = kw
        break
else:
    verbose_name = None

At this point, if verbose_name is None we know that the attribute was not set and we are ready to issue our first warning!

Issuing Django checks

To issue checks we need to register a function with the check framework:

from django.core import check checks

@checks.register(checks.Tags.models)
def run_custom_checks(app_configs, **kwargs):
    # implement check logic

Inside the function we implement the check logic and return a list of checks.

We want to warn the developer that a field is missing a verbose_name attribute, so once we find a field that has no verbose_name we create a CheckMessage of type Warning:

from django.core.checks import Warning

@checks.register(checks.Tags.models)
def run_custom_checks(app_configs, **kwargs):

    # inspect and parse models...

    return [(
        Warning(
            'Field has no verbose name',
            hint='Set verbose name on field {}.'.format(field.name),
            obj=field,
            id='H001',
        )
    )]

I assigned the code H00X to my warnings (guess why…). For each warning we can also add a hint to inform the developer on how to address the issue raised by the warning.

Putting it all together

To recap what we did so far:

Get the source code for a model using inspect.
Parse the model source code using ast and identify the field nodes.
Examine a field node and check if verbose_name is defined.
Register a function with the check framework and issue a Warning.

The skeleton of a function that checks a single model:

# common/checks.py

def check_model(model):
   """Check a single model.

   Yields (django.checks.CheckMessage)
   """
   model_source = inspect.getsource(model)
   model_node = ast.parse(model_source)

   for node in model_node.body[0].body:

       # Check if node is a model field.

       # Check if field has verbose name defined

       yield Warning(
            'Field has no verbose name',
            hint='Set verbose name on field {}.'.format(field.name),
            obj=field,
            id='H001',
        )

The next step is to implement a single function to iterate over all models, run our checks and register it with the Django check framework:

# common/checks.py

@checks.register(checks.Tags.models)
def check_models(app_configs, **kwargs):
    errors = []
    for app in django.apps.apps.get_app_configs():

        # Skip third party apps.
        if app.path.find('site-packages') > -1:
            continue

        for model in app.get_models():
            for check_message in check_model(model):
                errors.append(check_message)

    return errors

We use a little trick to skip models from third party apps. We assume that when installing third party apps using pip install they are installed in a directory called "site-packages".

The only thing left to do it to import this file somewhere in the code and that's it.

# app/__init__.py

from common.checks import *  # noqa

Let's see our new check in action:

$ ./manage.py check

SystemCheckError: System check identified some issues:

WARNINGS:
app.CustomerProfile.name: (H001) Field has no verbose name
HINT: Set verbose name on the field "name".

System check identified 1 issues (0 silenced).

Exactly what we wanted!

Custom Checks in the Real World

To give a sense of what you can do with Django checks, these are the checks we use in our code base:

H001: Field has no verbose name. This is the example we just saw.
H002: Verbose name should use gettext. Make sure verbose_name is always in the form of verbose_name=_('text'). If the value is not using gettext it will not be translated.
H003: Words in verbose name must be all upper case or all lower case. We decided to use only lower case in verbose names. Using lower case texts we were able to reuse more translations. One exception to the rule is acronyms such as API and ETL. The general rule we ended up with is making sure all words are either all lower or all upper case. For example, "etl run" is valid, "ETL run" is also valid, "Etl Run" is not valid.
H004: Help text should use gettext. Help text is displayed to the user in admin forms and detail views so it should use gettext and be translated.
H005: Model must define class Meta. The translation of the model name is defined in the model Meta class so every model must have a class Meta.
H006: Model has no verbose name. Model verbose names are defined in the class Meta and are displayed to the user in the admin so they should be translated.
H007: Model has no verbose name plural. Plural model names are used in the admin and are displayed to the user so they should be translated.
H008: Must set db_index explicitly on a ForeignKey field. This must be the most useful check we defined. This check forces the developer to explicitly set db_index on every ForeignKey field. I wrote in the past about how a database index is created implicitly for every foreign key field. By making sure the developer is aware of that and making him decide if an index is required or not, you are left with only the indexes you really need!

This is it, go piss off some colleagues!

source code

The complete source code for the checks above can be found in this gist.

9 Django Tips for Working with Databases

2018-01-29T00:00:00+02:00

ORMs offer great utility for developers but abstracting access to the database has its costs. Developers who are willing to poke around the database and change some defaults often find that great improvements can be made.

Aggregation with Filter

Prior to Django 2.0 if we wanted to get something like the total number of users and the total number of active users we had to resort to conditional expressions:

from django.contrib.auth.models import User
from django.db.models import (
    Count,
    Sum,
    Case,
    When,
    Value,
    IntegerField,
)

User.objects.aggregate(
    total_users=Count('id'),
    total_active_users=Sum(Case(
        When(is_active=True, then=Value(1)),
        default=Value(0),
        output_field=IntegerField(),
    )),
)

In Django 2.0 a filter argument to aggregate functions was added to make this a lot easier:

from django.contrib.auth.models import User
from django.db.models import Count, F

User.objects.aggregate(
    total_users=Count('id'),
    total_active_users=Count('id', filter=F('is_active')),
)

Nice, short and sweet.

If you are using PostgreSQL, the two queries will look like this:

SELECT
    COUNT(id) AS total_users,
    SUM(CASE WHEN is_active THEN 1 ELSE 0 END) AS total_active_users
FROM
    auth_users;

SELECT
    COUNT(id) AS total_users,
    COUNT(id) FILTER (WHERE is_active) AS total_active_users
FROM
    auth_users;

The second query uses the FILTER (WHERE …) clause.

QuerySet Results as `namedtuple`

I'm a big fan of namedtuples and apparently starting Django 2.0 so is the ORM.

In Django 2.0 a new attribute was added to values_list called named. Setting named to true will return the queryset as a list of namedtuples:

>>> user.objects.values_list(
    'first_name',
    'last_name',
)[0]
('Haki', 'Benita')

>>> user_names = User.objects.values_list(
    'first_name',
    'last_name',
    named=True,
)

>>> user_names[0]
Row(first_name='Haki', last_name='Benita')

>>> user_names[0].first_name
'Haki'

>>> user_names[0].last_name
'Benita'

Custom Functions

Django ORM is very powerful and feature-rich but it can't possibly keep up with all database vendors. Luckily the ORM lets us extend it with custom functions.

Say we have a Report model with a duration field. We want to find the average duration of all reports:

>>> from django.db.models import Avg
>>> Report.objects.aggregate(avg_duration=Avg('duration'))
{'avg_duration': datetime.timedelta(0, 0, 55432)}

That's great, but average alone tells us very little. Let's try to fetch the standard deviation as well:

>>>> from django.db.models import Avg, StdDev
>>> Report.objects.aggregate(
...     avg_duration=Avg('duration'),
...     std_duration=StdDev('duration'),
... )

ProgrammingError: function stddev_pop(interval) does not exist
LINE 1: SELECT STDDEV_POP("report"."duration") AS "std_dura...
               ^
HINT:  No function matches the given name and argument types.
You might need to add explicit type casts.

Oops… PostgreSQL does not support stddev on an interval field - we need to convert the interval to a number before we can apply STDDEV_POP to it.

One option is extracting epoch from the duration:

db=# SELECT
    AVG(duration),
    STDDEV_POP(EXTRACT(EPOCH FROM duration))
FROM
    report;

        avg       |    stddev_pop
------------------+------------------
   00:00:00.55432 | 1.06310113695549

(1 row)

So how can we implement this in Django? You guessed it - a custom function:

# common/db.py

from django.db.models import Func

class Epoch(Func):
   function = 'EXTRACT'
   template = "%(function)s('epoch' from %(expressions)s)"

And use our new function like this:

>>> from django.db.models import Avg, StdDev, F
>>> from common.db import Epoch
>>> Report.objects.aggregate(
...     avg_duration=Avg('duration'),
...     std_duration=StdDev(Epoch(F('duration'))),
... )

{'avg_duration': datetime.timedelta(0, 0, 55432),
'std_duration': 1.06310113695549}

Notice the use of the F expression in the call to Epoch.

Statement Timeout

This is probably the easiest and most important tip I can give. We are all humans and we make mistakes. We can't possibly handle each and every edge case so we must set boundaries.

Unlike other non-blocking app servers such as Tornado, asyncio or even Node, Django usually uses synchronous worker processes. This means that when a user executes a long running operation, the worker process is blocked and no one else can use it until it is done.

I'm sure no one is really running Django in production with just one worker process but we still want to make sure a single query is not hogging too much resources for too long.

In most Django apps the majority of time is spent waiting for database queries. So, setting a timeout on SQL queries is a good place to start.

We like setting a global timeout in our wsgi.py file like this:

# wsgi.py

from django.db.backends.signals import connection_created
from django.dispatch import receiver

@receiver(connection_created)
def setup_postgres(connection, **kwargs):
    if connection.vendor != 'postgresql':
        return

    # Timeout statements after 30 seconds.
    with connection.cursor() as cursor:
        cursor.execute("SET statement_timeout TO 30000;")

Why wsgi.py? This way it only affects worker processes and not out-of-band analytic queries, cron tasks, etc.

Hopefully, you are using persistent database connections, so this per-connection setup should not add overhead to each request.

The timeout can also be set at the user level:

db=#> alter user app_user set statement_timeout TO 30000;

ALTER ROLE

SIDE NOTE: The other common place we spent a lot of time at is networking. So make sure when you call a remote service to always set a timeout:

import requests

response = requests.get(
    'https://api.slow-as-hell.com',
    timeout=3000,
)

LIMIT

This is somewhat related to the last point about setting boundaries. Sometimes we want to let users produce reports and maybe export them to a spreadsheet. These types of views are usually the immediate suspects for any weird behavior in production.

It's not uncommon to encounter a user that thinks it's reasonable to export all sales since the dawn of time in the middle of the work day. It's also not uncommon for this same user to open another tab and try again when the first attempt "got stuck".

This is where LIMIT comes in.

Let's limit a certain query to no more than 100 rows:

# bad example

>>> data = list(Sale.objects.all())[:100]

This is the worst thing you can do. You just fetched all gazillion rows into memory just to return the first 100.

Let's try again:

data = Sale.objects.all()[:100]

This is better. Django will use the limit clause in the SQL to fetch only 100 rows.

Now let's say we added the limit, the users are under control and all is good. We still have one problem - the user asked for all the sales and we gave them 100. The user now thinks there are only 100 sales - this is wrong.

Instead of blindly returning the first 100 rows, let's make sure that if there are more than 100 rows (normally after filtering) we throw an exception:

LIMIT = 100

if Sales.objects.count() > LIMIT:
    raise ExceededLimit(LIMIT)

return Sale.objects.all()[:LIMIT]

This will work but we just added another query.

Can we do better? I think we can:

LIMIT = 100

data = Sale.objects.all()[:(LIMIT + 1)]
if len(data) > LIMIT:
    raise ExceededLimit(LIMIT)

return data

Instead of fetching 100 rows, we fetch 100 + 1 = 101 rows. If the 101 row exists it's enough for us to know there is more than 100 rows. Or in other words, fetching LIMIT + 1 rows is the least we need to make sure there are no more than LIMIT rows in the query result.

Remember the LIMIT + 1 trick, it can come pretty handy at times.

Select for update … of

This one we learned the hard way. We started getting errors in the middle of the night about transactions timing out due to locks in the database.

A common pattern for manipulating a transaction in our code would look like this:

from django.db import transaction as db_transaction

# ...

with db_transaction.atomic():
    transaction = (
        Transaction.objects
        .select_related(
            'user',
            'product',
            'product__category',
        )
        .select_for_update()
        .get(uid=uid)
    )

    # ...

Manipulating the transaction usually involves some properties from the user and the product so we often use select_related to force a join and save some queries.

Updating the transaction also involves obtaining a lock to make sure it's not being manipulated by anyone else.

Now, do you see the problem? NO? Neither did we.

We had some ETL processes running at night performing maintenance on the product and user tables. These ETLs performed updates and inserts to the tables so they also obtained locks on the tables.

So what was the problem? When select_for_update is used along with select_related, Django will attempt to obtain a lock on all the tables in the query.

The code we used to fetch the transaction tried to obtain a lock on both the transaction table and the users, product and category tables. Once the ETL locked the last three tables in the middle of the night transactions started to fail.

Once we had a better understanding of the problem we started looking for ways to lock only the necessary table - the transaction table. Luckily A new option to select_for_update just became available in Django 2.0:

from django.db import transaction as db_transaction

# ...

with db_transaction.atomic():
    transaction = (
        Transaction.objects
        .select_related(
            'user',
            'product',
            'product__category',
        )
        .select_for_update(
            of=('self',)
        )
        .get(uid=uid)
    )
    # ...

The of option was added to select_for_update. Using of we can explicitly state which tables we want to lock. self is a special keyword indicating we want to lock the model we are working on, in this case, the Transaction.

Currently, this feature is only available for the PostgreSQL and Oracle backends.

FK Indexes

When creating a model, Django will automatically create a B-Tree index on any foreign key. B-Tree indexes can get pretty heavy and sometimes they are not really necessary.

A classic example is a through model for an M2M relation:

class Membership(Model):
    group = ForeignKey(Group)
    user = ForeignKey(User)

In the model above Django will implicitly create two indexes - one for user and one for group.

Another common pattern in M2M models is adding a unique constraint on the two fields. In our case it means that a user can only be a member of the same group once:

class Membership(Model):
    group = ForeignKey(Group)
    user = ForeignKey(User)

    class Meta:
        unique_together = (
            ('group', 'user',)
        )

The unique_together will also create an index on both fields. So we get one model with two fields and three indexes.

Depending on the work we do with this model, many times we can dismiss the FK indexes and keep only the one created by the unique constraint:

class Membership(Model):
    group = ForeignKey(
        Group,
        db_index=False,
    )
    user = ForeignKey(
        User,
        db_index=False,
    )

    class Meta:
        unique_together = (
            ('group', 'user',)
        )

Removing redundant indexes will make insert and updates faster, plus, our database is now lighter which is always a good thing.

Order of columns in composite index

Indexes with more than one column are called composite indexes. In B-Tree composite indexes the first column is indexed using a tree structure. From the leafs of the first level a new tree is created for the second level and so on.

The order of the columns in the index is significant.

In the example above we would get a tree for groups first, and for each group another tree for all it's users.

The rule of thumb for B-Tree composite indexes is to make the secondary indexes as small as possible. In other words, columns with high cardinality (more distinct values) should come first.

In our example it's reasonable to assume there are more users than groups so puting the user column first will make the secondary index on group, smaller.

class Membership(Model):
    group = ForeignKey(
        Group,
        db_index=False,
    )
    user = ForeignKey(
        User,
        db_index=False,
    )

    class Meta:
        unique_together = (
            ('user', 'group', )
        )

This is just a rule of thumb and it should be taken with a grain of salt. The final indexing should be optimized for the specific use case. The main point here is to be aware of implicit indexes and the significance of the column order in composite indexes.

BRIN indexes

A B-Tree index is structured like a tree. The cost of looking up a single value is the height of the tree + 1 for the random access to the table. This makes B-Tree indexes ideal for unique constraints and (some) range queries.

The disadvantage of B-Tree index is its size - B-Tree indexes can get big.

It's not uncommon to think there are no alternatives but databases offer other types of indexes for specific use cases.

Starting with Django 1.11 there is a new Meta option for creating indexes on a model. This gives us an opportunity to explore other types of indexes.

PostgreSQL has a very useful type of index called BRIN (Block Range Index). Under some circumstances BRIN indexes can be more efficient than B-Tree indexes.

Let's see what the official documentation has to say first:

BRIN is designed for handling very large tables in which certain columns have some natural correlation with their physical location within the table.

To understand this statement it's important to understand how BRIN index works. As the name suggest, a BRIN index will create a mini index on a range of adjacent blocks in the table. The index is very small and it can only say if a certain value is definitely not in the range or if it might be in the range of indexed blocks.

Let's do a simplified example of how BRIN works to help us understand.

Say we have these values in a column, each is one block:

1, 2, 3, 4, 5, 6, 7, 8, 9

Let's create a range for each 3 adjacent blocks:

[1,2,3], [4,5,6], [7,8,9]

For each range we are going to keep the minimum and maximum value in the range:

[1–3], [4–6], [7–9]

Using this index, let's try to search for the value 5:

[1–3] - Definitely not here.
[4–6] - Might be here.
[7–9] - Definitely not here.

Using the index we limited our search to blocks 4–6.

Let's take another example, this time the values in the column are not going to be nicely sorted:

[2,9,5], [1,4,7], [3,8,6]

And this is our index with the minimum and maximum value in each range:

[2–9], [1–7], [3–8]

Let's try to search for the value 5:

[2–9] - Might be here.
[1–7] - Might be here.
[3–8] - Might be here.

The index is useless - not only did it not limit the search at all, we actually had to read more because we fetched both the index and the entire table.

Going back to the documentation:

…columns have some natural correlation with their physical location within the table.

This is key for BRIN indexes. To get the most out of it, the values in the column must be roughly sorted or clustered on disk.

Now back to Django, what field do we have that is often indexed and will most likely be naturally sorted on disk? That's right, I'm looking at you auto_now_add.

A very common pattern in Django models is this:

class SomeModel(Model):
    created = DatetimeField(
        auto_now_add=True,
    )

When auto_now_add is used Django will automatically populate the field with the current time when the row is created. A created field is usually also a great candidate for queries so it's often indexed.

Let's add a BRIN index on created:

from django.contrib.postgres.indexes import BrinIndex

class SomeModel(Model):
    created = DatetimeField(
        auto_now_add=True,
    )

    class Meta:
        indexes = (
            BrinIndex(fields=['created']),
        )

To get a sense of the difference in size I created a table with ~2M rows with a date field that is naturally sorted on disk:

B-Tree index: 37 MB
BRIN index: 49 KB

That's right, no mistake.

There are a lot more to consider when creating indexes than the size of the index. But now, with Django 1.11 support for indexes, we can easily integrate new types of indexes into our apps and make them lighter and faster.

How to Add a Text Filter to Django Admin

2018-01-02T00:00:00+02:00

When creating a new Django Admin page a common conversation between the developer and the support personal might sound like this:

Developer: Hey, I'm adding a new admin page for transactions. Can you tell me how you want to search for transactions?

Support: Sure, I usually just search by the username.

Developer: Cool.

search_fields = (
    user__username,
)

Anything else?

Support: I sometimes also want to search by the user email address.

Developer: OK.

search_fields = (
   user__username,
   user__email,
)

Support: And the first and last name of course.

Developer: Yeah, OK.

search_fields = (
    user__username,
    user__email,
    user__first_name,
    user__last_name,
)

Developer: Is that it?

Support: Well, sometimes I need to search by the payment voucher number.

Developer: OK.

search_fields = (
    user__username,
    user__email,
    user__first_name,
    user__last_name,
    payment__voucher_number,
)

Developer: Anything else?

Support: Some customers send their invoices and ask questions so I search by the invoice number as well.

Developer: FINE!

search_fields = (
    user__username,
    user__email,
    user__first_name,
    user__last_name,
    payment__voucher_number,
    invoice__invoice_number,
)

Developer: OK, are you sure this is it?

Support: Well, developers sometimes forward tickets to us and they use these long random strings. I'm never really sure what they are so I just search and hope for the best.

Developer: These are called UUID's.

search_fields = (
    user__username,
    user__email,
    user__first_name,
    user__last_name,
    payment__voucher_number,
    invoice__invoice_number,
    uid,
    user__uid,
    payment__uid,
    invoice__uid,
)

Developer: So is that it?

Support: Yes, for now…

The Problem With Search Fields

Django Admin search fields are great, throw a bunch of fields in search_fields and Django will handle the rest.

The problem with search field begins when there are too many of them.

When the admin user want to search by UID or email, Django has no idea this is what the user intended so it has to search by all the fields listed in search_fields. These "match any" queries have huge WHERE clauses and lots of joins and can quickly become very slow.

Using a regular ListFilter is not an option -ListFilter will render a list of choices from the distinct values of the field. Some fields we listed above are unique and the others have many distinct values - Showing choices is not an option.

Bridging the gap between Django and the user

We started thinking of ways we can create multiple search fields - one for each field or group of fields. We thought that if the user want to search by email or UID there is no reason to search by any other field.

After some thought we came up with a solution - a custom SimpleListFilter:

ListFilter allows for custom filtering logic.
ListFilter can have a custom template.
Django already has support for multiple ListFilters.

We wanted it to look like this:

A text list filter

Implementing `InputFilter`

What we want to do is have a ListFilter with a text input instead of choices.

Before we dive into the implementation, let's start from the end. This is how we want to use our InputFilter in a ModelAdmin:

class UIDFilter(InputFilter):
    parameter_name = 'uid'
    title = _('UID')

    def queryset(self, request, queryset):
        if self.value() is not None:
            uid = self.value()

            return queryset.filter(
                Q(uid=uid) |
                Q(payment__uid=uid) |
                Q(user__uid=uid)
            )

And use it like any other list filter in a ModelAdmin:

class TransactionAdmin(admin.ModelAdmin):

    # ...

    list_filter = (
        UUIDFilter,
    )

    # ...

We create a custom filter for the uuid field - UIDFilter.
We set the parameter_name in the URL to be uid. A URL filtered by uid will look like this /admin/app/transaction?uid=<uid>
If the user entered a uid we search by transaction uid, payment uid or user uid.

So far this is just like a regular custom ListFilter.

Now that we have a better idea of what we want let's implement our InputFilter:

class InputFilter(admin.SimpleListFilter):
    template = 'admin/input_filter.html'

    def lookups(self, request, model_admin):
        # Dummy, required to show the filter.
        return ((),)

We inherit from SimpleListFilter and override the template. We don't have any lookups and we want the template to render a text input instead of choices:

<!-- templates/admin/input_filter.html -->

{% load i18n %}

<h3>{% blocktrans with filter_title=title %} By {{ filter_title }} {% endblocktrans %}</h3>
<ul>
  <li>
    <form method="GET" action="">
        <input
           type="text"
           value="{{ spec.value|default_if_none:'' }}"
           name="{{ spec.parameter_name }}"/>
    </form>
  </li>
</ul>

We use similar markup to Django's existing list filter to make it native. The template renders a simple form with a GET action and a text field for the parameter. When this form is submitted the URL will be updated with the parameter name and the submitted value.

Play Nice With Other Filters

So far our filter works but only if there are no other filters. If we want to play nice with other filters we need to consider them in our form. To do that, we need to get their values.

The list filter has another function called "choices". The function accepts a changelist object that contains all the information about the current view and return a list of choices.

We don't have any choices, so we are going to use this function to extract all the filters that were applied to the queryset and expose them to the template:

class InputFilter(admin.SimpleListFilter):
    template = 'admin/input_filter.html'

    def lookups(self, request, model_admin):
        # Dummy, required to show the filter.
        return ((),)

    def choices(self, changelist):
        # Grab only the "all" option.
        all_choice = next(super().choices(changelist))
        all_choice['query_parts'] = (
            (k, v)
            for k, v in changelist.get_filters_params().items()
            if k != self.parameter_name
        )
        yield all_choice

To include the filters we add a hidden input field for each parameter:

<!-- templates/admin/input_filter.html -->

{% load i18n %}

<h3>{% blocktrans with filter_title=title %} By {{ filter_title }} {% endblocktrans %}</h3>
<ul>
  <li>
    {% with choices.0 as all_choice %}
    <form method="GET" action="">

        {% for k, v in all_choice.query_parts %}
        <input type="hidden" name="{{ k }}" value="{{ v }}" />
        {% endfor %}

        <input
            type="text"
            value="{{ spec.value|default_if_none:'' }}"
            name="{{ spec.parameter_name }}"/>

    </form>
    {% endwith %}
  </li>
</ul>

Now we have a filter with a text input that plays nice with other filters. The only thing left to do it to add a "clear" option.

To clear the filter we need a URL that include all filters except ours:

<!-- templates/admin/input_filter.html -->

...

<input
  type="text"
  value="{{ spec.value|default_if_none:'' }}"
  name="{{ spec.parameter_name }}"/>

{% if not all_choice.selected %}
  <strong><a href="{{ all_choice.query_string }}">⨉ {% trans 'Remove' %}</a></strong>
{% endif %}

...

Voilà!

This is what we get:

InputFilter with other filters and a remove button

The complete code of admin.py can be found in this gist and the complete code of the tempalte can be found in this gist.

Bonus

Search Multiple Words Similar to Django Search

You might have noticed that when searching multiple words Django find results that include at least one of the words and not all.

For example, if you search for a user "John Duo" Django will find both "John Foo" and "Bar Due". This is very convenient when searching for things like full name, product names and so on.

We can implement a similar condition using our InputFilter:

from django.db.models import Q

class UserFilter(InputFilter):
    parameter_name = 'user'
    title = _('User')

    def queryset(self, request, queryset):
        term = self.value()

        if term is None:
            return

        any_name = Q()
        for bit in term.split():
            any_name &= (
                Q(user__first_name__icontains=bit) |
                Q(user__last_name__icontains=bit)
            )

        return queryset.filter(any_name)

This is it!

Django Admin Range-Based Date Hierarchy

2017-12-11T00:00:00+02:00

A few weeks ago we encountered a major performance regression in one of our main admin pages. The page took more than 10 seconds to load (at best) and hit the query execution timeout at worst.

The page was an admin list view of a transactions model, one of the main models in our app. The model is used by support personal on a daily basis. It has millions of rows, and used several joins to display relevant information.

The most common use for the page was to filter transactions by a certain period, most commonly the last day. We used Django admin date hierarchy to drill down on the creation date of the records. When we investigated the issue, we found that the date hierarchy was the cause for most of the time spent loading the admin page.

Identifying the Problem

The filtered URL looked like this:

/admin/transactions/created__year=2017&created__month=11

We identified the "heavy" query as the one fetching the data to populate the list. The query performed the join with all the lookup tables and applied the filters we listed in the ModelAdmin. The relevant WHERE clause was:

    WHERE created BETWEEN
       '2017-01-01T00:00:00+00:00'::timestamptz AND
       '2017-12-31T23:59:59.999999+00:00'::timestamptz
    AND EXTRACT('month' FROM created AT TIME ZONE 'UTC') = 11)

When we inspected the execution plan of this query we found this snippet at the bottom:

Index Scan using ix on transactions_transaction (cost=0.43..90663.65 rows=4561 width=471)
    Index Cond: ((created >= '2017-01-01 02:00:00+02'::timestamp with time zone)
                 AND (created <= '2018-01-01 01:59:59.999999+02'::timestamp with time zone))
        Filter: (date_part('month'::text, timezone('UTC'::text, created)) = '11'::double precision)

PostgreSQL decided to use the index on the created column. The estimate was 4,561 rows and the cost estimate maxed at 90,000. This estimate was wildly inaccurate which made us drew our first conclusion:

The estimate made by the database is very low and inaccurate.

We first suspected that the low estimate was due to stale statistics on the column. We gathered stats on the column and tried again:

haki=# analyze transaction_transaction (created);

No change. Estimate remained the same.

We decided to give the query another look. We found this snippet in the WHERE clause a bit odd:

WHERE
    created BETWEEN
        '2017-01-01T00:00:00+00:00'::timestamptz
        AND '2017-12-31T23:59:59.999999+00:00'::timestamptz
    AND EXTRACT('month' FROM created AT TIME ZONE 'UTC') = 11

Django applied a filter on the full year and then used EXTRACT to filter only the month.

We decided to check what happens if we remove the EXTRACT function and instead simplify the condition:

WHERE created BETWEEN
    '2017-11-01T00:00:00+00:00'::timestamptz
    AND '2017-11-30T23:59:59.999999+00:00'::timestamptz

The execution plan for that query was:

Index Scan using ix on transactions_transaction (cost=0.43..1265.75 rows=13716 width=471)
    Index Cond: ((created >= ‘2017–11–01 02:00:00+02'::timestamp with time zone)
                  AND (created <= ‘2017–12–01 01:59:59.999999+02'::timestamp with time zone))

The estimate is still low but the cost estimate is now significantly lower. This execution plan helped us reach our second conclusion:

The way Django apply the filter makes it difficult for the database to optimize the query.

The Problem With the Way Date Hierarchy is Implemented

Django filters the queryset for a given level in the date hierarchy using a database function to extract the relevant date part.

A function is opaque to the database optimizer. If you have a range-based (BTREE) index on the field, using EXTRACT does not limit the range at all. The index is not utilized properly which might lead to a sub-optimal execution plan.

Once we had a better understanding of the problem we started discussing possible solutions.

Function based index

A function based index is an index on an expression. In our case, an appropriate function based index might look like this:

CREATE INDEX transactions_transaction_created_month_brin
ON transactions_transaction
USING BRIN(EXTRACT('month' FROM created AT TIME ZONE'UTC'));

The index made the query run faster but it came at a cost.

The downside to this approach is having to maintain additional indexes for each level of the hierarchy (day and month). Additional indexes slow down insert and update operations, and take up space.

Another downside is the index size. We used a BRIN index in this case to minimize the size of the index. The table is naturally clustered by the creation date so this is an ideal use case for a BRIN index. When this is not the case, a similar BTREE index can become quite heavy.

see also

More on how BRIN indexes work in 9-django-tips-for-working-with-databases.

Simplify the Condition Used by Django Date Hierarchy

We decided to see if we can simplify the condition used by Django to apply the date hierarchy.

To implement the filter differently we first need to understand how Django admin applies filters on a queryset.

In django/contrib/admin/views/main.py there is a function called get_filters. It might look scary at first but what it does is:

Extract all the query parameters from the URL.
Get all the ListFilter declared in the ModelAdmin and apply them one by one to the parameter list.
Each ListFilter receive the parameter list, takes what it needs and remove the value from the parameter list.
Any parameters left after all ListFilter's were applied are processed using the "default" Django filter. This is why, for example, it's possible to filter the queryset directly from the URL even when a ListFilter is not explicitly defined.

If you look at the date hierarchy query parameters you'll see that there is nothing special about them, they are just regular URL params. This sparked an idea:

Implement a ListFilter to grab the relevant date hierarchy parameters from the parameter list and apply a custom filter on the queryset.

The Implementation

Let's start by grabbing the date hierarchy fields from the parameter list:

class RangeBasedDateHierarchyListFilter(admin.ListFilter):
    title = ''

    def __init__(self, request, params, model, model_admin):
        self.date_hierarchy_field = model_admin.date_hierarchy
        self.date_hierarchy = {}

        date_hierarchy_field_re = re.compile(
            r'^{}__(day|month|year)$'.format(self.date_hierarchy_field)
        )

        for param in list(params.keys()):
            match = date_hierarchy_field_re.match(param)
            if match:
                period = match.group(1)
                self.date_hierarchy[period] = int(params.pop(param))

We take the date_hierarchy field name from the model_admin.In our case it was created.
We create a Regex pattern to identify the parameters in the URL. The pattern is always the name of the date hierarchy field + the period (day, month, year).
In our case the possible parameters are created__day, created__month and created__year.
We iterate the parameter list, pop any date hierarchy parameter that match our criteria and store the period and the value in a dict.

Now, this is where all the magic happens:

class RangeBasedDateHierarchyListFilter(admin.ListFilter):

    # ...

    def queryset(self, request, queryset):
        tz = timezone.get_default_timezone()
        from_date, to_date = get_date_range_for_hierarchy(self.date_hierarchy, tz)

        return queryset.filter(**{
            '{}__gte'.format(self.date_hierarchy_field): from_date,
            '{}__lt'.format(self.date_hierarchy_field): to_date,
        })

Django will call the queryset function of our ListFilter with a queryset, and the function is expected to return the filtered queryset.

In the example above this is happening in a separate function (so we can test it, you know...):

def get_date_range_for_hierarchy(date_hierarchy, tz):
    """Generate date range for date hierarchy.

    date_hierarchy <dict>:
        year (int)
        month (int or None)
        day (int or None)
    tz <timezone or None>:
        The timezone in which to generate the datetimes.
        If None, the datetimes will be naive.

    Returns (tuple):
        from_date (datetime.datetime, aware if tz is set) inclusive
        to_date (datetime.datetime, aware if tz is set) exclusive
    """
    from_date = datetime.datetime(
        date_hierarchy['year'],
        date_hierarchy.get('month', 1),
        date_hierarchy.get('day', 1),
    )

    if tz:
        from_date = tz.localize(from_date)

    if 'day' in date_hierarchy:
        to_date = from_date + datetime.timedelta(days=1)

    elif 'month' in date_hierarchy:
        assert from_date.day == 1
        to_date = (from_date + datetime.timedelta(days=32)).replace(day=1)

    else:
        to_date = from_date.replace(year=from_date.year + 1)

    return from_date, to_date

The function receives the dict we constructed in __init__ and returns a date range to filter on. The queryset function then applies a simpler range filter that our database can better utilize.

To use the simplified range condition in our ModelAdmin we need to add it as a list_filter:

class TransactionAdmin(admin.ModelAdmin):
    date_hierarchy = 'created'
    list_filter = (
        #...
        RangeBasedDateHierarchyListFilter,
    )

The Result

After we deployed this change the performance of the page improved drastically. Our database was happy, the support team was happy and so were we.

source code

See our package django-lightweight-date-hierarchy on github and pypi.

Scaling Django Admin Date Hierarchy

2017-10-06T00:00:00+03:00

package

We published a package called django-admin-lightweight-date-hierarchy which overrides Django Admin date_hierarchy template tag and eliminates all database queries from it.
For the implementation details and the shocking performance analysis read on.

If you are not familiar with Django Admin date_hierarchy you should, it's great. Set the date_hierarchy attribute of a ModelAdmin to a DateField:

from django.contrib import admin
from .models import Sale

@admin.register(Sale)
class SaleAdmin(admin.ModelAdmin):
    date_hierarchy = 'created'

And you get a nice drill-down menu at the top of the admin change list:

Django date_hierarchy in action

When selecting a year, Django will filter the data to the selected year, and present a list of months for which there is data in that year.

When selecting a month, Django will apply the filter and present the list of days for which there is data in that month.

`date_hierarchy` Behind the Scenes

To produce a list of dates for which there is data, Django has to perform a query. For example, to produce the list of years, the first level in the hierarchy, Django will execute the following query:

SELECT DISTINCT
    django_datetime_trunc('year',  "sales_sale"."created", 'UTC') AS "datetimefield"
FROM
    "sales_sale"
WHERE
    "sales_sale"."created" IS NOT NULL
ORDER BY
    "datetimefield" ASC;

When a year is selected, Django will execute a query to produce the next level in the hierarchy - months:

SELECT DISTINCT
    django_datetime_trunc('month', "sales_sale"."created", 'UTC') AS "datetimefield"
FROM
    "sales_sale"
WHERE
    "sales_sale"."created" BETWEEN '2017–01–01 00:00:00' AND '2017–12–31 23:59:59.999999'
    AND "sales_sale"."created" BETWEEN '2017–01–01 00:00:00' AND '2017–12–31 23:59:59.999999'
    AND "sales_sale"."created" IS NOT NULL
ORDER BY
    "datetimefield" ASC;

How Expensive is it?

The date hierarchy is a great feature but it comes at a price.

To illustrate the problem we created a simple Sale model with two fields - id and created, and populated it with ~1,000,000 rows.

Using django-admin-toolbar we can see how long it takes Django to produce the date hierarchy:

Breakdown of SQL queries executed by Django Admin

WOW! The page took a boggling ~8s to load, out of which 7.6 seconds are spent producing the date hierarchy!

Just for comparison, the exact same page without date_hierarchy:

Breakdown of SQL queries executed by Django Admin without date hierarchy

17ms. That's about 99.8% better.

A Possible Solution

Looking at the chart above it's clear that we have a problem. The date hierarchy is weighing our page making it nearly unusable.

Django need to execute a query because it only wants to show dates for which there is data. In our case, we have sales every day. Once we make this assumption we no longer have to query the data to produce a list of dates - we can just show them all!

The idea we came up with is:

if the user selected a month we show all of the days in the month.
If the user selected a year we show all of the months in the year.
If the user selected nothing we need to make an additional assumption - in our case we decided to show +-3 years from the current year. This is a compromise we were willing to make for the sake of performance and usability.

Now that we have the general idea let's dive into the implementation.

Implementation

Looking at the output above we can see that the queries originate from a template tag called {% date_hierarchy cl %}. The argument cl is the ChangeList created by the ModelAdmin.

The implementation for the date_hierarchy template tag can be found at admin_list.py.The interesting part is where the queries are executed.

Let's take a look at how Django produces a list of months for a given year:

# ...
year_field = '%s__year' % field_name

# ...
year_lookup = cl.params.get(year_field)

def link(filters):
  return cl.get_query_string(filters, [field_generic])

# ...

elif year_lookup:
  months = cl.queryset.filter(**{year_field: year_lookup})
  months = getattr(months, 'dates')(field_name, 'month')
  return {
    'show': True,
    'back': {
      'link': link({}),
      'title': _('All dates'),
    },
    'choices': [{
      'link': link({
        year_field: year_lookup,
        month_field: month.month,
      }),
      'title': capfirst(formats.date_format(month, 'YEAR_MONTH_FORMAT'))
    } for month in months]
  }
    # ...

Our date_hierarchy is set to the created. If we drill-down on year 2017 we get the following URL:

http://localhost:8000/admin/sales/sale/?created__year=2017

In the template tag year_field is created__year and year_lookup is 2017. The generated query applies the filter on created and fetches a list of months there is data for in year 2017 to the variable months.

Let's replace this bit and populate months with a list of all the months in the year instead:

# months = cl.queryset.filter(**{year_field: year_lookup})
# months = getattr(months, 'dates')(field_name, 'month')

# All months of selected year.
months = (
  datetime.date(int(year_lookup), month, 1)
  for month in range(1, 13)
)

After the change:

Queries executed by Django Admin after the change

Awesome! No queries.

And the list view:

All month of year 2017 are shown

Our little change worked! All the months are displayed and no queries are executed by the date hierarchy.

Let's do the same for days:

# days = cl.queryset.filter(**{year_field: year_lookup, month_field: month_lookup})
# days = getattr(days, dates_or_datetimes)(field_name, 'day')
# All days of month.

days_in_month = calendar.monthrange(int(year_lookup), int(month_lookup))[1]
first_day_of_month = datetime.date(int(year_lookup), int(month_lookup), 1)
days = (
  first_day_of_month + datetime.timedelta(days=i)
  for i in range(days_in_month)
)

We use the calendar module to find out how many days there are in a given month.

Let's handle the years. Remember, we fetch +-3 years from today:

# years = getattr(cl.queryset, dates_or_datetimes)(field_name, 'year')

# Three years in each direction.
today = get_today()
years = (
  datetime.date(y, 1, 1)
  for y in range(today.year - 3, today.year + 3 + 1)
)

Integration

Up until now we fiddled with Django's source but we can't really do that. First we override the template tag to make Django use our implementation.

Let's copy the function and register a template with the same name:

# app/templatetags/admin_list.py
from django.contrib.admin.templatetags.admin_list import register

@register.inclusion_tag('admin/date_hierarchy.html')
def date_hierarchy(cl):
  # ... (original implementation) ...

# settings.py

TEMPLATES = [{
  # ...
  'OPTIONS': {
    # ...
    'libraries': {
      # ...
      'admin': 'app.templatetags.admin_list',
    },
  },
}]

Now Django uses our template tag instead of his.

The problem with producing the date hierarchy is really an issue only for very large tables. We don't want to disable the existing behavior - we want to enable it only for very large tables.

Let's add an attribute on the ModelAdmin to turn the default drill-down behavior on and off:

@admin.register(Sale)
class SaleAdmin(admin.ModelAdmin):
    date_hierarchy = 'created'
    date_hierarchy_drilldown = False

When date_hierarchy_drilldown is set to False our new template tag will not execute queries. Otherwise, we preserve the original behavior.

To implement this we add the following at the start of our implementation of the date_hierarchy template tag:

date_hierarchy_drilldown = getattr(
    cl.model_admin,
    'date_hierarchy_drilldown',
    True,
)

Now we can re-enable the default behavior when date_hierarchy_drilldown=True.

For example, producing a list of months:

# ...
if date_hierarchy_drilldown:
    months = cl.queryset.filter(**{year_field: year_lookup})
    months = getattr(months, dates_or_datetimes)(field_name, 'month')
else:
    # All months of selected year.
    months = (
        datetime.date(int(year_lookup), month, 1)
        for month in range(1, 13)
    )
# ...

This is it! We've successfully scaled date hierarchy to handle millions of rows with a little compromise on UX.

Package

We found this approach useful in several projects so we decided to publish it as a package:

Using it is as simple as

Install it

$ pip install django-admin-lightweight-date-hierarchy

Add it to your INSTALLED_APPS:

INSTALLED_APPS = (
    'django_admin_lightweight_date_hierarchy',
)

Set date_hierarchy_drilldown to False on any ModelAdmin with date_hierarchy to prevent the default drill-down behavior:

@admin.register(MyModel)
class MyModelAdmin(admin.ModelAdmin):
  date_hierarchy = 'created'
  date_hierarchy_drilldown = False

To squeeze some more juice out of Django Admin check out this post as well:

Cheers!

How We Replaced Dozens of Test Fixtures With One Simple Function

2017-08-16T00:00:00+03:00

It all started when we added feature flags to our app. After some deliberation we created a "feature set" model with boolean fields for each feature:

class FeatureSet(models.Model):
    name = models.CharField(max_length=50)
    can_pay_with_credit_card = models.BooleanField()
    can_save_credit_card = models.BooleanField()
    can_receive_email_notifications = models.BooleanField()

We added a foreign key from the user account to the feature sets model, and created feature sets for "pro", "newbie" and "commercial" users.

To enforce the features we added tests in appropriate places. For example:

def pay_with_credit_card(self, user_account, amount):
    if not user_account.feature_set.can_pay_with_credit_card:
        raise FeatureDisabled('can_pay_with_credit_card')
    # ...

The Problem

At this point we had a large codebase and a lot of tests. Unfortunately, a lot of our tests were a relic from when we were using fixtures extensively.

The thought of having to update and add new fixtures was unacceptable. But, we still had to test the new features so we started writing tests like this:

def test_should_charge_credit_card(self):
    feature_set = user_account.feature_set
    feature_set.can_pay_with_credit_card = True
    feature_set.save(update_fields=['can_pay_with_credit_card'])
    pay_with_credit_card(user_account, 100)

def test_should_fail_when_feature_disabled(self):
    feature_set = user_account.feature_set
    feature_set.can_pay_with_credit_card = False
    with self.assertRaises(FeatureDisabled):
        pay_with_credit_card(self.user_account, 100)

We had a lot of tests to update and some of the features we added interrupted the flow of other tests which resulted in a mess!

The Context Manager

We already used context managers to improve our tests in the past, and we thought we can use one here to set features on and off:

from contextlib import contextmanager

@contextmanager
def feature(feature_set, feature_name, enabled):
    original_value = getattr(feature_set, feature_name)
    setattr(feature_set, feature_name, enabled)
    feature_set.save(update_fields=[feature_name])

    try:
        yield

    finally:
        setattr(feature_set, feature_name, original_value)
        feature_set.save(update_fields=[feature_name])

What does this context manager do?

Save the original value of the feature.
Set the new value for the feature.
Yields - this where our test code actually executes.
Set the feature back to the original value

This made our tests much more elegant:

def test_should_charge_credit_card(self):
    with feature(user_account.feature_set, can_pay_with_credit_card, True):
       pay_with_credit_card(user_account, 100)

def test_should_fail_when_feature_disabled(self):
    with feature(user_account.feature_set, can_pay_with_credit_card, False):
        with self.assertRaises(FeatureDisabled):
            pay_with_credit_card(self.user_account, 100)

**kwargs

This context manager has proven to be very useful for features so we thought... why not use it for other things as well?

We had a lot of methods involving more than one feature:

def test_should_not_send_notification(self):
    feature_set = user_account.feature_set
    with feature(feature_set, can_pay_with_credit_card, True):
        with feature(feature_set, can_receive_notifications, False):
            pay_with_credit_card(user_account, 100)

Or more than one object:

def test_should_not_send_notification_to_inactive_user(self):
    feature_set = user_account.feature_set
    user_account.user.is_active = False
    with feature(feature_set, can_receive_notifications, False):
        pay_with_credit_card(user_account, 100)

So we rewrote the context manager to accept any object and added support for multiple arguments:

@contextmanager
def temporarily(obj, **kwargs):
    original_values = {k: getattr(obj, k) for k in kwargs}

    for k, v in kwargs.items():
        setattr(obj, k, v)

    obj.save(update_fields=kwargs.keys())

    try:
        yield

    finally:
        for k, v in original_values.items():
            setattr(obj, k, v)

        obj.save(update_fields=original_values.keys())

The context manager can now accept multiple features, save the original values, set the new values and restore when we are done.

Testing became much easier:

def test_should_not_send_notification(self):
    with temporarily(
        user_account.feature_set,
        can_pay_with_credit_card=True,
        can_receive_notifications=False,
    ):
        pay_with_credit_card(user_account, 100)
    self.assertEquals(len(outbox), 0)

We can now use the function on other objects as well:

def test_should_fail_to_login_inactive_user(self):
    with temporarily(user, is_active=False):
        response = self.login(user)
    self.assertEqual(response.status_code, 400)

Profit!

The Hidden Performance Benefit

After a while getting comfortable with the new utility we noticed another performance benefit. In tests that had heavy setups we managed to move the setup from the test level to the class level.

To illustrate the difference let's test a function that sends an invoice to the users. Invoices are usually sent only when the transaction is complete. To create a complete transaction we need a lot of setup (choose products, checkout, issue payment etc).

This is a test that require a lot of setup:

class TestSendInvoice(TestCase):

    def setUp(self):
        self.user = User.objects.create_user( ... )
        self.transaction = Transaction.create(self.user,  ... )
        Transaction.add_product( ... )
        Transaction.add_product( ... )
        Transaction.checkout( ... )
        Transaction.request_payment( ... )
        Transaction.process_payment( ... )

    def test_should_not_send_invoice_to_commercial_user(self):
        self.user.type = 'commercial'
        mail.outbox = []
        Transaction.send_invoice(self.user)
        self.assertEqual(len(mail.outbox), 0)

    def test_should_attach_special_offer_to_pro_user(self):
        self.user.type = 'pro'
        mail.outbox = []
        Transaction.send_invoice(self.user)
        self.assertEqual(len(mail.outbox), 1)
        self.assertEqual(
            mail.outbox[0].subject,
            'Invoice and a special offer!'
        )

The setUp function need to execute before each test function because the test functions change the objects and that might create a dangerous dependency between test cases.

To prevent dependencies between test cases we need to make sure each test leaves the data exactly as it got it. Luckily, this is exactly what our new context manager does:

class TestSendInvoice(TestCase):

    @classmethod
    def setUpTestData(cls):
        cls.user = User.objects.create_user( ... )
        cls.transaction = Transaction.create(cls.user,  ... )
        Transaction.add_product( ... )
        Transaction.add_product( ... )
        Transaction.checkout( ... )
        Transaction.request_payment( ... )
        Transaction.process_payment( ... )

    def test_should_not_send_invoice_to_commercial_user(self):
        mail.outbox = []
        with temporarily(self.user, type='commercial'):
            Transaction.send_invoice(self.user)
        self.assertEqual(len(mail.outbox), 0)

    def test_should_attach_special_offer_to_pro_user(self):
        mail.outbox = []
        with temporarily(self.user, type='pro'):
            Transaction.send_invoice(self.user)
        self.assertEqual(len(mail.outbox), 1)
        self.assertEqual(mail.outbox[0].subject, 'Invoice and a special offer!')

We moved the setup code to setUpTestData. The setup code will execute only once for the entire test class resulting in quicker tests.

Final Words

The motivation for this context processor was our long unhealthy relationship with fixtures. As we scaled our app the fixtures became a burden. Having so many tests rely on them made it difficult to completely replace.

With the addition of features we knew we did not want to rely on fixtures any more and we looked for creative, more verbose and maintainable ways, of managing test data. Having a simple way to create different variations of an object for testing was exactly what we needed.

How to Manage Concurrency in Django Models

2017-07-07T00:00:00+03:00

The days of desktop systems serving single users are long gone. Web applications nowadays are serving millions of users at the same time. With many users comes a wide range of new problems: concurrency problems.

The Problem

To demonstrate common concurrency issues we are going to work on a bank account model:

class Account(models.Model):
    id = models.AutoField(
        primary_key=True,
    )
    user = models.ForeignKey(
        User,
    )
    balance = models.IntegerField(
        default=0,
    )

To get started we are going to implement a naive deposit and withdraw methods for an account instance:

def deposit(self, amount):
    self.balance += amount
    self.save()

def withdraw(self, amount):
    if amount > self.balance:
        raise errors.InsufficientFunds()
    self.balance -= amount
    self.save()

This seems innocent enough and it might even pass unit tests and integration tests on localhost. But, what happens when two users perform actions on the same account at the same time?

User A fetches the account
- balance is 100$
User B fetches the account
- balance is 100$
User B withdraws 30$
- balance is updated to 100$ - 30$ = 70$
User A deposits 50$
- balance is updated to 100$ + 50$ = 150$

What Happened Here?

User B asked to withdraw 30$ and user A deposited 50$. We expect the balance to be 120$, but we ended up with 150$.

Why Did it Happen?

At step 4, when user A updated the balance, the amount he had stored in memory was stale (user B had already withdrawn 30$). To prevent this situation from happening we need to make sure the resource we are working on is not altered while we are working on it.

Pessimistic Approach

The pessimistic approach dictates that you should lock the resource exclusively until you are finished with it. If nobody else can acquire a lock on the object while you are working on it, you can be sure the object was not changed.

To acquire a lock on a resource we use a database lock for several reasons:

(relational) databases are very good at managing locks and maintaining consistency.
The database is the lowest level in which data is accessed - acquiring the lock at the lowest level will protect the data from other processes modifying the data as well. For example, direct updates in the DB, cron jobs, cleanup tasks, etc.
A Django app can run on multiple processes (e.g workers). Maintaining locks at the app level will require a lot of (unnecessary) work.

Implementation

Let's implement a safe deposit and withdraw actions using a pessimistic approach:

@classmethod
def deposit(cls, id, amount):
    with transaction.atomic():
        account = (
            cls.objects
            .select_for_update()
            .get(id=id)
        )
        account.balance += amount
        account.save()

    return account

@classmethod
def withdraw(cls, id, amount):
    with transaction.atomic():
        account = (
            cls.objects
            .select_for_update()
            .get(id=id)
        )
        if account.balance < amount:
            raise errors.InsufficientFunds()
       account.balance -= amount
       account.save()

   return account

We use select_for_update on our queryset to tell the database to lock the object until the transaction is done.
Locking a row in the database requires a database transaction. We use Django's decorator transaction.atomic() to scope the transaction.
We use a classmethod instead of an instance method. To acquire the lock we need to tell the database to lock it. To achieve that we need to be the ones fetching the object from the database. When operating on self the object is already fetched and we don't have any guaranty that it was locked.
All the operations on the account are executed within the database transaction.

Let's see how the scenario from earlier is prevented with our new implementation:

User A asks to withdraw 30$:
- User A acquires a lock on the account
- Balance is 100$
User B asks to deposit 50$:
- Attempt to acquire lock on account fails (locked by user A)
- User B waits for the lock to release
User A withdraw 30$:
- Balance is 70$
- Lock of user A on account is released
User B acquires a lock on the account
- Balance is 70$
- New balance is 70$ + 50$ = 120$
Lock of user B on account is released, balance is 120$. Bug prevented!

What You Need to Know About `select_for_update`

You dont have to wait for the lock to release - In our scenario, user B waited for user A to release the lock. Instead of waiting, we can tell Django not to wait for the lock to release and raise a DatabaseError instead. To do that, we set select_for_update(nowait=True).
Select related objects are also locked - Using select_for_update with select_related locks the related objects as well.
For example, If we select_related the user along with the account, both the user and the account are locked. If during deposit someone is trying to update the user's first name, that update will fail because the user object is locked.
If you are using PostgreSQL or Oracle this might not be a problem soon, thanks to a new feature in the upcoming Django 2.0. In this version, select_for_update has an of option to explicitly state which of the tables in the query to lock.

Optimistic Approach

Unlike the pessimistic approach, the optimistic approach does not require a lock on the object. The optimistic approach assumes collisions are not very common, and dictates that one should only make sure there were no changes made to the object at the time it is updated.

Implementation

First, we add a column to keep track of changes made to the object:

version = models.IntegerField(default=0)

Then, when we update an object, we make sure the version did not change:

def deposit(self, id, amount):
    updated = Account.objects.filter(
        id=self.id,
        version=self.version,
    ).update(
        balance=balance + amount,
        version=self.version + 1,
    )
    return updated > 0

def withdraw(self, id, amount):
    if self.balance < amount:
        raise errors.InsufficientFunds()

    updated = Account.objects.filter(
        id=self.id,
        version=self.version,
    ).update(
        balance=balance - amount,
        version=self.version + 1,
    )
    return updated > 0

Let's break it down:

We operate directly on the instance (no classmethod).
We rely on the fact that the version is incremented every time the object is updated.
We update only if the version did not change:
- If the object was not modified since we fetched it than the object is updated.
- If it was modified than the query will return zero records and the object will not be updated.
Django returns the number of updated rows. If updated is zero it means someone else changed the object from the time we fetched it.

How is optimistic locking work in our scenario:

User A fetch the account:
- balance is 100$
- version is 0
User B fetch the account:
- balance is 100$
- version is 0
User B asks to withdraw 30$:
- Balance is updated to 100$ - 30$ = 70$
- Version is incremented to 1
User A asks to deposit 50$:
- The calculated balance is 100$ + 50$ = 150$
- The account does not exist with version 0 → nothing is updated

What You Need to Know About the Optimistic Approach:

Unlike the pessimistic approach, this approach requires an additional field and a lot of discipline. One way to overcome the discipline issue is to abstract this behavior. Some packages we've taken inspiration from are:
- django-fsm implements optimistic locking using a version field as described above.
- django-optimistic-lock seem to do the same.
In an environment with a lot of concurrent updates this approach might be wasteful.
The optimistic approach does not protect from modifications made to the object outside the app. If you have other tasks that modify the data directly (e.g no through the model), you need to make sure they use the version as well.
Using the optimistic approach, the function can fail and return false. In this case we will most likely want to retry the operation. Using the pessimistic approach with nowait=False the operation cannot fail, it will wait for the lock to release.

Which One Should I Use?

Like any great question, the answer is "it depends":

If your object has a lot of concurrent updates you are probably better off with the pessimistic approach.
If you have updates happening outside the ORM (for example, directly in the database) the pessimistic approach is safer.
If your method has side effects such as remote API calls or OS calls make sure they are safe. Some things to consider - can the remote call take a long time? Is the remote call idempotent (safe to retry)?

5 Ways to Make Django Admin Safer

2017-06-09T00:00:00+03:00

In this article I present 5 ways to protect the Django Admin from human errors and attackers.

Table of Contents

Change the URL
Visually Distinguish Environments
Name Your Admin Site
Separate the Django Admin From The Main Site
Add Two Factor Authentication (2FA)
Final Words

Change the URL

Every framework has a fingerprint and Django is no exception. A skilled developer, an attacker or even a tech savvy user can identify a Django site by looking at things like cookies and auth URLs.

Once a site is identified as a Django site, an attacker will most likely try /admin.

To make it harder to gain access we can change the "recommended" URL to something harder to guess.

In the base url.py of the app, register the admin site under a different url:

urlpatterns += i18n_patterns(
    url(r'^super-secret/', admin.site.urls, name='admin'),
)

Change "super-secret" to something you and your team can remember.

This is definitely not the only precaution you should take, but it is a good start.

Visually Distinguish Environments

Users and admins are not perfect and mistakes happen. When you have multiple environments such as development, QA, staging and production, it's not unlikely for an admin to perform a destructive operation in the wrong environment by accident (just ask gitlab).

To reduce the chance of mistakes, we mark different environments clearly in the admin:

Indicator in different environments

First you need to have some way of knowing which environment you are on. We have a variable called ENVIRONMENT_NAME we populate during deployment. We have another variable called ENVIRONMENT_COLOR for the indicator color.

To add the environment indicator to every page in the admin, override the base admin template:

<!-- app/templates/admin/base_site.html -->

{% extends "admin/base_site.html" %}

{% block extrastyle %}

<style type="text/css">
    body:before {
        display: block;
        line-height: 35px;
        text-align: center;
        font-weight: bold;
        text-transform: uppercase;
        color: white;
        content: "{{ ENVIRONMENT_NAME }}";
        background-color: {{ ENVIRONMENT_COLOR }};
    }
</style>

{% endblock %}

To make the ENVIRONMENT variables from settings.py available in the template we use a context processor:

# app/context_processors.py

from django.conf import settings

def from_settings(request):
    return {
        'ENVIRONMENT_NAME': settings.ENVIRONMENT_NAME,
        'ENVIRONMENT_COLOR': settings.ENVIRONMENT_COLOR,
    }

To register the context processor add the following in settings.py:

TEMPLATES = [{
    # ...
    'OPTIONS': {
        'context_processors': [
            # ...
            'app.context_processors.from_settings',
        ],
    # ...
    },
}]

Now when you open Django Admin you should see the indicator on top.

Name Your Admin Site

If you have multiple Django services that look the same, admins can easily get confused. To help admins be more aware of where they are, change the title:

# urls.py

from django.contrib import admin

admin.site.site_header = 'Awesome Inc. Administration'
admin.site.site_title = 'Awesome Inc. Administration'

And you get:

Django Admin site with a name and a title

For more exotic options look around in the docs.

Separate the Django Admin From The Main Site

Using the same codebase you can deploy two instances of the same Django app - one only for the admin and one only for the rest of the app.

This is controversial and not as easy as the other tips. The implementation is dependent on the configuration (e.g. if you are using gunicorn or uwsgi) so I won't go into the details.

Some reasons you might want to split the admin to its own instance are:

Deploy the admin inside a VPN (virtual private network) - If the admin is used only internally and you have a VPN it is good practice to have it inside the private network.
Remove unnecessary components from the main site - For example, the Django admin uses the messages framework. If the main site does not, you can remove that middleware. Another example is authentication - if the main site is an API backend using token authentication, you can remove a lot of templates configuration, session middleware, etc. and trim some fat from the request-response cycle.
Stronger authentication - If you want to strengthen the security of Django Admin you might want to provide a different authentication mechanism just for the admin. This is much easier on different instances with different settings.

We split the admin from the main site only in public facing sites. We don't bother with internal apps because it complicates the deployment and it has no benefit of being more secure.

Add Two Factor Authentication (2FA)

Two factor authentication became very popular lately as many sites started offering this option. 2FA performs authentication using two things:

Something you know - Usually a password.
Something you have - Usually a mobile app that generates a random number every 30 seconds (such as Authenticator by Google).

On first signup the user is usually asked to scan a barcode with the authenticator app. After this initial setup the app will start generating the one-time codes.

I don't usually recommend third party packages, but a couple of months ago we started using django-otp to implement 2FA in our admin site and it's working great for us. It's hosted on Bitbucket so you might have missed it.

The setup is pretty simple:

$ pip install django-otp
$ pip install qrcode

Add django-otp to the installed apps and the middleware:

# settings.py

INSTALLED_APPS = (
    'django_otp',
    'django_otp.plugins.otp_totp',
)

MIDDLEWARE = (
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django_otp.middleware.OTPMiddleware',
)

Name the issuer - this is the name users will see in the authenticator app, so make it distinguishable.

# settings.py

OTP_TOTP_ISSUER = 'Awesome Inc.'

Add 2FA authentication to the admin site:

# urls.py

from django_otp.admin import OTPAdminSite

admin.site.__class__ = OTPAdminSite

Now you have a secure admin page that looks like this:

Django Admin login with OTP token

To set up a new user create a "TOTP Device" from the Django Admin. Once you are done click the QR link and you will get a screen like that:

QR for setting up a new user

Have the user scan the QR code with the authenticator app on their personal device, and they will have a fresh code generated every 30 seconds.

Final Words

Making a Django admin safer and more secure doesn't have to be hard - you just have to pay attention. Some of the tips mentioned here are very easy to set up and they go a long way.

All You Need To Know About Prefetching in Django

2017-04-29T00:00:00+03:00

I have recently worked on a ticket ordering system for a conference. It was very important for the customer to see a table of orders including a column with a list of program names in each order:

The column requested by the users

The models looked (roughly) like this:

class Program(models.Model):
    name = models.CharField(
        max_length=20,
    )

class Price(models.Model):
    program = models.ForeignKey(
        Program,
    )
    from_date = models.DateTimeField()
    to_date = models.DateTimeField()

class Order(models.Model):
    state = models.CharField(
        max_length=20,
    )
    items = models.ManyToManyField(Price)

Program - a session, lecture or a conference day.
Price - Prices can change over time. One way to model changes over time is using a type 2 slowly changing dimension (SCD). The Price model represents the price of a program at a certain point in time.
Order - An order to one or more programs. Each item in the order is the price of the program at the time the order was made.

Before We Start

Throughout this article we are going to monitor the queries executed by Django. To log the queries add the following to the LOGGING settings in settings.py:

LOGGING = {
    # ...
    'loggers': {
        'django.db.backends': {
            'level': 'DEBUG',
        },
    },
}

What's The Problem?

Let's try to fetch the program names for a single order:

>>> o = Order.objects.filter(state='completed').first()

(0.002) SELECT ... FROM "orders_order"
WHERE "orders_order"."state" = 'completed'
ORDER BY "orders_order"."id" ASC LIMIT 1;

>>> [p.program.name for p in o.items.all()]

(0.002) SELECT ... FROM "events_price"
INNER JOIN "orders_order_items" ON ("events_price"."id" = "orders_order_items"."price_id")
WHERE "orders_order_items"."order_id" = 29; args=(29,)

(0.001) SELECT ... FROM "events_program"
WHERE "events_program"."id" = 8; args=(8,)

['Day 1 Pass']

To fetch completed orders we need one query.
To fetch the program names for each order we need two more queries.

I previously wrote about the N+1 problem and this is a classic case. If we need two queries for each order, the number of queries for 100 orders will be 1 + 100 * 2 = 201 queries, that's a lot!

Let's use Django to reduce the amount of queries:

>>> o.items.values_list('program__name')

(0.003) SELECT "events_program"."name" FROM "events_price"
INNER JOIN "orders_order_items" ON ("events_price"."id" = "orders_order_items"."price_id")
INNER JOIN "events_program" ON ("events_price"."program_id" = "events_program"."id")
WHERE "orders_order_items"."order_id" = 29 LIMIT 21;

['Day 1 Pass']

Great! Django performed a join between Price and Program and reduced the amount of queries to just one per order.

At this point instead of 201 queries we only need 101 queries for 100 orders. Can we do better?

Why Can't We Join?

The first question that should come to mind is "why can't we join the tables?"

If we have a foreign key we can use select_related or use snake case like we did above to fetch the related fields in a single query.

For example, we fetched the program name for a list of prices in a single query using values_list('program__name'). We were able to do that because each price is related to exactly one program.

If the relation between two models is many to many we can't do that. Every order has one or more related prices - if we join the two tables we get duplicate orders:

SELECT
    o.id AS order_id,
    p.id AS price_id
FROM
    orders_order o
    JOIN orders_order_items op ON (o.id = op.order_id)
    JOIN events_price p ON (op.price_id = p.id)
ORDER BY
    1,
    2;

 order_id | price_id
----------+----------
    45    |    38
    45    |    56
    70    |    38
    70    |    50
    70    |    77
    71    |    38

Orders 70 and 45 have multiple items so they come up more than once in the result - Django can't handle that.

Enter `prefetch_related`

Django has a nice, built-in way, of dealing with this problem called prefetch_related:

>>> o = Order.objects.filter(
...     state='completed',
... ).prefetch_related(
...     'items__program',
... ).first()

(0.002) SELECT ... FROM "orders_order"
WHERE "orders_order"."state" = 'completed'
ORDER BY "orders_order"."id" ASC LIMIT 1;

(0.001) SELECT ("orders_order_items"."order_id") AS "_prefetch_related_val_order_id", "events_price"...
FROM "events_price"
INNER JOIN "orders_order_items" ON ("events_price"."id" = "orders_order_items"."price_id")
WHERE "orders_order_items"."order_id" IN (29);

(0.001) SELECT "events_program"."id", "events_program"."name" FROM "events_program"
WHERE "events_program"."id" IN (8);

We told Django we intend to fetch items__program from the result set. In the second and third query we can see that Django fetched the through table orders_order_items and the relevant programs from events_program. The results of the prefetch are cached on the objects.

What happens when we try to fetch program names from the result?

>>> [p.program.name for p in o.items.all()]

['Day 1 Pass']

No additional queries - exactly what we wanted!

When using prefetch, it's important to work on the object and not on the query. Trying to fetch the program names with a query will produce the same outcome but will result in an additional query:

>>> o.items.values_list('program__name')

(0.002) SELECT "events_program"."name" FROM "events_price"
INNER JOIN "orders_order_items" ON ("events_price"."id" = "orders_order_items"."price_id")
INNER JOIN "events_program" ON ("events_price"."program_id" = "events_program"."id")
WHERE "orders_order_items"."order_id" = 29 LIMIT 21;

['Day 1 Pass']

At this point, fetching 100 orders requires only 3 queries. Can we do even better?

Introducing `Prefetch`

In version 1.7 Django introduced a new Prefetch object that extends the capabilities of prefetch_related.

The new object allows the developer to override the query used by Django to prefetch the related objects.

In our previous example Django used two queries for the prefetch - one for the through table and one for the program table. What if we could tell Django to join these two together?

>>> prices_and_programs = Price.objects.select_related('program')

>>> o = Order.objects.filter(
...     state='completed'
... ).prefetch_related(
...     Prefetch('items', queryset=prices_and_programs)
... ).first()

(0.001) SELECT ... FROM "orders_order"
WHERE "orders_order"."state" = 'completed'
ORDER BY "orders_order"."id" ASC LIMIT 1;

(0.001) SELECT ("orders_order_items"."order_id") AS "_prefetch_related_val_order_id",
"events_price"..., "events_program"...
INNER JOIN "events_program" ON ("events_price"."program_id" = "events_program"."id")
WHERE "orders_order_items"."order_id" IN (29);

We created a query that joins prices with programs. Than we told Django to use this query to prefetch the values. This is like telling Django that you intend to fetch both items and programs for each order.

Fetching program names for an order:

>>> [p.program.name for p in o.items.all()]

['Day 1 Pass']

No additional queries - it worked!

Taking It To The Next Level

When we talked earlier about the models we mentioned that the prices are modeled as an SCD table. This means we might want to query only active prices at a certain date.

A price is active at a certain date if it's between from_date and end_date:

>>> from django.utils import timezone
>>> now = timezone.now()

>>> active_prices = Price.objects.filter(
...     from_date__lte=now,
...     to_date__gt=now,
... )

Using the Prefetch object we can tell Django to store the prefetched objects in a new attribute of the result set:

>>> from django.utils import timezone
>>> now = timezone.now()
>>> active_prices_and_programs = (
...     Price.objects.filter(
...         from_date__lte=now,
...         to_date__gt=now,
...     ).select_related('program')
... )
>>> o = Order.objects.filter(
...     state='completed'
... ).prefetch_related(
...     Prefetch(
...         'items',
...         queryset=active_prices_and_programs,
...         to_attr='active_prices',
...     ),
... ).first()

(0.001) SELECT ... FROM "orders_order"
WHERE "orders_order"."state" = 'completed'
ORDER BY "orders_order"."id" ASC
LIMIT 1;

(0.001) SELECT ... FROM "events_price"
INNER JOIN "orders_order_items" ON ("events_price"."id" = "orders_order_items"."price_id")
INNER JOIN "events_program" ON ("events_price"."program_id" = "events_program"."id")
WHERE ("orders_order_items"."order_id" IN (29)
AND "events_price"."from_date" <= '2017–04–29T07:53:00.210537+00:00'::timestamptz
AND "events_price"."to_date" > '2017–04–29T07:53:00.210537+00:00'::timestamptz);

We can see in the log that Django performed only two queries, and the prefetch query now include the custom filter we defined.

To fetch the active prices we can use the new attribute defined in to_attr:

>>> [p.program.name for p in o.active_prices]

['Day 1 Pass']

No additional query!

Final Words

Prefetch is a very powerful feature of Django ORM. I strongly recommend going over the documentation, you are bound to strike a gem.

How to Turn Django Admin Into a Lightweight Dashboard

2017-03-31T00:00:00+03:00

source code

The complete source code for this article can be found in this gist.

Django Admin is a powerful tool for managing data in your app. However, it was not designed with summary tables and charts in mind. Luckily, the developers of Django Admin made it easy for us to customize.

This is what it's going to look like at the end:

Django admin dashboard

Why Would I Want To Do That

There are a lot of tools, apps and packages out there that can produce very nice looking dashboards. I personally found that unless the product is an actual dashboard, most of the time all you need is a simple summary table and a few charts.

Second, and just as important - no dependencies.

If all you need is a little boost to your admin interface this approach is definitely worth considering.

Setup

We are going to use a made up Sale model.

To harness the full power of Django Admin we are going to base our dashboard on a the built-in ModelAdmin.

To do that we need a model:

# models.py

class SaleSummary(Sale):
    class Meta:
        proxy = True
        verbose_name = 'Sale Summary'
        verbose_name_plural = 'Sales Summary'

A proxy model extends the functionality of another model without creating an actual table in the database.

Now that we have a model we can create the ModelAdmin:

# admin.py

from django.contrib import admin
from .models import SaleSummary

@admin.register(SaleSummary)
class SaleSummaryAdmin(ModelAdmin):
    change_list_template = 'admin/sale_summary_change_list.html'
    date_hierarchy = 'created'

Because we are using a standard ModelAdmin we can use its features. In this example I added a date_hierarchy to filter sales by creation date. We are going to use this later for the chart.

To keep the page looking like a "regular" admin page we extend Django's change_list template and place our content in the result_list block:

<!-- sales/templates/admin/sale_summary_change_list.html -->

{% extends "admin/change_list.html" %}

{% block content_title %}
    <h1> Sales Summary </h1>
{% endblock %}

{% block result_list %}
    <!-- Our content goes here... -->
{% endblock %}

{% block pagination %}{% endblock %}

This is what our page looks like at this point:

A bare Django admin dashboard

Adding a Summary Table

The context sent to the template is populated by the ModelAdmin in a function called changelist_view.

To render the table in the template we fetch the data in changelist_view and add it to the context:

# admin.py

class SaleSummaryAdmin(ModelAdmin):

    # ...

    def changelist_view(self, request, extra_context=None):
        response = super().changelist_view(
            request,
            extra_context=extra_context,
        )

        try:
            qs = response.context_data['cl'].queryset
        except (AttributeError, KeyError):
            return response

        metrics = {
            'total': Count('id'),
            'total_sales': Sum('price'),
        }

        response.context_data['summary'] = list(
            qs
            .values('sale__category__name')
            .annotate(**metrics)
            .order_by('-total_sales')
        )

        return response

Let's break it down:

Call super to let Django do its thing (populate headers, breadcrumbs, queryset, filters and so on).
Extract the queryset created for us from the context. At this point the query is filtered with any inline filters or date hierarchy selected by the user.
If we can't fetch the queryset from the context it's most likely due to invalid query parameters. In cases like this Django will redirect so we don't interfere and return the response.
Aggregate total sales by category and return a list (the "metrics" dict will become clear in the next section).

Now that we have the data in the context we can render it in the template:

<!-- sale_summary_change_list.html -->

{% load humanize %}

<!-- ... -->

{% block result_list %}

<div class="results">
    <table>

    <thead>
      <tr>
        <th>
          <div class="text">
            <a href="#">Category</a>
          </div>
        </th>
        <th>
          <div class="text">
            <a href="#">Total</a>
          </div>
        </th>
        <th>
          <div class="text">
            <a href="#">Total Sales</a>
          </div>
        </th>
        <th>
          <div class="text">
            <a href="#">
              <strong>% Of Total Sales</strong>
            </a>
          </div>
        </th>
      </tr>
    </thead>

    <tbody>
      {% for row in summary %}
      <tr class="{% cycle 'row1' 'row2' %}">
        <td> {{ row.sale__category__name }} </td>
        <td> {{ row.total | intcomma }} </td>
        <td> {{ row.total_sales | default:0 | intcomma }}$ </td>
        <td>
          <strong>
          {{ row.total_sales |
              default:0 |
              percentof:summary_total.total_sales }}
          </strong>
        </td>
      </tr>
      {% endfor %}
    </tbody>

  </table>
</div>

<!-- ... -->

{% endblock %}

The markup is important. To get the native Django look we need to render tables in the same way Django renders them.

This is what we have so far:

Django admin dashboard with just a table

A summary table is not much without a bottom line. We can use the metrics and do some Django ORM voodoo to quickly calculate the bottom line:

# admin.py

class SaleSummaryAdmin(ModelAdmin):
    # ...
    def changelist_view(self, request, extra_context=None):
        # ...
        response.context_data['summary_total'] = dict(
            qs.aggregate(**metrics)
        )
        return response

That's a pretty cool trick…

Lets add the bottom line to the table:

<!-- sale_summary_change_list.html -->

<div class="results">
    <table>

        <!-- ... -->

        <tr style="font-weight:bold; border-top:2px solid #DDDDDD;">
            <td> Total </td>
            <td> {{ summary_total.total | intcomma }} </td>
            <td> {{ summary_total.total_sales | default:0 }}$ </td>
            <td> 100% </td>
        </tr>

    </table>
</div>

This is starting to take shape:

Django admin dashboard with a summary table

Adding Filters

We are using a "regular" model admin so filters are already baked in. Let's add a filter by device:

# admin.py

class SaleSummaryAdmin(ModelAdmin):

    # ...

    list_filter = (
        'device',
    )

And the result:

Django admin dashboard with a filter

Adding a Chart

A dashboard is not complete without a chart so we are going to add a bar chart to show sales over time.

To build our chart we are going to use plain HTML and some good ol' CSS with flexbox. The data for the chart is going to be a time series of percents to use as the bar height.

Back to our changelist_view, we add the following:

# admin.py

from django.db.models.functions import Trunc
from django.db.models import DateTimeField

class SalesSummaryAdmin(ModelAdmin):

    # ...

    def changelist_view(self, request, extra_context=None):

        # ...

        summary_over_time = qs.annotate(
            period=Trunc(
                'created',
                'day',
                output_field=DateTimeField(),
            ),
        ).values('period')
        .annotate(total=Sum('price'))
        .order_by('period')

        summary_range = summary_over_time.aggregate(
            low=Min('total'),
            high=Max('total'),
        )
        high = summary_range.get('high', 0)
        low = summary_range.get('low', 0)

        response.context_data['summary_over_time'] = [{
            'period': x['period'],
            'total': x['total'] or 0,
            'pct': \
               ((x['total'] or 0) - low) / (high - low) * 100
               if high > low else 0,
        } for x in summary_over_time]

    return response

Let's add the bar chart to the template and style it a bit:

<!-- sale_summary_change_list.html -->

<div class="results">

    <!-- ... -->

    <h2> Sales over time </h2>

    <style>
    .bar-chart {
      display: flex;
      justify-content: space-around;
      height: 160px;
      padding-top: 60px;
      overflow: hidden;
    }
    .bar-chart .bar {
        flex: 100%;
        align-self: flex-end;
        margin-right: 2px;
        position: relative;
        background-color: #79aec8;
    }
    .bar-chart .bar:last-child {
        margin: 0;
    }
    .bar-chart .bar:hover {
        background-color: #417690;
    }

    .bar-chart .bar .bar-tooltip {
        position: relative;
        z-index: 999;
    }
    .bar-chart .bar .bar-tooltip {
        position: absolute;
        top: -60px;
        left: 50%;
        transform: translateX(-50%);
        text-align: center;
        font-weight: bold;
        opacity: 0;
    }
    .bar-chart .bar:hover .bar-tooltip {
        opacity: 1;
    }
    </style>

    <div class="results">
        <div class="bar-chart">
        {% for x in summary_over_time %}
            <div class="bar" style="height:{{x.pct}}%">
                <div class="bar-tooltip">
                    {{x.total | default:0 | intcomma }}<br>
                    {{x.period | date:"d/m/Y"}}
                </div>
            </div>
        {% endfor %}
        </div>
    </div>

</div>

For those of you not familiar with flexbox, that piece of CSS means "draw from the bottom up, pull to the left and adjust the width to fit".

This is how it looks like now:

Django admin dashboard with a basic chart

That's looking pretty good, but... Each bar in the chart represents a day. What will happen when we try to show data for a single day? Or several years?

Daily chart for several years

A chart like that is both unreadable and dangerous. Fetching so much data will flood the server and generate a huge HTML file.

Django Admin has a date hierarchy - let's see if we can use that to adjust the period of the bars based on the selected date hierarchy:

def get_next_in_date_hierarchy(request, date_hierarchy):
    if date_hierarchy + '__day' in request.GET:
        return 'hour'

    if date_hierarchy + '__month' in request.GET:
        return 'day'

    if date_hierarchy + '__year' in request.GET:
        return 'week'

    return 'month'

If the user filtered a single day each bar will be one hour (max 24 bars).
If the user selected a month each bar will be one day (max 31 bars).
If the user selected a year each bar will be one week (max 52 bars).
More than that and each bar will be one month.

Now we need just one small adjustment to the change list view:

# admin.py

class SalesSummaryAdmin(ModelAdmin):

    # ...

    def changelist_view(self, request, extra_context=None):

        # ...

        period = get_next_in_date_hierarchy(request, self.date_hierarchy)
        response.context_data['period'] = period

        summary_over_time = qs.annotate(
            period=Trunc('created', period, output_field=DateTimeField()),
        ).values('period')
        .annotate(total=Sum('price'))
        .order_by('period')

        # ...

The period argument passed to Trunc is now a parameter. The result:

Django admin dashboard chart with adjusted period

That's a beautiful trend...

Where Can We Take It From Here?

Now that you have all this spare time from not rolling your own dashboard you can:

How to Test Django Signals Like a Pro

2017-02-18T00:00:00+02:00

Django Signals are extremely useful for decoupling modules. They allow a low-level Django app to send events for other apps to handle without creating a direct dependency.

The Use Case

Let's say you have a payment module with a charge function. (I write a lot about payments, so I know this use case well.) Once a charge is made, you want to increment a total charges counter.

What would that look like using signals?

First, define the signal:

# signals.py

from django.dispatch import Signal

charge_completed = Signal(providing_args=['total'])

Then send the signal when a charge completes successfully:

# payment.py

from .signals import charge_completed

@classmethod
def process_charge(cls, total):

    # Process charge...

    if success:
        charge_completed.send_robust(sender=cls, total=total)

A different app, such as a summary app, can connect a handler that increments a total charges counter:

# summary.py

from django.dispatch import receiver

from .signals import charge_completed

@receiver(charge_completed)
def increment_total_charges(sender, total, **kwargs):
    total_charges += total

The payment module does not have to know the summary module or any other module handling completed charges. You can add many receivers without modifying the payment module.

For example, the following are good candidates for receivers:

Update the transaction status.
Send an email notification to the user.
Update the last used date of the credit card.

Testing Signals

Now that you got the basics covered, let's write a test for process_charge. You want to make sure the signal is sent with the right arguments when a charge completes successfully.

The best way to test if a signal was sent is to connect to it:

# test.py

from django.test import TestCase

from .payment import charge
from .signals import charge_completed

class TestCharge(TestCase):

    def test_should_send_signal_when_charge_succeeds(self):
        self.signal_was_called = False
        self.total = None

        def handler(sender, total, **kwargs):
            self.signal_was_called = True
            self.total = total

        charge_completed.connect(handler)
        charge(100)

        self.assertTrue(self.signal_was_called)
        self.assertEqual(self.total, 100)

        charge_completed.disconnect(handler)

We create a handler, connect to the signal, execute the function and check the args.

We use self inside the handler to create a closure. If we hadn't used self the handler function would update the variables in its local scope and we won't have access to them. We will revisit this later.

Let's add a test to make sure the signal is not called if the charge failed:

# test.py

def test_should_not_send_signal_when_charge_failed(self):
    self.signal_was_called = False

    def handler(sender, total, **kwargs):
        self.signal_was_called = True

    charge_completed.connect(handler)

    charge(-1)
    self.assertFalse(self.signal_was_called)

    charge_completed.disconnect(handler)

This is working but it's a lot of boilerplate! There must be a better way.

Enter Context Manager

Let's break down what we did so far:

Connect a signal to some handler.
Run the test code and save the arguments passed to the handler.
Disconnect the handler from the signal.

This pattern sounds familiar...

Let's look at what a (file) open context manager does:

Open a file.
Process the file.
Close the file.

And a database transaction context manager:

Open transaction.
Execute some operations.
Close transaction (commit / rollback).

It looks like a context manager can work for signals as well.

Before you start, think how you want to use a context manager to test signals:

with CatchSignal(charge_completed) as signal_args:
    charge(100)

self.assertEqual(signal_args.total, 100)

Nice, let's give it a try:

class CatchSignal:

    def __init__(self, signal):
        self.signal = signal
        self.signal_kwargs = {}

        def handler(sender, **kwargs):
            self.signal_kwrags.update(kwargs)

        self.handler = handler

    def __enter__(self):
        self.signal.connect(self.handler)
        return self.signal_kwrags

    def __exit__(self, exc_type, exc_value, tb):
        self.signal.disconnect(self.handler)

What we have here:

You initialized the context with the signal you want to "catch".
The context creates a handler function to save the arguments sent by the signal.
You create closure by updating an existing object (signal_kwargs) on self.
You connect the handler to the signal.
Some processing is done (by the test) between __enter__ and __exit__.
You disconnect the handler from the signal.

Let's use the context manager to test the charge function:

def test_should_send_signal_when_charge_succeeds(self):
    with CatchSignal(charge_completed) as signal_args:
        charge(100)

    self.assertEqual(signal_args['total'], 100)

This is better, but how would the negative test look like?

def test_should_not_send_signal_when_charge_failed(self):
    with CatchSignal(signal) as signal_args:
        charge(100)

    self.assertEqual(signal_args, {})

Yak, that's bad.

Let's take another look at the handler:

We want to make sure the handler function was invoked.
We want to test the args sent to the handler function.

Wait... I already know this function!

Enter Mock

Let's replace our handler with a Mock:

from unittest import mock

class CatchSignal:

    def __init__(self, signal):
        self.signal = signal
        self.handler = mock.Mock()

    def __enter__(self):
        self.signal.connect(self.handler)
        return self.handler

    def __exit__(self, exc_type, exc_value, tb):
        self.signal.disconnect(self.handler)

And the tests:

# test.py

def test_should_send_signal_when_charge_succeeds(self):
    with CatchSignal(charge_completed) as handler:
        charge(100)

    handler.assert_called_once_with(
        total=100,
        sender=mock.ANY,
        signal=charge_completed,
    )

def test_should_not_send_signal_when_charge_failed(self):
    with CatchSignal(charge_completed) as handler:
        charge(-1)

    handler.assert_not_called()

Much better!

You used the mock for exactly what it should be used for, and you don't need to worry about scope and closure.

Now that you have this working, can you make it even better?

Enter `contextlib`

Python has a utility module for handling context managers called contextlib.

Let's rewrite our context using contextlib:

from unittest import mock
from contextlib import contextmanager

@contextmanager
def catch_signal(signal):
    """Catch django signal and return the mocked call."""
    handler = mock.Mock()
    signal.connect(handler)
    yield handler
    signal.disconnect(handler)

I like this approach better because it's easier to follow:

The yield makes it clear where the test code is executed.
No need to save objects on self because the setup code (enter and exit) are in the same scope.

And that's it, 4 lines of code to rule them all! Profit!

Working With APIs the Pythonic Way

2017-01-05T00:00:00+02:00

Communication with external services is an integral part of any modern system. Whether it's a payment service, authentication, analytics or an internal one - systems need to talk to each other.

In this short article we are going to implement a module for communicating with a made-up payment gateway, step by step.

It used to be harder

The External Service

Let's start by defining an imaginary payment service.

To charge a credit card we need a credit card token, an amount to charge (in cents) and some unique ID provided by the client (us):

POST
{
    token: <string>,
    amount: <number>,
    uid: <string>,
}

If the charge was successful we get a 200 OK status with the data from our request, an expiration time for the charge and a transaction ID:

200 OK
{
    uid: <string>,
    amount: <number>,
    token: <string>,
    expiration: <string, isoformat>,
    transaction_id: <number>
}

If the charge was not successful we get a 400 status with an error code and an informative message:

400 Bad Request
{
    uid: <string>,
    error: <number>,
    message: <string>
}

There are two error codes we want to handle - 1 = refused, and 2 = stolen.

Naive Implementation

To get the ball rolling, we start with a naive implementation and build from there:

# payments.py

import uuid
import requests

PAYMENT_GATEWAY_BASE_URL = 'https://gw.com/api'
PAYMENT_GATEWAY_TOKEN = 'topsecret'

def charge(
    amount,
    token,
    timeout=5,
):
    """Charge.

    amount (int):
        Amount in cents to charge.
    token (str):
        Credit card token.
    timeout (int):
        Timeout in seconds.

    Returns (dict):
        New payment information.
    """
    headers = {
        "Authorization": "Bearer " + PAYMENT_GATEWAY_TOKEN,
    }

    payload = {
        "token": token,
        "amount": amount,
        "uid": str(uuid.uuid4()),
    }

    response = requests.post(
        PAYMENT_GATEWAY_BASE_URL + '/charge',
        json=payload,
        headers=headers,
        timeout=timeout,
    )
    response.raise_for_status()

return response.json()

90% of developer will stop here, so what is the problem?

Handling Errors

There are two types of errors we need to handle:

HTTP errors such as connection errors, timeout or connection refused.
Remote payment errors such as refusal or stolen card.

Our decision to use requests is an internal implementation detail. The consumer of our module shouldn't have to be aware of that.

To provide a complete API our module must communicate errors.

Let's start by defining custom error classes:

# errors.py

class Error(Exception):
    pass

class Unavailable(Error):
    pass

class PaymentGatewayError(Error):
    def __init__(self, code, message):
        self.code = code
        self.message = message

class Refused(PaymentGatewayError):
    pass

class Stolen(PaymentGatewayError):
    pass

I previously wrote about the benefits of using a base error class.

Let's add exception handling and logging to our function:

import logging

from . import errors

logger = logging.getLogger('payments')

def charge(
    amount,
    token,
    timeout=5,
):

    # ...

    try:
        response = requests.post(
            PAYMENT_GATEWAY_BASE_URL + '/charge',
            json=payload,
            headers=headers,
            timeout=timeout,
        )
        response.raise_for_status()

    except (requests.ConnectionError, requests.Timeout) as e:
        raise errors.Unavailable() from e

    except requests.exceptions.HTTPError as e:
        if e.response.status_code == 400:
            error = e.response.json()
            code = error['code']
            message = error['message']

        if code == 1:
            raise errors.Refused(code, message) from e
        elif code == 2:
            raise errors.Stolen(code, message) from e
        else:
            raise errors.PaymentGatewayError(code, message) from e

        logger.exception("Payment service had internal error.")
        raise errors.Unavailable() from e

Great! Our function no longer raises requests exceptions. Important errors such as stolen card or refusal are raised as custom exceptions.

Defining the Response

Our function returns a dict. A dict is a great and flexible data structure, but when you have a defined set of fields you are better off using a more targeted data type.

In every OOP class you learn that everything is an object. While it is true in Java land, Python has a lightweight solution that works better in our case - namedtuple.

A namedtuple is just like it sounds, a tuple where the fields have names. You use it like a class and it consumes less space (even compared to a class with slots).

Let's define a namedtuple for the charge response:

from collections import namedtuple

ChargeResponse = namedtuple('ChargeResponse', [
    'uid',
    'amount',
    'token',
    'expiration',
    'transaction_id',
])

If the charge was successful, we create a ChargeResponse object:

from datetime import datetime

# ...

def charge(
    amount,
    token,
    timeout=5,
):

    # ...

    data = response.json()

    charge_response = ChargeResponse(
        uid=uuid.UID(data['uid']),
        amount=data['amount'],
        token=data['token'],
        expiration=datetime.strptime(data['expiration'], "%Y-%m-%dT%H:%M:%S.%f"),
        transaction_id=data['transaction_id'],
    )

    return charge_response

Our function now returns a ChargeResponse object. Additional processing such as casting and validations can be added easily.

In the case of our imaginary payment gateway, we convert the expiration date to a datetime object. The consumer doesn't have to guess the date format used by the remote service (when it comes to date formats I am sure we all encountered a fair share of horrors).

By using a custom "class" as the return value we reduce the dependency in the payment vendor‘s serialization format. If the response was an XML, would we still return a dict? That's just awkward.

Using a Session

To skim some extra milliseconds from API calls we can use a session. Requests session uses a connection pool internally. Requests to the same host can benefit from that. We also take the opportunity to add useful configuration such as blocking cookies:

import http.cookiejar

# A shared requests session for payment requests.
class BlockAll(http.cookiejar.CookiePolicy):
    def set_ok(self, cookie, request):
        return False

payment_session = requests.Session()
payment_session.cookies.policy = BlockAll()

# ...

def charge(
    amount,
    token,
    timeout=5,
):
    # ...
    response = payment_session.post( ... )
    # ...

More Actions

Any external service, and a payment service in particular, has more than one action.

The first section of our function takes care of authorization, the request and HTTP errors. The second part handle protocol errors and serialization specific to the charge action.

The first part is relevant to all actions while the second part is specific only to the charge.

Let's split the function so we can reuse the first part:

import uuid
import logging
import requests
import http.cookiejar
from datetime import datetime


logger = logging.getLogger('payments')


class BlockAll(http.cookiejar.CookiePolicy):
    def set_ok(self, cookie, request):
        return False

payment_session = requests.Session()
payment_session.cookies.policy = BlockAll()


def make_payment_request(path, payload, timeout=5):
    """Make a request to the payment gateway.

    path (str):
        Path to post to.
    payload (object):
        JSON-serializable request payload.
    timeout (int):
        Timeout in seconds.

    Raises
        Unavailable
        requests.exceptions.HTTPError

    Returns (response)
    """
    headers = {
        "Authorization": "Bearer " + PAYMENT_GATEWAY_TOKEN,
    }

    try:
        response = payment_session.post(
            PAYMENT_GATEWAY_BASE_URL + path,
            json=payload,
            headers=headers,
            timeout=timeout,
        )
    except (requests.ConnectionError, requests.Timeout) as e:
        raise errors.Unavailable() from e

    response.raise_for_status()
    return response.json()


def charge(amount, token):
    """Charge credit card.

    amount (int):
        Amount to charge in cents.
    token (str):
        Credit card token.

    Raises
        Unavailable
        Refused
        Stolen
        PaymentGatewayError

    Returns (ChargeResponse)
    """
    try:
        data = make_payment_request('/charge', {
            'uid': str(uuid.uuid4()),
            'amount': amount,
            'token': token,
        })

    except requests.HTTPError as e:
        if e.response.status_code == 400:
            error = e.response.json()
            code = error['code']
            message = error['message']

            if code == 1:
                raise Refused(code, message) from e

            elif code == 2:
                raise Stolen(code, message) from e

            else:
                raise PaymentGatewayError(code, message) from e

        logger.exception("Payment service had internal error")
        raise errors.Unavailable() from e

    return ChargeResponse(
        uid=uuid.UID(data['uid']),
        amount=data['amount'],
        token=data['token'],
        expiration=datetime.strptime(data['expiration'], "%Y-%m-%dT%H:%M:%S.%f"),
        transaction_id=data['transaction_id'],
    )

This is the entire code.

There is a clear separation between "transport", serialization, authentication and request processing. We also have a well defined interface to our top level function charge.

To add a new action we define a new return type, call make_payment_request and handle the response the same way:

RefundResponse = namedtuple('RefundResponse', [
    'transaction_id',
    'refunded_transaction_id',
])


def refund(transaction_id):
    """Refund charged transaction.

    transaction_id (str):
        Transaction id to refund.

    Raises:

    Return (RefundResponse)
    """
    try:
        data = make_payment_request('/refund', {
            'uid': str(uuid.uuid4()),
            'transaction_id': transaction_id,
        })

    except requests.HTTPError as e:
        # TODO: Handle refund remote errors

    return RefundResponse(
        'transaction_id': data['transaction_id'],
        'refunded_transaction_id': data['refunded_transaction_id'],
    )

Profit!

Testing

The challenge with external APIs is that you can't (or at least, shouldn't) make calls to them in automated tests. I want to focus on testing code that uses our payments module rather than testing the actual module.

Our module has a simple interface so it's easy to mock. Let's test a made up function called charge_user_for_product:

# test.py

from unittest import TestCase
from unittest.mock import patch

from payment.payment import ChargeResponse
from payment import errors

def TestApp(TestCase):

    @mock.patch('payment.charge')
    def test_should_charge_user_for_product(self, mock_charge):
        mock_charge.return_value = ChargeResponse(
            uid='test-uid',
            amount=1000,
            token='test-token',
            expiration=datetime.datetime(2017, 1, 1, 15, 30, 7),
            transaction_id=12345,
        )
        charge_user_for_product(user, product)
        self.assertEqual(user.approved_transactions, 1)

    @mock.patch('payment.charge')
    def test_should_suspend_user_if_stolen(self, mock_charge):
        mock_charge.side_effect = errors.Stolen
        charge_user_for_product(user, product)
        self.assertEqual(user.is_active, False)

Pretty straight forward - no need to mock the API response. The tests are contained to data structures we defined ourselves and have full control of.

Note About Dependency Injection

Another approach to test a service is to provide two implementations: the real one, and a fake one. Then for tests, inject the fake one.

This is of course, how dependency injection works. Django doesn't do DI but it utilizes the same concept with "backends" (email, cache, template, etc). For example you can test emails in django by using a test backend, test caching by using in-memory backend, etc.

This also has other advantages in that you can have multiple "real" backends.

Whether you choose to mock the service calls as illustrated above or inject a "fake" service, you must have a proper interface.

Summary

We have an external service we want to use in our app. We want to implement a module to communicate with that external service and make it robust, resilient and reusable.

We worked the following steps:

Naive implementation - Fetch using requests and return a json response.
Handled errors - Defined custom errors to catch both transport and remote application errors. The consumer is indifferent to the transport (HTTP, RPC, Web Socket) and implementation details (requests).
Formalize the return value - Used a namedtuple to return a class-like type that represents a response from the remote service. The consumer is now indifferent to the serialization format as well.
Added a session - Skimmed off a few milliseconds from the request and added a place for global connection configuration.
Split request from action - The request part is reusable and new actions can be added more easily.
Test - Mocked calls to our module and replaced them with our own custom exceptions.

Timing Tests in Python For Fun and Profit

2016-11-09T00:00:00+02:00

I was preparing to push some changes a couple of days ago and as I usually do, I ran the tests. I sat back in my chair as the dots raced across the screen when suddenly I noticed that one of the dots linger. "OS is probably running some updates in the background or something" I said to myself, and ran the tests again just to be sure. I watched closely as the dots filled the screen and there it was again - I have a slow test!

Can you spot the slow test? Neither can I...

We are going to hunt down slow tests by reporting tests that take longer than a certain threshold.

The Basics

To get the ball rolling let's create a simple test case with a fast test and a slow test:

import time
import unittest

class SlowTestCase(unittest.TestCase):

    def test_should_run_fast(self):
        self.assertEqual(1, 1)

    def test_should_run_slow(self):
        time.sleep(0.5)
        self.assertEqual(1, 1)

Running this script from the command line produces the following output:

$ python -m unittest timing.py
..
Ran 2 tests in 0.502s

OK

I'm sorry unittest, but this is definitely not OK - 0.5s for two tests?

To figure out which tests are slow we need to measure the time it takes each test to execute.

A python unittest.TestCase has hooks that execute in the following order:

> setUpClass
    > setUp
        > test_*
    > tearDown
> tearDownClass

If we want to time a single test (test_*) we need to start a timer in setUp and stop it in tearDown:

import time
import unittest

class SlowTestCase(unittest.TestCase):

    def setUp(self):
        self._started_at = time.time()

    def tearDown(self):
        elapsed = time.time() - self._started_at
        print('{} ({}s)'.format(self.id(), round(elapsed, 2)))

    def test_should_run_fast(self):
        self.assertEqual(1, 1)

    def test_should_run_slow(self):
        time.sleep(0.5)
        self.assertEqual(1, 1)

This produces the following output:

$ python -m unittest timing.py

__main__.SlowTestCase.test_should_run_fast (0.0s)
.__main__.SlowTestCase.test_should_run_slow (0.5s)
.

Ran 2 tests in 0.503s
OK

Great! We got the timing for each test but we really want only the slow ones.

Let's say a slow test is a test that takes longer than 0.3s:

SLOW_TEST_THRESHOLD = 0.3

class SlowTestCase(unittest.TestCase):

    # ...

    def tearDown(self):
        elapsed = time.time() - self._started_at
        if elapsed > SLOW_TEST_THRESHOLD:
            print('{} ({}s)'.format(self.id(), round(elapsed, 2)))

And the output:

> python -m unittest timing.py
.__main__.SlowTestCase.test_should_run_slow (0.5s)
.
Ran 2 tests in 0.503s

OK

Awesome! We got exactly what we wanted but it is still incomplete. We are good developers so we are most likely dead lazy. We don't want to go around and update every test case - we need a more robust solution.

The Runner

One of the roles of the TestRunner is to print test results to an output stream. The runner uses a TestResult object to format the results. The unittest module comes with a default TextTestRunner and TextTestResult.

Let's implement a custom TestResult to report slow tests:

import time
from unittest.runner import TextTestResult

SLOW_TEST_THRESHOLD = 0.3

class TimeLoggingTestResult(TextTestResult):

    def startTest(self, test):
        self._started_at = time.time()
        super().startTest(test)

    def addSuccess(self, test):
        elapsed = time.time() - self._started_at

        if elapsed > SLOW_TEST_THRESHOLD:
            name = self.getDescription(test)
            self.stream.write("\n{} ({:.03}s)\n".format(name, elapsed))

        super().addSuccess(test)

Almost identical to what we already have but using different hooks. Instead of setUp we use testStart and instead of tearDown we use addSuccess.

The built-in TextTestRunner uses TextTestResult. To use a different TestResult we create an instance of TextTestRunner with our runner:

from unittest import TextTestRunner

if __name__ == '__main__':
    test_runner = TextTestRunner(resultclass=TimeLoggingTestResult)
    unittest.main(testRunner=test_runner)

And the output:

$ python runner.py
.
test_should_run_slow (__main__.SlowTestCase) (0.501s)
.
Ran 2 tests in 0.501s

OK

We get a nice report without having to make any changes to existing test cases.

Can We Do Better?

Right now we have a bunch of messages sprinkled around in random places across the screen. What if we could get a nice report with all the slow tests? Well, we can!

Let's start by making our TestResult store the timings without reporting them:

import time
from unittest.runner import TextTestResult

class TimeLoggingTestResult(TextTestResult):

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.test_timings = []

    def startTest(self, test):
        self._test_started_at = time.time()
        super().startTest(test)

    def addSuccess(self, test):
        elapsed = time.time() - self._test_started_at
        name = self.getDescription(test)
        self.test_timings.append((name, elapsed))
        super().addSuccess(test)

    def getTestTimings(self):
        return self.test_timings

The test result now holds a list of tuples containing the test name and the elapsed time. Moving over to our custom TestRunner:

# common/test/runner.py

import unittest

class TimeLoggingTestRunner(unittest.TextTestRunner):

    def __init__(self, slow_test_threshold=0.3, *args, **kwargs):
        self.slow_test_threshold = slow_test_threshold
        return super().__init__(resultclass=TimeLoggingTestResult, *args, **kwargs)

    def run(self, test):
        result = super().run(test)

        self.stream.writeln("\nSlow Tests (>{:.03}s):".format(self.slow_test_threshold))
        for name, elapsed in result.getTestTimings():
            if elapsed > self.slow_test_threshold:
                self.stream.writeln("({:.03}s) {}".format(elapsed, name))

        return result

Let's break it down:

We've replaced SLOW_TEST_THRESHOLD with a parameter to the init - Much cleaner.
We've set the appropriate TestResult class.
We've override run and add our custom "slow test" report.

This is what the output looks like (I added some slow tests to illustrate):

> python timing.py
.....
Ran 5 tests in 1.706s
OK

Slow Tests (>0.3s):
(0.501s) test_should_run_slow (__main__.SlowTestCase)
(0.802s) test_should_run_very_slow (__main__.SlowTestCase)
(0.301s) test_should_run_slow_enough (__main__.SlowTestCase)

Now that we have the timing data we can use that to generate interesting reports. We can sort by elapsed time, show potential time reduction and highlight sluggish tests.

How To Use This With Django

Django has its own test runner so we need to make some adjustments:

# common/test/runner.py

from django.test.runner import DiscoverRunner

# ...

class TimeLoggingTestRunner(DiscoverRunner):

    def get_resultclass(self):
        return TimeLoggingTestResult

And to make Django use our custom runner we set the following:

# settings.py

TEST_RUNNER = 'common.tests.runner.TimeLoggingTestRunner'

Go make some tests faster!

How to Add Custom Action Buttons to Django Admin

2016-11-02T00:00:00+02:00

We are big fans of the Django admin interface. It's a huge selling point for Django as it takes the load off developing a "back office" for support and day to day operations.

In the last post we presented a pattern we use often in our Django models. We used a bank account application with an Account and account Action models to demonstrate the way we handle common issues such as concurrency and validation. The bank account had two operations we wanted to expose in the admin interface - deposit and withdraw.

We are going to add buttons in the Django admin interface to deposit and withdraw from an account, and we are going to do it in less than 100 lines of code!

What will it look like at the end?

Django Admin interface with custom action buttons

Our custom actions are the nice looking deposit and withdraw buttons next to each account.

Why Not Use the Existing Admin Actions?

The built-in admin actions operate on a queryset. They are hidden in a dropbox menu in the top toolbar, and they are mostly useful for executing bulk operations. A good example is the default delete action. You mark several rows and select "delete rows" form the drop down menu. This is not very fleixble, and not suitable for some use cases.

Django built in actions

Another downside, is that actions are not available in the detail view. To add buttons to the detail view you need to override the template, A huge pain.

The Forms

First thing first, we need some data from the user to perform the action. Naturally, we need a form. We need one form for deposit, and one form for withdraw.

In addition to performing the action, we are going to add a nifty option to send a notification email to the account owner, informing him about an action made to his account.

All of our actions have common arguments like comment and send_email. The actions also handle success and failure in a similar way.

Let's start with a base form to handle a general action:

# forms.py

from django import forms
from common.utils import send_email

from . import errors

class AccountActionForm(forms.Form):
    comment = forms.CharField(
        required=False,
        widget=forms.Textarea,
    )
    send_email = forms.BooleanField(
        required=False,
    )

    @property
    def email_subject_template(self):
        return 'email/account/notification_subject.txt'

    @property
    def email_body_template(self):
        raise NotImplementedError()

    def form_action(self, account, user):
        raise NotImplementedError()

    def save(self, account, user):
        try:
            account, action = self.form_action(account, user)
        except errors.Error as e:
            error_message = str(e)
            self.add_error(None, error_message)
            raise

        if self.cleaned_data.get('send_email', False):
            send_email(
                to=[account.user.email],
                subject_template=self.email_subject_template,
                body_template=self.email_body_template,
                context={
                    "account": account,
                    "action": action,
                }
            )

    return account, action

Every action has a comment, and an option to send a notification if the action completed successfully.
The actual operation is executed when the form is saved. This is similar to how ModelForm works.
For logging and audit purposes, the caller must provide the user executing the action.
Required properties that are not implemented by derived forms, will raise NotImplementedError. This way, we make sure the developer gets an informative error message if she forgets to implement something.
Errors are handled at the base exception class level. Our models define a base error class, so we can easily catch all (and only) account related exceptions, and handle them appropriately.

Now that we have a simple base class, let's add a form to withdraw from an account. The withdraw we need to add an amount field:

# forms.py
from django.utils import timezone

from .models import Account, Action

class WithdrawForm(AccountActionForm):
    amount = forms.IntegerField(
        min_value=Account.MIN_WITHDRAW,
        max_value=Account.MAX_WITHDRAW,
        required=True,
        help_text='How much to withdraw?',
    )

    email_body_template = 'email/account/withdraw.txt'

    field_order = (
        'amount',
        'comment',
        'send_email',
    )

    def form_action(self, account, user):
        return Account.withdraw(
            id=account.pk,
            user=account.user,
            amount=self.cleaned_data['amount'],
            withdrawn_by=user,
            comment=self.cleaned_data['comment'],
            asof=timezone.now(),
        )

We extended the base AccountActionForm and added an amount field with proper validations.
We filled in the required attributes, email_body_template.
We implemented the form action using the classmethod from the previous post. The model takes care of locking the record, updating any calculated fields, and adding the proper action to the log.

The next step is to add a deposit action. Deposit requires amount, reference and reference type fields:

# forms.py

class DepositForm(AccountActionForm):
    amount = forms.IntegerField(
        min_value=Account.MIN_DEPOSIT,
        max_value=Account.MAX_DEPOSIT,
        required=True,
        help_text='How much to deposit?',
    )
    reference_type = forms.ChoiceField(
        required=True,
        choices=Action.REFERENCE_TYPE_CHOICES,
    )
    reference = forms.CharField(
        required=False,
    )

    email_body_template = 'email/account/deposit.txt'

    field_order = (
        'amount',
        'reference_type',
        'reference',
        'comment',
        'send_email',
    )

    def form_action(self, account, user):
        return Account.deposit(
            id=account.pk,
            user=account.user,
            amount=self.cleaned_data['amount'],
            deposited_by=user,
            reference=self.cleaned_data['reference'],
            reference_type=self.cleaned_data['reference_type'],
            comment=self.cleaned_data['comment'],
            asof=timezone.now(),
        )

So far we got the necessary forms to accept, validate and execute deposit and withdraw. The next step is to integrate it into Django Admin list view.

The Admin

Before we can add fancy buttons for actions, we need to set up a basic admin page for our Account model:

# admin.py
from django.contrib import admin

from .models import Account

@admin.register(Account)
class AccountAdmin(admin.ModelAdmin):
    date_heirarchy = (
        'modified',
    )
    list_display = (
        'id',
        'user',
        'modified',
        'balance',
        'account_actions',
    )
    readonly_fields = (
        'id',
        'user',
        'modified',
        'balance',
        'account_actions',
    )
    list_select_related = (
        'user',
    )

    def account_actions(self, obj):
        # TODO: Render action buttons

Side Note: We can make the list view much better by adding a link to the user, and to the account actions. We can add some search fields and many more. I previously wrote about performance considerations in the admin interface when scaling a Django app to hundreds of thousands of users. There are some nice tricks there that can make even this simple view much nicer.

Adding the Action Buttons

We want to add action buttons for each account, and have them link to a page with a form. Django has a function to register URLs in a list view. Let's use that function to add the routes for our custom actions:

# admin.py

from django.utils.html import format_html
from django.core.urlresolvers import reverse

class AccountAdmin(admin.ModelAdmin):

    # ...

    def get_urls(self):
        urls = super().get_urls()
        custom_urls = [
            url(
                r'^(?P<account_id>.+)/deposit/$',
                self.admin_site.admin_view(self.process_deposit),
                name='account-deposit',
            ),
            url(
                r'^(?P<account_id>.+)/withdraw/$',
                self.admin_site.admin_view(self.process_withdraw),
                name='account-withdraw',
            ),
        ]
        return custom_urls + urls

    def account_actions(self, obj):
        return format_html(
            '<a class="button" href="{}">Deposit</a>&nbsp;'
            '<a class="button" href="{}">Withdraw</a>',
            reverse('admin:account-deposit', args=[obj.pk]),
            reverse('admin:account-withdraw', args=[obj.pk]),
        )
    account_actions.short_description = 'Account Actions'
    account_actions.allow_tags = True

    def process_deposit(self):
        # TODO

    def process_withdraw(self):
        # TODO

We registered two urls, one for deposit and one for withdraw.
We referenced each route to a relevant function, process_deposit and process_withdraw. These function will render an intermediate page with the corresponding form, and will carry out the operation.
We added a custom field account_actions to render the buttons for each action. The benefit of using a "regular" admin field like account_actions, is that it is available in both the detail and the list view.

Let's move on to implement the functions to handle the actions:

# admin.py

from django.http import HttpResponseRedirect
from django.template.response import TemplateResponse

from .forms import DepositForm, WithdrawForm

class AccountAdmin(admin.ModelAdmin):

   # ...

   def process_deposit(self, request, account_id, *args, **kwargs):
        return self.process_action(
            request=request,
            account_id=account_id,
            action_form=DepositForm,
            action_title='Deposit',
        )

   def process_withdraw(self, request, account_id, *args, **kwargs):
        return self.process_action(
            request=request,
            account_id=account_id,
            action_form=WithdrawForm,
            action_title='Withdraw',
        )

   def process_action(
        self,
        request,
        account_id,
        action_form,
        action_title
   ):
        account = self.get_object(request, account_id)

        if request.method != 'POST':
            form = action_form()

        else:
            form = action_form(request.POST)
            if form.is_valid():
                try:
                    form.save(account, request.user)
                except errors.Error as e:
                    # If save() raised, the form will a have a non
                    # field error containing an informative message.
                    pass
                else:
                    self.message_user(request, 'Success')
                    url = reverse(
                        'admin:account_account_change',
                       args=[account.pk],
                        current_app=self.admin_site.name,
                    )
                    return HttpResponseRedirect(url)

        context = self.admin_site.each_context(request)
        context['opts'] = self.model._meta
        context['form'] = form
        context['account'] = account
        context['title'] = action_title

        return TemplateResponse(
            request,
            'admin/account/account_action.html',
            context,
        )

Deposit and withdraw are executed in a very similar way. To cary out both actions, we want to render a form in an intermediate page, and execute the action when the user submit the form.

process_action handles the form submission for both actions. The function accepts a form, the title of the action, and the account id. process_withdraw and process_deposit, are used to set the relevant context for each operation.

NOTE: There is some Django admin boilerplate here that is required by the Django admin site. No point in digging too deep into it because it's not relevant to us at this point.

To complete the process, we need a template for the intermediate page that contains the action form. We are going to base our template on an existing template used by Django Admin it self:

<!-- templates/admin/account/account_action.html -->

{% extends "admin/change_form.html" %}
{% load i18n admin_static admin_modify %}

{% block content %}

<div id="content-main">

  <form action="" method="POST">

    {% csrf_token %}
    {% if form.non_field_errors|length > 0 %}
      <p class="errornote">
          "Please correct the errors below."
      </p>
      {{ form.non_field_errors }}
    {% endif %}

    <fieldset class="module aligned">
      {% for field in form %}
        <div class="form-row">
          {{ field.errors }}
          {{ field.label_tag }}
          {{ field }}
          {% if field.field.help_text %}
          <p class="help">
            {{ field.field.help_text|safe }}
          </p>
          {% endif %}
        </div>
      {% endfor %}
    </fieldset>

    <div class="submit-row">
      <input type="submit" class="default" value="Submit">
    </div>

  </form>
</div>

{% endblock %}

This is it!

Staff members can now easily deposit and withdraw directly from the admin interface. No need to create an expensive dashboard or ssh to the server.

I promised we will do it in 100 lines and we did it in less!

Where Can We Take It From Here

Now that we nailed this technique we can pretty much do whatever we want with it. We have total control over the route and what's being rendered. The next step would be to abstract this functionality and put in a mixin, but, we will cross that bridge when we get there.

Credits

Big parts of the implementation are taken from the excellent (excellent!) package django-import-export. It saved us hours of "Can you just send me the data in Excel?" and we love it for it. If you are not familiar with it you should definitely check it out.

Bullet Proofing Django Models

2016-10-26T00:00:00+03:00

We recently added a bank account like functionality into one of our products. During the development we encountered some textbook problems and I thought it can be a good opportunity to go over some of the patterns we use in our Django models.

A Bank Account

This article was written in the order in which we usually address new problems:

Define the business requirements.
Write down a naive implementation and model definition.
Challenge the solution.
Refine and repeat.

Business Requirements

Each user can have only one account but not every user must have one.
The user can deposit and withdraw from the account up to a certain amount.
The account balance cannot be negative.
There is a max limit to the user's account balance.
The total amount of all balances in the app cannot exceed a certain amount.
There must be a record for every action on the account.
Actions on the account can be executed by the user from either the mobile app or the web interface and by support personnel from the admin interface.

Now that we have the business requirements we can start with a model definition.

Account Model

# models.py

import uuid

from django.conf import settings
from django.db import models

class Account(models.Model):
    class Meta:
        verbose_name = 'Account'
        verbose_name_plural = 'Accounts'

    MAX_TOTAL_BALANCES = 10000000

    MAX_BALANCE = 10000
    MIN_BALANCE = 0

    MAX_DEPOSIT = 1000
    MIN_DEPOSIT = 1

    MAX_WITHDRAW = 1000
    MIN_WITHDRAW = 1

    id = models.AutoField(
        primary_key=True,
    )
    uid = models.UUIDField(
        unique=True,
        editable=False,
        default=uuid.uuid4,
        verbose_name='Public identifier',
    )
    user = models.OneToOneField(
        settings.AUTH_USER_MODEL,
        on_delete=models.PROTECT,
    )
    created = models.DateTimeField(
        blank=True,
    )
    modified = models.DateTimeField(
        blank=True,
    )
    balance = models.PositiveIntegerField(
        verbose_name='Current balance',
    )

Let's break it down:

We use two unique identifiers - A private identifier which is an auto generated number (id) and a public id which is a uuid (uid). It's a good idea to keep enumerators private - They expose important information about our data such as how many accounts we have and we don't want that.
We use OneToOneField for the user - It's like a ForeignKey but with a unique constraint. This ensures a user cannot have more than one account.
We set on_delete=models.PROTECT - Starting with Django 2.0 this argument will be mandatory. The default is CASCADE - when the user is deleted the related account is deleted as well. In our case this doesn't make sense - imagine the bank "deleting your money" when you close an account. Setting on_delete=models.PROTECT will raise an IntegrityError when attempting to delete a user with an account.
You probably noticed that the code is very... "vertical". Wwe write like that because it makes git diffs look nicer.

Account Action Model

Now that we have an account model we can create a model to log actions made to the account:

# models.py

class Action(models.Model):
    class Meta:
        verbose_name = 'Account Action'
        verbose_name_plural = 'Account Actions'

    ACTION_TYPE_CREATED = 'CREATED'
    ACTION_TYPE_DEPOSITED = 'DEPOSITED'
    ACTION_TYPE_WITHDRAWN = 'WITHDRAWN'
    ACTION_TYPE_CHOICES = (
        (ACTION_TYPE_CREATED, 'Created'),
        (ACTION_TYPE_DEPOSITED, 'Deposited'),
        (ACTION_TYPE_WITHDRAWN, 'Withdrawn'),
    )

    REFERENCE_TYPE_BANK_TRANSFER = 'BANK_TRANSFER'
    REFERENCE_TYPE_CHECK = 'CHECK'
    REFERENCE_TYPE_CASH = 'CASH'
    REFERENCE_TYPE_NONE = 'NONE'
    REFERENCE_TYPE_CHOICES = (
        (REFERENCE_TYPE_BANK_TRANSFER, 'Bank Transfer'),
        (REFERENCE_TYPE_CHECK, 'Check'),
        (REFERENCE_TYPE_CASH, 'Cash'),
        (REFERENCE_TYPE_NONE, 'None'),
    )

    id = models.AutoField(
        primary_key=True,
    )
    user_friendly_id = models.CharField(
        unique=True,
        editable=False,
        max_length=30,
    )
    user = models.ForeignKey(
        settings.AUTH_USER_MODEL,
        on_delete=models.PROTECT,
        help_text='User who performed the action.',
    )
    created = models.DateTimeField(
        blank=True,
    )
    account = models.ForeignKey(
        Account,
    )
    type = models.CharField(
        max_length=30,
        choices=ACTION_TYPE_CHOICES,
    )
    delta = models.IntegerField(
        help_text='Balance delta.',
    )
    reference = models.TextField(
        blank=True,
    )
    reference_type = models.CharField(
        max_length=30,
        choices=REFERENCE_TYPE_CHOICES,
        default=REFERENCE_TYPE_NONE,
    )
    comment = models.TextField(
        blank=True,
    )

    # Fields used solely for debugging purposes.

    debug_balance = models.IntegerField(
        help_text='Balance after the action.',
    )

What do we have here?

Each record will hold a reference to the associated balance and the delta amount. A deposit of 100$ will have a delta of 100$, and a withdrawal of 50$ will have a delta of -50$. This way we can sum the deltas of all actions made to an account and get the current balance. This is important for validating our calculated balance.
We follow the same pattern of adding two identifiers - a private and a public one. The difference here is that reference numbers for actions are often used by users and support personnel to identify a specific action over the phone or in emails. A uuid is not user friendly- it's very long and it's not something users are used to see. I found a nice implementation of user-friendly ID's in django-invoice.
Two of the fields are only relevant for one type of action, deposit - reference and reference type. There are a lot of ways to tackle this issue - table inheritance and down casting, JSON fields, table polymorphism and the list of overly complicated solutions goes on. In our case we are going to use a sparse table.

Note about the design: Maintaining calculated fields in the model is usually bad design. Calculated fields such as the account's balance should be avoided whenever possible.

However, in our "real life" implementation there are additional action types and thousands of actions on each account - we treat calculated attribute as an optimization. Maintaining state poses some interesting challenges and we thought it can serve the purpose of this post so we decided to present it as well.

Challenges

Multiple Platforms

We have three client applications we need to support:

Mobile app - Uses an API interface to manage the account.
Web client - Uses either an API interface (if we have some sort of SPA), or good old server side rendering with Django forms.
Admin interface - Uses Django's admin module with Django forms.

Our motivation is to keep things DRY and self contained as possible.

Validation

We have two types of validations hiding in the business requirements:

Input validation such as "amount must be between X and Y", "balance cannot exceed Z", etc - these types of validation are well supported by Django and can usually be expressed as database constraints or django validations.

The second validation is a bit more complicated. We need to ensure the total amount of all balances in the entire system does not exceed a certain amount. This forces us to validate an instance against all other instances of the model.

Atomicity

Race conditions are a very common issue in distributed systems and ever more so in models that maintain state such as bank account (you can read more about race conditions in Wikipedia).

To illustrate the problem consider an account with a balance of 100$. The user connects from two different devices at the exact same time and issue a withdraw of 100$. Since the two actions were executed at the exact same time it is possible that both of them get a current balance of 100$. Given that both session see sufficient balance they will both get approved and update the new balance to 0$. The user withdrawn a total of 200$ and the current balance is now 0$ - we have a race condition and we are down 100$.

Logging / History

The log serves two purposes:

Log and Audit - Information about historical actions - dates, amounts, users etc.
Check Consistency - We maintain state in the model so we want to be able to validate the calculated balance by aggregating the action deltas.

The history records must be 100% immutable.

The Naive Implementation

Let's start with a naive implementation of deposit (this is not a good implementation):

class Account(models.Model):

    # ...

    def deposit(self, amount, deposited_by, asof):
        assert amount > 0

        if not self.MIN_DEPOSIT <= amount <= self.MAX_DEPOSIT:
            raise InvalidAmount(amount)

        if self.balance + amount > self.MAX_BALANCE:
            raise ExceedsLimit()

        total = Account.objects.aggregate(
            total=Sum('balance')
        )['total']
        if total + amount > self.MAX_TOTAL_BALANCES:
            raise ExceedsLimit()

        action = self.actions.create(
            user=deposited_by,
            type=Action.ACTION_TYPE_DEPOSITED,
            delta=amount,
            asof=asof,
        )

    self.balance += amount
    self.modified = asof

    self.save()

And let's add a simple endpoint for it using DRF @api_view:

# api.py

# ...
from django.db.models import transaction
# ...

@api_view('POST')
def deposit(request):
    try:
        amount = int(request.data\['amount'\])
    except (KeyError, ValueError):
        return Response(status=status.HTTP_400_BAD_REQUEST)

    with transaction.atomic():
        try:
            account = (
                Account.objects
                .select_for_update()
                .get(user=request.user)
            )
        except Account.DoesNotExist:
            return Response(status=status.HTTP_404_NOT_FOUND)

        try:
            account.deposit(
                amount=amount,
                deposited_by=request.user,
                asof=timezone.now(),
            )
        except (ExceedsLimit, InvalidAmount):
            return Response(status=status.HTTP_400_BAD_REQUEST)

        return Response(status=status.HTTP_200_OK)

So what is the problem?

Locking the account - An instance cannot lock itself because it had already been fetched. We gave up control over the locking and fetching so we have to trust the caller to properly obtain a lock - this is very bad design. Don't take my word for it, let's take a glimpse at Django's design philosophy:

Loose coupling A fundamental goal of Django's stack is loose coupling and tight cohesion. The various layers of the framework shouldn't "know" about each other unless absolutely necessary.

So is it really the business of our API, forms and django admin to fetch the account for us and obtain a proper lock? I think not.

Validation - The account has to validate itself against all other accounts - this just feels awkward.

A Better Approach

We need to hook into the process before the account is fetched (to obtain a lock) and in a place where it makes sense to validate and process more than one account.

Let's start with a function to create an Action instance and write it as a classmethod:

# models.py

from django.core.exceptions import ValidationError


class Action(models.Model):

    # ...

    @classmethod
    def create(
        cls,
        user,
        account,
        type,
        delta,
        asof,
        reference=None,
        reference_type=None,
        comment=None,
    ):
        """Create Action.

        user (User):
            User who executed the action.
        account (Account):
            Account the action executed on.
        type (str, one of Action.ACTION_TYPE_\*):
            Type of action.
        delta (int):
            Change in balance.
        asof (datetime.datetime):
            When was the action executed.
        reference (str or None):
            Reference number when appropriate.
        reference_type(str or None):
            Type of reference.
            Defaults to "NONE".
        comment (str or None):
            Optional comment on the action.

        Raises:
            ValidationError

        Returns (Action)
        """
        assert asof is not None

        if (type == cls.ACTION_TYPE_DEPOSITED and
            reference_type is None):
            raise errors.ValidationError({
                'reference_type': 'required for deposit.',
            })

        if reference_type is None:
            reference_type = cls.REFERENCE_TYPE_NONE

        # Don't store null in text field.

        if reference is None:
            reference = ''

        if comment is None:
            comment = ''

        user_friendly_id = generate_user_friendly_id()

        return cls.objects.create(
            user_friendly_id=user_friendly_id,
            created=asof,
            user=user,
            account=account,
            type=type,
            delta=delta,
            reference=reference,
            reference_type=reference_type,
            comment=comment,
            debug_balance=account.balance,
        )

What do we have here:

We used a classmethod that accepts all necessary data to validate and create the new instance. By *not* using the default manager's create function (Action.objects.create) we encapsulate all the business logic in the creation process.
We easily introduced custom validation and raise proper ValidationError.
We accept the creation time as an argument. That might seem a bit strange at first glance - why not use the built in auto_time_add? For starters It's much easier to test with predictable values. Second, as we are going to see in just a bit, we can make sure the modified time of the account is exactly the same as the action created time.

Before moving over to the implementation of the account methods let's define custom exceptions for our Account module:

# errors.py

class Error(Exception):
    pass

class ExceedsLimit(Error):
    pass

class InvalidAmount(Error):
    def __init__(self, amount):
        self.amount = amount

    def __str__(str):
        return 'Invalid Amount: {}'.format(amount)

class InsufficientFunds(Error):
    def __init__(self, balance, amount):
        self.balance = balance
        self.amount = amount

    def __str__(self):
        return 'amount: {}, current balance: {}'.format(
                self.amount, self.balance)

We define a base Error class that inherits from Exception. This is something we found very useful and we use it a lot. A base error class allows us to catch all errors coming from a certain module:

from account.errors import Error as AccountError

try:
   # action on account
except AccountError:
   # Handle all errors from account

A similar pattern can be found in the popular requests package.

Let's implement the method to create a new Account:

class Account(models.Model):

    # ...

    @classmethod
    def create(cls, user, created_by, asof):
        """Create account.

        user (User):
            Owner of the account.
        created_by (User):
            User that created the account.
        asof (datetime.datetime):
            Time of creation.

        Returns (tuple):
            [0] Account
            [1] Action
        """
        with transaction.atomic():
            account = cls.objects.create(
                user=user,
                created=asof,
                modified=asof,
                balance=0,
            )

            action = Action.create(
                user=created_by,
                account=account,
                type=Action.ACTION_TYPE_CREATED,
                delta=0,
                asof=asof,
            )

        return account, action

Pretty straight forward - create the instance, create the action and return them both.

Notice how we accept asof here as well - modified, created and the action creation time are all equal - you cant do that with auto_add and auto_add_now.

Now to the business logic:

# models.py

@classmethod
def deposit(
    cls,
    uid,
    deposited_by,
    amount,
    asof,
    comment=None,
):
    """Deposit to account.

    uid (uuid.UUID):
        Account public identifier.
    deposited_by (User):
        Deposited by.
    amount (positive int):
        Amount to deposit.
    asof (datetime.datetime):
        Time of deposit.
    comment(str or None):
       Optional comment.

    Raises
        Account.DoesNotExist
        InvalidAmount
        ExceedsLimit

    Returns (tuple):
        [0] (Account) Updated account instance.
        [1] (Action) Deposit action.
    """
    assert amount > 0

    with transaction.atomic():
        account = cls.objects.select_for_update().get(uid=uid)

        if not (cls.MIN_DEPOSIT <= amount <= cls.MAX_DEPOSIT):
            raise errors.InvalidAmount(amount)

        if account.balance + amount > cls.MAX_BALANCE:
            raise errors.ExceedsLimit()

        total = cls.objects.aggregate(total=Sum('balance'))\['total'\]
        if total + amount > cls.MAX_TOTAL_BALANCES:
            raise errors.ExceedsLimit()

        account.balance += amount
        account.modified = asof

        account.save(update_fields=[
            'balance',
            'modified',
        ])

        action = Action.create(
            user=deposited_by,
            account=account,
            type=Action.ACTION_TYPE_DEPOSITED,
            delta=amount,
            asof=asof,
        )

    return account, action

@classmethod
def withdraw(
    cls,
    uid,
    withdrawn_by,
    amount,
    asof
    comment=None,
):
    """Withdraw from account.

    uid (uuid.UUID):
        Account public identifier.
    withdrawn_by (User):
        The withdrawing user.
    amount (positive int):
        Amount to withdraw.
    asof (datetime.datetime):
        Time of withdraw.
    comment (str or None):
       Optional comment.

    Raises:
        Account.DoesNotExist
        InvalidAmount
        InsufficientFunds

    Returns (tuple):
        [0] (Account) Updated account instance.
        [1] (Action) Withdraw action.
    """
    assert amount > 0

    with transaction.atomic():
        account = cls.objects.select_for_update().get(uid=uid)

        if not (cls.MIN_WITHDRAW <= amount <= cls.MAX_WITHDRAW):
            raise InvalidAmount(amount)

        if account.balance - amount < cls.MIN_BALANCE:
            raise InsufficientFunds(amount, account.balance)

        account.balance -= amount
        account.modified = asof

        account.save(update_fields=[
            'balance',
            'modified',
        ])

        action = Action.create(
            user=withdrawn_by,
            account=account,
            type=Action.ACTION_TYPE_WITHDRAWN,
            delta=-amount,
            asof=asof,
        )

    return account, action

We can start to see the pattern here:

Acquire a lock on the account using select_for_update. This will lock the account row in the database and make sure no one can update the account instance until the transaction is completed (either committed or rolled-back).
Perform validation checks and raise proper exceptions - raising an exception will cause the transaction to rollback.
If all the validations passed update the state (current balance), set the modification time, save the instance and create the log (action).

So how does the model hold up to our challenges?

Multiple Platforms and Validation - We encapsulated all of our business logic including input and system wide validation inside the model method so consumers such as API, admin action or forms only need to handle exceptions and serialization / UI.
Atomicity - Each method obtains its own lock so there is no risk of race condition.
Logging / History - We created an action model and made sure each function registers the proper action.

Profit!

Testing

Our app will be incomplete without proper tests. I previously wrote about class based testings - we are going to take a slightly different approach but still have a base class with utility functions:

# tests/common.py

class TestAccountBase:
    DEFAULT = object()

    @classmethod
    def default(value, default_value):
        return default_value if value is cls.DEFAULT else value

    @classmethod
    def setUpTestData(cls):
        super().setUpTestData()

        # Set up some default values
        cls.admin = User.objects.create_superuser(
            'Admin',
            'admin',
            'admin@testing.test',
        )

        cls.user_A = User.objects.create_user(
            'user_A',
            'user_A',
            'A@testing.test',
        )

    @classmethod
    def create(
        cls,
        user=DEFAULT,
        created_by=DEFAULT,
        asof=DEFAULT
    ):
        user = cls.default(user, cls.user_A)
        created_by = cls.default(created_by, cls.admin)
        asof = cls.default(asof, timezone.now())

        account, action = Account.create(user, created_by, asof)
        return cls.account, action

    def deposit(
        self,
        amount,
        account=DEFAULT,
        deposited_by=DEFAULT,
        asof=DEFAULT,
        comment=DEFAULT,
    ):
        account = self.default(account, self.account)
        deposited_by = self.default(deposited_by, self.admin)
        asof = self.default(asof, timezone.now())
        comment = self.default(comment, 'deposit comment')

        self.account, action = Account.deposit(
            uid=account.uid,
            deposited_by=deposited_by,
            amount=amount,
            asof=asof,
        )

        self.assertEqual(action.type, Action.ACTION_TYPE_DEPOSITED)
        self.assertIsNotNone(action.user_friendly_id)
        self.assertEqual(action.created, asof)
        self.assertEqual(action.delta, amount)
        self.assertEqual(action.user, deposited_by)

        return action

    def withdraw(
        self,
        amount,
        account=DEFAULT,
        withdrawn_by=DEFAULT,
        asof=DEFAULT,
        comment=DEFAULT,
    ):
        account = self.default(account, self.account)
        withdrawn_by = self.default(withdrawn_by, self.admin)
        asof = self.default(asof, timezone.now())
        comment = self.default(comment, 'withdraw comment')

        self.account, action = Account.withdraw(
            uid=account.uid,
            withdrawn_by=withdrawn_by,
            amount=amount,
            asof=asof,
        )

        self.assertEqual(action.type, Action.ACTION_TYPE_WITHDRAWN)
        self.assertIsNotNone(action.user_friendly_id)
        self.assertEqual(action.created, asof)
        self.assertEqual(action.delta, amount)
        self.assertEqual(action.user, withdrawn_by)

        return action

To make testing easier to write we use utility functions to reduce the boilerplate of specifying the user, the account etc each time by providing default values and operating on self.account.

Lets use our base class to write some tests:

# tests/test_account.py

from unittest import mock
from django.test import TestCase

from .common import TestAccoutBase
from ..models import Account, Action
from ..errors import (
    InvalidAmount,
    ExceedsLimit,
    InsuficientFunds,
)

class TestAccount(TestAccountBase, TestCase):

    def setUp(self):
 **self.account, _ = cls.create()**

    def test_should_start_with_zero_balance(self):
        self.assertEqual(self.account.balance, 0)

    def test_should_deposit(self):
        self.deposit(100)
        self.assertEqual(self.account.balance, 100)
        self.deposit(150)
        self.assertEqual(self.account.balance, 250)

    def test_should_fail_to_deposit_less_than_minimum(self):
        with self.assertRaises(InvalidAmount):
            self.deposit(Account.MIN_DEPOSIT - 1)
        self.assertEqual(self.account.balance, 0)

    def test_should_fail_to_deposit_more_than_maximum(self):
        with self.assertRaises(InvalidAmount):
            self.deposit(Account.MAX_DEPOSIT + 1)
        self.assertEqual(self.account.balance, 0)

    @mock.patch('account.models.Account.MAX_BALANCE', 500)
    @mock.patch('account.models.Account.MAX_DEPOSIT', 502)
    def test_should_fail_to_deposit_more_than_max_balance(self):
        with self.assertRaises(ExceedsLimit):
            self.deposit(501)
        self.assertEqual(self.account.balance, 0)

    @mock.patch('account.models.Account.MAX_BALANCE', 500)
    @mock.patch('account.models.Account.MAX_DEPOSIT', 500)
    @mock.patch('account.models.Account.MAX_TOTAL_BALANCES', 600)
    def test_should_fail_when_exceed_max_total_balances(self):

        # Exceed max total balances for the same account

        self.deposit(500)
        with self.assertRaises(ExceedsLimit):
            self.deposit(500)
        self.assertEqual(self.account.balance, 500)

        # Exceed max total balances in other account

        other_user = User.objects.create_user('foo', 'bar', 'baz')
        other_account = self.create(user=other_user)

        with self.assertRaises(ExceedsLimit):
            self.deposit(200, account=other_account)
        self.assertEqual(other_account.balance, 0)

    def test_should_withdraw(self):
        self.deposit(100)
        self.withdraw(50)
        self.assertEqual(self.account.balance, 50)
        self.withdraw(30)
        self.assertEqual(self.account.balance, 20)

    def test_should_fail_when_insufficient_funds(self):
        self.deposit(100)
        with self.assertRaises(InsufficientFunds):
            self.withdraw(101)
        self.assertEqual(self.account.balance, 100)

Final Words

The classmethod approach has proved itself in our development for quite some time now. We found that it provides the necessary flexibility, readability and testability with very little overhead.

In this article we presented two common issues we encounter frequently - validation and concurrency. This method can be extended to handle access control (permissions) and caching (we have total control over the fetch, remember?), performance optimization (use select_related and update_fields...), audit and monitoring and additional business logic.

We usually support several interfaces for each model - admin interface for support, API for mobile and SPA clients, and a dashboard. Encapsulating the business logic inside the model reduced the amount of code duplication and required tests which leads to overall quality code that is easy to maintain.

In a follow up post I (might) present the admin interface for this model with some neat tricks (such as custom actions, intermediate pages etc) and possibly an RPC implementation using DRF to interact with the account as an API.

Keeping Tests DRY with Class Based Tests In Python

2016-08-20T00:00:00+03:00

Tests can be a bummer to write but even a bigger nightmare to maintain. When we noticed we are putting off simple tasks just because we were afraid to update some monster test case, we started looking for more creative ways to simplify the process of writing and maintaining tests.

In this article I will describe a class based approach to writing tests.

Before we start writing code let's set some goals:

Extensive - We want our tests to cover as many scenarios as possible. We hope a solid platform for writing tests will make it easier for us to adapt to changes and cover more grounds.
Expressive - Good tests tell a story. Issues become irrelevant and documents get lost but tests must always pass - this is why we treat our tests as specs. Writing good tests can help newcomers (and future self) to understand all the edge cases and micro-decisions made during development.
Maintainable - As requirements and implementations change we want to adapt quickly with as little effort as possible.

Enter Class Based Tests

Articles and tutorials about testing always give simple examples such as add and sub. I rarely have the pleasure of testing such simple functions. I'll take a more realistic example and test an API endpoint that does login:

POST /api/account/login
{
    username: <str>,
    password: <str>
}

The scenarios we want to test are:

User logins successfully.
User does not exist.
Incorrect password.
Missing or malformed data.
User already authenticated.

The input to our test is:

A payload, username and password.
The client performing the action, anonymous or authenticated.

The output we want to test is:

The return value, error or payload.
The response status code.
Side effects. For example, last login date after successful login.

After properly defining the input and output, we can write a base test class:

from unittest import TestCase
import requests

class TestLogin:
    """Base class for testing login endpoint."""

    @property
    def client(self):
        return requests.Session()

    @property
    def username(self):
        raise NotImplementedError()

    @property
    def password(self):
        raise NotImplementedError()

    @property
    def payload(self):
        return {
            'username': self.username,
            'password': self.password,
        }

    expected_status_code = 200
    expected_return_payload = {}

    def setUp(self):
        self.response = self.client.post('/api/account/login', json=payload)

    def test_should_return_expected_status_code(self):
        self.assertEqual(self.response.status, self.expected_status_code)

    def test_should_return_expected_payload(self):
        self.assertEqual(self.response.json(), self.expected_return_payload)

We defined the input, client and payload, and the expected output expected_*.
We performed the login action during test setUp. To let specific test cases access the result, we kept the response on the class instance.
We implemented two common test cases:
- Test the expected status code.
- Test the expected return value.

The observant reader might notice we raise a NotImplementedError exception from the properties. This way, if the test author forgets to set one of the required values for the test, they get a useful exception.

Lets use our TestLogin class to write a test for a successful login:

class TestSuccessfulLogin(TestLogin, TestCase):
    username = 'Haki',
    password = 'correct-password'
    expected_status_code = 200
    expected_return_payload = {
        'id': 1,
        'username': 'Haki',
        'full_name': 'Haki Benita',
    }

    def test_should_update_last_login_date_in_user_model(self):
        user = User.objects.get(self.response.data['id'])
        self.assertIsNotNone(user.last_login_date)

By just reading the code we can tell that a username and password are sent. We expect a response with a 200 status code, and additional data about the user. We extended the test to also check the last_login_date in our user model. This specific test might not be relevant to all test cases, so we add it only to the successful test case.

Lets test a failed login scenario:

class TestInvalidPassword(TestLogin, TestCase):
    username = 'Haki'
    password = 'wrong-password'
    expected_status_code = 401

class TestMissingPassword(TestLogin, TestCase):
    payload = {'username': 'Haki'}
    expected_status_code = 400

class TestMalformedData(TestLogin, TestCase):
    payload = {'username': [1, 2, 3]}
    expected_status_code = 400

A developer that stumbles upon this piece of code will be able to tell exactly what should happen for any type of input. The name of the class describe the scenario, and the names of the attributes describe the input. Together, the class tells a story which is easy to read and understand.

The last two tests set the payload directly (without setting username and password). This won't raise a NotImplementedError because we override the payload property directly, which is the one calling username and password.

A good test should help you find where the problem is.

Let's see the output of a failed test case:

FAIL: test_should_return_expected_status_code (tests.test_login.TestInvalidPassword)
------------------------------------------------------
Traceback (most recent call last):
  File "../tests/test_login.py", line 28, in test_should_return_expected_status_code
    self.assertEqual(self.response.status_code, self.expected_status_code)
AssertionError: 400 != 401
------------------------------------------------------

Looking at the failed test report, it is clear what went wrong. When the password is invalid we expect status code 401, but we received 400.

Let's make things a bit harder, and test an authenticated user attempting to login:

class TestAuthenticatedUserLogin(TestLogin, TestCase):
    username = 'Haki'
    password = 'correct-password'

    @property
    def client(self):
        session = requests.session()
        session.auth = ('Haki', 'correct-password')
        return session

    expected_status_code = 400

This time we had to override the client property to authenticate the session.

Putting Our Test To The Test

To illustrate how resilient our new test cases are lets see how we can modify the base class as we introduce new requirements and changes:

We have made some refactoring and the endpoint changed to /api/user/login:

class TestLogin:
    # ...
    def setUp(self):
        self.response = self.client.post(
            '/api/user/login',
            json=payload,
        )

Someone decided it can speed things up if we use a different serialization format (msgpack, xml, yaml):

class TestLogin:
    # ...
    def setUp(self):
        self.response = self.client.post(
            '/api/account/login',
            data=encode(payload),
        )

The product guys want to go global, and now we need to test different languages:

class TestLogin:
    language = 'en'

    # ...

    def setUp(self):
        self.response = self.client.post(
            '/{}/api/account/login'.format(self.language),
            json=payload,
        )

None of the changes above managed to break our existing tests.

Taking it a Step Further

A few things to consider when employing this technique.

Speed Things Up

setUp is executed for each test case in the class (test cases are the functions beginning with test_*). To speed things up, it is better to perform the action in setUpClass. This changes a few things. For example, the properties we used should be set as attributes on the class or as @classmethods.

Using Fixtures

When using Django with fixtures, the action should go in setUpTestData:

class TestLogin:
    fixtures = (
        'test/users',
    )

    @classmethod
    def setUpTestData(cls):
        super().setUpTestData()
        cls.response = cls.get_client().post('/api/account/login', json=payload)

Django loads fixtures at setUpTestData so by calling super the action is executed after the fixtures were loaded.

Another quick note about Django and requests. I've used the requests package but Django, and the popular Django restframework, provide their own clients. django.test.Client in Django's client, and rest_framework.test.APIClient is DRF's client.

Testing Exceptions

When a function raise an exception, we can extend the base class and wrap the action with try ... catch:

class TestLoginFailure(TestLogin):

    @property
    def expected_exception(self):
        raise NotImplementedError()

    def setUp(self):
        try:
            super().setUp()
        except Exception as e:
            self.exception = e

    def test_should_raise_expected_exception(self):
        self.assertIsInstance(
            self.exception,
            self.expected_exception
        )

If you are familiar with the assertRaises context, I haven't used it in this case because the test should not fail during setUp.

Create Mixins

Test cases are repetitive by nature. With mixins, we can abstract common parts of tests cases and compose new ones. For example:

TestAnonymousUserMixin - populates the test with anonymous API client.
TestRemoteResponseMixin - mock response from remote service.

The later, might look something like this:

from unittest import mock

class TestRemoteServiceXResponseMixin:
    mock_response_data = None

    @classmethod
    @mock.patch('path.to.function.making.remote.request')
    def setUpTestData(cls, mock_remote)
        mock_remote.return_value = cls.mock_response_data
        super().setUpTestData()

Conclusion

Someone once said that duplication is cheaper than the wrong abstraction. I couldn't agree more. If your tests do not fit easily into a pattern then this solution is probably not the right one. It's important to carefully decide what to abstract. The more you abstract, the more flexible your tests are. But, as parameters pile up in base classes, tests are becoming harder to write and maintain, and we go back to square one.

Having said that, we found this technique to be useful in various situations and with different frameworks (such as Tornado and Django). Over time it has proven itself as being resilient to changes and easy to maintain. This is what we set out to achieve and we consider it a success!

Things You Must Know About Django Admin As Your App Gets Bigger

2016-08-05T00:00:00+03:00

The Django admin is a very powerful tool. We use it for day to day operations,browsing data and support. As we grew some of our projects from zero to 100K+users we started experiencing some of Django's admin pain points - long response times and heavy load on the database.

In this short article I am going to share some simple techniques we use in our projects to make the Django admin behave as apps grow in size and complexity.

We use Django 1.8, Python 3.4 and PostgreSQL 9.4. The code samples are for Python 3.4 but they can be easily modified to work on 2.7 and other Django versions.

Before We Start

These are the main components in a Django Admin list view:

Example admin list view including some of the components discussed in the article

Logging

Most of Django's work is performing SQL queries so our main focus will be on minimizing the amount of queries. To keep track of query execution you can use one of the following:

django-debug-toolbar - Very nice utility that adds a little panel on the side of the screen with a list of SQL queries executed and other useful metrics.
If you don't like dependencies (like us) you can log SQL queries to the console by adding the following logger in settings.py:

LOGGING = {
    # ...
    'loggers': {
        'django.db.backends': {
            'level': 'DEBUG',
        },
    },
    # ...
}

The N+1 Problem

The N+1 problem is a well known problem in ORMs. To illustrate the problem let's say we have this schema:

class Category(models.Model):
    name = models.CharField(max_length=50)

    def__str__(self):
        return self.name

class Product(models.Model):
    name = models.CharField(max_length=50)
    category = models.ForeignKey(Category)

By implementing __str__ we tell Django that we want the name of the category to be used as the default description of the object. Whenever we print a category object, Django will fetch the name of the category.

A simple admin page for our Product model might look like this:

@admin.register(models.Product)
class ProductAdmin(admin.ModelAdmin):
    list_display = (
        'id',
        'name',
        'category',
    )

This seems innocent enough but the SQL log reveals the horror:

(0.000) SELECT COUNT(*) AS "__count" FROM "app_product"; args=()
(0.002) SELECT "app_product"."id", "app_product"."name", "app_product"."category_id"
        FROM "app_product" ORDER BY "app_product"."id" DESC LIMIT 100; args=()
(0.000) SELECT ... FROM "app_category" where "app_category"."id" = 1; args=(1)
(0.000) SELECT ... FROM "app_category" where "app_category"."id" = 2; args=(2)
(0.000) SELECT ... FROM "app_category" where "app_category"."id" = 1; args=(1)
(0.000) SELECT ... FROM "app_category" where "app_category"."id" = 4; args=(4)
(0.000) SELECT ... FROM "app_category" where "app_category"."id" = 3; args=(3)
...
(0.000) SELECT ... FROM "app_category" where "app_category"."id" = 2; args=(2)
(0.000) SELECT ... FROM "app_category" where "app_category"."id" = 99; args=(99)
(0.000) SELECT ... FROM "app_category" where "app_category"."id" = 104; args=(104)

Django first counts the objects (more on that later), then fetches the actual objects (limiting to the default page size of 100) and then passes the data on to the template for rendering. We used the category name as the description of the Category object, so for each product Django has to fetch the category name. This results in 100 additional queries.

To tell Django we want to perform a join instead of fetching the names of the categories one by one, we can use list_select_related:

@admin.register(models.Product)
class ProductAdmin(admin.ModelAdmin):
    list_display = (
        'id',
        'name',
        'category',
    )
    list_select_related = (
        'category',
    )

Now, the SQL log looks much nicer. Instead of 101 queries we have only 1:

(0.004) SELECT "app_product"."id", "app_product"."name",
"app_product"."category_id", "app_category"."id", "app_category"."name"
FROM "app_product"
INNER JOIN "app_category" on ("app_product"."category_id" = "app_category"."id")
ORDER BY "app_product"."id" DESC LIMIT 100; args=()

To understand the real impact of this setting consider the following. Django default page size is 100 objects. If you have one related fields you have ~101 queries. If you have two related objects displayed in the list view, you have ~201 queries and so on.

Fetching related fields in a join can only work for ForeignKey relations. If you wish to display ManyToMany relations it's a bit more complicated (and most of the time wrong, but keep reading).

Sometimes it can be useful to quickly navigate between objects. After trying for a while to teach support personnel to filter using URL parameters, we finally gave up and created two simple decorators.

`admin_link`

Create a link to a detail page of a related model:

def admin_change_url(obj):
    app_label = obj._meta.app_label
    model_name = obj._meta.model.__name__.lower()
    return reverse('admin:{}_{}_change'.format(
        app_label, model_name
    ), args=(obj.pk,))

def admin_link(attr, short_description, empty_description="-"):
    """Decorator used for rendering a link to a related model in
    the admin detail page.

    attr (str):
        Name of the related field.
    short_description (str):
        Name if the field.
    empty_description (str):
        Value to display if the related field is None.

    The wrapped method receives the related object and should
    return the link text.

    Usage:
        @admin_link('credit_card', _('Credit Card'))
        def credit_card_link(self, credit_card):
            return credit_card.name
    """
    def wrap(func):
        def field_func(self, obj):
            related_obj = getattr(obj, attr)
            if related_obj is None:
                return empty_description
            url = admin_change_url(related_obj)
            return format_html('<a href="{}">{}</a>', url, func(self, related_obj))
        field_func.short_description = short_description
        field_func.allow_tags = True
        return field_func
    return wrap

The decorator will render a link (<a href="...">...</a>) to the related model in both the list view and the detail view. If for example, we want to add a link from each product to its category detail page, we use the decorator like this:

@admin.register(models.Product)
class ProductAdmin(admin.ModelAdmin):
    list_display = (
        'id',
        'name',
        'category_link',
    )
    admin_select_related = (
        'category',
    )

    @admin_link('category', _('Category'))
    def category_link(self, category):
        return category

`admin_changelist_link`

More complicated links such as "all the products of a category" require a different implementation. We created a decorator that accepts a query string, and link to the list view of a related model:

def admin_changelist_url(model):
    app_label = model._meta.app_label
    model_name = model.__name__.lower()
    return reverse('admin:{}_{}_changelist'.format(app_label, model_name))

def admin_changelist_link(
    attr,
    short_description,
    empty_description='-',
    query_string=None
):
    """Decorator used for rendering a link to the list display of
    a related model in the admin detail page.

    attr (str):
        Name of the related field.
    short_description (str):
        Field display name.
    empty_description (str):
        Value to display if the related field is None.
    query_string (function):
        Optional callback for adding a query string to the link.
        Receives the object and should return a query string.

    The wrapped method receives the related object and
    should return the link text.

    Usage:

        @admin_changelist_link('credit_card', _('Credit Card'))
        def credit_card_link(self, credit_card):
            return credit_card.name
    """
    def wrap(func):
        def field_func(self, obj):
            related_obj = getattr(obj, attr)
            if related_obj is None:
                return empty_description
            url = admin_changelist_url(related_obj.model)
            if query_string:
                url += '?' + query_string(obj)
            return format_html('<a href="{}">{}</a>', url, func(self, related_obj))
        field_func.short_description = short_description
        field_func.allow_tags = True
        return field_func
    return wrap

To add a link from a category to of its products, we do the following in CategoryAdmin:

@admin.register(models.Category)
class CategoryAdmin(admin.ModelAdmin):
    list_display = (
        'id',
        'name',
        'products_link',
    )

    @admin_changelist_link('products', _('Products'),
                            query_string=lambda c: 'category_id={}'.format(c.pk))
    def products_link(self, products):
        return _('Products')

Be careful with the products argument. It is very tempting to do something like this:

# Bad example

@admin_changelist_link('products', _('Products'),
                        query_string=lambda c: 'category_id={}'.format(c.pk))
def products_link(self, products):
    # Dont do that!
    return 'see {} products'.format(products.count())

The example above will result in additional queries.

`readonly_fields`

In the detail page, Django creates an editable element for each field. Text and numeric fields will be rendered as regular input field. Choice fields and foreign key fields will be rendered as a <select> element. To render a select box Django has to do the following:

Fetch the options - the entire related model and their descriptions (remember the N+1 problem?).
Render the option list - one option for each related model instance.

A common scenario that is often overlooked, is foreign key to the User model. When you have 100 users you might not notice the load, but what happens when you suddenly have 100K users? The detail page will fetch the entire users table, and the option list will make the resulting HTML huge. We pay twice, first for the full table scan, and then for downloading the html file. Not to mention the memory required to generate the html file in the first place.

Having a select element with 100K options is not really usable. The easiest way to prevent Django from rendering a field as a <select> element is to mark it as readonly_fields:

@admin.register(SomeModel)
def SomeModelAdmin(admin.ModelAdmin):
    readonly_fields = (
        'user',
    )

This will render the description of the related model, without being able to change it in the admin.

Another option to prevent Django from rendering a select box, is to mark the field as raw_id fields.

@admin.register(SomeModel)
def SomeModelAdmin(admin.ModelAdmin):
    raw_id_fields = (
        'user',
    )

Using raw_id_fields, Django will render a special widget that shows the id of the value, and an option to open a list of all values in a popup window. This option is very useful when you want to edit a foreign key value.

Filters

We often use the admin interface as a day to day tool for general support. We found that most of the times we use the same filters: only active users, users registered in the last month, successful transactions ans so on. Once we realized that, we asked ourselves, why fetch the entire dataset if we are most likely to immediately apply a filter to it?. We started to look for a way to apply a default filter when entering the model list view.

`DefaultFilterMixin`

There are many approaches to apply default filters. Some approaches involve custom filters or injecting special query parameters to the request. We wanted to avoid those.

We found that the following approach to be simple and straightforward:

from urllib.parse import urlencode
from django.shortcuts import redirect

class DefaultFilterMixin:
    def get_default_filters(self, request):
        """Set default filters to the page.

        request (Request)

        Returns (dict):
            Default filter to encode.
        """
        raise NotImplementedError()


def changelist_view(self, request, extra_context=None):
    ref = request.META.get('HTTP_REFERER', '')
    path = request.META.get('PATH_INFO', '')

    # If already have query parameters or if the page
    # was referred from it self (by drilldown or redirect)
    # don't apply default filter.
    if request.GET or ref.endswith(path):
        return super().changelist_view(request, extra_context=extra_context)

    query = urlencode(self.get_default_filters(request))
    return redirect('{}?{}'.format(path, query))

If the list view was accessed from a different view, and no query params were specified, we generate a default query and redirect.

Let's apply a default filter to our product page to show only products created in the last month:

from django.utils import timezone

@admin.register(models.Product)
class ProductAdmin(DefaultFilterMixin, admin.ModelAdmin):
    date_hierarchy = 'created'

    def get_default_filters(self, request):
        now = timezone.now()
        return {
            'created__year': now.year,
            'created__month': now.month,
        }

If we drill down from within the page, or if we get to the page with query parameters, the default filter will not be applied.

Quick Bits

Some neat tricks we gathered over time.

`show_full_result_count`

Prevent Django from showing the total number of rows in the list view. Setting show_full_result_count=False saves a count(*) query on the queryset on every page load.

`defer`

When performing a query the entire resultset is put into memory for processing. If you have large columns in your model such as JSON or Text fields, it might be a good idea to defer them until you really need to use them. To defer fields override get_queryset.

Change the Admin Default URL Route

This is definitely not the only precaution you should take to protect your admin page, but it can make it harder for "curious" users to reach the login page.

In your main urls.py override the default admin route:

# urls.py

from django.conf.urls import include, url
from django.contrib import admin

urlpatterns = [
    url(r'^foo/', include(admin.site.urls)),
]

see also

I wrote a bunch of tips on how to make Django admin safer.

`date_hierarchy`

We found that this index can be used to improve queries generate with date hierarchy predicate in PostgresSQL 9.4:

CREATE INDEX yourmodel_date_hierarchy_ix ON yourmodel_table (
    extract('day' from created at time zone 'America/New_York'),
    extract('month' from created at time zone 'America/New_York'),
    extract('year' from created at time zone 'America/New_York')
);

Make sure to change table name, index name, the date hierarchy column and the time zone.

Conclusion

Even if you don't have 100K users and millions of records in the database, it is still important to keep the admin tidy. Bad code has this nasty tendency of biting you in the ass when you least expect it.