Django in Production: Lessons from 4 Years at Lobstr.io

Django is easy to start with. startproject, define some models, write views, ship it. But the Django that handles 100 requests per day and the Django that handles 100 requests per second look very different.

After four years of running Django in production at Lobstr.io, here are the patterns that survived - and the mistakes I made learning them.

Query Optimization: The N+1 Killer

The single most common Django performance issue. You write code that looks clean:

# This generates N+1 queries
users = User.objects.all()
for user in users:
    print(user.profile.company)  # Each .profile hits the database

With 1,000 users, that's 1,001 database queries. The fix is always the same - tell Django to fetch related data upfront:

# select_related: for ForeignKey / OneToOne (SQL JOIN)
users = User.objects.select_related('profile').all()

# prefetch_related: for ManyToMany / reverse ForeignKey (separate query)
users = User.objects.prefetch_related('orders').all()

In practice, I add select_related and prefetch_related to almost every queryset that touches related models. The rule is simple: if you access a related field in a loop, you need one of these.

Spotting N+1 in Production

Django Debug Toolbar is great for development, but in production, I use query logging:

# settings.py - enable for debugging, disable in production
LOGGING = {
    'version': 1,
    'handlers': {
        'console': {'class': 'logging.StreamHandler'},
    },
    'loggers': {
        'django.db.backends': {
            'level': 'DEBUG',
            'handlers': ['console'],
        },
    },
}

If you see the same query repeated 50 times in the logs, you've found your N+1.

Connection Pooling: Don't Hit the Database Cold

By default, Django opens a new database connection for every request and closes it when the request ends. At scale, the overhead of establishing connections (especially with SSL) adds up.

# settings.py
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'mydb',
        # Keep connections alive for 10 minutes instead of closing after each request
        'CONN_MAX_AGE': 600,
        'CONN_HEALTH_CHECKS': True,  # Django 4.1+ - verify connection before reuse
    }
}

CONN_MAX_AGE=600 keeps connections alive for 10 minutes. Combined with CONN_HEALTH_CHECKS, Django will reuse healthy connections and replace dead ones automatically.

For high-traffic apps, add django-db-connection-pool or PgBouncer for proper connection pooling beyond what Django's built-in CONN_MAX_AGE provides.

Signals vs. Celery: When to Use Which

Django signals (post_save, pre_delete, etc.) are tempting for side effects - send an email when a user signs up, update a cache when a model changes. But they become a maintenance nightmare:

# Seems clean, but...
@receiver(post_save, sender=Order)
def order_created(sender, instance, created, **kwargs):
    if created:
        send_confirmation_email(instance)  # Blocks the request
        update_inventory(instance)          # Blocks the request
        notify_warehouse(instance)          # Blocks the request

Problems:

1. Signals are invisible - code that triggers Order.save() has no idea these side effects run
2. They block the request - three slow operations happen synchronously
3. They're hard to test - you can't easily test Order.save() without also testing email, inventory, and notification logic

My rule: signals for cache invalidation, Celery for everything else.

# Better: explicit Celery tasks
class OrderCreateView(CreateView):
    def form_valid(self, form):
        order = form.save()
        # Explicit, visible, non-blocking
        send_confirmation_email.delay(order.id)
        update_inventory.delay(order.id)
        notify_warehouse.delay(order.id)
        return redirect('order-detail', pk=order.id)

With Celery, the side effects are visible in the view, don't block the response, and can be retried independently if they fail.

Caching: The Three Layers

I use three levels of caching depending on the data:

1. Per-View Cache (Redis)

For expensive queries that many users see:

from django.views.decorators.cache import cache_page

@cache_page(60 * 15)  # 15 minutes
def dashboard(request):
    # Expensive aggregation queries
    stats = Order.objects.aggregate(
        total=Sum('amount'),
        count=Count('id'),
    )
    return render(request, 'dashboard.html', stats)

2. Low-Level Cache (Specific Objects)

For data that changes per-user but is expensive to compute:

from django.core.cache import cache

def get_user_stats(user_id):
    cache_key = f'user_stats_{user_id}'
    stats = cache.get(cache_key)
    if stats is None:
        stats = compute_expensive_stats(user_id)
        cache.set(cache_key, stats, timeout=300)  # 5 minutes
    return stats

3. Database Query Cache (Queryset Caching)

For querysets that are hit repeatedly within a single request:

# Bad: hits the database twice
user_count = User.objects.count()
active_count = User.objects.filter(is_active=True).count()

# Better: evaluate once, filter in Python (if dataset is small)
users = list(User.objects.all())
user_count = len(users)
active_count = sum(1 for u in users if u.is_active)

Cache Invalidation

The hardest part. I use a signal (one of the few valid uses) to invalidate when models change:

@receiver(post_save, sender=Order)
@receiver(post_delete, sender=Order)
def invalidate_order_cache(sender, instance, **kwargs):
    cache.delete(f'user_stats_{instance.user_id}')
    cache.delete('dashboard_stats')

Bulk Operations

Never create/update objects in a loop:

# Bad: 1000 INSERT queries
for item in data:
    Product.objects.create(name=item['name'], price=item['price'])

# Good: 1 INSERT query
products = [Product(name=item['name'], price=item['price']) for item in data]
Product.objects.bulk_create(products, batch_size=500)

# For updates:
products = Product.objects.filter(category='electronics')
products.update(on_sale=True)  # 1 UPDATE query

# Or bulk_update for different values per object:
for product in products:
    product.price = calculate_new_price(product)
Product.objects.bulk_update(products, ['price'], batch_size=500)

bulk_create and bulk_update reduce thousands of queries to a handful. The batch_size parameter keeps memory and query size manageable.

Index Your Filters

Every filter(), exclude(), order_by(), and get() should have a corresponding database index. Django creates indexes for primary keys and unique fields automatically, but you need to add them for fields you filter on:

class Order(models.Model):
    user = models.ForeignKey(User, on_delete=models.CASCADE)
    status = models.CharField(max_length=20, db_index=True)
    created_at = models.DateTimeField(db_index=True)
    amount = models.DecimalField(max_digits=10, decimal_places=2)

    class Meta:
        indexes = [
            # Composite index for common query patterns
            models.Index(fields=['user', 'status']),
            models.Index(fields=['status', '-created_at']),
        ]

The composite indexes match your most common query patterns. If you frequently query Order.objects.filter(user=user, status='pending'), the ['user', 'status'] index makes that query use an index scan instead of a full table scan.

Middleware: Keep It Light

Every middleware runs on every request. I've seen middleware that does database queries, API calls, or heavy computation - and tanks the entire site's performance.

Rules:

• No database queries in middleware (use lazy evaluation if you must)
• No external API calls in middleware
• If you need per-request data, compute it lazily on first access
• Measure middleware execution time - it adds up

Structure for Scale

After four years, my Django project structure looks like this:

project/
├── config/                 # Settings, URLs, WSGI/ASGI
│   ├── settings/
│   │   ├── base.py
│   │   ├── development.py
│   │   └── production.py
│   ├── urls.py
│   └── celery.py
├── apps/
│   ├── users/
│   ├── orders/
│   └── notifications/
├── core/                   # Shared utilities
│   ├── mixins.py
│   ├── permissions.py
│   └── pagination.py
└── manage.py

Each app is self-contained. Shared logic lives in core/. Settings are split by environment. Celery config is next to the Django config.

Key Takeaways

• select_related and prefetch_related solve 90% of query performance issues
• Use CONN_MAX_AGE to avoid connection overhead per request
• Signals for cache invalidation, Celery for everything else
• Three-layer caching: per-view, low-level, and queryset
• Bulk operations for any create/update in a loop
• Index every field you filter on, including composite indexes
• Keep middleware lightweight - it runs on every single request

Django scales further than most people think. The framework isn't the bottleneck - it's always the queries.

These patterns are from production experience at Lobstr.io. Your mileage may vary depending on your specific workload and database.