Decoding Protobuf in the Wild: Reverse Engineering Binary API Responses

You're inspecting network traffic from a mobile app. The response body isn't JSON - it's a blob of binary data. No documentation, no .proto files, no schema. Welcome to Protobuf in the wild.

Protocol Buffers (Protobuf) is Google's binary serialization format, and it's everywhere - gRPC services, mobile apps, internal APIs. When you're reverse engineering an app that uses it, you need to decode it without the schema. Here's how.

Recognizing Protobuf

Before you can decode it, you need to identify it. Protobuf responses have a few tells:

• Content-Type header is often application/x-protobuf, application/grpc, or application/octet-stream
• The response body is binary (not human-readable)
• The first few bytes follow the wire format pattern - field numbers and wire types packed into varints

If you see a binary response from a Google, Facebook, or any modern mobile app backend, there's a good chance it's Protobuf.

The Wire Format

Protobuf encodes data as a series of key-value pairs. Each key is a varint that encodes both the field number and the wire type:

key = (field_number << 3) | wire_type

Wire types tell you how to read the value:

This is the foundation. Without a .proto schema, you won't know field names - but you can decode the structure.

Decoding Without a Schema

The protoc tool can decode raw Protobuf using --decode_raw:

cat response.bin | protoc --decode_raw

This gives you something like:

1: 12345
2: "John Doe"
3 {
  1: "[email protected]"
  2: 1
}
4: 1708300800

You can see the structure: field 1 is an integer (probably an ID), field 2 is a string (name), field 3 is a nested message (contact info), and field 4 looks like a Unix timestamp.

Python Tooling

For programmatic decoding, I use the blackboxprotobuf library:

import blackboxprotobuf
import requests

response = requests.get("https://api.example.com/data",
    headers={"Accept": "application/x-protobuf"})

# Decode without a schema
message, typedef = blackboxprotobuf.decode_message(response.content)
print(message)
# {1: 12345, 2: b'John Doe', 3: {1: b'[email protected]', 2: 1}}

# The typedef tells you the inferred types
print(typedef)
# {'1': {'type': 'int'}, '2': {'type': 'bytes'}, ...}

The library infers types from the wire format. It can't tell you that field 2 is a "username" - but it can tell you it's a string, and from context, you can figure the rest out.

Reconstructing the Schema

Once you've decoded enough responses, you can reconstruct a working .proto file:

syntax = "proto3";

message UserResponse {
  int64 user_id = 1;
  string name = 2;
  ContactInfo contact = 3;
  int64 created_at = 4;
}

message ContactInfo {
  string email = 1;
  bool verified = 2;
}

With this schema, you can use protoc to generate Python classes and decode/encode messages properly:

from generated_pb2 import UserResponse

user = UserResponse()
user.ParseFromString(response.content)
print(user.name)        # "John Doe"
print(user.contact.email)  # "[email protected]"

gRPC: Protobuf Over HTTP/2

Many modern APIs use gRPC, which is Protobuf over HTTP/2 with a specific framing format. gRPC responses have a 5-byte header before the Protobuf payload:

def decode_grpc_response(data):
    # Byte 0: compression flag (0 = none)
    compressed = data[0]
    # Bytes 1-4: message length (big-endian)
    length = int.from_bytes(data[1:5], 'big')
    # Bytes 5+: the actual Protobuf message
    protobuf_data = data[5:5 + length]
    return protobuf_data

Strip those 5 bytes and you're back to standard Protobuf decoding.

The Workflow

My process for any Protobuf-based target:

1. Capture traffic - Use mitmproxy with a mobile device or emulator to intercept requests
2. Identify Protobuf endpoints - Look for binary responses and x-protobuf content types
3. Raw decode - Use protoc --decode_raw or blackboxprotobuf to see the structure
4. Map fields to meaning - Cross-reference decoded values with what you see in the UI
5. Reconstruct the schema - Build a .proto file that matches the wire format
6. Generate code - Use protoc to generate Python classes for clean encode/decode

Common Pitfalls

• Nested messages vs. bytes - Wire type 2 is used for both strings and embedded messages. blackboxprotobuf sometimes guesses wrong - if a "bytes" field looks structured, try decoding it as a message
• Packed repeated fields - Arrays of numbers can be packed into a single length-delimited field. If you see an unexpectedly long bytes field, try decoding it as packed varints
• Signed integers - Protobuf uses ZigZag encoding for signed integers (sint32/sint64). If decoded values look wrong (huge numbers for what should be small negatives), you might be reading a sint as a uint
• Enums look like ints - You'll just see 0, 1, 2 - you need the app context to know what they mean

Key Takeaways

• Protobuf is everywhere in modern APIs, especially mobile apps and gRPC services
• You can always decode the structure without a schema - you just won't have field names
• protoc --decode_raw and blackboxprotobuf are your primary tools
• Reconstructing a .proto file from decoded responses is tedious but straightforward
• gRPC adds a 5-byte framing header - strip it before decoding

The binary format is designed for machines, not humans. But with the right tools and a systematic approach, you can read it just fine.

This is an educational overview for security research and authorized testing. Always ensure you have permission before testing any system.