Implementing BSON Format: From Theory to Practice

What is BSON?

BSON, short for Binary JSON, is a binary serialization format for JSON‑like documents. It was originally created for MongoDB to serve as both the on‑disk storage format and the wire protocol for client‑server communication. Unlike plain JSON text, BSON encodes data in a compact binary layout that adds explicit type information, length prefixes, and additional data types not present in standard JSON.

A BSON document consists of a sequence of bytes with a well‑defined structure: it starts with a 4‑byte little‑endian integer that tells you the total size of the document, followed by a series of elements. Each element is composed of a type byte, a cstring key, and the value encoded according to the type. The document ends with a trailing null byte (0x00).

Because BSON is self‑describing and length‑prefixed, parsers can scan through documents very efficiently, skip unknown fields, and extract nested values without parsing the whole structure first.

Why BSON Matters

🚀 Deploy your AI agent in 10 minutes

Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.

Try it free →

Understanding BSON is essential when working with MongoDB or when you need to exchange rich structured data with minimal overhead. Here are the key reasons why it matters:

Efficient scanning – Length prefixes allow parsers to jump over elements or whole documents without decoding them.
Richer type system – BSON adds dates, binary data, ObjectId, 32‑bit and 64‑bit integers, and more, all missing from plain JSON.
Compact wire format – Binary encoding reduces payload size compared to text‑based JSON, which matters for database protocols and high‑throughput messaging.
Schema‑less but typed – Every value carries its own type code, making documents self‑describing and easy to validate on the fly.
Interoperability – Official MongoDB drivers use BSON internally, and many third‑party tools rely on the BSON specification for data import/export.

BSON Specification Overview

The BSON specification defines a set of type codes that govern how each value is encoded. Below are the most commonly used types you’ll implement and encounter:

0x01 – Double (8 bytes IEEE 754)
0x02 – UTF‑8 string (length‑prefixed, null‑terminated)
0x03 – Embedded document (a full BSON document)
0x04 – Array (encoded as a document with keys "0", "1", …)
0x05 – Binary data (length‑prefixed, followed by a subtype byte and raw bytes)
0x07 – ObjectId (12 bytes, typically auto‑generated)
0x08 – Boolean (1 byte: 0x01 for true, 0x00 for false)
0x09 – UTC datetime (8 bytes int64, milliseconds since epoch)
0x0A – Null value (no content)
0x10 – Int32 (4 bytes signed integer)
0x12 – Int64 (8 bytes signed integer)

Every element starts with one of these type bytes, followed by the key encoded as a cstring (UTF‑8 bytes terminated by 0x00), and then the value bytes. Documents and arrays are themselves full BSON documents (with their own 4‑byte length prefix and trailing null). The entire top‑level document ends with a single null byte after all elements.

All multibyte numeric values (int32, int64, double, lengths) are stored in little‑endian byte order.

Implementing a BSON Encoder in Python

We’ll build a minimal BSON encoder step by step. Our goal is to convert a Python dictionary (with a few special types) into a valid BSON byte string. The implementation uses only the standard library (struct, datetime, os, etc.).

Encoding Primitives

The first building block is writing typed values into a byte buffer. We’ll use struct.pack with little‑endian format characters ("<i", "<q", "<d"). For example, an int32 is written as:

import struct

def encode_int32(buf, value):
    buf.extend(struct.pack("<i", value))

Similarly, a boolean is a single byte, and null has no content at all:

def encode_boolean(buf, value):
    buf.append(1 if value else 0)

def encode_null(buf):
    # nothing to append, just the type byte and key handled elsewhere
    pass

Encoding Strings and CStrings

BSON strings are length‑prefixed UTF‑8 sequences that include a trailing null byte. The length field counts the bytes of the string plus the null terminator.

def encode_string(buf, s):
    encoded = s.encode('utf-8') + b'\x00'
    length = len(encoded)   # includes the null terminator
    buf.extend(struct.pack("<i", length))
    buf.extend(encoded)

A key is written as a cstring (UTF‑8 bytes + null) without any length prefix:

def encode_cstring(buf, key):
    buf.extend(key.encode('utf-8') + b'\x00')

Encoding Documents and Arrays

An embedded document is a full BSON document, which means it starts with its own 4‑byte length, contains its elements, and ends with 0x00. Our encoder will recursively call itself for nested dictionaries.

Arrays in BSON are represented as documents with keys "0", "1", … matching the list indices. We’ll convert a Python list into such a dictionary and encode it as a document with type 0x04.

def encode_document(buf, doc):
    # Recursively encode the whole document (see full encoder)
    ...

def encode_array(buf, lst):
    doc = {str(i): v for i, v in enumerate(lst)}
    encode_document(buf, doc)   # wrapped as type 0x04

Putting the Encoder Together

Now we assemble a complete BSONEncoder class. It iterates over key‑value pairs, picks the correct type code, writes the key and value, and finally prepends the total document length.

import struct
import datetime
import os
import random

class ObjectId:
    """Simple ObjectId wrapper holding 12 bytes."""
    def __init__(self, bytes=None):
        if bytes is None:
            # generate 12 random bytes
            self._bytes = os.urandom(12)
        else:
            self._bytes = bytes
    def __bytes__(self):
        return self._bytes

class BSONEncoder:
    def encode(self, document):
        """Encode a dictionary into BSON bytes."""
        buf = bytearray()
        for key, value in document.items():
            self._encode_element(buf, key, value)
        buf.append(0x00)  # document terminator
        total_len = 4 + len(buf)  # length includes the 4-byte length field itself
        final = struct.pack("<i", total_len) + buf
        return bytes(final)

    def _encode_element(self, buf, key, value):
        # Determine type and write the type byte and key cstring
        if isinstance(value, dict):
            buf.append(0x03)   # embedded document
            buf.extend(key.encode('utf-8') + b'\x00')
            sub = self.encode(value)
            buf.extend(sub)    # already contains length prefix and terminator
        elif isinstance(value, list):
            buf.append(0x04)   # array
            buf.extend(key.encode('utf-8') + b'\x00')
            doc = {str(i): v for i, v in enumerate(value)}
            sub = self.encode(doc)
            buf.extend(sub)
        elif isinstance(value, str):
            buf.append(0x02)
            buf.extend(key.encode('utf-8') + b'\x00')
            self._encode_string(buf, value)
        elif isinstance(value, bool):
            buf.append(0x08)
            buf.extend(key.encode('utf-8') + b'\x00')
            buf.append(1 if value else 0)
        elif value is None:
            buf.append(0x0A)
            buf.extend(key.encode('utf-8') + b'\x00')
            # nothing further
        elif isinstance(value, datetime.datetime):
            buf.append(0x09)
            buf.extend(key.encode('utf-8') + b'\x00')
            ms = int(value.timestamp() * 1000)
            buf.extend(struct.pack("<q", ms))
        elif isinstance(value, bytes):
            buf.append(0x05)
            buf.extend(key.encode('utf-8') + b'\x00')
            subtype = 0   # generic binary subtype
            length = len(value)
            buf.extend(struct.pack("<i", length))
            buf.append(subtype)
            buf.extend(value)
        elif isinstance(value, ObjectId):
            buf.append(0x07)
            buf.extend(key.encode('utf-8') + b'\x00')
            buf.extend(bytes(value))
        elif isinstance(value, int):
            # Choose int32 or int64 based on value range
            if -2**31 <= value <= 2**31 - 1:
                buf.append(0x10)   # int32
                buf.extend(key.encode('utf-8') + b'\x00')
                buf.extend(struct.pack("<i", value))
            else:
                buf.append(0x12)   # int64
                buf.extend(key.encode('utf-8') + b'\x00')
                buf.extend(struct.pack("<q", value))
        elif isinstance(value, float):
            buf.append(0x01)
            buf.extend(key.encode('utf-8') + b'\x00')
            buf.extend(struct.pack("<d", value))
        else:
            raise TypeError(f"Unsupported type for key '{key}': {type(value)}")

    def _encode_string(self, buf, s):
        encoded = s.encode('utf-8') + b'\x00'
        length = len(encoded)   # includes terminator
        buf.extend(struct.pack("<i", length))
        buf.extend(encoded)

Implementing a BSON Decoder

The decoder reverses the process: it reads the 4‑byte length, then walks through elements until it hits the terminator. For each element it reads the type byte, extracts the cstring key, and decodes the value based on the type code.

Decoding Elements

We’ll build helper methods that consume bytes from a buffer and return the Python value together with the new buffer position. For example, decoding a double reads 8 bytes:

def decode_double(buf, pos):
    val = struct.unpack_from("<d", buf, pos)[0]
    return val, pos + 8

Strings require reading the length, then extracting UTF‑8 bytes up to the null terminator:

def decode_string(buf, pos):
    length = struct.unpack_from("<i", buf, pos)[0]
    pos += 4
    # string bytes = buf[pos : pos+length-1] (last byte is \x00)
    s = buf[pos : pos+length-1].decode('utf-8')
    return s, pos + length

Full Decoder Implementation

The BSONDecoder class contains a main decode method and a recursive _decode_document that handles embedded documents and arrays.

import struct
import datetime

class BSONDecoder:
    def decode(self, data):
        """Decode BSON bytes into a Python dict."""
        if len(data) < 4:
            raise ValueError("Invalid BSON: too short")
        total_len = struct.unpack_from("<i", data, 0)[0]
        # optional: verify total_len == len(data)
        buffer = data[4:]   # skip length prefix
        doc, consumed = self._decode_document(buffer)
        return doc

    def _decode_document(self, buf):
        """Decode a document from a buffer that starts with elements and ends with \x00."""
        doc = {}