← Back to DevBytes
Python July 09, 2026 14 min read

Fix 'MemoryError' in Python: Complete Troubleshooting Guide

Understanding Python's MemoryError

A MemoryError is raised when Python's memory manager cannot allocate enough memory for an operation. Unlike a typical exception that you can catch and gracefully handle, a MemoryError often indicates that your process has exhausted available RAM — and sometimes even swap space — leaving the interpreter in a precarious state.

The error message is deceptively simple:

Traceback (most recent call last):
  File "script.py", line 42, in <module>
    result = [i ** 2 for i in range(10**9)]
MemoryError

But the root cause can range from genuinely insufficient hardware to subtle code patterns that accidentally materialize enormous data structures in memory. This guide walks you through every facet of diagnosing and fixing MemoryError in Python.

What Exactly Triggers a MemoryError?

🚀 Deploy your AI agent in 10 minutes

Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.

Try it free →

Python manages memory through a private heap. When you create objects — integers, lists, dictionaries, class instances — the interpreter requests memory from the operating system. If the OS denies that request (because physical RAM and swap are exhausted, or because a per-process limit has been hit), Python raises MemoryError.

Importantly, MemoryError is not the same as a slow program or high CPU usage. It specifically means an allocation request failed. You may still have gigabytes of free RAM on the machine, but a single allocation for a multi-gigabyte contiguous block can fail if the OS cannot satisfy it.

Key Distinction: MemoryError vs. Other Resource Errors

Knowing this distinction prevents you from applying the wrong fix. If you see MemoryError, the problem is always about memory allocation.

Why Fixing MemoryError Matters

Ignoring or working around MemoryError without understanding its cause leads to several problems:

Properly addressing MemoryError makes your code robust, predictable, and ready for production environments where resources are finite.

Common Causes (and How to Spot Them)

1. Loading Entire Datasets Into Memory

The most frequent culprit. Reading a large CSV, JSON file, or database query result into a single list or DataFrame can easily exhaust RAM.

# Dangerous: loads entire file into memory
with open('massive_log.json') as f:
    data = json.load(f)  # MemoryError if file is multi-gigabyte

# Also dangerous: materializing a huge list
rows = [row for row in csv.reader(open('big.csv'))]  # All rows in RAM

2. Unbounded Accumulation in Loops

Lists, dicts, or sets that grow without limit inside long-running loops will eventually consume all available memory.

# Memory leak by accumulation
cache = {}
for user_id in infinitely_streaming_ids():
    cache[user_id] = expensive_computation(user_id)  # Dict grows forever

3. Exploding Cartesian Products

Cross joins, permutations, or combinatorial generators materialized as lists can produce exponentially large data structures.

from itertools import product

# Fine as an iterator — but materializing it is deadly
combinations = list(product(range(1000), range(1000), range(100)))  # 100 million tuples

4. Large Object Graphs with Circular References

Objects that reference each other cyclically can prevent the garbage collector from freeing memory promptly, causing retained memory to grow until allocation fails. This is especially dangerous when combined with __del__ methods that the GC cannot safely collect.

class Node:
    def __init__(self, name):
        self.name = name
        self.parent = None
        self.children = []

    def add_child(self, child):
        self.children.append(child)
        child.parent = self  # Circular reference created

5. Inefficient Data Structures

Using Python's default types without considering their memory footprint can waste enormous amounts of RAM. A Python int is 28 bytes (on 64-bit CPython), not 4. A list of 1 million integers uses ~8 MB just for pointers, plus 28 MB for the integer objects themselves.

6. Forking Processes (Copy-on-Write Blowup)

On Linux, multiprocessing with fork start method shares memory pages via copy-on-write. But when the child process modifies reference counts (which happens on almost any Python operation), pages are duplicated. A parent with 4 GB of state can cause each child to gradually consume 4 GB of its own.

Diagnosing MemoryError: Tools and Techniques

Before applying fixes, you must understand where memory is going. Use these approaches in order:

1. Check System and Process Limits

# Linux: check available memory
$ free -h
$ ulimit -v   # virtual memory limit (if set)
$ ulimit -m   # max resident set size (if set)

# macOS: check process limits
$ launchctl limit

If ulimit -v returns a small value (like 1 GB), your process may be hitting an artificial ceiling. Raise it with ulimit -v unlimited (or consult your container orchestrator if running in Docker/Kubernetes).

2. Profile Memory Usage with tracemalloc

Python's built-in tracemalloc module tracks allocations by line number. It's lightweight enough to run in production for short periods.

import tracemalloc

tracemalloc.start()

# ... run your code that triggers MemoryError ...

snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')

print("Top 10 memory allocations:")
for stat in top_stats[:10]:
    print(stat)
    # Shows file, line, and memory usage
    # e.g.:  script.py:42: size=856 MiB (+782 MiB), count=1000000, average=896 B

3. Use memory_profiler for Line-by-Line Analysis

The memory_profiler third-party package decorates functions to report memory usage per line.

# pip install memory-profiler

from memory_profiler import profile

@profile
def process_records(filepath):
    with open(filepath) as f:
        data = f.readlines()     # Shows memory increment
    parsed = [parse(line) for line in data]  # Shows another jump
    return parsed

4. Inspect Object Sizes with sys.getsizeof

For a quick sanity check on individual objects:

import sys

data = [1] * 1_000_000
print(f"List of 1M ints: {sys.getsizeof(data) / 1e6:.2f} MB")  # ~8 MB for list container

# But to get the full recursive size, use a helper:
def total_size(obj, seen=None):
    if seen is None:
        seen = set()
    obj_id = id(obj)
    if obj_id in seen:
        return 0
    seen.add(obj_id)
    size = sys.getsizeof(obj)
    if isinstance(obj, dict):
        size += sum(total_size(k, seen) + total_size(v, seen) for k, v in obj.items())
    elif isinstance(obj, (list, tuple, set, frozenset)):
        size += sum(total_size(item, seen) for item in obj)
    return size

print(f"Total size: {total_size(data) / 1e6:.2f} MB")  # ~36 MB (28 bytes per int + 8 bytes per pointer)

5. Monitor Real-Time with guppy3 or pympler

For long-running processes, take periodic heap snapshots to detect growth trends.

# pip install pympler

from pympler import tracker

tr = tracker.SummaryTracker()
# ... first snapshot ...
tr.print_diff()  # Shows what changed since last snapshot

Fix #1: Use Iterators and Generators Instead of Lists

The single most impactful fix: replace materialized sequences with lazy iterators. Generators compute values on demand and hold only one item at a time in memory.

# Before: list comprehension — all results in memory
squares = [i ** 2 for i in range(10**8)]  # MemoryError

# After: generator expression — yields one value at a time
squares_gen = (i ** 2 for i in range(10**8))  # No memory allocated yet
for sq in squares_gen:
    if sq > 1000:
        break  # Only computes what's needed, minimal RAM

Apply this pattern to file reading as well:

# Before: read entire file
with open('huge.log') as f:
    lines = f.readlines()  # All lines in memory
    for line in lines:
        process(line)

# After: iterate lazily
with open('huge.log') as f:
    for line in f:  # f is an iterator; one line at a time
        process(line)

For JSON, use ijson for streaming parsing:

import ijson

with open('massive.json', 'rb') as f:
    # Streaming: yields objects one at a time
    for record in ijson.items(f, 'records.item'):
        process(record)

Fix #2: Process Data in Chunks

When you must work with large collections, process them in fixed-size chunks. This bounds memory usage to a predictable ceiling.

import pandas as pd

# Read a large CSV in chunks
chunk_size = 100_000
results = []

for chunk in pd.read_csv('big_table.csv', chunksize=chunk_size):
    # chunk is a DataFrame with at most 100k rows
    filtered = chunk[chunk['value'] > 0]
    agg = filtered.groupby('category').sum()
    results.append(agg)  # Still accumulating, but much smaller

# Combine chunk-level aggregates
final = pd.concat(results).groupby('category').sum()

The chunking pattern works universally:

def chunked_iterable(iterable, size):
    """Yield fixed-size chunks from an iterable."""
    chunk = []
    for item in iterable:
        chunk.append(item)
        if len(chunk) >= size:
            yield chunk
            chunk = []
    if chunk:
        yield chunk

# Process 1 million records in chunks of 10,000
for batch in chunked_iterable(range(1_000_000), 10_000):
    processed = [expensive_op(x) for x in batch]
    save_to_db(processed)
    # Each batch uses bounded memory, then gets garbage collected

Fix #3: Use Memory-Efficient Data Structures

Use array.array for Homogeneous Numeric Data

Python's array module stores typed numeric values compactly (similar to C arrays).

import array

# Before: list of 10 million integers (~280 MB for ints + ~80 MB for list pointers)
nums_list = [i for i in range(10_000_000)]

# After: array of type 'i' (signed int, 4 bytes each) — only ~40 MB
nums_array = array.array('i', range(10_000_000))

Use NumPy Arrays for Numerical Work

NumPy arrays store data in contiguous C-style buffers with minimal overhead.

import numpy as np

# 10 million float64 values
arr = np.arange(10_000_000, dtype=np.float64)
print(f"NumPy array size: {arr.nbytes / 1e6:.2f} MB")  # 80 MB

# Equivalent Python list would be ~360+ MB

Use __slots__ for Class Instances

By default, Python class instances store attributes in a __dict__ which is a hash map with overhead. Using __slots__ stores attributes in a compact array, saving roughly 40-60% memory per instance.

# Before: each instance has __dict__ overhead
class DataPoint:
    def __init__(self, x, y, z, label):
        self.x = x
        self.y = y
        self.z = z
        self.label = label

# After: __slots__ eliminates __dict__ and __weakref__
class DataPointSlotted:
    __slots__ = ('x', 'y', 'z', 'label')
    def __init__(self, x, y, z, label):
        self.x = x
        self.y = y
        self.z = z
        self.label = label

# For 1 million instances, this saves hundreds of MB

Use bytes/bytearray Instead of Strings for Binary Data

If you're storing large blobs of text that don't need rich string operations, keep them as bytes.

Fix #4: Release References Explicitly

Python's garbage collector usually does the right thing, but in tight memory situations you can help by explicitly deleting references and calling gc.collect().

import gc

def heavy_computation():
    large_intermediate = [expensive_calc(i) for i in range(10**7)]
    result = summarize(large_intermediate)
    
    # Explicitly release the large intermediate list
    del large_intermediate
    gc.collect()  # Force collection before proceeding
    
    return result

For long-running loops, periodically clear accumulators:

accumulator = []
for i, item in enumerate(streaming_source()):
    accumulator.append(process(item))
    if i % 100_000 == 0:
        # Flush accumulator to disk and reset
        flush_to_disk(accumulator)
        accumulator.clear()  # or: accumulator = []
        gc.collect()

Fix #5: Break Circular References

Use weakref to avoid reference cycles that prevent garbage collection:

import weakref

class Parent:
    def __init__(self, name):
        self.name = name
        self.children = []  # Strong references to children

class Child:
    def __init__(self, name, parent):
        self.name = name
        # Use weak reference to avoid circular dependency
        self.parent = weakref.ref(parent)  # Does not increase reference count
    
    @property
    def parent_name(self):
        p = self.parent()
        return p.name if p else None

For objects with __del__ methods, the GC places them in a special list (gc.garbage) if they're part of a cycle. Avoid __del__ when possible, or use weakref to break cycles.

Fix #6: Use Disk-Backed Storage (Out-of-Core Processing)

When data doesn't fit in RAM, keep it on disk and access it incrementally.

Use SQLite for Structured Data

import sqlite3

# Instead of keeping everything in a dict
conn = sqlite3.connect('file:cache.db?mode=memory&cache=shared', uri=True)
conn.execute('CREATE TABLE IF NOT EXISTS cache (key TEXT PRIMARY KEY, value BLOB)')

def get_or_compute(key):
    cursor = conn.execute('SELECT value FROM cache WHERE key = ?', (key,))
    row = cursor.fetchone()
    if row:
        return row[0]
    value = expensive_computation(key)
    conn.execute('INSERT OR REPLACE INTO cache VALUES (?, ?)', (key, value))
    conn.commit()
    return value

Use Memory-Mapped Files

The mmap module lets you access a file as if it were a byte array, with the OS handling paging:

import mmap

with open('huge_data.bin', 'r+b') as f:
    with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_WRITE) as m:
        # m behaves like a bytearray, but OS pages data in/out on demand
        # You can access m[offset] without loading the whole file
        for offset in range(0, len(m), record_size):
            process(m[offset:offset+record_size])

Use Dask or Vaex for Out-of-Core DataFrames

import dask.dataframe as dd

# Dask splits the DataFrame into chunks, processing them lazily
df = dd.read_csv('s3://bucket/massive/*.csv')
result = df[df.value > 0].groupby('category').sum().compute()
# Only materializes final (hopefully smaller) result in memory

Fix #7: Optimize Multiprocessing Memory Sharing

When using multiprocessing, the fork start method can cause memory duplication. Switch to spawn (available in Python 3.4+) or use shared memory explicitly.

import multiprocessing as mp

# Option A: Use 'spawn' start method (no copy-on-write)
if __name__ == '__main__':
    mp.set_start_method('spawn')
    with mp.Pool(processes=4) as pool:
        results = pool.map(worker, tasks)

# Option B: Use shared memory for large read-only data
# (Python 3.8+ multiprocessing.shared_memory)
from multiprocessing.shared_memory import SharedMemory
import numpy as np

# Create shared NumPy array
a = np.zeros(1_000_000, dtype=np.float64)
shm = SharedMemory(create=True, size=a.nbytes)
shared_array = np.ndarray(a.shape, dtype=a.dtype, buffer=shm.buf)
np.copyto(shared_array, a)  # Copy data once into shared memory

# Workers can now access shared_array without duplicating it

Fix #8: Switch to 64-bit Python and Increase Virtual Memory

If you're running 32-bit Python, the process is limited to ~2-4 GB of addressable memory regardless of physical RAM. Verify with:

import sys
print(sys.maxsize)  # If ~2**31-1, you're on 32-bit

Switch to 64-bit Python to access the full machine address space. Additionally, ensure swap space is configured:

# Linux: create a swap file
$ sudo fallocate -l 8G /swapfile
$ sudo chmod 600 /swapfile
$ sudo mkswap /swapfile
$ sudo swapon /swapfile

# Verify
$ free -h

Note: Swap allows allocations to succeed but may make your program extremely slow if it causes thrashing. It's a safety net, not a performance solution.

Fix #9: Use Cloud-Scale Patterns (MapReduce, Streaming)

For truly massive datasets, redesign your architecture around streaming and distributed processing:

# Conceptual streaming pattern
def streaming_pipeline(source_stream):
    for record in source_stream:
        enriched = enrich(record)
        validated = validate(enriched)
        yield validated  # Downstream consumer processes and frees memory

Best Practices for Preventing MemoryError

1. Preallocate and Reuse Buffers

Avoid repeated allocation of large temporary objects. Preallocate once and reuse:

# Preallocate a buffer for processing
BUFFER_SIZE = 1_000_000
buffer = bytearray(BUFFER_SIZE)

def process_data(source):
    while True:
        n = source.readinto(buffer)  # Reuses the same buffer
        if n == 0:
            break
        handle(buffer[:n])

2. Set Hard Memory Limits

Use resource module (Unix) to enforce a ceiling and fail fast:

import resource

# Set max heap to 4 GB (in bytes)
resource.setrlimit(resource.RLIMIT_DATA, (4 * 1024**3, 4 * 1024**3))

3. Monitor and Alert on Memory Growth

In production, expose memory metrics and set alerts:

import os
import psutil

def memory_usage_gb():
    process = psutil.Process(os.getpid())
    return process.memory_info().rss / 1e9  # Resident Set Size in GB

# Log or expose as Prometheus metric
print(f"Current RSS: {memory_usage_gb():.3f} GB")

4. Use Context Managers for Resource Cleanup

Always wrap resource-intensive operations in context managers:

from contextlib import contextmanager

@contextmanager
def temporary_buffer(size):
    buf = bytearray(size)
    try:
        yield buf
    finally:
        del buf  # Guaranteed cleanup

5. Test Under Memory Constraints

Simulate low-memory conditions in CI to catch MemoryError before deployment:

# Run your test suite with limited memory
$ docker run --memory=512m --memory-swap=512m my-python-app pytest

6. Favor Functional Pipeline Patterns

Chain operations without materializing intermediate collections:

# Bad: materializes three large lists
step1 = [op1(x) for x in data]
step2 = [op2(x) for x in step1]
step3 = [op3(x) for x in step2]

# Good: generator pipeline
result = (op3(op2(op1(x))) for x in data)
for item in result:
    consume(item)

7. Profile Before Optimizing

Don't guess where memory goes — always profile first with tracemalloc or memory_profiler. Premature memory optimization can complicate code without solving the real bottleneck.

Emergency Fixes When MemoryError Strikes in Production

If you encounter MemoryError in a running system and need immediate mitigation:

  1. Restart the process — The quickest way to reclaim leaked memory
  2. Reduce worker count — Fewer concurrent workers means less aggregate memory pressure
  3. Enable swap temporarily — Buys time while you deploy a proper fix
  4. Drop caches — On Linux: sync; echo 3 > /proc/sys/vm/drop_caches (frees OS cache, not process memory)
  5. Roll back recent deployment — If the error started after a code change

Conclusion

MemoryError in Python is a signal that your code is attempting to allocate more memory than the system can provide — whether due to genuinely large data, inefficient data structures, or accidental materialization of lazy sequences. The fix is rarely a single line change; it requires understanding your data flow, choosing appropriate data structures (array, NumPy, __slots__), embracing iterators over lists, processing data in chunks, and sometimes redesigning your architecture around streaming or distributed patterns. Profile relentlessly with tracemalloc and memory_profiler before applying any fix, test under memory constraints, and set up monitoring so you catch memory growth before it becomes an outage. With the techniques in this guide — from generator expressions to shared memory multiprocessing — you now have a complete toolkit to diagnose, fix, and prevent MemoryError in any Python application.

🚀 Need a reliable AI agent for your project?

Deploy Hermes Agent in 10 minutes. Managed hosting, zero DevOps.

Get Started — $23.99/mo
← Back to all articles