Migrating from Redis to Memcached: Step-by-Step Guide

Introduction: Understanding the Shift from Redis to Memcached

Redis and Memcached are both battle-tested, in-memory key-value stores often used for caching, session storage, and transient data. However, they differ significantly in feature set and operational behavior. Redis offers a rich data model (strings, hashes, lists, sets, sorted sets, streams, geospatial indexes), persistence, replication, pub/sub, and Lua scripting. Memcached focuses on simplicity: it stores string-based key-value pairs, operates purely in RAM without persistence, and excels at high-throughput, low-latency caching.

Migrating from Redis to Memcached is a strategic choice. It can simplify your infrastructure, reduce memory overhead when you only need a basic cache, lower operational complexity, and sometimes yield better raw throughput for simple key-value workloads. However, the migration demands careful planning because features like list operations, sorted sets, or pub/sub must be replaced with other technologies or application-level logic.

This tutorial provides a complete step-by-step guide for migrating a typical application from Redis to Memcached, covering analysis, code translation, data migration, testing, and best practices.

Why Migrate from Redis to Memcached

Operational Simplicity

Memcached has a minimalistic design. It doesn’t require snapshotting, AOF persistence, or failover configuration. The server is lightweight, easy to deploy, and scales horizontally using client-side sharding (with libraries like libmemcached or proxy solutions). This reduces maintenance burden and eliminates concerns like disk-write contention or memory fragmentation from persistence.

Memory Efficiency for Plain Caching

When you only use Redis for simple key-value caching (no advanced data types), Redis's internal overhead (per-key metadata, dictionary structures, expiry management) can consume more memory than Memcached. Memcached's slab allocator is tuned for fixed-size chunks and often uses less memory for homogeneous workloads. Migrating can free up RAM and lower your instance sizes.

Pure Cache Semantics

Memcached is designed as a volatile cache; it will evict items when memory is exhausted, with no persistence guarantees. This aligns perfectly with caching use cases where data can be regenerated. Redis’s persistence features are irrelevant in such scenarios, and migrating removes the temptation to misuse Redis as a primary database for non-critical data.

Client Library Diversity and Multi-Language Support

Memcached has mature, fast clients in virtually every language, with consistent hashing and binary protocol support. Its simplicity leads to predictable behavior and easy debugging.

Planning and Assessing Your Redis Usage

Before writing a single line of migration code, inventory all Redis operations in your application. Look for:

• Simple GET/SET with optional TTL — directly migratable to Memcached.
• MGET/MSET (multi-get / multi-set) — Memcached supports get_multi and set_multi via most clients, though atomicity differs.
• INCR/DECR (atomic counters) — Memcached provides incr and decr commands (only for numeric string values).
• Data structures: lists, sets, sorted sets, hashes — These have no direct equivalent. You must either replace them with a dedicated database (e.g., PostgreSQL, Redis kept for those features), or implement application-level alternatives with multiple keys and careful handling.
• Pub/Sub or Streams — Migrate to a message broker like RabbitMQ, Kafka, or Redis kept solely for messaging.
• Lua scripts, transactions (MULTI/EXEC) — No Memcached equivalent. Decompose logic into idempotent operations or use a database for atomicity.
• Key expiration patterns, TTL refresh — Memcached supports expiration (in seconds) and touch to update expiration.

Document every Redis key prefix, its purpose, TTL, and how the data is regenerated. This audit will guide the translation.

Step 1: Setting Up Memcached

Install Memcached on your target environment. On Ubuntu/Debian:

sudo apt update
sudo apt install memcached
sudo systemctl enable memcached
sudo systemctl start memcached

On macOS via Homebrew:

brew install memcached
brew services start memcached

Default configuration listens on 127.0.0.1:11211. For production, adjust memory limit (-m 64 for 64MB), maximum connections (-c 1024), and binding to private network interfaces. You can edit /etc/memcached.conf (Debian/Ubuntu) or use command-line flags.

Verify connectivity using telnet or a quick script:

# telnet 127.0.0.1 11211
set test_key 0 3600 5
hello

The 0 is flags, 3600 expiration in seconds, 5 is the byte length of the value. Memcached will respond STORED.

Step 2: Translating Redis Commands to Memcached

Map each Redis operation to its Memcached equivalent. Here are the most common translations:

For operations without equivalents (e.g., LPUSH, SADD), you must redesign the feature. For example, a Redis list used as a queue can be replaced by a dedicated message queue (RabbitMQ, SQS). A set used for deduplication can be replaced by a database table with unique constraints plus a Memcached cache for frequently checked values.

Step 3: Data Migration Script

If you need to pre-seed the Memcached cache with existing Redis data (hot keys that are expensive to regenerate), you can write a migration script. This example uses Python with redis-py and pymemcache:

import redis
from pymemcache.client.base import Client

# Connect to Redis source
redis_client = redis.Redis(host='redis-host', port=6379, db=0)

# Connect to Memcached target (use consistent hashing for production)
memcached_client = Client(('memcached-host', 11211))

# Define a pattern to scan keys (e.g., "cache:*")
pattern = "cache:*"
cursor = 0
batch_size = 100

while True:
    cursor, keys = redis_client.scan(cursor=cursor, match=pattern, count=batch_size)
    for key in keys:
        # Retrieve value and TTL (in seconds)
        value = redis_client.get(key)
        if value is None:
            continue
        ttl = redis_client.ttl(key)
        # Redis returns -1 for no expiry, -2 if key doesn't exist (shouldn't happen)
        expire = 0 if ttl == -1 else max(1, ttl)  # memcached: 0 = no expiry

        # Store in Memcached as bytes (pymemcache expects bytes or str)
        memcached_client.set(key.decode('utf-8'), value, expire=expire)

    if cursor == 0:
        break

print("Migration completed.")

Important considerations:

• SCAN is iterative to avoid blocking Redis. Adjust batch_size for performance.
• Memcached expiration is in seconds, with 0 meaning "never expire" (but will be evicted under memory pressure). Redis TTL of -1 maps to 0.
• If keys contain binary data or special characters, ensure proper encoding.
• For large datasets, run this script during low-traffic windows and consider using pipelines (pymemcache supports set_multi).
• Only migrate cache data that can't be easily regenerated; otherwise, let the application warm Memcached naturally.

Step 4: Updating Application Code

Replace the Redis client library with a Memcached client in your application. The following Python example shows a typical cache layer before and after migration.

Original Redis-based cache service

import redis

class RedisCache:
    def __init__(self, host, port, db):
        self.client = redis.Redis(host=host, port=port, db=db)

    def get(self, key):
        return self.client.get(key)

    def set(self, key, value, ttl=None):
        if ttl:
            self.client.setex(key, ttl, value)
        else:
            self.client.set(key, value)

    def delete(self, key):
        self.client.delete(key)

    def increment(self, key, delta=1):
        return self.client.incr(key, delta)

New Memcached-based cache service

from pymemcache.client.base import Client
import time

class MemcacheCache:
    def __init__(self, host, port):
        # For production, use a hash-aware client like HashClient
        self.client = Client((host, port))

    def get(self, key):
        # Ensure key is str or bytes; pymemcache expects bytes
        key_bytes = key.encode('utf-8') if isinstance(key, str) else key
        result = self.client.get(key_bytes)
        return result  # Returns bytes or None

    def set(self, key, value, ttl=None):
        key_bytes = key.encode('utf-8') if isinstance(key, str) else key
        expire = ttl if ttl is not None else 0  # 0 = no expiry
        self.client.set(key_bytes, value, expire=expire)

    def delete(self, key):
        key_bytes = key.encode('utf-8') if isinstance(key, str) else key
        self.client.delete(key_bytes)

    def increment(self, key, delta=1):
        key_bytes = key.encode('utf-8') if isinstance(key, str) else key
        # Memcached incr returns the new value as int
        return self.client.incr(key_bytes, delta)

If your application uses Redis-specific features like HGETALL on a hash, you'll need to refactor. For example, a hash that stores user profile fields can be split into multiple keys like user:123:name, user:123:email, or stored as a JSON blob under a single key. Choose based on update patterns and size.

Example: Migrating a Redis hash to Memcached

# Before (Redis hash)
redis.hset("user:123", mapping={"name": "Alice", "email": "alice@example.com"})
profile = redis.hgetall("user:123")

# After (Memcached JSON blob)
import json
user_profile = {"name": "Alice", "email": "alice@example.com"}
memcached.set("user:123", json.dumps(user_profile).encode('utf-8'), expire=3600)

# Retrieval
data = memcached.get("user:123")
profile = json.loads(data.decode('utf-8')) if data else {}

For atomic updates to individual fields, you can store each field as a separate key and use set / get_multi. For counters inside a structure, consider a database-backed approach.

Step 5: Testing and Validation

Thorough testing is critical. Plan multiple phases:

• Unit Tests: Mock the cache layer and verify that your new Memcached client methods behave as expected (including edge cases: key not found, TTL behavior, binary data).
• Integration Tests: Run a local Memcached instance and test cache operations end-to-end with sample data. Validate that serialization/deserialization works correctly.
• Load Testing: Use a tool like memtier_benchmark or custom scripts to simulate production traffic against the Memcached setup. Measure latency, hit ratio, and memory usage. Compare to Redis baseline.
• Parallel Run (Dual Writes): Temporarily write to both Redis and Memcached while reading from Redis. Monitor Memcached for consistency. This allows a safe rollback.
• Shadow Traffic: In production, send a small percentage of read traffic to Memcached and log mismatches against Redis. Adjust until discrepancies are negligible.

Testing also validates your new expiration logic. Memcached eviction is based on slab class; under memory pressure, it may evict keys before their expiration. Ensure your application handles cache misses gracefully (regeneration or fallback).

Step 6: Cutover and Rollback Plan

Execute the final switch with a gradual approach:

1. Deploy the application with feature flags to toggle between Redis and Memcached reads.
2. Increase Memcached read percentage from 0% to 100% while monitoring error rates and latency.
3. Once confident, disable writes to Redis, stop the Redis instance, and decommission it.
4. Keep the old Redis data disk snapshot as a backup for a few days.

Always have a rollback strategy: if errors spike, revert the read toggle back to Redis and investigate. The dual-write phase ensures Redis data is still fresh.

Best Practices for Memcached

Use Consistent Hashing for Distributed Caching

When scaling Memcached across multiple nodes, client-side sharding with consistent hashing (e.g., pymemcache.HashClient or libmemcached) ensures minimal cache invalidation when nodes are added or removed. Avoid using a single large instance; instead, partition by key.

Namespace Your Keys and Avoid Collisions

Adopt a key naming convention similar to Redis: cache:user:123:profile. This prevents key collisions and simplifies debugging. Use colon or dash separators.

Set Sensible Expiration Times

Always set a TTL, even if data could theoretically live forever. Memcached's LRU eviction will remove items under memory pressure, but explicit TTLs help prevent stale data accumulation. Use values like 3600 (1 hour) for frequently updated data, 86400 for daily refreshes.

Monitor Memory Usage and Eviction Rate

Track bytes, evictions, and curr_items via stats command. Set alerts when evictions occur frequently — this indicates insufficient memory or a need to optimize keys. Adjust the -m flag or add nodes.

Handle Cache Misses with Grace

Never assume a cache hit. Implement a robust fallback: query the database, recalculate, and populate the cache. Use locking (e.g., add operation to prevent stampeding on a cold key) if needed.

Validate Key Size and Value Size

Memcached limits keys to 250 bytes and values to 1 MB by default (configurable). Ensure your application respects these limits. Compress large values (e.g., JSON) with zlib before storing if they approach the limit.

Use Binary Protocol When Possible

Binary protocol reduces parsing overhead and supports operations like get_multi more efficiently. Most clients default to text protocol; check your library’s configuration.

Conclusion

Migrating from Redis to Memcached is not merely swapping libraries — it’s a strategic decision that simplifies your caching layer, reduces operational overhead, and can improve memory efficiency for plain key-value workloads. By systematically auditing your Redis usage, translating commands, handling unsupported data types with application refactoring, and implementing a phased rollout with rigorous testing, you can achieve a smooth transition. Embrace Memcached’s simplicity: focus on key design, proper expiration, and robust fallback logic. With the steps and best practices outlined here, you’re well-equipped to decommission Redis and let Memcached drive your caching infrastructure reliably.