Designing a Real-Time Analytics with Redis Caching

Designing Real-Time Analytics with Redis Caching

What It Is

Real-time analytics involves processing and querying data as it arrives, typically within sub-second latency. When combined with Redis caching, you leverage an in-memory data store to handle high-velocity writes and fast reads, making it possible to compute metrics like page views, unique visitors, active users, and event counts on the fly. Redis provides specialized data structures (Sorted Sets, HyperLogLog, Bitmaps, Streams) that are ideally suited for real-time aggregation and time-window analysis without needing a heavy database.

Why It Matters

Low latency: Redis operations are in-memory, delivering microsecond response times for both writes and reads.
High throughput: Can handle millions of events per second on a single instance, making it perfect for real-time dashboards.
Built-in data structures: Sorted Sets for leaderboards and time-series, HyperLogLog for approximate unique counts, Bitmaps for boolean tracking, and Streams for event streaming.
Stateless and scalable: Redis can be clustered, replicated, and used as a caching layer in front of a primary database, reducing load and cost.
Time-to-live (TTL) support: Automatically expire old data, keeping memory usage predictable without manual cleanup.

How to Use It

Below we build a complete real-time analytics system using Node.js and Redis. The examples assume you have a Redis server running locally (default port 6379). Install the redis npm package:

npm install redis

1. Connecting to Redis

Create a reusable client module:

// redisClient.js
import { createClient } from 'redis';

const client = createClient({
    url: 'redis://localhost:6379'
});

client.on('error', (err) => console.error('Redis Client Error', err));
await client.connect();

export default client;

2. Real-Time Page View Counter (Sorted Set with Minute Buckets)

We track page views per minute using a sorted set where the score is the timestamp and the member is a page identifier. To get the count for the last 60 seconds, we use ZCOUNT.

// pageViews.js
import client from './redisClient.js';

// Record a page view for a given page
export async function recordPageView(pageId) {
    const now = Date.now();
    // Use a sorted set key per page, e.g., "pageviews:home"
    const key = `pageviews:${pageId}`;
    await client.zAdd(key, { score: now, value: `${now}:${Math.random()}` }); // unique member to allow duplicates
    // Set TTL to 2 minutes to automatically clean old data
    await client.expire(key, 120);
}

// Get count of views in last N seconds
export async function getPageViewsLastNSeconds(pageId, seconds = 60) {
    const key = `pageviews:${pageId}`;
    const min = Date.now() - seconds * 1000;
    const max = Date.now();
    const count = await client.zCount(key, min, max);
    return count;
}

3. Unique Visitor Tracking (HyperLogLog)

HyperLogLog uses constant memory (~12KB) to estimate unique elements. We track unique visitors per page per day.

// uniqueVisitors.js
import client from './redisClient.js';

export async function recordVisitor(pageId, visitorId) {
    const key = `unique:${pageId}:${new Date().toISOString().slice(0,10)}`; // daily key
    await client.pfAdd(key, visitorId);
    // Expire after 2 days
    await client.expire(key, 172800);
}

export async function getUniqueVisitorsToday(pageId) {
    const key = `unique:${pageId}:${new Date().toISOString().slice(0,10)}`;
    const count = await client.pfCount(key);
    return count;
}

4. Event Stream Processing with Redis Streams

Redis Streams allow reliable event ingestion and consumer groups. We'll create a simple producer and consumer for real-time events.

// eventProducer.js
import client from './redisClient.js';

export async function publishEvent(eventType, data) {
    const streamKey = 'analytics:events';
    const event = {
        type: eventType,
        data: JSON.stringify(data),
        timestamp: Date.now().toString()
    };
    await client.xAdd(streamKey, '*', event);
}

// eventConsumer.js – runs as a separate process
import client from './redisClient.js';
import { recordPageView, recordVisitor } from './analytics.js'; // assume imported

const groupName = 'analytics-consumers';
const consumerName = `consumer-${Math.random().toString(36).slice(2,8)}`;

async function startConsumer() {
    const streamKey = 'analytics:events';
    // Create consumer group (if not exists)
    try {
        await client.xGroupCreate(streamKey, groupName, '$', { MKSTREAM: true });
    } catch (e) {
        // group already exists
    }

    console.log(`Consumer ${consumerName} started`);

    while (true) {
        const results = await client.xReadGroup(
            groupName,
            consumerName,
            { key: streamKey, id: '>' },
            { COUNT: 10, BLOCK: 5000 }
        );

        if (results) {
            for (const { messages } of results) {
                for (const msg of messages) {
                    const { type, data, timestamp } = msg.message;
                    console.log(`Processing event: ${type}`);

                    // Route to analytics functions
                    if (type === 'pageview') {
                        const { pageId } = JSON.parse(data);
                        await recordPageView(pageId);
                    } else if (type === 'visitor') {
                        const { pageId, visitorId } = JSON.parse(data);
                        await recordVisitor(pageId, visitorId);
                    }

                    // Acknowledge the message
                    await client.xAck(streamKey, groupName, msg.id);
                }
            }
        }
    }
}

startConsumer();

5. Express API to Serve Real-Time Metrics

// server.js
import express from 'express';
import { recordPageView, getPageViewsLastNSeconds } from './pageViews.js';
import { recordVisitor, getUniqueVisitorsToday } from './uniqueVisitors.js';
import { publishEvent } from './eventProducer.js';

const app = express();
app.use(express.json());

// Record a page view (via event stream)
app.post('/api/event', async (req, res) => {
    const { type, pageId, visitorId } = req.body;
    if (type === 'pageview') {
        await publishEvent('pageview', { pageId });
    } else if (type === 'visitor') {
        await publishEvent('visitor', { pageId, visitorId });
    }
    res.send({ status: 'ok' });
});

// Get current page view count (last 60 seconds)
app.get('/api/pageviews/:pageId', async (req, res) => {
    const count = await getPageViewsLastNSeconds(req.params.pageId, 60);
    res.json({ pageId: req.params.pageId, viewsLast60s: count });
});

// Get unique visitors today
app.get('/api/unique/:pageId', async (req, res) => {
    const count = await getUniqueVisitorsToday(req.params.pageId);
    res.json({ pageId: req.params.pageId, uniqueVisitorsToday: count });
});

app.listen(3000, () => console.log('Server on port 3000'));

Best Practices

Use TTL aggressively: Real-time data is ephemeral. Set expiration on keys (e.g., 2x the window size) to prevent memory bloat.
Choose the right eviction policy: If Redis runs out of memory, set maxmemory-policy allkeys-lru to evict least recently used keys.
Batch writes with pipelines: When recording many events, use client.multi() or pipeline to reduce round trips.
Avoid hot keys: For high-traffic pages, shard keys by time or use multiple keys (e.g., pageviews:home:2025-03-27:14 instead of a single key).
Use Redis Streams for reliable event ingestion: Consumer groups with acknowledgements ensure no data loss during crashes.
Monitor memory and CPU: Use INFO command and Redis Dashboard to catch growing memory or slow queries.
Consider approximate data structures: HyperLogLog (unique counts) and Bloom filters (membership) trade a small error for huge memory savings.
Keep analytic keys separate from application cache keys: Use a different Redis database (SELECT) or instance to avoid interference.
Pre-aggregate where possible: Instead of storing every event, use sorted sets with minute-level buckets and periodically roll up into hourly/daily summaries.

Conclusion

Redis provides a powerful, lightweight foundation for building real-time analytics systems. By leveraging its specialized data structures—Sorted Sets for time-window counts, HyperLogLog for unique approximations, Bitmaps for boolean tracking, and Streams for event ingestion—you can implement dashboards, live counters, and user activity monitors with minimal infrastructure. The key is to design for ephemeral data, choose appropriate eviction strategies, and use batching and pipelining to maximize throughput. With the code examples and best practices outlined here, you can start building your own real-time analytics layer that scales gracefully under high load.