What is Erlang for System Programming?
Erlang is a functional, concurrent programming language originally designed by Ericsson for building large-scale, fault-tolerant telecommunications systems. When we talk about Erlang for system programming, we refer to using Erlang's unique runtime model — lightweight processes, message passing, immutable data, and supervision trees — to build the foundational infrastructure that keeps services running: databases, message brokers, protocol handlers, monitoring agents, and distributed coordination layers.
Unlike C or Rust, Erlang does not give you raw pointer access or zero-cost memory management. Instead, it gives you something arguably more valuable for distributed systems: a process-based concurrency model where failure is compartmentalized, state is isolated, and the entire system can heal itself at runtime. System programming in Erlang means constructing the skeleton of a resilient software organism, not just writing code that talks to hardware.
Why Erlang Matters for System Programming
🚀 Deploy your AI agent in 10 minutes
Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.
Try it free →Modern systems are distributed, concurrent, and must tolerate failure. Traditional system programming languages excel at squeezing out performance but often leave fault tolerance and hot code upgrades as afterthoughts. Erlang inverts this priority:
- Lightweight processes — spawning hundreds of thousands of concurrent activities without OS thread overhead
- Preemptive scheduling — the BEAM VM gives every process its fair share of CPU time, preventing starvation
- Process isolation — a crash in one process cannot corrupt another's memory
- Supervision trees — structured error recovery that restarts failed components automatically
- Hot code upgrades — swap running code without downtime, essential for systems that must never stop
- Built-in distribution — nodes communicate transparently across machines with the same message-passing semantics
- Immutable data — eliminates whole categories of concurrency bugs at the language level
For system programmers coming from C, Go, or Rust, Erlang offers a radically different mental model: instead of protecting shared state with locks, you eliminate shared state entirely. Each process owns its data, and communication happens solely through asynchronous message passing.
Core Concepts You Must Understand
The Process as the Unit of Computation
In Erlang, a process is not an OS thread. It is a lightweight abstraction managed by the BEAM virtual machine. Each process has its own mailbox, heap, and stack. Creating one costs microseconds and a few kilobytes of memory. This allows architectures where every connection, every timer, every background task runs in its own isolated process.
%% Spawn a process and capture its PID
Pid = spawn(fun() ->
receive
{From, ping} ->
From ! pong
end
end),
%% Send a message
Pid ! {self(), ping},
%% Wait for the reply
receive
pong -> io:format("Got pong back!~n")
end.Message Passing and the Mailbox
Every process has a mailbox — a FIFO queue of incoming messages. The receive expression pattern-matches against messages, blocking until a matching pattern arrives. This is the foundation of all synchronization in Erlang. There are no mutexes, no condition variables, no channels with complex semantics — just pattern matching on a queue.
%% A simple state machine as a process
-module(connection).
-export([start/0]).
start() ->
spawn(fun() -> loop(#{}) end).
loop(State) ->
receive
{connect, ClientId} ->
NewState = maps:put(ClientId, connected, State),
io:format("Client ~p connected~n", [ClientId]),
loop(NewState);
{disconnect, ClientId} ->
NewState = maps:remove(ClientId, State),
io:format("Client ~p disconnected~n", [ClientId]),
loop(NewState);
{get_state, From} ->
From ! {state, State},
loop(State);
stop ->
io:format("Shutting down~n"),
ok
end.Supervision Trees
The supervision tree is Erlang's answer to defensive programming. Instead of writing try/catch blocks everywhere, you structure your system as a hierarchy of workers and supervisors. When a worker crashes, its supervisor decides what to do: restart it, restart all siblings, or escalate the failure upward. This creates self-healing systems where transient failures are recovered automatically.
-module(my_supervisor).
-behaviour(supervisor).
-export([start_link/0, init/1]).
start_link() ->
supervisor:start_link({local, ?MODULE}, ?MODULE, []).
init([]) ->
%% Child specification: restart strategy, module, args
Children = [
#{id => worker1,
start => {worker_module, start_link, []},
restart => permanent,
shutdown => 5000,
type => worker,
modules => [worker_module]}
],
%% Strategy: one_for_one restarts only the crashed child
{ok, {{one_for_one, 5, 10}, Children}}.Building a TCP Server from Scratch
Let's build a real system programming example: a non-blocking TCP server that handles thousands of concurrent connections, each in its own Erlang process. This demonstrates how Erlang naturally maps network programming onto its process model.
-module(tcp_server).
-export([start/1, acceptor/1]).
start(Port) ->
%% Start the acceptor process
spawn(?MODULE, acceptor, [Port]).
acceptor(Port) ->
{ok, ListenSocket} = gen_tcp:listen(Port, [
binary,
{packet, 0},
{active, false},
{reuseaddr, true}
]),
io:format("Listening on port ~p~n", [Port]),
accept_loop(ListenSocket).
accept_loop(ListenSocket) ->
{ok, Socket} = gen_tcp:accept(ListenSocket),
%% Spawn a dedicated process for this connection
Pid = spawn(fun() -> handle_client(Socket) end),
%% Transfer ownership of the socket to the handler process
gen_tcp:controlling_process(Socket, Pid),
%% Immediately accept the next connection
accept_loop(ListenSocket).
handle_client(Socket) ->
%% Set the socket to active mode so we receive data as messages
inet:setopts(Socket, [{active, once}]),
receive
{tcp, Socket, Data} ->
io:format("Received: ~p~n", [Data]),
%% Echo back
gen_tcp:send(Socket, Data),
%% Reactivate for next message
inet:setopts(Socket, [{active, once}]),
handle_client(Socket);
{tcp_closed, Socket} ->
io:format("Client disconnected~n"),
ok;
{tcp_error, Socket, Reason} ->
io:format("Error: ~p~n", [Reason]),
ok
end.This server never blocks on a single connection. The acceptor loop spawns a new process for each client and immediately returns to accepting. Each handler process owns its socket exclusively — no shared state, no connection pool to manage, no thread starvation. The BEAM scheduler preemptively switches between thousands of these handler processes, giving each a fair slice of CPU time.
Distributed System Programming with Erlang Nodes
Erlang's distribution model treats remote processes almost identically to local ones. Two Erlang runtime instances (nodes) connect via a shared cookie, and once connected, you can spawn processes on remote nodes and send messages across the network transparently.
%% Start two nodes on the same machine for testing:
%% erl -sname node1@localhost -setcookie secret
%% erl -sname node2@localhost -setcookie secret
%% On node1@localhost:
%% Connect to node2
net_kernel:connect('node2@localhost').
%% Spawn a process on node2
RemotePid = spawn('node2@localhost', fun() ->
receive
{From, ping} -> From ! {self(), pong}
end
end).
%% Send a message across the network
RemotePid ! {self(), ping}.
receive
{RemotePid, pong} -> io:format("Got pong from remote node!~n")
end.For system programming, this means you can build distributed message buses, replicated state stores, and cluster-wide supervision without reaching for external libraries like Zookeeper or etcd — though those have their place. Erlang's native distribution is sufficient for many system-level coordination tasks.
Hot Code Upgrades: Deploy Without Downtime
Few features distinguish Erlang for system programming more than hot code upgrades. You can load new versions of a module into a running system without stopping processes. The BEAM VM keeps two versions of each module in memory; processes running the old version continue until they make an external call, at which point they transparently transition to the new code.
-module(upgrade_demo).
-export([start/0, loop/1]).
start() ->
spawn(fun() -> loop(0) end).
loop(Count) ->
receive
increment ->
loop(Count + 1);
{get, From} ->
From ! {count, Count},
loop(Count)
end.
%% To upgrade at runtime:
%% 1. Compile the new version
%% 2. Load it into the running system:
%% code:purge(upgrade_demo),
%% code:load_file(upgrade_demo).
%% 3. Existing process continues; next external call uses new codeFor systems that must run continuously — telecom switches, financial matching engines, IoT gateways — this capability is revolutionary. You fix bugs, add features, and optimize performance without scheduling maintenance windows.
Working with C via NIFs and Ports
Sometimes you need raw performance or access to OS-specific functionality. Erlang offers two escape hatches for system programming: Native Implemented Functions (NIFs) for synchronous, fast operations, and Ports for asynchronous communication with external OS processes.
NIFs: Inline Native Code
/* nif_example.c — compile as a shared library */
#include <erl_nif.h>
static ERL_NIF_TERM fast_hash(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) {
ErlNifBinary bin;
if (!enif_inspect_binary(env, argv[0], &bin)) {
return enif_make_badarg(env);
}
/* Simple FNV-style hash */
unsigned long hash = 5381;
for (size_t i = 0; i < bin.size; i++) {
hash = ((hash << 5) + hash) + bin.data[i];
}
return enif_make_uint64(env, hash);
}
static ErlNifFunc nif_funcs[] = {
{"fast_hash", 1, fast_hash}
};
ERL_NIF_INIT(nif_example, nif_funcs, NULL, NULL, NULL, NULL)-module(nif_example).
-export([fast_hash/1]).
-on_load(init/0).
init() ->
erlang:load_nif("./nif_example", 0).
fast_hash(_Binary) ->
erlang:nif_error("NIF not loaded").NIFs run on the scheduler thread, so they must return within a millisecond. For longer operations, use dirty NIFs (scheduled on dedicated dirty scheduler threads) or Ports.
Ports: External Process Communication
%% Start an external C program as a port
start_port() ->
Port = open_port({spawn, "./external_worker"}, [
{packet, 4},
binary,
use_stdio
]),
%% Send data to the external process
Port ! {self(), {command, <<"do_work">>}},
%% Receive result
receive
{Port, {data, Result}} ->
io:format("Port returned: ~p~n", [Result])
end.Ports communicate via standard I/O and are truly isolated — if the external process crashes, the Erlang port receives a close signal but the VM continues running. This is ideal for wrapping unsafe C libraries or interfacing with hardware.
ETS and Mnesia: In-Memory Storage for System State
System programming often requires fast, shared lookup tables. Erlang provides ETS (Erlang Term Storage) — an in-memory key-value store with support for ordered_set, set, bag, and duplicate_bag table types. ETS tables live in the VM's memory and can be accessed by any process with the table reference.
%% Create an ETS table for a connection registry
Table = ets:new(connections, [
named_table,
public,
set,
{keypos, 1},
{write_concurrency, true},
{read_concurrency, true}
]).
%% Insert a record
ets:insert(connections, {client_42, {ip, "10.0.1.42"}, {port, 5678}}).
%% Lookup
[{client_42, {ip, IP}, {port, Port}}] = ets:lookup(connections, client_42).Mnesia builds on ETS and disk_log to provide a distributed, replicated database with transaction support — all within the Erlang VM. For system-level services like cluster membership, configuration storage, or service discovery, Mnesia eliminates external database dependencies.
%% Create a Mnesia schema and table
mnesia:create_schema([node()]),
mnesia:start(),
mnesia:create_table(service_registry, [
{attributes, record_info(fields, service)},
{ram_copies, [node()]},
{local_content, true}
]).
%% Transactional insert
mnesia:transaction(fun() ->
mnesia:write(#service{name = auth, host = "10.0.0.1", port = 8080})
end).Best Practices for Erlang System Programming
1. Let It Crash
Don't write defensive code that handles every conceivable error inline. Instead, design processes that fail cleanly and rely on supervisors to restore them. A process handling a malformed message should crash and let its supervisor spawn a fresh instance. This keeps error-handling logic centralized and the system predictable.
%% Good: crash on unexpected input
handle_message({valid, Data}) ->
process(Data);
handle_message(Unexpected) ->
%% Crash deliberately — supervisor will restart
error({unexpected_message, Unexpected}).
%% Bad: swallowing errors
handle_message(_Anything) ->
try process(_Anything) catch _ -> ok end,
%% Now the process is in an unknown state — dangerous!
ok.2. Keep Process State Minimal
A process should hold only the state it genuinely needs. Large state bloats process memory and slows garbage collection. Offload bulk data to ETS tables or external stores, keeping the process as a coordinator rather than a data bucket.
3. Use OTP Behaviours Religiously
The OTP framework (gen_server, gen_statem, supervisor, application) provides battle-tested abstractions. Don't roll your own process loops when gen_server gives you standardized call/cast/info handling, timeout management, and system messages for free.
-module(my_service).
-behaviour(gen_server).
%% API
-export([start_link/0, get_value/0, set_value/1]).
%% gen_server callbacks
-export([init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2]).
start_link() ->
gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
init([]) ->
{ok, #{value => 0}}.
get_value() ->
gen_server:call(?MODULE, get).
set_value(V) ->
gen_server:cast(?MODULE, {set, V}).
handle_call(get, _From, State) ->
{reply, maps:get(value, State), State}.
handle_cast({set, V}, State) ->
{noreply, State#{value => V}}.
handle_info(_Msg, State) ->
{noreply, State}.
terminate(_Reason, _State) ->
ok.4. Monitor and Link Strategically
Use monitor for one-way observation (you want to know if a process dies without affecting it) and link for bidirectional fate-sharing (if either process dies, the other gets notified). Choose based on whether the relationship is supervisory or observational.
%% Monitor: get notified when target dies, but don't affect it
Ref = monitor(process, TargetPid),
receive
{'DOWN', Ref, process, TargetPid, Reason} ->
io:format("Target died: ~p~n", [Reason])
end.
%% Link: bidirectional — if either dies, the other receives exit signal
link(TargetPid),
%% If TargetPid crashes with reason normal, we receive: {'EXIT', TargetPid, normal}5. Profile Before Optimizing
The BEAM VM ships with powerful profiling tools: fprof, eprof, cprof, and the observer GUI. Before writing NIFs or restructuring your architecture, profile to find actual bottlenecks. Often the culprit is not CPU but message queue buildup or excessive process creation.
%% Profile a function call
fprof:apply(fun() -> my_module:expensive_operation() end, []),
fprof:profile(),
fprof:analyse({dest, file, "profile_results.txt"}).6. Handle Backpressure in Message-Passing Systems
When a process mailbox grows faster than it can be processed, the system eventually runs out of memory. Implement backpressure by monitoring mailbox size and applying load shedding or flow control.
%% Check mailbox length before processing
check_backpressure() ->
case process_info(self(), message_queue_len) of
{message_queue_len, Len} when Len > 1000 ->
%% Apply backpressure: slow down or drop messages
timer:sleep(100),
check_backpressure();
_ ->
ok
end.7. Test for Concurrency Bugs with Concuerror
Traditional unit tests miss race conditions. Use tools like Concuerror or stateful property-based testing to systematically explore interleavings. Erlang's deterministic scheduling within a single VM makes this surprisingly tractable.
Real-World System Programming Use Cases
- RabbitMQ — a message broker written entirely in Erlang, handling millions of concurrent connections with per-connection processes and supervision trees for queue reliability
- WhatsApp — the backend infrastructure uses Erlang for its messaging core, leveraging hot code upgrades to push changes to thousands of nodes without downtime
- CouchDB — a document database where Erlang manages the storage engine, view indexing, and cluster coordination
- Elixir's Phoenix Framework — while Elixir is a different language, it runs on the same BEAM VM, demonstrating that Erlang's system programming primitives power modern web frameworks handling WebSocket floods
- EMQX — an MQTT broker for IoT that handles 100K+ concurrent device connections on modest hardware, thanks to Erlang's process-per-connection model
Getting Started: Your First System Service
Let's build a complete, supervised system service — a simple key-value store with TCP interface, supervision, and proper shutdown. This ties together everything we've covered.
%% kv_store.erl — the application module
-module(kv_store).
-behaviour(application).
-export([start/2, stop/1]).
start(_Type, _Args) ->
%% Start the top-level supervisor
case kv_sup:start_link() of
{ok, Pid} -> {ok, Pid};
Other -> Other
end.
stop(_State) ->
ok.%% kv_sup.erl — the supervisor
-module(kv_sup).
-behaviour(supervisor).
-export([start_link/0, init/1]).
start_link() ->
supervisor:start_link({local, ?MODULE}, ?MODULE, []).
init([]) ->
Children = [
#{id => kv_server,
start => {kv_server, start_link, []},
restart => permanent,
shutdown => 5000,
type => worker,
modules => [kv_server]},
#{id => kv_tcp_listener,
start => {kv_tcp_listener, start_link, [9900]},
restart => permanent,
shutdown => 5000,
type => worker,
modules => [kv_tcp_listener]}
],
{ok, {{one_for_one, 5, 10}, Children}}.%% kv_server.erl — the core state process (gen_server)
-module(kv_server).
-behaviour(gen_server).
-export([start_link/0, get/1, put/2, delete/1, stop/0]).
-export([init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2]).
start_link() ->
gen_server:start_link({local, ?MODULE}, ?MODULE, [], []).
init([]) ->
%% Initialize with empty ETS table for persistence
ets:new(kv_data, [named_table, public, set, {keypos, 1}]),
{ok, #{}}.
get(Key) ->
gen_server:call(?MODULE, {get, Key}).
put(Key, Value) ->
gen_server:call(?MODULE, {put, Key, Value}).
delete(Key) ->
gen_server:call(?MODULE, {delete, Key}).
stop() ->
gen_server:cast(?MODULE, stop).
handle_call({get, Key}, _From, State) ->
Result = case ets:lookup(kv_data, Key) of
[{Key, Value}] -> {ok, Value};
[] -> {error, not_found}
end,
{reply, Result, State};
handle_call({put, Key, Value}, _From, State) ->
ets:insert(kv_data, {Key, Value}),
{reply, ok, State};
handle_call({delete, Key}, _From, State) ->
ets:delete(kv_data, Key),
{reply, ok, State}.
handle_cast(stop, State) ->
{stop, normal, State}.
handle_info(_Msg, State) ->
{noreply, State}.
terminate(_Reason, _State) ->
ets:delete(kv_data),
ok.%% kv_tcp_listener.erl — TCP frontend
-module(kv_tcp_listener).
-export([start_link/1]).
start_link(Port) ->
{ok, spawn_link(fun() -> listen(Port) end)}.
listen(Port) ->
{ok, LSock} = gen_tcp:listen(Port, [
binary, {packet, 4}, {active, false}, {reuseaddr, true}
]),
accept_loop(LSock).
accept_loop(LSock) ->
case gen_tcp:accept(LSock) of
{ok, Sock} ->
spawn(fun() -> handle(Sock) end),
accept_loop(LSock);
{error, closed} -> ok;
{error, Reason} ->
io:format("Accept error: ~p~n", [Reason]),
accept_loop(LSock)
end.
handle(Sock) ->
case gen_tcp:recv(Sock, 0) of
{ok, Data} ->
%% Parse: <<"GET key">> or <<"PUT key value">>
Response = process_command(Data),
gen_tcp:send(Sock, Response),
handle(Sock);
{error, closed} ->
ok;
{error, Reason} ->
io:format("Recv error: ~p~n", [Reason]),
ok
end.
process_command(<<"GET ", Key/binary>>) ->
case kv_server:get(Key) of
{ok, Value} -> <<"OK ", Value/binary, "\n">>;
{error, _} -> <<"NOTFOUND\n">>
end;
process_command(<<"PUT ", Rest/binary>>) ->
[Key, Value] = binary:split(Rest, <<" ">>),
kv_server:put(Key, Value),
<<"OK\n">>;
process_command(<<"DELETE ", Key/binary>>) ->
kv_server:delete(Key),
<<"OK\n">>;
process_command(_) ->
<<"ERROR\n">>.This complete example demonstrates the layered architecture of Erlang system programming: an application module starts a supervisor, which manages a gen_server for business logic and a TCP listener for network access. If the listener crashes, the supervisor restarts it. If the server crashes, ETS data might be lost (in this simple example), but the supervisor brings it back cleanly. The entire system shuts down gracefully when the application stops.
Conclusion
Erlang for system programming represents a fundamentally different approach to building infrastructure software. Rather than fighting concurrency with locks and shared memory, Erlang embraces isolation and message passing. Rather than writing exhaustive error handlers, it structures systems as supervision hierarchies that heal themselves. The result is code that maps naturally to the problem domain of distributed, fault-tolerant services — code that runs for years without restart, accepts updates while running, and scales across cores and machines with minimal ceremony.
For developers accustomed to C, Go, or Rust, the shift requires letting go of micro-control in favor of macro-resilience. You trade pointer arithmetic for process arithmetic, mutexes for mailboxes, and manual memory management for garbage-collected per-process heaps. The payoff is a system programming toolkit uniquely suited to the challenges of the modern, always-on, globally distributed software landscape. Whether you're building the next great message broker, an IoT platform handling millions of devices, or the control plane for a distributed database, Erlang's primitives — processes, supervision, hot upgrades, and transparent distribution — deserve a prominent place in your engineering arsenal.