Menu

Distributed Systems / Consensus Engineering

RaftNova

The point of the system is not simply a distributed store. It is to implement consensus correctly — Figure 8, WAL persistence, votedFor durability, and pipelined replication — and to make every invariant visible and testable.

RaftNova is a correctness-first distributed key-value store implementing the complete Raft consensus algorithm in Go without any consensus library. Five nodes coordinate via gRPC, persist state through a WAL with CRC32-IEEE integrity, replicate log entries with etcd-style pipelining, and expose a React 19 dashboard that visualizes election state, replication lag, and cluster topology in real time.

GoRaftgRPCWALProtobufReactD3

Pipelined Throughput

~7,400

ops/sec — etcd-style concurrent pipeline, 3-node cluster

Sequential P50

2.1ms

WAL fsync + gRPC round-trip + majority ACK, 3-node

Cluster Size

5 nodes

tolerates 2 simultaneous failures with 3-node majority

The Core Problem

Distributed agreement is not hard. Correct distributed agreement is.

Most distributed KV stores hide the hard part behind library calls. RaftNova implements the full Raft paper from scratch — election, log replication, WAL durability, crash recovery — with every correctness invariant enforced in code, not assumed by the framework.

The three canonical failure modes that make consensus hard: split votes from correlated election timeouts, log divergence after partition, and the Figure 8 problem where a previous term's entry can be silently overwritten after leader change. All three are addressed explicitly.

Pipelined replication (etcd-style) means WAL fsync and gRPC replication happen concurrently on the leader — not sequentially. This is what separates ~7,400 ops/s from ~1,200 ops/s under concurrent load.

Naive implementation

~1,200 ops/s

sequential fsync + replicate

RaftNova (pipelined)

~7,400 ops/s

etcd-style concurrent pipeline

RaftNova system architecture — 5-node Raft cluster with gRPC replication, WAL, and KV store
Expand Full Detail

Five nodes in a Raft cluster. Each node runs the full stack: HTTP API, Raft consensus engine, WAL, and in-memory KV store. The dashboard polls /status at 250ms to build the live cluster view.

Architecture

System Components

Seven files. Each owns a precise correctness boundary. Each has a defined failure mode.

RaftNova full architecture — 5-node pentagon, gRPC links, HTTP API, WAL, KV store layers
Expand Full Detail

Full system: Client → HTTP API (307 redirect for non-leaders) → Raft Consensus → WAL (CRC32 + fsync) → KV Store state machine. Dashboard polls /status at 250ms intervals.

FileResponsibility
node.goRaft state machine — election, replication, commit, apply
replication.goPipelined follower sync, matchIndex/nextIndex bookkeeping, commitIndex advance
wal.goAppend-only persistence with CRC32 integrity and fsync
log.go1-indexed in-memory Raft log with truncation
http.goREST API — PUT/GET/DELETE /kv, /status, /healthz, /simulate-crash
grpc.goLazy-connect gRPC transport with connection pooling
store.goIn-memory map[string]string state machine with Apply()

Write Path

The ordering is not arbitrary — violating it loses data.

Every write follows this exact 6-step pipeline. Steps 3 and 4 run concurrently — this is the etcd-style pipelining that drives the throughput difference.

01
Client PUT → HTTP API

Non-leader nodes return 307 Redirect to leader's address. Only leader proceeds.

Redirect boundary →
02
SubmitCommand(entry)

Entry appended to in-memory log. Assigned log index and current term.

Log append →
03
WAL.Append(entry) + fsync()

Protobuf-encoded entry written with CRC32-IEEE checksum. fsync() before replication begins.

Durability record →
04
SendAppendEntries (parallel)

Pipelined gRPC replication to all followers simultaneously. Does not wait for WAL on followers before sending.

Replication fork →
05
Majority ACK → advanceCommitIndex()

Once ⌈N/2⌉+1 peers confirm matchIndex >= entry.Index, commitIndex advances. Term check (Figure 8) enforced here.

Commit point →
06
applyLoop() → KV Store.Apply()

Background goroutine applies committed entries to in-memory map. lastApplied catches up to commitIndex.

State machine →

The Lag Metric

commitIndex − lastApplied is the observable replication lag. The dashboard surfaces this per-node in real time. A lag above zero means the state machine is catching up to the committed log. A sustained lag on a follower indicates a slow disk or network issue.

RaftNova write path — 8-step consensus pipeline from Client PUT to 200 OK
Expand Full Detail

The 8-stage consensus pipeline: Client → HTTP (307 redirect) → Append to Log → WAL Write + gRPC Replicate (parallel) → Majority ACK → Advance commitIndex → Apply to KV Store → 200 OK.

RaftNova leader election state machine — Follower, Candidate, Leader transitions
Expand Full Detail

Three-state FSM: FOLLOWER → CANDIDATE on election timeout → LEADER on majority votes. Any node discovering a higher term immediately reverts to FOLLOWER. Candidate → FOLLOWER on higher term prevents split-brain.

Leader Election

One mechanism prevents all split-brain scenarios.

FOLLOWER

Waiting state — election timer running (150–300ms randomized)

Resets timer on any valid AppendEntries or RequestVote

CANDIDATE

Increments term, votes for self, sends RequestVote to all peers

Moves to LEADER on majority, back to FOLLOWER on higher term

LEADER

Sends heartbeats every 50ms. Accepts client writes. Replicates log.

On election: submits no-op entry (§5.4.2) to commit pending entries

Split Vote Prevention

Randomized election timeouts (150–300ms uniform distribution) make correlated timeouts statistically improbable. The first node to time out has a head start on gathering votes.

No-Op on Election

A new leader immediately submits a no-op log entry in its own term. This forces the Figure 8 term check to pass, committing any pending entries from the previous leader's term.

WAL Format

Every acknowledged byte is on disk. No exceptions.

The WAL uses a binary record format: 4-byte record type (LE uint32) + 4-byte payload length + Protobuf-encoded payload + 4-byte CRC32-IEEE checksum. The CRC32 is computed over the full header + payload.

On replay after a crash, any record with a mismatched CRC32 halts replay at that point. The log self-heals to the last consistent record — partial writes at the tail are silently truncated.

Record Layout

[RecordType: 4 bytes LE uint32  ]
[PayloadLen: 4 bytes LE uint32  ]
[Payload:    N bytes Protobuf   ]
[CRC32:      4 bytes LE uint32  ]
 ↑ computed over all three above

Record Types

0x01  LogEntry   { term, index, command }
0x02  State      { currentTerm, votedFor }

File rotation at 64 MB threshold
→ wal-000001.log, wal-000002.log, ...

Durability

fsync() after every record

Batch Optimization

AppendMany() — single fsync for N entries

RaftNova WAL record binary format — RecordType, PayloadLength, Protobuf payload, CRC32
Expand Full Detail

WAL binary record: 12 bytes fixed header + N bytes Protobuf payload. CRC32-IEEE covers all fields. fsync() after every write. File rotation at 64 MB.

Correctness Invariants

A crash at any point leaves a consistent state.

These invariants are enforced in code with inline comments at the enforcement site — not in documentation that diverges from the implementation.

System Guarantees

currentTerm only increasesMonotonic term ensures stale leaders step down immediately on any RPC contact
votedFor persisted before vote replyPrevents double-voting across crashes — written to WAL before RequestVoteResponse is sent
Committed entries never deletedLog truncation only removes entries with index > commitIndex
Figure 8 term check on commitadvanceCommitIndex() skips any entry whose term != currentTerm — previous terms commit indirectly
No-op on electionNew leader submits a nil command in currentTerm, forcing prior-term entries to commit safely
Reads only from leaderFollowers return 307 redirect. linearizable reads guaranteed by leader exclusivity
matchIndex only increasesOnce a follower confirms an entry, that entry is never unconfirmed — matchIndex is monotone
WAL fsync before ackNo client ever receives 200 OK for a write whose WAL record is not on disk

Engineering Honesty

5 Real Bugs Found, Fixed, and Documented

These are actual bugs discovered during development, in chronological order. Not hypothetical edge cases. Not textbook examples. Real failures that surfaced under real conditions — some of which took hours to trace because the symptom pointed to the wrong module.

LivenessBug 1 of 5

Election Timer Not Resetting on AppendEntries

Symptom

Under write load, followers would initiate elections even while receiving valid heartbeats from the elected leader. The cluster entered repeated election storms, stalling all writes.

Root Cause

The AppendEntries RPC handler processed log entries and responded correctly — but forgot to call resetElectionTimer(). The follower's election countdown ran to zero despite receiving valid leader contact.

Fix

One call to resetElectionTimer() at the top of the AppendEntries handler. The fix is one line. The root-cause hunt took two hours because the symptom (split-vote storms) looked identical to a vote-counting bug.

Fix
// BEFORE: timer never reset on valid heartbeats
func (n *Node) handleAppendEntries(req *pb.AppendEntriesRequest) *pb.AppendEntriesResponse {
  // ... process entries ...
  return &pb.AppendEntriesResponse{Success: true}
}

// AFTER: one line fixes liveness
func (n *Node) handleAppendEntries(req *pb.AppendEntriesRequest) *pb.AppendEntriesResponse {
  n.resetElectionTimer() // ← this line was missing
  // ... process entries ...
  return &pb.AppendEntriesResponse{Success: true}
}
LESSON

In Raft, the election timer is the heartbeat of a follower's trust in the leader. Missing a single reset breaks the entire liveness guarantee — the cluster works until it doesn't, then fails visibly under load.

SafetyBug 2 of 5

Double-Counting Votes from Duplicate RPC Replies

Symptom

Occasionally a node declared itself leader after receiving what it believed was a majority vote — but only 2 distinct peers had actually voted for it. The cluster briefly had two leaders.

Root Cause

When a RequestVote RPC timed out, the candidate retried. The original reply then arrived after the retry's reply — both replies were counted, inflating the vote total for that peer.

Fix

Added a votesReceived map[NodeID]bool to the election goroutine. Only the first response from each peer is counted. Subsequent responses for the same peer are discarded.

Fix
// BEFORE: duplicate replies double-counted
votesGranted := 1 // self-vote
for reply := range replyChan {
  if reply.VoteGranted {
    votesGranted++
  }
}

// AFTER: track per-peer, count once
votesGranted := 1
votesReceived := map[string]bool{}
for reply := range replyChan {
  if votesReceived[reply.PeerID] { continue }
  votesReceived[reply.PeerID] = true
  if reply.VoteGranted {
    votesGranted++
  }
}
LESSON

The network can deliver duplicate messages. Idempotency is not optional in distributed systems — every counter that affects a safety property must be deduplicated at the source.

CorrectnessBug 3 of 5

WAL Replay Not Restoring votedFor

Symptom

After a crash and restart, a node voted for two different candidates in the same term — directly violating Raft's election-safety property. In theory, two leaders could have been elected.

Root Cause

WAL replay on startup correctly reconstructed currentTerm from persisted state records. It did not reconstruct votedFor. The node restarted believing it had never voted in any term.

Fix

The WAL replay path now fully restores both currentTerm and votedFor atomically from the same state record. Both fields are written together on every state change.

Fix
// BEFORE: replay restored term but not votedFor
func (n *Node) replayWAL(entries []WALEntry) {
  for _, e := range entries {
    if e.Type == WALStateRecord {
      n.currentTerm = e.Term
      // votedFor never restored ← BUG
    }
  }
}

// AFTER: both fields restored together
func (n *Node) replayWAL(entries []WALEntry) {
  for _, e := range entries {
    if e.Type == WALStateRecord {
      n.currentTerm = e.Term
      n.votedFor = e.VotedFor // ← restored
    }
  }
}
LESSON

Every piece of Raft state that affects safety must survive a crash. Missing votedFor is not a performance bug — it is a correctness violation that can elect two leaders in the same term.

SafetyBug 4 of 5

commitIndex Advance Without Term Check (Figure 8)

Symptom

After a leader change, a key written under term 3 returned a stale value. The new leader had silently committed an old entry from term 3 without verifying it was safe to do so.

Root Cause

advanceCommitIndex() counted how many peers had matchIndex >= N, then committed. It did not check whether log[N].term == currentTerm. This violates the Raft paper's Figure 8 safety rule: a leader can only commit entries from the current term.

Fix

Added a term check inside advanceCommitIndex(): `if log.TermAt(idx) != currentTerm { continue }`. Entries from previous terms can only be committed indirectly — by committing a current-term entry that follows them.

Fix
// BEFORE: committed entries from any term
func (n *Node) advanceCommitIndex() {
  for idx := n.commitIndex + 1; idx <= n.log.LastIndex(); idx++ {
    if n.countMajority(idx) {
      n.commitIndex = idx // ← UNSAFE for old-term entries
    }
  }
}

// AFTER: only commit current-term entries
func (n *Node) advanceCommitIndex() {
  for idx := n.commitIndex + 1; idx <= n.log.LastIndex(); idx++ {
    if n.log.TermAt(idx) != n.currentTerm { continue } // ← Figure 8
    if n.countMajority(idx) {
      n.commitIndex = idx
    }
  }
}
LESSON

Figure 8 is the hardest part of the Raft paper for a reason. The term check is non-obvious, and violating it allows data to silently disappear: an entry replicated to a majority in term 3 can be overwritten by a term-4 leader who hasn't seen it.

DurabilityBug 5 of 5

nextIndex Initialization Truncating Ahead Followers

Symptom

After leader election, keys that had been written and acknowledged by the old leader were absent from the cluster. The new leader had silently deleted committed entries from followers.

Root Cause

New leaders initialize nextIndex[peer] = lastLogIndex + 1 based on their own log length. If a follower had a longer log than the new leader (having received replication before the leader crashed), the AppendEntries probe sent empty entries — and the follower's conflicting suffix was truncated.

Fix

AppendEntries truncation logic now only removes conflicting entries — those whose term does not match. Extra valid entries beyond the leader's current index are preserved. The leader discovers each follower's true state through the probe protocol.

Fix
// BEFORE: truncated valid follower entries
func applyAppendEntries(leaderPrevIdx int, entries []Entry) {
  n.log.Truncate(leaderPrevIdx + 1) // ← deleted committed entries
  n.log.Append(entries...)
}

// AFTER: only truncate conflicting entries
func applyAppendEntries(leaderPrevIdx int, entries []Entry) {
  for i, e := range entries {
    idx := leaderPrevIdx + 1 + i
    if idx < n.log.Length() && n.log.TermAt(idx) != e.Term {
      n.log.Truncate(idx) // only truncate at conflict
      break
    }
  }
  n.log.AppendIfMissing(entries...)
}
LESSON

Log length is not a proxy for log correctness. A newly elected leader must discover each follower's actual state through the AppendEntries consistency check, not assume all followers have exactly its own log.

Performance

Benchmarks — 100-byte values, 5-node cluster

Sequential mode pays the full fsync + gRPC round-trip latency per write. The latency floor is: WAL fsync (leader) + gRPC to 2 peers + WAL fsync (peers) + majority ACK — approximately 2.8ms on commodity hardware. Pipelined mode overlaps these costs across concurrent writes.

Benchmark Results

WorkloadClusterThroughputP50P99Note
Concurrent writes (pipelined)3-node~7,400 ops/sFull etcd-style pipelining
Sequential writes (no pipeline)3-node~1,200 ops/s2.1ms8.4msWAL + gRPC round-trip
Sequential writes (no pipeline)5-node~900 ops/s2.8ms11.2ms2 extra peers in quorum
01

The 2.8ms Sequential Write Floor

Durability over throughput

Sequential writes are hard-capped by the physical cost of two fsync() barriers (leader WAL + majority follower WALs) plus gRPC round-trip latency. 2.8ms P50 on a 5-node cluster is close to the hardware floor for unbatched synchronous consensus on commodity NICs. No amount of CPU tuning reduces this — the bottleneck is network + disk.

02

Pipelining Multiplies Throughput 6×

etcd-style concurrent pipeline

In pipelined mode, the leader sends the next AppendEntries before waiting for the previous one's ACK. WAL fsync and gRPC replication overlap across concurrent writes. 7,400 ops/s vs 1,200 ops/s is not a tuning win — it is a structural win from overlapping I/O that would otherwise serialize.

03

5-Node vs 3-Node Latency Cost

Quorum size matters

Adding two more nodes to the quorum increases P99 from 8.4ms to 11.2ms in sequential mode. This is the raw cost of requiring one additional network round-trip for majority. The leader must wait for 3 of 5 nodes vs 2 of 3. Each additional peer in the quorum path adds latency proportional to the slowest peer in the majority.

04

Read Latency: Zero Extra Hops

Leader-only reads

Reads are served directly from the leader's in-memory KV store. No disk reads. No quorum required. The cost is: non-leaders pay one 307 redirect round-trip. The leader serves the value from its local map in microseconds. This is intentional — linearizability at the cost of a redirect is a better tradeoff than stale reads from followers.

Design Tradeoffs

DecisionWhyCost
WAL fsync on every recordNo acknowledged write can be lost across crash~1-5ms per write; sequential throughput bounded by disk
Leader-only readsLinearizability — no stale reads possibleNon-leaders pay one 307 redirect round-trip
In-memory KV storeO(1) reads after apply; no disk on read pathState is volatile; WAL replay required on restart
Pipelined replicationOverlaps disk I/O and network I/O across concurrent writesOut-of-order ACKs require careful matchIndex bookkeeping
Single goroutine owns Raft stateEliminates data races; all invariants hold triviallyNo parallelism in consensus path; throughput bounded by one core
gRPC transportMultiplexed, streaming-ready, protobuf-typed RPCsConnection overhead; not ideal for sub-millisecond latency targets

Running It

How to Run RaftNova

The cluster ships with a React 19 + D3 live dashboard that visualizes election state, log replication lag, and cluster topology in real time — the same observability surface a production Raft cluster would need.

Live Demo

RaftNova Testing Arena

A deployed 5-node Raft cluster with the full observability dashboard. Write keys, kill nodes, trigger elections, observe replication lag — all live.

Launch Arena

Docker Compose (5-node cluster)

# Spin up the full 5-node cluster
docker-compose up --build -d

# Verify all nodes are healthy
curl http://localhost:8080/status
curl http://localhost:8081/status

# Nodes available at:
# HTTP: ports 8080–8084
# gRPC: ports 9090–9094

Build from Source

# Requires Go 1.22+ and protoc
go build -o bin/server ./cmd/server

# Run local 5-node cluster (PowerShell)
./start_servers.ps1

# Run tests
go test -v ./internal/...
go test -v -timeout 120s ./tests/chaos/...

# Dashboard
cd dashboard && npm install && npm run dev
# http://localhost:5173

HTTP API Reference

All requests to any node — non-leaders return 307 redirect.

Write a key

curl -X PUT http://localhost:8080/kv/mykey \
  -d '{"value": "hello"}'

Read a key (linearizable)

curl http://localhost:8080/kv/mykey
# → {"value": "hello"}

Cluster status

curl http://localhost:8080/status
# → {node_id, state, term, commit_index,
#    last_applied, peers, leader_id}

Chaos — crash simulation

curl -X POST http://localhost:8080/simulate-crash
# Triggers crash scenario for observability
curl http://localhost:8080/healthz  # liveness probe

Dashboard Panels

🔵

NodeCard

Per-node state chip: LEADER / FOLLOWER / CANDIDATE. Live term, commitIndex, lastApplied, and replication lag (commitIndex − lastApplied).

📈

CommitChart

Rolling 30-second sparkline of commit_index per node. Election events marked as vertical lines on the timeline.

🕸️

TopologyGraph

D3 force-directed graph of peer edges derived from status.peers. Lag severity visualized as edge color overlay.

✍️

WriteTester

Target-node selector + 307 redirect observability. Send writes, observe redirects to leader, verify replication.

💥

ChaosPanel

Trigger simulate-crash endpoint with configurable batch flood writes. Observe election and recovery in real time.

📡

Cluster Polling

All 5 nodes polled at 250ms via /status. Global cluster view assembled client-side from node responses.

📊

Lag Monitor

commitIndex − lastApplied surfaced per-node. Sustained lag indicates follower disk or network pressure.

🔁

Election Markers

Term changes detected via CommitChart delta. Visual event markers show when elections occur and which node wins.

Next Case

NullRing