A Bare-Metal, Homogeneous-Node Architecture for a Distributed Social Network

Design goals & principles

The architecture is governed by six principles. Every later decision is traceable to one of them.

P1 — Homogeneous commodity hardware

A single server SKU is procured in volume. Any machine can fill any role. This collapses procurement, sparing, capacity modeling, and firmware management into one problem instead of five, and it means a failed database node and a failed cache node are replaced from the same shelf of cold spares. The cost is a modest efficiency loss versus role-optimized SKUs; §2 explains why a balanced node makes that loss small.

P2 — Shared-nothing

No component is a singleton whose loss stops the system. There is no shared SAN, no primary-only database, no single load balancer. State is partitioned and replicated; the loss of any one owner is survivable by a peer.

P3 — Distributed at every layer

Sharding and replication are applied independently at each tier — edge, LB, app, cache, data, storage, stream, search — rather than relying on one tier to protect the others. A cache-tier failure must not become a database outage.

P4 — Failure domains are explicit

Disk < node < rack < colo. Every placement decision — replica sets, erasure-coded stripes, shard replicas — is aware of this hierarchy and spreads copies to tolerate the loss of a whole domain, not just a single machine.

P5 — Cattle, not pets

Machines are anonymous and interchangeable. They PXE-boot an immutable image, receive a role from the scheduler, and are drained and reimaged rather than hand-repaired. No machine holds unreplicated state that would make its loss expensive.

P6 — Cost discipline as a product requirement

Because the platform is funded by subscriptions rather than advertising (see the companion product brief), infrastructure cost is a first-class constraint, not an afterthought. Owned commodity iron at steady-state utilization, amortized over 4–5 years, sets a far lower and more predictable marginal cost per active user than metered cloud — which is what lets the subscription price stay low.

The reference node

Every physical machine is identical. Rather than optimize separate SKUs for compute, memory, and storage, we specify one balanced node: enough RAM to host cache and database working sets (and to feed a ZFS ARC), a flash tier of NVMe for hot data and metadata, a spinning-disk capacity tier for bulk media, and enough cores to run application containers. The scheduler then decides, per machine and per day, which of those capabilities to actually use.

Roughly 15 TB of NVMe flash plus ~288 TB of spinning disk and 256 GB of RAM per node — on the order of 300 TB raw per box — means the same machine can act as a flash-and-disk Ceph OSD host, a MySQL shard replica, or a Memcached instance without reconfiguration. Storage-heavy and compute-heavy services coexist on the fleet; the scheduler simply avoids co-locating two hot roles — a memory-hungry ZFS OSD and a cache or database instance, say — on the same machine. Where a workload genuinely needs a different shape, we prefer more identical nodes over a second SKU, trading a little per-U density for the operational simplicity of one part number.

Intra-colo network fabric

Within each colo, nodes are wired in a leaf-spine (Clos) fabric. Every rack has a top-of-rack (ToR) leaf switch; every leaf connects to every spine. This gives non-blocking, uniform east-west bandwidth — critical because a social workload is dominated by internal traffic: an application node fanning a single page render out to dozens of cache, database, and storage nodes.

• Routing: L3 to the ToR with BGP and ECMP. Each node has two 25 GbE uplinks to two leaves; the loss of a leaf halves a rack's bandwidth but never isolates it.
• Failure domain = rack: a rack is the smallest unit the placement layer treats as a correlated-failure boundary (shared ToR, shared PDU pair). Replica sets never place two copies in the same rack.
• Addressing: nodes get stable identities from the provisioning system, not from their physical slot, so a machine can be moved or reimaged without reconfiguring dependents.

Racks are provisioned in N+1: the fabric and power are sized so that one rack per row can be lost (or taken down for maintenance) with headroom to spare. Dual PSUs on A/B feeds mean a single PDU or feed failure drops no nodes.

Multi-colo topology & the quorum problem

The platform runs active-active across two production colocation facilities, A and B, chosen for geographic and network diversity (different metros, different transit providers, independent power grids). Users are steered to the nearest healthy colo; each colo holds a full, serving copy of the data.

We therefore add a witness site, W: a minimal footprint (a handful of R1 nodes or even leased capacity) that hosts no user-serving traffic and stores only consensus voters and metadata replicas. Quorum-bearing systems are deployed as 2 : 2 : 1 across A : B : W. Now the loss of any one site still leaves a majority (e.g., A + W = 3 of 5), so the survivors elect a leader and keep committing, while the third site's replicas simply rejoin when it returns. This is the classic three-availability-zone pattern, executed with one cheap arbiter instead of three equal data centers.

Replication & consistency

Most cross-colo replication is asynchronous: a write commits locally and streams to the peer within tens of milliseconds. This keeps user-facing latency low and tolerates the inter-colo RTT. The consequence is a small replication lag window and the possibility of conflicting concurrent writes to the same key in different colos. We handle this by partitioning write authority by shard: each data shard has a home colo that owns writes for its key range, and the peer serves reads and stands by to take over. Cross-shard reads are served locally from the replica. This gives read-anywhere locality with single-writer-per-shard simplicity, avoiding multi-master conflict resolution for the core social graph.

Homogeneous pool & the scheduler

Because the hardware is uniform, the fleet is presented to operators as a single pool of interchangeable capacity. A cluster scheduler — Kubernetes on bare metal, or HashiCorp Nomad for a lighter footprint — bin-packs services onto nodes, honors placement constraints (spread replicas across racks and colos), and reschedules workloads off any node that fails. This is what turns "distributed at every layer" from a diagram into a running property: fault tolerance is the scheduler continuously re-satisfying placement constraints as machines come and go.

Provisioning is netboot-first: a node powers on, PXE-boots an immutable OS image (managed with a tool such as Tinkerbell or MAAS), registers with the scheduler, and is handed work. There is no per-machine hand configuration to drift. Reimaging is the repair of first resort.

Traffic entry & load balancing

User traffic enters through three cooperating stages, all running on R1 nodes.

Stage 1 — Anycast + GeoDNS

A small set of public VIPs is announced by BGP anycast from both colos. The nearest colo, by network topology, naturally receives the user; if a colo withdraws its announcement (planned or on failure), traffic reconverges to the survivor within seconds. GeoDNS provides a coarser second layer of steering and a fast way to drain a site.

Stage 2 — L4 load balancing

Behind each VIP is a tier of L4 balancers using ECMP plus consistent hashing, in the style of Google's Maglev [5] — packet-forwarding data planes (XDP/eBPF) that spread connections across L7 proxies while keeping a given flow pinned to one backend. L4 nodes are stateless and horizontally scalable; ECMP across several of them means losing one reshuffles only its share of flows.

Stage 3 — L7 reverse proxy

An L7 tier (Envoy or HAProxy) terminates TLS, applies routing and rate limiting, and load-balances to application backends by health and locality. This is also where a web application firewall and per-endpoint circuit breaking live. Because every stage is a horizontally scaled pool with health checking, there is no single balancer to lose.

Stateless application tier

Application servers are stateless: any request can be served by any app instance, because no session or user state lives on the machine. Session and ephemeral state are externalized to the cache tier; durable state lives in the data and storage tiers. This is what makes the app tier trivially fault-tolerant — an instance dying mid-request costs one retry, and the scheduler starts a replacement elsewhere.

A single page render is a fan-out: the app node gathers a user's profile, their graph neighborhood, recent posts from followed accounts, notification counts, and media URLs, typically touching dozens of cache and data backends in parallel before composing a response. Two implications follow — the internal fabric must be fast and non-blocking (§3), and every backend call needs a tight timeout with a graceful degradation path (render the page with a stale or empty module rather than block on one slow shard).

Cache & social-graph serving

At social-network read volumes, the database is protected by a large distributed cache. We run two complementary caching layers.

Look-aside object cache

A fleet-wide Memcached/Redis tier holds rendered fragments, session data, and hot rows, keyed and distributed by consistent hashing so that adding or removing a cache node only remaps its neighborhood of keys. The scaling patterns here follow Facebook's published Memcache work [3]: lease tokens to prevent thundering-herd stampedes on a cold key, and careful invalidation on write. Hot keys are replicated across several cache nodes to spread load and survive a node loss.

Graph-aware serving layer

The social graph — users, edges (friend/follow), and the objects hung off them (posts, likes, comments) — is the core data structure and the hottest read path. We front the sharded database with a graph cache in the spirit of Facebook's TAO [1]: a write-through, read-optimized layer that understands objects and associations, serves the overwhelming majority of graph reads from memory, and keeps its cache coherent with the underlying shards. This is what lets a friend-of-friend or feed query hit RAM instead of fanning out to disk.

Data tier

The durable core — accounts, profiles, edges, posts, comments — is stored in a horizontally sharded relational database, sharded by user ID so that a given user's data and immediate graph co-locate on one shard. Both Facebook and MySpace scaled on sharded MySQL; the pattern is proven. Two implementation paths fit this architecture:

Path A — Sharded MySQL under Vitess (recommended baseline)

Vitess [7] sits in front of a fleet of MySQL instances and provides sharding, query routing, connection pooling, online resharding, and topology management. Each shard is a replica set — one primary plus replicas, spread across racks and colos — with automated failover (semi-synchronous replication so a committed write survives a primary loss). This path is mature, well-understood operationally, and keeps the familiar relational model.

Path B — Distributed SQL (Raft-native)

Alternatively, a Raft-based distributed SQL store — CockroachDB, TiDB, or YugabyteDB — makes multi-colo replication, sharding, and failover intrinsic properties of the database rather than bolted-on. Each range is a Raft group replicated 2:2:1 across A:B:W, so surviving a site loss is automatic. This is architecturally cleaner against P2/P3/P4, at the cost of a younger operational surface and the need to respect quorum latency on writes. It is also the path that most directly consumes the witness site from §4.

Either way, the invariants hold: shards are replicated across failure domains, writes for a shard are owned by one colo, no shard has a single point of failure, and the fleet's identical nodes host the replicas.

Object & media storage

Photos, video, and — given the MySpace-style music focus — audio tracks are the largest data class by volume and the most bandwidth-hungry to serve. Storing billions of small-to-medium blobs efficiently, and the petabytes of media behind them affordably, is its own problem — and the one tier where all-flash economics stop making sense. This tier is therefore explicitly temperature-tiered: a fast flash tier for hot and metadata-heavy access, and a dense spinning-disk tier, managed by ZFS, for the warm and cold bulk that dominates capacity.

A temperature hierarchy

Media access is heavily skewed: a small, recently-uploaded or currently-trending set is read constantly, while the long tail — most of the archive by volume — is touched rarely. We map that skew onto three physical tiers, all present on every R1 node:

• Tier 0 — RAM (ZFS ARC + app cache): the hottest working set and metadata in flight, served from memory.
• Tier 1 — NVMe flash: hot objects, the Haystack index, database WAL/DB, and — for the disk tier beneath it — ZFS metadata, small blocks, and the synchronous-write log. Everything latency- or IOPS-critical lives here.
• Tier 2 — HDD capacity: the warm and cold bulk — the long tail of photos and the mass of video and audio — at roughly one-fifth the cost per terabyte of flash.

Objects are written hot to flash and age down to the disk tier as their access temperature falls; a re-heated object is re-promoted or simply served from the delivery cache (below). Tiering is a placement policy over the object store, not a manual migration chore.

Substrate — Ceph / RADOS with device classes

The distributed substrate remains Ceph [4], but its OSDs now span both media. Ceph tags each OSD with a device class — nvme for the flash tier, hdd for the disk tier — and CRUSH rules target each pool at a class, so hot replicated pools and indexes land on flash while warm/cold pools land on disk. CRUSH still places data pseudo-randomly but deterministically, with full awareness of the failure hierarchy — spreading copies across racks and colos and self-healing when a disk or node dies, all without a central metadata bottleneck. The bulk warm/cold pools use erasure coding (an 8+3 stripe: 11 fragments, any 8 reconstruct the object) on the disk tier for roughly 1.4× storage overhead instead of 3× replication, spread so that losing a rack — or a colo — loses no object. This cross-node erasure code is the design's only bulk-redundancy layer.

The capacity tier — ZFS on spinning disk

Spinning disk is dense and cheap but slow at random access, and its silent-corruption rate matters when data sits untouched for years. We therefore manage each node's HDDs with ZFS [11], which turns a shelf of slow disks into a checksummed, flash-accelerated capacity pool:

• Flash-accelerated metadata & small IO. A mirrored slice of the node's NVMe serves as a ZFS special vdev — all filesystem metadata, and any block below a tunable threshold (e.g. 64 KB), are placed on flash automatically — alongside a SLOG for low-latency synchronous writes and an optional L2ARC read cache. The result is HDD $/TB with flash latency on exactly the operations spinning disk is worst at: metadata lookups, small-object reads, and sync commits.
• End-to-end integrity. Every block is checksummed and periodic scrubs catch bit-rot before it spreads. When ZFS detects a bad block on a single-disk OSD it cannot repair locally, it fails the read and Ceph reconstructs the object from another erasure fragment — detection local, repair cluster-wide.
• Compression & snapshots. Transparent lz4/zstd compression reclaims capacity at negligible CPU cost, and copy-on-write snapshots make cheap, consistent backup and zfs send replication targets.

Hot media — a Haystack-style blob store

For the hottest, most numerous small objects (profile photos, thumbnails), a generic object store spends too much metadata and too many IOPS per tiny file. Following Facebook's Haystack design [2], hot media are packed into large append-only volumes with a compact in-memory index, collapsing per-photo metadata to a few bytes and turning a read into a single seek. This runs as a service on the flash tier of the same nodes; as volumes cool they are demoted to the ZFS-backed disk tier, while Ceph erasure coding holds the durable copy throughout.

Delivery

In front of storage sits a caching/edge tier — Varnish/nginx caching nodes at each colo and, optionally, a handful of additional POPs — so that popular media is served from cache close to the user and never re-reads the durable store. This owned edge can be augmented by a commercial CDN for global reach without changing the origin design.

Event stream & feed fan-out

Writes — a post, a like, a follow — are appended to a distributed log (Apache Kafka, or Redpanda for a lighter footprint) whose partitions are replicated across nodes and colos. The log is the backbone for everything asynchronous: notifications, feed generation, search indexing, and analytics all consume from it, decoupled from the write path so a slow consumer never blocks a user.

The feed problem — hybrid fan-out

Generating each user's home feed is the signature hard problem of a social network, and it is a choice between two strategies:

• Fan-out-on-write (push): when a user posts, workers push the post reference into every follower's precomputed feed. Reads are then cheap. But a user with millions of followers creates a write storm.
• Fan-out-on-read (pull): a feed is assembled at read time by querying the accounts a user follows. Writes are cheap; reads for users following many accounts are expensive.

We use a hybrid: push for ordinary accounts (the common case, cheap reads), and pull for high-fan-out "celebrity" accounts whose posts are merged in at read time. This bounds both the write amplification of push and the read cost of pull. Feed workers consume the event log and write per-user feed lists into the cache tier, backed by the data tier.

Search & coordination

Search

User, post, and tag search runs on a distributed OpenSearch/Elasticsearch cluster — indices sharded and replicated across nodes and colos, fed from the event log so the index trails writes by seconds. Search is a read-scaled service that never sits on the primary write path.

Coordination & service discovery

Cluster membership, configuration, leader election, and service discovery run on a quorum store — etcd or ZooKeeper — deployed, like every other consensus system, as 2:2:1 across A:B:W. Service discovery and health checking (Consul, or the scheduler's native mechanism) let the stateless tiers find healthy backends without hardcoded addresses, which is what allows any node to be reimaged or moved without touching its dependents.

High-availability / fault-tolerance matrix

Each layer earns its resilience independently. The table summarizes the redundancy mechanism and the largest failure domain each layer survives without user-visible outage.

* Cache on colo loss Losing a colo cold-evicts that site's cache working set; the survivor serves correctly but with a temporary latency spike and elevated database load as caches refill. This is a performance event, not an outage — provided the data tier was sized to absorb the refill (see §14).

Capacity planning under colo loss

Active-active across two colos hides an uncomfortable arithmetic: to serve full load after losing one of two colos, each colo must be able to carry 100% of traffic — i.e., the pair runs at ≤ 50% utilization in steady state. That is effectively 2× the serving capacity you'd need if you never had to fail over.

Practical planning rules for this design:

• Serving tiers (app, LB, cache): provision so either colo alone meets peak. Accept the 50% steady-state ceiling, or run hotter and accept graceful degradation (shed non-essential modules) during a failover.
• Data tier: size the surviving colo's replicas to absorb both the redirected read traffic and the cache-refill surge; this is often the binding constraint after a colo loss.
• Storage: budget erasure-coding overhead (~1.4×) on the disk tier plus enough free space that a full node's fragments can rebuild onto survivors without filling any pool; size the NVMe special vdev for the metadata + small-block footprint (typically a low-single-digit percentage of raw capacity) so it never spills to disk; and note that resilvering a 24 TB HDD is hours of I/O — Ceph parallelizes the rebuild across many nodes, but the free-space and IOPS headroom must exist for it.
• Sparing: keep a cold-spare pool sized to expected concurrent failures plus lead time for replacements; because there is one SKU, this pool covers every tier at once.

Failure walkthroughs

A disk fails

Ceph detects the dead OSD, marks it out, and re-replicates or reconstructs its data from surviving fragments onto other nodes per the CRUSH map. No object becomes unavailable; the only effect is background rebuild I/O. The physical disk is swapped at leisure.

A node fails

The scheduler notices the missing node and reschedules its stateless workloads elsewhere within seconds. Any stateful role it held (a DB replica, a cache instance, OSDs) was already replicated; the data tier promotes a replica if it was a primary, and storage rebuilds the lost fragments. The node is reimaged or pulled and replaced with a cold spare — same part number.

A rack fails (ToR or PDU)

Because replica sets and erasure stripes never place a majority of copies in one rack, every affected shard and object still has a serving copy. Traffic and rebuild shift to other racks; capacity drops by one rack's worth until it returns. N+1 rack provisioning absorbs this.

A colo fails

The lost colo withdraws its anycast announcement; users reconverge to the survivor in seconds. Quorum systems retain a majority (survivor + witness = 3 of 5) and elect new leaders for any shards the dead colo owned. The survivor serves all traffic — cold caches refill under elevated DB load for a few minutes, then latency normalizes. This is a capacity and latency event, not an outage, exactly because §14's headroom exists.

The A–B link partitions (both colos up, cut off from each other)

The witness is the tiebreaker. Whichever production colo can still reach W forms a majority and keeps serving as the write authority; the isolated colo, unable to reach a majority, steps its quorum-bearing roles down to read-only rather than accept conflicting writes. When the link heals, the stepped-down side rejoins and catches up. This is precisely the scenario a two-site design cannot handle safely, and the reason the witness is non-negotiable.

Provisioning & observability

Provisioning

Bare-metal lifecycle is netboot-driven: a bare-metal provisioner (Tinkerbell, MAAS, or Metal³) discovers a powered-on node via its BMC, installs an immutable OS image, and enrolls it in the scheduler. Configuration is declarative and version-controlled (Terraform for topology, Ansible for any host-level config); there is no interactive machine setup and therefore no configuration drift. A machine is "cattle" — described entirely by code, replaceable by reimage.

Observability

The fleet is instrumented uniformly: metrics via Prometheus with a long-term, cross-colo store (Thanos, Mimir, or VictoriaMetrics); logs via Loki or an ELK stack; traces via OpenTelemetry into Jaeger or Tempo; alerting via Alertmanager. Because roles are software overlays, dashboards are keyed by service, not by machine — you watch "the cache tier," not "server 47." Health checks feed both the load balancers (eject a bad backend) and the scheduler (reschedule off a bad node), closing the loop from detection to remediation without a human in the fast path.

Security & privacy

The platform's product promise is privacy-first and ad-free, so privacy is an architectural property, not a policy statement:

• No third-party trackers or data brokerage anywhere in the request path — there is nothing to embed because there is no ad or data-sale business to serve.
• Encryption in transit everywhere: TLS to users, mutual TLS between internal services (a service mesh or Consul Connect issues and rotates the certificates).
• Encryption at rest at the storage layer for durable media and database volumes.
• Data minimization & portability: collect only what the product needs, and provide genuine export — a differentiator this audience actively checks for.
• Segmentation: the public edge is isolated from data and storage tiers; only the app and proxy tiers bridge them, and secrets live in a dedicated manager (Vault) rather than on hosts.
• User-supplied HTML/CSS (the MySpace-style profile customization) is the single largest application-layer attack surface. It must be aggressively sandboxed — strict Content-Security-Policy, server-side sanitization, no arbitrary JavaScript, and a whitelist of embeddable media sources — to avoid re-creating the profile-injection and worm problems the original customizable networks suffered. This is an application concern layered above the infrastructure in this paper, but it is called out because it is where this particular product is most likely to be attacked.

Limitations & future work

• Two colos are a starting point, not an end state. The witness solves correctness, but §14's 2× serving overhead is capital-inefficient; a third production colo is the natural next investment and improves both efficiency and blast radius.
• Homogeneity has a ceiling, now pushed further out. This revision folds a spinning-disk capacity tier into the one SKU (§2, §10), recovering most of the density a dedicated storage chassis would offer without a second part number. A purpose-built JBOD is still denser per U, so at extreme media scale a dedicated cold-storage SKU may eventually earn its place — but that decision is deferred far further than it was in v0.1.
• The hybrid storage stack adds moving parts. ZFS beneath Ceph costs some write amplification versus raw-device BlueStore; the ZFS ARC competes for RAM with cache and database roles on the same box (a scheduler placement constraint, §5); and large-HDD resilvers are slow — mitigated by keeping bulk redundancy in Ceph's cross-node erasure code rather than local raidz, but not eliminated.
• Celebrity fan-out remains the perennial social-scale hazard; the hybrid strategy bounds it but very large accounts still need care (rate limiting, dedicated read replicas for their association lists).
• Async cross-colo replication means a narrow window of un-replicated writes can be lost if a colo dies abruptly. Shards requiring stronger guarantees can opt into synchronous quorum writes at a latency cost; most social data does not need it.
• Bare metal shifts undifferentiated heavy lifting onto the operator. Hardware supply, RMA, DC remote-hands, and capacity lead times become your problem. This is the deliberate trade for cost control and data sovereignty; a hybrid burst-to-cloud path for demand spikes is worth keeping in reserve.

The result is a system whose resilience comes not from any one clever component but from the same discipline applied at every layer: partition it, replicate it across failure domains, put it on an identical anonymous machine, and let software decide the rest.