CoWrite: Distributed File System

The problem

Collaborative editing looks simple from the outside: several people type into the same document and it just works. Underneath, it is distributed systems in miniature: concurrent writers who must not clobber each other, storage machines that can die with a write in flight, and users who should only ever see the files they have been granted. Most software gets these guarantees for free from a database or a framework, which means most engineers never have to actually build them.

CoWrite is my attempt to solve those problems directly, in C, over raw TCP sockets. No framework, no database, no off-the-shelf coordination service. Every lock, ticket, replication task, and failover decision in the system is code I wrote and can defend line by line. The goal was not to compete with Google Docs; it was to earn the semantics that make Google Docs feel effortless.

Interactive: locks, tickets, and a crash

This is the real topology: two docs_client users on the left, the name manager brokering every operation in the middle, and a primary/backup storage pair on the right holding the document notes. Solid wires carry control RPCs; dashed wires are the data plane, where file content actually flows. Run a scenario, or crash the primary yourself and watch the name manager recover.

How it works

CoWrite is three programs speaking a small length-prefixed JSON protocol over TCP:

• Name Manager (bin/nm) is the coordinator. It owns all file metadata, the ACLs, and the file → primary/backup map, and it brokers every operation through short-lived capability tickets (30-second TTL). It is multithreaded: a thread per connection, plus background workers for replication and ticket expiry.
• Storage Server (bin/ss) holds file content on disk, enforces sentence-level write locks, performs commit-time merges, and keeps per-file undo snapshots and named checkpoints. Multiple instances register with the NM, which assigns primary/backup roles across them.
• docs_client is the interactive REPL that authenticates with a username and drives every command.

The split that matters is control plane versus data plane. Clients never move file content through the name manager. A client asks the NM for an operation; the NM checks the ACL and answers with a storage-server endpoint plus a ticket; the client then connects to the storage server directly and presents the ticket. The NM stays a small metadata service instead of becoming an I/O bottleneck, and a storage server never has to take a client's word for anything: it honors only valid, unexpired tickets.

Concurrency & conflicts

Write locks are sentence-granular: a WRITE is scoped to a sentence index, so writers on different sentences of the same file proceed concurrently and only same-sentence writers serialize (the loser gets ERR_LOCKED and retries). A write session records a baseline sentence; on commit, the storage server re-reads the current file, relocates the target sentence by its content and surrounding context, and merges the edit in. If the baseline can no longer be found because someone changed it underneath the writer, the commit returns ERR_CONFLICT. To be precise about what that is: conflict detection is heuristic and string-based, not version-numbered. While a write is in progress, UNDO, REVERT, and DELETE on that file are refused with ERR_LOCKED, so rollback can never race a half-finished edit.

Replication & failover

On CREATE, the NM assigns a primary storage server round-robin and, when a second server is available, a backup. After every mutating operation the primary sends SS_FILE_UPDATE to the NM, which enqueues an asynchronous replication task: the backup pulls the authoritative content from the primary using a read ticket the NM hands it via NM_SYNC. Failure detection is connection-drop based. When a storage server's control connection to the NM dies, the NM walks every file whose primary was that server, promotes the backup to primary in place, and asynchronously re-replicates to restore a new backup. Because clients re-resolve the endpoint on every operation, the failover is transparent to them: the next ticket simply points at the promoted server.

Design decisions & trade-offs

Capability tickets instead of proxying. The obvious design routes all I/O through the coordinator; it is simpler and it is also how you build a bottleneck. Tickets keep the NM out of the data path entirely while still centralizing authorization. A storage server never trusts a client's claim of access, only a valid ticket the NM minted. The 30-second TTL bounds how long a stale grant can live. The cost is a slightly larger protocol surface and a two-hop dance for every operation.

Sentence granularity. Locking is a spectrum. Word-level locks would thrash, with every keystroke-sized edit acquiring and releasing state, and file-level locks would kill the entire point of collaboration by serializing all writers. Sentences are the unit people actually edit, so that is where the lock lives: coarse enough to be cheap, fine enough that two writers rarely collide.

Asynchronous replication. The backup is brought up to date after the commit is acknowledged, not before. That choice buys availability and low write latency, since a writer never waits on the backup, and it is an acceptable trade for a document editor. The cost is a real one, and the failover scenario in the demo shows where it bites: a freshly promoted backup may be missing the primary's very latest writes. I chose to pay that cost knowingly rather than pretend the system is synchronously consistent.

Honest scope

• Failure detection is connection-drop / RPC-failure based. There is no active heartbeat, so a hung-but-still-connected storage server is not detected.
• Replication is asynchronous and each file keeps a single backup, so a freshly promoted backup may lag the primary's most recent writes.
• Conflict detection is heuristic and string-based (content + context matching), not version-vector or CRDT based.
• The README's tested-limits table (max clients, failover time, replication lag) is deliberately left unfilled, because I don't quote numbers I haven't measured.

Try it yourself

Requires GCC or Clang, make, a POSIX environment, and OpenSSL headers (the storage server links -lcrypto). Start the pieces in order so each can register with the previous one:

make                    # produces bin/nm, bin/ss, bin/docs_client

# 1) name manager
bin/nm 9000

# 2) storage servers - start a second one to exercise replication + failover
bin/ss ss1 127.0.0.1 9100 127.0.0.1 9000 ./ss_data
bin/ss ss2 127.0.0.1 9200 127.0.0.1 9000 ./ss_data2

# 3) client
bin/docs_client 127.0.0.1 9000

A short session: create a file, edit a sentence, snapshot it, and roll back.

$ bin/docs_client 127.0.0.1 9000
Username: alice
Connected as alice. Type 'help' for commands.

docs> CREATE notes
Created notes

docs> WRITE notes 0
write> 0 hello world
write> ETIRW
Write committed.

docs> READ notes
--- notes ---
hello world
-------------

docs> CHECKPOINT notes v1
Checkpoint 'v1' saved for notes.

docs> REVERT notes v1
Reverted notes to checkpoint 'v1'.