Distributed Locking

In a single-server application, you use a mutex or semaphore to prevent two threads from concurrently modifying the same data. But in a distributed system with multiple servers, a local mutex is useless—Server A's lock is invisible to Server B. You need a Distributed Lock: a lock that is visible and enforceable across all servers in the cluster.

1. Use Cases

• Preventing Double Processing: An e-commerce system uses a message queue. If two consumer servers pick up the same "Process Payment" message, the customer gets charged twice. A distributed lock on the orderId ensures only one server processes it.
• Scheduled Jobs: A cron job runs on all 10 application servers. Without a distributed lock, the same daily report email is sent 10 times.
• Resource Coordination: Only one server should be rebuilding the search index at any time.

2. Implementation with Redis (Redlock)

Redis is the most popular backend for distributed locks due to its speed and atomic operations.

Simple Lock:

SET resource_name my_random_value NX PX 30000

• NX: Only set if the key does NOT already exist (mutual exclusion).
• PX 30000: Auto-expire after 30 seconds (prevents deadlocks if the lock holder crashes).
• my_random_value: A unique identifier so only the lock holder can release it.

Release:

Use a Lua script to atomically check and delete:

if redis.call("get", KEYS[1]) == ARGV[1] then
    return redis.call("del", KEYS[1])
end

This ensures that Server A doesn't accidentally release a lock that was already acquired by Server B after Server A's lock expired.

Redlock Algorithm:

For higher reliability, the Redlock algorithm acquires locks across multiple independent Redis instances (e.g., 5 instances). The lock is considered acquired only if a majority (at least 3 out of 5) of instances grant it. This tolerates individual Redis node failures.

3. Implementation with ZooKeeper

ZooKeeper uses ephemeral sequential nodes for distributed locking:

1. Each client creates an ephemeral sequential node under a lock path (e.g., /locks/resource/lock-0000000001).
2. The client with the lowest sequence number holds the lock.
3. Other clients watch the node immediately preceding theirs.
4. When the lock holder finishes and deletes its node (or crashes, causing the ephemeral node to disappear), the next client in sequence is notified and acquires the lock.

4. Challenges

• Clock Drift: If Server A acquires a lock with a 30-second expiry, but Server A's clock is running 5 seconds fast, the lock might expire before Server A finishes its work.
• Network Delays: A server might acquire a lock, experience a long garbage collection pause, and resume after the lock has already expired and been acquired by another server.
• Fencing Tokens: To solve the above issues, the lock service issues a monotonically increasing fencing token with each lock acquisition. The protected resource rejects any request with a token lower than the highest token it has already seen.