Cambridge, Massachusetts, United States; Palo Alto, California, United States; Singapore; Tokyo, Japan•Cambridge MA - US•Palo Alto CA, US•Singapore•Tokyo, JP
Remote
Senior
Full Time
databaseClickHouseMooseFSSeaweedFSTiDBElasticsearchC++RustGodistributed systems
Requirements
- •Strong C++, C and Rust programming skills
- •Ability to work in Go
- •Experience with large codebases of millions of lines
- •Deep knowledge of database storage engines including merges or compaction, replication logs, query execution
- •Understanding of distributed consensus (Raft or Paxos) and failure scenarios
- •Knowledge of Linux storage internals: page cache, fsync, FUSE, file systems, NVMe and HDD behavior
- •Experience running databases or storage on hundreds of nodes and leading recovery during incidents
- •Debugging skills with gdb, perf and sanitizers
- •Ability to measure changes on production data before considering them successful
What You'll Do
- •Build storage and query features in ClickHouse, such as replication over a shared file system, and storage policies that place data by its merge history
- •Keep up with upstream ClickHouse LTS releases, keep CI green and ship our own releases regularly
- •Make MooseFS and SeaweedFS highly available and durable: Raft-based metadata servers, follower reads, an intent log, automatic re-replication
- •Find performance problems in production profiles and fix them in the code, from query plans to FUSE clients
- •Trace data corruption to its root cause, even when it’s in compression, the memory allocator or the compiler, and repair the damaged data
- •Roll out releases and storage migrations across about 1,200 servers in three regions, without downtime
- •Run and tune TiDB, TiKV, TiCDC and Elasticsearch: scheduling, replication lag, backups, disk balance
- •Check risky designs with TLA+ and fault injection, and automate whatever repeats with watchdogs, balancers and dashboards
- •Build an SRE agent that watches our database clusters, diagnoses incidents and proposes fixes
Nice to Have
- •Contributions to an open-source database or file system
- •Knowledge of OCaml
- •Experience with Puppet or similar configuration management tools
Benefits
- •Real ownership of large-scale data systems
- •Small team with no layers of process
- •Hardware to do the job properly
- •Value honesty, results and simple solutions over complicated ones
