This job has closed.

Pearster · 1 day ago

SR Technical Operations Engineer (Solana)

United States

Full-time

Remote

Senior Level

Pearster is a cloud-based infrastructure provider powering the global blockchain ecosystem. They are seeking a Senior Technical Operations Engineer (Solana) to maintain and optimize core systems, ensuring the reliability and performance of Solana in production environments.

Cloud SecurityConsultingInformation Technology

Responsibilities

Run validators: Deploy, upgrade, and tune Agave/Jito; minimize missed slots; maintain healthy voting and high leader performance

Operate high-throughput RPC: Set smart connection and queue limits, optimize PubSub fan-out and backpressure, and ensure indexers are efficiently served without starving nodes

Extract performance from hardware: Select optimal servers, tune BIOS, kernel, NIC, and NVMe configurations, and validate performance gains through profiling and metrics

Automate everything: Implement reproducible images, manage fleet changes with Terraform and Ansible, create snapshot pipelines, verify state-sync and replay processes, and build automated release systems

Lead incidents (SEV0–2): Quickly isolate issues, execute safe roll-forwards or roll-backs, publish clear root cause analyses, and implement preventive measures to avoid recurrences

Collaborate with the ecosystem: Reproduce complex bugs, share performance traces, and contribute targeted patches upstream when beneficial

Code where it counts: Develop and extend tools for snapshots, replay/load, and state-sync verification; patch client bugs impacting production and upstream relevant fixes when valuable

Qualification

Linux systemsAgave/Jito operationsGoPythonHardware performance engineeringObservability toolsKey managementTransaction sending logicSoft skills

Required

Linux systems + kernel tuning: NUMA, IRQ affinity, hugepages, cpusets, I/O schedulers, sysctl; filesystem/NVMe layout; BIOS/firmware setup (C-states, power governors); NIC queues/offloads (RSS/RPS/XPS, GRO/LRO/TSO)

Hardware performance engineering: Choose and tune CPU/RAM/NVMe/NIC; measure replay throughput, p95/p99 RPC latency, IOPS/egress—and push them lower/faster

Agave/Jito operations: Build from source; manage feature gates and config flags; snapshots (create/consume), ledger compaction/repair/replay health; accounts-DB tuning; version management

Read protocols & surfaces: Operate and tune JSON-RPC (HTTP/WS), gRPC, and PubSub; design connection pools, concurrency limits, caching, timeouts, and backpressure that hold under peak

Transaction sending logic: Understand direct-to-TPU (QUIC) vs RPC sendTransaction; preflight/simulation trade-offs; priority fees and compute budget tuning; leader-schedule awareness

Go or Python (plus Bash): Build small, sharp tools/CLIs (snapshot/restore pipelines, state-sync verification, health checks, replay/load harnesses)

Observability that matters: SLOs/error budgets; Prometheus/Grafana; alerts that page only when users hurt (RPC latency, PubSub backlog, missed leader slots, replay stalls)

Key management & safety: KMS/HSM/Vault; authority rotations; secure backups; tested DR paths; controlled, auditable change windows