Edge & Node · 1 day ago

Data Systems Engineer, Rust

United States

Full-time

Remote

Mid Level

$100K/yr - $500K/yr

Maximize your interview chances

Software

Insider Connection @Edge & Node

Discover valuable connections within the company who might provide insights and potential referrals.
Get 3x more responses when you reach out via email instead of LinkedIn.

Responsibilities

Leveraging and contributing to Apache DataFusion, closely monitoring upstream developments and becoming familiar with its codebase.

Implementing storage strategies for blockchain and other web3 data sources, while leveraging industry standard formats and keeping in mind their tradeoffs.

Developing the internals of a streaming processing system. Given the streaming nature of blockchains, you must understand incremental techniques to update outputs with each new block.

Applying resource isolation in a multi-tenant environment for resources such as CPU, RAM and storage.

Building data transformation pipelines specifically for blockchain data and implementing operations like aggregations and joins for streamlined usage.

Optimizing compaction and garbage collection processes for performance and resource efficiency particularly in multi-tenant and high-throughput database systems.

Developing supportive tooling, including tracing and profiling.

Qualification

Find out how your skills align with this job's requirements. If anything seems off, you can easily click on the tags to select or unselect skills to reflect your actual expertise.

RustApache DataFusionDatabase system internalsDistributed data systemsResource managementData ingestionData transformation pipelinesStreaming processing systemGarbage collection optimizationTracingProfiling tools