Truckstop · 2 days ago

Data Engineer Lead

United States

Full-time

Remote

Senior Level

$136K/yr - $150K/yr

6+ years exp

Wonder how qualified you are to the job?

Maximize your interview chances

Information TechnologyInternet

Insider Connection @Truckstop

Discover valuable connections within the company who might provide insights and potential referrals, giving your job application an inside edge.

Responsibilities

Create, automate, and maintain complex data pipelines from multiple source systems using migration and orchestration technology.

Extract, transform, and load data from various sources using SQL, Python, and cloud data technologies.

Interact with API and Kafka technology to ingest and transform data into Snowflake for Product and Data Science team end uses.

Write and optimize queries, perform data modeling, and work with Product to engineer performant data pipelines.

Collaborate with various teams to improve data security, data capture, and data infrastructure.

Resolve data-related technical issues, respond to incidents, and perform root cause analysis of critical outages.

Assist with documentation of environments supporting products and provide on-call support when necessary.

Qualification

Find out how your skills align with this job's requirements. If anything seems off, you can easily click on the tags to select or unselect skills to reflect your actual expertise.

SQLPythonSnowflakeMS SQLPostgreSQLETLMatillionApache AirflowFiveTranKafkaSparkSoftware EngineeringData InfrastructureAutomationLoggingMetricsData WarehousingPerformance OptimizationSnowflake ArchitectureAzureDbtData ModelingSource to Target MappingData Schema DesignSnowflake ClusteringMicro-partitioningAzure CloudAWS CloudData ScienceMachine Learning

Required

Bachelor's degree or equivalent professional experience required; Computer Science or Engineering preferred.

Demonstrated expertise in SQL and Python required.

Minimum of +6 years of experience required.

Experience with Snowflake, MS SQL, and PostgreSQL required.

Experience building and optimizing reliable, idempotent data pipelines (streaming or batch) using ETL tools such as Matillion, Apache Airflow, FiveTran, Kafka and Spark required.

A successful history applying software engineering principles to build data infrastructure tools/libraries to automate and scale data pipelines.

Understanding of applying logging and metrics to monitor and detect data pipeline performance issues.

Advanced understanding of data warehousing architecture including performance optimization and tuning, specifically within Snowflake and Azure.

Working knowledge of dbt data modeling and source to target mapping with a demonstrated understanding of data schema design, Snowflake clustering and micro-partitioning.

Experience working in Azure and AWS cloud-based environment.