200+ applicants

Company

Truckstop · 1 day ago

Lead Data Engineer

United States

Full-time

Remote

Senior Level, Lead/Staff

$139K/yr - $155K/yr

8+ years exp

Maximize your interview chances

Information TechnologyInternet

Insider Connection @Truckstop

Discover valuable connections within the company who might provide insights and potential referrals.
Get 3x more responses when you reach out via email instead of LinkedIn.

Responsibilities

Create, automate, and maintain data pipelines. Assemble complex data sets that meet business requirements.

Extract, transform, and load data from a variety of data sources using SQL and big data tools and technologies.

Identify, design, and implement internal process improvement including automating manual processes, optimizing data delivery, re-designing infrastructure for greater reliability and performance.

Write and optimize queries, write lambda functions, perform source to target mapping and data modeling.

Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metrics.

Work as a member of the data engineering chapter on a development squad to deliver product features and resolve data related technical issues.

Work as a member of the data management center of excellence on data quality projects.

Work with information security to keep our data secure.

Work with data architects, database administrators, and data engineers to continually improve our data systems.

Work with IT operations to resolve data related technical issues and respond to data related major incidents.

Create data tools to assist data analysts and data scientists in building and optimizing our product into an innovative industry leader.

Troubleshoot operations and database issues for internal and external customers.

Perform incident resolution and root cause analysis of critical outages. Implement solutions to systematic failures. Provide on-call support, including after hours.

Assist with documentation of the environments that support our products.

Other duties as assigned.

Qualification

Find out how your skills align with this job's requirements. If anything seems off, you can easily click on the tags to select or unselect skills to reflect your actual expertise.

SQLPythonETL toolsData pipeline optimizationData warehousingData modelingMachine LearningCloud environmentsRelational DatabasesNoSQL DatabasesLinuxKubernetesLambda functionsIncident resolutionDocumentation

Required

Bachelor's degree or equivalent professional experience required; Computer Science or Engineering preferred.

Strong experience in SQL and Python.

Experience with relational SQL and NoSQL Databases; MS SQL, PostgreSQL, Snowflake, MongoDB, MySQL, MariaDB, Elastic stack.

Experience building and optimizing reliable, idempotent data pipelines (streaming or batch) using ETL tools such as Matillion, Apache NiFi, Kafka and Spark.

A successful history applying software engineering principles to build data infrastructure tools/libraries to automate and scale data pipelines.

Understanding of applying logging and metrics to monitor and detect data pipeline performance issues.

Solid understanding of Data warehousing theory and concepts.

Working knowledge of data modeling and source to target mapping with a demonstrated understanding of data schema design and indexes, when to apply fact-based vs. time-based vs. relational and when to apply NoSQL approaches such as key-value, document, and column data stores.

Understanding of Statistical and Machine Learning (ML) terminology with experience deploying and monitoring ML-models in a production environment.

Experience working in a modern cloud-based environment a plus; Azure, AWS, Docker, Kubernetes.

Experience with Linux.

Familiarity with Kubernetes, Lambda functions, Cloud Watch, SNS, S3, Glacier.

Minimum of 8 or more years of experience.