193 applicants

Company

Original Job Post

ATD · 2 days ago

Senior Big Data Engineer

United States

Contract

Remote

Senior Level

$75/hr - $75/hr

5+ years exp

Wonder how qualified you are to the job?

Maximize your interview chances

FinanceFinancial Exchanges

Insider Connection @ATD

Discover valuable connections within the company who might provide insights and potential referrals, giving your job application an inside edge.

Responsibilities

Optimize and troubleshoot Databricks workflows

Develop data pipelines and data orchestration tools

Ingest and transform data (batch and streaming) with data quality validation

Utilize Spark programming in Python and Pyspark

Work with GCP services such as Google Cloud Storage, BigQuery, Pub/Sub, and Dataflow

Lead problem-solving, root cause analysis, and issue resolution

Apply expertise in Healthcare or Health Insurance Industry

Collaborate with application development tools and software development lifecycle

Qualification

Find out how your skills align with this job's requirements. If anything seems off, you can easily click on the tags to select or unselect skills to reflect your actual expertise.

SparkPythonData engineeringEtlGcpGoogle cloud storageBigqueryPub/subDataflowHealthcare industryHealth insurance industryApplication developmentSoftware development life cycleDatabricks certified data engineer professionalProblem solvingLeadership

Required

Bachelor’s degree in Information Technology, Computer Science or closely related field

5 years of experience as a developer

3 years of experience in Databricks technology and be experienced in working with Databricks notebooks, clusters, and jobs. They should be able to optimize and troubleshoot Databricks workflows.

Deep understanding of Spark and its core concepts. They should be proficient in Spark programming Python, Pyspark

Strong data engineering skills, including experience with data ingestion (batch and streaming), data transformation, and data quality validation. They should be familiar with ETL and have hands-on experience with data pipelines and data orchestration tools.

Expertise with GCP, including services like Google Cloud Storage, BigQuery, Pub/Sub, and Dataflow.

Problem solving, root cause analysis, and issue resolution

Demonstrated leadership skills

Experience in Healthcare or Health Insurance Industry

Experience with Databricks, GCP, application development tools, business processes, software development life cycle

Data Engineer: 5 years

Databricks: 5 years

Python: 5 years

Google Cloud Platform: 5 years

Databricks Certified Data Engineer Professional

Benefits

401(k)

Dental insurance

Health insurance

Company

ATD

A pioneer of automated trading technology, Automated Trading Desk, LLC (ATD) develops state of the art trading solutions for its broker-dealer subsidiaries.

Founded in 1988

Mount Pleasant, South Carolina, USA

501-1,000 employees