Apply on Employer Site

ICF · 3 months ago

Senior Data Engineer (USA/Remote)

United States

Full-time

Remote

Senior Level

$99K/yr - $168K/yr

7+ years exp

ICF is a mission-driven company dedicated to improving lives and making the world a better place. They are seeking a seasoned Senior Data Engineer to design, develop, and maintain scalable data pipelines and facilitate data access and integration using various technologies.

ConsultingInformation TechnologyProfessional Services

No H1B

Security Clearance Required

U.S. Citizen Only

Responsibilities

Design, develop, and maintain scalable data pipelines using Spark, Hive, and Airflow

Develop and deploy data processing workflows on the Databricks platform

Develop API services to facilitate data access and integration

Create interactive data visualizations and reports using AWS QuickSight

Build required infrastructure for optimal extraction, transformation and loading of data from various data sources using AWS and SQL technologies

Monitor and optimize the performance of data infrastructure and processes

Develop data quality and validation jobs

Assemble large, complex sets of data that meet non-functional and functional business requirements

Write unit and integration tests for all data processing code

Work with DevOps engineers on CI, CD, and IaC

Read specs and translate them into code and design documents

Perform code reviews and develop processes for improving code quality

Improve data availability and timeliness by implementing more frequent refreshes, tiered data storage, and optimizations of existing datasets

Maintain security and privacy for data at rest and while in transit

Other duties as assigned

Qualification

PythonPySparkAWSDatabricksSparkHiveSQLData governanceDevOpsAgile methodology

Required

Bachelor's degree in computer science, engineering or related field

7+ years of hands-on software or data development experience

4+ years of data pipeline experience using Python, PySpark and cloud technologies

2 years working in Spark and Hive or similar large data environments

Candidate must be able to obtain and maintain a Public Trust clearance

Candidate must reside in the US, be authorized to work in the US, and work must be performed in the US

Must have lived in the US 3 full years out of the last 5 years

Travel of up to once a quarter US domestically is required

Preferred

U.S. Citizenship or Green Card is highly prioritized due to federal contract requirements

Experience building job workflows with the Databricks platform (Strongly Preferred)

Strong understanding of AWS products including S3, Redshift, RDS, EMR, AWS Glue, AWS Glue DataBrew, Jupyter Notebooks, Athena, QuickSight, EMR, and Amazon SNS

Familiar with work to build processes that support data transformation, workload management, data structures, dependency and metadata

Experienced in data governance process to ingest (batch, stream), curate, and share data with upstream and downstream data users

Experienced in data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the ground up

Demonstrated understanding using software and tools including relational NoSQL and SQL databases including Cassandra and Postgres; workflow management and pipeline tools such as Airflow, Luigi and Azkaban; stream-processing systems like Spark-Streaming and Storm; and object function/object-oriented scripting languages including Scala, C++, Java and Python

Familiar with DevOps methodologies, including CI/CD pipelines (Github Actions) and IaC (Terraform)

Experience with Agile methodology, using test-driven development