Apply on Employer Site

Sev1Tech LLC · 1 month ago

Senior Databricks Platform Engineer

US-VA-Arlington

Full-time

Onsite

Senior Level

Sev1Tech LLC is seeking a highly experienced and skilled Senior Data Lake Engineer to join their team. This role involves establishing and configuring an enterprise-level Databricks solution to support federal customer data lake initiatives, ensuring scalability, reliability, and performance of data infrastructure.

ConsultingCyber SecurityInformation TechnologyInfrastructureSecurity

Growth Opportunities

No H1B

Security Clearance Required

U.S. Citizen Only

Responsibilities

Design, implement, and configure enterprise Data Lake solutions on AWS using Databricks, ensuring scalability, reliability, and performance

Work directly with cross-functional teams to gather requirements, analyze data integration needs, and translate them into actionable Data Lake architecture components

Build and configure Databricks workspaces, clusters, notebooks, jobs, and Delta Lake storage, integrating with AWS services such as S3, IAM, and KMS

Develop end-to-end data ingestion pipelines to extract, transform, and load data using Databricks, Apache Spark, and AWS services

Implement and maintain security controls including IAM policies, cluster security configurations, KMS-based encryption, and data masking to protect sensitive data

Create and optimize data transformation workflows, Delta Lake tables, and ETL/ELT processes to ensure high data quality and performance

Monitor and tune Databricks clusters and job workloads for reliability, cost optimization, and performance, leveraging autoscaling and workload management

Apply data governance best practices including cataloging, metadata management, and data lineage using Databricks Unity Catalog and AWS-native capabilities

Collaborate with cloud infrastructure teams to configure supporting AWS components such as S3 storage, networking, logging, monitoring, and access controls

Maintain detailed technical documentation including solution designs, data flow diagrams, configuration standards, and operational procedures

Stay up to date with advancements in Databricks, Delta Lake, Spark, and AWS services, integrating new features that improve automation and efficiency

Partner with data engineers, analysts, and scientists to implement data models and reusable transformation patterns within Databricks and AWS

Troubleshoot and resolve ingestion failures, cluster performance issues, pipeline errors, and data quality problems across AWS and Databricks environments

Ensure compliance with data governance, privacy, and security requirements by applying secure architecture patterns and validating controls throughout the data lifecycle

Support the evaluation and hands-on testing of new AWS and Databricks features or services that enhance the Data Lake environment

Qualification

DatabricksAWSApache SparkData governanceSQLData ingestion pipelinesPythonScalaJavaTechnical guidanceData securityData modelingRelevant certificationsProblem-solvingCommunication skillsMentorship

Required

Bachelor's degree in computer science, information technology, or a related field. Equivalent experience will also be considered

Proven experience in building and configuring enterprise-level data lake solutions using Databricks in an AWS or Azure environment

In-depth knowledge of Databricks architecture, including workspaces, clusters, storage, notebook development, and automation capabilities

Strong expertise in designing and implementing data ingestion pipelines, data transformations, and data quality processes using Databricks

Experience with big data technologies such as Apache Spark, Apache Hive, Delta Lake, and Hadoop

Solid understanding of data governance principles, data modeling, data cataloging, and metadata management

Hands-on experience with cloud platforms like AWS or Azure, including relevant services like S3, EMR, Glue, Data Factory, etc

Proficiency in SQL and one or more programming languages (Python, Scala, or Java) for data manipulation and transformation

Knowledge of data security and privacy best practices, including data access controls, encryption, and data masking techniques

Strong problem-solving and analytical skills, with the ability to identify and resolve complex data-related issues

Excellent interpersonal and communication skills, with the ability to collaborate effectively with technical and non-technical stakeholders

Experience in a senior or lead role, providing technical guidance and mentorship to junior team members

Eligibility/Clearance Requirements: Must be eligible to obtain a Department of Homeland Security EOD clearance (Requirements 1. US Citizenship, 2. Favorable Background Investigation)

Preferred

Relevant certifications such as Databricks Certified Developer or Databricks Certified Professional are highly desirable

Active DHS/CISA suitability - 1st priority

Any DHS badge + DoD Top Secret - 2nd choice

DoD Top Secret + willingness to obtain DHS/CISA suitability - 3rd choice (it can take 10-60 days to obtain suitability – work can only begin once suitability is fully adjudicated)