PLATEAU GRP ยท 1 month ago
Azure Databricks Engineer
Fairfax, VA
Full-time
Onsite
Senior Level
5+ years expPlateau Software Inc is seeking a highly skilled Azure Databricks Engineer to support a large-scale data modernization initiative for the Federal Deposit Insurance Corporation (FDIC). The selected candidate will design, develop, and optimize enterprise-grade data lakehouse solutions using Azure Databricks, collaborating with cross-functional teams to deliver secure, scalable, and high-performing data infrastructure.
Consumer ElectronicsGovernmentInformation TechnologySoftware
No H1B
Security Clearance RequiredU.S. Citizen Only
Responsibilities
Design, build, and maintain scalable ETL/ELT pipelines using Azure Databricks, Delta Lake, and Apache Spark to process large volumes of structured and unstructured data
Integrate data from multiple sources using Azure Data Factory, Azure Data Lake Storage, and Azure Synapse Analytics
Architect and optimize data lakehouse solutions leveraging Delta Lake format and partitioning strategies
Implement data modeling best practices for efficient data storage, retrieval, and analytical performance
Configure and manage Databricks workspaces, clusters, and compute resources
Implement security controls, access management, and cost optimization strategies aligned with FDIC standards
Develop streaming data solutions using Structured Streaming and implement batch processing jobs for transformation, aggregation, and integration
Ensure performance, scalability, and reliability of both real-time and batch data workflows
Collaborate with data scientists to deploy machine learning models and integrate them into data pipelines
Implement MLflow for model lifecycle management and automate ML pipelines for training and inference
Monitor pipeline performance and system health across Databricks and Azure environments
Troubleshoot job failures, implement data quality checks, and optimize Spark jobs for improved efficiency and reduced processing cost
Qualification
Required
BA/BS/MS in Computer Science, Engineering, Data Science, or equivalent experience
Over 5 years of experience as a Data Engineer, with 3+ years specifically in Azure Databricks and related Azure data services
This position requires US Citizenship and the ability to pass/obtain any required government background check or security clearance. An active security is preferred
Expertise in Apache Spark, Delta Lake, Python, SQL, and Scala
Hands-on experience designing, developing, and optimizing large-scale data pipelines in cloud environments
Technical proficiency with: Azure Data Factory, Azure Data Lake Storage, and Azure Synapse Analytics
ETL/ELT processes, data modeling, and big data architectures
Strong understanding of data governance, data security, and cloud operations in enterprise environments
Experience working within Agile delivery frameworks
Preferred
Microsoft Certified: Azure Data Engineer Associate (DP-203) or Databricks Certified Data Engineer certification
Experience supporting federal data modernization or enterprise cloud transformation programs
Familiarity with DevOps/CI-CD pipelines, GitHub Enterprise, and Infrastructure as Code (Terraform)
Experience supporting AI/ML initiatives and integrating data science workflows within Azure Databricks
Company
PLATEAU GRP
Plateau Grp provides innovative and agile IT solutions to customer challenges.
51-200 employees
Funding
Current Stage
Growth StageLeadership Team
Brad Kleinfelder
Founder/CEO
Vishwenath Kizhapandal
President & CEO
Recent News
Premium Times
2025-04-22
2025-01-28
Company data provided by crunchbase