Apply on Employer Site

hackajob · 1 day ago

Lead Data Engineer – Platform

West Chester, PA

Full-time

Onsite

Lead/Staff

$120K/yr - $180K/yr

8+ years exp

hackajob is collaborating with Comcast, a Fortune 30 global media and technology company, to connect exceptional tech professionals for the role of Lead Data Engineer – Platform. This position involves overseeing the enterprise data platform strategy, collaborating with various teams to enhance data pipelines and analytics workloads, while providing technical leadership and ensuring platform governance and security.

Artificial Intelligence (AI)Generative AIHuman ResourcesRecruitingSoftware

No H1B

Responsibilities

Define and evolve Comcast’s enterprise data platform strategy across Databricks, AWS, Kafka, and database ecosystems

Develop architectural patterns, reference implementations, reusable frameworks, and platform guardrails

Lead decisions on compute orchestration, storage tiering, data access models, and multi-region resiliency

Serve as SME for Databricks architecture, including security model, cluster governance, Unity Catalog, Delta Lake strategy, and optimization frameworks

Act as the highest-level escalation point for complex platform issues

Set engineering standards and coding conventions for all data engineering domains

Mentor senior engineers and architects; promote best practices and thought leadership

Evaluate emerging technologies to enhance scalability and operational efficiency

Own platform governance, including workspace lifecycle, cluster policy enforcement, and compliance

Drive adoption of monitoring and observability tools (ELK, CloudWatch, Databricks native logging)

Implement FinOps strategies for cost management and forecasting

Ensure platform resilience, disaster recovery readiness, and high availability

Lead globally distributed platform engineering teams (onshore, offshore, vendor partners)

Partner with Security, Enterprise Architecture, and CloudOps to align standards and governance

Drive quarterly planning, roadmap execution, and platform enhancements

Communicate risks, KPIs, and modernization opportunities to executive stakeholders

Qualification

DatabricksAWSKafkaDistributed SystemsPythonSQLShell scriptingOracleMongoDBStakeholder InfluenceTechnical LeadershipCollaborationCommunication

Required

8-10+ years of experience in Data Engineering, Platform Engineering, Distributed Systems, or Software Engineering

Deep hands-on expertise in Databricks engineering and administration: Cluster policies & cost governance; Workspace architecture; Unity Catalog & data access models; Delta Lake performance tuning; Job orchestration & pipeline optimization

Advanced AWS experience with: S3 lifecycle tiers, encryption, cross-account architectures; MSK provisioning, tuning, scaling, monitoring; EC2, VPC networking, IAM role design; Lambda for serverless automation

Strong background in backend/datastore technologies (Oracle, MongoDB)

Expert-level knowledge of Kafka, Spark, distributed computing, and real-time ingestion frameworks

Proficiency in Python, SQL, and Shell scripting

Prior leadership experience managing platform teams and offshore delivery models

Proven ability to define and enforce platform standards across large organizations

Excellent communication and stakeholder influence skills

Bachelor's Degree