Capgemini · 1 week ago
Lead Data Engineer - Real Time Data Processing
Atlanta, GA
Full-time
Hybrid
Senior Level, Lead/Staff
$130K/yr - $135K/yr
10+ years expCapgemini is a global business and technology transformation partner, and they are seeking a seasoned Lead Software Engineer to architect, build, and scale real time data processing platforms. The role involves designing streaming microservices, governing data quality, and mentoring engineers while collaborating with various stakeholders to deliver resilient systems.
ConsultingInformation TechnologyInsurTechIT ManagementSoftware
Responsibilities
Own design & delivery of high throughput, low latency streaming solutions using technologies like Confluent Kafka, Apache Flink, Hazelcast, Kafka Streams, Kafka Connect, and Schema Registry
Design and implement microservices and event driven systems with robust ETL/ELT pipelines for real time ingestion, enrichment, and delivery
Establish distributed caching and in memory data grid patterns (e.g., Redis, Hazelcast) to optimize read/write performance and session/state management
Define and operationalize event gateways / event grids for event routing, fan out, and reliable delivery
Lead data governance initiatives-standards for metadata, lineage, classifications, retention, access controls, and compliance (PII/PCI/SOX/GDPR as applicable)
Drive CI/CD best practices (pipelines, automated testing, progressive delivery) to enable safe, frequent releases; champion DevSecOps and "shift left" testing
Set SLOs/SLAs, track observability (tracing, metrics, logs), and optimize performance at scale (throughput, backpressure, state, checkpointing)
Work with Security, Platform, and Cloud teams on networking, IAM, secrets, certificates, and cost optimization
Mentor engineers, conduct design reviews, and enforce coding standards and reliability patterns
Guide platform and delivery roadmap
Qualification
Required
10+ years in software engineering; 5+ years designing large-scale real time or event driven platforms
Expert with Confluent Kafka (brokers, partitions, consumer groups, Schema Registry, Kafka Connect), Flink (DataStream/Table API, stateful ops, checkpointing), Hazelcast, and/or Kafka Streams
Strong in ETL/ELT design, streaming joins/windows, exactly once semantics, and idempotent processing
Experience with microservices (Java/Python), REST/gRPC, protobuf/Avro, and contract-first development
Hands-on with distributed caching and in memory data grids; performance tuning and eviction strategies
Cloud experience in any one or more cloud platforms Azure/AWS/GCP; containers, Docker, Kubernetes
Experience in production-grade CI/CD (Jenkins, Bamboo, Harness or similar), Infrastructure as Code (Terraform/Helm)
Robust observability (Prometheus/Grafana/OpenTelemetry, Splunk/ELK or similar), and resilience patterns (circuit breakers, retries, DLQs)
Practical data governance: metadata catalogs, lineage, encryption, RBAC
Excellent communication; ability to lead design, influence stakeholders, and guide cross-functional delivery
Core competencies to include Architectural Thinking, Systems Design, Operational Excellence, Security & Compliance, Team Leadership, Stakeholder Management
Preferred
Experience with CDC, Kafka Connect custom connectors, Flink SQL, Beam
Streaming ML or feature stores integration (online/offline consistency)
Multi region / disaster recovery for streaming platforms
Experience with Zero downtime migrations, blue/green, and canary deployments
Benefits
Flexible work
Healthcare including dental, vision, mental health, and well-being programs
Financial well-being programs such as 401(k) and Employee Share Ownership Plan
Paid time off and paid holidays
Paid parental leave
Family building benefits like adoption assistance, surrogacy, and cryopreservation
Social well-being benefits like subsidized back-up child/elder care and tutoring
Mentoring, coaching and learning programs
Employee Resource Groups
Disaster Relief
Medical, dental, and vision coverage (or provincial healthcare coordination in Canada)
Retirement savings plans (e.g., 401(k) in the U.S., RRSP in Canada)
Life and disability insurance
Employee assistance programs
Company
Capgemini
Capgemini is a software company that provides consulting, technology, and digital transformation services.
10001+ employees
H1B Sponsorship
Capgemini has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (2856)
2024 (3012)
2023 (3424)
2022 (4392)
2021 (3311)
2020 (5871)
Funding
Current Stage
Public CompanyTotal Funding
$4.72B2025-09-18Post Ipo Debt· $4.72B
1999-04-01IPO
Leadership Team
Aiman Ezzat
CEO, Capgemini Group
Anirban Bose
CEO of Americas Strategic Business Unit
Recent News
2026-01-08
2026-01-06
2026-01-05
Company data provided by crunchbase