SIGN IN
Lead Cloudera Streaming Architect (CDP | NiFi | Kafka | Flink | Kudu | SSB) jobs in United States
info-icon
This job has closed.
company-logo

HCL Global Systems Inc · 5 days ago

Lead Cloudera Streaming Architect (CDP | NiFi | Kafka | Flink | Kudu | SSB)

positionUnited States
timeFull-time
remoteRemote
seniorityLead/Staff
date8+ years exp
HCL Global Systems Inc is seeking a Lead Cloudera Streaming Architect with extensive experience in the Cloudera CDP streaming stack. The role involves designing, delivering, and optimizing mission-critical real-time data pipelines at enterprise scale, utilizing technologies such as NiFi, Kafka, and Flink.
B2BConsultingHuman ResourcesInformation TechnologySoftwareStaffing Agency
check
H1B Sponsor Likelynote

Responsibilities

Architect and build real-time data pipelines using the full Cloudera Data Platform (CDP) streaming suite: NiFi → Kafka → Flink → Kudu/Impala → SSB
Own architectural decisions, patterns, and best practices for streaming, CDC, state management, schema evolution, and exactly-once delivery
Develop complex NiFi flows involving controller services (DBCP/JDBC), stateful processors, record processors, schema registry integrations, batch-to-stream conversions, and high-volume ingestion patterns
Build and optimize Flink SQL or DataStream API jobs with: ? Kafka sources/sinks ? event-time windows ? watermarks ? state management ? checkpointing / savepoints ? exactly-once guarantees
Design and tune Kudu tables (PKs, partitioning, distribution, upserts, deletes, merges)
Build and deploy streaming SQL jobs using Cloudera SQL Stream Builder (SSB)
You must be able to deliver the following four core use cases immediately: 1. NiFi → Snowflake → Impala/Kudu ingestion pipeline 2. Kafka → Flink streaming (real-time processing) 3. Flink → Kafka sink with exactly-once semantics 4. CDC ingestion via NiFi, Flink CDC, or SSB (incremental keys, late events, deletes)
Tune NiFi, Kafka, and Flink clusters for performance, throughput, and stability
Implement schema governance, error handling, back-pressure strategies, and replay mechanisms
Work closely with platform engineers to optimize CDP components and CDF deployments
Provide architectural guidance, documentation, and mentorship to engineering teams

Qualification

Cloudera CDPApache NiFiApache KafkaApache FlinkApache KuduSQL Stream BuilderChange Data CaptureKubernetesCI/CDSecuritySoft Skills

Required

Hands-on, production-grade experience with Cloudera CDP / CDF
Experience with CDP Public Cloud or Private Cloud Base
Experience with Cloudera Flow Management (NiFi + NiFi Registry)
Experience with Cloudera Streams Messaging (Kafka, SMM)
Experience with Cloudera Stream Processing (Flink, SSB)
Experience with Kudu / Impala ecosystem
Advanced experience with Apache NiFi including building complex flows
Experience with QueryDatabaseTable / GenerateTableFetch / MergeRecord
Experience with record-based processors & schema registry
Experience with JDBC / DBCP controller services
Experience with stateful processors & incremental ingestion
Experience with NiFi → Snowflake integration
Experience with NiFi → Kudu ingestion patterns
Experience with Apache Kafka including brokers, partitions, retention, replication, consumer groups
Experience with schema registry (Avro/JSON)
Experience with designing topics for high-throughput streaming
Experience with Apache Flink including Flink SQL + DataStream API
Experience with event-time processing, watermarks, windows
Experience with checkpointing, savepoints, state backends
Experience with Kafka source/sink connectors
Experience with exactly-once semantics
Experience with Apache Kudu including table design (PKs, partition strategies)
Experience with upserts, deletes, merge semantics
Experience with integration with Impala
Experience with SQL Stream Builder (SSB) including creating jobs, connectors, materialized views
Experience with deploying and monitoring Flink SQL jobs in CDP
Experience with CDC (Change Data Capture) via NiFi or Flink CDC or SSB
Experience with handling late-arriving events
Experience with handling deletes, updates, schema evolution
Experience with incremental key tracking
8+ years in data engineering / streaming
3–5+ years specifically with CDP/CDF streaming
Strong SQL and distributed system fundamentals

Preferred

Experience in financial services, healthcare, telecom, or other high-volume industries
Kubernetes experience running NiFi/Kafka/Flink operators
Snowflake ingestion patterns (staging, Copy Into)
Experience with Debezium
CI/CD for data pipelines
Security (Kerberos, Ranger, Atlas)

Company

HCL Global Systems Inc

twittertwitter
Glassdoor3.8
company-logo
HCL Global Systems is a staffing and recruiting company providing consulting and business solutions.
dateFounded in 2005
location
Farmington Hills, Michigan, USA
people-size1001-5000 employees
web-link
http://hclglobal.com

H1B Sponsorship

HCL Global Systems Inc has a track record of offering H1B sponsorships. Please note that this does not guarantee sponsorship for this specific role. Below presents additional info for your reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (265)
2024 (285)
2023 (378)
2022 (390)
2021 (470)
2020 (736)

Funding

Current Stage
Late Stage

Leadership Team

A
Andy HCL
Cheif Marketing Officer
linkedin
Company data provided by crunchbase