HCL Global Systems Inc · 14 hours ago
Lead Cloudera Streaming Architect (CDP | NiFi | Kafka | Flink | Kudu | SSB)
United States
Full-time
Remote
Lead/Staff
8+ years expHCL Global Systems Inc is seeking a Lead Cloudera Streaming Architect with deep, hands-on experience across the Cloudera CDP streaming stack. The role involves designing, delivering, and optimizing mission-critical real-time data pipelines at enterprise scale, requiring expertise in various streaming technologies and architectures.
B2BConsultingHuman ResourcesInformation TechnologySoftwareStaffing Agency
Responsibilities
Architect and build real-time data pipelines using the full Cloudera Data Platform (CDP) streaming suite: NiFi → Kafka → Flink → Kudu/Impala → SSB
Own architectural decisions, patterns, and best practices for streaming, CDC, state management, schema evolution, and exactly-once delivery
Develop complex NiFi flows involving controller services (DBCP/JDBC), stateful processors, record processors, schema registry integrations, batch-to-stream conversions, and high-volume ingestion patterns
Build and optimize Flink SQL or DataStream API jobs with: ? Kafka sources/sinks ? event-time windows ? watermarks ? state management ? checkpointing / savepoints ? exactly-once guarantees
Design and tune Kudu tables (PKs, partitioning, distribution, upserts, deletes, merges)
Build and deploy streaming SQL jobs using Cloudera SQL Stream Builder (SSB)
Deliver the following four core use cases immediately: 1. NiFi → Snowflake → Impala/Kudu ingestion pipeline 2. Kafka → Flink streaming (real-time processing) 3. Flink → Kafka sink with exactly-once semantics 4. CDC ingestion via NiFi, Flink CDC, or SSB (incremental keys, late events, deletes)
Tune NiFi, Kafka, and Flink clusters for performance, throughput, and stability
Implement schema governance, error handling, back-pressure strategies, and replay mechanisms
Work closely with platform engineers to optimize CDP components and CDF deployments
Provide architectural guidance, documentation, and mentorship to engineering teams
Qualification
Required
Hands-on, production-grade experience with Cloudera CDP / CDF including CDP Public Cloud or Private Cloud Base
Cloudera Flow Management (NiFi + NiFi Registry)
Cloudera Streams Messaging (Kafka, SMM)
Cloudera Stream Processing (Flink, SSB)
Kudu / Impala ecosystem
Advanced Apache NiFi experience including building complex flows, QueryDatabaseTable / GenerateTableFetch / MergeRecord, record-based processors & schema registry, JDBC / DBCP controller services, stateful processors & incremental ingestion, NiFi → Snowflake integration, NiFi → Kudu ingestion patterns
Apache Kafka experience including Kafka brokers, partitions, retention, replication, consumer groups, schema registry (Avro/JSON), designing topics for high-throughput streaming
Apache Flink experience including Flink SQL + DataStream API, event-time processing, watermarks, windows, checkpointing, savepoints, state backends, Kafka source/sink connectors, exactly-once semantics, Flink CDC a plus
Apache Kudu experience including table design (PKs, partition strategies), upserts, deletes, merge semantics, integration with Impala
SQL Stream Builder (SSB) experience including creating jobs, connectors, materialized views, deploying and monitoring Flink SQL jobs in CDP
CDC (Change Data Capture) experience including CDC via NiFi or Flink CDC or SSB, handling late-arriving events, handling deletes, updates, schema evolution, incremental key tracking
8+ years in data engineering / streaming
3–5+ years specifically with CDP/CDF streaming
Strong SQL and distributed system fundamentals
Preferred
Experience in financial services, healthcare, telecom, or other high-volume industries
Kubernetes experience running NiFi/Kafka/Flink operators
Snowflake ingestion patterns (staging, Copy Into)
Experience with Debezium
CI/CD for data pipelines
Security (Kerberos, Ranger, Atlas)
Company
HCL Global Systems Inc
HCL Global Systems is a staffing and recruiting company providing consulting and business solutions.
1001-5000 employees
H1B Sponsorship
HCL Global Systems Inc has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (265)
2024 (285)
2023 (378)
2022 (390)
2021 (470)
2020 (736)
Funding
Current Stage
Late StageLeadership Team
A
Andy HCL
Cheif Marketing Officer
Company data provided by crunchbase