Tredence Inc. · 2 weeks ago
Principal Hadoop Architect
San Jose, CA
Full-time
Hybrid
Lead/StaffTredence Inc. is seeking a Principal Hadoop Architect to serve as the central authority for a large-scale Big Data ecosystem. The role involves defining standards for data ingestion, storage, and processing, while ensuring optimization for cloud evolution and maintaining architectural integrity.
AnalyticsArtificial Intelligence (AI)ConsultingInformation Technology
Hiring Manager
Aamir Malik
Responsibilities
Define and enforce 'Blueprints' for Hive schemas, Spark configurations, and Kafka topics to be used across all engineering and analyst teams
Maintain the official 'Big Data Playbook,' detailing approved design patterns for batch vs. real-time processing
Lead the Architecture & Design Review Board (ADRB) to vet new data projects, ensuring they don't introduce technical debt or inefficient resource patterns
Identify 'heavy-hitter' queries and inefficient YARN resource allocations. Implement mandatory partitioning and bucketing standards to reduce HDFS overhead
Implement tiered storage (Hot/Warm/Cold) policies. Enforce standard file formats (Parquet/Avro) to optimize compression and predicate push-down
Establish data retention and archival standards to prevent 'Data Swamp' growth, ensuring we only store what provides value
Lead the effort to decouple storage (HDFS) from compute (YARN) through architectural standards, making future cloud migration a 'plug-and-play' exercise
Encourage the use of abstraction layers and APIs so that downstream applications aren't hard-coded to specific Hadoop versions
Provide guidance on moving localized workloads toward Kubernetes/Docker-friendly designs
Design a robust 'Quotas and Queues' system to ensure a single team's rogue Spark job doesn't crash the cluster for everyone else
Standardize Apache Ranger policies and Kerberos implementation across all nodes
Qualification
Required
Expert Level Hadoop: Mastery of the Cloudera/Hortonworks stack, specifically Hive LLAP, YARN, and HDFS
Standardization Experience: Proven track record of creating Enterprise Design Standards used by multiple engineering teams
Processing Frameworks: Deep knowledge of Spark (Core/SQL) optimization and Kafka event-driven architecture
Tooling Mastery: Experience with Apache foundation services such as Apache Atlas for lineage and Apache Ranger for centralized security
Soft Skills: Ability to influence senior leadership and guide diverse engineering teams without direct reporting authority
Company
Tredence Inc.
Tredence is a global data science solutions provider focused on solving the last mile problem in AI.
Founded in 2013
1001-5000 employees
H1B Sponsorship
Tredence Inc. has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (143)
2024 (103)
2023 (103)
2022 (74)
2021 (69)
2020 (75)
Funding
Current Stage
Late StageTotal Funding
$205MKey Investors
Advent InternationalChicago Pacific Founders
2022-12-22Series B· $175M
2020-12-10Series A· $30M
Leadership Team
Shub Bhowmick
Co-founder and CEO
Shashank Dubey
Co-Founder & Chief Revenue Officer
Recent News
2026-01-06
2025-11-13
Company data provided by crunchbase