Nexus Cognitive · 2 months ago
Big Data Developer
United States
Full-time
Remote
Mid, Senior Level
4+ years expNexus Cognitive is focused on modernizing data ecosystems, and they are seeking a Big Data Developer to support the migration of legacy applications to open-source frameworks. The role involves optimizing data processing pipelines and ensuring alignment with enterprise data standards.
AnalyticsBusiness IntelligenceCloud ComputingInformation TechnologyInternet of ThingsMachine LearningSoftware
Responsibilities
Analyze, refactor, and modernize Spark/MapReduce/Hive/Tez jobs for execution within NexusOne’s managed Spark and Trino environments
Design, build, and optimize batch and streaming pipelines using Spark, NiFi, and Kafka
Convert existing ETL jobs and DAGs from Cloudera/MAPR ecosystems to open-source equivalents
Collaborate with Data Engineers and Architects to define new data ingestion and transformation patterns
Tune performance across large-scale data processing workloads (partitioning, caching, resource allocation)
Implement data quality and validation frameworks to ensure consistency during migration
Support code reviews, performance tests, and production readiness validation for migrated workloads
Document conversion approaches, dependencies, and operational runbooks
Partner with Wells Fargo application SMEs to ensure domain alignment and business continuity
Qualification
Required
4–8 years of experience in big data engineering or application modernization in enterprise settings
Strong technical expertise across distributed data systems, open-source frameworks, and hybrid data environments
Analyze, refactor, and modernize Spark/MapReduce/Hive/Tez jobs for execution within NexusOne's managed Spark and Trino environments
Design, build, and optimize batch and streaming pipelines using Spark, NiFi, and Kafka
Convert existing ETL jobs and DAGs from Cloudera/MAPR ecosystems to open-source equivalents
Collaborate with Data Engineers and Architects to define new data ingestion and transformation patterns
Tune performance across large-scale data processing workloads (partitioning, caching, resource allocation)
Implement data quality and validation frameworks to ensure consistency during migration
Support code reviews, performance tests, and production readiness validation for migrated workloads
Document conversion approaches, dependencies, and operational runbooks
Partner with Wells Fargo application SMEs to ensure domain alignment and business continuity
Core Frameworks: Apache Spark, PySpark, Airflow, NiFi, Kafka, Hive, Iceberg, Oozie
Programming Languages: Python, Scala, Java
Data Formats & Storage: Parquet, ORC, Avro, S3, HDFS
Orchestration & Workflow: Airflow, DBT
Performance Optimization: Spark tuning, partitioning strategies, caching, YARN/K8s resource tuning
Testing & Validation: Great Expectations, Deequ, SQL-based QA frameworks
Observability & Monitoring: Datadog, Grafana, Prometheus
Preferred
Prior experience with Cloudera, MAPR, or Hadoop ecosystems, transitioning to open-source frameworks
Exposure to hybrid or cloud-native environments (AWS, GCP, or Azure)
Familiarity with regulated environments (financial services, telecom, healthcare) is a plus
Company
Nexus Cognitive
Nexus Cognitive develops data analytics and digital-maturity software.
51-200 employees
H1B Sponsorship
Nexus Cognitive has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (1)
2024 (2)
2023 (1)
Funding
Current Stage
Growth StageTotal Funding
unknownKey Investors
Insight Partners
2024-06-04Series Unknown
Leadership Team
Steve Roberts
Partner
Recent News
Company data provided by crunchbase