Apply on Employer Site

Tata Consultancy Services · 6 hours ago

Databricks Engineer

Fallon, NV

Full-time

Onsite

Senior Level, Lead/Staff

$100K/yr - $120K/yr

8+ years exp

Tata Consultancy Services is a leading global IT services, consulting, and business solutions organization. They are seeking a Databricks Engineer to design, develop, and optimize data pipelines and workflows, ensuring high-quality data solutions for their clients.

Business Information SystemsConsultingInformation TechnologyIT Management

H1B Sponsor Likely

Responsibilities

Focus on standing up/adopting tools that will enable both internal and external customers. Key leader/contributor in the data and software and platform and data warehouse development life cycle (i.e. design, development, documentation, testing, deployment, support)

Work efficiently within a high security, PII and PCI-DSS environments

Key leader/contributor in the data and software and platform and data warehouse development life cycle (i.e. design, development, documentation, testing, deployment, support)

Support architects and engineers as they design and build effective, agile applications and platforms

Design, develop, and optimize batch and real-time data pipelines using Medallion Architecture, preferably on Databricks cloud data platforms

Build data transformation workflows using dbt, with a strong focus on Test-Driven Development (TDD) and modular design

Implement and manage CI/CD pipelines using GitLab and Jenkins, enabling automated testing, deployment, and monitoring of data workflows

Drive DataOps practices by integrating testing, monitoring, versioning, and collaboration into every phase of the data pipeline lifecycle

Build scalable and reusable data models that support business analytics and dashboarding in Power BI

Develop and support real-time data streaming pipelines (e.g., using Kafka, Spark Structured Streaming) for near-instant data availability

Establish and implement data observability practices, including monitoring data quality, freshness, lineage, and anomaly detection across the platform

Plan and own deployments, migrations, and upgrades across data platforms and pipelines to minimize service impacts, including developing and executing mitigation plans

Collaborate with stakeholders to understand data requirements and deliver reliable, high-impact data solutions

Document pipeline architecture, processes, and standards, promoting consistency and transparency across the team

Apply exceptional problem-solving and analytical skills to troubleshoot complex data and system issues

Demonstrate excellent written and verbal communication skills when collaborating across technical and non-technical teams

Qualification

DatabricksPythonSpark SQLCI/CDDbtPower BICloud databasesData observabilityReal-time processingCommunication skills

Required

Experience Required - 8+ Years

Tenured in the fields of Computer Science/Engineering or Software Engineering

Bachelor's degree in computer science, or a related technical field including programming with 15+ years of demonstrated experience in software development and delivery

Deep hands-on experience with Databricks and Python

Spark and SQL Expertise: Writing performant Spark SQL and PySpark transformations. Optimizing joins, window functions, and aggregations. Handling large-scale data with partitioning and caching strategies

Experience with real-time data processing and streaming pipelines

Proficient in both Delta Live Tables (DLT) and Workflows – Understanding of Lakeflow Declarative Pipelines is an asset

DevOps and CI/CD - Using Databricks Asset Bundles (DABs) for deployment - Integrating with Git, Bitbucket, and Jenkins for version control and automation

Proven track record using dbt for building robust, testable data transformation workflows following TDD

Strong grasp of cloud-based database infrastructure (AWS, Azure, or GCP)

Skilled in developing insightful dashboards and scalable data models using Power BI

Expert in SQL development and performance optimization

Demonstrated success in building and maintaining data observability tools and frameworks

Proven ability to plan and execute deployments, upgrades, and migrations with minimal disruption to operations