Ender-IT · 5 days ago
DataBricks Data Engineer
Oldwick, NJ
Full-time
Onsite
Mid, Senior Level
4+ years expEnder-IT is seeking a DataBricks Data Engineer to assist with the transition to the Databricks platform. The role involves designing and developing data pipelines, optimizing workflows, and ensuring data quality within the data engineering processes.
Staffing & Recruiting
Hiring Manager
Akash Kasaragod
Responsibilities
Assist with leading the team's transition to the Databricks platform and utilize the newer features of Delta Live Tables, Workflows etc
Design and develop data pipelines that extract data from Oracle, load it into the data lake, transform it into the desired format, and load it into Databricks data lake house
Optimize data pipelines and data processing workflows for performance, scalability, and efficiency
Implement data quality checks and validations within data pipelines to ensure the accuracy, consistency, and completeness of data
Help create and maintain documentation for data mappings, data definitions, architecture and data flow diagrams
Build proof-of-concepts to determine viability of possible new processes and technologies
Deploy and manage code in non-prod and prod environments
Investigate and troubleshoot data related issues and fix or provide solutions to fix defects
Identify and resolve performance bottlenecks, which could include suggesting ways to optimize and performance tune databases and queries to enhance query performance
Qualification
Required
Bachelor's Degree in Computer Science, Data Science, Software Engineering, Information Systems, or related quantitative field
4+ years of experience working as a Data Engineer, ETL Engineer, Data/ETL Architect or similar roles
3+ years working with Databricks with knowledge and expertise of data structures, data storage and change data capture gained from prior production implementations of data pipelines, optimizations, and best practices
Solid continuous experience in Python
3+ years of experience in Kimball dimensional modeling (star-schema comprising of facts, type1 and type2 dimensions, aggregates, etc.) with solid understanding of ELT/ETL
3+ years of solid experience writing SQL and PL/SQL code
2+ years of experience with Airflow
2+ years of experience working with NoSQL databases: MongoDB, Cosmos DB, DocumentDB or similar
2+ years of experience with CI/CD utilizing git/Azure DevOps
Experience with storage formats including Parquet/Arrow/Avro
Preferred
Current/active Databricks Data Engineer/Analyst certification a BIG plus
3+ years working with relational databases (Oracle preferred)
2+ years of cloud experience (Azure preferred)
Company
Ender-IT
The Intersection of Talent, Health, and Technology Ender IT is a specialized staffing partner dedicated to providing mission-critical talent across the most demanding industry verticals.
11-50 employees
Funding
Current Stage
Early StageLeadership Team
Jaspal Singh Nandra
Founder| Startup Pioneer
Company data provided by crunchbase