APN Consulting, Inc. · 6 hours ago
Data Engineer – Databricks
Oldwick, NJ
Full-time
Hybrid
Mid Level
4+ years expAPN Consulting, Inc. is a progressive IT staffing and services company offering innovative business solutions. They are seeking a Data Engineer to design, build, and optimize scalable data solutions, engaging with cross-functional teams to enhance data pipelines and ensure data reliability.
ConsultingFacility ManagementHealth CareInformation TechnologyIT ManagementService IndustryStaffing Agency
Responsibilities
Assist with leading the team's transition to the Databricks platform and utilize the newer features of Delta Live Tables, Workflows etc
Design and develop data pipelines that extract data from Oracle, load it into the data lake, transform it into the desired format, and load it into Databricks data lakehouse
Optimize data pipelines and data processing workflows for performance, scalability, and efficiency
Implement data quality checks and validations within data pipelines to ensure the accuracy, consistency, and completeness of data
Help create and maintain documentation for data mappings, data definitions, architecture and data flow diagrams
Build proof-of-concepts to determine viability of possible new processes and technologies
Deploy and manage code in non-prod and prod environments
Investigate and troubleshoot data related issues and fix or provide solutions to fix defects
Identify and resolve performance bottlenecks, which could include suggesting ways to optimize and performance tune databases and queries to enhance query performance
Qualification
Required
Bachelor's Degree in Computer Science, Data Science, Software Engineering, Information Systems, or related quantitative field
4 plus years of experience working as a Data Engineer, ETL Engineer, Data/ETL Architect or similar roles
Must hold a current/active Databricks Data Engineer/Analyst certification
4+ years of solid continuous experience in Python
3+ years working with Databricks with knowledge and expertise of data structures, data storage and change data capture gained from prior production implementations of data pipelines, optimizations, and best practices
3+ years of experience in Kimball dimensional modeling (star-schema comprising of facts, type1 and type2 dimensions, aggregates, etc.) with solid understanding of ELT/ETL
3+ years of solid experience writing SQL and PL/SQL code
2+ years of experience with Airflow
3+ years of experience working with relational databases (Oracle preferred)
2+ years of experience working with NoSQL databases: MongoDB, Cosmos DB, DocumentDB or similar
2+ years of cloud experience (Azure preferred)
Experience with CI/CD utilizing git/Azure DevOps
Experience with storage formats including Parquet/Arrow/Avro
Effectively collaborate with team members while being able to work independently with minimal supervision
Must have a creative mindset, knack to solve complex problems, passion to work with data, and a positive attitude
Ability to collaborate within and across teams of different technical knowledge to support delivery and educate end users on data products
Expert problem-solving skills, including debugging skills, allowing the determination of sources of issues in unfamiliar code or systems
Preferred
ETL / ELT tools: Spark, Kafka, Azure Data Factory (ADF)
Languages: R, Java, Scala
Databases: Redis, Elasticsearch
Company
APN Consulting, Inc.
APN Consulting, Inc. is a technology and talent solutions company helping enterprises maximize the value of their IT investments.
201-500 employees
H1B Sponsorship
APN Consulting, Inc. has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (2)
2024 (5)
2022 (1)
2021 (1)
2020 (2)
Funding
Current Stage
Growth StageLeadership Team
Vedant Pathak
Founder/CEO
Company data provided by crunchbase