Credible · 1 day ago
Senior Data Engineer
New York, NY
Full-time
Remote
Mid, Senior Level
3+ years exp
Maximize your interview chancesComputer Software
Insider Connection @Credible
Get 3x more responses when you reach out via email instead of LinkedIn.
Responsibilities
Collaborate with cross-functional teams to define the data engineering strategy aligned to business objectives, including data modeling that unifies data assets across a range of source systems used to manage the operations of our partnering hospitals.
Define and execute processes needed to develop, test, deploy, and maintain high quality data pipelines.
Oversee the end-to-end development of data pipelines from source data extraction through to production-grade analytical dataset delivery, ensuring data quality and security throughout the pipeline.
Continuously monitor and optimize data processing performance and efficiency.
Identify and address bottlenecks, optimize query performance, and improve overall system stability.
Establish and enforce data quality management policies, data access controls, and data privacy standards.
Stay abreast of the latest developments in engineering tools and best practices.
Provide guidance to the team about technical challenges.
Maintain clear and comprehensive documentation of data pipelines, architecture, and processes to ensure knowledge sharing and team continuity.
Evaluate and manage relationships with third-party vendors and tools, making informed decisions about when to leverage external solutions.
Qualification
Find out how your skills align with this job's requirements. If anything seems off, you can easily click on the tags to select or unselect skills to reflect your actual expertise.
Required
3+ years in data engineering roles in a production environment
Advanced proficiency in Python and SQL for data engineering
Up-to-date knowledge of and 1+ years of experience using Databricks for Lakehouse management
Deep understanding of data modeling, data architecture, and data integration best practices
Strong hands-on experience with Apache Spark
Familiarity with data governance, security, and privacy principles
Comfort using git or equivalent to manage the software development life cycle
Exceptional ability to learn and use new software development techniques and tools
Ability to manage multiple projects simultaneously
High energy, humble team player with “get it done” attitude, seeking collaboration with colleagues
Preferred
Experience with the Azure cloud ecosystem
Experience developing production-ready, real-time machine learning model serving pipelines
Comfort developing in the Apache Spark Structured Streaming paradigm
Experience working in a private equity-backed services company
Experience deploying machine learning models with MLFlow or equivalent
Experience developing CI/CD pipelines
Company
Credible
Welcome to Credible, the next generation of ATS platforms, providing employers with cutting-edge technology to find their next great hire in as little as one day.
11-50 employees
Funding
Current Stage
Early StageCompany data provided by crunchbase
