Apply on Employer Site

INSPYR Solutions · 3 hours ago

Data Engineer

Sunrise, FL

Contract

Onsite

Senior Level

$55/hr - $60/hr

5+ years exp

INSPYR Solutions is a national expert in delivering flexible technology and talent solutions. They are seeking a Data Engineer to administer and optimize their Databricks Lakehouse environment, develop data platform infrastructure, and implement data ingestion patterns.

Information TechnologyProfessional ServicesStaffing Agency

No H1B

Responsibilities

Administer, optimize, and scale our Databricks Lakehouse environment, ensuring high performance, cost efficiency, and operational excellence

Develop, maintain, and enhance our data platform infrastructure and security configurations using Terraform . This includes provisioning Databricks workspaces, SQL Endpoints, Unity Catalog objects, and network components

Manage and enforce Unity Catalog for data governance, access control, and metadata management

Implement and manage CI/CD pipelines for data pipelines, dbt projects, and infrastructure deployments using GitHub Actions

Automate operational tasks, monitoring, and alerting for the data platform

Implement and enforce DevSecOps principles, working closely with security teams to ensure compliance and manage/rotate credentials securely

Design and implement data ingestion patterns into Databricks using Delta Lake , optimizing for large-scale data processing and storage

Develop, optimize, and troubleshoot complex Spark jobs (PySpark/Scala) for data processing and transformation within Databricks

Manage and extend data ingestion pipelines using Airbyte (or similar modern tools like Fivetran, Stitch), including configuring connectors, monitoring syncs, and ensuring data quality and reliability from diverse source systems (e.g., ERP, CRM, marketing, supply chain)

Orchestrate and automate data pipelines and dbt models using Databricks Workflows and potentially integrating with other orchestration tools

Collaborate with Analytics Engineers to translate business requirements into efficient and scalable data models using dbt (Data Build Tool)

Implement dbt best practices for modularity, testing, documentation, and version control, ensuring seamless integration with Databricks

Partner effectively with Analytics Engineers, Data Scientists, and business stakeholders to deliver high-quality data solutions

Provide technical guidance and mentorship to junior team members, and champion data engineering best practices, code quality, and documentation standards