Apply on Employer Site

Curotec · 5 months ago

Senior Data Engineer

United States

Full-time

Remote

Senior Level

Curotec is an IT Services company seeking a Senior Data Engineer to support the ingestion, processing, and synchronization of data across their analytics platform. This role focuses on using Python Notebooks to ingest data via APIs into Microsoft Fabric's Data Lake and Data Warehouse, ensuring data availability and accuracy for reporting needs.

Artificial Intelligence (AI)Cloud Data ServicesCloud ManagementConsultingEnterprise ApplicationsInformation TechnologyOutsourcingSaaSSoftwareSoftware Engineering

Growth Opportunities

Responsibilities

Build and maintain Python Notebooks to ingest data from third-party APIs

Design and implement Medallion layer architecture (Bronze, Silver, Gold) for structured data organization and progressive data refinement

Store and manage data within Microsoft Fabric's Data Lake and Warehouse using delta parquet file formats

Set up data pipelines and sync key datasets to Azure Synapse Analytics

Develop PySpark-based data transformation processes across Bronze, Silver, and Gold layers

Collaborate with developers, analysts, and stakeholders to ensure data availability and accuracy

Monitor, test, and optimize data flows for reliability and performance

Document processes and contribute to best practices for data ingestion and transformation

Qualification

PythonPySparkMedallion architectureMicrosoft FabricAzure Synapse AnalyticsRESTful APIsDelta LakeData warehousingCloud workflowsData modelingAI coding toolsFivetranAribyteRiverly