Apply on Employer Site

Life Science Connect · 3 days ago

Staff Data Engineer (EL Focus - Azure/Snowflake)

Horsham, PA

Full-time

Onsite

Senior Level, Lead/Staff

7+ years exp

Life Science Connect is dedicated to uniting life sciences professionals and suppliers to accelerate research, development, and manufacturing. They are seeking a Staff Data Engineer to lead the technical backbone of their data ingestion strategy, focusing on building and optimizing data pipelines using Azure and Snowflake.

Content MarketingLife ScienceSalesSales Enablement

No H1B

Responsibilities

EL Pipeline Architecture & Execution

Ingestion Architecture: Own the design, development, and optimization of scalable data ingestion pipelines using Azure Data Factory (ADF). You will move beyond basic "drag-and-drop" configurations to build resilient, parameterized frameworks

Complex Source Integration: Design robust pipelines for high-volume, complex sources including Salesforce, Google Analytics (GA4), and internal APIs. You will be responsible for building custom connectors (using Python/Azure Functions) when native ADF connectors encounter API limits or sampling constraints

Snowflake Landing: Architect efficient loading patterns into Snowflake (Snowpipe, External Stages), ensuring that the "Raw Layer" is optimized for cost and performance before transformation begins

Analytics Collaboration & Schema Governance

The "dbt" Bridge: Act as the primary partner to the Analytics Engineering team. You will collaborate on Raw Layer schema design, ensuring that data lands in a structure that is easily consumable by dbt, preventing "garbage in" scenarios

Data Reliability: Provide the downstream teams with thoroughly documented reliable raw data feeds. You are the guarantee that the data in the warehouse matches the source of truth

Pipeline Orchestration & Optimization

Advanced Orchestration: Design dependency-aware pipeline orchestrations that manage the full data lifecycle, ensuring data arrives in the correct order and at the required frequency

Performance Tuning: Continuously monitor pipeline performance (latency, throughput) and optimize ADF resource allocation to control costs without sacrificing speed

Engineering Standards & Security

CI/CD Implementation: Define and lead the implementation of CI/CD pipelines for data workflows. You will enforce automated testing and deployment processes using Git/GitHub, treating infrastructure as code

Security & Compliance: Implement security best practices within the ingestion layer, specifically regarding Azure Key Vault for credential management and PIPL/GDPR compliance for PII handling

Qualification

Azure Data FactorySnowflakePythonSQLCI/CD ImplementationAPI IntegrationData IngestionStakeholder FocusPerformance TuningSchema Governance

Required

7+ years of professional experience in data engineering with a focus on high-volume production pipelines

Expert-level proficiency with Azure Data Factory (ADF)

Proficient in Python and SQL, capable of writing custom scripts for API interactions, data validation, and complex logic

Understanding of integrating with complex SaaS APIs (Salesforce, GA4), including handling rate limits, pagination, and token management programmatically

Extensive experience loading data into Snowflake and understanding the architectural implications of loading patterns on warehouse costs

Understanding that the 'customer' is the Analytics Engineer and delivering clean, reliable raw data