Apply on Employer Site

Kaygen, Inc. · 2 months ago

Sr Data Engineer -- pipeline

San Diego, CA

Full-time

Onsite

Senior Level

$90/hr - $95/hr

KAYGEN is an emerging leader in providing top talent for technology-based staffing services. They are seeking a Senior Data Engineer to design, develop, and optimize data pipelines using Databricks and Medallion architecture, ensuring high-quality data flow from diverse sources.

AnalyticsBig DataData ManagementInformation ServicesInformation TechnologyStaffing Agency

Growth Opportunities

H1B Sponsor Likely

Responsibilities

Design, develop, and optimize data pipelines using Databricks and Medallion architecture for structured and unstructured data

Implement data ingestion strategies from diverse sources (SAP, TrackWise, Salesforce, and others) ensuring high-quality, reliable data flow

Ingest and process large volumes of data from Azure Blob Storage into Databricks for transformation and analytics

Build and maintain silver and gold layers for curated, analytics-ready datasets

Apply advanced statistical methods , including: Anomaly Detection for identifying irregular patterns in operational and quality data. Forecasting for demand, defect trends, and resource planning. Root Cause Analysis to identify drivers of failures or complaints. Risk Modeling to quantify and predict potential quality or operational risks. Predictive Analytics for proactive decision-making and maintenance. Pattern Detection to uncover hidden relationships in large datasets

Develop and deploy NLP solutions for text-based data analysis, leveraging linguistic schemas for structured interpretation of unstructured data

Create Power BI dashboards with: Workflow automation (notifications, reminders, messaging). AI-driven insights using prepped datasets for predictive and prescriptive analytics

Implement near real-time tracking systems and alert mechanisms for critical events and operational KPIs

Collaborate with cross-functional teams to align data solutions with business objectives and ensure visibility across the end-to-end operating cycle

Document data workflows, maintain traceability, and ensure audit readiness for all data engineering activities

Leverage knowledge of Quality Management Systems (QMS) to support compliance and continuous improvement

Qualification

Data pipelinesDatabricksMedallion architectureStatistical methodsData ingestion strategiesAzure Blob StorageNLP solutionsPower BI dashboardsQuality Management SystemsAnomaly DetectionForecastingRoot Cause AnalysisRisk ModelingPredictive AnalyticsPattern DetectionWorkflow automationAI-driven insightsEnd-to-end operating cycle

Required

Design, develop, and optimize data pipelines using Databricks and Medallion architecture for structured and unstructured data

Implement data ingestion strategies from diverse sources (SAP, TrackWise, Salesforce, and others) ensuring high-quality, reliable data flow

Ingest and process large volumes of data from Azure Blob Storage into Databricks for transformation and analytics

Build and maintain silver and gold layers for curated, analytics-ready datasets

Apply advanced statistical methods, including: Anomaly Detection for identifying irregular patterns in operational and quality data

Forecasting for demand, defect trends, and resource planning

Root Cause Analysis to identify drivers of failures or complaints

Risk Modeling to quantify and predict potential quality or operational risks

Predictive Analytics for proactive decision-making and maintenance

Pattern Detection to uncover hidden relationships in large datasets

Develop and deploy NLP solutions for text-based data analysis, leveraging linguistic schemas for structured interpretation of unstructured data

Create Power BI dashboards with: Workflow automation (notifications, reminders, messaging)

AI-driven insights using prepped datasets for predictive and prescriptive analytics

Implement near real-time tracking systems and alert mechanisms for critical events and operational KPIs

Collaborate with cross-functional teams to align data solutions with business objectives and ensure visibility across the end-to-end operating cycle

Document data workflows, maintain traceability, and ensure audit readiness for all data engineering activities

Leverage knowledge of Quality Management Systems (QMS) to support compliance and continuous improvement

Benefits

Free Healthcare Insurance

Vision and Dental Insurance

401(k) Retirement Plan

Free Life Insurance

Sick Time Off

Certifications

Mentorship Program

Referrals

Family and Wellness benefits

Continuous Growth and Career Development

Company

Kaygen, Inc.

Kaygen, an award-winning technology consulting and talent firm, assists organizations to achieve digital transformation.

Founded in 2003

Irvine, California, USA

51-200 employees

https://kaygen.com

H1B Sponsorship

Kaygen, Inc. has a track record of offering H1B sponsorships. Please note that this does not guarantee sponsorship for this specific role. Below presents additional info for your reference. (Data Powered by US Department of Labor)

Distribution of Different Job Fields Receiving Sponsorship

Represents job field similar to this job

Trends of Total Sponsorships

2025 (2)

2024 (4)

2023 (2)

2022 (3)

2021 (8)

2020 (16)

Funding

Current Stage

Growth Stage

Leadership Team

Anshul Chaturvedi

Founder & CEO

Rashmi Chaturvedi

Co-founder & President

Company data provided by crunchbase