Confidential Jobs · 4 days ago
SAP Qualtrics Data Integration Engineer
Confidential Jobs is seeking an AWS Data Engineer specializing in Qualtrics Integration. This role involves designing, building, and maintaining scalable data pipelines that support Qualtrics survey ingestion and reporting, with a focus on serverless AWS data engineering and API integrations.
Responsibilities
Design and maintain ETL pipelines using AWS Glue (PySpark)
Develop AWS Lambda functions (Python / Node.js) for serverless data processing
Manage AWS S3 for Qualtrics input/output storage and optimized data access
Orchestrate workflows using AWS Step Functions and MWAA (Airflow)
Query structured datasets using AWS Athena
Monitor pipelines using CloudWatch logs and metrics
Transform and aggregate Qualtrics datasets (CSV, JSON, XML)
Merge multiple source files (e.g., dealer master + employee files) into unified hierarchies
Automate ingestion, transformation, validation, and report generation
Implement reusable, scalable ETL frameworks
Integrate with Qualtrics APIs to extract raw survey data and response files
Implement event-driven processing (S3 triggers → Lambda)
Connect AWS pipelines with CRM, ERP, BI tools , or downstream platforms
Perform data validation and quality checks prior to Qualtrics ingestion
Generate formatted output files aligned to business-defined templates
Support ad-hoc analysis using SQL and Athena
Develop robust Python scripts for Glue, Lambda, and automation tasks
Write optimized SQL queries for structured data access
Use Bash/Shell scripting for file movement and preparation
Configure IAM roles and permissions securely
Implement Infrastructure as Code (Terraform / CloudFormation)
Support CI/CD pipelines for data workflows
Qualification
Required
Strong experience in AWS Data Engineering & Serverless Architecture
Hands-on expertise with AWS Glue, Lambda, S3, Athena, Step Functions
Experience with MWAA (Airflow) for orchestration
Strong Python and SQL skills
Experience integrating Qualtrics API or structured survey data
Ability to troubleshoot pipeline failures and performance issues
Preferred
Experience with Terraform or CloudFormation
CI/CD for data pipelines
Experience supporting BI tools (Power BI, Tableau, etc.)
Knowledge of data governance and security best practices