RSM Solutions, Inc ยท 2 weeks ago
Data Integration Engineer
RSM Solutions, Inc is a company focused on recruiting technical talent and is seeking a Data Integration Engineer. In this role, you will design, build, and operate data pipelines to ensure data is accurate and analytics-ready, working closely with a team of data-centric individuals.
Responsibilities
Design, develop, and deploy incremental and full load pipelines using SSIS, Spark, Runbooks and Azure Data Factory to ingest data into landing, raw, and curated layers of the Data Vault
Build CDC (change data capture) solutions to minimize latency for downstream reporting and ML features
Automate schema evolution and metadata population for hubs, links, and satellites
Implement validation rules, unit tests, and data quality frameworks to enforce referential integrity and conformance to business rules
Maintain a requirements traceability matrix and publish data lineage documentation Metadata Management / SSAS models. This includes partnering with this team's BA to translate user stories into technical interfaces and mapping specs
Create CI/CD pipelines (Azure DevOps, Git) to version ETL code, infrastructure as code, and automated tests
Develop PowerShell/.NET utilities to orchestrate jobs, manage secrets, and push metrics to Grafana or Azure Monitor
Benchmark and tune Spark, SQL, and SSIS performance; recommend index strategies, partitioning, and cluster sizing strategies for cost/performance balance
Stay current with emerging integration patterns (e.g., event driven architectures, Delta Lake) and propose pilots for adoption
Qualification
Required
4+ years building data integration with MS SQL, SSIS, and Spark
At least 2 years of ML Cluster build experience
At least 2 years of experience with Data Vault
Strong T SQL, Python/Scala for Spark, PowerShell/.NET scripting; working knowledge of MongoDB aggregation, SSAS tabular models, and Git CI/CD
Excellent problem solving, communication, and stakeholder management abilities
Preferred
Data Vault 2.0 certification a plus