Software Guidance & Assistance, Inc. (SGA, Inc.) · 11 hours ago
Senior Data Engineer
New York, NY
Full-time
Onsite
Senior Level
$90/hr - $110/hrSoftware Guidance & Assistance, Inc. (SGA) is a technology and resource solutions provider seeking a Senior Data Engineer for a contract assignment with a premier Financial Services client. The role involves migrating applications to the cloud, developing scalable ETL/ELT pipelines, and collaborating with stakeholders to meet data requirements.
Information TechnologyStaffing Agency
Hiring Manager
Dean McGowan
Responsibilities
Work on migrating applications from an on-premises location to the cloud service providers
Develop products and services on the latest technologies through contributions in development, enhancements, testing and implementation
Develop, modify, extend code for building cloud infrastructure, and automate using CI/CD pipeline
Partners with business and peers in the pursuit of solutions that achieve business goals through an agile software development methodology
Perform problem analysis, data analysis, reporting, and communication
Work with peers across the system to define and implement best practices and standards
Assess applications and help determine the appropriate application infrastructure patterns
Use the best practices and knowledge of internal or external drivers to improve products or services
Design, develop, and optimize scalable ETL/ELT pipelines using ETL/ELT pipelines using Databricks and Apache Spark (PySpark and Scala)
Build data ingestion workflows from various sources (structured, semi-structured, and unstructured)
Develop reusable components and frameworks for efficient data processing
Implement best practices for data quality, validation, and governance
Collaborate with data architects, analysts, and business stakeholders to understand data requirements
Tune Spark jobs for performance and scalability in a cloud-based environment
Maintain robust data lake or Lakehouse architecture
Ensure high availability, security, and integrity of data pipelines and platforms
Support troubleshooting, debugging, and performance optimization in production workloads
Qualification
Required
Develop, and optimize scalable ETL/ELT pipelines using Databricks and Apache Spark (PySpark and Scala)
Experience in developing data pipeline solutions to ingest and exploit new and existing data sources
Expertise in leveraging SQL, programming language like Python and ETL tools like Databricks
Perform code reviews to ensure requirements, optimal execution patterns and adherence to established standards
Expertise in AWS Compute (EC2, EMR), AWS Storage (S3, EBS), AWS Databases (RDS, DynamoDB), AWS Data Integration (Glue)
Advanced understanding of Container Orchestration services including Docker and Kubernetes, and a variety of AWS tools and services
Good understanding of AWS Identify and Access management, AWS Networking and AWS Monitoring tools
Proficiency in CI/CD and deployment automation using GITLAB pipeline
Proficiency in Cloud infrastructure provisioning tools e.g., Terraform
Proficiency in one or more programming languages e.g., Python, Scala
Experience in Starburst, Trino and building SQL queries in federated architecture
Good knowledge of Lake house architecture
Company
Software Guidance & Assistance, Inc. (SGA, Inc.)
For more than 40 years, SGA has delivered technology and resource solutions that Fortune 500 organizations, mid-size companies, state and local governments, and higher education institutions rely on to achieve their goals/missions.
Founded in 1981
201-500 employees
H1B Sponsorship
Software Guidance & Assistance, Inc. (SGA, Inc.) has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (13)
2024 (12)
2023 (19)
2022 (25)
2021 (41)
2020 (44)
Funding
Current Stage
Growth StageLeadership Team
Peter Taylor
CEO
Andrew Gallagher - CCWP
Senior Vice President - Strategic Partnerships
Company data provided by crunchbase