Software Guidance & Assistance, Inc. (SGA, Inc.) · 12 hours ago
Big Data Engineer
Rockville, MD
Full-time
Hybrid
Senior Level
$70/hr - $76/hr
5+ years expSoftware Guidance & Assistance, Inc. (SGA) is searching for a Big Data Engineer for a Contract assignment with one of their premier Regulatory clients. The role involves designing, developing, and maintaining large-scale data processing pipelines using Big Data technologies while collaborating with cross-functional teams to meet business requirements.
Information TechnologyStaffing Agency
Responsibilities
Design, develop, and maintain large-scale data processing pipelines using Big Data technologies (e.g., Hadoop, Spark, Python, Scala)
Implement data ingestion, storage, transformation, and analysis of solutions that are scalable, efficient, and reliable
Stay current with industry trends and emerging Big Data technologies to continuously improve the data architecture
Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions
Optimize and enhance existing data pipelines for performance, scalability, and reliability
Develop automated testing frameworks and implement continuous testing for data quality assurance
Conduct unit, integration, and system testing to ensure the robustness and accuracy of data pipelines
Work with data scientists and analysts to support data-driven decision-making across the organization
Ability to write and maintain automated unit, integration, and end-to-end tests
Monitor and troubleshoot data pipelines in production environments to identify and resolve issues
Qualification
Required
Bachelor's degree in Computer Science, Information Systems or related discipline with at least five (5) years of related experience, or equivalent training and/or work experience; Master's degree and past Financial Services industry experience preferred
Demonstrated technical expertise in Object Oriented and database technologies/concepts which resulted in deployment of enterprise quality solutions
Past experience with developing enterprise quality solutions in an iterative or Agile environment
Extensive knowledge of industry leading software engineering approaches including Test Automation, Build Automation and Configuration Management frameworks
Strong written and verbal technical communication skills
Demonstrated ability to develop effective working relationships that improved the quality of work products
Should be well organized, thorough, and able to handle competing priorities
Ability to maintain focus and develop proficiency in new skills rapidly
Ability to work in a fast paced environment
Experience with object oriented programming languages such as Java, Scala or Python
Experience with Big data technologies such as Hadoop, Spark, Hive & Trino
Evaluate understanding of common issues like: Data skew and strategies to mitigate it
Working with massive data volumes in PetaBytes
Troubleshooting job failures due to resource limitations, bad data, scalability challenged
Proficiency in crafting effective prompts for AI coding assistants and analysis tools
Experience redesigning development processes to leverage AI capabilities
Ability to interpret AI-generated insights and translate them into actionable team improvements
Experience leading teams through AI adoption and workflow transformation
Assess comfort with SQL window functions, multi-table joins, aggregations
Provide examples or ask them to write/optimize SQL queries on the spot
Probe how they handle edge cases like NULLs, duplicates, ordering, etc
Test their understanding of Spark's core architecture — executors, tasks, stages, DAG
Focus on Spark performance tuning techniques: partitioning, caching, broadcast joins, etc
Ask scenario-based questions on troubleshooting slow running/stuck jobs or resource issues in Spark
Explore their experience optimizing Spark jobs for large-scale datasets
Check exposure to AWS services like S3, EMR, Glue, Lambda, Athena, etc
Ask how they've used S3 with Spark (e.g., dealing with file formats, consistency issues)
Assess ability to write clean, modular, and performant code
Look for experience in functional programming concepts (e.g., immutability, higher-order functions)
Ask about real-world use cases where they wrote scalable data processing code
Evaluate understanding of collections, concurrency, and memory management
Preferred
Experience with managing production data pipelines/ETL systems
Experience with CI/CD
Experience writing test cases
AWS certifications
Company
Software Guidance & Assistance, Inc. (SGA, Inc.)
For more than 40 years, SGA has delivered technology and resource solutions that Fortune 500 organizations, mid-size companies, state and local governments, and higher education institutions rely on to achieve their goals/missions.
201-500 employees
H1B Sponsorship
Software Guidance & Assistance, Inc. (SGA, Inc.) has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (13)
2024 (12)
2023 (19)
2022 (25)
2021 (41)
2020 (44)
Funding
Current Stage
Growth StageLeadership Team
Peter Taylor
CEO
Andrew Gallagher - CCWP
Senior Vice President - Strategic Partnerships
Company data provided by crunchbase