This job has closed.

U.S. Bank · 3 days ago

Senior Data Engineering Specialist

Charlotte, NC

Full-time

Hybrid

Senior Level, Lead/Staff

$105K/yr - $124K/yr

10+ years exp

U.S. Bank is on a journey to help customers and businesses make better financial decisions. The Senior Data Engineering Specialist will guide the migration to Azure cloud, design scalable data lake solutions, and manage data governance and security across cloud environments.

BankingFinancial ServicesInsuranceMortgageWealth Management

H1B Sponsor Likely

Responsibilities

Guide team to migrate from on Prem Cloudera to Azure cloud environment

Designed and implemented scalable data lake solutions using Snowflake and Databricks, Developed and optimized data pipelines for ingestion, transformation, and storage

Managed data governance, quality, and security across cloud environments and Implemented performance tuning, automation, and CI/CD for data workflows

Collaborated with cross-functional teams to support cloud migration activities

Install, configure, manage, and monitor Cloudera Hadoop clusters, ensuring high availability, performance, and security. This includes managing HDFS, YARN, and other ecosystem components

Tune Hadoop, Hive, and Spark jobs and configurations for optimal performance, efficiency, and resource utilization. This includes optimizing queries, managing partitions, and leveraging in-memory capabilities

Diagnose and resolve issues related to Linux servers, networks, cluster health, job failures, and performance bottlenecks. Provide on-call support and collaborate with other teams to ensure smooth operations

Implement and manage security measures within the Cloudera environment, including Kerberos, Apache Ranger, and Atlas, to ensure data governance and compliance

Setup and manage HashiCorp Vault for secure keys and secrets management

Utilize CyberArk for privileged access management and secure administrative tasks on the cluster

Migrate Hadoop, Hive, and Spark data and applications to Azure cloud services such as Azure Synapse Analytics, Azure Databricks, or Snowflake. Ensure data integrity, performance tuning, and validation

Develop scripts (e.g., shell, Ansible, Python) for automating administrative tasks, deployments, and monitoring. Work with users to develop, debug, optimize Hive/Spark/Python programs that connect to the Cloudera environment

Create and maintain documentation for system configurations, operational procedures, and troubleshooting knowledge bases

Work closely with the Cloudera vendor to stay current with the latest releases, perform upgrades, and address vulnerabilities

Qualification

Data Lake ArchitectureCloud MigrationHadoop EcosystemPerformance OptimizationData GovernanceAzure Cloud ServicesAutomationScriptingLinux ProficiencyCollaboration SkillsProblem-Solving SkillsDocumentation Skills

Required

Advanced degree in Computer Science, Engineering, or related field

Deep expertise in Data Engineering and Management technologies, synthetic data, automation, advanced analytics

10+ years of hands-on experience in Data engineering, Cloud platform management, and performance optimization

Hands-on experience with Hadoop, Hive, Spark, and migration of Big Data into Azure cloud services

Working knowledge and hands-on experience in Data Integration and Data Lake Architectures with Databricks and Snowflake platforms

Working knowledge of Microsoft Azure cloud and big data migration to cloud platforms

Experience with HashiCorp Vault for secrets and keys management

Experience with CyberArk for privileged access and administrative security tasks

Proficiency in Linux, clustering, and distributed systems

Expertise in Hive and Spark for data processing and analytics

Expertise in Hadoop ecosystem components such as HDFS, YARN, Hive, Spark, and Sqoop

Proficiency in languages such as shell, Ansible, C/C++, Java, and Python, Pyspark for automating workflows, deployments, and monitoring

Expertise in Linux, Network, Python scripting, DNS, Kerberos, LDAP/AD, MySQL, PostgreSQL, JupyterHub

Experience in creating and maintaining documentation for system configurations, operational procedures, and troubleshooting knowledge bases

Strong problem-solving skills and the ability to diagnose and resolve system failures and performance bottlenecks

Excellent communication and collaboration skills to work effectively with cross-functional teams

Preferred

Strong experience with Cloudera Hadoop distribution administration

Strong problem-solving, analytical skills

Deep understanding of banking, operations, risk, and compliance—pairing technical fluency with business and regulatory acumen

Experience on change management process and addressing PCI and PII data requirements

Experience in cloud-native architectures (Azure/AWS), Kubernetes, Docker, and containerized workflows

Experience in implementing secure systems with secure API design, IAM integration, Kerberos, Active Directory, and Azure SSO

Benefits

Healthcare (medical, dental, vision)

Basic term and optional term life insurance

Short-term and long-term disability

Pregnancy disability and parental leave

401(k) and employer-funded retirement plan

Paid vacation (from two to five weeks depending on salary grade and tenure)

Up to 11 paid holiday opportunities

Adoption assistance

Sick and Safe Leave accruals of one hour for every 30 worked, up to 80 hours per calendar year unless otherwise provided by law

Company

U.S. Bank

Glassdoor3.6

At U.S.

Founded in 1968

Minneapolis, Minnesota, USA

10001+ employees

http://www.usbank.com

H1B Sponsorship

U.S. Bank has a track record of offering H1B sponsorships. Please note that this does not guarantee sponsorship for this specific role. Below presents additional info for your reference. (Data Powered by US Department of Labor)

Distribution of Different Job Fields Receiving Sponsorship

Represents job field similar to this job

Trends of Total Sponsorships

2023 (20)