Federal Reserve Bank of New York ยท 2 days ago
Data Engineer
Wonder how qualified you are to the job?
Financial Services
Insider Connection @Federal Reserve Bank of New York
Responsibilities
Design, develop, monitor, and maintain data pipelines in an AWS Gov Cloud ecosystem with AWS, Databricks, Delta Lake, and Trino as the underlying platforms.
Collaborate with cross-functional teams to understand data needs and translate them into effective data pipeline solutions.
Establish data quality checks and ensure data integrity and accuracy throughout the data lifecycle.
Automate testing of the data pipelines and configure as part of CICD.
Optimize data processing and query performance for large-scale datasets within AWS and Databricks environments.
Document data engineering processes, architecture, and configurations.
Troubleshooting and debugging data-related issues on the AWS Databricks platform.
Integrating Databricks with other AWS products such as SNS, SQS, and MSK.
Qualification
Find out how your skills align with this job's requirements. If anything seems off, you can easily click on the tags to select or unselect skills to reflect your actual expertise.
Required
Bachelor's or master's degree in computer science, Information Technology, or a related field.
Strong technical proficiency to contribute to data engineering task in an AWS and Databricks ecosystem.
Strong technical proficiency with Spark, Trino, Python, PySpark and SQL
Proven ability in Gitlab with CI/CD.
Proven ability in AWS Services like S3, RDS, Lambda, SQS, SNS, MSK is required.
Strong SQL skills to perform data analysis and understanding of source data.
Proven ability with data pipeline orchestration tools
Proven ability with ETL tools and Relational databases
Proven ability to troubleshoot complex data issues and implement effective solutions.
Proven ability in staying updated with industry trends and emerging technologies in data engineering.
Benefits
Fully paid Pension plan and 401k with Generous Match
Comprehensive Insurance Plans (Medical, Dental and Vision including Flexible Spending Accounts and HSA)
Subsidized Public Transportation Program
Tuition Assistance Program
Onsite Fitness & Wellness Center
Company
Federal Reserve Bank of New York
The Federal Reserve Bank of New York is a regional reserve bank that sets monetary policy and maintains financial stability in New York.
Funding
Current Stage
Late StageLeadership Team
Recent News
2024-04-07
2024-04-07
Company data provided by crunchbase