58 applicants

Company

Original Job Post

State Street Global Advisors · 3 days ago

Data Engineer

Boston, Massachusetts

Full-time

Onsite

Senior Level

$115K/yr - $182K/yr

5+ years exp

Wonder how qualified you are to the job?

Maximize your interview chances

FinanceFinancial Services

Insider Connection @State Street Global Advisors

Discover valuable connections within the company who might provide insights and potential referrals, giving your job application an inside edge.

Responsibilities

Design, develop, and maintain scalable data pipelines using pyspark on Databricks.

Implement and optimize stream processing workflows using Kafka for real-time data ingestion and processing.

Utilize Parquet and Avro-formatted data files for efficient storage and retrieval.

Leverage Databricks platform on AWS to build and manage data processing workflows and analytics.

Harness the power of Databricks Delta Lake and Parquet files for data warehousing and query optimization.

Collaborate closely with data analysts and scientists to provide reliable data solutions.

Implement robust testing methodologies and contribute to the pyspark/Python ecosystem.

Monitor data pipelines, identify and resolve issues, and ensure data integrity.

Stay up-to-date with the latest trends and technologies in data engineering.

Qualification

Find out how your skills align with this job's requirements. If anything seems off, you can easily click on the tags to select or unselect skills to reflect your actual expertise.

Data EngineeringPySparkPythonShell ScriptingSoftware EngineeringDevelopment LifecycleWorkflow ManagementAirflowStream ProcessingKafkaAvroData SerializationDatabricksAWSData ProcessingData WarehousingDelta LakeParquet FilesSQLRelational DatabasesTestingUnit TestsIntegration TestsEnd-to-End TestsPyPI PackagesTechnical ConceptsNon-Technical StakeholdersCloud PlatformsAzureLarge Datasets

Required

Bachelor’s or master’s degree in computer science or a related field.

Minimum 5 years of real world Data Engineering experience working on large scale data projects.

Strong proficiency in pySpark, Python and shell scripting, with a focus on software engineering best practices and a deep understanding of development lifecycle.

Experience working with workflow management tools such as Airflow

Experience with stream processing technologies, preferably Kafka.

Familiarity with Avro data serialization format and its usage in data engineering workflows.

Expertise in using Databricks platform on AWS for data processing and analytics.

Solid understanding of data warehousing concepts and experience with Delta Lake and Parquet files.

Proficiency in SQL and experience with relational databases.

Strong testing skills, with experience in implementing and executing unit tests, integration tests, and end-to-end tests using Python packages such as pytest.

Familiarity with the Python ecosystem, including PyPI packages and their integration into data engineering workflows.

Excellent problem-solving skills and ability to work in a fast-paced, collaborative environment.

Strong communication skills and ability to effectively communicate complex technical concepts to non-technical stakeholders.

Working experience with Databricks and pyspark

Proficiency in writing complex SQLs

Working experience with cloud platforms like AWS or Azure (preferably AWS)

Working Experience with Airflow

Experience working with very large datasets