Legalist · 2 days ago

Senior Software Engineer

San Francisco, CA

Full-time

Hybrid

Senior Level

4+ years exp

Wonder how qualified you are to the job?

Maximize your interview chances

Financial Services

Insider Connection @Legalist

Discover valuable connections within the company who might provide insights and potential referrals, giving your job application an inside edge.

Responsibilities

Help to design and implement the architecture of a large-scale data pipeline system

Design, implement, and maintain various components of ELT data processing infrastructure

Collaborate with product and business teams to understand/anticipate requirements for greater functionality and impact in data processing systems

Lead development of data architecture best practices, standards, design review processes, and creation of repeatable design documents

Scale architectural design work through data engineering, data governance, and solutions architecture

Develop and maintain all aspects of a data processing pipeline end to end (Extract, Load, Transform, Monitor, Document)

Clean and wrangle data from multiple disparate structured and semi-structured sources

Utilize cloud environments like GCP, AWS, containerization tools like Docker, and orchestration tools like Kubernetes

Understand different types of databases (Postgres, Redis, Google Datastore, Google Bigtable, etcd)

Apply computer science basics: Algorithm, OOP, SQL, ORM, Recursion, Memoization, Caching, Common Design Patterns

Qualification

Find out how your skills align with this job's requirements. If anything seems off, you can easily click on the tags to select or unselect skills to reflect your actual expertise.

PythonData WranglingHigh-throughput Distributed SystemData WarehousingAirbyteDBTBigqueryRedshiftData Processing PipelineETLCloud EnvironmentsGCPAWSDockerKubernetesMessage BrokersKafkaRabbitMQGoogle Pub/SubDatabasesPostgresRedisGoogle DatastoreGoogle BigtableEtcdComputer Science BasicsAlgorithmOOPSQLORM

Required

4+ Years experience with Python for automation and data wrangling

Production experience in high-throughput distributed system

Familiarity with data warehousing concepts, tools and monitoring highly recommended (Airbyte, DBT, Bigquery, Redshift)

Ability to develop and maintain all aspects of a data processing pipeline end to end (Extract, Load, Transform, Monitor, Document)

Experience cleaning and wrangling data from multiple disparate structured and semi structured sources including (HTML, XML, REST, GraphQL, PDF, JSON, and spreadsheets.)

Experience with cloud environments like GCP, AWS, as well as containerization tools like Docker and orchestration such as kubernetes or others.

Familiarity with message brokers such as Kafka, RabbitMQ, Google Pub/Sub

Good understanding of different types of databases (Postgres, Redis, Google Datastore, Google Bigtable, etcd)

Mastery of computer science basics: Algorithm, OOP, SQL, ORM, Recursion, Memoization, Caching, Common Design Patterns