Acceler8 Talent · 2 days ago

Software Engineer

United States

Full-time

Remote

Mid Level

$200K/yr - $600K/yr

Wonder how qualified you are to the job?

Maximize your interview chances

Staffing and Recruiting

Hiring Manager

Anthony Vitti

Insider Connection @Acceler8 Talent

Discover valuable connections within the company who might provide insights and potential referrals, giving your job application an inside edge.

Responsibilities

Designing and implementing multimodal web crawlers for scraping and indexing petabytes of data.

Creating large-scale data processing pipelines using tools such as Ray, Apache Spark, Apache Flink, and Google BigQuery.

Scaling deduplication techniques across different modalities and applying heuristic and model-based methods for parsing and filtering crawled data.

Identifying new data sources for integration into pre/post-training datasets.

Qualification

Find out how your skills align with this job's requirements. If anything seems off, you can easily click on the tags to select or unselect skills to reflect your actual expertise.

Distributed computingParallel processingData architecture designScalable architecturesLLM data pipeline developmentWeb scrapingData loadingAttention to detailReliabilityRigorous testing