Apply on Employer Site

New York Life Insurance Company · 3 days ago

Senior Associate - Security Data Lake Engineer

New York, United States

Full-time

Hybrid

Mid, Senior Level

$121K/yr - $172K/yr

5+ years exp

New York Life Insurance Company is seeking a Security Data Lake Engineer to design, implement, and optimize their enterprise-scale security data lake and advanced analytics capabilities. The role involves building scalable data pipelines and collaborating with various teams to enhance detection and analytics capabilities, focusing on cybersecurity outcomes.

FinanceFinancial ServicesInsurance

H1B Sponsor Likely

Responsibilities

Architect, build, and manage data pipelines across AWS services (S3, Kinesis, Lambda), APIs, and Cribl

Ensure data lake performance and usability through indexing, query optimization, and aggregation pipeline management for 3.5TB+ daily volumes

Act as the dedicated owner to maintain data quality and prevent “data swamp” degradation

Establish a Data Lake Center of Excellence (C4E) to empower decentralized analytics development while maintaining governance and standards

Collaborate with security teams to define, develop, and deploy data-driven use cases

Build ML pipelines for anomaly detection, vector search, semantic search, and other advanced security analytics

Integrate and optimize Elasticsearch, vector databases, and KNN/percolator search capabilities

Develop and maintain Jupyter Notebook workflows for data wrangling, modeling, and analysis

Apply machine learning and natural language processing techniques, including retrieval-augmented generation (RAG)

Partner with AI adoption initiatives to ensure secure and effective use of enterprise data for AI/ML

Stay current with emerging technologies and techniques in data science, big data, and cloud analytics

Work closely with SOC Engineers, Threat Hunters, and data platform teams to integrate log sources and telemetry into the data lake

Provide guidance and mentorship across Cyber teams on building analytics use cases

Serve as the in-house expert to reduce reliance on external consulting spend, ensuring institutional knowledge is retained within NYL

Qualification

AWS servicesData engineeringMachine learningPythonElasticsearchData pipeline optimizationJupyter NotebooksNatural language processingData quality managementCollaborationMentorship

Required

5+ years of hands-on experience in data engineering, big data platforms, or applied data science

Proficiency with AWS services (S3, Kinesis, Lambda, Bedrock) and enterprise-scale analytics design

Strong programming skills in Python and/or Scala; experience with Jupyter Notebooks

Experience with indexing, query optimization, and data pipeline optimization at scale (3.5TB+ daily)

Advanced knowledge of Elasticsearch including advanced search methods (KNN vector search, percolator reverse search), and performance tuning, platform maintenance, etc

Experience applying ML techniques for anomaly detection, semantic search, embeddings, and NLP

Bachelor's degree in Computer Science, Data Science, Cybersecurity, or related field

Preferred

Familiarity with Cribl for data routing and transformation

Advanced degree in mathematics, computer science, or related empirical sciences

Previous experience in cybersecurity environments or willingness to learn security context quickly

Certifications in AWS Data Engineering, Machine Learning, or Big Data technologies a plus

Benefits

Leave programs

Adoption assistance

Student loan repayment programs

Company

New York Life Insurance Company

For over 180 years, we’ve helped turn your biggest dreams into milestones that last a lifetime.

Founded in 1845

New York, New York, USA

10001+ employees

https://www.newyorklife.com/

H1B Sponsorship

New York Life Insurance Company has a track record of offering H1B sponsorships. Please note that this does not guarantee sponsorship for this specific role. Below presents additional info for your reference. (Data Powered by US Department of Labor)

Distribution of Different Job Fields Receiving Sponsorship

Represents job field similar to this job

Trends of Total Sponsorships

2025 (149)

2024 (99)

2023 (85)

2022 (77)

2021 (48)

2020 (65)