HyperFi · 19 hours ago
AI Systems & Data Engineer
Hybrid (San Francisco, California, US)
Full-time
Hybrid
Senior Level
5+ years expHyperFi is building a fast, flexible platform to manage real-world complexity, and they are seeking an AI Systems & Data Engineer to join their team. This role involves designing and operating data pipelines in Databricks, ingesting unstructured data, and leveraging it to build AI agents.
Computer Software
No H1B
Responsibilities
Design and operate Databricks pipelines in Python to ingest and normalize large-scale unstructured data
Build streaming and batch ingestion using Auto Loader, Delta Live Tables, and Workflows
Model and maintain AI-ready lakehouse tables with Delta Lake and Unity Catalog
Prepare retrieval and context datasets for RAG and agent systems
Orchestrate Temporal-based workflows to coordinate data prep, validation, and AI handoff
Enforce data quality, lineage, and access controls across pipelines
Optimize PySpark jobs for performance, reliability, and cost
Integrate pipeline outputs into production AI systems and APIs
Monitor freshness, schema drift, and pipeline health
Qualification
Required
5-7 years of experience building production-grade ML, data, or AI systems
Strong grasp of prompt engineering, context construction, and retrieval design
Comfortable working in LangChain and building agents
Experience with PySpark and Databricks to handle real-world data scale
Ability to write testable, maintainable Python with clear structure
Understanding of model evaluation, observability, and feedback loops
Excited to push from prototype → production → iteration
Familiarity with Databricks Data Intelligence Platform which unifies data warehousing and AI use cases on a single platform
Knowledge of Unity Catalog for open and unified governance of data, analytics, and AI on the lakehouse
Understanding of data security concerns related to AI and how to mitigate them using the Databricks AI Security Framework (DASF)
Confident English skills to collaborate clearly and effectively with teammates
Preferred
Have built scalable agent-like workflows on the Databricks platform
Have worked on semantic chunking, vector search, or hybrid retrieval strategies
Can walk us through a real-world prompt failure and how you fixed it
Have contributed to OSS tools or internal AI platforms
Think of yourself as both an engineer and a systems designer
Are familiar with the concept of a data lakehouse architecture
Benefits
Flexible hours
Async-friendly culture
Engineering-led environment
Company
HyperFi
HyperFi is building an AI-powered operating system to transform how the world’s most connected companies buy, sell, and scale technology.
2-10 employees
Funding
Current Stage
Early StageCompany data provided by crunchbase