LILT AI · 1 month ago
AI Benchmark Engineer | Native Language Specialist - Arabic - Remote
LILT is a language technology company building multilingual AI and human-verified services. The AI Benchmark Engineer and Native Language Specialist will design, implement, validate, and calibrate Terminal-Bench tasks that evaluate large language models on multilingual software challenges, with a focus on Arabic language processing, locale, encoding, and terminal workflows.
Artificial Intelligence (AI)Enterprise SoftwareProfessional ServicesMachine LearningNatural Language ProcessingProductivity ToolsTranslation Service
H1B Sponsor Likely
Responsibilities
Task Engineering: Evaluating Coding Agents
Asset Creation: Build realistic task environments using datasets and files in your native language. Crucially, these assets must remain in the target language to genuinely measure multilingual handling
Prompting & Translation: finding failure points where AI does not work, in your native language
Implementation & Verification: Support the development of robust solutions (reference implementations) and write highly reliable, deterministic verifier scripts (using rubric-based judging only when strictly necessary)
Calibration & Execution: Analyze execution logs and calibrate task difficulty (Easy to Very Hard) using standard Terminal-Bench run configurations against various model tiers (Haiku, Sonnet, Opus)
Quality Assurance: Participate in a rigorous, 4-layer human quality control process (creation, human review, calibration review, and audit) alongside automated LLM-based checks to ensure fairness, grammatical accuracy, and benchmark integrity
Qualification
Required
• Experience: 5+ years of industry experience in software engineering
• Background: Proven track record at leading technology companies and/or graduation from top-tier engineering universities
• Language: Native or near-native fluency, with a deep understanding of its grammar, register, and phrasing rules
High English proficiency
• Technical Stack: Strong proficiency in Python, standard shell scripting, and data processing
• Workflow: Extensive experience with Terminal/CLI-based development workflows and a working familiarity with coding agents
• Domain Expertise: Deep technical understanding of multilingual text processing pitfalls, including:
+ Encoding/decoding robustness and Unicode normalization
+ Locale-dependent conventions (collation, casing, non-Gregorian dates)
+ Text I/O, toolchain interoperability, and safe string operations
+ (For specific languages) Bidirectional/RTL handling, font fallbacks, and rendering/typography in UI or artifacts
• Requires reliable availability and commitment. Once you accept a task, we expect quality work and on-time delivery. Most tasks require a minimum of 2 hours per day or 10 hours per week. If your schedule is unpredictable, this may not be the right fit
• Geographic restrictions may apply. We cannot engage contractors in regions subject to international embargo or sanctions
Benefits
As an independent contractor, work when you want, as much or as little as you want. No fixed hours, no check-ins, no micromanaging.
Competitive rates, prompt payments, no chasing invoices.
Contribute to cutting-edge AI and language technology that is shaping how humans and machines communicate.
Join a global community of linguists, subject matter experts, and language professionals who are advancing human knowledge together.
As a Lilt contractor you get access to diverse, innovative projects that expand your portfolio and sharpen your skills across industries and domains.
Work on diverse projects from anywhere, any time you want.
Build your professional network in a supportive community.
Company
LILT AI
Translation and dataset creation for businesses and governments.
H1B Sponsorship
LILT AI has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
86%
Represents job field similar to this job
Engineering and Development
Customer Service and Support
Management and Executive
Trends of Total Sponsorships
*2026 (3)
2023 (2)
2022 (1)
2021 (3)
2020 (5)
Funding
Current Stage
Growth StageTotal Funding
$162.92MKey Investors
Four Rivers GroupIntel CapitalSequoia Capital
2025-06-24Undisclosed· $25M
2025-06-10Undisclosed· $45.42M
2022-04-07Series C· $55M
Leadership Team
Spence Green
CEO
John DeNero
Co-Founder and Chief Scientist
Recent News
Clear Ventures
2026-09-22
2026-08-18
2026-08-18
Company data provided by crunchbase