Oracle · 1 day ago
QA Engineer (Automation)- Dallas, TX
4835 Lyndon B Johnson fwy, Suite 540 Dallas, TX 75244, Dallas, Texas, US
Full-time
Onsite
Senior Level, Lead/Staff
$38K/yr - $133K/yr
10+ years expOracle is seeking a QA Automation Engineer who is ready to move beyond traditional 'Pass/Fail' testing. In this role, you will design and build automation frameworks specifically for Agentic AI products, focusing on evaluating the performance of autonomous agents and ensuring they provide accurate outputs.
Data GovernanceData ManagementEnterprise SoftwareInformation TechnologySaaSSoftware
Responsibilities
Develop automation strategies for probabilistic outputs, using model-based evaluation to 'test the tester.'
Create and maintain 'Golden Datasets' to benchmark agent performance across different versions of prompts and models
Build automated tests to verify that agents call the correct functions/APIs with the right parameters in complex multi-step workflows
Monitor how subtle changes in prompt engineering or model updates (e.g., moving from GPT-4 to Claude 3.5) affect the product’s reliability
Integrate performance testing into the CI/CD pipeline to track agent reasoning time and cost-efficiency
Develop automated checks to identify and report AI hallucinations, bias, or 'jailbreak' attempts
Work closely with AI Engineers to translate 'vague' business requirements into measurable, automated test cases
Qualification
Required
10+ years in QA Automation, with a recent focus on AI/ML or LLM-based applications
Expert-level Python skills (the industry standard for AI testing) and experience with testing frameworks like Pytest
Familiarity with AI evaluation frameworks such as LangSmith, DeepEval, RAGAS, or Promptfoo
Deep experience with Playwright, Selenium, or Cypress for UI, but a heavy focus on API-level testing and database validation
Understanding that AI testing often requires 'scoring' (e.g., 85% accuracy) rather than a simple binary pass/fail
Ability to work with SQL and JSON to validate data retrieved by agents during RAG (Retrieval-Augmented Generation) processes
Preferred
Experience testing Multi-Agent Systems (where one agent tests another)
Knowledge of Prompt Engineering and how it influences software behavior
Background in Investment Banking or Fintech (if applicable) to understand high-stakes data accuracy
Benefits
Medical
Vision
Dental benefits
401k retirement plan
Variable pay/incentives
Paid time off
Paid holidays
Company
Oracle
Oracle is an integrated cloud application and platform services that sells a range of enterprise information technology solutions.
10001+ employees
H1B Sponsorship
Oracle has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (1271)
2024 (846)
2023 (995)
2022 (1192)
2021 (985)
2020 (755)
Funding
Current Stage
Public CompanyTotal Funding
$25.75BKey Investors
Sequoia Capital
2025-09-24Post Ipo Debt· $18B
2025-02-03Post Ipo Debt· $7.75B
1986-03-12IPO
Leadership Team
Esteban Rubens
Healthcare Field CTO
G
Gerard Warrens
Field CTO, Business Strategy and Transformative Technologies
Recent News
2026-01-09
2026-01-07
Company data provided by crunchbase