QA Engineer - AI Systems

About This Role

We were looking for a detail-oriented QA Engineer to ensure the quality and reliability of our AI-powered applications. This was a unique role at the intersection of traditional QA and the emerging field of AI testing.

Unlike conventional software testing, AI systems require specialized approaches—testing LLM outputs for accuracy, consistency, and safety; validating AI agent behaviors across diverse scenarios; and ensuring graceful handling of edge cases that traditional systems never encounter.

What You Would Do

• Design comprehensive test plans for AI applications, including LLM-based features and agent systems
• Develop automated testing frameworks specifically for evaluating LLM outputs and behaviors
• Create test datasets and evaluation benchmarks for measuring AI quality
• Identify edge cases, failure modes, and potential safety issues in AI systems
• Build regression test suites to catch quality degradation across model updates
• Collaborate with engineers to establish quality gates and improve system reliability
• Document testing procedures, results, and quality metrics
• Participate in prompt engineering reviews to catch issues before deployment
• Monitor production AI systems for quality degradation and anomalies

Requirements

• 2+ years of QA/testing experience with strong fundamentals in test methodology
• Familiarity with AI/ML concepts and how LLMs work
• Experience with test automation tools (Playwright, Cypress, Selenium, or similar)
• Strong analytical and problem-solving skills
• Excellent attention to detail and systematic approach to testing
• Good written communication for documentation and bug reports
• Experience with API testing

Nice to Have

• Experience testing LLM-based or conversational AI applications
• Knowledge of Python or JavaScript for test automation
• Background in linguistics, content analysis, or data quality
• Understanding of prompt engineering techniques
• Experience with statistical analysis for quality metrics

What We Offered

• Competitive salary in USD (adjusted for LATAM cost of living)
• Fully remote position with flexible hours
• Opportunities to pioneer AI testing methodologies
• Learning and development budget
• Collaborative team environment across time zones
• Path to senior QA or QA lead roles