Overview

Responsibilities:
  • Develop and execute AI model evaluation strategies, ensuring accuracy, consistency, and fairness
  • Implement automated and manual testing for LLM-based applications
  • Collaborate with the AI Engineer to integrate testing into early-stage development
  • Build and manage test datasets, ensuring high-quality, diverse, and balanced samples
  • Develop synthetic data pipelines to enhance model evaluation
  • Design and maintain hallucination, bias, and robustness detection frameworks
  • Define and track AI performance metrics (e.g., factual accuracy, coherence, latency, response quality)
  • Work closely with AI engineers to debug failures, identify root causes, and optimize model performance
  • Provide feedback on prompt effectiveness, suggest improvements, and collaborate with the Prompt Engineer to refine prompts
  • Implement continuous monitoring tools to track AI model drift, performance degradation, and unexpected failures
  • Develop and maintain comprehensive test reports, summarizing findings and recommendations
Required Qualifications:
  • Experience with AI/ML testing frameworks and LLM evaluation methodologies
  • Strong understanding of LLM behaviors, biases, failure modes, and edge cases
  • Proficiency in Python and familiarity with ML testing frameworks (e.g., PyTest, Unittest, custom ML evaluation tools)
  • Experience with test dataset management and annotation tools
  • Familiarity with synthetic data generation and adversarial testing techniques
  • Strong problem-solving and debugging skills to analyze AI failures and inconsistencies
  • Strong English language proficiency with the ability to evaluate AI-generated text and improve prompts
Note:

✨ Our intelligent job search engine discovered this job and republished it for your convenience.
Please be aware that the job information may be incorrect or incomplete. The job announcement remains the property of its original publisher. To view the original job and its full details, please visit the job's URL on the owner’s page.

Please clearly mention that you have heard of this job opportunity on https://ijob.am.