Overview

We are looking for a Platform Engineer with experience in CI/CD automation, cloud observability, and AI platform operations to help build and maintain quality evaluation pipelines for agent based systems. In this role, you will integrate automated evaluation frameworks into deployment workflows, develop monitoring solutions, and support continuous quality validation across production environments.
Project Overview:
The project focuses on establishing a scalable evaluation and monitoring platform for AI agents and intelligent applications. The solution provides automated deployment gates, online quality assessment, operational visibility, and regression detection capabilities to help maintain high standards of reliability and performance.

Responsibilities:
  • Integrate AWS AgentCore Evaluation API into CI/CD pipelines to support automated deployment quality gates
  • Configure and maintain on demand evaluation workflows as part of build and release processes
  • Develop and support GitHub Actions workflows for automated evaluation execution and deployment validation
  • Configure and manage online evaluation modes with appropriate sampling strategies
  • Build and maintain CloudWatch dashboards to track evaluation metrics for agents and evaluators over time
  • Design and configure CloudWatch alarms to identify quality degradation and operational issues
  • Collaborate with AI and platform engineering teams to define evaluation thresholds and quality standards
  • Monitor evaluation results and contribute to regression detection and release validation processes
  • Support observability, reporting, and operational monitoring initiatives across AI platforms
  • Create and maintain technical documentation related to evaluation workflows, monitoring, and deployment automation
Required Qualifications:
  • 3+ years of experience in platform engineering, DevOps engineering, or a related field
  • Experience integrating quality validation processes into CI/CD pipelines
  • Hands on experience designing and maintaining deployment gates for automated release workflows
  • Experience building and supporting AWS CloudWatch dashboards and monitoring solutions
  • Experience configuring CloudWatch alarms and operational alerting strategies
  • Knowledge of cloud infrastructure, automation, and deployment best practices
  • Experience working with monitoring, observability, and platform operations tools
  • Strong analytical and troubleshooting skills
  • Good written and verbal communication skills
Nice To Have:
  • Experience with AWS AgentCore Evaluation online mode configuration and sampling strategies
  • Experience using evaluations to compare runtime versions and identify regressions
  • Knowledge of CloudWatch Embedded Metric Format (EMF) for structured metric collection and reporting
  • Experience with AI platform monitoring and evaluation frameworks
  • Exposure to automated quality assurance processes for machine learning or agent based systems
  • Experience supporting enterprise scale cloud platforms and deployment pipelines
Note:

✨ Our intelligent job search engine discovered this job and republished it for your convenience.
Please be aware that the job information may be incorrect or incomplete. The job announcement remains the property of its original publisher. To view the original job and its full details, please visit the job's URL on the owner’s page.

Please clearly mention that you have heard of this job opportunity on https://ijob.am.