Overview
We are looking for a Senior AI Engineer to design and build production-grade conversational AI systems in a multi-tenant SaaS environment. You will own end-to-end AI solutions, focusing on prompt pipelines, structured outputs, evaluation frameworks, and production reliability while ensuring high-quality AI experiences at scale. This role requires hands-on experience shipping production LLM systems to real users, not only building prototypes or proof-of-concept solutions.
Project Overview:
You will contribute to building a production-grade conversational AI platform designed for a multi-tenant SaaS environment. The platform focuses on delivering reliable, scalable, and observable AI-driven interactions, transforming complex user inputs into structured and actionable outputs for real users.
- Design, build, and maintain production conversational AI systems operating in a multi-tenant SaaS environment
- Develop and optimize prompt pipelines and structured output workflows to ensure reliability and consistency
- Design end-to-end AI solutions from requirements gathering through deployment and production support
- Build and maintain evaluation frameworks including deterministic, online, and LLM-as-a-judge approaches
- Implement observability pipelines including tracing, latency monitoring, token tracking and prompt and output logging
- Monitor production systems and continuously improve quality, latency, safety, and operational efficiency using evaluation and observability data
- Collaborate with Product Managers and engineering teams to define requirements and deliver production-ready AI solutions
- Drive architectural decisions and evaluate trade-offs to improve system scalability and maintainability
- Independently propose technical approaches, refine requirements with Product Managers, and execute with limited supervision
- Proven experience building production LLM systems used by real customers
- Ability to clearly describe a specific production AI or conversational system personally built and delivered
- Experience designing AI systems end to end and delivering production-ready applications
- Strong experience developing conversational AI applications
- Understanding of prompt engineering, structured outputs, and completion logic
- Experience implementing production observability including tracing, latency monitoring, token tracking, and logging pipelines
- Experience building evaluation frameworks including deterministic evaluations, online evaluations, and LLM-as-a-judge methodologies
- Experience measuring regressions and making engineering decisions based on evaluation results
- Experience improving structured output reliability in production systems under real latency constraints
- Hands-on experience with LangGraph or LangChain
- Experience working with production AI workloads in AWS with focus on scalability, reliability, and cost optimization
- Ability to drive technical decisions, evaluate trade-offs, and collaborate with Product Managers to refine requirements
- English working proficiency
- Experience with Voice AI technologies including speech-to-text and text-to-speech
- Experience building real-time conversational agents and turn-detection systems
- Familiarity with vision-language models
- Experience with Amazon Bedrock AgentCore and OpenAI APIs
- Experience working with MCP and A2A communication protocols
- Familiarity with observability tools such as Arize, LangSmith, or Braintrust
✨ Our intelligent job search engine discovered this job and republished it for your convenience.
Please be aware that the job information may be incorrect or incomplete. The job announcement remains the property of its original publisher. To view the original job and its full details, please visit the job's URL on the owner’s page.
Please clearly mention that you have heard of this job opportunity on https://ijob.am.
