AI Quality Analyst

About the role

Finom is looking for an AI Quality Analyst to act as the gatekeeper of AI safety, performance, and deterministic behavior across non‑deterministic multi‑agent systems. You will work in a highly cross‑functional environment, collaborating with domain experts, MLOps, and backend engineers to ensure AI outputs meet rigorous quality standards.

Key responsibilities

• Design, implement, and maintain scalable evaluation pipelines (Evals) for large language models (LLMs) and agent graphs using modern tooling such as LangSmith, DeepEval, Ragas, or Opik.
• Collaborate with domain experts to build, version, and sanitize gold‑standard datasets, synthetic evaluation profiles, and edge‑case testing matrices that reflect real‑world business scenarios.
• Define, monitor, and enforce quality KPIs across multi‑agent workflows, focusing on tool‑calling accuracy, intent‑recognition safety, structured output formatting, and retrieval‑augmented generation (RAG) precision.
• Lead failure and hallucination analyses on production outputs, implementing LLM‑as‑Judge patterns, validation metrics, and guardrail heuristics while mitigating baseline evaluation bias.
• Partner with MLOps and backend engineering teams to embed evaluation gates into CI/CD pipelines, ensuring continuous quality assurance before deployment.

Required profile

• Strong analytical mindset with experience in AI/ML evaluation and quality assurance.
• Ability to work collaboratively in a fast‑paced startup environment.
• Excellent communication skills to translate technical findings into actionable recommendations.

Required skills

• LangSmith
• DeepEval
• Ragas
• Opik