AI Conversation Research & Evaluation Specialist

About the role

Gramian Consultancy, a boutique consultancy focused on IT professional services and engineering talent solutions, is looking for experienced Subject Matter Experts (SMEs) in psychology, social work, counseling, qualitative research, human-computer interaction, communication, conflict resolution, coaching and related advisory fields. You will review conversations between people and AI models, check whether enough context was gathered before recommendations were made, spot missed opportunities for clarification and give structured feedback to improve conversational quality.

Key responsibilities

• Review AI conversations covering real-world interpersonal situations and everyday decision-making.
• Identify missing context and recommend follow-up questions that would improve AI responses.
• Assess whether gathering more information would materially change an AI-generated recommendation.
• Distinguish between information that affects a decision and non-essential detail.
• Conduct qualitative reviews and document findings using structured evaluation guidelines.
• Check whether conversations consider personal, psychological, social, financial, practical and organizational factors.
• Review conversations about relationships, work, education, family, wellbeing, planning and resource constraints.
• Collaborate with research teams to keep assessments consistent and evidence-based, and meet project deadlines.

Required profile

• At least 3-5 years of relevant direct professional experience.
• Pathway 1: Licensed social worker (MSW or equivalent; LCSW, LICSW, LMSW or equivalent) with experience in client assessment, case review, interviewing or counseling.
• Pathway 2: Counselor, therapist or psychologist (master's with LMFT/LPC/LCPC or equivalent, or PhD/PsyD in clinical, counseling, social or I-O psychology) with client-facing experience.
• Pathway 3: Advanced expertise (MA, PhD or equivalent) in HCI, communication, anthropology, sociology, qualitative research or a close field, with experience in qualitative research, interviews, conversation analysis or thematic analysis.
• Pathway 4: Mediation, coaching, student services, organizational consulting or similar advisory field, with at least 5 years of advisory or client-facing experience.