Graduate AI Test Automation Engineer

Graduate AI Test Automation Engineer role focused on developing Python-based testing tools, automating LLM evaluation, benchmarking AI agents and measuring model quality.

About Reply:

Reply specialises in designing and implementing solutions based on new communication channels and digital media. Structured as a network of highly specialised companies, Reply supports leading European industrial groups in sectors such as Telco & Media, Industry and Services, Banking and Insurance, and Public Administration, helping define and develop business models enabled by new paradigms such as AI, Cloud Computing, Digital Media, and the Internet of Things. Reply’s services include Consulting, System Integration, and Digital Services.    

Role Overview:

We are looking for a Graduate AI Test Automation Engineer with expertise in Python and a strong interest in Artificial Intelligence who will support the development of an innovative testing framework for generative AI agents. You will be involved in designing tools for test data generation, automating evaluation processes, and contributing to AI model quality metrics (such as accuracy, bias, consistency and more).

This position will allow you to join a cutting-edge project shaping AI testing standards. You’ll work with experts in AI, testing, and data governance and will be offered opportunities for quick career progression in a high-impact, AI-driven environment.

Responsibilities:

• As an AI Test Automation Engineer, you will develop and maintain Python-based test automation tools

• You’ll collaborate on framework design and third-party tool integration

• In this role, you’ll define and implement test strategies for LLM behavior validation

• Support continuous testing, benchmarking, and KPI tracking

About the candidate:

• You’re on track to complete (or have already completed) a Bachelor’s or Master’s degree in Computer Science, AI, or in a related field (2.1 min grade)

• Strong interest in AI-driven software testing and development

• You’re proficient in Python and data libraries

• Familiarity with LLMs and testing tools (PyTest, Playwright, etc.) would be a major asset for this role

• Initial experience with datasets and evaluation metrics (BLEU, ROUGE, etc.) is beneficial

• You can demonstrate an analytical mindset and enjoy following a collaborative approach to solving problems

Reply is an Equal Opportunities Employer and committed to embracing diversity in the workplace. We provide equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type regardless of age, sexual orientation, gender, identity, pregnancy, religion, nationality, ethnic origin, disability, medical history, skin colour, marital status or parental status or any other characteristic protected by the Law.