Back to job offers
Apply Now!

AI Test Engineer

  • Seniority:

    Senior

  • Work model:

    Remote

  • City:

    Poland

  • Language:

    English

  • Job Description:

    We are seeking a highly skilled AI Test Engineer specializing in testing Generative (and non-Generative) AI applications to join our team. The ideal candidate will have expertise in both traditional software testing and specialized testing methodologies for Large Language Models (LLMs)-based and AI-based systems. This role requires deep understanding of LLM behaviors, and testing frameworks to ensure the reliability, fairness, and effectiveness of our AI-powered solutions.

  • Key Responsibilities:

    • Generative AI Testing Strategy: Design and implement comprehensive testing strategies specifically tailored for LLM-based applications, including evaluation of model responses, RAG pipelines accuracy, and overall system reliability
    • Quality Assurance Framework Development: Utilize testing frameworks that address both traditional software quality aspects and AI-specific concerns such as output consistency, contextual accuracy, and ethical compliance; co-create and maintain such frameworks when required
    • Test Automation Development: Design and implement automated testing solutions for continuous evaluation of LLM applications, including integration tests, performance tests, and specialized AI behavior tests
    • Edge Case Analysis: Identify and develop test scenarios for edge cases in LLM behavior, including handling of ambiguous inputs, potential biases, and unexpected response patterns
    • Bias and Fairness Testing: Design and execute tests to identify potential biases in model outputs and ensure fair treatment across different user groups and use cases
    • Security Testing: Collaborate within development teams to test for potential vulnerabilities specific to LLM applications, including prompt injection, data leakage, and other AI-specific security concerns
    • Test Documentation: Create and maintain comprehensive test documentation testing strategy, test cases, and testing guidelines specific to AI applications and compliant with practices; document the analysis of requirements and risks and develop tests accordingly
    • Performance Testing: Collaborate with other engineers to conduct thorough performance testing of GenAI applications, including response time analysis, load testing, and resource utilization monitoring

  • Profile Requirements:

    • Experience: 5+ years of experience in software testing, with at least 1 year focused on genAI applications
    • Technical Skills: Strong proficiency in Python and testing frameworks (pytest, allure); experience with AI testing tools and frameworks
    • AI Knowledge: Understanding of LLM architectures, RAG systems, and common failure modes in AI applications
    • Test Automation: Experience with test automation frameworks and GitHub Actions / GitLab CI/CD pipelines; ability to design and implement automated testing solutions for AI applications
    • Ability to analyze requirements, identify test scenarios, and prioritize testing activities based on risk and impact
    • Cloud Platforms: Practical experience with testing applications on cloud platforms (AWS) and working with cloud-based AI services
    • Monitoring Tools: Experience with monitoring and observability tools, log analysis, and performance metrics tracking (Grafana)
    • ALM tools: Experienced in mandatory ALM tools (Jira, GitLab, GitHub)
    • Security Testing: Understanding of security testing methodologies, particularly in the context of AI applications
    • Hold B.Sc., B.Eng., M.Sc., M.Eng. in Computer Science, Software Engineering, or equivalent degree with strong background in testing methodologie

  • Perks and benefits:

    • Comprehensive benefits – enjoy Udemy for Business, private medical care, Multisport card, veterinary package, language lessons, and shopping vouchers.
    • Career growth – access opportunities for professional development and learning, including perks related to our official partnerships with global IT giants: Microsoft, AWS, Snowflake, Salesforce & more.
    • Global collaboration – work with a diverse, international team.
    • Innovative environment – be part of a forward-thinking and growth-oriented workplace.
    • Engaging community – Work with passionate professionals and participate in team-building events, hackathons, and CSR initiatives to make an impact beyond work.
    • Team-building events including our company tradition (annual company event in Mazury).
    • A pleasant surprise to start your journey with us in the form of a welcome pack.

AI Test Engineer

Apply Now!