Overview

Are you fascinated by how artificial intelligence handles complex software engineering? Mindrift is seeking talented professionals for an exciting project-based opportunity as an AI Evaluation Engineer. In this role, you will help build the datasets used to evaluate state-of-the-art AI coding agents. This is a fully remote, worldwide engagement where you collaborate with leading tech companies to test and improve AI systems. Please note that this is a flexible, project-based opportunity rather than a permanent employment position.

Key Responsibilities

Build realistic developer environments featuring virtual companies, complete codebases, infrastructure, and rich context such as tickets, documentation, and historical conversations.
Design engaging coding tasks from intermediate states of these environments by crafting precise prompts and defining clear criteria for success.
Write comprehensive tests that verify agent solutions while accommodating valid approaches and correctly rejecting flawed ones.
Iterate continuously on tasks and tests based on rigorous quality assurance feedback.
Analyze AI agent failures and refine benchmarks to ensure evaluation sets remain fair, robust, and truly challenging.

Requirements

Strong proficiency in Python programming.
Demonstrated experience in software quality assurance or application security.
Substantial background in developer environments and practical coding workflows.
Fluency in English with the ability to submit your CV and proficiency level in English.
A passion for pushing the boundaries of what artificial intelligence can achieve in software development.

What You'll Gain

The chance to shape the future of artificial intelligence by evaluating next-generation coding agents.
Total geographic freedom with a 100% remote working model available worldwide.
Collaborative project-based engagements with top tier technology companies.
Valuable hands-on experience at the intersection of software engineering and machine learning.