Research Scientist - Multi-modal LLMs

At Encord, we're building the AI infrastructure of the future. One of the biggest challenges AI companies face today is data quality. The success of any AI application relies heavily on the quality of its training data, yet for most teams, this crucial step is both the most costly and time-consuming. We’re here to change that.

As former computer scientists, physicists, and quants, we’ve experienced firsthand how a lack of tools to prepare quality training data impedes progress in building AI. We believe AI is at a stage similar to the early days of computing or the internet—where the potential is clear, but the surrounding tools and processes are still catching up. That's why we started Encord.

We are a talented and ambitious team of 60, working at the cutting edge of computer vision and deep learning. Backed by $30M in Series B funding from top investors like CRV and Y Combinator, we’re one of the fastest-growing companies in our space. Our platform is consistently rated the best by our customers, and we have big plans ahead. We’re looking for a Research Scientist to help our customers get the right data faster, easier, and cheaper.

As a Research Scientist focusing on multi-modal LLMs, you'll be allowing all the data, metadata, and embeddings that live in our system to be explored, used, and analyzed in ways no one thought possible. Although starting narrow with “smaller” multi-modal problems like, e.g., improving similarity searches via metadata, we have high ambitions for this role. You'll progressively work on harder problems that will improve user experience, surface the right (personalized) analytics to every customer, and put our users in the driver's seat of a data development platform that can do things much beyond today’s standards. Tasks can be i) fine-tuning models to understand how our platform is used by customers, ii) employing LLM reasoning to assist customers in their data analysis tasks, and iii) Building tools for customers to interface naturally with our platform. All to put the power in the hands of anyone using Encord.

You'll follow the latest research and accelerate state-of-the-art technologies to enrich customers’ data journeys. This role offers a great growth opportunity, with the potential to lead a bigger team of scientists over time in our efforts to build the ultimate data development platform

What you will be doing:

• Building, fine-tuning, and experimenting with multi-modal LLMs to surface potential actions and analytical conclusions in a data-driven manner.
• Developing scalable and novel ways to personalize LLMs based on information from our data development platform.
• Build sophisticated RAG systems on other types of data than the usual text documents.
• Follow the latest machine learning research to identify and apply new methods that improve our processes or the user experience.
• Ensure our customers have the world’s most powerful AI-powered data development platform.

Skills for the job:

• A PhD or similarly strong academic background in machine learning, with 2+ years of hands-on experience in with LLM fine-tuning, RAG systems, and prompt engineering.
• Proficiency with frameworks like PyTorch, Tensorflow, JAX, Pandas, and OpenCV.
• A solid understanding of transformer models and their common variants, loss functions, and pitfalls.
• A quick learner with a structured, organized approach to problem-solving.
• Excellent communication skills with an ability to uncover use cases and solve problems efficiently.
• Ambitious and self-motivated, with a proven track record of top performance in academic or professional settings.

Bonus skills:

• Experience working with data in the order of millions.
• Familiarity with using (and adapting) models like LLaMa and LLaVa.
• Experience with image-to-text embedding models like CLIP and SigLIP.
• Familiarity with cloud-based model training and inference.

- Competitive salary, commission, and equity in a high-growth business.

- A collaborative, in-person culture with most of the team working in the office 3+ days a week (engineers typically work on-site Wednesdays).

- 25 days annual leave + public holidays.

- An annual learning and development budget to help you grow your skills.

- Company lunches twice a week and regular socials, including bi-annual off-sites.

At Encord, you’ll have the unique opportunity to be part of a fast-growing startup with a clear mission and vision. You’ll work on real-world AI use cases across a variety of industry verticals and get hands-on experience with cutting-edge computer vision and deep learning technologies. This is a role where you'll grow quickly, take ownership of projects, and help shape the future of our company.