AI Evaluation Engineer

Company:

Application Deadline: 26.04.26

People are the very DNA of DNB. Since 1822, bright minds have worked together to find the best solutions for our customers. Today, DNB is much more than Norway's largest bank – we are a technology-driven financial institution that continuously connects people and ideas to knowledge and capital in new ways.

Diversity is part of who we are, and inclusion is something we actively choose every single day. We promise to do our best to make you feel at home. A job at Norway's largest financial group offers professional challenges in an exciting work environment with many opportunities for development.

AI Evaluation Engineer

About us
AI Tech is DNB’s new division within Technology & Services, created to accelerate our shift from AI experimentation to real, measurable impact. We bring together deep technical expertise, modern AI platforms, and hands‑on delivery to scale agentic AI across the group. We move fast, learn fast, and deliver real outcomes.

We expect every member of AI Tech to be a role model in the everyday use of AI —using AI proactively in coding, automating tasks, improving documentation, and accelerating problem‑solving. You help raise the overall AI maturity across DNB by demonstrating what AI‑first engineering looks like.

What you will do

Designing evaluation frameworks for agentic systems. Ensuring quality, coverage, safety, efficiency, and regulatory compliance
Building LLM-as-judge pipelines: prompt design, calibration, consistency validation
Integrating evaluation into CI/CD: automated regression detection when context (prompts, data sources etc.), models or services change
Running evaluation experiments when there are relevant changes or additions
Constructing and maintaining evaluation datasets (input, ground truth etc.).
Working with domain experts to translate requirements into measurable criteria
Developing shared evaluation tooling and patterns that teams across DNB can adopt

Background

Relevant background from evaluating AI / ML systems
Proficiency in Python for experiments and evaluation
Knowledge of observability, logging, and dataset curation
Experience running structured experiments spanning many roles
Familiarity with AI tooling for developer productivity (Claude Code, Copilot etc.)
Experience with agentic evaluation techniques (using code, LLMs and humans)
Experience with agentic evaluation solutions (AgentCore, Foundry, MLflow etc.)

Tech stack example:
Python, Strands, AWS AgentCore, AWS Bedrock, MCP, Mlflow, OTel, Docker, GitHub Actions

What you bring
You bring an evidence driven mindset and back your claims and decisions with data and numbers. You act as a role model for rigorous and responsible AI testing, setting a high bar for quality, safety, and trustworthiness in everything you build. You communicate and collaborate effectively across roles and teams, and you take true ownership to the team’s goals.

What’s in it for you?
You’ll work on challenging and meaningful tasks in a strong engineering culture with solid opportunities for professional growth and career development. We offer attractive pension and insurance schemes, as well as employee benefits on DNB’s products. You’ll also have access to company cabins across Norway, sports, cultural and social activities, and a wide range of employee discounts. We support flexibility in everyday work through flexible working hours, a hybrid way of working, extra days off, and reduced working hours from May to August (summertime).

Contact Persons: Mari Sand Frogner, Service Owner | Mari.Sand.Frogner@dnb.no | +47 41475454 / Olav Lognvik, Engineering Manager | Olav.Lognvik@dnb.no | +47 40019700
Application Deadline: 26.04.26

In this job application process, you only need to upload your resume (CV) and briefly answer some job-related questions. Cover letter is not required but you can upload this as an attachment if desired.

In DNB, we carry out background checks to verify that the information provided in your CV and other documentation is correct. Background checks are generally performed by an external independent third party. Former employers are typically contacted to check previous positions and periods of employment, while educational institutions are asked to confirm marks. No background check will be conducted without your prior consent, and you will receive more detailed information about this, if applicable. For positions that require an authorisation and/or approval of suitability, a police certificate of good conduct will be required.