Reward Model
A model trained to predict human preferences, used in RLHF pipelines to guide the training of language models toward generating outputs that align with human values.
In plain language
An AI trained to judge what good responses look like, based on human preferences. It scores the main AI's outputs as "helpful" or "harmful", guiding the AI to improve.
Why this matters
Reward models are critical to deployment because they encode the values that guide AI behaviour in production. Your implementation should address how human preferences are collected, who provides them, and how disagreement is managed.
Relevance
ImplementationReward models must be developed with documented human feedback processes, tested for alignment with organisational values and monitored for drift over time.
Related terms
Putting reward model into practice in your organisation?
Ready to transform your AI strategy?
Partner with Australia's AI strategy and governance specialists. From adoption roadmaps to ISO 42001 audit readiness.