Writing Evaluation Callbacks

During optimization ropt determines which variable vectors need values for the objectives and optional nonlinear-constraints. A compute step does not compute these values itself — it delegates to an Evaluator instance that you supply. This page describes the evaluators ropt provides and how to write the evaluation code they wrap.

The evaluators

Evaluator	Interface
`BatchEvaluator`	Batch: `f(variables_2d, context)` → `EvaluationBatchResult`.
`FunctionEvaluator`	Per-row: `f(variables_1d, context)` → `EvaluationFunctionResult`.
`ParallelEvaluator`	Parallel evaluation via an `Executor` — see Parallel Evaluation.

The first two run synchronously in the calling thread; ParallelEvaluator dispatches work to an Executor and is described in Parallel Evaluation.

Evaluators are not safe for concurrent use

An evaluator raises a WorkflowError if two threads execute its eval method at the same time. Serial reuse is allowed: a single evaluator instance may be shared by several compute steps that run one after another, even on different threads (for example, reusing one FunctionEvaluator across nested inner optimizations to keep batch ids counting). Do not share one evaluator across steps that run in parallel; give each parallel step its own evaluator. For the constraints on where each layer of a nested workflow may run, see Nested workflows and process boundaries. The parallelism of ParallelEvaluator happens below eval — it dispatches tasks to an executor, so its own eval is still called on a single thread. An evaluator cannot be transferred to another process: it holds a lock, so serializing one fails and the submission is refused with an ExecutionError.

You write evaluation code for two of them. BatchEvaluator takes a callback that receives the full 2-D batch of variable vectors; FunctionEvaluator wraps a simpler function called once per row. Both are covered below.

Writing a batch callback

Evaluation callbacks must adhere to the EvaluationBatchCallback protocol. For instance an evaluator function should look like this:

from numpy.typing import NDArray
import numpy as np
from ropt.evaluation import EvaluationBatchContext, EvaluationBatchResult


def my_evaluator(
    variables: NDArray[np.float64],
    context: EvaluationBatchContext,
) -> EvaluationBatchResult:
    ...

variables has shape (n_rows, n_variables). Each row is a separate variable vector to evaluate.
context carries per-row metadata (see below) plus the immutable EnOptContext for the run.
The return value should be an EvaluationBatchResult object that packages objective values, and optionally constraint values and per-row metadata. A failed row is marked by writing numpy.nan into objectives; there is no separate error field.

The callback receives all variable vectors at once as a 2-D NumPy array, so the rows can be evaluated in a single vectorized pass rather than in an explicit Python loop.

What is in `EvaluationBatchContext`

The EvaluationBatchContext dataclass exposes:

Field	Meaning
`context`	The full `EnOptContext` (read-only).
`active`	A boolean array indicating which rows actually need evaluation.
`realizations`	Integer realization index for each row.
`perturbations`	Integer perturbation index per row, negative for unperturbed rows. `None` if no perturbations are used.
`batch_id`	Integer identifying the current evaluation batch.
`metadata`	The metadata the run was started with, or `None`.

Use realizations to pick the right per-realization model parameters (an uncertainty draw, a different simulation deck, etc.). Use active to skip rows that are not needed: utility methods get_active_evaluations and insert_inactive_results help filter the input and re-expand the output.

Returning results

EvaluationBatchResult stores:

Field	Meaning
`objectives`	Objective values, shape `(n_rows, n_objectives)`.
`constraints`	Optional constraint values, shape `(n_rows, n_nonlinear_constraints)`.
`metadata`	Optional per-row metadata dict; not used by `ropt`.
`batch_id`	Batch label (default `0`).

constraints is required when nonlinear_constraints is configured in the problem. metadata is stored verbatim on the resulting Results object, which is what links results back to the input vectors that produced them. batch_id defaults to 0; all results will carry this label unless you set it yourself. For auto-incrementing IDs pass a BatchIdCounter (or any Callable[[], int]) to the batch_id_callback argument of FunctionEvaluator or ParallelEvaluator; for raw BatchEvaluator callbacks set it yourself.

Inactive rows (where active is False) should have their result values set to zero. Rows where an evaluation failed should be set to np.nan (see Handling partial failures below).

Returning constraints

def evaluator(variables, context):
    obj = ...    # shape (n_rows, n_objectives)
    con = ...    # shape (n_rows, n_nonlinear_constraints)
    return EvaluationBatchResult(objectives=obj, constraints=con)

The constraint values are compared to the lower_bounds / upper_bounds declared in NonlinearConstraintsConfig.

Handling partial failures

If your evaluator cannot compute a given objective, set the corresponding entry in the objectives field to np.nan. ropt treats NaN rows as failed evaluations; the realization_min_success and perturbation_min_success settings determine whether the optimization can recover. For example:

def evaluator(variables, context):
    n_rows, n_obj = variables.shape[0], 1
    obj = np.full((n_rows, n_obj), np.nan)
    for row in range(n_rows):
        try:
            obj[row, 0] = simulate(variables[row])
        except SimulationError:
            pass  # leave NaN
    return EvaluationBatchResult(objectives=obj)

Combined with the realization_min_success field of RealizationsConfig, this allows the optimization to continue as long as enough realizations succeed.

Using `FunctionEvaluator`

When your evaluation function naturally works on a single variable vector at a time — for instance when it calls an external simulator once per realization — the FunctionEvaluator offers a simpler alternative. Instead of receiving the full 2-D batch and managing the loop yourself, you write a function that takes a single 1-D variable vector and returns the objective (and optional constraint) values for that row. The FunctionEvaluator handles the batching, the active-row filtering, and the assembly of the final EvaluationBatchResult.

A function passed to FunctionEvaluator must follow the EvaluationFunctionCallback protocol:

from numpy.typing import NDArray
import numpy as np

from ropt.components.evaluators import (
    EvaluationFunctionContext,
    EvaluationFunctionResult,
)


def my_function(
    variables: NDArray[np.float64],
    context: EvaluationFunctionContext,
) -> EvaluationFunctionResult:
    ...

variables is a 1-D array for a single evaluation row.

context is an EvaluationFunctionContext dataclass identifying the evaluation. It exposes:

Field	Meaning
`realization`	Integer realization index for this row.
`perturbation`	Perturbation index, or `-1` when unperturbed.
`batch_id`	Integer identifying the current evaluation batch.
`eval_idx`	Row index within the batch.
`metadata`	The metadata the run was started with, or `None`.

metadata is the dictionary passed to the compute step's run method, and is the channel for handing your own per-run data (a run ID, a case name, a path) to the evaluation function. Every evaluation of the run sees the same dictionary, so treat it as read-only; for a process or HPC executor its contents must be picklable.

The return value is an EvaluationFunctionResult dataclass with the following fields:

Field	Meaning
`objectives`	Scalar or 1-D array of length `n_objectives`.
`constraints`	Optional scalar or 1-D array of length `n_nonlinear_constraints`.
`metadata`	Optional `dict[str, Any]`; stored verbatim in the batch result.

Each metadata entry is forwarded into EvaluationBatchResult.metadata for the corresponding row.

A key need not be set by every row. Rows that do not set it get np.nan for numeric values and None otherwise, so a numeric column with missing rows is widened to float64 — numpy has no integer NaN. A column set by every row keeps its natural dtype. Mixing strings and non-strings under one key raises a ValueError.

A value may be a 1-D array rather than a scalar, which gives the key an extra user-defined axis named after the key. Every row must then return the same number of entries, and the key may not be named after an AxisName value or batch_id.

Wrap the function in a FunctionEvaluator and give the evaluator to a compute step (see Optimization Workflows):

from ropt.components.evaluators import FunctionEvaluator

evaluator = FunctionEvaluator(function=my_function)

Using `ParallelEvaluator`

The evaluators above run each function call sequentially in the current thread. For parallel evaluation — whether via worker threads, separate processes, or an HPC cluster — use ParallelEvaluator. See Parallel Evaluation for the evaluator and the available executors.

Reusing objectives and constraints

When defining multiple objectives, you may need to reuse the same underlying computation. For example, a total objective could consist of the mean of the realizations plus their standard deviation. Rather than evaluating all realizations twice, compute them once and return the values for both objectives from a single evaluator call.

Writing Evaluation Callbacks

The evaluators

Writing a batch callback

What is in EvaluationBatchContext

Returning results

Returning constraints

Handling partial failures

Using FunctionEvaluator

Using ParallelEvaluator

What is in `EvaluationBatchContext`

Using `FunctionEvaluator`

Using `ParallelEvaluator`