Nested Optimization

Some problems split naturally into two layers: a few variables that are awkward for one method — integers, say, or switches — and the rest, which a gradient method handles well. Nested optimization solves them in two loops. An outer run varies the awkward variables, and every outer evaluation runs a complete inner optimization over the remaining ones, returning the best value it reached.

This page also covers four things that are not nesting

Nesting itself is a niche, but this page puts four things that are not into one program small enough to read end to end: choosing a pool per layer, and why one of them has to stay on threads; collecting results from runs that overlap in time, through a shared group; reusing an evaluation the optimizer has already made; and moving the expensive layer to a cluster by changing a single line.

Nothing in ropt is dedicated to this: the outer evaluation function calls optimize itself. What needs care is the plumbing around it — which variables each layer owns, which pool each layer evaluates on, and how to get the results out.

flowchart TB
    subgraph proc["your process"]
        direction LR
        outer["outer optimize<br/>(integer variables)"]
        subgraph tpool["outer thread_pool — 2 threads"]
            direction TB
            in1["outer eval →<br/>inner optimize"]
            in2["outer eval →<br/>inner optimize"]
        end
        grp["shared group →<br/>DataFrameHandler"]
        outer --> in1
        outer --> in2
        in1 -.->|"results"| grp
        in2 -.->|"results"| grp
    end
    subgraph ppool["inner process_pool — 2 workers"]
        direction TB
        w1["worker"]
        w2["worker"]
    end
    in1 ==>|"batch"| w1
    in2 ==>|"batch"| w2

Solid arrows start a run, thick arrows carry evaluations out to a worker process, and dotted arrows carry results back to the handler. Both pools are drawn: the outer one is inside your process, the inner one is not. The sections below work through each part.

The runnable script for this page is examples/simple/nested_optimization.py. It needs the polars extra.

Splitting the variables

Both layers describe the same variable vector; each is handed the half it may change. The mask field does this, and the two masks are complements:

DIM = 4
REALIZATIONS = 5
MASK = [True, True, False, False]
INNER_CONFIG: dict[str, Any] = {
    "variables": {
        "variable_count": DIM,
        "perturbation_magnitudes": 1e-6,
        "mask": MASK,
        "lower_bounds": 0.0,
        "upper_bounds": 10.0,
    },
    "realizations": {"weights": [1.0] * REALIZATIONS},
    "optimizer": {"max_functions": 8},
}

OUTER_CONFIG: dict[str, Any] = {
    "variables": {
        "variable_count": DIM,
        "mask": np.logical_not(MASK),
        "lower_bounds": 0.0,
        "upper_bounds": 10.0,
        "types": VariableType.INTEGER,
    },
    "realizations": {"weights": [1.0]},
    "backend": {
        "method": "differential_evolution",
        "options": {"rng": 4},
        "parallel": True,
        "max_iterations": 4,
    },
}

A masked-out variable keeps the value it was given and is not passed to the optimizer, so the inner run treats the outer variables as constants and the outer run never touches the inner ones. The two layers can otherwise be configured completely differently — here the outer is integer-valued and gradient-free, while the inner is continuous and uses an ensemble of five realizations. See variables for the field, and Discrete and Mixed-Integer Variables for the outer method.

The outer evaluation function

An outer evaluation is an inner optimization. The function receives the outer variables, merges them into a full vector, runs optimize, and returns the best objective it found:

key = (context.realization, *variables.tolist())
if key in memo:
    return memo[key]

result = optimize(
    INNER_CONFIG,
    np.where(MASK, INITIAL_VALUES, variables),
    function,
    pool=pool,
    handlers=[group],
    # Within a batch only (batch_id, eval_idx) is unique: several rows share
    # a realization, so realization alone would not identify the caller.
    metadata={"outer_batch": context.batch_id, "outer_eval": context.eval_idx},
)
assert result.results is not None
assert result.results.target_objective is not None
memo[key] = float(result.results.target_objective)
return memo[key]

np.where(MASK, INITIAL_VALUES, variables) builds the inner start point: the inner variables begin where they always do, and the outer ones carry the values being tried. The metadata tags every inner result with the outer evaluation that caused it, which is what makes the collected results traceable. The memo lookup above it is covered in Reusing an outer evaluation.

Two pools, not one

Each layer evaluates on its own pool, and that is a requirement rather than a preference:

# A plain dict reaches the outer evaluations because they run on threads, in
# this process; on a process pool each worker would get an empty copy.
memo: dict[tuple[float, ...], float] = {}
with session() as active:
    inner_pool = active.process_pool(workers=2, bundle_size=0)
    outer_pool = active.thread_pool(workers=2)
    group = active.shared_handlers(tables)
    optimize(
        OUTER_CONFIG,
        INITIAL_VALUES,
        partial(
            inner_optimization,
            pool=inner_pool,
            group=group,
            function=partial(rosenbrock, a=a, b=b),
            memo=memo,
        ),
        pool=outer_pool,
    )

The outer pool is a thread pool. Outer evaluations therefore stay inside this process, where the inner pool and the shared handler group are live objects; on a process pool they would arrive as copies, which ropt refuses. The inner pool is a process pool, which is where the real work goes.

That is also where a cluster belongs. Swapping the inner pool for an hpc_pool is the whole change — the inner evaluations become cluster jobs, and the layer above is untouched:

inner_pool = active.hpc_pool(workers=50)

Sending whole optimizations to the cluster, by putting the outer pool on processes or HPC, is possible but gives something different. Each outer evaluation is then copied into a job, and neither the inner pool nor the shared group can travel with it: the job has to open its own session and collect its own results, and return them as data for you to combine. Use that when the outer evaluations are genuinely independent; use the shape above when you want one pool and one table across all of them.

The inner pool takes bundle_size=0, which sends a whole inner batch to one worker as a single task. Within one inner run that gives up parallelism completely — its five realizations are evaluated one after another — but the parallelism here comes from the layer above: two outer evaluations run at once, each feeding the inner pool one task, so both workers are busy and each batch costs one transfer instead of five. See How a batch is split across workers.

Handing the inner run the pool it is already running on would deadlock, and ropt refuses it rather than hanging — see Pools inside an evaluation for why.

Reusing an outer evaluation

The outer variables are integers and the outer method is population-based, so the optimizer proposes points it has already evaluated. Each repeat would run a complete inner optimization for a value that is already known. The two lines at the top of the evaluation function return the stored value instead. The memo is an ordinary dictionary, created beside the pools and passed to the function through partial.

The key is exact. Integer variables arrive as whole-numbered doubles and are re-proposed unchanged, so a repeated vector is identical bit for bit. The realization index is part of the key because a memo keyed on the variables alone gives the wrong value for a problem with more than one realization.

The dictionary is reachable only because the outer evaluations run on threads, in this process. On a process_pool or an hpc_pool the function is copied into a worker, which starts from an empty dictionary and discards it when it finishes, without raising. See the rule to settle first.

Two outer evaluations run at the same time, so both can find the same key missing and both compute it. The stored value is the same either way, so the cost is one repeated evaluation. A lock would close that window and serialize the outer layer with it.

A repeat starts no inner run, so the table described next holds rows for the outer evaluations that were computed, not for every outer evaluation.

Collecting results from runs that overlap

The inner runs are concurrent, so a plain handler cannot collect them: a handler is claimed by one run at a time, and the second run to claim it is refused. A shared group can, because it routes every run's results through one dispatcher. Feed it a DataFrameHandler and every inner evaluation from every inner run lands in one table, keyed by the outer evaluation it belongs to.

Reading the answer takes one more step than usual:

best = frame.sort("Objective").row(0, named=True)
variables = [best[f"Variable,{idx}"] for idx in range(DIM)]

The outer result is not the place to look. The outer layer only ever sees its own variables and holds the inner ones at their initial values, so the optimum over all variables exists only in the collected inner results.