Nested Optimization

Some problems split naturally into two layers: a few variables that are awkward for one method — integers, say, or switches — and the rest, which a gradient method handles well. Nested optimization solves them in two loops. An outer optimization varies the awkward variables, and every outer evaluation runs a complete inner optimization over the remaining ones, returning the best value it reached.

This page also covers four things that are not nesting

Nesting itself is a niche, but this page puts four things that are not into one program small enough to read end to end: choosing a pool per layer, and why one of them has to stay on threads; collecting results from optimizations that overlap in time, in one handler; reusing an evaluation the optimizer has already made; and moving the expensive layer to a cluster by changing a single line.

Nothing in ropt is dedicated to this: the outer evaluation function calls optimize itself. What needs care is the plumbing around it — which variables each layer owns, which pool each layer evaluates on, and how to get the results out.

flowchart TB
    subgraph proc["your process"]
        direction LR
        outer["outer optimize<br/>(integer variables)"]
        subgraph touter["outer thread_pool — 2 threads"]
            direction TB
            in1["outer eval →<br/>inner optimize"]
            in2["outer eval →<br/>inner optimize"]
        end
        grp["shared<br/>DataFrameHandler"]
        outer --> in1
        outer --> in2
        in1 -.->|"results"| grp
        in2 -.->|"results"| grp
    end
    subgraph pinner["inner process_pool — 2 workers"]
        direction TB
        w1["worker"]
        w2["worker"]
    end
    in1 ==>|"batch"| w1
    in2 ==>|"batch"| w2

Solid arrows start an optimization, thick arrows carry evaluations out to a worker process, and dotted arrows carry results back to the handler. Both pools are drawn: the outer one is inside your process, the inner one is not. The sections below work through each part.

The runnable script for this page is examples/nested_optimization.py. It needs the polars extra.

Splitting the variables

Both layers describe the same variable vector; each is handed the half it may change. The mask field does this, and the two masks are complements:

DIM = 4
REALIZATIONS = 5
MASK = [True, True, False, False]
INNER_CONFIG: dict[str, Any] = {
    "variables": {
        "variable_count": DIM,
        "perturbation_magnitudes": 1e-6,
        "mask": MASK,
        "lower_bounds": 0.0,
        "upper_bounds": 10.0,
    },
    "realizations": {"weights": [1.0] * REALIZATIONS},
    "optimizer": {"max_functions": 8},
}

OUTER_CONFIG: dict[str, Any] = {
    "variables": {
        "variable_count": DIM,
        "mask": np.logical_not(MASK),
        "lower_bounds": 0.0,
        "upper_bounds": 10.0,
        "types": "integer",
    },
    "realizations": {"weights": [1.0]},
    "backend": {
        "method": "differential_evolution",
        "options": {"rng": 4},
        "parallel": True,
        "max_iterations": 4,
    },
}

A masked-out variable keeps the value it was given and is not passed to the optimizer, so the inner optimization treats the outer variables as constants and the outer optimization never touches the inner ones. The two layers can otherwise be configured completely differently — here the outer is integer-valued and gradient-free, while the inner is continuous and uses an ensemble of five realizations. See variables for the field, and Discrete and Mixed-Integer Variables for the outer method.

The outer evaluation function

An outer evaluation is an inner optimization. The function receives the outer variables, merges them into a full vector, runs optimize, and returns the best objective it found:

key = (context.realization, *variables.tolist())
if key in memo:
    return memo[key]

result = optimize(
    INNER_CONFIG,
    np.where(MASK, INITIAL_VALUES, variables),
    function,
    pool=pool,
    handlers=[tables],
    # Within a batch only (batch_id, row_idx) is unique: several rows share
    # a realization, so realization alone would not identify the caller.
    metadata={"outer_batch": context.batch_id, "outer_eval": context.row_idx},
)
if result.exit_code is ExitCode.TOO_FEW_REALIZATIONS:
    memo[key] = np.nan
else:
    assert result.results is not None
    assert result.results.target_objective is not None
    memo[key] = float(result.results.target_objective)
return memo[key]

np.where(MASK, INITIAL_VALUES, variables) builds the inner start point: the inner variables begin where they always do, and the outer ones carry the values being tried. The metadata tags every inner result with the outer evaluation that caused it, which is what makes the collected results traceable. The memo lookup above it is covered in Reusing an outer evaluation.

An inner optimization that ends with TOO_FEW_REALIZATIONS has no result, and the outer evaluation reports NaN for it. That is the same value an ordinary evaluation function returns for a realization that failed, so the outer optimization applies its own realization_min_success to it.

When an inner optimization fails

The inner optimization is started from the outer evaluation function, so aborting the outer optimization also aborts it. An exception it raises is raised in the outer evaluation function and aborts nothing outside the inner optimization. The evaluation function can catch it and return NaN for that realization:

try:
    result = optimize(inner_config, start, inner_function, pool=pool)
except RuntimeError:
    return numpy.nan

An exception that the outer evaluation function does not catch ends the outer optimization, as any exception from an evaluation function does. When the outer optimization is aborted, the inner optimizations are aborted with it. See Failures and Aborts for the rules that apply to every call.

Two pools, not one

Each layer evaluates with its own pool, and that is a requirement rather than a preference:

# A plain dict reaches the outer evaluations because they run on threads, in
# this process; on a process pool each worker would get an empty copy.
memo: dict[tuple[float, ...], float] = {}
with session() as s:
    optimize(
        OUTER_CONFIG,
        INITIAL_VALUES,
        partial(
            inner_optimization,
            pool=s.process_pool(workers=2),
            tables=tables,
            function=partial(rosenbrock, a=a, b=b),
            memo=memo,
        ),
        pool=s.thread_pool(workers=2),
    )

The outer pool is a thread pool. Outer evaluations therefore stay inside this process, where the inner pool and the shared handler are live objects; with a process pool they would arrive as copies, which ropt refuses. The inner pool is a process pool, which is where the real work goes.

That is also where a cluster belongs. Swapping the inner pool for an hpc_pool is the whole change — the inner evaluations become cluster jobs, and the layer above is untouched:

inner_pool = s.hpc_pool(workers=50, workdir="/scratch/my-run")

Sending whole optimizations to the cluster, by putting the outer evaluations on processes or HPC, is possible but gives something different. Each outer evaluation is then copied into a job, and neither the inner pool nor the shared handler can travel with it: the job has to open its own session and collect its own results, and return them as data for you to combine. Use that when the outer evaluations are genuinely independent; use the shape above when you want one pool and one table across all of them.

Each inner evaluation is sent to the inner pool separately. Two outer evaluations can run at once, sharing the inner pool's two workers; each inner optimization's realizations compete for those workers. See How many workers.

Handing the inner optimization the pool it is already running on would deadlock, and ropt refuses it rather than hanging — see Pools inside an evaluation for why.

Reusing an outer evaluation

In the example, the outer variables are integers and the outer method is population-based. The optimizer may propose points it has already evaluated. Each repeat would run a complete inner optimization for a value that is already known. The two lines at the top of the evaluation function return the stored value instead. The memo is an ordinary dictionary, created beside the pools and passed to the function through partial.

The key is exact. Integer variables arrive as whole-numbered doubles and are re-proposed unchanged, so a repeated vector is identical bit for bit. The realization index is part of the key because a memo keyed on the variables alone gives the wrong value for a problem with more than one realization.

The dictionary is reachable only because the outer evaluations run on threads, in the current process. With a process pool or an HPC pool the function is copied into a worker, which starts from an empty dictionary and discards it when it finishes, without raising. See Which pool should I use?.

Two outer evaluations run at the same time, so both can find the same key missing and both compute it. The stored value is the same either way, so the cost is one repeated evaluation. A lock would close that window and serialize the outer layer with it.

A repeat starts no inner optimization, so the table described next holds rows for the outer evaluations that were computed, not for every outer evaluation.

Collecting results from optimizations that overlap

The inner optimizations are concurrent, and one handler collects them all: it takes a lock around every event it handles, so two optimizations never write the same event at the same time. Give every inner optimization the same DataFrameHandler and every inner evaluation from every inner optimization lands in one table, keyed by the outer evaluation it belongs to.

The rows of one event stay together, but the events of overlapping inner optimizations arrive in thread order, so their rows interleave and the order differs from run to run. The outer batches do not overlap: an outer batch finishes before the next one starts. Sorting on the two keys groups the rows by inner optimization:

frame = tables["inner"].sort(["Outer-batch", "Outer-eval"], maintain_order=True)

The pair of keys identifies one inner optimization, because the row index starts again in every outer batch. maintain_order=True makes the sort stable, so the rows of one inner optimization keep the order in which they arrived, which is the order of its batches. Without it, polars may reorder rows with equal keys. Calculations that do not use the row order, such as finding the best objective, do not need the sort.

With the pandas engine, use frame = tables["inner"].sort_values(["Outer-batch", "Outer-eval"]) instead.

Reading the answer takes one more step than usual:

best = frame.sort("Objective").row(0, named=True)
variables = [best[f"Variable,{idx}"] for idx in range(DIM)]

For pandas, select the best row with best = frame.sort_values("Objective").iloc[0]; the variable extraction is unchanged.

The outer result is not the place to look. The outer layer only ever sees its own variables and holds the inner ones at their initial values, so the optimum over all variables exists only in the collected inner results.