Optimizer Backends

A backend is the bridge to an external optimization library. ropt ships with a SciPy-based backend and an external backend for running optimizers in a separate Python process; additional backends are provided by plugin packages (see Installation).

See Running the optimizer in a separate process for when and how to use the external backend.

ropt.backend

Public API for optimizer backend implementations.

Backends define how ropt runs an optimization algorithm against an OptimizationProblem. A backend requests function and gradient evaluations through the core callback interface, and advances the optimization from an initial variable vector toward a solution.

Core Interface

All backend implementations inherit from the Backend base class, which defines construction from a BackendConfig, a validation hook (validate_options), and the single entry point start.

Integration with Optimization

Backends are accessed via an EnOptContext object through its backend field. A backend is instantiated either directly as an object or via a BackendConfig object, which is used by the plugin system to create an instance based on the configured backend method string. The context itself never reaches the backend: ropt reduces it to an OptimizationProblem when the run starts.

During execution, a backend uses the OptimizerCallback interface to request objective, constraint, and gradient evaluations from the ropt core.

Built-in and Custom Backends

ropt includes two built-in backends:

SciPyBackend: Uses optimization methods provided by SciPy.
ExternalBackend: Delegates the optimization loop to an external executable or process.

Users can implement custom backends by subclassing Backend. Those subclasses can be instantiated directly and passed into an EnOptContext object through its backend field. Registering a custom backend with the plugin system is optional and only required when the backend should be selected and configured via BackendConfig objects instead of being instantiated explicitly by the user.

Backend

Bases: ABC

Abstract base class for optimizer backend implementations.

All concrete backend implementations must inherit from this class and implement the required lifecycle and validation methods. A backend is responsible for configuring a concrete optimization algorithm, interacting with the ropt evaluation pipeline through an OptimizerCallback, and executing the main optimization loop.

What a backend receives

What a backend is asked to solve arrives as an OptimizationProblem: already scaled, and already reduced to the free variables. A backend therefore neither scales nor masks anything; ropt unscales and expands results for reporting.

Non-linear constraints arrive normalized, as values that are non-negative when the constraint is satisfied, so a backend compares them against zero and never handles a bound. See OptimizerCallbackResult. A backend whose algorithm expects the opposite convention negates the values and their gradients.

Lifecycle

Instantiation via __init__: Called with a backend configuration object.
Validation via validate_options: Called to verify that the configured backend options are supported.
Execution via start: Called with the problem to solve and the callback that evaluates it.

Subclasses must implement:

__init__: Stores backend configuration and performs lightweight setup.
start: Runs the optimization algorithm.
validate_options: Verifies that backend-specific options are valid.

Subclasses may optionally override:

bypasses_python_output: Declares that the optimizer prints below the Python level.

Process-global state

A backend shares its process with everything else in the program, including other optimizations running at the same time. While a run is in progress it must therefore not change the working directory, the environment, sys.stdout or sys.stderr, or file descriptors 1 and 2. A backend wrapping a library that requires this, or that prints where it cannot be redirected per run, must document that it cannot run concurrently in-process and direct users to the external backend. See What a backend may not change for the full contract.

methods `class-attribute`

methods: MethodSpec

The optimization algorithms this class provides.

Either a set of names, which the registry matches case-insensitively, or a predicate for classes that cannot enumerate them. Include "default" in the set if this class has one. See MethodSpec.

init `abstractmethod`

__init__(backend_config: BackendConfig) -> None

Create a new backend instance.

Called during instantiation. Subclasses should store the configuration and perform any lightweight initialization. Validation and problem-dependent setup should usually be deferred to validate_options and start.

Method name with prefix

backend_config.method may be prefixed in the form "backend/method". Implementations should account for this when parsing the method name.

Handling the default method

backend_config.method may be set to "default", in which case it should be mapped to the backend's actual default method.

Parameters:

Name	Type	Description	Default
`backend_config`	`BackendConfig`	Configuration object specifying the backend method and any method-specific options.	required

start `abstractmethod`

start(
    problem: OptimizationProblem,
    optimizer_callback: OptimizerCallback,
    *,
    evaluation_policy: Literal[
        "speculative", "separate", "auto"
    ],
    output_dir: Path | None,
) -> None

Run the optimization algorithm on the given problem.

Starts the backend's main optimization loop. During execution, the implementation uses optimizer_callback to request any objective, constraint, or gradient evaluations its algorithm needs.

Called at most once per backend instance.

Parameters:

Name	Type	Description	Default
`problem`	`OptimizationProblem`	The problem to solve, in free-variable space.	required
`optimizer_callback`	`OptimizerCallback`	Callback used to request objective, constraint, and gradient evaluations from the `ropt` core.	required
`evaluation_policy`	`Literal['speculative', 'separate', 'auto']`	Whether functions and gradients should be asked for together (`"speculative"`), in separate calls (`"separate"`), or as the algorithm happens to need them (`"auto"`).	required
`output_dir`	`Path \| None`	Directory for any files the optimizer writes, or `None` if none was configured.	required

bypasses_python_output `property`

bypasses_python_output: bool

Indicate whether the optimizer prints without going through Python.

Compiled optimizers commonly write to file descriptors 1 and 2 directly instead of through sys.stdout, which puts their output beyond reach of the capture ropt applies for the stdout and stderr settings of OptimizerConfig. A backend wrapping such a library should override this to return True, and ropt then redirects the descriptors as well.

The answer may differ per method, in which case return it based on the configured method. Return True for the whole backend when unsure: the cost is that capture briefly rewires process-global state, whereas the cost of being wrong the other way is output escaping to the terminal.

See Writing a Plugin.

Returns:

Type	Description
`bool`	`True` if the optimizer writes output below the Python level.

validate_options `abstractmethod`

validate_options() -> None

Validate backend-specific options for the configured method.

Checks that the options supplied through the BackendConfig object have the expected type, contain only supported keys, and satisfy any method-specific value constraints.

Concrete backends should implement validation logic for the methods they support, potentially using schema-validation tools such as Pydantic.

The raised exception must be a ValueError, or derive from a ValueError.

Note

Backend options may be represented as a dictionary or list, depending on the backend. This method should verify that the type matches what the backend expects and raise a ValueError with a clear message when it does not.

Raises:

Type	Description
`ValueError`	If the provided options are invalid.

OptimizationProblem

The problem a backend is asked to solve.

Everything on this object is scaled and lives in free-variable space: the variables that variables.mask fixes are gone, the bounds and the linear constraints are reduced to the ones that remain, and the values delivered through the OptimizerCallback use the same space. A backend therefore neither scales nor masks anything.

See Backend for the rest of the contract.

Attributes:

Name	Type	Description
`initial_values`		The starting point, shape `(variable_count,)`.
`lower_bounds`		The lower bounds, shape `(variable_count,)`.
`upper_bounds`		The upper bounds, shape `(variable_count,)`.
`variable_types`		The type of each variable, shape `(variable_count,)`.

init

__init__(
    context: EnOptContext, initial_values: NDArray[float64]
) -> None

Reduce a context and a full-space starting point to a problem.

Parameters:

Name	Type	Description	Default
`context`	`EnOptContext`	The optimization context.	required
`initial_values`	`NDArray[float64]`	The values of all variables, including fixed ones.	required

variable_count `property`

variable_count: int

The number of free variables.

Returns:

Type	Description
`int`	The number of variables the optimizer sees.

linear_constraints `property`

linear_constraints: (
    tuple[
        NDArray[float64],
        NDArray[float64],
        NDArray[float64],
        NDArray[bool_],
    ]
    | None
)

The linear constraints, reduced to the problem the optimizer solves.

Rows that cannot constrain the optimization are dropped: a row is kept only if it has a non-zero coefficient on a free variable and at least one finite bound. The columns of the fixed variables are removed, and the contribution those variables make is folded into the bounds.

The four arrays are aligned, with one entry per surviving constraint. equality marks the constraints whose bounds coincide; it follows the bounds as configured, so scaling cannot change it. Pass the tuple to split_linear_constraints to get the same normalized form as the non-linear constraints.

Returns:

Type	Description
`tuple[NDArray[float64], NDArray[float64], NDArray[float64], NDArray[bool_]] \| None`	`(coefficients, lower_bounds, upper_bounds, equality)`, or `None`.

nonlinear_equalities `property`

nonlinear_equalities: NDArray[bool_] | None

Which non-linear constraint values are equalities.

One flag per value delivered through the OptimizerCallback: a constraint with a finite lower and a finite upper bound delivers two values, an equality one, and a constraint with no finite bound none.

Returns:

Type	Description
`NDArray[bool_] \| None`	A flag per value, or `None` if there are no non-linear constraints.

validate_supported_constraints

validate_supported_constraints(
    method: str,
    supported_constraints: dict[str, set[str]],
    required_constraints: dict[str, set[str]],
) -> None

Raise if this problem's constraints do not suit the chosen method.

Constraint types are identified by the keys "bounds", "linear:eq", "linear:ineq", "nonlinear:eq" and "nonlinear:ineq".

Parameters:

Name	Type	Description	Default
`method`	`str`	The name of the optimization method used.	required
`supported_constraints`	`dict[str, set[str]]`	Maps each constraint type to the methods that support it.	required
`required_constraints`	`dict[str, set[str]]`	Maps each constraint type to the methods that require it.	required

Raises:

Type	Description
`UnsupportedError`	If a constraint present in the problem is not supported by the method, or a constraint the method requires is absent.

ropt.backend.scipy.SciPyBackend

Bases: Backend

Backend implementation using SciPy optimization algorithms.

Implements the Backend interface to expose optimization algorithms from scipy.optimize to ropt.

The algorithm is selected via the method field of the BackendConfig object. Not every algorithm accepts every kind of problem:

Method	Gradient	Bounds	Linear eq.	Linear ineq.	Non-linear eq.	Non-linear ineq.	Integer
Nelder-Mead
Powell
CG
BFGS
Newton-CG
L-BFGS-B
TNC
COBYLA
COBYQA
SLSQP
trust-constr
differential_evolution		required

A method without a gradient asks only for function values, so no perturbations are evaluated for it. Configuring a method with a constraint it does not support raises UnsupportedError when the run starts. differential_evolution requires bound constraints on all variables, and is the only method that handles integer variables; the others silently treat them as continuous.

SciPy prints its own progress report when verbose asks for it. trust-constr is the only method with reporting levels of its own; for the rest the setting is on or off. Note that disp is a no-op for l-bfgs-b and nelder-mead in recent SciPy releases, which therefore report nothing.

Algorithm-specific options are passed through the options dictionary. Click on the common options or the method name for the corresponding scipy.optimize documentation:

Common Options:

disp, maxiter, keep_feasible

The keep_feasible option is used to maintain feasibility with respect to bound, linear and non-linear constraints, by passing it to the constraint handling code of the underlying SciPy optimizer. Some algorithms may choose to ignore this option.

Hessian Options:

hess, exception_strategy, min_curvature, min_denominator, init_scale

These options are used to configure the Hessian approximation method for the optimizer. The hess option specifies the type of Hessian approximation to use ("BFGS" or "SR1"), while the other options provide additional parameters for the chosen method.

Method-specific Options:

Method	Options
Nelder-Mead	maxfev, xatol, fatol, adaptive
Powell	maxfev, xtol, ftol
CG	gtol, norm, eps, finite_diff_rel_step, c1, c2
BFGS	gtol, norm, eps, finite_diff_rel_step, xrtol, c1, c2
Newton-CG	xtol, eps, c1, c2
L-BFGS-B¹	disp, maxcor, ftol, gtol, eps, maxfun, iprint, maxls, finite_diff_rel_step
TNC²	maxfun, eps, scale, offset, maxCGit, eta, stepmx, accuracy, minfev, ftol, xtol, gtol, rescale, finite_diff_rel_step, ~~maxiter~~
COBYLA	rhobeg, tol, catol
COBYQA	maxfev, f_target, feasibility_tol, initial_tr_radius, final_tr_radius, scale
SLSQP	ftol, eps, finite_diff_rel_step
trust-constr	gtol, xtol, barrier_tol, sparse_jacobian, initial_tr_radius, initial_constr_penalty, initial_barrier_parameter, initial_barrier_tolerance, factorization_method, finite_diff_rel_step, verbose
differential_evolution	strategy, popsize, tol, mutation, recombination, rng, polish, init, atol, updating

Notes:

Options in italics override a common option with a different type or behavior.
Options with ~~strikethrough~~ indicate a common option that is not supported.

init

__init__(backend_config: BackendConfig) -> None

Initialize the SciPy backend.

Parameters:

Name	Type	Description	Default
`backend_config`	`BackendConfig`	The backend configuration.	required

Raises:

Type	Description
`UnsupportedError`	If the requested method is not supported.

ropt.backend.external.ExternalBackend

Bases: Backend

Backend implementation that runs an optimizer in a separate process.

Implements the Backend interface by spawning a child process to run a delegate backend. The child process performs the optimization independently and communicates back through queues to request function evaluations, report optimizer states, and propagate errors.

Method naming

Unlike other backends, the method field of BackendConfig must include both the plugin and method name in one of these forms:

external/plugin-name/method-name
external/method-name

The external/ prefix is stripped before the remainder is forwarded to the delegate plugin. Standard plugin-name/method-name resolution without the prefix is not supported by this backend.

Note

The problem is sent to the child process by serializing it, so everything the delegate needs must be serializable, including the delegate class itself. The standard library can send anything that can be looked up by name, which covers the built-in plugins and any plugin class defined in an importable module. Installing the optional cloudpickle extra lifts that restriction, so plugins and plugin instances of classes defined inside a function or a notebook can be sent as well.

init

__init__(backend_config: BackendConfig) -> None

Initialize the external backend.

Parameters:

Name	Type	Description	Default
`backend_config`	`BackendConfig`	The backend configuration; its `method` field must be prefixed with `external/`.	required

ropt.backend.utils

Utility functions for use by optimizer backend plugins.

This module provides helpers for resolving the requested reporting level, checking what the optimizer writes below the Python level, unique output path construction, and splitting linear constraints into the same normalized form as the non-linear constraints delivered through the callback.

The problem itself, including its constraints, arrives as an OptimizationProblem. Every array these helpers accept or return is scaled, as is everything else a backend sees.

resolve_verbosity

resolve_verbosity(
    *, verbose: bool | int | None
) -> int | None

Resolve how much the optimizer should report.

Normalizes the verbose field of BackendConfig into a single value, so that backends need not distinguish True from 1 themselves:

Result	Meaning
`None`	Report at the optimizer's own default level.
`0`	Do not report.
`n`	Report at level `n`, clamped to what the optimizer offers.

This says nothing about where the output goes: that is decided by the stdout and stderr settings of OptimizerConfig.

Parameters:

Name	Type	Description	Default
`verbose`	`bool \| int \| None`	The `verbose` field of the backend configuration.	required

Returns:

Type	Description
`int \| None`	The reporting level, or `None` for the optimizer's own default.

collect_native_output

collect_native_output(run: Callable[[], None]) -> str

Collect the output a callable writes below the Python level.

Runs run with sys.stdout and sys.stderr replaced, so that everything written through Python is diverted, and with file descriptors 1 and 2 pointing at a temporary file. Whatever reaches that file was therefore written without going through Python.

Use this to check a backend's bypasses_python_output declaration from its test suite: a non-empty result means the declaration must be True for the method that was run.

Parameters:

Name	Type	Description	Default
`run`	`Callable[[], None]`	A callable that runs an optimization with the backend under test.	required

Returns:

Type	Description
`str`	Everything the callable wrote below the Python level.

create_output_path

create_output_path(
    base_name: str,
    base_dir: Path | None = None,
    name: str | None = None,
    suffix: str | None = None,
) -> Path

Construct a unique output path, appending an index if necessary.

Builds a path from the provided components. If the resulting path already exists on disk, a three-digit counter suffix (for example -001, -002) is appended or incremented until a non-existing path is found.

The path is assembled as: <base_dir>/<base_name>[-<name>][-<index>][<suffix>]

base_dir is created (including parents) if it does not exist.

Parameters:

Name	Type	Description	Default
`base_name`	`str`	Base file or directory name.	required
`base_dir`	`Path \| None`	Parent directory. If `None`, the path is relative to the current working directory.	`None`
`name`	`str \| None`	Optional label appended to `base_name` with a `-` separator.	`None`
`suffix`	`str \| None`	Optional file extension including the leading dot	`None`

Returns:

Type	Description
`Path`	A `pathlib.Path` that does not currently exist on disk.

split_linear_constraints

split_linear_constraints(
    coefficients: NDArray[float64],
    lower_bounds: NDArray[float64],
    upper_bounds: NDArray[float64],
    equality: NDArray[bool_],
) -> tuple[
    NDArray[np.float64],
    NDArray[np.float64],
    NDArray[np.bool_],
]

Split linear constraints into values compared against zero.

Produces the same form as the non-linear constraint values delivered through the OptimizerCallback, so that a backend that handles both together can concatenate them. Entry k evaluates to coefficients[k] @ variables - offsets[k], which is non-negative when the constraint is satisfied. A constraint with a finite lower and a finite upper bound contributes two entries, and an equality one.

The arguments are those returned by linear_constraints.

Parameters:

Name	Type	Description	Default
`coefficients`	`NDArray[float64]`	The constraint coefficients.	required
`lower_bounds`	`NDArray[float64]`	The lower bounds.	required
`upper_bounds`	`NDArray[float64]`	The upper bounds.	required
`equality`	`NDArray[bool_]`	Which constraints have coinciding bounds.	required

Returns:

Type	Description
`tuple[NDArray[float64], NDArray[float64], NDArray[bool_]]`	A tuple of `(coefficients, offsets, equality)`, with one entry per value.

Optimizer Backends

ropt.backend

Backend

methods class-attribute

__init__ abstractmethod

start abstractmethod

bypasses_python_output property

validate_options abstractmethod

OptimizationProblem

__init__

variable_count property

linear_constraints property

nonlinear_equalities property

validate_supported_constraints

ropt.backend.scipy.SciPyBackend

__init__

ropt.backend.external.ExternalBackend

__init__

ropt.backend.utils

resolve_verbosity

collect_native_output

create_output_path

split_linear_constraints

methods `class-attribute`

init `abstractmethod`

start `abstractmethod`

bypasses_python_output `property`

validate_options `abstractmethod`

init

variable_count `property`

linear_constraints `property`

nonlinear_equalities `property`

init

init