Skip to content

Optimization API

ropt

The main ropt module, a library for ensemble based optimization.

Import the names used to run an optimization directly from this package, for example from ropt import optimize, session. See Running Optimizations for a walkthrough.

A name is re-exported here when a program written against ropt has to spell it out: the functions it calls, the classes it instantiates or subclasses, and the types it puts in its own signatures. A name that is only reached through a value an optimization or evaluation returns keeps its own module. The fields of a FunctionResults are in ropt.results, the remaining enumerations in ropt.enums, and the classes describing the configuration dictionary in ropt.config.

Exceptions follow the same rule. Every error that can reach a caller derives from RoptError and is re-exported here; the control-flow signals that ropt raises and catches itself stay in ropt.exceptions.

Nothing about an optimization or evaluation depends on where it is called from. The functions optimize, evaluate, evaluate_batch and run take an optional pool argument that says where the work happens: without one it runs in-process on the calling thread, with a WorkerPool it runs on that pool's workers and belongs to the pool's session. This holds wherever the call is made from, including a thread you spawn yourself.

Pools come from a session, which releases them when it closes.

The remaining names imported from ropt are documented on their own pages: ExitCode; Results, FunctionResults, GradientResults, results_to_pandas and results_to_polars; and RoptError with WorkflowError, ExecutionError, UnsupportedError and AbortedError.

Running optimizations

ropt.optimize

optimize(
    config: dict[str, Any],
    x0: ArrayLike,
    function: EvaluationFunction,
    *,
    pool: WorkerPool | None = None,
    handlers: Sequence[EventHandler] | None = None,
    result_callback: ResultCallback | None = None,
    constraint_tolerance: float = 1e-10,
    metadata: dict[str, Any] | None = None,
    f0: FunctionResults | None = None,
    g0: GradientResults | None = None,
    report_gradients: bool = False,
) -> OptimizationResult

Run a single optimization.

The evaluations run on the workers of pool, or on the calling thread without one. See Running Optimizations for a walkthrough.

Parameters:

Name Type Description Default
config dict[str, Any]

The optimization configuration.

required
x0 ArrayLike

The initial variable vector.

required
function EvaluationFunction

The per-realization evaluation function.

required
pool WorkerPool | None

The pool to evaluate on, or None to evaluate in-process.

None
handlers Sequence[EventHandler] | None

Optional handlers, called in the order listed.

None
result_callback ResultCallback | None

Optional callback invoked per evaluation.

None
constraint_tolerance float

The tolerance within which a constraint holds.

1e-10
metadata dict[str, Any] | None

Default metadata passed to evaluations and attached to results.

None
f0 FunctionResults | None

Optional function results at x0.

None
g0 GradientResults | None

Optional gradient results at x0.

None
report_gradients bool

Whether result_callback also receives gradient results.

False

Returns:

Type Description
OptimizationResult

Raises:

Type Description
AbortedError

If the optimization was aborted.

WorkflowError

If the session of pool has closed.

Evaluating without optimizing

ropt.evaluate

evaluate(
    config: dict[str, Any],
    variables: ArrayLike,
    function: EvaluationFunction,
    *,
    pool: WorkerPool | None = None,
    handlers: Sequence[EventHandler] | None = None,
    result_callback: ResultCallback | None = None,
    metadata: dict[str, Any] | None = None,
) -> EvaluationResult[FunctionResults]

Evaluate a single variable vector, without optimizing.

The evaluations run on the workers of pool, or on the calling thread without one. See Running Optimizations for a walkthrough.

Parameters:

Name Type Description Default
config dict[str, Any]

The optimization configuration.

required
variables ArrayLike

The variable vector to evaluate.

required
function EvaluationFunction

The per-realization evaluation function.

required
pool WorkerPool | None

The pool to evaluate on, or None to evaluate in-process.

None
handlers Sequence[EventHandler] | None

Optional handlers, called in the order listed.

None
result_callback ResultCallback | None

Optional callback invoked with the results.

None
metadata dict[str, Any] | None

Default metadata passed to evaluations and attached to results.

None

Returns:

Type Description
EvaluationResult[FunctionResults]

An evaluation result.

Raises:

Type Description
AbortedError

If the evaluation was aborted.

ValueError

If variables is not a single vector.

WorkflowError

If the session of pool has closed.

ropt.evaluate_batch

evaluate_batch(
    config: dict[str, Any],
    variables: ArrayLike,
    function: EvaluationFunction,
    *,
    pool: WorkerPool | None = None,
    handlers: Sequence[EventHandler] | None = None,
    result_callback: ResultCallback | None = None,
    metadata: dict[str, Any] | None = None,
) -> EvaluationResult[tuple[FunctionResults, ...]]

Evaluate a batch of variable vectors, without optimizing.

Each row of variables is one vector, and the results come back in the same order. The evaluations run on the workers of pool, or on the calling thread without one. See Running Optimizations for a walkthrough.

Parameters:

Name Type Description Default
config dict[str, Any]

The optimization configuration.

required
variables ArrayLike

The variable vectors to evaluate, one per row.

required
function EvaluationFunction

The per-realization evaluation function.

required
pool WorkerPool | None

The pool to evaluate on, or None to evaluate in-process.

None
handlers Sequence[EventHandler] | None

Optional handlers, called in the order listed.

None
result_callback ResultCallback | None

Optional callback invoked with each evaluation.

None
metadata dict[str, Any] | None

Default metadata passed to evaluations and attached to results.

None

Returns:

Type Description
EvaluationResult[tuple[FunctionResults, ...]]

An evaluation result.

Raises:

Type Description
AbortedError

If the evaluation was aborted.

ValueError

If variables is not a 2-D matrix.

WorkflowError

If the session of pool has closed.

Running your own work

ropt.run

run(
    work: Callable[[], T], *, pool: WorkerPool | None = None
) -> T
run(
    work: Sequence[Callable[[], T]],
    *,
    pool: WorkerPool | None = None,
) -> tuple[T, ...]
run(
    work: Callable[[], T] | Sequence[Callable[[], T]],
    *,
    pool: WorkerPool | None = None,
) -> T | tuple[T, ...]

Run one or more arbitrary callables.

Pass a single zero-argument callable to run one call and get its result, or a sequence to get a tuple of results in the order of work. Bind arguments with functools.partial. With a pool the callables run concurrently on its workers, and must be picklable for a process or HPC pool. Without a pool they run one after another on the calling thread. See Running your own work for a walkthrough.

Parameters:

Name Type Description Default
work Callable[[], T] | Sequence[Callable[[], T]]

A single zero-argument callable, or a sequence of them.

required
pool WorkerPool | None

The pool to run on, or None to run in-process.

None

Returns:

Type Description
T | tuple[T, ...]

The single result, or a tuple of results in the order of work.

Raises:

Type Description
AbortedError

If an abort arrived before every callable finished.

ExecutionError

If the machinery could not run a call.

WorkflowError

If the session of pool has closed.

Evaluation functions

ropt.EvaluationFunctionContext dataclass

Context for a single function evaluation.

Attributes:

Name Type Description
realization int

The realization index.

perturbation int

The perturbation index (-1 when unperturbed).

function_id int

The function evaluation index within the batch (-1 when the row is not a function evaluation).

batch_id int

Integer identifying the current evaluation batch.

row_idx int

Row index within the batch.

metadata dict[str, Any]

Metadata for this batch, including any event-handler updates.

ropt.EvaluationFunctionResult dataclass

Result of a single function evaluation.

Attributes:

Name Type Description
objectives NDArray[float64] | float

The objective values as an array.

constraints NDArray[float64] | float | None

Optional constraint values as an array.

metadata dict[str, Any]

Extra information for this evaluation (default: empty).

Sessions and pools

ropt.session

session() -> Session

Open a session that owns the pools it builds.

Build pools with the session's factories, and pass a pool to the optimizations and evaluations that should use its workers. Closing the session releases every pool it built, so most code needs no further cleanup:

with session() as s:
    pool = s.thread_pool(workers=4)
    result = optimize(config, x0, objective, pool=pool)

An optimization started with optimize without a pool evaluates in-process and needs no session. See Sessions for an overview.

A call that fails aborts the work it started, and its exception is raised where the call was made. Not caught, it ends the with block, and closing the session aborts whatever is still running. Failures and Aborts describes each case.

Returns:

Type Description
Session

A context manager binding the Session.

ropt.Session

An open session, and the factories that build its pools.

A pool needs an owner that releases its workers, which is why pools are built here rather than constructed on their own. Everything a session creates is returned to the caller and passed on explicitly; nothing is discovered from the surroundings.

Bind the session with as and call its factories inside the block. See Sessions for an overview.

An optimization, evaluation or run call given a pool of the session runs on that pool's workers and belongs to this session. One without a pool in the session's with block, or inside a call made there, runs in-process and is aborted with the session.

A session may be opened inside another, and pools from different sessions never interact; a session opened inside another session's with block, or inside a call made there, is aborted with it. A session is single use — once closed it cannot be reopened.

abort aborts the calls given its pools or made in its with block, which is what a caller on another thread — a signal handler, a user interface — calls to bring them down.

__init__

__init__() -> None

Initialize the session.

__enter__

__enter__() -> Self

Open the session.

Returns:

Type Description
Self

The session itself.

Raises:

Type Description
WorkflowError

If the session was already opened.

__exit__

__exit__(*_exc: object) -> None

Close the session, aborting its calls and releasing every pool.

Work still running with its pools is waited for, five seconds at most in total, which gives the programs it started with run_program the time to end. Worker processes still busy after that are killed; threads are no longer waited for.

abort

abort() -> None

Abort the calls given its pools or made in its with block.

Each optimization, evaluation and run call under way raises AbortedError with ExitCode.USER_ABORT. The session stays aborted: a call given one of its pools afterwards, or made in its with block, is aborted as soon as it starts.

Calling it is thread-safe, and safe when nothing is running. See Aborting from outside for which calls are reached and when.

thread_pool

thread_pool(*, workers: int = 1) -> WorkerPool

Create a pool that runs evaluations in worker threads.

Use this where an evaluation spends its time waiting, or in a library that releases the GIL. See Running Optimizations for a walkthrough.

Parameters:

Name Type Description Default
workers int

The number of worker threads.

1

Returns:

Type Description
WorkerPool

A pool backed by worker threads.

Raises:

Type Description
WorkflowError

If this session has closed.

process_pool

process_pool(
    *,
    workers: int = 1,
    max_tasks_per_child: int | None = None,
) -> WorkerPool

Create a pool that runs evaluations in worker processes.

The evaluation function must be picklable. See Running Optimizations for a walkthrough.

An abort reaches an evaluation that is still running, as it does on a thread pool: aborting turns True, and a program started with run_program is terminated. Workers still busy five seconds after an abort, or after the session closes, are killed.

Parameters:

Name Type Description Default
workers int

The number of worker processes.

1
max_tasks_per_child int | None

Evaluations before a worker is replaced.

None

Returns:

Type Description
WorkerPool

A pool backed by worker processes.

Raises:

Type Description
WorkflowError

If this session has closed.

hpc_pool

hpc_pool(
    *,
    workdir: Path | str,
    workers: int = 1,
    interval: float | None = None,
    config_path: Path | str | None = None,
    cluster: str | None = None,
    queue: str | None = None,
    template: str | None = None,
    scheduler: str | None = None,
    cores: int = 1,
    memory_max: int | str | None = None,
    run_time_max: int | None = None,
    submit_options: dict[str, Any] | None = None,
    extra_polls: int | None = None,
    query_retries: int = 30,
    cleanup: bool = True,
) -> WorkerPool

Create a pool that runs evaluations on an HPC cluster.

Interfaces with a cluster queue (for example Slurm) through pysqa; requires the ropt[hpc] extra. Each evaluation is a job started as its own command, so the evaluation function must live in a module the compute nodes can import, or the ropt[cloudpickle] extra must be installed. The cluster is selected from cluster/queue: give a queue to search for its cluster, a cluster to use its default queue, or both to be explicit.

The queue "local" without a cluster runs each job as a process on this machine instead, with the same job files and the same rule for the evaluation function. It needs neither pysqa nor a configuration, and it ignores cores, memory_max, run_time_max, submit_options and query_retries. It is an error if a configured cluster also has a queue named "local"; pass that cluster to submit to it.

A template is the alternative to a configuration: it cannot be combined with config_path, cluster or queue. See Running on an HPC cluster for the configuration layout and Slurm examples.

Parameters:

Name Type Description Default
workdir Path | str

The shared-filesystem working directory.

required
workers int

The maximum number of concurrent cluster jobs.

1
interval float | None

Seconds between polls of the cluster (default: 1, or 0.1 for local jobs).

None
config_path Path | str | None

The pysqa configuration directory.

None
cluster str | None

The cluster name.

None
queue str | None

The queue name.

None
template str | None

A submission-script template.

None
scheduler str | None

The queueing system a template is written for.

None
cores int

The number of CPUs per job.

1
memory_max int | str | None

The memory per job.

None
run_time_max int | None

The run time per job.

None
submit_options dict[str, Any] | None

Extra variables for the submission script.

None
extra_polls int | None

Extra polls to wait for a work item's result (default: 30, or 0 for local jobs).

None
query_retries int

Times the cluster is re-polled for a result.

30
cleanup bool

Whether to remove the job files afterwards.

True

Returns:

Type Description
WorkerPool

A pool backed by an HPC cluster, or by local job processes.

Raises:

Type Description
WorkflowError

If this session has closed.

ropt.WorkerPool

The workers that evaluations are given to.

Built by a session factory such as thread_pool, and released when that session closes. A pool has no methods of its own: pass it as the pool argument of optimize, evaluate, evaluate_batch or run, which then run their work on its workers, nested in its session. See Evaluating in Parallel for a walkthrough.

This is the type to annotate with when your own code takes a pool.

__init__

__init__(session: Session, executor: Executor) -> None

Initialize a pool owned by a session.

Parameters:

Name Type Description Default
session Session

The session that built this pool.

required
executor Executor

The executor its evaluations run on.

required

ropt.aborting

aborting() -> bool

Return whether the call that runs the calling code is being aborted.

Code that a call runs — an evaluation function, an event handler, a report callback, or a function given to run — can call this to stop early once that call is aborted. In the with block of a session it returns whether the session is aborted. It returns False outside any call and session block, which includes the jobs of an HPC pool. In the worker processes of a process pool it turns True within a fraction of a second of the abort. See Failures and Aborts.

Returns:

Type Description
bool

Whether the call that runs the calling code is being aborted.

ropt.run_program

run_program(
    cmd: Sequence[str | PathLike[str]],
    *,
    cwd: str | PathLike[str] | None = None,
    env: Mapping[str, str] | None = None,
    stdout: ProgramOutput = None,
    stderr: ProgramOutput = None,
) -> int

Run an external program, and stop it when its caller is stopped.

Blocks until the program exits. The program, and everything it started, is terminated when the optimization, evaluation or run call that runs the caller is aborted, or the session in whose with block it is called; when an exception such as KeyboardInterrupt interrupts the call; or when the calling process ends. An abort raises AbortedError. Code outside any such call and session block is not reached by an abort.

stdout and stderr take None to inherit the caller's, a path to write to, an open file, a file descriptor, or subprocess.DEVNULL; stderr also takes subprocess.STDOUT. A relative path is relative to cwd. Standard input is empty. See Running External Programs.

Parameters:

Name Type Description Default
cmd Sequence[str | PathLike[str]]

The program and its arguments.

required
cwd str | PathLike[str] | None

The working directory, or None to inherit it.

None
env Mapping[str, str] | None

The environment, or None to inherit it.

None
stdout ProgramOutput

Where standard output goes.

None
stderr ProgramOutput

Where standard error goes.

None

Returns:

Type Description
int

The exit status of the program, negative for a signal as in subprocess.

Raises:

Type Description
AbortedError

If the caller was aborted.

OSError

If the program could not be started.

TypeError

If cmd is a single string.

ValueError

If stdout or stderr is subprocess.PIPE.

Handlers

ropt.EventHandler

Bases: ABC

Abstract base class for event handlers.

A concrete handler reacts to the events emitted by the run it is attached to, by implementing handle_event. Handlers may store state using dictionary-like access ([]).

Note

A handler's handle_event serializes itself: a call from a second thread waits for the first to finish, so the same handler may be attached to several runs at once. It is not re-entrant, so a call that reaches the same handler again on the same stack raises WorkflowError rather than deadlocking. See Handling Results for usage and pitfalls.

__init__

__init__() -> None

Initialize the EventHandler.

event_types abstractmethod property

event_types: set[EnOptEventType]

The event types that are handled.

Returns:

Type Description
set[EnOptEventType]

A set of event types that are handled.

handle_event

handle_event(event: EnOptEvent) -> None

React to an emitted event.

Calls from other threads are serialized: the second waits for the first to finish.

Parameters:

Name Type Description Default
event EnOptEvent

The event object.

required

Raises:

Type Description
WorkflowError

If this handler is already running on this call stack.

__getitem__

__getitem__(key: str) -> Any

Retrieve a stored value by key (handler[key]).

Parameters:

Name Type Description Default
key str

The string key identifying the value to retrieve.

required

Returns:

Type Description
Any

The value associated with the specified key.

Raises:

Type Description
AttributeError

If key does not exist in the stored values.

__setitem__

__setitem__(key: str, value: Any) -> None

Store or update a value in the internal state (handler[key] = value).

Parameters:

Name Type Description Default
key str

The string key identifying the value to store (must be an identifier).

required
value Any

The value to associate with the key.

required

Raises:

Type Description
AttributeError

If key is not a valid identifier.

ropt.CallbackHandler

Bases: EventHandler

Forward selected events to a callback.

The callback receives an EnOptEvent, not an individual result. Its return value is ignored. See Callback handlers for usage.

__init__

__init__(
    *,
    event_types: set[EnOptEventType],
    callback: Callable[[EnOptEvent], None],
) -> None

Initialize the CallbackHandler.

Parameters:

Name Type Description Default
event_types set[EnOptEventType]

The set of event types to respond to.

required
callback Callable[[EnOptEvent], None]

The callable to invoke for matching events.

required

event_types property

event_types: set[EnOptEventType]

The event types that are handled.

Returns:

Type Description
set[EnOptEventType]

A set of event types that are handled.

ropt.ResultsHandler

Bases: EventHandler

Track a single optimization result based on selection criteria.

Listens for FINISHED_EVALUATION events and retains either the best (lowest weighted objective) or most recent valid result. Optionally filters by constraint tolerance. The selected result is accessible via the result property or handler["results"].

The gradient computed at the selected result, if there is one, is available as handler["gradient"]. It may arrive in a later event than the result it belongs to, and stays None for a point where no gradient was computed.

See Result Handlers for full details on selection criteria and scaling.

__init__

__init__(
    *,
    what: Literal["best", "last"] = "best",
    constraint_tolerance: float | None = None,
    filter: Callable[[Results], bool] | None = None,
) -> None

Initialize the ResultsHandler.

Constraint violations are compared in the domain the optimizer works in, so a scale applies to them as well.

Parameters:

Name Type Description Default
what Literal['best', 'last']

Criterion for selecting results ('best' or 'last').

'best'
constraint_tolerance float | None

Optional threshold for constraint violations.

None
filter Callable[[Results], bool] | None

Optional callable to filter results based on custom logic.

None

result property

result: FunctionResults | None

The selected (best or last) result, or None if none is available.

gradient property

gradient: GradientResults | None

The gradient at the selected result, or None if there is none.

event_types property

event_types: set[EnOptEventType]

The event types that are handled.

Returns:

Type Description
set[EnOptEventType]

A set of event types that are handled.

ropt.HistoryHandler

Bases: EventHandler

Collect all optimization results into a tuple.

Listens for FINISHED_EVALUATION events and appends every Results object to a growing tuple accessible via the results property or handler["results"].

See Result Handlers for full details on scaling and accumulation behavior.

__init__

__init__() -> None

Initialize the HistoryHandler.

results property

results: tuple[Results, ...]

All results collected so far, in the order received.

event_types property

event_types: set[EnOptEventType]

The event types that are handled.

Returns:

Type Description
set[EnOptEventType]

A set of event types that are handled.

ropt.DataFrameHandler

Bases: EventHandler

Build pandas or polars DataFrames from optimization results.

Collects FunctionResults and GradientResults into named tables. Tables are defined via add_table with a column specification.

The frame library is chosen with the engine argument. The "polars" engine, the default, produces long-format tables in which batch and axis labels are ordinary leading columns; the "pandas" engine produces tables indexed by those instead.

Access tables via dictionary syntax: handler["functions"].

Warning

Tables are generated on the fly from internal data when accessing them in this way. When multiple accesses are needed, it is more efficient to first store them in a variable.

See Result Handlers for the column specification, the available fields, and callback details.

__init__

__init__(
    *, engine: DataFrameEngine = "polars", sep: str = ","
) -> None

Initialize a default table event handler.

Parameters:

Name Type Description Default
engine DataFrameEngine

Frame library used to build the tables.

'polars'
sep str

Separator used in column names.

','

Raises:

Type Description
ValueError

If engine is not "pandas" or "polars".

UnsupportedError

If the selected frame library is not installed.

engine property

engine: DataFrameEngine

The frame library used to build the tables.

Returns:

Type Description
DataFrameEngine

Either "pandas" or "polars".

set_callback

set_callback(
    callback: Callable[[Path | None], None],
) -> None

Set a function to call whenever the tables are updated.

The callback receives the output directory configured for the optimization (OptimizerConfig.output_dir, None if it is not set), and reads the tables from this handler. A callback that performs blocking operations, for example writing tables to disk, holds up the optimization that emitted the event.

Parameters:

Name Type Description Default
callback Callable[[Path | None], None]

A function that is called when the tables are updated.

required

add_table

add_table(
    name: str,
    table_type: Literal["functions", "gradients"],
    columns: dict[str, str],
) -> None

Register a new table to be populated from incoming results.

Column keys are result-field attribute names, in dotted attribute syntax.

Parameters:

Name Type Description Default
name str

Key under which the table is stored and looked up.

required
table_type Literal['functions', 'gradients']

Fill from "functions" or from "gradients".

required
columns dict[str, str]

Mapping from field paths to display titles.

required

get_tables

get_tables() -> dict[str, pd.DataFrame | pl.DataFrame]

Return the tables stored in the event handler.

Returns:

Type Description
dict[str, DataFrame | DataFrame]

A dictionary mapping table names to their corresponding tables.

Warning

Tables are generated on the fly from internal data. When multiple access is needed, it is more efficient to first store them in a variable.

event_types property

event_types: set[EnOptEventType]

The event types that are handled.

Returns:

Type Description
set[EnOptEventType]

A set of event types that are handled.

__getitem__

__getitem__(key: str) -> Any

Retrieve a of a table from the event handler.

Warning

The table is generated on the fly from internal data hen multiple access are needed, it is more efficient to first store them in a variable.

Parameters:

Name Type Description Default
key str

The string key identifying the table to retrieve.

required

Returns:

Type Description
Any

The table associated with the specified key.

Raises:

Type Description
AttributeError

If the requested table does not exist.

add_column

add_column(table: str, name: str, title: str) -> None

Add a column to a given table.

Parameters:

Name Type Description Default
table str

The name of the table to add the column to.

required
name str

The name of the field to add as a column, using attribute syntax.

required
title str

The title of the column to add.

required

Result objects

ropt.OptimizationResult dataclass

The outcome of a single optimization.

An optimization ends at one evaluation, the best one it found, and that evaluation is carried unchanged on results. See Running Optimizations for a walkthrough.

Attributes:

Name Type Description
exit_code ExitCode

Why the optimization terminated.

results FunctionResults | None

The best evaluation, or None if there was no valid result.

gradient GradientResults | None

The gradient at that evaluation, if one was computed there.

ropt.EvaluationResult dataclass

The outcome of a single evaluation.

What results holds depends on which function produced it: one FunctionResults from evaluate, or one per vector from evaluate_batch. An evaluation is a single batch: it produces every result, or raises AbortedError when it is aborted.

Attributes:

Name Type Description
results T

What the evaluation produced.

Callback types

ropt.EvaluationFunction

Bases: Protocol

The call signature for an evaluation function.

__call__

__call__(
    variables: NDArray[float64],
    context: EvaluationFunctionContext,
) -> FunctionValue

Evaluate the objectives and constraints for a single variable vector.

Parameters:

Name Type Description Default
variables NDArray[float64]

The 1-D variable vector for this evaluation.

required
context EvaluationFunctionContext

The context identifying the evaluation.

required

Returns:

Type Description
FunctionValue

An evaluation result, a scalar, or a sequence of values.

ropt.ResultCallback module-attribute

ResultCallback = Callable[[FunctionResults], bool | None]

Called with each function evaluation; returning True stops the optimization.

Results later in the same batch are not passed on.

With report_gradients set, the callback is also called with each GradientResults, and must accept those as well. This alias describes the default, so a callback that takes gradients too widens its own annotation.