Simple API
ropt.simple
The high-level convenience API for running optimizations.
This module builds on the low-level ropt primitives. Import its names
directly, for example from ropt.simple import optimize, session. See
Running Optimizations for a walkthrough.
Enumerations used in the configuration and results (for example
ExitCode and VariableType)
are not re-exported here; import them from ropt.enums.
Nothing about a run depends on where it is called from. Where its evaluations
happen is decided by the pool it is given with pool=, and which handlers see
its results by the handlers= it is given. A session
hands out both; a run given no pool evaluates in-process. This holds wherever
the run is started from, including a thread you spawn yourself.
Running optimizations
ropt.simple.optimize
optimize(
config: dict[str, Any],
x0: ArrayLike,
function: EvaluationFunction,
*,
pool: WorkerPool | None = None,
handlers: Sequence[EventHandler | SharedHandlers]
| None = None,
report: ReportCallback | None = None,
constraint_tolerance: float = 1e-10,
metadata: dict[str, Any] | None = None,
) -> OptimizeResult
Run a single optimization.
See Running Optimizations for a walkthrough.
A pool or group that is closed — because it was closed directly, or
because its session ended — is refused here with a
WorkflowError, as is one carried into
a worker process, where it cannot work at all.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
dict[str, Any]
|
The optimization configuration. |
required |
x0
|
ArrayLike
|
The initial variable vector. |
required |
function
|
EvaluationFunction
|
The per-realization evaluation function. |
required |
pool
|
WorkerPool | None
|
The pool to evaluate on, from a session factory
such as
|
None
|
handlers
|
Sequence[EventHandler | SharedHandlers] | None
|
Optional result handlers, mixing two kinds. An
|
None
|
report
|
ReportCallback | None
|
An optional callback invoked with an
|
None
|
constraint_tolerance
|
float
|
The tolerance within which a constraint is considered satisfied. Violations are compared in the domain the optimizer works in, so a scale applies to them as well. |
1e-10
|
metadata
|
dict[str, Any] | None
|
An optional dictionary attached to every
|
None
|
Returns:
| Type | Description |
|---|---|
OptimizeResult
|
A |
ropt.simple.optimize_many
optimize_many(
config: dict[str, Any] | Sequence[dict[str, Any]],
x0: ArrayLike,
function: EvaluationFunction
| Sequence[EvaluationFunction],
*,
pool: WorkerPool | None = None,
handlers: Sequence[SharedHandlers] | None = None,
report: ReportCallback
| Sequence[ReportCallback]
| None = None,
limit: int | None = None,
constraint_tolerance: float = 1e-10,
metadata: dict[str, Any]
| Sequence[dict[str, Any]]
| None = None,
) -> tuple[OptimizeResult, ...]
Run several optimizations concurrently, sharing one pool.
Each of config, x0, and function may be a single value (used for
every run) or a sequence (one per run). Sequences set the number of runs and
must agree in length; single values are broadcast. A single x0 is a 1-D
vector; a per-run sequence of x0s is a 2-D matrix with one vector per row.
A sequence that is empty gives no runs, and returns no results.
The runs execute concurrently on driver threads and all evaluate on the
same pool, so its workers are shared between them; limit bounds how
many run simultaneously. Without a pool the runs still overlap, but each
evaluation runs in-process on its own driver thread. See
Parallel Execution and Many Runs
for a walkthrough, and Failure in one run
for what happens when one raises.
Unlike optimize, this takes no local handlers: a local handler belongs
to one run at a time and these runs overlap, so handlers= accepts only
SharedHandlers groups, which every run feeds
together. report=, being local by nature, is the opposite: it is given per
run, or broadcast to all of them.
A pool or group that is closed — because it was closed directly, or
because its session ended — is refused here with a
WorkflowError, as is one carried into
a worker process, where it cannot work at all.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
dict[str, Any] | Sequence[dict[str, Any]]
|
The configuration, or one per run. |
required |
x0
|
ArrayLike
|
The initial variable vector, or one per row. |
required |
function
|
EvaluationFunction | Sequence[EvaluationFunction]
|
The evaluation function, or one per run. |
required |
pool
|
WorkerPool | None
|
The pool every run evaluates on, from a session
factory such as
|
None
|
handlers
|
Sequence[SharedHandlers] | None
|
Optional |
None
|
report
|
ReportCallback | Sequence[ReportCallback] | None
|
An optional callback invoked with an
|
None
|
limit
|
int | None
|
The maximum number of runs to execute at once. |
None
|
constraint_tolerance
|
float
|
The tolerance within which a constraint is considered satisfied. Violations are compared in the domain the optimizer works in, so a scale applies to them as well. |
1e-10
|
metadata
|
dict[str, Any] | Sequence[dict[str, Any]] | None
|
An optional dictionary attached to every
|
None
|
Returns:
| Type | Description |
|---|---|
tuple[OptimizeResult, ...]
|
One |
Raises:
| Type | Description |
|---|---|
WorkflowError
|
If |
Evaluating without optimizing
ropt.simple.evaluate
evaluate(
config: dict[str, Any],
variables: ArrayLike,
function: EvaluationFunction,
*,
pool: WorkerPool | None = None,
handlers: Sequence[EventHandler | SharedHandlers]
| None = None,
report: ReportCallback | None = None,
metadata: dict[str, Any] | None = None,
) -> EvaluateResult
Evaluate a single variable vector without optimizing.
Use evaluate_many to evaluate several
vectors at once. See Running Optimizations for a
walkthrough.
A pool or group that is closed — because it was closed directly, or
because its session ended — is refused here with a
WorkflowError, as is one carried into
a worker process, where it cannot work at all.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
dict[str, Any]
|
The optimization configuration. |
required |
variables
|
ArrayLike
|
The variable vector to evaluate. |
required |
function
|
EvaluationFunction
|
The per-realization evaluation function. |
required |
pool
|
WorkerPool | None
|
The pool to evaluate on, from a session factory such as
|
None
|
handlers
|
Sequence[EventHandler | SharedHandlers] | None
|
Optional result handlers, mixing local
|
None
|
report
|
ReportCallback | None
|
An optional callback invoked with an |
None
|
metadata
|
dict[str, Any] | None
|
An optional dictionary attached to the emitted
|
None
|
Returns:
| Type | Description |
|---|---|
EvaluateResult
|
An |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
ropt.simple.evaluate_many
evaluate_many(
config: dict[str, Any],
variables: ArrayLike,
function: EvaluationFunction,
*,
pool: WorkerPool | None = None,
handlers: Sequence[EventHandler | SharedHandlers]
| None = None,
report: ReportCallback | None = None,
metadata: dict[str, Any] | None = None,
) -> tuple[EvaluateResult, ...]
Evaluate a batch of variable vectors without optimizing.
Each row of variables is one variable vector; the results are returned in
the same order. See Running Optimizations for a
walkthrough.
A pool or group that is closed — because it was closed directly, or
because its session ended — is refused here with a
WorkflowError, as is one carried into
a worker process, where it cannot work at all.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
dict[str, Any]
|
The optimization configuration. |
required |
variables
|
ArrayLike
|
The variable vectors to evaluate, one per row. |
required |
function
|
EvaluationFunction
|
The per-realization evaluation function. |
required |
pool
|
WorkerPool | None
|
The pool to evaluate on, from a session factory such as
|
None
|
handlers
|
Sequence[EventHandler | SharedHandlers] | None
|
Optional result handlers, mixing local
|
None
|
report
|
ReportCallback | None
|
An optional callback invoked with an |
None
|
metadata
|
dict[str, Any] | None
|
An optional dictionary attached to every emitted
|
None
|
Returns:
| Type | Description |
|---|---|
tuple[EvaluateResult, ...]
|
One |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Sessions and pools
ropt.simple.session
Open a background session that pools and shared handlers run on.
The session owns one event loop, on a daemon thread, for as long as the block is open. Build pools on it with its factories, and pass them to the runs that should use them:
Closing the session releases every pool it created, so most code needs no further cleanup. See Running Optimizations for a walkthrough.
Returns:
| Type | Description |
|---|---|
Session
|
A context manager owning the session, which binds the |
Session
|
|
ropt.simple.Session
An open session, and the factories that build on it.
A session owns one background event loop, on its own daemon thread. Pools need that loop to run on, which is why they are built here rather than constructed on their own. Everything a session hands out is returned to the caller and passed on explicitly; nothing is discovered from the surroundings.
Bind the session with as and call its factories inside the block:
Sessions are objects, so opening one inside another is unremarkable: each gets its own loop, and pools from different sessions never interact. A session is single use — once closed it cannot be reopened.
__enter__
Open the session's event loop.
Returns:
| Type | Description |
|---|---|
Self
|
The session itself. |
Raises:
| Type | Description |
|---|---|
WorkflowError
|
If the session was already opened. |
thread_pool
Create a pool that runs evaluations in worker threads.
See Running Optimizations for a walkthrough.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
workers
|
int
|
The number of worker threads. |
1
|
bundle_size
|
int
|
How many evaluations go to a worker as one task, |
1
|
Returns:
| Type | Description |
|---|---|
WorkerPool
|
A pool backed by a thread pool. |
process_pool
Create a pool that runs evaluations in worker processes.
The evaluation function must be picklable. See Running Optimizations for a walkthrough.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
workers
|
int
|
The number of worker processes. |
1
|
bundle_size
|
int
|
How many evaluations go to a worker as one task. Every
task is transferred to a worker separately, and the
evaluations within one run after another, so this is a
trade between spreading a batch and the cost of moving
it. The default of 1 gives every evaluation its own
task, spreading a batch as widely as the workers allow;
a larger value groups that many per task; and |
1
|
Returns:
| Type | Description |
|---|---|
WorkerPool
|
A pool backed by a process pool. |
local_pool
local_pool(
*,
workers: int = 1,
workdir: Path | str | None = None,
retries: int = 0,
bundle_size: int = 1,
) -> WorkerPool
Create a pool that runs each evaluation as a separate local process.
Between a process pool and a cluster: every evaluation gets an
interpreter of its own, which can be stopped outright and whose output
is captured to a file, but there is no queueing system and nothing to
install. This is the local stand-in for
hpc_pool: the same job shape, so an
evaluation function that works here works there.
Each job is a fresh command rather than a re-import of your script, so
the evaluation function must live in a module the job can import, or
the ropt[cloudpickle] extra must be installed — which is the
recommended way to use this pool.
POSIX only. See Running Optimizations for a walkthrough.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
workers
|
int
|
The maximum number of concurrent local jobs. |
1
|
workdir
|
Path | str | None
|
The directory holding each evaluation's files. The default is a temporary directory that is removed when the pool closes — unless an evaluation failed, in which case it is kept, with that evaluation's output in it, and its path logged. |
None
|
retries
|
int
|
Extra polls to wait for a result. The default of |
0
|
bundle_size
|
int
|
How many evaluations go to a worker as one task, |
1
|
Returns:
| Type | Description |
|---|---|
WorkerPool
|
A pool backed by local processes. |
hpc_pool
hpc_pool(
*,
workers: int = 1,
cores: int = 1,
cluster: str | None = None,
queue: str | None = None,
workdir: Path | str | None = None,
config_path: Path | str | None = None,
template: str | None = None,
scheduler: str | None = None,
memory_max: int | str | None = None,
run_time_max: int | None = None,
submit_options: dict[str, Any] | None = None,
retries: int = 30,
bundle_size: int = 1,
) -> WorkerPool
Create a pool that runs evaluations on an HPC cluster.
Interfaces with a cluster queue (for example Slurm) through pysqa;
requires the ropt[hpc] extra. Each evaluation is a job started as its
own command, so the evaluation function must live in a module the
compute nodes can import, or the ropt[cloudpickle] extra must be
installed — which is the recommended way to use this pool. Develop
against local_pool first: it has the
same shape and the same rule, without a cluster. The cluster is selected
from cluster/queue: give a queue to search for its cluster, a
cluster to use its default queue, or both to be explicit.
A template is the alternative to all of that: it submits without a
configuration, so it cannot be combined with config_path, cluster or
queue. See Running on an HPC
cluster for the
configuration layout and Slurm examples.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
workers
|
int
|
The maximum number of concurrent cluster jobs. |
1
|
cores
|
int
|
The number of CPUs per job. |
1
|
cluster
|
str | None
|
The cluster name. |
None
|
queue
|
str | None
|
The queue name. |
None
|
workdir
|
Path | str | None
|
The shared-filesystem working directory. |
None
|
config_path
|
Path | str | None
|
The |
None
|
template
|
str | None
|
A submission-script template. |
None
|
scheduler
|
str | None
|
The queueing system a |
None
|
memory_max
|
int | str | None
|
The memory per job. |
None
|
run_time_max
|
int | None
|
The run time per job. |
None
|
submit_options
|
dict[str, Any] | None
|
Extra variables for the submission script. |
None
|
retries
|
int
|
Number of retries for polling the cluster for results. |
30
|
bundle_size
|
int
|
How many evaluations go to a worker as one task, |
1
|
Returns:
| Type | Description |
|---|---|
WorkerPool
|
A pool backed by an HPC cluster. |
serial_pool
Create a pool that evaluates in-process, on the calling thread.
Identical to the free serial_pool function,
except that this pool is closed when the session closes, so runs cannot
keep using it afterwards. Prefer the free function for a pool that
should outlive any session.
Returns:
| Type | Description |
|---|---|
WorkerPool
|
A pool without an executor. |
shared_handlers
shared_handlers(
*handler: EventHandler,
threaded: EventHandler | Sequence[EventHandler] = (),
report: ReportCallback | None = None,
) -> SharedHandlers
Group result handlers that several runs share.
Pass the group to every run that should feed it, in handlers=. Each
handler then sees the results of all those runs, serialized across them,
which is what makes accumulating over concurrent runs safe. A run may
feed several groups, and mix them with handlers of its own.
A handler joins one group at a time, and a handler that was ever passed to a run as a local handler cannot join a group at all; decide per handler whether it is local or shared. See Running Optimizations for a walkthrough.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
handler
|
EventHandler
|
The result handlers to share, each run on the session's event-loop thread. |
()
|
threaded
|
EventHandler | Sequence[EventHandler]
|
Handlers (one, or a sequence) to run on a worker thread instead of the loop. This only helps handlers that spend real time in blocking, GIL-releasing I/O (files, databases, network); for in-memory work it gives no benefit under CPython's GIL. See Result Handlers. |
()
|
report
|
ReportCallback | None
|
An optional callback invoked with an |
None
|
Returns:
| Type | Description |
|---|---|
SharedHandlers
|
A |
ropt.simple.WorkerPool
The workers that run evaluations, and the batch IDs they share.
Created by a session factory such as
thread_pool. Every run using the same
pool draws its batch IDs from the same counter, so concurrent runs never
produce the same batch ID.
A pool lives until its session closes, which releases it. Release it earlier
with close, or by using it as a context
manager. See Running Optimizations for a
walkthrough.
A pool built by serial_pool has no executor and
no workers to release; it exists so that runs sharing one batch-ID sequence
can say so, whether or not they run in parallel.
__init__
__init__(
executor: Executor | None = None,
session: _Session | None = None,
bundle_size: int = 1,
) -> None
Initialize the pool.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
executor
|
Executor | None
|
The started executor, or |
None
|
session
|
_Session | None
|
The session that owns the pool, if it has one. |
None
|
bundle_size
|
int
|
Evaluations per worker task, |
1
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
closed
property
Whether the pool has been released.
Returns:
| Type | Description |
|---|---|
bool
|
|
executor
property
The executor that runs this pool's evaluations.
Returns:
| Type | Description |
|---|---|
Executor | None
|
The executor, or |
batch_ids
property
The counter every run on this pool draws its batch IDs from.
Returns:
| Type | Description |
|---|---|
BatchIdCounter
|
The batch ID counter. |
bundle_size
property
How many evaluations are sent to a worker as one task.
Returns:
| Type | Description |
|---|---|
int
|
The number of evaluations per task, |
close
Release the pool's workers without waiting for the session to close.
See Releasing a pool early for when this matters and the resulting lifecycle.
__enter__
ropt.simple.serial_pool
Create a pool that evaluates in-process, on the calling thread.
A serial pool has no workers: it carries only the batch-ID counter that the runs sharing it draw from. Use it to give concurrent runs one batch-ID sequence without running their evaluations in parallel, or as an explicit way to say that a run should evaluate in-process.
It needs no session, and needs no releasing.
Returns:
| Type | Description |
|---|---|
WorkerPool
|
A pool without an executor. |
Offloading work to a pool
ropt.simple.offload
offload(
work: Callable[[], _T] | Sequence[Callable[[], _T]],
*,
pool: WorkerPool | None = None,
) -> _T | tuple[_T, ...]
Offload one or more callables to a pool.
Pass a single zero-argument callable to run one call and return its result,
or a sequence of callables to run them concurrently (they may be entirely
different functions) and return a tuple of results in the order of work.
Bind arguments with functools.partial.
Without a pool, or with a serial_pool, the
callables run inline on the calling thread, one after another. Code that may
or may not have a pool to hand therefore needs no fallback: pass whatever it
has, including None. The callables must be picklable for a process or HPC
pool.
See Running Optimizations for a walkthrough.
A handler in a shared group runs on the pool's own event loop and cannot
wait on it; offloading from there raises a
WorkflowError. So does a pool that is
closed, or one carried into a worker process.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
work
|
Callable[[], _T] | Sequence[Callable[[], _T]]
|
A single zero-argument callable, or a sequence of them. |
required |
pool
|
WorkerPool | None
|
The pool to dispatch to, or |
None
|
Returns:
| Type | Description |
|---|---|
_T | tuple[_T, ...]
|
The single result, or a tuple of results in the order of |
Aggregating results across runs
Session.shared_handlers builds the
group; the group itself is a SharedHandlers object.
ropt.simple.SharedHandlers
A group of result handlers that several runs share.
Created by shared_handlers and
passed to the runs that should feed it. Each handler in the group sees the
results of every one of those runs, sequential or concurrent, and the
group's dispatcher serializes them, so a handler that accumulates across
runs needs no locking of its own.
A group lives until its session closes, which releases it and its handlers.
Release it earlier with close, or by
using it as a context manager. See
Running Optimizations for a walkthrough.
__init__
Initialize the group and start its dispatcher.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
entries
|
Sequence[tuple[EventHandler, bool]]
|
The handlers, each with whether to run it in a thread. |
required |
session
|
_Session
|
The session whose event loop the dispatcher runs on. |
required |
closed
property
Whether the group has been released.
Returns:
| Type | Description |
|---|---|
bool
|
|
attach_to
Forward the events of a run's compute step to this group's handlers.
A fresh forwarding handler is added per step (one handler cannot serve several steps), carrying only the event types the group's handlers want.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
step
|
ComputeStep[Any]
|
The compute step whose events feed the shared handlers. |
required |
close
Release the group's handlers without waiting for the session to close.
See Sharing a handler across concurrent runs for when this matters and the resulting lifecycle.
__enter__
Result objects
ropt.simple.EvaluateResult
dataclass
The values one variable vector produced.
Returned by evaluate, once per input vector by
evaluate_many, and handed to a report
callback once per function evaluation of a run. See
Running Optimizations for a walkthrough.
Attributes:
| Name | Type | Description |
|---|---|---|
variables |
NDArray[float64] | None
|
The variable vector these values belong to, or |
target_objective |
float | None
|
The weighted objective, or |
objectives |
NDArray[float64] | None
|
The individual objective values, shape |
constraints |
NDArray[float64] | None
|
The constraint values, shape |
results |
FunctionResults | None
|
The full low-level
|
ropt.simple.OptimizeResult
dataclass
Bases: EvaluateResult
The outcome of a single optimization run.
An optimization ends at one evaluation, the best one it found, so this is
that evaluation's EvaluateResult together
with the exit code of the run that reached it. See
Running Optimizations for a walkthrough.
Attributes:
| Name | Type | Description |
|---|---|---|
variables |
NDArray[float64] | None
|
The optimal variable vector, or |
target_objective |
float | None
|
The weighted objective at the optimum, or |
objectives |
NDArray[float64] | None
|
The individual objective values at the optimum, shape
|
constraints |
NDArray[float64] | None
|
The nonlinear constraint values at the optimum, or |
results |
FunctionResults | None
|
The full low-level
|
exit_code |
ExitCode
|
The exit code describing how the optimization terminated. |
Callback types
ropt.simple.EvaluationFunction
Bases: Protocol
The call signature for a high-level evaluation function.
__call__
Evaluate the objectives and constraints for a single variable vector.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
variables
|
NDArray[float64]
|
The 1-D variable vector for this evaluation. |
required |
context
|
EvaluationFunctionContext
|
The context identifying the evaluation. |
required |
Returns:
| Type | Description |
|---|---|
FunctionValue
|
|
FunctionValue
|
a scalar, or a flat sequence of objectives followed by constraints. |
ropt.simple.ReportCallback
module-attribute
Re-exported for convenience
These names are re-exported from ropt.simple (so simple-API code imports them
from one place), but they are the low-level classes and are documented with the
components:
- the evaluation context and result:
EvaluationFunctionContext,EvaluationFunctionResult; - the result handlers:
EventHandler,HistoryHandler,ResultsHandler,DataFrameHandler.