Skip to content

Running External Programs

An evaluation function often runs a simulator or a solver as a separate program. Start it with run_program, and the program stops when the optimization is aborted, with any pool, together with every program it started. A program started with subprocess.run or os.system keeps running after an abort, and with a thread pool the evaluation waits for it.

The example script examples/programs.py runs a simulator from the objective function. The simulator is examples/simulator.py, a stand-in for a real one: it reads the variables from a JSON input file, prints its progress, and writes the objective to a JSON output file. When the input sets hang, it sleeps for an hour instead.

Running the simulator

The objective function writes the input file into a temporary directory, runs the simulator there, and reads the output file:

def objective(
    variables: NDArray[np.float64],
    context: EvaluationFunctionContext,
    *,
    hang_realization: int | None = None,
) -> float:
    """Run the simulator for one realization and read its objective.

    Args:
        variables:        The variable vector to evaluate.
        context:          Identifies the realization being evaluated.
        hang_realization: A realization whose simulator stops responding.

    Returns:
        The objective, or NaN if the simulator failed.
    """
    with tempfile.TemporaryDirectory() as workdir:
        cwd = Path(workdir)
        simulator_input = {
            "variables": variables.tolist(),
            "target": TARGETS[context.realization],
            "hang": context.realization == hang_realization,
        }
        (cwd / "input.json").write_text(json.dumps(simulator_input), encoding="utf-8")
        status = run_program(
            [sys.executable, SIMULATOR, "input.json", "output.json"],
            cwd=cwd,
            stdout="simulator.log",
            stderr=subprocess.STDOUT,
        )
        if status != 0:
            return np.nan
        output = json.loads((cwd / "output.json").read_text(encoding="utf-8"))
        return float(output["objective"])

The cwd argument sets the working directory of the simulator, so the relative file names refer to the temporary directory. Each evaluation uses a directory of its own, so evaluations that run at the same time do not overwrite each other's files. The output of the simulator, standard output and standard error together, goes to simulator.log in that directory.

run_program blocks until the program exits, and returns its exit status. A status of zero usually means success; a negative status -N means that the program was ended by signal N, as in subprocess. The objective returns NaN for a failed simulation, which marks the evaluation as failed; see Failures and Aborts.

Running the optimization

The optimization runs the four realizations with a thread pool, so up to four simulators run at the same time:

with session() as s:
    result = optimize(
        CONFIG, INITIAL_VALUES, objective, pool=s.thread_pool(workers=4)
    )

Aborting a hanging simulator

In the second run the simulator of realization 2 does not finish. A timer calls Session.abort after two seconds:

hanging = partial(objective, hang_realization=2)
exit_code = None
start = time.monotonic()
with session() as s:
    timer = threading.Timer(TIME_LIMIT, s.abort)
    timer.start()
    try:
        optimize(CONFIG, INITIAL_VALUES, hanging, pool=s.thread_pool(workers=4))
    except AbortedError as error:
        print(f"aborted after {time.monotonic() - start:.1f} s: {error}")
        exit_code = error.exit_code
    finally:
        timer.cancel()

The abort terminates the hanging simulator, and the optimization raises AbortedError about two seconds after it started. If the objective started the simulator with subprocess.run, the abort would not reach the simulator: the session would not close until the simulator exited, an hour later. The same applies to an abort by Ctrl-C, or by an exception elsewhere in the session.

Arguments

The command is a list of the program and its arguments, as in subprocess.run. A single string raises a TypeError: there is no shell, so a string cannot be split into arguments.

Argument Description
cwd The working directory of the program, by default that of your program.
env The environment of the program, by default that of your program.
stdout Where standard output goes.
stderr Where standard error goes.

stdout and stderr take one of:

  • None, the default: the output goes where the output of your program goes.
  • A path, as a string or a Path: the file is created, or emptied if it exists, and the output is written to it. A relative path is relative to cwd.
  • An open file, or a file descriptor.
  • subprocess.DEVNULL: the output is discarded.
  • subprocess.STDOUT, for stderr only: standard error goes where standard output goes.

subprocess.PIPE raises a ValueError: to process the output, write it to a file and read the file. Standard input is empty.

When the program is stopped

run_program never stops the program on its own account. It stops the program only when its caller is stopped, in one of three ways:

  • The caller is aborted, for example by Ctrl-C, by Session.abort, or by a failure elsewhere in the session. run_program then raises AbortedError; see Failures and Aborts for how that reaches the caller. Called when the caller is already being aborted, run_program raises at once and does not start the program.
  • An exception interrupts the call, for example KeyboardInterrupt in the main thread. The exception is re-raised.
  • The process that called run_program ends, for example when a process pool kills a worker that is still busy five seconds after an abort, or when an HPC job is cancelled.

In each case the program and every program it started receive SIGTERM, and those still running five seconds later receive SIGKILL.

The cleanup described above applies wherever you call run_program. What differs is how the call is interrupted:

  • Within a session, Session.abort reaches run_program through the session's with block or through work using one of its pools.
  • Outside a session, there is no session abort to react to. In the main thread, Ctrl-C can still interrupt run_program with KeyboardInterrupt and trigger cleanup.
  • On a thread started outside the ropt API, for example with threading.Thread, the session's abort is not inherited automatically. See copying the thread's context to make aborts reach it.
  • Inside an HPC job, run_program does not receive the submitting call's abort request. Instead, cancelling the job ends the process that called run_program, which triggers cleanup.

Programs that leave the process group

On POSIX systems, including Linux and macOS, run_program starts the external program in a separate process group. Cleanup terminates that group, including child programs that remain in it. A child that creates a new process group or session is not terminated by this cleanup. On Windows and other non-POSIX systems, only the external program itself is terminated; its child programs may keep running.