镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

Handle solids hanging during multi-process - #18

Open
alexvalentine94 wants to merge 5 commits into
ukaea:devfrom
alexvalentine94:feature/solid_timeout
Open

alexvalentine94 wants to merge 5 commits into
ukaea:devfrom
alexvalentine94:feature/solid_timeout

Conversation

@alexvalentine94

Copy link
Copy Markdown
Collaborator

Per-solid timeout and log file fix

Summary

A single solid stuck inside OpenCASCADE (an endless loop in IntPatch_WLine::ComputeVertexParameters) could hang a whole multiprocess decomposition, because ProcessPoolExecutor cannot stop one worker. Separately, spawned workers wiped analysis/logging.log.

Changes

  • Per-solid time limit: new split_solid_timeout decomposition parameter (default 300 s, 0 = no limit). A solid that runs past it has its worker stopped and replaced, and is saved to failed_to_convert with the reason. The remaining solids carry on.
  • New pool: run_with_timeout in decomposemultiprocess.py replaces ProcessPoolExecutor in Decomposer.run. It also replaces workers that die (e.g. a segfault), and returns results in input order.
  • Shared failure handling: saving a failed solid is moved into save_failed_solid, which both paths use. Behaviour is unchanged.
  • Log file fix: a spawned worker imports mcio before parent_process() is set, with sys.argv copied from the main process, so each new worker truncated the log (causing lost lines and NUL bytes). __init__.py now also checks sys.orig_argv for --multiprocessing-fork.
  • Fixed missing spaces in the "conversion failed" message.

Not changed

The decomposition logic itself (splitting, booleans, surfaces, void, output writers). The time limit doesn't apply to single-core or debug runs.

Testing

  • New tests/unittests/test_decomposemultiprocess.py (9 tests): a stuck task, all workers stuck, a crashed worker, an exception in the task, result order, no limit, and a log-file regression test that fails on the old code.
  • Full suite: 361 passed, 16 skipped.
  • On a ~3,800 solid model that used to hang, the run now finishes in about 5 minutes, with 4 solids stopped by the limit and a complete log.

Notes

  • Branched from feature/ruff_fix, so merge that first.
  • The run still stops with "conversion failed" if any solid fails, as before.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant