镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

Orchestration instance stuck in a partial RunAsync execution cycle after a request to Abort #812

Open

Description

@emilvel

The RunAsync of an orchestration calls a suborchestration, waits for the result, then acts upon it, and finally calls ContinueAsNew. In the implementation of the system, there are situation where we want to Abort the orchestration instance.
However, it seems that if that request to abort via TerminateInstanceAsync is made around (possibly immediately after) the sub orchestration completion, the orchestration continues the RunAsync exection all the way to the end where it calls ContinueAsNew...At that point in time, the runtime reports this exception:

System.InvalidOperationException: Multiple ExecutionCompletedEvent found, potential corruption in state storage
at DurableTask.Core.OrchestrationRuntimeState.SetMarkerEvents(HistoryEvent historyEvent) in //src/DurableTask.Core/OrchestrationRuntimeState.cs:line 256
at DurableTask.Core.TaskOrchestrationDispatcher.ProcessWorkflowCompletedTaskDecision(OrchestrationCompleteOrchestratorAction completeOrchestratorAction, OrchestrationRuntimeState runtimeState, Boolean includeDetails, Boolean& continuedAsNew) in /
/src/DurableTask.Core/TaskOrchestrationDispatcher.cs:line 816
at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemAsync(TaskOrchestrationWorkItem workItem) in //src/DurableTask.Core/TaskOrchestrationDispatcher.cs:line 399
at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemSessionAsync(TaskOrchestrationWorkItem workItem) in /
/src/DurableTask.Core/TaskOrchestrationDispatcher.cs:line 194
at DurableTask.Core.WorkItemDispatcher`1.ProcessWorkItemAsync(WorkItemDispatcherContext context, Object workItemObj) in /_/src/DurableTask.Core/WorkItemDispatcher.cs:line 459

followed by trace events about abandoning a couple of control messages as follows:
8c06fe74-7f26-44af-8c8e-74b747e2ea2c: Message [ExecutionTerminated] with ID e03ebdff-c5b9-4e7f-a43f-405cb180a7c2 has been dequeued 26 times and is now considered poison
8c06fe74-7f26-44af-8c8e-74b747e2ea2c: Message [SubOrchestrationInstanceCompleted#1] with ID 9e975302-87f8-4f46-8cb6-bce368f99778 has been dequeued 26 times and is now considered poison

Eventually, every 10 minutes (the max of visibility timeout set on messages) this repeats in a loop - our custom trace messages following the sub-orchestration completion are logged, followed by above exception, and the abandonment of the 2 control messages.

It appears that the ExecutionTerminated event and the completion of the RunAsync with ContinueAsNew end up in conflict causing the return of that message batch back to the control queue for retry?

Activity

  1. jviau commented on Oct 26, 2022

    @jviau
    Contributor

    Chris Gillum (@cgillum) I don't understand AzureStorage implementation that well, but it appears that it is an issue with resolving both an ExecutionTerminated and ContinueAsNew event in the same batch. Do you have any insight on this?

  2. jviau commented on Nov 15, 2022

    @jviau
    Contributor

    Found we have an existing issue that is being tracked in the durable functions repo: Azure/azure-functions-durable-extension#2264

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions