Repository navigation
Cancellation of Socket.ConnectAsync intermitently hangs #42198
Description
When Socket.ConnectAsync is cancelled it might lead to a deadlock.
The cancellation code for MultipleConnectAsync tries to acquire a lock which is held by the connectAsync continuation (triggered by DoDnsCallback). The continuation then tries to dispose of the cancellation registration which will wait for all the callbacks if they've already been triggered.
This is easily reproducible with our HTTP stress suite (for HTTP 1.1 runs longer than 30 mins).
The biggest impact of this issue is that it prevents us from getting reasonable results from HTTP 1.1 stress pipeline on master. The pipeline gets cancelled on timeout and no results are available (AzDO empty logs).
Activity
[-]Socket.CancelAsync intermitently hangs[/-][+]Socket.CancelConnectAsync intermitently hangs[/+]geoffkizer commentedon Sep 14, 2020
In AttemptConnection, when ConnectAsync completes synchronously, we are calling InternalConnectCallback directly. This is bad since the caller of AttemptConnection should be holding the lock here.
We should instead return to the caller and do the call to InternalConnectCallback outside the lock.
Additionally, we should add asserts wherever we take the lock, but especially in the callbacks like InternalConnectCallback, that the lock is not already held by this thread -- basically,
Debug.Assert(!Monitor.IsEntered(_lockObject)).And if a method is assuming the caller is holding the lock, like AttemptConnection is, we should add an assert that the lock is actually held. This is valuable not just to detect bugs, but also to document the expectation of the function.
[-]Socket.CancelConnectAsync intermitently hangs[/-][+]Cancelltion of Socket.ConnectAsync intermitently hangs[/+][-]Cancelltion of Socket.ConnectAsync intermitently hangs[/-][+]Cancellation of Socket.ConnectAsync intermitently hangs[/+]HTTP stress test improvements (#42313)
riberk commentedon Oct 19, 2020
Is there a workaround? I have the same problem and i don`t know how to solve it from outside of the Socket class
ManickaP commentedon Nov 6, 2020
The only workaround is to avoid cancellation during socket connect. I don't know your exact problem so I cannot help much further at this point.