Kernel drops subshells messages if a subshell no longer exists. #1557

Description

A shell message addressed to a subshell that no longer exists is dropped. The kernel logs Invalid message with a KeyError and sends nothing back (no reply nor error). A client that waits for the message to complete waits forever.

This is easy to hit because delete_subshell_request travels on the control channel while the messages it invalidates travel on the shell channel. A client that sends a shell message on a subshell and then deletes that subshell has no way to order the two, so the delete often wins.

The routing happens in Kernel.shell_channel_thread_main, where the except Exception around the send swallows the message:

The get_shell_channel_to_subshell_socket call ends at a cache lookup, which raises KeyError once the subshell is gone:

JupyterLab hits this with comms. Comm messages go to a per comm target subshell, and closing the last comm on a target sends comm_close on that subshell and then deletes it. The comm_close is lost about 70% of the time, and the front end hangs on a promise that never settles. See jupyterlab/jupyterlab#19722.

Reproduce

The script below opens a subshell, sends comm_close on it, then deletes the subshell. It reports how many times the kernel never published the matching idle status. On ipykernel 7.3.0 with Python 3.12.14 it lost 12 and 13 of 20 rounds on two runs. I used Claude Opus 5 to generate and run this reporducer.

import queue, sys, time
from jupyter_client.manager import KernelManager


def send(kc, channel, msg_type, content, subshell_id=None):
    msg = kc.session.msg(msg_type, content)
    if subshell_id is not None:
        msg["header"]["subshell_id"] = subshell_id
    kc.session.send(channel, msg)
    return msg["header"]["msg_id"]


def wait_for(get_msg, msg_id, predicate, timeout):
    deadline = time.time() + timeout
    while time.time() < deadline:
        try:
            msg = get_msg(timeout=max(0.05, deadline - time.time()))
        except queue.Empty:
            return None
        if msg["parent_header"].get("msg_id") == msg_id and predicate(msg):
            return msg
    return None


def is_idle(msg):
    return msg["msg_type"] == "status" and msg["content"]["execution_state"] == "idle"


km = KernelManager(kernel_name="python3")
km.start_kernel()
kc = km.client()
kc.start_channels()
kc.wait_for_ready(timeout=60)

lost = 0
for i in range(20):
    mid = send(kc, kc.control_channel.socket, "create_subshell_request", {})
    reply = wait_for(kc.get_control_msg, mid, lambda m: True, 10)
    subshell_id = reply["content"]["subshell_id"]

    mid = send(kc, kc.shell_channel.socket, "comm_open",
               {"comm_id": f"comm-{i}", "target_name": "test", "data": {}}, subshell_id)
    assert wait_for(kc.get_iopub_msg, mid, is_idle, 5) is not None

    close_id = send(kc, kc.shell_channel.socket, "comm_close",
                    {"comm_id": f"comm-{i}", "data": {}}, subshell_id)
    send(kc, kc.control_channel.socket, "delete_subshell_request",
         {"subshell_id": subshell_id})

    answered = wait_for(kc.get_iopub_msg, close_id, is_idle, 5) is not None
    lost += 0 if answered else 1
    print(f"round {i}: {'ok' if answered else 'lost'}", flush=True)

print(f"{lost}/20 messages never reported idle")
kc.stop_channels()
km.shutdown_kernel(now=True)

The kernel log shows one of these per lost message:

[IPKernelApp] ERROR | Invalid message
Traceback (most recent call last):
  File ".../ipykernel/kernelbase.py", line 591, in shell_channel_thread_main
    socket = subshell_manager.get_shell_channel_to_subshell_socket(subshell_id)
  File ".../ipykernel/subshell_manager.py", line 107, in get_shell_channel_to_subshell_socket
    return self.get_shell_channel_to_subshell_pair(subshell_id).from_socket
  File ".../ipykernel/subshell_manager.py", line 92, in get_shell_channel_to_subshell_pair
    return self._cache[subshell_id].shell_channel_to_subshell
KeyError: '...'

Expected behavior

The client should be able to tell that the message will not be answered. Two options, either one closes the hole:

1. Answer the message. Publish busy and idle for it and send an error reply with status: "error" naming the missing subshell, in the same way as an unknown message type.
2. Drain before deleting. Make delete_subshell_request wait for the messages already queued for that subshell, so a message that reached the kernel before the delete is still processed.

Option 2 alone still leaves a window for a message that arrives after the delete, so option 1 is needed in any case.

Related: _process_control_request catches the KeyError from _delete_subshell for an unknown subshell and returns an error reply, so deletes report the problem and shell messages do not.

Context

• ipykernel: 7.3.0
• jupyter_client: 8.9.1
• Python: 3.12.14
• Operating system: Linux

Related:

• Example Tests are flaky because of subshell issues in ipykernel v7 jupyterlab/jupyterlab#19722
• ipykernel 7.0.0 release with subshells #1438