Repository navigation
Operator stops reconciling permanently after its watch connections drop while idle #936
Description
Affected Stackable version
26.7
Affected Trino version
481
Current and expected behavior
The trino, hive and airflow operators flip from recoverable watch-stream errors to a permanent failed to start watching object: client error (Connect) loop and never reconcile again until the pod is restarted. While wedged the operator looks healthy to Kubernetes: the process is running, its conversion webhook still answers.
Expected
A controller whose watch connection is dropped re-establishes it and continues reconciling, as it does for the first several drops.
Actual
ERROR kube_client::client::builder: failed with error client error (Connect)
WARN stackable_operator::logging::controller: Queued reconcile resulted in an error
controller.name="trinocluster.trino.stackable.tech"
error=failed to start watching object: ServiceError: client error (Connect)
error.sources=[ServiceError: client error (Connect), client error (Connect), deadline has elapsed]
The problems seems to be upstream in kube-rs: https://github.com/kube-rs/kube/blob/main/kube-runtime/src/controller/mod.rs#L1704
Possible explanation for this issue: All watch triggers get merged into a single stream that then gets joint backoff - so if an operator watches e.g. 7 resources, the backoff fires instantly 6 times, which quickly leads to high sleep durations that ultimately go beyond the timeout.
While kube-rs has a long timeout configured (>200 secs) on the observed cluster some kind of proxy for the apiserver seems to terminate idle watches after 60 seconds, which is reliably triggered for the mentioned operators.
Possibly related: kube-rs/kube#1915
As mentioned in the beginning, this potentially affects all our operators
Possible solution
Possible Workarounds:
Smaller, constant backoff
use kube::runtime::utils::Backoff;
struct ConstantBackoff(Duration);
impl Iterator for ConstantBackoff {
type Item = Duration;
fn next(&mut self) -> Option<Duration> { Some(self.0) }
}
impl Backoff for ConstantBackoff {
fn reset(&mut self) {}
}
Controller::new(api, watcher::Config::default())
.trigger_backoff(ConstantBackoff(Duration::from_secs(2)))
Reduce timeout on watcher side:
Maybe as an optional parameter or something.
const WATCH_TIMEOUT_SECS: u32 = 40;
fn watch_config() -> watcher::Config {
watcher::Config::default().timeout(WATCH_TIMEOUT_SECS)
}
Additional context
No response
Environment
No response
Would you like to work on fixing this bug?
None
Activity
lfrancke commentedon Aug 28, 2026
Whatever we might fix upstream or in our code: Should something like this not also trigger a condition that marks our operator as unhealthy?
sweb commentedon Aug 28, 2026
A regular probe would not detect this - you would have to check the connectivity between controller and api server.
Techassi commentedon Sep 3, 2026
A few thoughts on this: