[inureyes]

Summary

FIDO security keys (sk-ssh-ed25519@openssh.com, sk-ecdsa-sha2-nistp256@openssh.com) now work for host keys, host certificates, user keys and user certificates, matching OpenSSH 10.3p1 built without its internal provider. The built-in internal provider is #331; it plugs into the provider seam added here, and until then SecurityKeyProvider internal prints a diagnostic and authentication continues with the next identity.

• SecurityKeyProvider is a runtime option: library path, internal, none, or $VAR (default $SSH_SK_PROVIDER, printed raw by -G like the reference). An unset variable or none skips sk-* identities at debug level, as ssh does. id_ed25519_sk and id_ecdsa_sk are default identities, and an sk-* file is checked against the provider before any passphrase prompt.
• The sk-api (0x000a0000) library is loaded in process with dlopen(RTLD_NOW | RTLD_LOCAL), validated (version, sk_enroll/sk_sign/sk_load_resident_keys) and cached per canonical path for the process lifetime. OpenSSH isolates the provider in ssh-sk-helper to keep it out of its pledged client; bssh has no such sandbox, so a helper would add a private protocol without changing what runs as the user. The trade-off, documented in the compatibility status: a started provider call cannot be cancelled, and the provider can read bssh's memory.
• Provider calls are serialized by a FIFO async lock taken before spawn_blocking, so queued hosts hold no blocking thread and a host that hits ConnectTimeout while queued never reaches the provider. Every response pointer and length is checked (Ed25519 requires a 64-byte signature), an empty application is rejected, the response is zeroed and freed as sshsk_free_sign_response does, and PINs are zeroed and never logged.
• A sk_sign failure after the server accepted the key prints security key signing failed for <path>: ... and authentication continues with the next identity on the same connection, as in OpenSSH: the signer returns the request unsigned, russh 0.64.1 sends nothing for it, and bssh drops the exchange that would otherwise wait for a reply.
• -Q key-plain, key, key-cert, key-sig and the default HostKeyAlgorithms and PubkeyAcceptedAlgorithms match the reference ssh -Q and ssh -G except for the webauthn-sk-ecdsa-sha2-nistp256@openssh.com pair, which ssh-key cannot verify and bssh therefore does not advertise.
• sk-* host keys and certificates verify through ssh-key's sk verifiers (SHA256(application) || flags || counter || SHA256(message)); like OpenSSH's client, no flag policy applies to host keys.
• New, also closing a gap that predates this PR: the server's host key or certificate must match the negotiated host-key algorithm (OpenSSH kex_verify_host_key, SSH_ERR_KEY_TYPE_MISMATCH), so a HostKeyAlgorithms restriction also limits which trusted key type a server may present.

Test plan

• Full cargo fmt --check, cargo clippy -- -D warnings, cargo clippy --lib --tests -- -D warnings, cargo test --lib and cargo test --tests -- --skip integration_test pass locally on 20d38c4 (the head before the docs-only commit).
• OpenSSH regress locally: hostkey-agent (both sk-* certificate phases), keytype, limit-keytype, agent, connect, proxy-connect, sshcfgparse, hostkey-rotate, rekey pass.
• CI regress: Ubuntu 69/70 and macOS 66/67 (was 68/70 and 65/67); the remaining failure is agent-restrict (feat(agent): bind destination-constrained agent keys with session-bind@openssh.com #327).
• New tests cover $VAR expansion, the internal diagnostic, load and validation failures including lazy symbols, the library cache, signature encoding, the decline-and-continue path against a scripted server, signer queueing and cancellation, the algorithm lists, the certificate-only host-key list, and host-key type matching; each was checked to fail when its behavior is reverted.
• musl: cargo check --target aarch64-unknown-linux-musl -p libloading passes, and a failed dlopen on a static build says so. A full musl build of bssh needs a musl cross compiler for aws-lc-sys, which this host lacks, so the next release workflow run is the first full build check of the musl targets.

Follow-up candidates: bssh-server verification of sk-* user keys, bssh-keygen enrollment and resident keys, webauthn-sk-* verification, and host-key checks on re-key (russh does not pass the re-key host key to the handler).

Closes #328

[inureyes]

Implementation Review Summary

Intent

FIDO sk-* host keys, host certificates and user keys through an external SecurityKeyProvider, matching OpenSSH 10.3p1 built without the internal provider (#328).

Findings Addressed

• A sk_sign failure after PK_OK ended the connection; OpenSSH moves on to the next identity. russh 0.64.1 does block on Msg::Signed after a signer Err, but sends nothing when the signer returns the request unchanged, so the sk signer now declines that way and authenticate_declinable drops the stuck handle-side exchange. SecurityKeySignFailed is retryable (HIGH, d4a4811)
• Provider calls could run concurrently from several spawn_blocking threads (multi-host auth); sk-api promises no thread safety and OpenSSH never does this. Calls are now serialized process-wide inside the blocking task (HIGH, d4a4811)
• id_ed25519_sk and id_ecdsa_sk were printed by -G but never tried at runtime; they are now default identities, and an sk file is checked against the provider from its public half before any passphrase prompt (HIGH, d4a4811)
• tests/ssh_compat_output_test.rs still expected securitykeyprovider to be unsupported; it now uses VisualHostKey and asserts SecurityKeyProvider produces no diagnostic (HIGH, 5618c12)

Remaining Items

• Touch and PIN prompts name the key by SSH name (sk-ssh-ed25519@openssh.com) where OpenSSH prints ED25519-SK; the touch notice is suppressed in BatchMode (LOW)
• A PIN containing NUL is dropped without zeroization; SecurityKeyIdentity Debug prints the key handle (LOW)
• -G casignaturealgorithms and hostbasedacceptedalgorithms still print the raw defaults including webauthn-sk-* (LOW, pre-existing)

Verification

• All stated requirements implemented
• No placeholder/mock code remaining
• Integrated into project code flow
• Project conventions followed
• Existing modules reused where applicable
• No unintended structural changes
• Tests pass (narrow lib groups, ssh_compat_output_test, ssh_config_dump_test, bin query, clippy, fmt; regress not run in this worktree)

[inureyes]

Security and Performance Review

Fixed

• HIGH (7ca198d): PROVIDER_CALL_LOCK was a std mutex taken inside each spawn_blocking task. Every host queued behind a touch pinned a blocking thread, so a large --parallel exhausted tokio's blocking pool and stalled tokio::fs and DNS for all connections; a host that hit ConnectTimeout while queued still called sk_sign later and asked for a touch nobody was waiting for. The lock is now a FIFO tokio mutex taken before spawn_blocking, with the owned guard moved into the task. Two new tests fail with the old lock.

Remaining

• MEDIUM, pre-existing: the received host key type is never compared with the negotiated host-key algorithm. russh 0.64.1 verifies against whatever key the server sends (client/kex.rs 297-341), and check_server_key does not compare it with names.key, which kex_done already records. OpenSSH rejects this case (kex.c:1197, SSH_ERR_KEY_TYPE_MISMATCH). As a result a HostKeyAlgorithms restriction, including one that excludes sk-*, does not limit which trusted key type the server can use.
• MEDIUM: Library::new uses RTLD_LAZY, while sshsk_open uses RTLD_NOW (ssh-sk.c:139). A provider with an unresolved symbol loads fine, then aborts the whole bssh process at the first sk_sign.
• MEDIUM, accepted design: an in-flight sk_sign cannot be cancelled. A hung provider holds the lock until it returns, so no other sk signature in that process can proceed. Queued waiters now give up at ConnectTimeout. The provider also runs with access to all of bssh's in-memory credentials, where OpenSSH confines it to ssh-sk-helper.
• LOW: the provider is dlopened and dlclosed per sk identity per connection. Caching one handle per path would avoid repeated initializers and dlclose hazards such as TLS destructors left pointing into unmapped code.
• LOW: preflight_security_key_file reads and parses every identity file again on every connection, including software keys. It never prompts or decrypts.
• LOW: encode_signature does not enforce the 64-byte Ed25519 sig_r length that sshsk_ed25519_sig checks, and an empty application is not rejected the way sshsk_sign rejects it. free_response at external.rs:229 has no SAFETY comment.

Verified

FFI struct layout and prototypes match sk-api.h. The response is freed exactly once on every path. The fn pointer cannot outlive its Library, because the Arc moves into the blocking task. The provider path comes only from ssh_config, -o, or the environment, which matches OpenSSH. The decline path cannot mis-sign or reorder a request: russh sends nothing when the length is unchanged, auth_method is taken, and the signer never returns Err. sk host signatures cover the application hash, flags, and counter. Users without sk keys are offered the same keys as before.