Repository navigation
Conversation
* Acks for completed client RPCs were banked on the homa_peer, which is keyed by destination address only, and piggybacked on any later packet to that server. Since f30a0ef packets go out on a homa_route, keyed by source address, mark, bound device, etc., so a client can reach one server from several source addresses. * The server looks an ack up by the source address of the packet that carries it (homa_rpc_find_server). An ack for an RPC sent from address B that rides out on a packet from address A matches nothing and is dropped. The server then needs a NEED_ACK/ACK exchange two ticks later to free the RPC, and that ACK can drain more misrouted acks from the shared bank. * With 3 source addresses on one client (two CloudLab xl170 nodes, 25 Gbps, cp_node, 100-byte RPCs) there was a NEED_ACK for 2 RPCs in 3, and 284 Kops at 711 us median latency vs 472 Kops at 31 us with one address. Nothing is lost for the application. * Fix it on the client: RPC ids are only unique per client host, so the server needs the address; naming another address in the ack would add 16 bytes to every DATA header. * Move num_acks and acks to homa_route, guarded by route->lock (which existed but was never taken). The lock moves to the end of the struct, on its own cache line together with the bank. * Keep the trylock wrapper and miss metrics the peer lock had, now homa_route_lock() and route_ack_lock_misses/_miss_cycles: all sockets with the same routing parameters share one route, so this lock is as contended as the peer lock was. * Remove peer->lock; rename homa_peer_add_ack/get_acks to homa_route_add_ack/get_acks; update util/metrics.py and protocol.md. * Acks banked on a route that homa_route_validate() replaces or the GC evicts are dropped and recovered by NEED_ACK, as for an evicted peer today. * Update the ack unit tests and add one that checks that two routes to one peer keep separate banks.
|
Interesting points. Having a peer ID distinct from address sounds like a good idea, but how would this be generated? |
|
Thanks John. To clarify, this PR is a bug fix. Before f30a0ef, a The peer ID (or more complete multi-homing support) is a longer-term |
|
Yep, I understood about the bug; it's just taken me a while to get to it. I will be pushing your fix shortly. Thanks for pointing out the problem and providing a fix. |
Acks are banked on the homa_peer, which is keyed by the server's address,
but since f30a0ef they go out on packets sent over a homa_route, which
also depends on the source address, mark, bound device etc. The server
matches an ack with homa_rpc_find_server(hsk, saddr, id), where saddr is
the source of the packet carrying it. So when a client reaches one server
from several source addresses, an ack that leaves from the wrong address
matches nothing, and the server RPC waits for a NEED_ACK.
This moves the ack bank to homa_route, under route->lock (initialized but
never taken until now), so an ack always leaves from the address its RPC
used. The server side and the wire format don't change.
Two xl170 nodes, 25 Gbps, cp_node with 100-byte RPCs, the client using
3 source addresses:
With one source address the difference is within 1 % either way, about
the spread between module loads. Scripts and raw logs:
https://github.com/breakertt/HomaModule/tree/fix-artifact/bind-ack-with-route
A question this raised: should Homa identify a client by something other
than its address?
The server keys an RPC by (client address, id). That works because ids
are unique per client host and, until f30a0ef, all packets to a peer went
out on the peer's single dst, so even a multi-homed host sent to a given
peer from one source address at a time. Now one host can use several
addresses for the same server, and this patch only makes the client go
along with the server's view that each address is a separate client. The
same assumption breaks in other places:
replaces an obsolete dst with a fresh lookup, and for a socket without
a bound address that lookup can pick a different source address
(failover, an address being removed, a routing change). The server then
can't match the RPC's later packets to it (more DATA starts a second
RPC), and whatever it sends for the original RPC still goes to the old
address. This could already happen before f30a0ef, when
homa_get_dst() re-resolved a stale peer dst with the parameters of
whichever socket was sending.
migration, multipath) needs the same thing.
make many clients share one address. The server's key doesn't include
the client port, and every client starts its ids at 2, so two clients
behind one address that use the same id at the same time are treated
as one RPC.
A peer ID would fix these at the root: RPCs are keyed by (peer ID, id),
and an address only says where to send.
Let me know what you think.