Repository navigation
node: unix sockets are slow in case of lots of small buffers #5095
Description
While running the node_redis benchmarks with unix sockets I see a huge performance decrease opposed to TCP while pipelining lots of small buffers (4byte || 4kb) to the stream. The performance will drop to 50% of TCP while benchmarking the small buffers and increase by about 50% by using 4mb buffers opposed to TCP.
Writing strings to the stream stays about the same or increases the performance a tiny bit.
I tried to work out where the difference came from but this is handled in C and I have no clue about C.
As a side note: node_redis mixes buffers and strings while writing buffers to the stream. Strictly only using buffers does not change the outcome though.
clients: 1, NodeJS: 5.5.0, Redis: 3.0.5, parser: hiredis, connected by: socket
SET 4B str, 1/1 min/max/avg: 0.01/ 2.77/ 0.02 2501ms total, 54347 ops/sec
SET 4B str, batch 50/1 min/max/avg: 0.09/ 3.41/ 0.11 2501ms total, 430688 ops/sec
SET 4B buf, 1/1 min/max/avg: 0.02/ 3.11/ 0.03 2501ms total, 37721 ops/sec
SET 4B buf, batch 50/1 min/max/avg: 0.49/ 4.43/ 0.59 2501ms total, 83826 ops/sec
SET 4KiB str, 1/1 min/max/avg: 0.02/ 3.52/ 0.02 2501ms total, 39837 ops/sec
SET 4KiB str, batch 50/1 min/max/avg: 0.25/ 1.96/ 0.29 2501ms total, 172891 ops/sec
SET 4KiB buf, 1/1 min/max/avg: 0.02/ 2.79/ 0.03 2501ms total, 35443 ops/sec
SET 4KiB buf, batch 50/1 min/max/avg: 0.53/ 6.28/ 0.67 2501ms total, 73870 ops/sec
SET 4MiB str, 1/1 min/max/avg: 3.54/ 20.85/ 4.41 2501ms total, 226 ops/sec
SET 4MiB str, batch 20/1 min/max/avg: 67.20/ 89.91/ 75.32 2561ms total, 266 ops/sec
SET 4MiB buf, 1/1 min/max/avg: 1.25/ 4.78/ 1.37 2501ms total, 724 ops/sec
SET 4MiB buf, batch 20/1 min/max/avg: 24.51/ 36.13/ 26.51 2520ms total, 754 ops/sec
clients: 1, NodeJS: 5.5.0, Redis: 3.0.5, parser: hiredis, connected by: tcp
SET 4B str, 1/1 min/max/avg: 0.02/ 5.21/ 0.03 2501ms total, 34392 ops/sec
SET 4B str, batch 50/1 min/max/avg: 0.10/ 3.37/ 0.11 2501ms total, 439584 ops/sec
SET 4B buf, 1/1 min/max/avg: 0.04/ 3.61/ 0.05 2501ms total, 19350 ops/sec
SET 4B buf, batch 50/1 min/max/avg: 0.25/ 6.18/ 0.29 2501ms total, 170292 ops/sec
SET 4KiB str, 1/1 min/max/avg: 0.03/ 5.49/ 0.03 2501ms total, 29333 ops/sec
SET 4KiB str, batch 50/1 min/max/avg: 0.27/ 6.38/ 0.32 2501ms total, 152879 ops/sec
SET 4KiB buf, 1/1 min/max/avg: 0.04/ 3.63/ 0.06 2501ms total, 17617 ops/sec
SET 4KiB buf, batch 50/1 min/max/avg: 0.31/ 3.92/ 0.36 2501ms total, 136026 ops/sec
SET 4MiB str, 1/1 min/max/avg: 4.01/ 20.96/ 5.29 2501ms total, 189 ops/sec
SET 4MiB str, batch 20/1 min/max/avg: 78.62/117.93/ 87.70 2544ms total, 228 ops/sec
SET 4MiB buf, 1/1 min/max/avg: 1.82/ 10.09/ 2.28 2502ms total, 436 ops/sec
SET 4MiB buf, batch 20/1 min/max/avg: 38.21/ 51.92/ 41.36 2523ms total, 484 ops/sec
Ping @trevnorris
Activity
mscdex commentedon Feb 5, 2016
What if you explicitly use
.cork()/.uncork()innode_redisto batch writes?BridgeAR commentedon Feb 5, 2016
@mscdex this is already in use but it does not change anything by removing it.
GeorgeBailey commentedon Feb 6, 2016
Thanks for bringing this up. I had a similar experience in Node 0.10.36 about 11 months ago, but opted to use STDIO for my project.
mcollina commentedon Apr 13, 2016
The way currently cork it's being currently done in node_redis is not the most efficient (as it is right now). Basically it calls cork/uncork multiple times during the same synchronous flow, effectively crossing the C++ barrier multiple times. @BridgeAR have you tried calling
uncorkinnextTickand see if there is any difference? That will likely batch up all your writes as they were a single one.BridgeAR commentedon Apr 13, 2016
Using nextTick does not change the performance in my tests. It stays in the tolerance range after each test run. There might be a increase that I can't measure well because of the noise but it won't be significant.
mcollina commentedon Apr 13, 2016
Run the same thing with http://npm.im/0x, and upload the results somewhere, so we can have a look.
It might even be OS specific (maybe some tuning in the kernel). Be careful of not running it on OS X, I had some surprises when running stuff on Linux for production.
src: enable writev for pipe handles on Unix
src: enable writev for pipe handles on Unix
src: enable writev for pipe handles on Unix
src: enable writev for pipe handles on Unix
src: enable writev for pipe handles on Unix
src: enable writev for pipe handles on Unix
src: enable writev for pipe handles on Unix
src: enable writev for pipe handles on Unix
src: enable writev for pipe handles on Unix