Put *Sync methods behind a flag in some future major version

No, I am not writing this from a mental hospital, please read till the bottom.

The purpose of this is to discourage using *Sync versions of the methods that could be async.

I propose to put all or at least a part of *Sync methods behind a runtime flag in some future major version of io.js and to split documentation, moving all *Sync methods to a separate page. This of course would be semver-major.

Atm, there are *Sync methods defined in zlib, fs, child_process, crypto modules and additionaly used in repl and module modules.
To allow usage inside io.js itself (i.e. for require), they could be moved to «private» methods (beginning with a _ sign).

*Sync methods suggest bad practices (see 2), but when someone with full understanding of the consequences needs them, they could be constructed from userspace. See https://github.com/abbr/deasync, it creates synchronous methods from async methods.
If deasync module is not good enough for this, this could be done when there is would be a good enough solution.
When a newcomer begins writing something using io.js, he or she goes to the documentation looking how to do something, sees *Sync methods without any warnings there and almost certanly begins with using them, because that's what he or she is used to. That's easier for a newcomer than spending a few minutes reading how he or she should actually do stuff. And don't blame the newcomer, it's the presense of *Sync methods in the documentation that suggests to him or her that it's an ok way to do things. This results in a big pile of bad code by the time when the person understands that it should be rewritten. And people don't like to rewrite code for no visible reson, leaving this code to be legacy (see 3).
When someone writes synchronous code (see 2) it limits how he or she can use async functions without rewriting most part of the logic, so he or she comes complaining about that there should be a *Sync version of everything out there (as the presense of *Sync versions in the core suggests it). See Is there a sync version encapsulating libmagic in node.js ? mscdex/mmmagic#32 and motivation behind https://github.com/abbr/deasync.

It's completely broken either way. Even for «simple one-time scripts». See zlib: memory leak with gunzipSync #1479 — a person was doing something like files.forEach( …zlib.gzipSync(…) …), in what I suppose was a simple script. What could possibly go wrong? Memory usage has gone completely bad in his script. And even manual calls to gc() do not help. Testcase:

'use strict';

var zlib = require('zlib');
var data = 'abcdefghijklmnopqrstuvwxyz';
var gzipped = zlib.gzipSync(data);

function call() {
  var contents = zlib.gunzipSync(gzipped);
}

for (var i = 0; i < 100000; i++) {
   call();
   if (i % 1000 === 0) {
       gc();gc();gc();gc();
       console.log(i + ' ' + JSON.stringify(process.memoryUsage()));
   }
}

People go to the doc, see *Sync versions (see 2), use them — then everyone are telling them that they are using io.js wrong just based on that fact: gripe: deprecating fs.exists/existsSync #1592 (comment).

if you're using existsSync inside your server I'd argue you're not using io.js correctly to begin with
One can argue again that using *Sync versions of methods is ok in scripts and that that's simplier, but:
1. See 4.
2. For one-time, throw-away scripts you can turn that flag on.
3. Promises-based code is as clean as *Sync-based code. Promises-based code with accurate error handling is much cleaner that *Sync-based code with accurate error handling. Using Promisify won't be needed once Promises go to the core (see Feature Request: Every async function returns Promise #11).

This will require all public modules that are using *Sync methods either to rewrite things using async methods or require/include something like https://github.com/abbr/deasync to be compatible.

This can be done separately for various core modules/methods. For example, zlib.*Sync are maybe the worst of them and it looks to me that they are not actively used in public modules.

I think synchronous methods do have their place, and shouldn't be moved behind a flag. You just shouldn't be using synchronous calls in hot code. require() itself is synchronous.

-1. Sync functions are very useful in various cases.

deasync is definitely the worst idea ever. 1. It doesn't actually make asynchronous calls blocking 2. It's inherently broken.

@cjihrig

@cjihrig I was not talking about require, I was talking about *Sync methods — synchoronous variants of those methods that are async by themselves.

@vkurchatkin Checked the code. Isn't it possible to create an actual lock instead of a loop?

By the way, zlib.*Sync seems to be almost unused in comparison with fs.*Sync. For example, in node_modules of the project I have currently opened, fs.*Sync is used in 636 files while zlib.*Sync is not used at all.

@ChALkeR

@ChALkeR I'm not sure how lock can help? To run async functions you need an event loop. It can't be the main loop, as unrelated requests would be processed as well. Something like this would work: nodejs/node-v0.x-archive#7323

+1

Only good use for sync functions is for one-line scripts, so adding a flag for it seems reasonable.

But usage of sync functions in any public modules should definitely be discouraged. Not only because of the event loop blocking, but because sync functions can't be replaced with async functions should the need for it arise.

For example, Jade uses readFileSync to read templates internally. What if I want to load them from a database instead? If it was an async readFile, I'd just monkey-patched require('fs'), but with readFileSync it just ain't happening.

Even synchronous require is a big pain in the ass if you want to require something from a remote http server or a zip archive instead of a filesystem. We've talked about that before.

I believe sync functions should be removed completely when ES7 async/await support lands in v8. Until then we should be gradually deprecating them.

@vkurchatkin

@vkurchatkin nodejs/node-v0.x-archive#7323 looks like a cleaner solution to me than the current deasync module.

Once again: I do not propose to remove *Sync just now, I am talking about masking them in some future major version, when Promises will be treated as first-class citizens etc. And this is not about all synchronous methods, it's about those methods that have a normal async version.

This might require a better way to build a sync version of an async function than the deasync module, if it is not good enough.

I'd rather have an async-only mode that warns/throws (an option for both). Creating CLI scripts that require runtime flags is a pain in the ass.

I'm -1 on putting Sync methods behind a flag. There are plenty of valid use cases for them.
If anything, maybe we could add --trace-blocking that prints a message whenever the event loop is blocked on i/o.

@jonathanong

@jonathanong That woldn't do anything. It would not lower *Sync usage in modules, it wouldn't discourage newcomers from using *Sync methods, it wouldn't motivate people to port their legacy code to async. It wouldn't solve any problem highlighted in the inital post.

You could just grep for [a-z]*Sync\( if you want to just find all usages of them, there is no need for that to be traced.

@piscisaureus

@piscisaureus They introduce problems because they seem to be an easy solution at a first glance but result in huge problems later (see 2, 3). Even for use cases that look like valid there might be problems (see 4). Such flag might be useful, but this is a different question from the one that I want to discuss here. A relevant thing would be to propose some other way of discouraging newcomers (and not-so-newcomers) from (mis-)using *Sync methods.

@piscisaureus, @vkurchatkin Please, list specific use cases for this to be constructive.

Relevant: #5 (comment)

To name some: scripting, bootstrapping, logging, using IO inside synchronous code. Sometimes blocking is fine. async-await + promises would be nice, but it doesn't solve all the problems.

Also, point by point:

1 -

when someone with full understanding of the consequences needs them, they could be constructed from userspace

No, they couldn't

2 - let's fix the docs

3 -

When someone writes synchronous code (see 2) it limits how he or she can use async functions without rewriting most part of the logic

This point as against removing Sync functions, if I understand it right

4 - this looks bad, actually

5 - See 2

6 -

Promises-based code is as clean as *Sync-based code.

No, it's not.

@vkurchatkin

@vkurchatkin

when someone with full understanding of the consequences needs them, they could be constructed from userspace

No, they couldn't

By this (added some time ago, after you shared deasync considerations):

If deasync module is not good enough for this, this could be done when there is would be a good enough solution.

I meant that this whole issue would have to wait until that problem (constructing sync methods from async methods) is solved. When and if that problem would be solved, the answer would be «Yes, they could».
Yes, fixing the docs is part of the issue, it's mentioned at the very top. Doing only that would not solve the issue completely, but will work as a partial solution.
No, it's a point for discouraging being contaminated by *Sync at the first place and against making it look like every async function should have a sync variant. No one is going to do that in all the user-contributed modules, they are generally async-only (for stuff that should be async), without sync variants.
Yes. zlib.*Sync is bad, broken and not actively used, it should be the first candidate for deprecating IMO. But the same should be true for all heavy memory-using *Sync code.
It's just an example for that you can not predict when sync code goes wild. Everything was made with garbage collection and async in mind.
See 2.
When Promises would be first-class citizens in io.js, it would be. And do not forget about error handling, if you take error handling into account, Promises-based code even now looks much cleaner that *Sync-based code. The latter would be a bunch of try-catches.

@trevnorris

@trevnorris Your example demonstrates my point on the common misperception and doesn't address the reasoning for this issue (performance degradation in high-concurrency scenarios).

Here's an example that does:

var http = require('http')
var foo = require('./packageWithBuriedSync').foo

// Note the ignorant all-to-common omission of Sync in the name
var bar = require('./packageWithBuriedSync').bar

function asyncHandler(req, res) {
  var i = 3
  foo(sendEnd)
  foo(sendEnd)
  foo(sendEnd)

  function sendEnd() {
    if (!--i) res.end()
  }
}

function syncHandler(req, res) {
  bar()
  bar()
  bar()
  res.end()

  // To eliminate performance questions about closure creation
  function sendEndNoop() {}
}

http.createServer(asyncHandler).listen(8000)

// packageWithBuriedSync.js
var child_process = require('child_process')
var exec = child_process.exec
var execSync = child_process.execSync

module.exports = {
  foo: function (callback) {
    exec('ls', callback)
  },
  bar: function () {
    return execSync('ls')
  }
}

Using ab -k -n 10000 -c 100 http://127.0.0.1:8000/

exec('ls') Time taken for tests:

1 calls: 26.429 seconds
2 calls: 48.974 seconds
3 calls: 77.823 seconds
4 calls: 99.995 seconds

execSync('ls') Time taken for tests:

1 calls: 53.748 seconds
2 calls: 103.953 seconds
3 calls: 154.733 seconds
4 calls: 218.769 seconds

Conclusion: For IO-bound cases[1], an ignorant buried *Sync consistently decreased performance by 100%.

@Qard Yes. See @rlidwka's comment though.

@benjamingr This is solved. The issue is the current stance is hostile to performance and an anti-pattern for new developers, thus the reasoning for proposing inclusion in core.

[1] Due to my MBA's flash drive, filesystem latency was extremely low and async performance had relative parity with *Sync with -c 100. An appropriate test should determine the point at which IO latency and concurrency begin to penalize performance.

My preference is to close this in favor of a more pragmatic solution like #1674.

Indeed. It doesn't really stop people from using sync functions where they
shouldn't it's just a way for people that care about that sort of thing to
get warnings and maybe a stack trace of where it came from, so they can
just avoid using those bad modules.

Like I said, not a fix, but it's something that can be done right now to
help optimise the performance of your code.
On May 11, 2015 1:01 PM, "Alex Kocharin" notifications@github.com wrote:

Writing such a module is trivial. However, forcing all the people who
misuse Sync to install it is not as easy. :(

—
Reply to this email directly or view it on GitHub
#1665 (comment).

Please let's not have an argument about who does or doesn't know more node..

@Fishrock123

@Fishrock123 you're right, for what it's worth I was not making fun of @CrabDude nor was I particularly critical of his level of expertise. I was just really amused by the tone of his comments. Still am, but going to keep it at a more professional tone from now on.

@CrabDude I'm sorry if I offended you, but you got to see where I'm coming from. I never once criticized you or your ability to write serverside JavaScript. The only thing my comments were about were the tone you replied to trev (and later me).

@ChALkeR for what it's worth it was never personal, I don't recall interacting with him before.

@CrabDude

@CrabDude You successfully demonstrated a case where using a sync alternative is slower. Those examples are plentiful. The point of my benchmark is to 1) demonstrate that async is not always faster and 2) show we shouldn't pretend we know what's best for the app developers. There are legitimate use cases for sync operations. Recall again that if you want to log information in an uncaughtException before the process goes down that it must be done synchronously.

execSync() was added after a lot of community members reached out and requested it. Even then it took a while to make it into core because of the scale of the change that was required (see fa4eb47 and e8df267). Removing it after all that would be to spit in the community's face.

@trevnorris

I don't think you two actually disagree about anything at this point.

@trevnorris demonstrated the Sync isn't a performance anti-pattern in non-concurrent situations.
@CrabDude demonstrated that Sync is a performance anti-pattern in concurrent situations.

Neither of you disagreed about those points. Both of you agree that it would be nice to track these performance issues in real code.

I think we should ofcus on #1674 at this point.

@trevnorris

I think that @trevnorris is correct here:

Let's not assume we're more intelligent than the developers writing their own apps and remove functionality because it's more theoretically correct.

I initially wanted some discussion on this and I am glad that happened and that #1674 was created.

But it misses the problem behind this issue that I specially highligthed in the first (actually second) sentance of the issue. And the discussion somewhy almost missed the documentation changes that I proposed, so I opened a separate issue for that: #1684.

I guess this issue could be closed now if there are no objections.

@trevnorris

@trevnorris

Let's make sure we're talking about the same things...

It's stupid for us to force this type of paradigm on the user.

Stupid to remove *Sync completely? Yes. I differ from the OP in that I'm advocating for requiring opt-in in all scenarios where it wouldn't be "stupid" (i.e., first tick and exit).

execSync() was added after a lot of community members reached out and requested it. [...] Removing it after all that would be to spit in the community's face.

Unnecessary inflammatory language aside, *Sync calls are a liability and an asynchronous IO-bound anti-pattern, what node.js is optimized for. It's a "spit in the community's face" to put anti-patterns that undermine the runtime in core in the first place. Appeasing a vocal minority to support non-concurrent, non-asynchronous scenarios is harmful to node.js primary use case. This is precisely what I mean when I point to your unfamiliarity with application developers' dilemma. What's "stupid" is the proliferation of liabilities and anti-patterns in core without considerations for education, allowances for documentation, and meaningful support for avoidance. I think we're now all on the same page here.

Let's not assume we're more intelligent than the developers writing their own apps and remove functionality because it's more theoretically correct.

First, facts are not theory. It is factually correct, demonstrated above and acknowledge by yourself that blocking calls are a major performance penalty for IO-bound tasks, which arguably account for the vast majority of node's use (numbers would be useful here). Second, the addition of *Sync to appease a vocal minority and the pernicious liabilities it creates for the ecosystem is naive. domains, of which you are intimately familiar, perfectly illustrates another core anti-pattern that developers consistently ignorantly misused resulting in its removal despite "the scale of the change that was required." In your opinion, was that also a "spit in the community's face"?

Let's just put the performance issue to bed. [...] You successfully demonstrated a case where using a sync alternative is slower. Those examples are plentiful.

Precisely. So we're in agreement that *Sync is anti-performant in high-concurrency scenarios, which again is node's core focus, and only faster in non-concurrent scenarios.

we shouldn't pretend we know what's best for the app developers.

At face value, this seems reasonable, yet in this context, it's misguided. Reductio ad absurdum:

Who are we to say developers prefer performance over features?
Who are we to say developers prefer non-blocking calls in a non-blocking runtime?
Who are we to say http should be in core when it's perfectly implementable in userland?
Who are we to say how modules or packages should be loaded?

You'll notice for all of the above, core does not preclude userland from implementing their own features, blocking calls, http alternatives, or module/package systems. It does however ship solutions to further enable the best non-blocking JavaScript IO runtime, which *Sync calls after startup are deleterious to. Did you equally object that execSync belonged in userland and if so, why do you now defend its inclusion with such forceful language?

There are legitimate use cases for sync operations.

Addressed at the beginning of the conversation.

#1674 and #1684 are excellent steps in the right directly, though I would prefer to see the logic that continues to support the proliferation of *Sync calls in core to die. It was not long ago when the node community loudly advocated that a primary reason for node's success over Twisted, EventMachine, etc... was the lack of legacy (blocking) IO calls. It seems we're prone to repeat the mistakes of the past in the name of pragmatism, un-opinionatedness or "community".

Got replaced by #1674 (#1707) for the opt-in warnings for using *Sync after the first tick and #1684 for the docs.

TC voted against *Sync «deprecation» and (if I got things correctly) for the opt-in warnings and documentation changes.

If there are any other ideas, those could go to separate issues.

Navigation Menu

Repository navigation

Put `*Sync` methods behind a flag in some future major version #1665

42 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Put *Sync methods behind a flag in some future major version #1665

Description

Activity

cjihrig commented on May 9, 2015

vkurchatkin commented on May 9, 2015

ChALkeR commented on May 9, 2015

vkurchatkin commented on May 9, 2015

rlidwka commented on May 9, 2015

ChALkeR commented on May 9, 2015

jonathanong commented on May 9, 2015

piscisaureus commented on May 9, 2015

ChALkeR commented on May 9, 2015

ChALkeR commented on May 9, 2015

ChALkeR commented on May 9, 2015

vkurchatkin commented on May 9, 2015

ChALkeR commented on May 9, 2015

42 remaining items

CrabDude commented on May 11, 2015

CrabDude commented on May 11, 2015

Qard commented on May 11, 2015

Fishrock123 commented on May 11, 2015

benjamingr commented on May 11, 2015

trevnorris commented on May 12, 2015

benjamingr commented on May 12, 2015

ChALkeR commented on May 12, 2015

CrabDude commented on May 12, 2015

ChALkeR commented on May 15, 2015

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Put `*Sync` methods behind a flag in some future major version #1665