Repository navigation
Consider explicit string formatting module #43382
Description
Activity
What's the motivation for this? Why does util need to have color formatting and all of these other bells and whistles, taking an opinionated stance on these things?
Say what you want about Marak, but don't pin that on other packages that do ANSI colors, please. Your PR is just moving well established ecosystem packages into Node core for... no apparent reason. Not only is this unnecessarily bloating Node's util module, but you're kind of disregarding a lot of work over the last decade.
And to be completely honest, it's a bit insulting you copied almost entirely the Chalk API, seemingly without any concern for how the maintainers would feel about it.
Reacted by Chris PunchesIt’s not for no reason; most of the world uses CJS and will for the foreseeable future, and there’s no longer a maintained package available that works with it - and node often moves packages into core, as does the language - that’s the goal of a platform. A package is successful when it’s obsoleted by the platform.
Reacted by Nick Ribal, Matteo Collina, Kirill Groshkov, Carlos Fuentes, Sophie L, Bo Lingen and Corentin GirardReacted by Josh Junon and Chris PunchesI too welcome that funcionality like ANSI colors is moving into core. Every dependency is a potential security risk waiting to happen. Besides that, reducing the number of dependencies is also good for performance and stability.
@Qix- what would be a better way to interact assuming there is merit in this functionality in core? Core cares a bunch about the community, the ecosystem and maintainers like you. As someone who maintained several libraries whose functionality was subsumed by core I can totally see how this isn't clear-cut and it's nice/flattering but also frustrating.
The best case scenario for Node.js would be if you helped maintain and consult on these APIs in core (if Node decides this belongs in core, which can also be through vendoring a blessed module under the node namespace) which would ideally enable you to keep benefiting from them on one hand (for example through sponsorhsips) but would also simplify things for users.
I do see the tension here and I acknowledge core has a lot of power over the ecosystem, the ecosystem is really important to Node's success and the last thing most team members I talked to about this want to do is hurt the authors. (I will not comment about Marak since our policy is not to discuss specific people and moderation in public)
I am not sure what the best scenario for you is and what would work well for you - so please let me know. Knowing @BridgeAR I don't think there was any intention to insult you (but Ruben can weigh in), the fact he liked the chalk API so much he picked it (or something similar) is a sign of respect (similarly to how you maintaining a library that allows debugging using Node's internal API is not seen as bad in any way).
On the other hand, I would ask core members (and specifically @BridgeAR who is working on the PR) not to land this until good-faith discussion with @Qix- has exhausted itself and we've both listened to what they have to say and reached consensus amongst ourselves. I know this is common practice in Node anyway but I just want to explicitly point it out.
Reacted by Feng Yu, Tierney Cyren, Kirill Groshkov and Ruy AdornoHere's my take. If Node.js is moving popular modules out of the ecosystem into core for the sake of API compatibility...
- What place does NPM serve in all of this? Is it that the NPM/package ecosystem no longer in good graces with the Node core team? This change will effectively render a decade of hard work and dedication to the ecosystem as moot, as this could have been added whenever libuv's tty module was added (it's been around since the 0.12 days or maybe even sooner, if memory serves). If it weren't the case, then why not work with package maintainers to create some sort of contingencies about what to do if another
colors-like situation happens again? - How many top-10 packages are going to be stolen (yes, strong word - the PR copies chalk's API and functionality explicitly, as the PR author has stated directly)? Why isn't it common practice to ping the maintainers for which this will affect? You offered to allow us to maintain the standard libs, but that feels more like a rebuttal to my post above rather than an upfront nicety.
- If the issue is CJS vs ESM, then why does Node have an LTS release schedule to begin with? Because Chalk's approach has been to follow the Node LTS cycle as guidance for things that would otherwise break users during a major version migration. ESM was one of those, if memory serves (@sindresorhus please correct me if I'm wrong). Once Node supported it, and the last remaining LTS version that didn't support it went out of the LTS cycle, that's when Sindre moved all packages to ESM and bumped their majors. If you need to stay on a version of Chalk that requires CJS, it means you're also on a version of Node that is no longer supported as well. We did our due diligence to keep our packages clean of bitrot and now we're being punished for it.
- If this "was discussed at the collaborator summit", why weren't the actual people writing the actual code involved? Our emails are listed in commits. Our issue trackers are open. We're clearly both extremely active on GitHub. Node.js committee members know who I am on Discord, if memory serves. There's no reason you couldn't have reached out to discuss what this whole thing would look like. You just didn't bother. Not very collaborative, if you ask me.
The original post's template seems to ask "what problem with this solve?" More contextually, "what problem is solved by invalidating Chalk from the ecosystem as a long-standing and well trusted package, and moving it into core with the exact same API and functionality? Perhaps this seems snarky, but this is precisely what the question is getting at.
Node.js has multiple formatting related APIs such as util.inspect(), util.format(), util.formatWithOptions(), util.stripVTControlCharacters(str) and multiple util.inspect sub APIs such as colors, custom, styles and defaultOptions.
We recently added util.parseArgs() to the util module and it becomes pretty big overall, while not all really being connected with each other.
I don't see how this answers the question at all. What is the point of pulling these modules into core and inflating the standard library? This question remains unanswered.
Admittedly, I had no idea that
util.stripVTControlCharacters()existed. I already immediately knew the answer, but I wanted to know which regex or parser Node uses for this.node/lib/internal/util/inspect.js
Lines 224 to 232 in 027c288
Yep. That regex pattern that we spent collective days refining, testing, and hardening. Vendored into node, without notifying the maintainers (again, @sindresorhus correct me if I'm wrong - I certainly wasn't notified).
This regex has had ReDos vulnerabilities reported for it before. How does Node intend to maintain this regex? What was the point of pulling this into core? What is the security benefit of increasing vulnerability surface area?
I'm not here to have the license battle because I'll lose that. That's what the license is there for. I'm happy we as a community have defaulted to something like the MIT. I have no problem with anyone vendoring or privately using my code in such cases.
This still feels quite slimy, because the very platform we invested our time in has now absorbed our hard work into core and completely invalidated any reason for people to refer to our package and to receive updates and to report to us any issues they find to get support and fixes quickly and directly. The Node core team has unilaterally agreed, then, that the open source model for their own ecosystem does not work. I would say this is an oversight in thought processes within the committee, but there's already precedent here with
ansi-regex.Though the fact there is precedent here does not mean it's alright. It feels like the Node committee is trying to EEE us - pretty deliberately. I had to make sure that Node wasn't also acquired somehow by Microsoft when writing this, and it turns out that the OpenJS foundation is owned by a bunch of corporations, including Microsoft. The merger of the Node.js foundation into OpenJS happened in 2019. Node 16.11 - the version where
ansi-regexwas "vendored" - happened in 2021, after that. I'll leave thing there, as presented.What is the feature you are proposing to solve the problem?
It is also considered to add new APIs such as a nicer abstraction for colors. To do so, I would like to create a new module that only contains string formatting APIs including the above. These APIs should always be kept supported from the util module as reference to the new module.
This makes little sense; I have questions.
- How will this be maintained?
- How will new contributions to
chalkbe incorporated? - Are you going to "vendor"
chalkdirectly, like you didansi-regex? Since the PR doesn't seem to do that. No credit either, aside from the PR comment itself. - How many times has
ansi-regexbeen explicitly updated to reflect changes in the original package? - Is my package
colornext? Should I be letting it rot even more so as to not get it stolen by the Node.js committee when I upgrade it to ESM, too? - Is there rivalry between TC-39 and the OpenJS committee? Why have ESM modules if they'll prompt Node to bypass their own ecosystem with arguments like ". . .the established packages are not reliable[;] One has gone ESM-only. . ."?
- When does
debug- a package currently blocked by Node.js's lacking module loaders for ESM - go next? - Has any effort been made to assess the impact on sponsorships/open collectives/etc to the afflicted maintainers?
- Are these decisions being made with corporate interest? Not that I expect any honest answer given the laundry list of Big Tech listed as founders (no mention of Mozilla by the way, one of the few browser vendors that actually tries to compete these days).
I don't want to pull this card very often but this is one of the few places I will. I've been a diligent, happy proponent of Node.js since 0.10 was released. I've been a maintainer of several top-10 packages for almost, if not over, 10 years at this point. I've brought Node.js to many companies, supported and contributed to countless repositories on GitHub for over a decade now, I've even contributed directly to libuv and Node.js.
If this gets merged, I will consider myself directly and forcibly pushed out of the Node.js community by the Node and OpenJS committees. I'm not sure how much you actually care about this, but for me, as having attributed a large percentage of my career's time and success to this ecosystem and language, it will be the end of an era.
Maybe I'm overreacting. But this feels wrong.
Reacted by Darío Blanco Iturriaga, Vadim Demedes, Benjamin Gruenbaum, Kipras Melnikovas, Arka Pratim Chaudhuri and Corentin GirardReacted by Sophie LReacted by Bo Lingen- What place does NPM serve in all of this? Is it that the NPM/package ecosystem no longer in good graces with the Node core team? This change will effectively render a decade of hard work and dedication to the ecosystem as moot, as this could have been added whenever libuv's tty module was added (it's been around since the 0.12 days or maybe even sooner, if memory serves). If it weren't the case, then why not work with package maintainers to create some sort of contingencies about what to do if another
I intentionally opened the PR as draft and I reached out to @sindresorhus to let them know about it when I did that. Using the same API as chalk is indeed a form of great respect for the work that has been done as being proven to be intuitive and good.
I think it would be great to get the chance to discuss the general that has now come up due to me opening the draft for adding further coloring functionality to core in more depth. There was definitely no intention to do any harm when it was brought up at the collaborator summit to add more coloring functionality to core. @Qix- I agree that things could have gone better. I hope we can get back to good terms as I personally hugely appreciate your great work in the ecosystem!
Reacted by Benjamin Gruenbaum, Kirill Groshkov, Moshe Atlow, Josh Junon and Ruy AdornoI am going to close the Draft for now until we are able to resolve this topic.
Reacted by Benjamin GruenbaumI would like to add a few fundamental points.
Should Node.js produce output that has colors?
My answer to this question is a strong yes, as everybody likes colors in their terminal.
The best way to achieve this is to either:- copy the code of popular modules that had this functionality battle-tested and embed it into core
- add those modules to
deps/, use them internally but do not expose them to everybody, forcing people to download them again from npm even if they have them on disk already? - add those modules to
deps/, use them internally and expose this functionality externally - implement everything from scratch
I would really like to know @Qix- opinion on this topic. I think solving this would shape Node.js future.
ESM migration
Most of the real-world team I have worked in the last two-three years have no plan to migrate to ESM as there is no business benefit for them. Very few managers would allocate budget to the migration: it's significantly simpler to switch to a dependency that still support CJS.
The transition to ESM is taking significantly more time than what others have expected. ESM is still lacking a few fundamental features that makes CJS great. Very few companies or individuals are willing to invest the time and effort needed to fill the gap.
The download numbers of chalk and other libraries tells the same story: most users are not upgrading to ESM, as most people are staying on an old version which is a great solution short term. However, I don't expect to migrate any of my code to ESM-only anytime soon, therefore I had to seek an alternative. The chalk team decided that it was not interesting for them to support my use case: this is their decision to make. This decision to go ESM-only created significant issues in the community and a high number of people has asked me what to do.
What can Node.js core do? This functionality is a really good candidate for inclusion in core, because almost everybody wants it, include core itself.
Core vs Ecosystem
I don't see a conflict with bundling some feature in core vs have it in the ecosystem. Existing, proven technologies are extremely hard to displace. As an example, jQuery is still in used by the vast majority of websites.
The ecosystem can move at a speed of innovation that can be hardly matched by a project that has a LTS cycle of 3 years. I don't think chalk or any other library would be impacted at all.
Governance of Node.js
Node.js is governed by its collaborators, not by "Big Tech". The best way to have a say in these decision is to... contribute. People are willing to contribute new features to core: should the collaborators block them or facilitate them?
If everything is blocked, then the project will stagnate because very few will be willing to contribute back regularly.Reacted by Jordan Harband, Benjamin Gruenbaum, Kirill Groshkov, Moshe Atlow, Sophie L and Arnaud BenhamdineThanks for engaging @Qix- , Node.js is a distributed project and I only represent one person in it - a maintainer but not in a leadership position like Matteo or Ruben above. I've still been involved long enough to know maintainers typically have a say here.
Both these people above (Matteo and Ruben) are ecosystem maintainers themselves. I'm an ecosystem maintainer whose work was integrated into core before - I believe we do have some perspective. I also strongly believe your involvement is a net-positive for all parties here. You raise some good points .
What place does NPM serve in all of this? Is it that the NPM/package ecosystem no longer in good graces with the Node core team?
I've had a module I maintain (bluebird) integrated into core in the past. A lot of the ideas we've had there like long stack traces, better debuggability as well as performance (V8 uses the bluebird benchmark themselves) were upstreamed into core. I think a big part of the difference in my experience from yours is that I was a part of most of it and so was Petka (the original author).
How many top-10 packages are going to be stolen (yes, strong word - the PR copies chalk's API and functionality explicitly, as the PR author has stated directly)? Why isn't it common practice to ping the maintainers for which this will affect? You offered to allow us to maintain the standard libs, but that feels more like a rebuttal to my post above rather than an upfront nicety.
I think the project's communication here did leave a lot to be desired. It is now clear (following Ruben's message) there was an intention to ping maintainers for feedback before merging it but it looks like that wasn't communicated well initially.
I think this is regretful. I think there was no intention to harm chalk and as you can (hopefully) see from member responses after your initial comment: people understand we've made a mistake here and we'd like to improve in good faith.
I can't apologize on behalf of the whole project - but I will as myself: I'm sorry about the project taking external work from the ecosystem without attempting to contact maintainers, even though this was a draft PR the fact maintainers were to be contacted should have been made explicit and done sooner and you shouldn't have found out about this from outside the project. We've done better in the past and I will personally ping maintainers when I see this in the future.
Moreover, I am sorry the project to my knowledge didn't consider the implications on the maintainers of that work before. This was a mistake and didn't give enough space to the support the project has received from the ecosystem and how fundamental it was and is to its success. I vouch to personally call it into attention when I see these issues in the future.
That said, I also don't consider this theft. I do believe the fact the functionality provided is now (possibly) considered important, common and fundamental enough to be in core is a good thing. I believe we have to think more about how we celebrate the hard work put into these pieces of code.
If this "was discussed at the collaborator summit", why weren't the actual people writing the actual code involved?
I know this doesn't make up for past miscommunications but let me personally invite you to attend the next collaborator summits (in person or virtually). If you'd like to participate in or host a meeting about colors in terminals that could be a great win.
To answer some of the questions I can:
Is there rivalry between TC-39 and the OpenJS committee? Why have ESM modules if they'll prompt Node to bypass their own ecosystem with arguments like ". . .the established packages are not reliable[;] One has gone ESM-only. . ."?
Generally no. TC39 and Node.js (and it's technical steering committee) generally work well together and collaborate in good faith (at least in the last couple of years). There are some disagreements and different goals but the project has also helped standardize a bunch of things in the language (like Myles contributing a large chunk of top-level await).
When does debug - a package currently blocked by Node.js's lacking module loaders for ESM - go next?
Would it be OK to open a separate issue (or ping me at an existing one?) to discuss this and what node would have to do?
Has any effort been made to assess the impact on sponsorships/open collectives/etc to the afflicted maintainers?
I don't think that enough was done and I consider this (as mentioned above) a mistake on the project. I know some work was done on helping maintainers as part of pkgjs but I don't think this particular area was considered.
If it's OK with you I'll open a separate issue in the Node.js TSC repo and ping the technical steering committee?
Are these decisions being made with corporate interest? Not that I expect any honest answer given the laundry list of Big Tech listed as founders (no mention of Mozilla by the way, one of the few browser vendors that actually tries to compete these days).
Not that I am aware of. While I work for one of the member companies (Microsoft) my employer is actually not involved with my contributions to the project which I do on my spare time (I actually have a clause in my contract that allows this) and my work in Microsoft is on entirely different areas (not Node.js related at all).
I can't read peoples' minds so I can't speak for other members (+I didn't attend the last summit unfortunately) but generally corporate interests are mostly in line of "We run clouds like Azure or GCP so it's good that everyone has the same Node.js and it's maintained in an open way and not by a single company". The OpenJS foundation started as the Node.js foundation and since Mozilla is not Node cloud-vendor I suspect they didn't join - but I certainly hope they do!
I don't want to pull this card very often but this is one of the few places I will. I've been a diligent, happy proponent of Node.js since 0.10 was released. I've been a maintainer of several top-10 packages for almost, if not over, 10 years at this point. I've brought Node.js to many companies, supported and contributed to countless repositories on GitHub for over a decade now, I've even contributed directly to libuv and Node.js.
It is truly regretful in my opinion you feel you have to note this to have your voice heard.
If this gets merged, I will consider myself directly and forcibly pushed out of the Node.js community by the Node and OpenJS committees. I'm not sure how much you actually care about this, but for me, as having attributed a large percentage of my career's time and success to this ecosystem and language, it will be the end of an era.
I personally care about this and I feel many others in the project do too.
Maybe I'm overreacting. But this feels wrong.
Your feedback has been very valuable and your perspective brings a lot to the table. I hope it's clear we're trying to listen and engage in good faith.
Reacted by Moshe Atlow, yakov116 and Corentin GirardHow many top-10 packages are going to be stolen
Absolutely zero. What was discussed at the collaborator summit was driving a process for bundling and distributing certain open source ecosystem modules with the node binary. Modules that, as it turns out, are already distributed with the node binary thanks to other things that are bundled there. These would be either modules node itself uses or are considered core dependencies for most node.js apps. We discussed the possibility of how we would do that, clearly identifying that we would need to define a process for deciding what, when, and how. The issue I opened here #43413 starts to discuss one proposal for that process.
Not that I expect any honest answer given the laundry list of Big Tech listed as founders
Let's be absolutely clear: there are no corporate overlords in Node.js, and the suggestion in this comment is insulting, quite frankly. The Foundation does not make the decisions here, and in fact, has absolutely no standing in this conversation at all. What happens in this project is driven by individual contributors doing what they think is right for the project. Reasonable people can disagree and we have processes for that. Just because one individual contributor opens a PR proposing something, it doesn't mean "the project" or the Foundation or nameless Big Tech overlords are doing something. It only means there is a proposal from one individual on the table for discussion.
There is lots of reasonable discussion and debate to be had around whether and how to vendor bundle ecosystem modules in core. Lots of constructive feedback to have. But let's not throw around accusations of theft, and force, etc when it's literally just a conversation among individuals who disagree on how code should be distributed.
Reacted by Bo LingenReacted by Matteo Collina, Jordan Harband, Kirill Groshkov, Carlos Fuentes, Sophie L and Ruy AdornoShould Node.js produce output that has colors?
I feel like this is a tangent point that was introduced while discussing a much more serious issue, so, sorry if I'm butting in, and doubly sorry if I've misunderstood, but please do not add color codes to log generation facilities.
This works fine for development and toy environments but in professional or robust environments where logs are shipped to an aggregator, pumped through a content agnostic event stream, SIEM, etc, all this does is add work for the logging infrastructure and it comes across when we see it the same way that using a gmail account for a customer-facing client contact point does.
The producing system should be making logs that are agnostic of how those logs will be rendered and focus purely on the content of those logs (vs. presentation), because they're not guaranteed to be the same system (which system produces logs vs. which system is rendering those same logs). By adding functionality that makes it easier to colorize log output, it would be creating incentive for developers who don't know better to build potentially (unlikely) beautiful bad practices. And yes, colorizing log output is a bad practice.
This is coming from the system domain not software development domain (i.e. I am not a nodejs developer, I am a person that builds around the apps that would be created that use this functionality).
If I have misunderstood what's being discussed please delete this, I recognize the seriousness of the broader discussion taking place. Just, again, please don't add color content to your logs so as to deter normalization of the pain it creates.
12 remaining items
It would be beneficial if the ESM migration plan was discussed in a separate issue and we bring back the original topic: adding an explicit string formatting module to core.
@benjamingr could you open a separate issue to discuss the impact of core decisions on individual/module sponsorships?
Reacted by Beth Griggs, Benjamin Gruenbaum and Kirill GroshkovThis definitely is still an important use case to meet in core, I think, despite the complexity.
Reacted by Colin Ihrig, Matteo Collina, Ruy Adorno and Hemanth HMbump
What is the problem this feature will solve?
Node.js has multiple formatting related APIs such as
util.inspect(),util.format(),util.formatWithOptions(),util.stripVTControlCharacters(str)and multipleutil.inspectsub APIs such ascolors,custom,stylesanddefaultOptions.We recently added
util.parseArgs()to the util module and it becomes pretty big overall, while not all really being connected with each other.What is the feature you are proposing to solve the problem?
It is also considered to add new APIs such as a nicer abstraction for colors. To do so, I would like to create a new module that only contains string formatting APIs including the above. These APIs should always be kept supported from the util module as reference to the new module.
@nodejs/util @nodejs/tooling @nodejs/tsc any opinions?