Gitteh needs a primary maintainer!

I had an epiphany this week: I've been selfish when it comes to Gitteh.

I've acknowledged several times in issues / emails that I don't really have enough time for Gitteh. But then, like most developers, I'm presented with a problem of some kind and want to solve it. Then suddenly Gitteh seems interesting again, and I feel all inspired and start making some promises to do things that I generally don't.

Today, I think node-gitteh holds the most promise out of most git libraries for Node. Here's why:

It's under the official libgit2 umbrella and has compatible licencing.
The latest version is stable and has a decent suite of tests.
It has a decent amount of interest (260 stars, 37 forks, 10 watchers).
It's had a lot of blood, sweat, and tears poured into it :)

I think the design choices made for gitteh are sound, there's been some quibbling regarding some implementation choices (such as CS), but I think these can be addressed in the short-term easily enough.

SO. What I need is for someone to step up to the plate. I need someone who wants to become the primary maintainer for gitteh.

Please mention your interest here, and we'll hash it out. The person I pick to be the primary maintainer will ideally be someone who's shown a healthy amount of interest in gitteh over the past few months and has some Github projects demonstrating an understanding of Node, and C++ (read: best candidate would be someone who's written a Node native module before). Once I find the right person, I will ensure they are given contributor access to this repository.

@FrozenCow

@FrozenCow @lushzero @jmendeth @ben @iamwilhelm - Who's it gonna be fellas? :)

@ben

Does it need to be just one?

[...] has some Github projects demonstrating an understanding of Node, and C++ [...]

I have some overall experience in it, I've mostly written Robotskirt (Node.JS wrapper for the now-frozen Sundown library), as you can see in the contributors page; from that I derived the V8U set of microutilities for writing V8 addons. And some other small modules, like parport.js (high-level access to parallell ports from Node).

But I wouldn't like to hold that responsibility alone.

PS: @ben is already a maintainer. :)

I have experience with C++ and I have experience with Node/JS. However I don't have any experience writing C++ addons for Node (apart from the PR I did here), so I don't think I'm fit for the job.

Does it need to be just one?

I agree, 2 or more would be best.

However I don't have any experience writing C++ addons for Node (apart from the PR I did here)

you'll get experience from gitteh! :D

@samcday

@samcday That's good to hear. I'd hate to see it languish. I have a vested interest in making sure gitteh is up to date and up to par, as I use it in cubehero.com. However, I haven't written a node native module before, I'm not against spending the time to learn, however. I think as one of the maintainer, I'd probably start off doing things like making sure documentation is up to date, and making sure PRs and issues don't just sit there.

I think as one of the maintainer, I'd probably start off doing things like making sure documentation is up to date, and making sure PRs and issues don't just sit there.

That's the spirit! It's clear that we need several maintainers. 😃

I'm afraid I have to bow out for the maintainer role. I'm a bit overcommitted just now, and more projects are coming down the pipeline. I'll keep watching this, and feel free to summon me for libgit2 questions.

I'm afraid I have to bow out for the maintainer role. I'm a bit overcommitted just now, and more projects are coming down the pipeline. I'll keep watching this, and feel free to summon me for libgit2 questions.

I understand. But don't worry, we'll still @ summon you. ;)

I'm pretty much out on gitteh at this point and working on sgit. To me a lot of the fundamental technological decisions made in gitteh were sound at the time they were made but not now. Libgit2 is very different now so they don't make sense anymore, i.e. C++ async, thread handling and thread locking which is 70% of gitteh's code.

@lushzero

@lushzero Fair enough, I'm not sure I'd want someone working on gitteh if they make statements like that without backing it up with proper reasoned arguments ;)

@jmendeth @FrozenCow @iamwilhelm Thanks for indicating interest. I wrote up the initial issue pretty fast, and didn't mean for it to sound like it precludes more than one active maintainer. What I meant by it is I need someone (or someones, hehe) to be involved in gitteh enough so that it keeps moving along, as it's not really something I realistically have time for. That said, I plan on remaining involved, and may be able to contribute some more features over the coming weeks / months. We'll see.

@iamwilhelm I didn't actually realise you were using gitteh in anger somewhere like that. That's pretty cool! Gitteh is actually something I wrote for a project I was working on a while ago, but I've since scrapped that project and I think that's one of the biggest reasons why I haven't had much time for gitteh.

@ben Understandable!

Okay so the guys who are interested, sounds like we need to divvy up some responsibilities. Here's a quick rough cut of what might work:

@jmendeth + @FrozenCow work on getting bindings up to date with latest libgit2.
@iamwilhelm Get new version of documentation up, implement some higher level methods in gitteh userland.

Whatcha think?

Yeah, that sounds great to me.

But I think that, before trying to start working, we should set a simpler
build system (i.e. remove coffeescript), so that we (and future contributors)
can focus better on it.

There's also things we can simplify on gitteh's code, now that LibUV is
standard in every Node.

@lushzero

Since @lushzero was working on a fork, out of interest I thought I'd look around whether there were other git libraries for node based on libgit2. I found https://github.com/nodegit/nodegit/tree/wip . Does anyone know anything about that project? They seem to do things very similar to what node-gitteh does and still active. I'm not sure how far along they are, but they do have treebuilder stuff in there, which was something I wanted to put in gitteh too.

With sgit, there are 3 projects doing the exact same thing with slight in their API and the way it works. I'm a bit torn at what to do here. Focus our efforts on one project? Keep developing multiple projects side-by-side? Collaborate a bit? I'm still interested in the new methods @lushzero is suggesting, because the current way to implement methods in gitteh is quite hard as a beginner. With the code being in a better state, it would be easier for people to contribute.

That being said (back on-topic), I don't mind being a maintainer, though I do not know how long it will be of interest for me. I'm working on a project for myself that uses gitteh and for that I'm adding features that are needed, but I don't know how long that project will last. I'll likely lose interest/become less active once that project dies. So, short-term: yes. long-term: I'm not sure.

@lushzero

@lushzero @FrozenCow @samcday

My thoughts:

@lushzero has interesting views (which I don't understand yet; I want to talk about them on a dedicate place) which can help improve Gitteh as well as SGit.
I understand that @lushzero didn't back his arguments on his comment; it's very natural since this is not the adequate place to talk about implementation issues, so we'd better open a dedicate issue to talk about it.
Nodegit was a quick & low-level binding. Lately it has improved a lot, but Gitteh was has been a serious high-level approach since the beginning. So we shouldn't focus much on nodegit.
@FrozenCow About your interest matter, I have to say: this is what happens to everyone. 😄 Even @samcday admits he once lost interest (and now Gitteh seems interesting again). I also wanted to do a rewrite and actually started it (and developed it for some time), and it has some functionality, but now it sits there, abandoned.

So, that "interest" problem is normal on everyone, what (I think) it's important here is that we inform others about it before leaving. Other than that, I don't see any problem in being a maintainer.

PS. Sorry for my ugly English.

I thought I might introduce myself as I'm now making active contributions to nodegit. I'm the author of the wip branch https://github.com/nodegit/nodegit/tree/wip

The wip branch is a rewrite of nodegit from the ground up. It uses codegen to generate the node bindings automatically, from a JSON description of the API. The wip branch is, I believe, already more comprehensive than both nodegit/master and node-gitteh. IMO, codegen is a better approach than writing bindings by hand because it requires less manual work to add support for a new function -- and libgit2's API is huge! Other advantages include: a bug fixed in one place is a bug fixed in all places, etc.

This wip version isn't exactly quick and low-level, although it follows this simple[istic] design philosophy: it aims to implement something like a 1:1 mapping between the node API and the C API, with a layer added in pure Javascript that is more "convenient". For example, there is a wrapper around the treebuilder to make it simple to make deep (nested) changes to trees https://github.com/nodegit/nodegit/blob/wip/example/new-commit.js and there are event emitters for walking history https://github.com/nodegit/nodegit/blob/wip/example/walk-history.js.

All of that "convenience" is implemented in Javascript so people can make nodegit as friendly as possible without having to write C++ -- and at the same time, since nearly 100% of the libgit2 functionality is available to them, people are no longer blocked waiting for someone else to implement some function (like clone or treebuilder) in C++.

Even the 1:1 "raw" API is not as bad as it sounds. It's still object-oriented in style, and is asynchronous exactly and only where libgit2 (potentially) does blocking IO:

  git.Repo.open(path.resolve(__dirname, '../.git'), function(error, repo) {
    if (error) throw error;

    repo.getCommit('59b20b8d5c6ff8d09518454d4dd8b7b30f095ab5', function(error, commit) {
      if (error) throw error;

      commit.getEntry('README.md', function(error, entry) {
        if (error) throw error;

        entry.getBlob(function(error, blob) {
          if (error) throw error;

          console.log(entry.name(), entry.sha(), blob.size());
        });
      });
    });
  });

Note that in the above, code, nothing is part of the "convience" API -- everything is 1:1 and automatically generated. Preliminary documentation of this raw API is here: http://www.nodegit.org/nodegit/ . Documentation of the convenience layer is forthcoming (but check the examples at https://github.com/nodegit/nodegit/tree/wip/example )

I'd like to explore the idea of whether we can merge these two projects. That is, whether we can just take the codegen rewrite of nodegit, and once it is stable and well documented, call it libgit2/WHATEVER and deprecate the previous two projects. One project has the benefit of: one implementation, more maintainers, fewer bugs, better documentation.

So that's my hope.

@nkallen

Hi @nkallen,

that explains many things (I remember nodegit to be a lot hackish and have a raw API, and the results with codegen really impressed me). However, going from manual to auto is a huge change, and has to be studied accordingly (it's also a big phylosophy change).

You see, Gitteh now wants to remodellate itself and needs some people who care of it (reason of this thread). So I think the best approach would be to let @samcday decide, then wait a bit so we can simplify and get things to stability again, and then we would all debate about the fusion you're proposing ---I'm serious about getting a single project with all our efforts combined.

Conclusion: we need some time to get Gitteh itself to normality, and then we can talk about that change. :)

My thoughts.

@nkallen

@nkallen @lushzero

So I coded a little program that will asynchronously run that
git log emulation test on the last wip nodegit. It'll use global.gc
and memwatch to take snapshot diffs and display them beautifully
on your console. You'll be able to see what type is exactly leaking.
It has a variable number of iterations.

You should totally try it out: https://github.com/proxy/gist.github.com/jmendeth/6434819

In my test, the repository is only opened one time. Everything
else is repeated. I think I'll change it to constantly open and close
the repository, but I think this is a good place to start.

My results

I have tried to put console.logs around to make sure they are looping,
also tried putting timeouts so that the GC can clean everything, but
the results are always the same:

Logically, the number of nodes allocated before is always the same.
At 50 iterations (default), the balance is negative (there's less memory
after the test than before). The details don't list any nodegit types, which
means all the classes Repo, RevWalk, History, EventEmitter are
correctly deallocated. Only generic types like Code, Array, Object, appear.
Exactly the same at 100 iterations. "Exactly the same" means the output
is exactly. the. same. in both cases. Same numbers, same types.

At 500 iterations, we see some nodegit-specific types leaking:

 Commit: +6 (which is the number of commits in the repo)
 EventEmitter: +1
 Oid: +1
 Repo: +1 (yeah, that makes sense)
 RevWalk: +1

At 5000, the results are exactly the same as in 500.

You can see the results here: https://github.com/proxy/gist.github.com/jmendeth/6440106

@jmendeth

@jmendeth -- This script is awesome though, thanks! I'd love it if we can continue this discussion over here nodegit/nodegit#84 . As @lushzero rightly mentioned, at this level of technical detail we are derailing the broader discussion.

Some quick observations:

memwatch detects leaks in js code only; it won't detect missing frees in the C++ code. Valgrind is still the only way to do that AFAIK.
You can tell in advance there is no leak because the leftover objects aren't O(N) where N is the number of iterations. The 6 objects must be phantoms:
For the leftover 6 Commit objects, this is simply due to the fact that v8 does partial garbage-collections. https://developers.google.com/v8/design?csw=1#garb_coll:

processes only part of the object heap in most garbage collection cycles. This minimizes the impact of stopping the application.

I don't know how to force a full gc, but if you incrementally call gc() as in https://github.com/proxy/gist.github.com/nkallen/6442106#file-gistfile1-js-L4 , even for 5000 runs there are no nodegit objects left over

it won't detect missing frees in the C++ code. Valgrind is still the only way to do that AFAIK.

Yes. :(

this is simply due to the fact that v8 does partial garbage-collections.

hum, I thought that global.gc() would force a full garbage-collection;
if that's not true, maybe it's a better idea to use memwatch's own GC:

To aid in speedier debugging, memwatch provides a gc() method to force V8 to do a full GC and heap compaction.

@lushzero

@lushzero, before we can use sgit, it first needs to be fully asynchronous. If it is not, weird behavior will crop up and comparing it to other libraries wouldn't be fair (whether we're comparing memory, cpu, complexity does not matter). Or am I still missing something? Is sgit already async? I was under the impression that v8 objects couldn't be used outside of v8's main thread, so I presumed it was not.

Anyway, I've dabbled a bit with nodegit a bit. It seems to work pretty well, although I have found some minor issues (documentation slightly wrong and one usecase not correctly implemented), but all in all I'm pretty impressed. I like that it's just a 1-to-1 mapping of libgit2, which means you can just lookup libgit2's documentation if you want to have details about a particular function. It might also be a good idea to include those mappings into the documentation where possible.

I haven't used all functions that I used for gitteh yet, but for now I'm very positive about the current work. Compared to gitteh it already has more up-to-date documentation, it is easier to find the function you need and it was easy to find the function that was causing a bug and fix it. This is mostly due to the 1-to-1 mapping and the fact that there is documentation (although still minimal and wrong in places).

Or am I still missing something? Is sgit already async? I was under the impression that v8 objects couldn't be used outside of v8's main thread, so I presumed it was not.

No, no, you're right. The functions don't return until all the LibGit2 operations (which are blocking) have been done.
LibGit2 is synchronous.

There are many potentially dangerous things done in SGit. Just to list an example:

const git_oid *oid;
{...}
oid = git_commit_oid(wcommit);
{...}
git_oid *w_oid = (git_oid*) oid;

The first lines save the OID of a commit, which is a pointer to a const git_oid. The last line casts that pointer, stripping out the const, which is nasty. Then it modifies the contents of oid through the new pointer.

LibGit2 gives you const pointers because you are expected to not modify them (the memory they point to)! oid contains a pointer to the area where LibGit2 holds the commit's info. That area is expected to always contain the commit's OID. If you change that area, LibGit2 will think the commit suddenly has another OID which will break caching, produce crashes if you call certain other funtions on that commit and other nasty things!

Another bad idea is that git_log outputs this:

{
  'OID of the tree of first commit': {
    'author': '...',
    'date': '...',
    'message': '...',
    '...': '...'
  },
  'OID of the tree of second commit': {
    'author': '...',
    'date': '...',
    'message': '...',
    '...': '...'
  },
  '...': '...'
}

This doesn't allow to iterate over the commits in order; since keys in a hash don't have a concrete order, the original order in which commits were walked is lost.

It's not useful to put the OIDs of the tree as hash keys, because what uniquely identifies a commit object is the commit's OID! I want to list commits, not trees. Also, since two commits can share the same tree (which is what happens when you do a revert; you have the same tree as in the other commit) setting both values on the same key will result in one of the two commits lost.

This is the particular problem with sgit; the API is too concrete. We need more abstract APIs like the ones provided in Gitteh or nodegit. If SGit had a Commit type in its API, for example, the avobe problems wouldn't exist, since git_log would return an array of Commit objects. Also the same object would be used i.e. when committing, resulting in a modular API.

@jmendeth

@jmendeth you raise a valid bug regarding the the keys being the tree OID's rather than the commit OID's. Sorting is intended to be a parameter just as it is in the git CLI. The pointer cast you mention is ugly but not an immediate cause for concern as the function is self contained. It should be cleaned up though. Sgit is definitely 0.1, I'm just starting to cleanup some things now that I'm committed to the project. Things like removing the dependency on boost, getting travis builds to work, unit tests, documentation, etc. The code itself has had less than a half-day of coding on it.

I think there are two very different approaches, I absolutely do not want to expose the libgit2 api into JS space. I think something like the sqlite addon is an outstanding example to work towards. It's almost automagical but also very fast and memory efficient. Having a large list of 1x1 C api mappings to me it's just not nodelike and I don't like the memory "leaking/consumption" and context switching problems it raises. Try to creating hundreds of repos each with a thousand commits and look at memory (which is very close to my project's needs) and performance with any of the options here.

The only way nodegit could work for me would be to manually call the GC or implement additional cleanup in the runtime of the addon and there are numerous problems with both approaches. My scenario involves creating lots of repositories and creating lots of commits and then doing data analysis of those. I'm focusing on an API that is very close to the git command line both for ease of use and ease of documentation. That lends itself to about 20 main api functions and parameterized options arrays.

@FrozenCow No doubt async support needs to be implemented. Having said that though there are a couple of different ways to do it each with various tradeoffs. I haven't determined what the best approach is yet. I might suggest trying it, as is, so that you can at least see what the impact looks like for your project. I see a lot of comment about the async issues but little data, I need to see more data to make a good decision of how to implement it best. For the scenario I have, the impact of sync is much less than I would have thought it would be for the currently supported operations. If you're on NFS it's going to be potentially catastrophoc. I would be very interested in feedback on semi-realworld cases to implement as benchmarks and tests to see the impact of various tradeoffs in implementing async support.

My next steps are to implement object gets, resolve the tree OID issue jmendeth brought up and implement git_index_add_all, at that point I think I can construct a good benchmark that will inform as to the best means of implementing async.

Look up disk seek http://julianhyde.blogspot.com/2010/11/numbers-everyone-should-know.html and you will have your answer

@lushzero

@lushzero Doing things async in node shouldn't be a discussion. Say you're creating a http server in node that does things with git. Now in that same script you want to clone a repository like the Linux kernel. Cloning a repo of that scale takes a while, say a couple of minutes. If cloning is done synchronously your server is not responing, since no other Javascript code can be ran on v8's thread, since you're using that thread for cloning.

Cloning is just an example that takes a long time, but any operation will suspend the whole node process if done synchronously. That's why the gitteh and nodegit implementations are so much more complex compared to sgit, they need to do everything in a separate thread.

@FrozenCow

@FrozenCow Just to clarify I'm not debating async or not async, I'm debating the precise implementation for async. What level of abstraction is employed that wraps what api function, the number of threads used, how the libgit2 threads interplay, etc. From a cursory review both gitteh and nodegit seem to use the same approximate abstraction regarding async and it's very simple. I just started experimenting and benchmarking for sgit async this afternoon.

Just an update for everyone following along. Enough people are productively using the nodegit wip branch without segfaults or noticeable memory leaks that I've merged to master. Furthermore, the documentation is now all to date -- it correctly merges the "raw" and "convenience" layer together. Feel free to use master as the npm artifact for now.

I will publish a nodegit 0.2.0 release in a week or so if it seems like the code is stable enough.

I updated sgit to operate asynchronously and better reflect git CLI naming conventions. Best of luck to you all.

Just wondering 8 months later: I'm about to start a project that needs git and I have a rough time figuring out which of these projects to use. Has the view on that changed since 8 months back?

@walterheck

@walterheck this project is still unmaintained at this point. nodegit might be a better approach.

@walterheck

Had a feeling yeah, already started looking at nodegit here:
http://www.nodegit.org/

On Mon, May 5, 2014 at 8:54 AM, Sam notifications@github.com wrote:

@walterheck https://github.com/walterheck this project is still
unmaintained at this point. nodegit https://github.com/nodegit/nodegitmight be a better approach.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/68#issuecomment-42163014
.

Best regards,

Walter Heck
CEO / Founder OlinData http://olindata.com/?src=wh_gapp - Open Source
Training & Consulting

Check out our upcoming Drupal and Puppet
trainingshttp://olindata.com/?src=wh_gapp#calendar-icon-title

Navigation Menu

Repository navigation

Gitteh needs a primary maintainer! #68

31 remaining items

My results

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Gitteh needs a primary maintainer! #68

Description

Activity

samcday commented on Aug 31, 2013

mildsunrise commented on Aug 31, 2013

FrozenCow commented on Aug 31, 2013

mildsunrise commented on Aug 31, 2013

iamwilhelm commented on Aug 31, 2013

mildsunrise commented on Aug 31, 2013

ben commented on Aug 31, 2013

mildsunrise commented on Aug 31, 2013

lushzero commented on Sep 1, 2013

samcday commented on Sep 1, 2013

mildsunrise commented on Sep 1, 2013

FrozenCow commented on Sep 1, 2013

mildsunrise commented on Sep 1, 2013

nkallen commented on Sep 1, 2013

mildsunrise commented on Sep 1, 2013

31 remaining items

mildsunrise commented on Sep 4, 2013

My results

nkallen commented on Sep 4, 2013

mildsunrise commented on Sep 4, 2013

FrozenCow commented on Sep 4, 2013

mildsunrise commented on Sep 4, 2013

mildsunrise commented on Sep 4, 2013

lushzero commented on Sep 4, 2013

nkallen commented on Sep 5, 2013

FrozenCow commented on Sep 5, 2013

lushzero commented on Sep 5, 2013

nkallen commented on Sep 6, 2013

lushzero commented on Sep 9, 2013

walterheck commented on May 5, 2014

samcday commented on May 5, 2014

walterheck commented on May 5, 2014

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions