Hades Blag blog posts with tag intersection git

Your Own Git Hosting: Gitolite

2010-12-02T17:53:52Z

So, let’s start with setting up gitolite to host your repositories and provide SSH access to them. There is also a tool called gitosis, but it is not as supported and featurefull as gitolite.

So what does it actually do? In essence, all the developers that will access your repositories will login to your server as user “git”, run server-side git daemon and sync data with it. Gitolite will determine if a user has access to perform operations he is going to. Ok, but why bother with all this crap, and not just create a bunch of users on the server and tell them where is the repository? There is a couple of reasons.

Firstly, you may not want to give your developers shell access to your server. Why wouldn’t you? Who knows, probably because they are stupid or malicious, or whatever. With gitolite they won’t get a shell access, because they will only be able to login with their SSH keys and that SSH keys will be restricted to invoking gitolite only. At this point you may want to read up SSH manual on keys and command restrictions, if you do not yet know about it.

Secondly, you may want to impose certain access restrictions on them. Gitolite allows to restrict read write and rewrite down to per-branch level.

Thirdly, with local multiuser git access you have to fiddle a little with umasks or write a hook that fixes the permissions, but that’s unimportant and uninteresting.

Fourthly, you may even not have a root access. That’s right, gitolite allows you to setup multi-user repositories on systems where you have only one non-root account.

So let’s go ahead with installation. The INSTALL doc is rather straightforward and you should in fact read it instead of this blog post. I am not offering the text below as a setup instructions, but merely as a log of how I installed it on my server.

First off, we require some git on our server. Please be careful to use as fresh git as possible, because it generally tends to gets even more super awesome with time. Also gitolite requires at least git 1.6.2 at the time of writing of this post. Unless you use especially sucky distribution (ahem debian ahem), you should be fine with git from your repository. I just told Portage to install me one:

    emerge -av git

Although gitolite has an ebuild in Gentoo, I set it up manually and will walk you along the procedure. First, lets obtain the sources:

    git clone git://github.com/sitaramc/gitolite.git

This will create a “gitolite” directory with gitolite sources. Let’s create a system directory for gitolite. I use /opt for all non-Portage packages, so let’s go ahead with:

    GITOLITE=/opt/gitolite
    sudo mkdir -p ${GITOLITE}/{bin,conf,hooks}
    sudo chown -R `whoami` ${GITOLITE}/{bin,conf,hooks}

You may of course use another set of directories if you want. Next, we let gitolite install itself wherever we want it:

    ./src/gl-system-install ${GITOLITE}/{bin,conf,hooks}

This takes three arguments: directory for binaries, directory for configs and directory for hooks. Also this is the same command you would use to update your gitolite installation after you git pull the new sources.

We would now require a user account for git. Let’s create them!

    sudo groupadd -r git
    sudo useradd -d /srv/git -g git -m -r -s `which bash` git

This creates a system (-r) group git and a system (-r) user git with home directory (-d) /srv/git and shell (-s) bash, in group (-g) git, and creates its home directory (-m). Now login into your new git account. Don’t forget to add your ${GITOLITE}/bin directory to the .bashrc if it is not in the PATH:

    sudo su — git
    echo PATH='${PATH}':/opt/gitolite/bin/ >> .bashrc
    source .bashrc
    which gl-setup

Now copy over your public key to a file called username.pub and finalize gitolite setup by running:

    gl-setup username.pub

After that, follow the instructions, they should be fairly straightforward. Congratulations, you’re done! The gitolite’s configuration is stored in a Git repository under gitolite, so you’d want to clone it from your computer (i.e. where you have the private key for username.pub):

    git clone git@yourservername:gitolite-admin
    cd gitolite-admin
    vim conf/gitolite.conf

The file gitolite.conf is in fact your gitolite config file. After you edit it, save it, commit it and push to the gitolite-admin repository on your server, gitolite checks the config and makes appropriate changes to the repositories. To add users, just place their keys into keydir directory. It’s that simple! You probably would want to read official admin doc just about now.

Your Own Git Hosting: Prologue

2010-09-07T15:49:55Z

As you might know, Git itself does not deal with collaboration, letting you choose any adequate model. So how does one go about publishing his repository? Or letting someone push into their repository? The answers are multiple and confusing. Official Git Manual gives a dry summary of the basic options, which you may read now, or at your leisure. I’ll just note the few key points.

Limiting the discussion to network exchange, Git supports three basic protocols: git protocol, HTTP and SSH. All of them support pulling and pushing, however each of them has drawbacks. Git protocol does not support any kind of authentication, so it is mostly used for read-only public access. HTTP is slow and stupid (which means that it allows only file transfer, so you can’t use Git hooks). SSH is great in all respects, but it implies that every person that has access to your repository has an account on the server, where it is hosted.

In the majority of the cases, two protocols are set up for Git repository: public read-only git protocol access and private read/write SSH access. However I can name you upon request at least one real-world case, when git protocol was successfully used for unauthenticated read/write access :). HTTP protocol is usually used in those morbid cases where you can in no conceivable way get rid of it (imagine a firewall that blocks everything except port 80 or 443).

So how do you setup a Git hosting? There are a lot of web Git hostings (gitorious.org, github.com), and project hostings with Git support (sf.net) out there. Mostly they provide free hosting for open-source projects and charge a fee for private repositories.

However, public Git hostings can not satisfy everyone. There are closed projects, that could not be trusted to a third-party. There are closed projects that do not have spare money for Git hosting. Also, it is hard to integrate hosted Git repository with other development tools (buildbot, bugzilla, etc.), mainly because custom hooks are not supported. So that brings us to the following problem.

Given:

a server.

Required:

hosting of any number of Git repositories;
public read-only access to a subset of them via git protocol;
read/write access for authorized users via SSH;
web interface to the repositories.

In the following posts, I will tell you how I solved the problem for myself using the following tools:

gitolite for SSH access management;
git-daemon for anonymous git protocol access;
cgit for web interface.

Please stay tuned!

P.S: I’ve heard some concerns regarding cgit not working properly in some cases. I can neither confirm this nor disprove (since I rarely use web frontends for code browsing), but I’ll look into it.

Git Is Your Friend not a Foe Vol. 4: Rebasing

2010-03-03T21:02:23Z

This time I’ll talk about more complex things, which give developers more power over their life. Actually, they just look complex. In fact these are quite natural operations over Git commit history structure, which was described already in my previous posts and gazillions of posts by other people.

So, let’s start with the simplest. Mr. Cleverhead sometimes fails to remember which branch he is currently on. He commits to a branch master, while in fact he should have committed that to staging. What can he do to fix it? He decides to run git show, save the commit as a patch, then checkout branch staging, apply patch there and commit with the same commit message. Well, it happens so that Git already has a command that does exactly this automatically! It is called git cherry-pick. Besides fixing Mr. Cleverhead’s reputation, it is also quite often used for example to backport commits to release branches.

Note, however, that this still requires Mr. Cleverhead to remove his commit from master. We believe in him.

The next big complex thing is a branch-wise cherry-pick. This is called a rebase and often casts a great deal of confusion upon novice Git users. But actually rebase is to cherry-pick as is multiplication to addition: by doing rebase you just cherry-pick a series of commits on another branch. Although git-rebase manual page is ugly and unfriendly, it tells the same basic thing: rebase is a series of cherry-picks, followed by a branch reset.

Note the desaturated old commits. Despite Git changed the branch head, it didn’t remove these old commits. They are still accessible through reflog, or by their SHA1 ids, in case you realise you’ve made a mistake.

There are many use-cases for git rebase, from Mr. Cleverhead committing several commits in a row to a wrong branch, to complex integration and release cycle management workflows. See, for example, http://nvie.com/git-model.

There is also an interactive mode for git rebase. And it is truly awesome! It allows you to edit your branch in any way you want: remove commits, edit commits, squash commits, even change commit order. So try it out. Now.

The post would be incomplete if I didn’t mention git filter-branch. It is basically the same as rebase, but it usually affects the whole branch up to the initial commit and for every commit Git performs a certain action. It is very complex and powerful tool that is quite rarely needed and quite more rarely used. Its uses include: removing a file from the whole project history (for example because of license issues, or because Mr. Cleverhead accidentally added his grandmother’s recipe book five years ago); fixing author’s name in commit messages; creating a separate Git repository from a subdirectory. All this comes free with Git and doesn’t require you to spend weeks doing this any other clumsy way.

Last thing I would like to mention today is pull-rebasing. This refers to the following problem: by default git pull means git fetch followed by git merge, which is perfectly fine if you haven’t committed anything locally. But if you have, this creates a merge commit. This is perfectly fine either. But if you do small commits often, or work on a big feature in your branch and merge master often to be in sync with updates, this will create a so-called “loopy history”, which usually pisses people off. Especially Linus. So if you have committed a small fix that you can’t push because Mrs. Slowrunner managed to push a commit before you, use git pull --rebase. This will rebase your work upon Mrs. Slowrunner’s work and no merge commit will be created. If you work on a topic branch and would like to sync it with master, simply run git rebase master. This will reduce pissed off people count too.

Note, however, that rebase rewrites history! This means, that if you have published your work somewhere, you shouldn’t rebase it, unless you have warned people that they may expect history rewriting. This is usually refered to as throw-away branches (for example branch pu of git.git). If you have published your work, rewritten it and try to publish it again, git push won’t let you. If you force it to, expect angry people to come to your house with your local analogue of baseball bats.

Regarding angry people: some workflows require topic branches to be squashed to a single commit before merging to mainline. This is easy in Git: just use the --squash option of git merge, and everyone will be happy. If you want to squash only some of them (for example, make 5 commits out of 20), you can use aforementioned interactive rebase.

This pretty much concludes this series of Git posts. If you feel that I have left something unveiled, please tell me that! I appreciate all the comments.

All posts about Git

Git Is Your Friend not a Foe Vol. 3: Refs and Index

2010-03-03T20:44:18Z

Let’s take a walk along Git repository structure. The central square is Git Object Database. Objects reference each other by 160-bit unique IDs with a certain semantics (for example, commit-object references its parent commit(s) and the tree that corresponds to project’s root directory; tree-object references blob-objects that correspond to file content and tree-objects that correspond to subdirectories; etc., see gittutorial-2(7) for details). For the sake of simplicity, let’s forget about trees and blobs for now, and look at commits only.

We now have a bunch of commits that know who were their parents. We can trace history from any given commit back to the very beginning. But how do we know what is the current state of things? What was the latest commit in the history? To answer that let’s look at Git refs (short for references). They are basically named references for Git commits. There are two major types of refs: tags and heads. Tags are fixed references that mark a specific point in history, for example v2.6.29. On the contrary, heads are always moved to reflect the current position of project development.

Now we know what is happening in the project. But to know what is happening right here, right now there is a special reference called HEAD. It serves two major purposes: it tells Git which commit to take files from when you checkout and it tells Git where to put new commits when you commit. When you run git checkout ref it points HEAD to the ref you’ve designated and extracts files from it. When you run git commit it creates a new commit object, which becomes a child of current HEAD. Normally HEAD points to one of the heads, so everything works out just fine.

But if you checkout a specific commit instead of a branch, your HEAD starts pointing at this commit. This is referred to as detached head and you may be told that you are not on a branch (git branch says “(no branch)”). This is perfectly fine, but if you commit anything to it, your commits won’t have a known ref, so if you checkout another branch, you can lose them.

Having said about committing, can’t help stopping by the process of committing itself. You may already know that the Git’s “add” operation differs from almost every other VCS in that you have to “add” not only files that are not yet known to Git, but also files that you have just modified. This is because Git takes content for next commit not from your working copy, but from a special temporary area, called index. This allows finer control over what is going to be committed. You can not only exclude some files from commit, you can exclude even certain pieces of files from commit (try git add -i). This helps developers stick to atomic commits principle.

And if you have inhuman ability of creating only perfect committs and need stupid VCS only to obey your orders, then you can just use option “-a” for git-commit. And I envy you.

Another special kind of refs are remotes. Whenever you run git fetch, it asks the remote repository, what heads and tags does it have, downloads missing objects (if any) and stores remote refs under refs/remotes prefix. The remote heads are displayed if you run git branch -r.

Some of your branches (notably master) may be what is called tracking branches. That means that a certain branch “tracks” its remote counterpart. Physically that means that when you run git pull on that branch, the corresponding remote branch gets automatically merged into your local branch. Fairly recent versions of Git set up tracking automatically when you checkout a remote branch (for example, git checkout -b stable origin/stable). Note, however, that sometimes it’s better to rebase instead of merge.

But that’s a whole new story…

Volume 4, on cherry-picking and rebasing

All posts about Git

Git Is Your Friend not a Foe Vol. 2: Branches

2010-03-03T20:42:12Z

So if you worked with some version control systems for a bit, you’ve probably heard of a concept called branches. It is quite a simple concept: you can perform several development processes in parallel without them interfering with each other. Most projects use branches for experimental features that could set hell loose and for backporting bugfixes to older releases. Subversion and CVS people usually dislike branches, because they involve lots of uninteresting and painful work that they don’t want to do. That is easily explained by the way branches are implemented there.

As you might know, branches in SVN are implemented in a very interesting fashion. They are not, in fact, implemented at all. SVN branch is just a folder, which is created when a branch is started. If you want to merge it back, you need to remember the revision number, when you created the branch, and use that magical number in a complex “svn merge” command. But still, SVN project history remains a straight line.

What’s wrong with this way of interpreting the branch concept? Nothing. It’s completely fine, if you don’t want to work with branches. And you should want to work with branches, because they are actually awesome! Especially if implemented as in Git.

Instead of SVN-y linear development history, Git’s commit history is a more complex structure: each commit can have multiple parents and multiple children. In computer science this is called a Directed Acyclic Graph (and if this rings any bells you may want to read Tv’s article “Git for Computer Scientists”). In practice that means that you are not restricted to developing upon the latest revision in project’s history. Instead, you can take any existing commits and start creating commits off it. If you want to merge them, you create a commit that is a child of two commits (such children are called merge commits).

This way you get a graph with several commits with no children (let’s call them branch heads for now). Every commit has a reference to its parent. So if we take the branch head, we can trace back the project history to the very beginning. This is why the Git branch is simply a reference to its head (you can go ahead and look into the files in .git/refs/heads directory of any of your Git repositories).

Most of the time you will have a checked-out branch (the special reference HEAD points to the current branch, see .git/HEAD for example). When you commit something, your commits are attached to it and the branch reference is moved to the new commit. Simple. But sometimes your HEAD may point to something other than a branch head (for example, when you checkout an older revision by its ID or tag). This is called a detached head. It is a very simple, important and confusing situation. There’s nothing wrong about it, but it hides a peril: if you commit to a detached head, Git creates a new commit and attaches it to the current commit, forming a branch. But this branch has no name! It will just grow sideways as a normal branch without a name. Here’s what’s wrong with it:

it is confusing, because commits do not go to master, or whatever branch you had checked out before;
if you check out another branch, you won’t be able to return to this branch by its name, it simply doesn’t have any.

Note, that Git won’t let you lose data easily and won’t force you to do unneeded work. Let me tell you what to do in case you have committed to a detached head. Suppose you created just one single commit to a detached head and now just sit and look at it. You have two options:

create a new branch with the current commit as a head: git branch branchname,
attach the new commit to another branch (suppose it is branch master): remember the ID of the new commit, checkout the required branch (git checkout master), cherry-pick your commit to it (git cherry-pick id).

If you find yourself in the second situation (you’ve just committed to a detached head, checked out another branch and don’t remember the commit id), you may use Git’s reflog (git reflog, or git log -g). This will list the history of your HEAD (checkouts, commits and the such), where you can take commit ID and use it wisely.

Merging is an important part of Git workflow. You will, in fact, do merges frequently even if you don’t use branches other than master, provided you use more than one repository. That is because master of one repository and master of another repository are, in general, different branches. So when you do push or pull to/from another repository you do a merge. Git differentiates two merge types (suppose you attempt to merge branch B into branch A):

fast-forward merge. This happens when B is a direct descendant of A. This is resolved trivially: Git simply moves reference A to point to B,
non fast-forward merge. This covers all the remaining cases, and requires a merge commit to be created (merge commit is a commit with at least two parents).

This differentiation is important because the fast-forward merge can be performed automatically without human intervention. That’s why this is the only merge possible during Git push. The non fast-forward merge may result in edit conflicts (the situation when two lines of development changed the same line of the same file differently), so a human intervention may be required. This is what is meant by a (not immediately clear) git-push message “remote rejected: non fast-forward”: sorry, I can’t push your modifications, because remote branch has diverged, please resolve this manually. Most often this occurs when another developer managed to push his changes first. In this case just run “git pull”, resolve conflicts (if any), then run “git push”. Less often this occurs when a remote branch has been changed completely (for example, branch pu of Git Git repository is changed very frequently and is not supposed to be developed upon). This means that either you or the remote repository owner screwed up, so you’d better talk to each other. Sometimes this occurs when you try to push to a completely unrelated repository. So just be careful there.

I should note here, that the “—force” option to git-push along with +refspec notation is not going to solve your problems automagically. It will simply destroy the remote history, replacing with your own. So you should never use it, unless you know exactly what you are doing.

Next up: rebasing and staging area.

Volume 1, on the distributed VCS concepts

All posts about Git