ssh troubleshooting

In this document:


This document should help you troubleshoot ssh-related problems in accessing gitolite after the install has completed successfully.

In addition, I strongly recommend reading this document -- it's a very detailed look at how gitolite uses ssh's features on the server side. Most people don't know ssh as well as they think they do; even if you don't have any problems right now, it's worth skimming over.

In addition to both these documents, there's now a program called sshkeys-lint that you can run on your client. Run it without arguments to get help on how to run it and what inputs it needs.

the most common problems that an admin will see

Ironically, these problems only happen to the person who installed gitolite using easy-install, and has utterly failed to read/heed the message that shows up at the end of running that command. This is because only the admin has two ssh keys to the server (see "basic ssh troubleshooting for the main admin" section below for more on this).

Both these problems are caused by using the wrong key, thus bypassing gitolite completely:

Let us recap the message that appears on a successful run of the "easy-install" program; it looks something like this (with suitable values substituted for <user>, <server>, and <port>):

IMPORTANT NOTE -- PLEASE READ!!!
    *Your* URL for cloning any repo on this server will be
        gitolite:reponame.git

    *Other* users you set up will have to use
        <user>@<server>:reponame.git
    However, if your server uses a non-standard ssh port, they should use
        ssh://<user>@<server>:<port>/reponame.git

    If this is your first time installing gitolite, please also:
        tail -31 src/gl-easy-install
    for next steps.

The first error above happens if you use git@server:reponame instead of gitolite:reponame. All your repos are actually in a subdirectory pointed to by $REPO_BASE in the rc file (default: repositories). Gitolite internally prefixes this before calling the actual git command you invoked, but since you're bypassing gitolite completely, this prefixing does not happen, and so the repo is not found.

The second error happens if you use git@server:repositories/reponame.git (assuming default $REPO_BASE setting) -- that is, you used the full unix path. Since the "prefixing" mentioned above is not required, the shell finds the repo and clones ok. But when you push, gitolite's update hook kicks in, and fails to run because you some of the environment variables it is expecting are not present.


basic ssh troubleshooting

I assume the gitolite server is called "server" and the user hosting all the gitolite repos is "git". I will also be using "sitaram" as the gitolite username of the admin.

Unless specifically mentioned, all these commands are run on the user's or admin's workstation, not on the server.

passphrases versus passwords

When you create an ssh keypair, you have the option of protecting it with a passphrase. When you subsequently use that keypair to access a remote host, your local ssh client needs to unlock the corresponding private key, and ssh will probably ask for the passphrase you set when you created the keypair.

Do not confuse or mistake this prompt (Enter passphrase for key '/home/sitaram/.ssh/id_rsa':) for a password prompt from the remote server!

You have two choices to avoid this prompt every time you try to access the remote. The first is to create keypairs without a passphrase (just hit enter when prompted for one). Be sure to add a passphrase later, once everything is working, using ssh-keygen -p.

The second is to use ssh-agent (or keychain, which in turn uses ssh-agent) or something like that to manage your keys. Other than the next section, further discussion of this is out of scope of this document.

ssh-agent problems

  1. Run ssh-add -l. If this responds with either "The agent has no identities." or "Could not open a connection to your authentication agent.", skip this section.

  2. However, if it lists some keys, like this:

    2048 fc:c1:48:1e:06:31:97:a4:8b:fc:37:b2:76:14:c7:53 /home/sitaram/.ssh/id_rsa (RSA)
    2048 d2:e0:7f:fa:1a:89:22:41:bb:06:d9:ff:a7:27:36:5c /home/sitaram/.ssh/sitaram (RSA)
    

    then run ls ~/.ssh and make sure that all the keypairs you have there are represented in the ssh-add -l output.

  3. If you find any keypairs in ~/.ssh that are not represented in the ssh-add -l output, add them. For instance, if ssh-add -l showed me only the id_rsa key, but I also had a sitaram (and sitaram.pub) keypair, I'd run ssh-add ~/.ssh/sitaram to add it.

This is because ssh-agent has a quirk: if ssh-add -l shows any keys at all, ssh will only use those keys. Even if you explicitly specify an unlisted key using ssh -i or an identityfile directive in the config file, it won't use it.

basic ssh troubleshooting for the main admin

You're the "main admin" if you're trying to access gitolite from the same workstation and user account where you ran the "easy install" command. You should have two keypairs in your ~/.ssh directory. The pair called id_rsa (and id_rsa.pub) was probably the first one you created, and you used this to get passwordless (pubkey based) access to the server (which was a pre-requisite for running the easy install command).

The second keypair has the same name as the last argument in the easy install command you ran (in my case, sitaram and sitaram.pub). It was probably created by the easy install script, and is the key used for gitolite access.

In addition, you should have a "gitolite" paragraph in your ~/.ssh/config, looking something like this:

host gitolite
     user git
     hostname server
     identityfile ~/.ssh/sitaram

If any of these are not true, you did something funky in your install; email me or hop onto #git and hope for the best ;-)

Otherwise, run these checks:

  1. ssh git@server should get you a command line.

    If it asks you for a password, then your id_rsa keypair changed after you ran the easy install, or someone fiddled with the ~/.ssh/authorized_keys file on the server.

    If it prints gitolite version and access info, you managed to overwrite the id_rsa keypair with the sitaram keypair, or something equally weird.

  2. ssh gitolite info should print some gitolite version and access info. If you get the output of the GNU info command instead, you probably reused your id_rsa keypair as your sitaram keypair, or overwrote the sitaram keypair with the id_rsa keypair.

There are many ways to fix this, depending on where and what the damage is. The most generic way (and therefore time-taking) is to re-install gitolite from scratch:

That's a long sequence but it should work.

basic ssh troubleshooting for a normal user

For a normal user, life is much simpler. They should have only one pubkey, which was previously sent to the gitolite admin to add into the admin repo's keydir as "user.pub", and then "user" given permissions to some repo.

ssh git@server info should get you gitolite version and access info. If it asks you for a password, your pubkey was not sent to the server properly. Check with your admin.

If it gets you the GNU info command output, you have shell access. This means you had command line access to the server before you were added as a gitolite user. If you send that same key to your gitolite admin to include in the admin repo, it won't work. For reasons why, see below.

details

Here's how it all hangs together.

files on the server

files on client

why two keys on client

Why do I (the admin) need two different keypairs?

There are two types of access the admin will make to the server: a normal login, to get a shell prompt, and gitolite access (clone/fetch/push etc). The first access needs an authkeys line without any "command=" restrictions, while the second requires a line with such a restriction.

And we can't use the same key for both because there is no way to disambiguate them; the ssh server will always (always) pick the first one in sequence when the key is offered by the ssh client.

So the next question is usually "I have other ways to get a shell on that account, so why do I need a key for shell access at all?"

The answer to this is that the "easy install" script, being written for the most general case, needs shell access via ssh to do its stuff.

If you really, really, want to get rid of the extra key, here's a transcript that should have enough info to get you going (but it helps to know ssh well):

That should do it.

more complex ssh setups

What do you need to know in order to create more complex ssh setups (for instance if you have two gitolite servers you are administering)?

two gitolite servers to manage?

giving shell access to gitolite users

We've managed (thanks to an idea from Jesse Keating) to make it possible for a single key to allow both gitolite access and shell access.

This is done by copying the pubkey (to which you want to give shell access) to the server and running either

cd $HOME/.gitolite  # assuming default $GL_ADMINDIR in ~/.gitolite.rc
src/gl-tool shell-add ~/foo.pub

or

gl-tool shell-add ~/foo.pub

The first method is to be used if you used the user-install mode, while the second method is for the system-install followed by user-setup mode (see doc/0-INSTALL.mkd, section on "install methods", for more on this)

IMPORTANT UPGRADE NOTE: previous implementations of this feature were crap. There was no easy/elegant way to ensure that someone who had repo admin access would not manage to get himself shell access.

Giving someone shell access requires that you should have shell access in the first place, so the simplest way is to enable it from the server side only.