[git] Update Git submodule to latest commit on origin

I have a project with a Git submodule. It is from an ssh://... URL, and is on commit A. Commit B has been pushed to that URL, and I want the submodule to retrieve the commit, and change to it.

Now, my understanding is that git submodule update should do this, but it doesn't. It doesn't do anything (no output, success exit code). Here's an example:

$ mkdir foo
$ cd foo
$ git init .
Initialized empty Git repository in /.../foo/.git/
$ git submodule add ssh://user@host/git/mod mod
Cloning into mod...
user@host's password: hunter2
remote: Counting objects: 131, done.
remote: Compressing objects: 100% (115/115), done.
remote: Total 131 (delta 54), reused 0 (delta 0)
Receiving objects: 100% (131/131), 16.16 KiB, done.
Resolving deltas: 100% (54/54), done.
$ git commit -m "Hello world."
[master (root-commit) 565b235] Hello world.
 2 files changed, 4 insertions(+), 0 deletions(-)
 create mode 100644 .gitmodules
 create mode 160000 mod
# At this point, ssh://user@host/git/mod changes; submodule needs to change too.
$ git submodule init
Submodule 'mod' (ssh://user@host/git/mod) registered for path 'mod'
$ git submodule update
$ git submodule sync
Synchronizing submodule url for 'mod'
$ git submodule update
$ man git-submodule 
$ git submodule update --rebase
$ git submodule update
$ echo $?
0
$ git status
# On branch master
nothing to commit (working directory clean)
$ git submodule update mod
$ ...

I've also tried git fetch mod, which appears to do a fetch (but can't possibly, because it's not prompting for a password!), but git log and git show deny the existence of new commits. Thus far I've just been rm-ing the module and re-adding it, but this is both wrong in principle and tedious in practice.

In your project parent directory, run:

git submodule update --init

Or if you have recursive submodules run:

git submodule update --init --recursive

Sometimes this still doesn't work, because somehow you have local changes in the local submodule directory while the submodule is being updated.

Most of the time the local change might not be the one you want to commit. It can happen due to a file deletion in your submodule, etc. If so, do a reset in your local submodule directory and in your project parent directory, run again:

git submodule update --init --recursive

Here's an awesome one-liner to update everything to the latest on master:

git submodule foreach 'git fetch origin --tags; git checkout master; git pull' && git pull && git submodule update --init --recursive

Thanks to Mark Jaquith

If you are looking to checkout master branch for each submodule -- you can use the following command for that purpose:

git submodule foreach git checkout master

Plain and simple, to fetch the submodules:

git submodule update --init --recursive

And now proceed updating them to the latest master branch (for example):

git submodule foreach git pull origin master

the simplest way to handle git projects containing submodules is to always add

--recurse-submodules 

at the end of each git command example:

git fetch --recurse-submodules

another

git pull --update --recurse-submodules

etc...

git pull --recurse-submodules

This will pull all the latest commits.

Git 1.8.2 features a new option, --remote, that will enable exactly this behavior. Running

git submodule update --remote --merge

will fetch the latest changes from upstream in each submodule, merge them in, and check out the latest revision of the submodule. As the documentation puts it:

--remote

This option is only valid for the update command. Instead of using the superproject’s recorded SHA-1 to update the submodule, use the status of the submodule’s remote-tracking branch.

This is equivalent to running git pull in each submodule, which is generally exactly what you want.

Your main project points to a particular commit that the submodule should be at. git submodule update tries to check out that commit in each submodule that has been initialized. The submodule is really an independent repository - just creating a new commit in the submodule and pushing that isn't enough. You also need to explicitly add the new version of the submodule in the main project.

So, in your case, you should find the right commit in the submodule - let's assume that's the tip of master:

cd mod
git checkout master
git pull origin master

Now go back to the main project, stage the submodule and commit that:

cd ..
git add mod
git commit -m "Updating the submodule 'mod' to the latest version"

Now push your new version of the main project:

git push origin master

From this point on, if anyone else updates their main project, then git submodule update for them will update the submodule, assuming it's been initialized.

In my case, I wanted git to update to the latest and at the same time re-populate any missing files.

The following restored the missing files (thanks to --force which doesn't seem to have been mentioned here), but it didn't pull any new commits:

git submodule update --init --recursive --force

This did:

git submodule update --recursive --remote --merge --force

It seems like two different scenarios are being mixed together in this discussion:

Scenario 1

Using my parent repository's pointers to submodules, I want to check out the commit in each submodule that the parent repository is pointing to, possibly after first iterating through all submodules and updating/pulling these from remote.

This is, as pointed out, done with

git submodule foreach git pull origin BRANCH
git submodule update

Scenario 2, which I think is what OP is aiming at

New stuff has happened in one or more submodules, and I want to 1) pull these changes and 2) update the parent repository to point to the HEAD (latest) commit of this/these submodules.

This would be done by

git submodule foreach git pull origin BRANCH
git add module_1_name
git add module_2_name
......
git add module_n_name
git push origin BRANCH

Not very practical, since you would have to hardcode n paths to all n submodules in e.g. a script to update the parent repository's commit pointers.

It would be cool to have an automated iteration through each submodule, updating the parent repository pointer (using git add) to point to the head of the submodule(s).

For this, I made this small Bash script:

git-update-submodules.sh

#!/bin/bash

APP_PATH=$1
shift

if [ -z $APP_PATH ]; then
  echo "Missing 1st argument: should be path to folder of a git repo";
  exit 1;
fi

BRANCH=$1
shift

if [ -z $BRANCH ]; then
  echo "Missing 2nd argument (branch name)";
  exit 1;
fi

echo "Working in: $APP_PATH"
cd $APP_PATH

git checkout $BRANCH && git pull --ff origin $BRANCH

git submodule sync
git submodule init
git submodule update
git submodule foreach "(git checkout $BRANCH && git pull --ff origin $BRANCH && git push origin $BRANCH) || true"

for i in $(git submodule foreach --quiet 'echo $path')
do
  echo "Adding $i to root repo"
  git add "$i"
done

git commit -m "Updated $BRANCH branch of deployment repo to point to latest head of submodules"
git push origin $BRANCH

To run it, execute

git-update-submodules.sh /path/to/base/repo BRANCH_NAME

Elaboration

First of all, I assume that the branch with name $BRANCH (second argument) exists in all repositories. Feel free to make this even more complex.

The first couple of sections is some checking that the arguments are there. Then I pull the parent repository's latest stuff (I prefer to use --ff (fast-forwarding) whenever I'm just doing pulls. I have rebase off, BTW).

git checkout $BRANCH && git pull --ff origin $BRANCH

Then some submodule initializing, might be necessary, if new submodules have been added or are not initialized yet:

git submodule sync
git submodule init
git submodule update

Then I update/pull all submodules:

git submodule foreach "(git checkout $BRANCH && git pull --ff origin $BRANCH && git push origin $BRANCH) || true"

Notice a few things: First of all, I'm chaining some Git commands using && - meaning previous command must execute without error.

After a possible successful pull (if new stuff was found on the remote), I do a push to ensure that a possible merge-commit is not left behind on the client. Again, it only happens if a pull actually brought in new stuff.

Finally, the final || true is ensuring that script continues on errors. To make this work, everything in the iteration must be wrapped in the double-quotes and the Git commands are wrapped in parentheses (operator precedence).

My favourite part:

for i in $(git submodule foreach --quiet 'echo $path')
do
  echo "Adding $i to root repo"
  git add "$i"
done

Iterate all submodules - with --quiet, which removes the 'Entering MODULE_PATH' output. Using 'echo $path' (must be in single-quotes), the path to the submodule gets written to output.

This list of relative submodule paths is captured in an array ($(...)) - finally iterate this and do git add $i to update the parent repository.

Finally, a commit with some message explaining that the parent repository was updated. This commit will be ignored by default, if nothing was done. Push this to origin, and you're done.

I have a script running this in a Jenkins job that chains to a scheduled automated deployment afterwards, and it works like a charm.

I hope this will be of help to someone.

Note, while the modern form of updating submodule commits would be:

git submodule update --recursive --remote --merge --force

The older form was:

git submodule foreach --quiet git pull --quiet origin

Except... this second form is not really "quiet".

See commit a282f5a (12 Apr 2019) by Nguy?n Thái Ng?c Duy (pclouds).
^{(Merged by Junio C Hamano -- gitster -- in commit f1c9f6c, 25 Apr 2019)}

submodule foreach: fix "<command> --quiet" not being respected

Robin reported that

git submodule foreach --quiet git pull --quiet origin

is not really quiet anymore.
It should be quiet before fc1b924 (submodule: port submodule subcommand 'foreach' from shell to C, 2018-05-10, Git v2.19.0-rc0) because parseopt can't accidentally eat options then.

"git pull" behaves as if --quiet is not given.

This happens because parseopt in submodule--helper will try to parse both --quiet options as if they are foreach's options, not git-pull's.
The parsed options are removed from the command line. So when we do pull later, we execute just this

git pull origin

When calling submodule helper, adding "--" in front of "git pull" will stop parseopt for parsing options that do not really belong to submodule--helper foreach.

PARSE_OPT_KEEP_UNKNOWN is removed as a safety measure. parseopt should never see unknown options or something has gone wrong. There are also a couple usage string update while I'm looking at them.

While at it, I also add "--" to other subcommands that pass "$@" to submodule--helper. "$@" in these cases are paths and less likely to be --something-like-this.
But the point still stands, git-submodule has parsed and classified what are options, what are paths.
submodule--helper should never consider paths passed by git-submodule to be options even if they look like one.

And Git 2.23 (Q3 2019) fixes another issue: "git submodule foreach" did not protect command line options passed to the command to be run in each submodule correctly, when the "--recursive" option was in use.

See commit 30db18b (24 Jun 2019) by Morian Sonnet (momoson).
^{(Merged by Junio C Hamano -- gitster -- in commit 968eecb, 09 Jul 2019)}

submodule foreach: fix recursion of options

Calling:

git submodule foreach --recursive <subcommand> --<option>

leads to an error stating that the option --<option> is unknown to submodule--helper.
That is of course only, when <option> is not a valid option for git submodule foreach.

The reason for this is, that above call is internally translated into a call to submodule--helper:

git submodule--helper foreach --recursive \
   -- <subcommand> --<option>

This call starts by executing the subcommand with its option inside the first level submodule and continues by calling the next iteration of the submodule foreach call

git --super-prefix <submodulepath> submodule--helper \
  foreach --recursive <subcommand> --<option>

inside the first level submodule. Note that the double dash in front of the subcommand is missing.

This problem starts to arise only recently, as the PARSE_OPT_KEEP_UNKNOWN flag for the argument parsing of git submodule foreach was removed in commit a282f5a.
Hence, the unknown option is complained about now, as the argument parsing is not properly ended by the double dash.

This commit fixes the problem by adding the double dash in front of the subcommand during the recursion.

Note that, before Git 2.29 (Q4 2020), "git submodule update --quiet"^(man) did not squelch underlying "rebase" and "pull" commands.

See commit 3ad0401 (30 Sep 2020) by Theodore Dubois (tbodt).
^{(Merged by Junio C Hamano -- gitster -- in commit 300cd14, 05 Oct 2020)}

submodule update: silence underlying merge/rebase with "--quiet"

^{Signed-off-by: Theodore Dubois}

Commands such as

$ git pull --rebase --recurse-submodules --quiet  

produce non-quiet output from the merge or rebase.
Pass the --quiet option down when invoking "rebase" and "merge".

Also fix the parsing of git submodule update^(man) -v.

When e84c3cf3 ("git-submodule.sh: accept verbose flag in cmd_update to be non-quiet", 2018-08-14, Git v2.19.0-rc0 -- merge) taught "git submodule update"^(man) to take "--quiet", it apparently did not know how ${GIT_QUIET:+--quiet} works, and reviewers seem to have missed that setting the variable to "0", rather than unsetting it, still results in "--quiet" being passed to underlying commands.

@Jason is correct in a way but not entirely.

update

Update the registered submodules, i.e. clone missing submodules and checkout the commit specified in the index of the containing repository. This will make the submodules HEAD be detached unless --rebase or --merge is specified or the key submodule.$name.update is set to rebase or merge.

So, git submodule update does checkout, but it is to the commit in the index of the containing repository. It does not yet know of the new commit upstream at all. So go to your submodule, get the commit you want and commit the updated submodule state in the main repository and then do the git submodule update.

If you don't know the host branch, make this:

git submodule foreach git pull origin $(git rev-parse --abbrev-ref HEAD)

It will get a branch of the main Git repository and then for each submodule will make a pull of the same branch.