How can I debug git git-shell related problems

Question

How can I have some debug information regarding git git-shell    I had a problem  that user1 could clone a repository without problem  while user2 could clone only an empty one  I had set GIT TRACE 1  but nothing useful was told   Finally  after a long trial and error  it turned out that it was a permission problem on a file  An appropriate error message could short-circuit this problem

User · Answer

Git 2 22  Q2 2019  introduces trace2 with commit ee4512e by Jeff Hostetler   trace2  create new combined trace facility   Create a new unified tracing facility for git  The eventual intent is to replace the current trace printf  and trace performance  routines with a unified set of git trace2  routines  In addition to the usual printf-style API  trace2 provides higer-level event verbs with fixed-fields allowing structured data to be written  This makes post-processing and analysis easier for external tools  Trace2 defines 3 output targets  These are set using the environment variables  quot GIT TR2 quot    quot GIT TR2 PERF quot   and  quot GIT TR2 EVENT quot   These may be set to  quot 1 quot  or to an absolute pathname  just like the current GIT TRACE    Note  regarding environment variable name  always use GIT TRACExxx  not GIT TRxxx  So actually GIT TRACE2  GIT TRACE2 PERF or GIT TRACE2 EVENT  See the Git 2 22 rename mentioned later below  What follows is the initial work on this new tracing feature  with the old environment variable names    GIT TR2 is intended to be a replacement for GIT TRACE and logs command summary data   GIT TR2 PERF is intended as a replacement for GIT TRACE PERFORMANCE  It extends the output with columns for the command process  thread  repo  absolute and relative elapsed times  It reports events for child process start stop  thread start stop  and per-thread function nesting   GIT TR2 EVENT is a new structured format  It writes event data as a series of JSON records    Calls to trace2 functions log to any of the 3 output targets enabled without the need to call different trace printf  or trace performance  routines   See commit a4d3a28  21 Mar 2019  by Josh Steadmon  steadmon    Merged by Junio C Hamano -- gitster -- in commit 1b40314  08 May 2019   trace2  write to directory targets   When the value of a trace2 environment variable is an absolute path referring to an existing directory  write output to files  one per process  underneath the given directory  Files will be named according to the final component of the trace2 SID  followed by a counter to avoid potential collisions  This makes it more convenient to collect traces for every git invocation by unconditionally setting the relevant trace2 envvar to a constant directory name    See also commit f672dee  29 Apr 2019   and commit 81567ca  commit 08881b9  commit bad229a  commit 26c6f25  commit bce9db6  commit 800a7f9  commit a7bc01e  commit 39f4317  commit a089724  commit 1703751  15 Apr 2019  by Jeff Hostetler  jeffhostetler    Merged by Junio C Hamano -- gitster -- in commit 5b2d1c0  13 May 2019  The new documentation now includes config settings which are only read from the system and global config files  meaning repository local and worktree config files and -c command line arguments are not respected   Example    git config --global trace2 normalTarget   log normal   git version git version 2 20 1 155 g426c96fcdb  yields   cat   log normal 12 28 42 620009 common-main c 38                  version 2 20 1 155 g426c96fcdb 12 28 42 620989 common-main c 39                  start git version 12 28 42 621101 git c 432                         cmd name version  version  12 28 42 621215 git c 662                         exit elapsed 0 001227 code 0 12 28 42 621250 trace2 tr2 tgt normal c 124 atexit elapsed 0 001265 code 0  And for performance measure    git config --global trace2 perfTarget   log perf   git version git version 2 20 1 155 g426c96fcdb  yields   cat   log perf 12 28 42 620675 common-main c 38                    d0   main                       version                                                   2 20 1 155 g426c96fcdb 12 28 42 621001 common-main c 39                    d0   main                       start                 0 001173                            git version 12 28 42 621111 git c 432                           d0   main                       cmd name                                                  version  version  12 28 42 621225 git c 662                           d0   main                       exit                  0 001227                            code 0 12 28 42 621259 trace2 tr2 tgt perf c 211           d0   main                       atexit                0 001265                            code 0   As documented in Git 2 23  Q3 2019   the environment variable to use is GIT TRACE2  See commit 6114a40  26 Jun 2019  by Carlo Marcelo Arenas Bel  n  carenas   See commit 3efa1c6  12 Jun 2019  by   var Arnfj  r   Bjarmason  avar    Merged by Junio C Hamano -- gitster -- in commit e9eaaa4  09 Jul 2019  That follows the work done in Git 2 22  commit 4e0d3aa  commit e4b75d6  19 May 2019  by SZEDER G  bor  szeder    Merged by Junio C Hamano -- gitster -- in commit 463dca6  30 May 2019   trace2  rename environment variables to GIT TRACE2    For an environment variable that is supposed to be set by users  the GIT TR2  env vars are just too unclear  inconsistent  and ugly  Most of the established GIT   environment variables don t use abbreviations  and in case of the few that do  GIT DIR  GIT COMMON DIR  GIT DIFF OPTS  it s quite obvious what the abbreviations  DIR and OPTS  stand for  But what does TR stand for  Track  traditional  trailer  transaction  transfer  transformation  transition  translation  transplant  transport  traversal  tree  trigger  truncate  trust  or       The trace2 facility  as the  2  suffix in its name suggests  is supposed to eventually supercede Git s original trace facility  It s reasonable to expect that the corresponding environment variables follow suit  and after the original GIT TRACE variables they are called GIT TRACE2  there is no such thing is  GIT TR   All trace2-specific config variables are  very sensibly  in the  trace2  section  not in  tr2   OTOH  we don t gain anything at all by omitting the last three characters of  quot trace quot  from the names of these environment variables  So let s rename all GIT TR2  environment variables to GIT TRACE2   before they make their way into a stable release    Git 2 24  Q3 2019  improves the Git repository initialization  See commit 22932d9  commit 5732f2b  commit 58ebccb  06 Aug 2019  by Jeff King  peff    Merged by Junio C Hamano -- gitster -- in commit b4a1eec  09 Sep 2019   common-main  delay trace2 initialization   We initialize the trace2 system in the common main   function so that all programs  even ones that aren t builtins  will enable tracing  But trace2 startup is relatively heavy-weight  as we have to actually read on-disk config to decide whether to trace  This can cause unexpected interactions with other common-main initialization  For instance  we ll end up in the config code before calling initialize the repository    and the usual invariant that the repository is never NULL will not hold  Let s push the trace2 initialization further down in common-main  to just before we execute cmd main      Git 2 24  Q4 2019  makes also sure that output from trace2 subsystem is formatted more prettily now  See commit 742ed63  commit e344305  commit c2b890a  09 Aug 2019   commit ad43e37  commit 04f10d3  commit da4589c  08 Aug 2019   and commit 371df1b  31 Jul 2019  by Jeff Hostetler  jeffhostetler    Merged by Junio C Hamano -- gitster -- in commit 93fc876  30 Sep 2019  And  still Git 2 24 See commit 87db61a  commit 83e57b0  04 Oct 2019   and commit 2254101  commit 3d4548e  03 Oct 2019  by Josh Steadmon  steadmon    Merged by Junio C Hamano -- gitster -- in commit d0ce4d9  15 Oct 2019   trace2  discard new traces if target directory has too many files Signed-off-by  Josh Steadmon   trace2 can write files into a target directory  With heavy usage  this directory can fill up with files  causing difficulty for trace-processing systems  This patch adds a config option  trace2 maxFiles  to set a maximum number of files that trace2 will write to a target directory  The following behavior is enabled when the maxFiles is set to a positive integer   When trace2 would write a file to a target directory  first check whether or not the traces should be discarded  Traces should be discarded if   there is a sentinel file declaring that there are too many files OR  the number of files exceeds trace2 maxFiles  In the latter case  we create a sentinel file named git-trace2-discard to speed up future checks     The assumption is that a separate trace-processing system is dealing with the generated traces  once it processes and removes the sentinel file  it should be safe to generate new trace files again  The default value for trace2 maxFiles is zero  which disables the file count check  The config can also be overridden with a new environment variable  GIT TRACE2 MAX FILES    And Git 2 24  Q4 2019  teach trace2 about git push stages  See commit 25e4b80  commit 5fc3118  02 Oct 2019  by Josh Steadmon  steadmon    Merged by Junio C Hamano -- gitster -- in commit 3b9ec27  15 Oct 2019   push  add trace2 instrumentation Signed-off-by  Josh Steadmon   Add trace2 regions in transport c and builtin push c to better track time spent in various phases of pushing   Listing refs Checking submodules Pushing submodules Pushing refs    With Git 2 25  Q1 2020   some of the Documentation technical is moved to header   h files  See commit 6c51cb5  commit d95a77d  commit bbcfa30  commit f1ecbe0  commit 4c4066d  commit 7db0305  commit f3b9055  commit 971b1f2  commit 13aa9c8  commit c0be43f  commit 19ef3dd  commit 301d595  commit 3a1b341  commit 126c1cc  commit d27eb35  commit 405c6b1  commit d3d7172  commit 3f1480b  commit 266f03e  commit 13c4d7e  17 Nov 2019  by Heba Waly  HebaWaly    Merged by Junio C Hamano -- gitster -- in commit 26c816a  16 Dec 2019   trace2  move doc to trace2 h Signed-off-by  Heba Waly   Move the functions documentation from Documentation technical api-trace2 txt to trace2 h as it s easier for the developers to find the usage information beside the code instead of looking for it in another doc file  Only the functions documentation section is removed from Documentation technical api-trace2 txt as the file is full of details that seemed more appropriate to be in a separate doc file as it is  with a link to the doc file added in the trace2 h  Also the functions doc is removed to avoid having redundandt info which will be hard to keep syncronized with the documentation in the header file    although that reorganization had a side effect on another command  explained and fixed with Git 2 25 2  March 2020  in commit cc4f2eb  14 Feb 2020  by Jeff King  peff    Merged by Junio C Hamano -- gitster -- in commit 1235384  17 Feb 2020    With Git 2 27  Q2 2020   Trace2 enhancement to allow logging of the environment variables  See commit 3d3adaa  20 Mar 2020  by Josh Steadmon  steadmon    Merged by Junio C Hamano -- gitster -- in commit 810dc64  22 Apr 2020   trace2  teach Git to log environment variables Signed-off-by  Josh Steadmon Acked-by  Jeff Hostetler   Via trace2  Git can already log interesting config parameters  see the trace2 cmd list config   function   However  this can grant an incomplete picture because many config parameters also allow overrides via environment variables  To allow for more complete logs  we add a new trace2 cmd list env vars   function and supporting implementation  modeled after the pre-existing config param logging implementation    With Git 2 27  Q2 2020   teach codepaths that show progress meter to also use the start progress   and the stop progress   calls as a  quot region quot  to be traced  See commit 98a1364  12 May 2020  by Emily Shaffer  nasamuffin    Merged by Junio C Hamano -- gitster -- in commit d98abce  14 May 2020   trace2  log progress time and throughput Signed-off-by  Emily Shaffer   Rather than teaching only one operation  like  git fetch   how to write down throughput to traces  we can learn about a wide range of user operations that may seem slow by adding tooling to the progress library itself  Operations which display progress are likely to be slow-running and the kind of thing we want to monitor for performance anyways  By showing object counts and data transfer size  we should be able to make some derived measurements to ensure operations are scaling the way we expect   And  With Git 2 27  Q2 2020   last-minute fix for our recent change to allow use of progress API as a traceable region  See commit 3af029c  15 May 2020  by Derrick Stolee  derrickstolee    Merged by Junio C Hamano -- gitster -- in commit 85d6e28  20 May 2020   progress  call trace2 region leave   only after calling  enter   Signed-off-by  Derrick Stolee   A user of progress API calls start progress   conditionally and depends on the display progress   and stop progress   functions to become no-op when start progress   hasn t been called  As we added a call to trace2 region enter   to start progress    the calls to other trace2 API calls from the progress API functions must make sure that these trace2 calls are skipped when start progress   hasn t been called on the progress struct  Specifically  do not call trace2 region leave   from stop progress   when we haven t called start progress    which would have called the matching trace2 region enter      That last part is more robust with Git 2 29  Q4 2020   See commit ac900fd  10 Aug 2020  by Martin   gren  none    Merged by Junio C Hamano -- gitster -- in commit e6ec620  17 Aug 2020   progress  don t dereference before checking for NULL Signed-off-by  Martin   gren   In stop progress    we re careful to check that p progress is non-NULL before we dereference it  but by then we have already dereferenced it when calling finish if sparse  p progress   And  for what it s worth  we ll go on to blindly dereference it again inside stop progress msg    We could return early if we get a NULL-pointer  but let s go one step further and BUG instead  The progress API handles NULL just fine  but that s the NULL-ness of  p progress  e g   when running with --no-progress  If p progress is NULL  chances are that s a mistake  For symmetry  let s do the same check in stop progress msg    too    With Git 2 29  Q4 2020   there is even more trace  this time in a Git development environment  See commit 4441f42  09 Sep 2020  by Han-Wen Nienhuys  hanwen    Merged by Junio C Hamano -- gitster -- in commit c9a04f0  22 Sep 2020   refs  add GIT TRACE REFS debugging mechanism Signed-off-by  Han-Wen Nienhuys   When set in the environment  GIT TRACE REFS makes git print operations and results as they flow through the ref storage backend  This helps debug discrepancies between different ref backends  Example    GIT TRACE REFS  quot 1 quot    git branch 15 42 09 769631 refs debug c 26         ref store for  git 15 42 09 769681 refs debug c 249        read raw ref  HEAD  0000000000000000000000000000000000000000    gt  refs heads ref-debug  type 1  0 15 42 09 769695 refs debug c 249        read raw ref  refs heads ref-debug  3a238e539bcdfe3f9eb5010fd218640c1b499f7a    gt  refs heads ref-debug  type 0  0 15 42 09 770282 refs debug c 233        ref iterator begin  refs heads   0x1  15 42 09 770290 refs debug c 189        iterator advance  refs heads b4  0  15 42 09 770295 refs debug c 189        iterator advance  refs heads branch3  0    git now includes in its man page   GIT TRACE REFS Enables trace messages for operations on the ref database  See GIT TRACE for available trace output options    With Git 2 30  Q1 2021   like die   and error    a call to warning   will also trigger a trace2 event  See commit 0ee10fd  23 Nov 2020  by Jonathan Tan  jhowtan    Merged by Junio C Hamano -- gitster -- in commit 2aeafbc  08 Dec 2020

User · Answer

Git 2 9 x 2 10  Q3 2016  adds another debug option  GIT TRACE CURL   See commit 73e57aa  commit 74c682d  23 May 2016  by Elia Pinto  devzero2000   Helped-by  Torsten B  gershausen  tboegi   Ramsay Jones   Junio C Hamano  gitster   Eric Sunshine  sunshineco   and Jeff King  peff    Merged by Junio C Hamano -- gitster -- in commit 2f84df2  06 Jul 2016        http c  implement the GIT TRACE CURL environment variable      Implement the GIT TRACE CURL environment variable to allow a greater degree of detail of GIT CURL VERBOSE  in particular the complete transport header and all the data payload exchanged    It might be useful if a particular situation could require a more thorough debugging analysis     The documentation will state   GIT TRACE CURL      Enables a curl full trace dump of all incoming and outgoing data      including descriptive information  of the git transport protocol    This is similar to doing curl --trace-ascii on the command line       This option overrides setting the GIT CURL VERBOSE environment variable      You can see that new option used in this answer  but also in the Git 2 11  Q4 2016  tests   See commit 14e2411  commit 81590bf  commit 4527aa1  commit 4eee6c6  07 Sep 2016  by Elia Pinto  devzero2000    Merged by Junio C Hamano -- gitster -- in commit 930b67e  12 Sep 2016        Use the new GIT TRACE CURL environment variable instead   of the deprecated GIT CURL VERBOSE    GIT TRACE CURL true git clone --quiet  HTTPD URL smart repo git

User · Answer

For older git versions  1 8 and before   I could find no suitable way to enable SSH debugging in an older git and ssh versions   I looked for environment variables using ltrace -e getenv     and couldn t find any combination of GIT TRACE or SSH DEBUG variables that would work   Instead here s a recipe to temporarily inject  ssh -v  into the git- ssh sequence     echo   usr bin ssh -v        gt  tmp ssh   chmod  x  tmp ssh   PATH  tmp   PATH  git clone       rm -f  tmp ssh   Here s output from git version 1 8 3 with ssh version OpenSSH 5 3p1  OpenSSL 1 0 1e-fips 11 Feb 2013 cloning a github repo      echo   usr bin ssh -v        gt  tmp ssh  chmod  x  tmp ssh  PATH  tmp   PATH       GIT TRACE 1 git clone https   github com qneill cliff git       rm -f  tmp ssh  2 gt  amp 1   tee log trace  built-in  git  clone   https   github com qneill cliff git  trace  run command   git-remote-https   origin   https   github com qneill cliff git  Cloning into  cliff     OpenSSH 5 3p1  OpenSSL 1 0 1e-fips 11 Feb 2013 debug1  Reading configuration data  home q neill  ssh config debug1  Reading configuration data  etc ssh ssh config debug1  Applying options for   debug1  Connecting to github com         Transferred  sent 4120  received 724232 bytes  in 0 2 seconds Bytes per second  sent 21590 6  received 3795287 2 debug1  Exit status 0 trace  run command   rev-list   --objects   --stdin   --not   --all  trace  exec   git   rev-list   --objects   --stdin   --not   --all  trace  built-in  git  rev-list   --objects   --stdin   --not   --all

User · Answer

try this one   GIT TRACE 1 git pull origin master

User · Answer

Have you tried adding the verbose  -v  operator when you clone   git clone -v git   git kernel org pub scm     linux-2 6 my2 6

User · Answer

For even more verbose output use following   GIT CURL VERBOSE 1 GIT TRACE 1 git pull origin master

User · Answer

Debugging  Git has a fairly complete set of traces embedded which you can use to debug your git problems   To turn them on  you can define the following variables    GIT TRACE for general traces  GIT TRACE PACK ACCESS for tracing of packfile access  GIT TRACE PACKET for packet-level tracing for network operations  GIT TRACE PERFORMANCE for logging the performance data  GIT TRACE SETUP for information about discovering the repository and environment it   s interacting with  GIT MERGE VERBOSITY for debugging recursive merge strategy  values  0-5   GIT CURL VERBOSE for logging all curl messages  equivalent to curl -v   GIT TRACE SHALLOW for debugging fetching cloning of shallow repositories    Possible values can include    true  1 or 2 to write to stderr  an absolute path starting with   to trace output to the specified file    For more details  see  Git Internals - Environment Variables    SSH  For SSH issues  try the following commands   echo  ssh -vvv        gt  ssh  amp  amp  chmod  x ssh GIT SSH   PWD ssh  git pull origin master   or use ssh to validate your credentials  e g   ssh -vvvT git github com   or over HTTPS port   ssh -vvvT -p 443 git ssh github com   Note  Reduce number of -v to reduce the verbosity level     Examples    GIT TRACE 1 git status 20 11 39 565701 git c 350               trace  built-in  git  status     GIT TRACE PERFORMANCE  PWD gc log git gc Counting objects  143760  done        head gc log  20 12 37 214410 trace c 420             performance  0 090286000 s  git command   git   pack-refs   --all   --prune  20 12 37 378101 trace c 420             performance  0 156971000 s  git command   git   reflog   expire   --all         GIT TRACE PACKET true git pull origin master 20 16 53 062183 pkt-line c 80           packet         fetch lt  93eb028c6b2f8b1d694d1173a4ddf32b48e371ce HEAD 0multi ack thin-pack side-band side-band-64k ofs-delta shallow no-progress include-tag multi ack detailed symref HEAD refs heads master agent git 2 2 6 5 update-ref-initial-update-1494-g76b680d

User · Answer

If its over SSH  you can use the following   For a higher debug level for type -vv or -vvv for debug level 2 and 3 respectively     Debug level 1 GIT SSH COMMAND  ssh -v  git clone  lt repositoryurl gt     Debug level 2 GIT SSH COMMAND  ssh -vv  git clone  lt repositoryurl gt     Debug level 3 GIT SSH COMMAND  ssh -vvv  git clone  lt repositoryurl gt    This is mainly useful to handle public and private key problems with the server  You can use this command for any git command  not only  git clone

[git] How can I debug git/git-shell related problems?

`trace2`: create new combined trace facility

`trace2`: write to directory targets

`trace2`: rename environment variables to GIT_TRACE2*

common-main: delay trace2 initialization

`trace2`: discard new traces if target directory has too many files

`push`: add trace2 instrumentation

`trace2`: move doc to `trace2.h`

`trace2`: teach Git to log environment variables

`trace2`: log progress time and throughput

`progress`: call `trace2_region_leave()` only after calling `_enter()`

`progress`: don't dereference before checking for `NULL`

`refs`: add `GIT_TRACE_REFS` debugging mechanism

`GIT_TRACE_REFS`

Examples related to git

Examples related to debugging

Examples related to trace

[git] How can I debug git/git-shell related problems?

trace2: create new combined trace facility

trace2: write to directory targets

trace2: rename environment variables to GIT_TRACE2*

common-main: delay trace2 initialization

trace2: discard new traces if target directory has too many files

push: add trace2 instrumentation

trace2: move doc to trace2.h

trace2: teach Git to log environment variables

trace2: log progress time and throughput

progress: call trace2_region_leave() only after calling _enter()

progress: don't dereference before checking for NULL

refs: add GIT_TRACE_REFS debugging mechanism

GIT_TRACE_REFS