Associative arrays in Shell scripts

Question

We required a script that simulates Associative arrays or Map like data structure for Shell Scripting  any body

User · Answer

I modified Vadim s solution with the following                                                                          Bash v3 does not support associative arrays   and we cannot use ksh since all generic scripts are on bash   Usage  map put map name key value   function map put       alias    1  2    3       map get map name key    return value   function map get       if type -p    1  2          then             alias    1  2    awk -F        print  2          fi      map keys map name     return map keys   function map keys       alias -p   grep  1   cut -d    -f1   awk -F  1    print  2         The change is to map get in order to prevent it from returning errors if you request a key that doesn t exist  though the side-effect is that it will also silently ignore missing maps  but it suited my use-case better since I just wanted to check for a key in order to skip items in a loop

User · Answer

For Bash 3  there is a particular case that has a nice and simple solution   If you don t want to handle a lot of variables  or keys are simply invalid variable identifiers  and your array is guaranteed to have less than 256 items  you can abuse function return values  This solution does not require any subshell as the value is readily available as a variable  nor any iteration so that performance screams  Also it s very readable  almost like the Bash 4 version   Here s the most basic version   hash index         case  1 in          foo   return 0            bar   return 1            baz   return 2       esac    hash vals   foo val              bar val              baz val     hash index  foo  echo   hash vals        Remember  use single quotes in case  else it s subject to globbing  Really useful for static frozen hashes from the start  but one could write an index generator from a hash keys    array   Watch out  it defaults to the first one  so you may want to set aside zeroth element   hash index         case  1 in          foo   return 1            bar   return 2            baz   return 3       esac    hash vals                  sort of like returning null nil for a non existent key             foo val              bar val              baz val     hash index  foo     echo   hash vals         It can t get more readable than this   Caveat  the length is now incorrect   Alternatively  if you want to keep zero-based indexing  you can reserve another index value and guard against a non-existent key  but it s less readable   hash index         case  1 in          foo   return 0            bar   return 1            baz   return 2                return 255       esac    hash vals   foo val              bar val              baz val     hash index  foo        -ne 255     amp  amp  echo   hash vals        Or  to keep the length correct  offset index by one   hash index         case  1 in          foo   return 1            bar   return 2            baz   return 3       esac    hash vals   foo val              bar val              baz val     hash index  foo     echo   hash vals       - 1

User · Answer

Another option  if portability is not your main concern  is to use associative arrays that are built in to the shell  This should work in bash 4 0  available now on most major distros  though not on OS X unless you install it yourself   ksh  and zsh   declare -A newmap newmap name   Irfan Zulfiqar  newmap designation  SSE newmap company   My Own Company   echo   newmap company   echo   newmap name     Depending on the shell  you may need to do a typeset -A newmap instead of declare -A newmap  or in some it may not be necessary at all

User · Answer

Shell have no built-in map like data structure  I use raw string to describe items like that   ARRAY        item A attr1 attr2 attr3       item B attr1 attr2 attr3                when extract items and its attributes   for item in    ARRAY      do     item name   echo    item   awk -F       print  1        item attr1   echo    item   awk -F       print  2        item attr2   echo    item   awk -F       print  3         echo    item name       echo    item attr1       echo    item attr2   done   This seems like not clever than other people s answer  but easy to understand for new people to shell

User · Answer

Now answering this question   Following scripts simulates associative arrays in shell scripts  Its simple and very easy to understand   Map is nothing but a never ending string that has keyValuePair saved as  --name Irfan --designation SSE --company My SP Own SP Company  spaces are replaced with   SP   for values  put         if           3    then exit 1  fi     mapName  1  key  2  value  echo  3   sed -e  s    SP  g       eval map       mapName        map   echo   map    sed -e  s -- key        g   -- key  value      eval  mapName     map       get         mapName  1  key  2  valueFound  false       eval map    mapName      for keyValuePair in   map       do         case   keyValuePair  in             -- key    value  echo   keyValuePair    sed -e  s                                   valueFound  true          esac         if     valueFound      true     then break  fi     done     value  echo  value   sed -e  s  SP    g      put  newMap   name   Irfan Zulfiqar  put  newMap   designation   SSE  put  newMap   company   My Own Company   get  newMap   company  echo  value  get  newMap   name  echo  value   edit  Just added another method to fetch all keys   getKeySet         if           1         then          exit 1       fi      mapName  1        eval map       mapName         keySet              echo  map               sed -e  s         g  -e  s         --  1 g

User · Answer

Bash4 supports this natively   Do not use grep or eval  they are the ugliest of hacks   For a verbose  detailed answer with example code see  https   stackoverflow com questions 3467959

User · Answer

To add to Irfan s answer  here is a shorter and faster version of get   since it requires no iteration over the map contents   get         mapName  1  key  2      map    mapName      value    echo  map  sed -e  s   --  key               1   -e  s  SP    g

User · Answer

You can use dynamic variable names and let the variables names work like the keys of a hashmap      For example  if you have an input file with two columns  name  credit  as the example bellow  and you want to sum the income of each user   Mary 100 John 200 Mary 50 John 300 Paul 100 Paul 400 David 100   The command bellow will sum everything  using dynamic variables as keys  in the form of map   person    while read -r person money    map  person   money    done  lt   lt  cat INCOME REPORT log    To read the results   set   grep map   The output will be   map David 100 map John 500 map Mary 150 map Paul 500   Elaborating on these techniques  I m developing on GitHub a function that works just like a HashMap Object  shell map     In order to create  HashMap instances  the shell map function is able create copies of itself under different names  Each new function copy will have a different  FUNCNAME variable   FUNCNAME then is used to create a namespace for each Map instance     The map keys are global variables  in the form  FUNCNAME DATA  KEY  where  KEY is the key added to the Map  These variables are dynamic variables   Bellow I ll put a simplified version of it so you can use as example       bin bash  shell map          local METHOD   1       case  METHOD in     new          local NEW MAP   2             loads shell map function declaration         test -n    declare -f shell map      return            declares in the Global Scope a copy of shell map  under a new name          eval      shell map  2              put          local KEY   2            local VALUE   3             declares a variable in the global scope         eval   FUNCNAME  DATA   KEY    VALUE             get          local KEY   2          local VALUE    FUNCNAME  DATA   KEY           echo     VALUE              keys          declare   grep -Po     lt    FUNCNAME  DATA   w                      name          echo  FUNCNAME            contains key          local KEY   2          compgen -v   FUNCNAME  DATA   KEY   gt   dev null  amp  amp  return 0    return 1            clear all          while read var  do               unset  var         done  lt   lt  compgen -v   FUNCNAME  DATA              remove          local KEY   2          unset   FUNCNAME  DATA   KEY             size          compgen -v   FUNCNAME  DATA   KEY    wc -l                       echo  unsupported operation   1            return 1            esac     Usage   shell map new credit credit put Mary 100 credit put John 200 for customer in  credit keys   do      value  credit get  customer             echo  customer  customer has  value  done credit contains key  Mary   amp  amp  echo  Mary has credit

User · Answer

Yet another non-bash-4  i e   bash 3  Mac-compatible  way   val of key         case  1 in          A1   echo  aaa             B2   echo  bbb             C3   echo  ccc               echo  zzz        esac    for x in  A1   B2   C3   D4   do     y   val of key   x       echo   x   gt   y  done   Prints   A1   gt  aaa B2   gt  bbb C3   gt  ccc D4   gt  zzz   The function with the case acts like an associative array  Unfortunately it cannot use return  so it has to echo its output  but this is not a problem  unless you are a purist that shuns forking subshells

User · Answer

Another non-bash 4 way      bin bash    A pretend Python dictionary with bash 3  ARRAY    cow moo           dinosaur roar           bird chirp           bash rock     for animal in    ARRAY        do     KEY   animal          VALUE   animal         printf   s likes to  s  n    KEY    VALUE  done  echo -e    ARRAY 1       is an extinct animal which likes to   ARRAY 1      n    You could throw an if statement for searching in there as well  if     var     blah      or whatever

User · Answer

several years ago I wrote script library for bash which supported associative arrays among other features  logging  configuration files  extended support for command line argument  generate help  unit testing  etc   The library contains a wrapper for associative arrays and automatically switches to appropriate model  internal for bash4 and emulate for previous versions   It was called shell-framework and hosted at origo ethz ch but today the resource is closed  If someone still needs it I can share it with you

User · Answer

Bash v3 does not support associative arrays   and we cannot use ksh since all generic scripts are on bash   Usage  map put map name key value   function map put       alias    1  2    3       map get map name key    return value   function map get       alias    1  2    awk -F       print  2          map keys map name     return map keys   function map keys       alias -p   grep  1   cut -d    -f1   awk -F  1    print  2         Example   mapName   basename  0  map  map put  mapName  name   Irfan Zulfiqar  map put  mapName  designation   SSE   for key in   map keys  mapName  do     echo   key     map get  mapName  key  done

User · Answer

What a pity I did not see the question before - I ve wrote library shell-framework which contains among others the maps Associative arrays   The last version of it can be found here    Example      bin bash   include map library shF PATH TO LIB   usr lib shell-framework  source    shF PATH TO LIB  map    simple example get put putMapValue  mapName   mapKey1   map Value 2  echo  mapName mapKey1     getMapValue  mapName   mapKey1      redefine old value to new putMapValue  mapName   mapKey1   map Value 1  echo  after change mapName mapKey1     getMapValue  mapName   mapKey1      add two new pairs key values and print all keys putMapValue  mapName   mapKey2   map Value 2  putMapValue  mapName   mapKey3   map Value 3  echo -e  mapName keys are  n  getMapKeys  mapName      create new map putMapValue  subMapName   subMapKey1   sub map Value 1  putMapValue  subMapName   subMapKey2   sub map Value 2    and put it in mapName under key  mapKey4  putMapValue  mapName   mapKey4   subMapName    check if under two key were placed maps echo  is map mapName mapKey3   -   if isMap    getMapValue  mapName   mapKey3      then echo Yes  else echo No  fi   echo  is map mapName mapKey4   -   if isMap    getMapValue  mapName   mapKey4      then echo Yes  else echo No  fi     print map with sub maps printf   s n     mapToString  mapName

User · Answer

Late reply  but consider addressing the problem in this way  using the bash builtin read as illustrated within the code snippet from a ufw firewall script that follows  This approach has the advantage of using as many delimited field sets  not just 2  as are desired   We have used the   delimiter because port range specifiers may require a colon  ie 6001 6010      usr bin env bash  readonly connections                                       192 168 1 4 24 tcp 22                               192 168 1 4 24 tcp 53                               192 168 1 4 24 tcp 80                               192 168 1 4 24 tcp 139                               192 168 1 4 24 tcp 443                               192 168 1 4 24 tcp 445                               192 168 1 4 24 tcp 631                               192 168 1 4 24 tcp 5901                               192 168 1 4 24 tcp 6566     function set connections        local range proto port     for fields in   connections         do             IFS      read -r range proto port  lt  lt  lt    fields              ufw allow from   range  proto   proto  to any port   port      done    set connections

User · Answer

I think that you need to step back and think about what a map  or associative array  really is  All it is is a way to store a value for a given key  and get that value back quickly and efficiently  You may also want to be able to iterate over the keys to retrieve every key value pair  or delete keys and their associated values   Now  think about a data structure you use all the time in shell scripting  and even just in the shell without writing a script  that has these properties  Stumped  It s the filesystem   Really  all you need to have an associative array in shell programming is a temp directory  mktemp -d is your associative array constructor   prefix   basename --   0   map   mktemp -dt   prefix   echo  gt   map  key somevalue value   cat   map  key    If you don t feel like using echo and cat  you can always write some little wrappers  these ones are modelled off of Irfan s  though they just output the value rather than setting arbitrary variables like  value      bin sh  prefix   basename --   0   mapdir   mktemp -dt   prefix   trap  rm -r   mapdir   EXIT  put                 3    amp  amp  exit 1   mapname  1  key  2  value  3     -d    mapdir    mapname        mkdir    mapdir    mapname     echo  value  gt    mapdir    mapname    key      get                 2    amp  amp  exit 1   mapname  1  key  2   cat    mapdir    mapname    key      put  newMap   name   Irfan Zulfiqar  put  newMap   designation   SSE  put  newMap   company   My Own Company   value   get  newMap   company   echo  value  value   get  newMap   name   echo  value   edit  This approach is actually quite a bit faster than the linear search using sed suggested by the questioner  as well as more robust  it allows keys and values to contain -     space  qnd   SP     The fact that it uses the filesystem does not make it slow  these files are actually never guaranteed to be written to the disk unless you call sync  for temporary files like this with a short lifetime  it s not unlikely that many of them will never be written to disk   I did a few benchmarks of Irfan s code  Jerry s modification of Irfan s code  and my code  using the following driver program      bin sh  mapimpl  1 numkeys  2 numvals  3        mapimpl  sh        lt - fix broken stack overflow syntax highlighting  for    i   0    i  lt   numkeys   i    1    do     for    j   0    j  lt   numvals   j    1        do         put  newMap   key i   value j          get  newMap   key i      done done   The results          time   driver sh irfan 10 5      real    0m0 975s     user    0m0 280s     sys     0m0 691s        time   driver sh brian 10 5      real    0m0 226s     user    0m0 057s     sys     0m0 123s        time   driver sh jerry 10 5      real    0m0 706s     user    0m0 228s     sys     0m0 530s        time   driver sh irfan 100 5      real    0m10 633s     user    0m4 366s     sys     0m7 127s        time   driver sh brian 100 5      real    0m1 682s     user    0m0 546s     sys     0m1 082s        time   driver sh jerry 100 5      real    0m9 315s     user    0m4 565s     sys     0m5 446s        time   driver sh irfan 10 500      real    1m46 197s     user    0m44 869s     sys     1m12 282s        time   driver sh brian 10 500      real    0m16 003s     user    0m5 135s     sys     0m10 396s        time   driver sh jerry 10 500      real    1m24 414s     user    0m39 696s     sys     0m54 834s        time   driver sh irfan 1000 5      real    4m25 145s     user    3m17 286s     sys     1m21 490s        time   driver sh brian 1000 5      real    0m19 442s     user    0m5 287s     sys     0m10 751s        time   driver sh jerry 1000 5      real    5m29 136s     user    4m48 926s     sys     0m59 336s

User · Answer

Adding another option  if jq is available   export NAMES        Mary     100        John     200        Mary     50        John     300        Paul     100        Paul     400        David     100      export NAME David echo  NAMES   jq --arg v   NAME      v     tr -d

User · Answer

I ve found it true  as already mentioned  that the best performing method is to write out key vals to a file  and then use grep awk to retrieve them  It sounds like all sorts of unnecessary IO  but disk cache kicks in and makes it extremely efficient -- much faster than trying to store them in memory using one of the above methods  as the benchmarks show    Here s a quick  clean method I like   hinit         rm -f  tmp hashmap  1    hput         echo   2  3   gt  gt   tmp hashmap  1    hget         grep    2    tmp hashmap  1   awk    print  2        hinit capitols hput capitols France Paris hput capitols Netherlands Amsterdam hput capitols Spain Madrid  echo  hget capitols France  and  hget capitols Netherlands  and  hget capitols Spain    If you wanted to enforce single-value per key  you could also do a little grep sed action in hput

[bash] Associative arrays in Shell scripts

Examples related to bash

Examples related to shell

Examples related to hashtable

Examples related to associative-array