How to atomically delete keys matching a pattern using Redis

Question

In my Redis DB I have a number of prefix  lt numeric id gt  hashes   Sometimes I want to purge them all atomically  How do I do this without using some distributed locking mechanism

User · Answer

I am using below command in redis 3 2 8  redis-cli KEYS  YOUR KEY PREFIX    xargs redis-cli DEL   You can get more help related to keys pattern search from here  - https   redis io commands keys  Use your convenient glob-style pattern as per your requirement like  YOUR KEY PREFIX  or YOUR KEY PREFIX   or any other   And if any of you have integrated Redis PHP library than below function will help you   flushRedisMultipleHashKeyUsingPattern   YOUR KEY PATTERN       function call  function flushRedisMultipleHashKeyUsingPattern  pattern                           if  pattern                       return true                              redisObj    this- gt redis               getHashes    redisObj- gt keys  pattern               if  empty  getHashes                     response   call user func array array  amp  redisObj   del     getHashes     setting all keys as parameter of  del  function  Using this we can achieve  redisObj- gt del  key1   key2                             Thank you

User · Answer

Ad of now  you can use a redis client and perform first SCAN  supports pattern matching  and then DEL each key individually   However  there is an issue on official redis github to create a patter-matching-del here  go show it some love if you find it useful

User · Answer

TODO You think it s command not make sense bu some times Redis command like DEL not working correct and comes to the rescue of this redis-cli KEYS  quot   quot    xargs -i redis-cli EXPIRE    1 it s life hack

User · Answer

Execute in bash   redis-cli KEYS  prefix      xargs redis-cli DEL   UPDATE  Ok  i understood  What about this way  store current additional incremental prefix and add it to all your keys  For example   You have values like this   prefix prefix actuall   2 prefix 2 1   4 prefix 2 2   10   When you need to purge data  you change prefix actuall first  for example set prefix prefix actuall   3   so your application will write new data to keys prefix 3 1 and prefix 3 2  Then you can safely take old values from prefix 2 1 and prefix 2 2 and purge old keys

User · Answer

I support all answers related to having some tool or execute Lua expression   One more option from my side   In our production and pre-production databases there are thousands of keys  Time to time we need to delete some keys  by some mask   modify by some criteria etc  Of course  there is no way to do it manually from CLI  especially having sharding  512 logical dbs in each physical    For this purpose I write java client tool that does all this work  In case of keys deletion the utility can be very simple  only one class there   public class DataCleaner        public static void main String args              String keyPattern   args 0           String host   args 1           int port   Integer valueOf args 2            int dbIndex   Integer valueOf args 3             Jedis jedis   new Jedis host  port            int deletedKeysNumber   0          if dbIndex  gt   0               deletedKeysNumber    deleteDataFromDB jedis  keyPattern  dbIndex             else               int dbSize   Integer valueOf jedis configGet  databases   get 1                for int i   0  i  lt  dbSize  i                     deletedKeysNumber    deleteDataFromDB jedis  keyPattern  i                                    if deletedKeysNumber    0                System out println  There is no keys with key pattern      keyPattern     was found in database with host      host                        private static int deleteDataFromDB Jedis jedis  String keyPattern  int dbIndex            jedis select dbIndex           Set lt String gt  keys   jedis keys keyPattern           for String key   keys               jedis del key               System out println  The key      key     has been deleted from database index      dbIndex                      return keys size

User · Answer

Other answers may not work if your key contains special chars - Guide CLASSMETADATA  1  for instance  Wrapping each key into quotes will ensure they get properly deleted   redis-cli --scan --pattern sf     awk   print  1     sed  s      s         xargs redis-cli del

User · Answer

Spring RedisTemplate itself provides the functionality  RedissonClient in the latest version has deprecated the  deleteByPattern  functionality   Set lt String gt  keys   redisTemplate keys  geotag      redisTemplate delete keys

User · Answer

Disclaimer  the following solution doesn t provide atomicity   Starting with v2 8 you really want to use the SCAN command instead of KEYS 1   The following Bash script demonstrates deletion of keys by pattern      bin bash  if      -ne 3    then   echo  Delete keys from Redis matching a pattern using SCAN  amp  DEL    echo  Usage   0  lt host gt   lt port gt   lt pattern gt     exit 1 fi  cursor -1 keys     while    cursor -ne 0    do   if    cursor -eq -1     then     cursor 0   fi    reply  redis-cli -h  1 -p  2 SCAN  cursor MATCH  3    cursor  expr   reply        0-9   0-9         keys   reply   0-9   0-9      redis-cli -h  1 -p  2 DEL  keys done    1  KEYS is a dangerous command that can potentially result in a DoS  The following is a quote from its documentation page      Warning  consider KEYS as a command that should only be used in production environments with extreme care  It may ruin performance when it is executed against large databases  This command is intended for debugging and special operations  such as changing your keyspace layout  Don t use KEYS in your regular application code  If you re looking for a way to find keys in a subset of your keyspace  consider using sets    UPDATE  a one liner for the same basic effect -    redis-cli --scan --pattern    foo bar      xargs -L 100 redis-cli DEL

User · Answer

Starting with redis 2 6 0  you can run lua scripts  which execute atomically  I have never written one  but I think it would look something like this  EVAL  return redis call  del   unpack redis call  keys   ARGV 1      0 prefix  YOUR PREFIX e g delete me         Warning  As the Redis document says  because of performance maters  keys   command should not use for regular operations in production  this   command is intended for debugging and special operations  read   more   See the EVAL documentation

User · Answer

A version using SCAN rather than KEYS  as recommended for production servers  and --pipe rather than xargs    I prefer pipe over xargs because it s more efficient and works when your keys contain quotes or other special characters that your shell with try and interpret  The regex substitution in this example wraps the key in double quotes  and escapes any double quotes inside   export REDIS HOST your hostname com redis-cli -h   REDIS HOST  --scan --pattern  YourPattern    gt   tmp keys time cat  tmp keys   perl -pe  s       g s   DEL    s           redis-cli -h   REDIS HOST  --pipe

User · Answer

For those who were having trouble parsing other answers    eval  for   k in ipairs redis call  keys   key   pattern    do redis call  del  k  end  0   Replace key   pattern with your own pattern and enter this into redis-cli and you are good to go   Credit lisco from  http   redis io commands del

User · Answer

poor man s atomic mass-delete   maybe you could set them all to EXPIREAT the same second - like a few minutes in the future - and then wait until that time and see them all  self-destruct  at the same time   but I am not really sure how atomic that would be

User · Answer

Please use this command and try    redis-cli --raw keys   PATTERN    xargs redis-cli del

User · Answer

I tried most of methods mentioned above but they didn t work for me  after some searches I found these points    if you have more than one db on redis you should determine the database using -n  number  if you have a few keys use del but if there are thousands or millions of keys it s better to use unlink because unlink is non-blocking while del is blocking  for more information visit this page unlink vs del also keys are like del and is blocking   so I used this code to delete keys by pattern    redis-cli -n 2 --scan --pattern   your pattern     xargs redis-cli -n 2 unlink

User · Answer

If you use windows environment please follow this steps and it will definitely works   Download GOW from here - https   github com bmatzelle gow wiki  because xargs command doesn t works in windows   Download redis-cli for Windows  detailed explanation is here - https   medium com  binary10111010 redis-cli-installation-on-windows-684fb6b6ac6b   Run cmd and open directory where redis-cli stores  example  D  Redis Redis-x64-3 2 100   if you want to delete all keys which start with  quot Global ProviderInfo quot  execute this query  it s require to change bold parameters  host  port  password  key  and write yours  because of this is only example   redis-cli -h redis test com -p 6379 -a redispassword --raw keys  quot Global ProviderInfo  quot    xargs redis-cli -h redis test com -p 6379 -a redispassword del

User · Answer

Below command worked for me   redis-cli -h redis host url KEYS   abcd     xargs redis-cli -h redis host url DEL

User · Answer

I think what might help you is the MULTI EXEC DISCARD  While not 100  equivalent of transactions  you should be able to isolate the deletes from other updates

User · Answer

Here s a completely working and atomic version of a wildcard delete implemented in Lua  It ll run much faster than the xargs version due to much less network back-and-forth  and it s completely atomic  blocking any other requests against redis until it finishes  If you want to atomically delete keys on Redis 2 6 0 or greater  this is definitely the way to go   redis-cli -n  some db  -h  some host name  EVAL  return redis call  DEL   unpack redis call  KEYS   ARGV 1             0 prefix    This is a working version of  mcdizzle s idea in his answer to this question  Credit for the idea 100  goes to him   EDIT  Per Kikito s comment below  if you have more keys to delete than free memory in your Redis server  you ll run into the  too many elements to unpack  error  In that case  do   for   k in ipairs redis call  keys   ARGV 1    do      redis call  del   k   end   As Kikito suggested

User · Answer

mcdizle s solution is not working it works only for one entry   This one works for all keys with same prefix  EVAL  for i  name in ipairs redis call  KEYS   ARGV 1    do redis call  DEL   name   end  0 prefix    Note  You should replace  prefix  with your key prefix

User · Answer

itamar s answer is great  but the parsing of the reply wasn t working for me  esp  in the case where there are no keys found in a given scan  A possibly simpler solution  directly from the console   redis-cli -h HOST -p PORT  --scan --pattern  prefix      xargs -n 100 redis-cli DEL   This also uses SCAN  which is preferable to KEYS in production  but is not atomic

User · Answer

I just had the same problem  I stored session data for a user in the format   session sessionid key-x - value of x session sessionid key-y - value of y session sessionid key-z - value of z   So  each entry was a seperate key-value pair  When the session is destroyed  I wanted to remove all session data by deleting keys with the pattern session sessionid   - but redis does not have such a function   What I did  store the session data within a hash  I just create a hash with the hash id of session sessionid and then I push key-x  key-y  key-z in that hash  order did not matter to me  and if I dont need that hash anymore I just do a DEL session sessionid and all data associated with that hash id is gone  DEL is atomic and accessing data writing data to the hash is O 1

User · Answer

FYI   only using bash and redis-cli not using keys  this uses scan  works well in cluster mode not atomic  Maybe you only need to modify capital characters  scan-match sh    bin bash rcli  quot  YOUR PATH redis-cli quot   default server  quot YOUR SERVER quot  default port  quot YOUR PORT quot  servers   rcli -h  default server -p  default port cluster nodes   grep master   awk   print  2     sed  s         if   x quot  1 quot      quot x quot     then      startswith  quot DEFAULT PATTERN quot  else     startswith  quot  1 quot  fi MAX BUFFER SIZE 1000 for server in  servers  do      cursor 0     while          r   rcli -h  server -p  default port scan  cursor match  quot  startswith  quot  count  MAX BUFFER SIZE           cursor  echo  r   cut -f 1 -d             nf  echo  r   awk   print NF            if    nf -gt 1    then             for x in  echo  r   cut -f 1 -d    --complement   do                  echo  x             done         fi            cursor    0        do               done done  clear-redis-key sh    bin bash STARTSWITH  quot  1 quot   RCLI YOUR PATH redis-cli HOST YOUR HOST PORT 6379 RCMD  quot  RCLI -h  HOST -p  PORT -c  quot     scan-match sh  STARTSWITH   while read -r KEY   do      RCMD del  KEY  done  Run at bash prompt     clear-redis-key sh key head pattern

User · Answer

This is not direct answer to the question  but since I got here when searching for my own answers  I ll share this here   If you have tens or hundreds of millions of keys you have to match  the answers given here will cause Redis to be non responsive for significant amount of time  minutes    and potentially crash because of memory consumption  be sure  background save will kick in in the middle of your operation    The following approach is undeniably ugly  but I didn t find a better one  Atomicity is out of question here  in this case main goal is to keep Redis up and responsive 100  of the time  It will work perfectly if you have all your keys in one of databases and you don t need to match any pattern  but cannot use http   redis io commands FLUSHDB because of it s blocking nature   Idea is simple  write a script that runs in a loop and uses O 1  operation like http   redis io commands SCAN or http   redis io commands RANDOMKEY to get keys  checks if they match the pattern  if you need it  and http   redis io commands DEL them one by one    If there is a better way to do it  please let me know  I ll update the answer   Example implementation with randomkey in Ruby  as a rake task  a non blocking substitute of something like redis-cli -n 3 flushdb   desc  Cleanup redis  task cleanup redis   environment do   redis   Redis new        connection to target database number which needs to be wiped out   counter   0   while key   redis randomkey                    puts  Deleting   counter     key       redis del key      counter    1   end end

User · Answer

You can also use this command to delete the keys -  Suppose there are many types of keys in your redis like-    xyz category fpc 12   xyz category fpc 245   xyz category fpc 321   xyz product fpc 876   xyz product fpc 302   xyz product fpc 01232    Ex-  xyz category fpc  here xyz is a sitename  and these keys are        related to products and categories of a E-Commerce site and generated by FPC   If you use this command as below-  redis-cli --scan --pattern  key     xargs redis-cli del   OR  redis-cli --scan --pattern  xyz category fpc     xargs redis-cli del   It deletes all the keys like  xyz category fpc   delete 1  2 and 3 keys   For delete other 4  5 and 6 number keys use  xyz product fpc  in above command   If you want to Delete Everything in Redis  then follow these Commands-  With redis-cli    FLUSHDB - Removes data from your connection s CURRENT database  FLUSHALL - Removes data from ALL databases    For Example - in your shell   redis-cli flushall redis-cli flushdb

User · Answer

If you have space in the name of the keys  you can use this in bash   redis-cli keys  pattern       xargs -L1 -I     echo         xargs redis-cli del

[redis] How to atomically delete keys matching a pattern using Redis

Examples related to redis