[hash] How to "EXPIRE" the "HSET" child key in redis?

I need to expire all keys in redis hash, which are older than 1 month.

You can. Here is an example.

redis 127.0.0.1:6379> hset key f1 1
(integer) 1
redis 127.0.0.1:6379> hset key f2 2
(integer) 1
redis 127.0.0.1:6379> hvals key
1) "1"
2) "1"
3) "2"
redis 127.0.0.1:6379> expire key 10
(integer) 1
redis 127.0.0.1:6379> hvals key
1) "1"
2) "1"
3) "2"
redis 127.0.0.1:6379> hvals key
1) "1"
2) "1"
3) "2"
redis 127.0.0.1:6379> hvals key

Use EXPIRE or EXPIREAT command.

If you want to expire specific keys in the hash older then 1 month. This is not possible. Redis expire command is for all keys in the hash. If you set daily hash key, you can set a keys time to live.

hset key-20140325 f1 1
expire key-20140325 100
hset key-20140325 f1 2

You could use the Redis Keyspace Notifications by using psubscribe and "__keyevent@<DB-INDEX>__:expired".

With that, each time that a key will expire, you will get a message published on your redis connection.

Regarding your question basically you create a temporary "normal" key using set with an expiration time in s/ms. It should match the name of the key that you wish to delete in your set.

As your temporary key will be published to your redis connection holding the "__keyevent@0__:expired" when it expired, you can easily delete your key from your original set as the message will have the name of the key.

A simple example in practice on that page : https://medium.com/@micah1powell/using-redis-keyspace-notifications-for-a-reminder-service-with-node-c05047befec3

doc : https://redis.io/topics/notifications ( look for the flag xE)

You can expire Redis hashes in ease, Eg using python

import redis
conn = redis.Redis('localhost')
conn.hmset("hashed_user", {'name': 'robert', 'age': 32})
conn.expire("hashed_user", 10)

This will expire all child keys in hash hashed_user after 10 seconds

same from redis-cli,

127.0.0.1:6379> HMSET testt username wlc password P1pp0 age 34
OK
127.0.0.1:6379> hgetall testt
1) "username"
2) "wlc"
3) "password"
4) "P1pp0"
5) "age"
6) "34"
127.0.0.1:6379> expire testt 10
(integer) 1
127.0.0.1:6379> hgetall testt
1) "username"
2) "wlc"
3) "password"
4) "P1pp0"
5) "age"
6) "34"

after 10 seconds

127.0.0.1:6379> hgetall testt
(empty list or set)

This is not possible, for the sake of keeping Redis simple.

Quoth Antirez, creator of Redis:

Hi, it is not possible, either use a different top-level key for that specific field, or store along with the filed another field with an expire time, fetch both, and let the application understand if it is still valid or not based on current time.

Regarding a NodeJS implementation, I have added a custom expiryTime field in the object I save in the HASH. Then after a specific period time, I clear the expired HASH entries by using the following code:

client.hgetall(HASH_NAME, function(err, reply) {
    if (reply) {
        Object.keys(reply).forEach(key => {
            if (reply[key] && JSON.parse(reply[key]).expiryTime < (new Date).getTime()) {
                client.hdel(HASH_NAME, key);
            }
        })
    }
});

If your use-case is that you're caching values in Redis and are tolerant of stale values but would like to refresh them occasionally so that they don't get too stale, a hacky workaround is to just include a timestamp in the field value and handle expirations in whatever place you're accessing the value.

This allows you to keep using Redis hashes normally without needing to worry about any complications that might arise from the other approaches. The only cost is a bit of extra logic and parsing on the client end. Not a perfect solution, but it's what I typically do as I haven't needed TTL for any other reason and I'm usually needing to do extra parsing on the cached value anyways.

So basically it'll be something like this:

In Redis:

hash_name
- field_1: "2021-01-15;123"
- field_2: "2021-01-20;125"
- field_2: "2021-02-01;127"

Your (pseudo)code:

val = redis.hget(hash_name, field_1)
timestamp = val.substring(0, val.index_of(";"))

if now() > timestamp:
  new_val = get_updated_value()
  new_timestamp = now() + EXPIRY_LENGTH
  redis.hset(hash_name, field_1, new_timestamp + ";" + new_val)
  val = new_val
else:
  val = val.substring(val.index_of(";"))

// proceed to use val

The biggest caveat imo is that you don't ever remove fields so the hash can grow quite large. Not sure there's an elegant solution for that - I usually just delete the hash every once in a while if it feels too big. Maybe you could keep track of everything you've stored somewhere and remove them periodically (though at that point, you might as well just be using that mechanism to expire the fields manually...).

You can use Sorted Set in redis to get a TTL container with timestamp as score. For example, whenever you insert a event string into the set you can set its score to the event time. Thus you can get data of any time window by calling zrangebyscore "your set name" min-time max-time

Moreover, we can do expire by using zremrangebyscore "your set name" min-time max-time to remove old events.

The only drawback here is you have to do housekeeping from an outsider process to maintain the size of the set.

There is a Redisson java framework which implements hash Map object with entry TTL support. It uses hmap and zset Redis objects under the hood. Usage example:

RMapCache<Integer, String> map = redisson.getMapCache('map');
map.put(1, 30, TimeUnit.DAYS); // this entry expires in 30 days

This approach is quite useful.

We had the same problem discussed here.

We have a Redis hash, a key to hash entries (name/value pairs), and we needed to hold individual expiration times on each hash entry.

We implemented this by adding n bytes of prefix data containing encoded expiration information when we write the hash entry values, we also set the key to expire at the time contained in the value being written.

Then, on read, we decode the prefix and check for expiration. This is additional overhead, however, the reads are still O(n) and the entire key will expire when the last hash entry has expired.

This is possible in KeyDB which is a Fork of Redis. Because it's a Fork its fully compatible with Redis and works as a drop in replacement.

Just use the EXPIREMEMBER command. It works with sets, hashes, and sorted sets.

EXPIREMEMBER keyname subkey [time]

You can also use TTL and PTTL to see the expiration

TTL keyname subkey

More documentation is available here: https://docs.keydb.dev/docs/commands/#expiremember

You could store key/values in Redis differently to achieve this, by just adding a prefix or namespace to your keys when you store them e.g. "hset_"

• Get a key/value GET hset_key equals to HGET hset key

• Add a key/value SET hset_key value equals to HSET hset key

• Get all keys KEYS hset_* equals to HGETALL hset

• Get all vals should be done in 2 ops, first get all keys KEYS hset_* then get the value for each key

• Add a key/value with TTL or expire which is the topic of question:

 SET hset_key value
 EXPIRE hset_key

Note: KEYS will lookup up for matching the key in the whole database which may affect on performance especially if you have big database.

Note:

• KEYS will lookup up for matching the key in the whole database which may affect on performance especially if you have big database. while SCAN 0 MATCH hset_* might be better as long as it doesn't block the server but still performance is an issue in case of big database.

• You may create a new database for storing separately these keys that you want to expire especially if they are small set of keys.

Thanks to @DanFarrell who highlighted the performance issue related to KEYS

Redis does not support having TTL on hashes other than the top key, which would expire the whole hash. If you are using a sharded cluster, there is another approach you could use. This approach could not be useful in all scenarios and the performance characteristics might differ from the expected ones. Still worth mentioning:

When having a hash, the structure basically looks like:

hash_top_key
  - child_key_1 -> some_value
  - child_key_2 -> some_value
  ...
  - child_key_n -> some_value

Since we want to add TTL to the child keys, we can move them to top keys. The main point is that the key now should be a combination of hash_top_key and child key:

{hash_top_key}child_key_1 -> some_value
{hash_top_key}child_key_2 -> some_value
...
{hash_top_key}child_key_n -> some_value

We are using the {} notation on purpose. This allows all those keys to fall in the same hash slot. You can read more about it here: https://redis.io/topics/cluster-tutorial

Now if we want to do the same operation of hashes, we could do:

HDEL hash_top_key child_key_1 => DEL {hash_top_key}child_key_1

HGET hash_top_key child_key_1 => GET {hash_top_key}child_key_1

HSET hash_top_key child_key_1 some_value => SET {hash_top_key}child_key_1 some_value [some_TTL]

HGETALL hash_top_key => 
  keyslot = CLUSTER KEYSLOT {hash_top_key}
  keys = CLUSTER GETKEYSINSLOT keyslot n
  MGET keys

The interesting one here is HGETALL. First we get the hash slot for all our children keys. Then we get the keys for that particular hash slot and finally we retrieve the values. We need to be careful here since there could be more than n keys for that hash slot and also there could be keys that we are not interested in but they have the same hash slot. We could actually write a Lua script to do those steps in the server by executing an EVAL or EVALSHA command. Again, you need to take into consideration the performance of this approach for your particular scenario.

Some more references:

• https://redis.io/commands/cluster-keyslot
• https://redis.io/commands/cluster-getkeysinslot
• https://redis.io/commands/eval