some interesting redis-cli commands and usages

redis-cli is the command line interface to redis and is used as the client provided by redis. It is normally used to connect to redis and query redis as below.

redis-cli -h 127.0.0.1 -p 6379
127.0.0.1:6379> randomkey
"myhllkey"
127.0.0.1:6379> type myhllkey
string

However there are other ways in which redis-cli can be useful in exploring redis

1) finding out the general redis stats.

redis-3.2.4 $ src/redis-cli --stat -i 2
------- data ------ --------------------- load -------------------- - child -
keys       mem      clients blocked requests            connections          
6          985.97K  1       0       21 (+0)             5           
6          985.97K  1       0       22 (+1)             5           
6          985.97K  1       0       23 (+1)             5           
6          985.97K  1       0       24 (+1)             5           
6          985.97K  1       0       25 (+1)             5  

The above command "redis-cli --stat -i 2"will keep on returning the general info on the redis server after every 2 seconds.
This info includes number of keys, memory, clients, blocked clients, no of requests and connections.
The paramater 'i' is the time interval in seconds after which the redis-cli will query redis. The default value of i is 1 so that "src/redis-cli --stat" will give the data after every 1 second.

2) finding out the big keys in redis.

   "redis-cli --bigkeys" command will scan the redis db and will start giving you information about the biggest keys of each type that it encounters as it scans the redis db.
Finally it will give information about the numbers of keys of each type found, with its average key members and average key size. This is a very useful command because periodically running it and analysing its output can sometimes give a clue if your redis data size is always increasing, and to potentially big keys.
The output for me was as below.

# Scanning the entire keyspace to find biggest keys as well as
# average sizes per key type.  You can use -i 0.1 to sleep 0.1 sec
# per 100 SCAN commands (not usually needed).

[00.00%] Biggest string found so far 'mystringkey' with 13 bytes
[00.00%] Biggest string found so far 'myhllkey' with 90 bytes
[00.00%] Biggest zset   found so far 'mysortedsetkey' with 6 members
[00.00%] Biggest list   found so far 'mylistkey' with 7 items
[00.00%] Biggest set    found so far 'mysetkey' with 4 members
[00.00%] Biggest hash   found so far 'myhashkey' with 2 fields

-------- summary -------

Sampled 6 keys in the keyspace!
Total key length in bytes is 59 (avg len 9.83)

Biggest string found 'myhllkey' has 90 bytes
Biggest   list found 'mylistkey' has 7 items
Biggest    set found 'mysetkey' has 4 members
Biggest   hash found 'myhashkey' has 2 fields
Biggest   zset found 'mysortedsetkey' has 6 members

2 strings with 103 bytes (33.33% of keys, avg size 51.50)
1 lists with 7 items (16.67% of keys, avg size 7.00)
1 sets with 4 members (16.67% of keys, avg size 4.00)
1 hashs with 2 fields (16.67% of keys, avg size 2.00)
1 zsets with 6 members (16.67% of keys, avg size 6.00)

3) scanning keys with pattern

redis-cli --scan --pattern "*list*" will find out all the keys which have the word 'list' in them. Also, it will internally use 'scan' and not keys, and will not block other commands for large amount of time even if the database size is very large.

4) repeatedly querying redis after every interval for a certain number of times.
"redis-cli -r 5 -i 1 set a b" will set the value of key 'a' to be 'b' 5 times after an interval of 1 sec.

redis-cli --help can be used to find out all the options in redis cli.
It has various options like outputting the data in csv format, for finding out the redis latency, system latency, reading arguments from standard input etc.

:)

finding out the memory occupied by a single key in redis

Many times while using redis, we need to find out how much memory is occupied by a particular key.

This is very useful because redis keeps everything in memory and we need to make the best use of whatever memory we have to reduce the infrastructure cost.

Redis provides a command 'DEBUG' which can be used for this purpose.

This command gives us information like size about a particular key. However this length is the serialized length(i.e. the bytes used to store this key in the rdb backup file).

The below shows how we can find the information about two keys(one string and one hash) in redis.

In the above, the serialized length is the number of bytes which are used by redis for storing the value in the backup file.

More often than not, we are not interested in the serialized length but the memory used by that key.

Unfortunately redis does not provide any such information by its own(as per my knowledge), but thanks to the open source community, an excellent such utility 'redis-rdb-tools' is written which gives us a very good estimate of the memory used by redis for a particular key.

The link to the utility is here.

Redis stores different keys in different formats, and the utility reverse engineers a key with its type and value, to get the approximate amount of memory used by a key. Although the utility underestimates the memory occupied by a key and the real memory used by a key is a little higher(can be higher by upto 20%), but even then it's very important because it can find out the large keys, and it can generate a CSV with the information about all the keys. The CSV file generated can be further analyzed for more insights into the data.

Usage:

After installing the redis-rdb-tools as described here, we can use it to find out the memory used by a key.

finding out memory for a key from running redis.

redis-3.2.4 $ redis-memory-for-key -s localhost -p 6379 mystringkey

Key "mystringkey"

Bytes 88

Type string

redis-3.2.4 $ redis-memory-for-key -s localhost -p 6379 myhashkey

Key "myhashkey"

Bytes 115

Type hash

Encoding ziplist

Number of Elements 2

Length of Largest Element 6

redis-3.2.4 $ 

finding out memory for a key from a rdb file.

redis-3.2.4 $ rdb -c memory dump.rdb -k mystringkey

database,type,key,size_in_bytes,encoding,num_elements,len_largest_element

0,string,"mystringkey",88,string,13,13

finding out memory for all keys for a pattern.

redis-3.2.4 $ rdb -c memory dump.rdb -k my.*

database,type,key,size_in_bytes,encoding,num_elements,len_largest_element

0,list,"mylistkey",219,quicklist,7,6

0,sortedset,"mysortedsetkey",143,ziplist,6,5

0,hash,"myhashkey",115,ziplist,2,6

0,string,"mystringkey",88,string,13,13

0,string,"myhllkey",168,string,90,90

0,set,"mysetkey",452,hashtable,4,6

finding out memory for all keys for a pattern and exporting to a csv file.

redis-3.2.4 $ rdb -c memory dump.rdb -k my.* -f memory.csv

redis-3.2.4 $ head memory.csv 

database,type,key,size_in_bytes,encoding,num_elements,len_largest_element

0,list,"mylistkey",219,quicklist,7,6

0,sortedset,"mysortedsetkey",143,ziplist,6,5

0,hash,"myhashkey",115,ziplist,2,6

0,string,"mystringkey",88,string,13,13

0,string,"myhllkey",168,string,90,90

0,set,"mysetkey",452,hashtable,4,6

After we have the csv file from the above method, we can do all sorts of analysis after the excel tricks by finding out the largest N keys of a particular type, keys occupying highest amount of memory etc.

:)

how to make redis more proactive in expiring keys.

Redis expires keys in two ways, active way, and passive way.

In the 'passive' way, when a user tries to get a key, redis will check whether the key should have expired, and if the key should be expired, redis will expire the key and return to the user that the key does not exist. This is called 'passive' way because redis did not 'actively' expired the key by itself, but only 'passively' or 'lazily' expired it when it came to know about it on user's request.

In the 'active' way, redis will run background job which will randomly get a sample of keys, and will expire whichever keys need to be expired. Further if it sees that more than 75% of those keys should be expired, it will further get another sample. Lets call this process 'keys expiring process'. The process is described here.

It means that at any point, on an average, there are a maximum of 25% keys which should have expired but did not. Needless to say, each of these keys occupies memory.

However when we restart redis, it loads all the keys, and will expire all the keys which should expire.

However there is a way for us to direct redis to be more 'proactive' in expiring the keys.
In the redis configuration, there is a parameter named 'hz', which is the number of times redis will run the process to remove the expiring keys in a second('keys expiring process' described above).

The default value is 10, so that redis runs 10 such processes in a second.
As per redis.conf file, although the value of 'hz' should be less than 500, it is recommended to keep the value less than 100.

We can increase this value to some value, like 50.

redis-3.0.6 $redis-cli config get hz

1) "hz"

2) "10"

redis-3.0.6 $ cat redis.conf | grep 'hz '

hz 10

redis-3.0.6 $ redis-cli config set hz 50

OK

redis-3.0.6 $ 

It is important to note that increasing the 'hz' value can lead to a slight increase in the CPU usage, so it is recommended to check the CPU usage after increasing it.

When we did it in our production stack, we saw that the number of keys reduced to some point and became stable, and same was the case with memory. CPU usage did not show any spike so, we were quite pleased to recover some more memory. 

:)

Redis - How to take individual nodes in redis cluster down for maintainence.

Redis is a very fast caching server, keeps all the data in RAM, and it is able to serve requests with sub microseconds latency.
Considering it keeps all the data in memory, if we need to keep more data than what fits in a single machine's memory, we create a redis cluster.
In a redis cluster, the data is distributed among its N masters, where N >=3. Also it is recommended to replicate the data, so that each master node has a backup node in case it goes down. The same backup node can be used and be promoted to master in case we have to bring down the master node for maintenance.

The main command used for this is the CLUSTER FAILOVER command.
This command should always be executed on a slave node, and it promotes the slave in a cluster to a master, (and master becomes the slave).

Below, we will show how to bring the master nodes of redis down one by one for maintainence(or for other activities)
Lets say you have the cluster in running mode with 3 masters and 3 slaves.

for port in {7000..7005}; do ../src/redis-server $port/redis.conf ; done ; 

I have modified the conf files so that above will start the 6 instances of redis in cluster mode, in daemon threads, and with appropriate logging.
After that, the cluster will be created using

../src/redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005

The cluster can be verified using cluster nodes or cluster info command.

../src/redis-cli -p 7001 cluster nodes

cluster-test $../src/redis-cli -p 7001 cluster nodes
daabc7bdc914f73682e07ed40769766ecee06c8f 127.0.0.1:7000 master - 0 1485859005956 1 connected 0-5460
276ad9dbd675f7e3183ab808549f1436d91e56b3 127.0.0.1:7001 myself,master - 0 0 2 connected 5461-10922
2a2e22e2e03bbc934d972a247b4a9c5c0b341747 127.0.0.1:7005 slave 5c0585a57362901380c392b4f3041777e169adec 0 1485859005653 6 connected
5c0585a57362901380c392b4f3041777e169adec 127.0.0.1:7002 master - 0 1485859006463 3 connected 10923-16383
ad7b5e40d14c4f63b2e1b7a901a3c258920d8898 127.0.0.1:7003 slave daabc7bdc914f73682e07ed40769766ecee06c8f 0 1485859005451 4 connected
ffc3155ca303da41825d42db3d109b67525e1f5a 127.0.0.1:7004 slave 276ad9dbd675f7e3183ab808549f1436d91e56b3 0 1485859005653 5 connected

../src/redis-cli -p 7001 cluster info

cluster-test $ ../src/redis-cli -p 7001 cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:2
cluster_stats_messages_sent:9298
cluster_stats_messages_received:9081

The general rules are these.

1. A slave can be brought down without a problem.
2. A master should be first converted to a slave before bringing it down.
3. After converting a master to a slave, always verify it.

For bringing down a slave like the instance running on 7005, we can simple stop redis, using shutdown or by killing the process(ideally shutdown should be renamed command).

../src/redis-cli -p 7005 shutdown

After doing the maintainence activity, we can again start redis using.

../src/redis-server 7005/redis.conf

For bringing down a master(like instance on 7002), we need to convert it into a slave and then stop the redis server.

This is done by running the cluster failover command on the slave of the master.

To find out which instance is the slave of the 7002 instance, we can execute the info replication command on 7002 instance

cluster-test $ redis-cli -p 7002 info replication

# Replication

role:master

connected_slaves:1

slave0:ip=127.0.0.1,port=7005,state=online,offset=5729,lag=0

master_repl_offset:5729

repl_backlog_active:1

repl_backlog_size:1048576

repl_backlog_first_byte_offset:5716

repl_backlog_histlen:14

cluster-test $ 

From the info replication command above, we can see that the slave of instance on port 7002 is instance on 7005.

So, we run the cluster failover command on the slave(7005), so that it becomes the master, and 7002 becomes the slave.

cluster-test $ ../src/redis-cli -p 7005 cluster failover

OK

cluster-test $ ../src/redis-cli -p 7005 cluster failover

(error) ERR You should send CLUSTER FAILOVER to a slave

Note that many times the cluster failover command returns "OK", but the slave is not promoted to a master, because the slave may not be in sync with the master. So, it is strongly recommended to run the command again so that we can be sure that 7005 has converted to the master.

We can also check it by executing the info replication command.

cluster-test $ redis-cli -p 7005 info replication

# Replication

role:master

connected_slaves:1

slave0:ip=127.0.0.1,port=7002,state=online,offset=16762,lag=1

master_repl_offset:16762

repl_backlog_active:1

repl_backlog_size:1048576

repl_backlog_first_byte_offset:16693

repl_backlog_histlen:70

After that, since the 7002 instance is a slave, it can be brought down by the usual shutdown command(or by killing the redis process or by stopping the service).

../src/redis-cli -p 7002 shutdown

:)

Normally, shutdown is one of the dangerous commands which should be disabled or atleast renamed in the redis servers to avoid the misuse.