Redis - How to take individual nodes in redis cluster down for maintainence.

Redis is a very fast caching server, keeps all the data in RAM, and it is able to serve requests with sub microseconds latency.
Considering it keeps all the data in memory, if we need to keep more data than what fits in a single machine's memory, we create a redis cluster.
In a redis cluster, the data is distributed among its N masters, where N >=3. Also it is recommended to replicate the data, so that each master node has a backup node in case it goes down. The same backup node can be used and be promoted to master in case we have to bring down the master node for maintenance.

The main command used for this is the CLUSTER FAILOVER command.
This command should always be executed on a slave node, and it promotes the slave in a cluster to a master, (and master becomes the slave).

Below, we will show how to bring the master nodes of redis down one by one for maintainence(or for other activities)
Lets say you have the cluster in running mode with 3 masters and 3 slaves.

for port in {7000..7005}; do ../src/redis-server $port/redis.conf ; done ; 

I have modified the conf files so that above will start the 6 instances of redis in cluster mode, in daemon threads, and with appropriate logging.
After that, the cluster will be created using

../src/redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005

The cluster can be verified using cluster nodes or cluster info command.

../src/redis-cli -p 7001 cluster nodes

cluster-test $../src/redis-cli -p 7001 cluster nodes
daabc7bdc914f73682e07ed40769766ecee06c8f 127.0.0.1:7000 master - 0 1485859005956 1 connected 0-5460
276ad9dbd675f7e3183ab808549f1436d91e56b3 127.0.0.1:7001 myself,master - 0 0 2 connected 5461-10922
2a2e22e2e03bbc934d972a247b4a9c5c0b341747 127.0.0.1:7005 slave 5c0585a57362901380c392b4f3041777e169adec 0 1485859005653 6 connected
5c0585a57362901380c392b4f3041777e169adec 127.0.0.1:7002 master - 0 1485859006463 3 connected 10923-16383
ad7b5e40d14c4f63b2e1b7a901a3c258920d8898 127.0.0.1:7003 slave daabc7bdc914f73682e07ed40769766ecee06c8f 0 1485859005451 4 connected
ffc3155ca303da41825d42db3d109b67525e1f5a 127.0.0.1:7004 slave 276ad9dbd675f7e3183ab808549f1436d91e56b3 0 1485859005653 5 connected

../src/redis-cli -p 7001 cluster info

cluster-test $ ../src/redis-cli -p 7001 cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:2
cluster_stats_messages_sent:9298
cluster_stats_messages_received:9081

The general rules are these.

1. A slave can be brought down without a problem.
2. A master should be first converted to a slave before bringing it down.
3. After converting a master to a slave, always verify it.

For bringing down a slave like the instance running on 7005, we can simple stop redis, using shutdown or by killing the process(ideally shutdown should be renamed command).

../src/redis-cli -p 7005 shutdown

After doing the maintainence activity, we can again start redis using.

../src/redis-server 7005/redis.conf

For bringing down a master(like instance on 7002), we need to convert it into a slave and then stop the redis server.

This is done by running the cluster failover command on the slave of the master.

To find out which instance is the slave of the 7002 instance, we can execute the info replication command on 7002 instance

cluster-test $ redis-cli -p 7002 info replication

# Replication

role:master

connected_slaves:1

slave0:ip=127.0.0.1,port=7005,state=online,offset=5729,lag=0

master_repl_offset:5729

repl_backlog_active:1

repl_backlog_size:1048576

repl_backlog_first_byte_offset:5716

repl_backlog_histlen:14

cluster-test $ 

From the info replication command above, we can see that the slave of instance on port 7002 is instance on 7005.

So, we run the cluster failover command on the slave(7005), so that it becomes the master, and 7002 becomes the slave.

cluster-test $ ../src/redis-cli -p 7005 cluster failover

OK

cluster-test $ ../src/redis-cli -p 7005 cluster failover

(error) ERR You should send CLUSTER FAILOVER to a slave

Note that many times the cluster failover command returns "OK", but the slave is not promoted to a master, because the slave may not be in sync with the master. So, it is strongly recommended to run the command again so that we can be sure that 7005 has converted to the master.

We can also check it by executing the info replication command.

cluster-test $ redis-cli -p 7005 info replication

# Replication

role:master

connected_slaves:1

slave0:ip=127.0.0.1,port=7002,state=online,offset=16762,lag=1

master_repl_offset:16762

repl_backlog_active:1

repl_backlog_size:1048576

repl_backlog_first_byte_offset:16693

repl_backlog_histlen:70

After that, since the 7002 instance is a slave, it can be brought down by the usual shutdown command(or by killing the redis process or by stopping the service).

../src/redis-cli -p 7002 shutdown

:)

Normally, shutdown is one of the dangerous commands which should be disabled or atleast renamed in the redis servers to avoid the misuse.

Redis Cluster monitoring - part 1 - node monitoring script

Redis is an in-memory database used for caching which provides very high performance and can run uninterrupted for months. Considering redis stores all the data in memory, if our data size is more than the memory for a single machine, we have to distribute the data on various machines. This is where Redis Cluster comes in, and it provides us a way to distribute the data on different machines, add/remove new machines etc.

However, once we have a large numbers of nodes in a redis cluster, it becomes imperative to continuously monitor the state and health of each redis cluster node/system because the chances of failure of one or more nodes increase.

Also, it helps to have automatic scripts which can monitor the redis nodes and alert in case it senses an error or that some memory/connection threshold has reached.

We should have the following monitoring in a cluster.

1. monitoring individual nodes.
2. monitoring overall cluster health.
3. monitoring stats on redis nodes.
4. viewing the redis stats over time on some graph

Monitoring individual nodes basically involves monitoring the redis process running on of each node and restarts it if it stops.

Monitoring overall cluster health is required because it is possible that one of the machine is down, so that its monitoring script running on it cannot send an alert. In this case, the global redis monitoring script should try to do a basic insert in each cluster stack, and if it fails, it should trigger the email alert.

Monitoring stats on redis nodes is required so that we don't have to wait for the things to go bad in redis, and we can identify whenever the threshold is reached for various indices in redis. This involves automatic monitoring of individual redis nodes, for the connections, memory, replication lag etc.

Finally, we need to have the stats of various redis nodes represented in terms of graphs. This is required to identify uneven patterns in the data usage/access and to have a global view of how the redis stack is used.

In this part, we will go through how we can monitor individual nodes.

Individual nodes can be monitored by a shell script, essentially a shell script will run every 30 seconds or so and will see if a redis server is running along with a port(s), if the redis server is not running on the ports defined, it restarts the redis server.

This can be achieved using a simple shell script as below.

#!/bin/bash

START_PORT=7000

END_PORT=7003

error=0

ports=''

checkRedis(){

        count=`ps -ef | grep "redis-server" | grep ":$1" | wc -l`

        if [[ $count -ne 1 ]]

        then

                error=1

                ports="$ports $1"

                echo "starting redis on port $1"

                # start redis either by redis-server or by service if redis is installed as a service

                # service redis-$1 start

                src/redis-server cluster-test/$1/redis.conf

        fi

}

for ((i=START_PORT;i<=END_PORT;i++)); do

    checkRedis $i

done

if [[ $error -eq 1 ]]

then

        echo "need to send mail that redis was started on ports $ports"

fi

The above script should run on each machine having one or many redis nodes. If a machine has redis running on different ports, they can be specified. In the above script, we specified that the redis will be running on ports 7000, 7001, 7002, 7003.

The above script can be saved in a file like 'monitorIndividualNodes.sh' and can be run every 2 minutes in crontab using

*/2 * * * * sh /redis/monitorIndividualNodes.sh

The script can be configured to run every interval, like every minute or so through crontab or any other trustworthy scheduling service, and will check whether the redis server is running on predefined ports on those machines. If it is not running, it will start the redis. Optionally, it should also send an email to alert the concerned.

Also, even in case of system restart, cron will run the script appropriately and all the redis instances will start.

Considering redis is very stable, and does not stop unless there is a machine restart, we don't have to worry about receiving too many emails. :)

In the next part, we will see the script to monitor the overall health of the cluster. This can be useful in case one or more machines are down as as result of which the monitoring script of individual nodes cannot run on them and no alert is generated by them.

Happy redising. :)

Redis Cluster - How to fix a cluster which is in broken state after migration

Sometimes in redis cluster, we need to expand the redis cluster by adding more machines. This is accomplished by adding more machines, making them as part of the cluster and resharding the existing cluster as explained here.

However, many times, the cluster is stuck in an inconsistent state when there is an error in resharding. This can happen because of many reasons like sudden termination of reshard script, redis timeout error while moving slots(in case the keys are too big), or if a key already exists in the target database(because it involves migration).

There is no quick fix to these problems, and one would have to understand the internals of how a slot movement happens.

But there is a general rule on what can be done to try to fix if a redis cluster is stuck in an inconsistent state during reshard/migration.

The following two are important ways to fix a cluster which was broken because of migration.


Run the Fix Script

Fixing a resharding error can be done by running the fix script provided by redis.
It can be run using

./redis-trib.rb fix 127.0.0.1:7000

We will need to change the ip address and port as per your configuration. Also, you only need to provide the ip address/port of only a single node which is part of the cluster. The configuration of the other nodes are read automatically by the script.

If the above cannot fix the cluster state, then you can follow the below step.

Setting the Cluster slots to a particular node.

Manually checking the keys in the unstable slot, and setting the slot to be served by a particular slot. We can execute the "cluster nodes" command on all the nodes and see if any slot in set in a migrating/importing state. If we are sure a slot belongs to a particular node and the node holds and serves the data for that slot, we can set the slot to that node by executing the cluster setslot <slot> node <nodeid> command as described here.

If a node 127.0.0.1:7000 does not have the correct configuration as per cluster nodes command executed on it, and it shows that the slot 1000 is with some other node, but for all other nodes, it shows that it is with the node with node id abcdefgghasgdkashd, then we need to correct the configuration of that node(127.0.0.1:7000). It can be executing like  the following.

redis-cli -h 127.0.0.1 -p 7001 cluster setslot 1000 node abcdefgghasgdkashd

The above command just assures the node 127.0.0.1:7000 that the slot 1000 is served by the node with node id abcdefgghasgdkashd, and that the node 127.0.0.1 should correct its configuration to affect the below.

Note that you need to run it if you are sure that all other nodes agree to it with their cluster nodes command, and you are sure that the data for slot 1000 resides in the node abcdefgghasgdkashd.

redis cluster - slave sync with master failed - client scheduled to be closed ASAP for overcoming of output buffer limits.

Sometimes while working with redis cluster, and while the master-slave replication happens, sometimes slave is not able to sync itself to the master.
While syncing, the slave's connection to the master breaks again and again.

The following are the logs on the slave instance(IP:Port are masked).

10 Nov 17:09:20.387 * Connecting to MASTER IP1:PORT1
10 Nov 17:09:20.387 * MASTER <-> SLAVE sync started
10 Nov 17:09:20.387 * Non blocking connect for SYNC fired the event.
10 Nov 17:09:20.387 * Master replied to PING, replication can continue...
10 Nov 17:09:20.388 * Trying a partial resynchronization (request 479e75d20ac03150de943a24d9b61300b6fa59d0:266086101935).
10 Nov 17:09:20.894 * Full resync from master: 479e75d20ac03150de943a24d9b61300b6fa59d0:266259284057
10 Nov 17:09:20.894 * Discarding previously cached master state.
10 Nov 17:10:31.281 * MASTER <-> SLAVE sync: receiving 3811948917 bytes from master
10 Nov 17:10:40.138 * MASTER <-> SLAVE sync: Flushing old data
10 Nov 17:11:11.933 * MASTER <-> SLAVE sync: Loading DB in memory
10 Nov 17:12:37.115 * MASTER <-> SLAVE sync: Finished with success
10 Nov 17:12:37.849 # Connection with master lost.
10 Nov 17:12:37.849 * Caching the disconnected master state.
10 Nov 17:12:38.866 * Connecting to MASTER IP1:PORT1
10 Nov 17:12:38.866 * MASTER <-> SLAVE sync started
10 Nov 17:12:38.866 * Non blocking connect for SYNC fired the event.

The error keeps repeating itself.
Somehow the error above does not give much clue. Just when the master-slave sync succeeds, connection with master is lost.

Looking at the master logs give more clue.
Below are master logs.

10 Nov 17:09:20.383 * Slave IP2:PORT2 asks for synchronization
10 Nov 17:09:20.383 * Unable to partial resync with slave IP2:PORT2 for lack of backlog (Slave request was: 266086101935).
10 Nov 17:09:20.383 * Starting BGSAVE for SYNC with target: disk
10 Nov 17:09:20.889 * Background saving started by pid 37651
10 Nov 17:10:30.557 * DB saved on disk
10 Nov 17:10:30.844 * RDB: 283 MB of memory used by copy-on-write
10 Nov 17:10:31.276 * Background saving terminated with success
10 Nov 17:10:39.561 * Synchronization with slave IP2:PORT2 succeeded
10 Nov 17:11:09.060 # Client id=2292912 addr=IP2:23666 fd=233 name= age=109 idle=9 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=7104 omem=127259816 events=rw cmd=psync scheduled to be closed ASAP for overcoming of output buffer limits.
10 Nov 17:11:09.160 # Connection with slave IP2:PORT2 lost.
10 Nov 17:12:38.861 * Slave IP2:PORT2 asks for synchronization
10 Nov 17:12:38.861 * Unable to partial resync with slave IP2:PORT2 for lack of backlog (Slave request was: 266261610586).
10 Nov 17:12:38.861 * Starting BGSAVE for SYNC with target: disk


The Key part is cmd=psync scheduled to be closed ASAP for overcoming of output buffer limits.

It seems to show that the slave connection was closed as it reached output buffer limits for the master.
Somehow we need to fix this.
There is a setting in the conf file
client-output-buffer-limit slave 256mb 64mb 60

Reading the excellent documentation seems to suggest that the two limits, 256 mb and 64 mb are two limits, hard limits and soft limits. If the client connection for slave exceeds 256 mb or exceeds 64 mb continuously for 60 seconds.

We can either increase this setting, and restart the master server, but it seems too much of an effort and a risk and restarting a master may make its outdated slave trying to become a master.
Thankfully, this can be changed at runtime by executing the following command on master server.

redis-cli -h MASTER_IP -p MASTER_PORT CONFIG get client-output-buffer-limit

It gives the output as 

1) "client-output-buffer-limit"

2) "normal 0 0 0 slave 268435456 67108864 60 pubsub 33554432 8388608 60"

Notice that the hard and soft limits for slave is 256 MB/64 MB (268435456/67108864 bytes)

We will update it to  1 GB/256 MB(1073741824/268435456 bytes) by executing the following command.

redis-cli -h MASTER_IP -p MASTER_PORT CONFIG set client-output-buffer-limit "normal 0 0 0 slave 1073741824 268435456 60 pubsub 33554432 8388608 60"

Note that depending on how far your slave is lagging behind the master and how fast your slave is able to consume the data from the master while syncing, you may have to increase the limit to more than 1GB/256 MB. Just be sure to go through the excellent documentation of client-output-buffer-limit in redis.conf.

After executing this command, we see that the slave succeeds in syncing with the master and does not give any other error.  

Below are the logs on slave after executing the command.

10 Nov 17:15:58.692 * MASTER <-> SLAVE sync started

10 Nov 17:15:58.692 * Non blocking connect for SYNC fired the event.

10 Nov 17:15:58.692 * Master replied to PING, replication can continue...

10 Nov 17:15:58.692 * Trying a partial resynchronization (request 479e75d20ac03150de943a24d9b61300b6fa59d0:266435075281).

10 Nov 17:15:59.195 * Full resync from master: 479e75d20ac03150de943a24d9b61300b6fa59d0:266612963059

10 Nov 17:15:59.195 * Discarding previously cached master state.

10 Nov 17:17:10.490 * MASTER <-> SLAVE sync: receiving 3811953308 bytes from master

10 Nov 17:17:19.613 * MASTER <-> SLAVE sync: Flushing old data

10 Nov 17:17:53.203 * MASTER <-> SLAVE sync: Loading DB in memory

10 Nov 17:19:19.332 * MASTER <-> SLAVE sync: Finished with success

10 Nov 18:01:31.139 * Background saving started by pid 16989

:)

How to move a single slot in redis cluster

In redis cluster, there are a total of 16384 logical slots, which are divided between multiple masters.
Often, when we need to add nodes to the redis cluster, we need to reshard the data.
This is done by redis-trib.rb utility file provided by redis.
However the redis-trib.rb utility can move N no of slots between masters, but it does not provide a function to move a single slot.
Moving a slot normally involves 3 steps as described here.

1. setting the destination node to receive a slot.
2. setting the source node to migrate a slot.
3. migrating the data from source node to destination node.
4. communicating source and destination nodes that the destination node is the owner of the slot.

Below is shown an existing cluster where the slots are equally divided between nodes.

Also, each node has some data.

We will try to move a slot, say slot no 102, from redis running on port 7000 to redis running on port 7001

The java code for it is below.

import java.util.List;

import com.google.common.collect.Lists;

import com.lambdaworks.redis.MigrateArgs;

import com.lambdaworks.redis.RedisClient;

import com.lambdaworks.redis.RedisURI;

import com.lambdaworks.redis.api.sync.RedisCommands;

import com.lambdaworks.redis.cluster.models.partitions.ClusterPartitionParser;

import com.lambdaworks.redis.cluster.models.partitions.Partitions;

import com.lambdaworks.redis.cluster.models.partitions.RedisClusterNode;

import com.lambdaworks.redis.codec.Utf8StringCodec;

import com.lambdaworks.redis.protocol.CommandArgs;

public class RedisMoveSingleSlot {

public static void main(String[] args){

String sourceHost = "127.0.0.1";

int sourcePort = 7000;

String destHost = "127.0.0.1";

int destPort = 7001;

int slotToMove = 102;

moveSlot(sourceHost, sourcePort, destHost, destPort, slotToMove);

}

private static void moveSlot(String sourceHost, int sourcePort, String destHost, int destPort, int slot) {

// create connections to source and destinations.

RedisClient sourceClient = RedisClient.create(RedisURI.create(sourceHost, sourcePort));

RedisClient destinationClient = RedisClient.create(RedisURI.create(destHost, destPort));

RedisCommands<String, String> source = sourceClient.connect().sync();

RedisCommands<String, String> destination = destinationClient.connect().sync();

// get the source node Id and destination Node Id.

String sourceNodeId = source.clusterMyId();

String destinationNodeId = destination.clusterMyId();

//STEP 1, set destination to be importing slot from source.

destination.clusterSetSlotImporting(slot, sourceNodeId);

//STEP 2, set source to be migrating slot to destination.

source.clusterSetSlotMigrating(slot, destinationNodeId);

// STEP 3 starts, we need to move all the keys from source to destination.

List<String> keys = source.clusterGetKeysInSlot(slot, Integer.MAX_VALUE);

// move keys in batches of 100, 

int keySize = 100;

System.out.println("Slot : " + slot + " : got keys of size : " + keys.size() );

if(keys != null && keys.size() > 0){

List<List<String>> listOfListOfKeys = Lists.partition(keys, keySize);

int noOfKeysMoved = 0;

for(List<String> listOfKeys : listOfListOfKeys){

MigrateArgs<String> migrateArgs = new MigrateArgs<>();

migrateArgs.keys(listOfKeys);

migrateArgs.replace();

CommandArgs<String, String> args1 = new CommandArgs<String, String>(new Utf8StringCodec());

migrateArgs.build(args1);

// migrate the keys from source to destination in db 0(default), and will some timeout set, here set to 5000 secs.

source.migrate(destHost, destPort, 0, 5000000, migrateArgs);

noOfKeysMoved = noOfKeysMoved + listOfKeys.size();

System.out.println("totally moved " + noOfKeysMoved + " for slot " + slot);

}

//Thread.sleep(5000);

}

//STEP 3 finished.

// STEP 4, set cluster set slot on source and destination and close the source and destinations.
source.clusterSetSlotNode(slot, destinationNodeId);

destination.clusterSetSlotNode(slot, destinationNodeId);

source.close();
destination.close();

// STEP 4 done.

System.out.println("Moved slot " + slot);

}

}

After the code is executed, it moved 2 keys(which belonged to slot 102) and slot no 102 from redis on 7000 port to redis on 7002 port.

Below was the output of "cluster nodes" and "dbsize" command on each nodes.

Note that the slot 102 moved. Also the no of keys belonging to redis with 7001 port increased by 2.