Monday, 30 January 2017

Redis - setting redis query timeouts using Lettuce in java

While writing applications, sometimes we need to get the data within a particular time, and if the time taken by the redis or the networks is more than a particular time, the application may not want to wait that long for the data to come back and may take some other route.

This is particularly important in scenarios like getting the user profiles while writing adservers where the overall response time is expected to be less than 100 ms, and if any component takes more than a few milliseconds, it should be discarded and the flow should continue without it. In the above scenarios, to make sure that the requests are served as fast as possible, while having a backup plan(how to get the required data in case the timeout happens from redis), it is a good idea to set the timeout.

Normally redis is very fast, and can serve data in a few microseconds(1 microsecond is a thousandth of a millisecond), but it is possible that the network between the redis and the application is slow and it is leading to slow queries.

The Redis client library in Java, Lettuce, provides a way for the application to set a timeout on the query. This way, the application for only wait for that time for the response to come back, otherwise it will proceed by throwing a timeout exception.

You will need the following entry in pom.xml file.

<dependency>
    <groupId>biz.paluch.redis</groupId>
    <artifactId>lettuce</artifactId>
    <version>4.3.0.Final</version>
</dependency>


The below code demonstrates the same.

public static void main(String[] args) {
String key = "testKey";
String value = null;

RedisURI uri = RedisURI.create("localhost", 6379);
RedisClient client = RedisClient.create(uri);

/*
* Normal way
*/
RedisCommands<String, String> commands = client.connect().sync();
System.out.println(commands.get(key));


/* 
* Reactive way
*/
RedisReactiveCommands<String, String> reactiveCommand = client.connect().reactive();
rx.Observable<String> observable = reactiveCommand.get(key);

try{
value = observable.timeout(100, TimeUnit.MILLISECONDS).toBlocking().singleOrDefault(null);
} catch (RuntimeException e) {
if (e.getCause() instanceof TimeoutException){
System.out.println("timeout exception thrown");
}
}

System.out.println(value);
reactiveCommand.close();



The above code shows both ways to get the data using Lettuce Client, ie. Normal way and the reactive way.
In the normal way, we get the data from redis regardless of how long redis(and the network) take to return the data.
However, in the reactive way, if the value of "testkey" in not returned within 100 ms, a Timeout exception wrapped inside RuntimeException is thrown.

The application may handle the exception and take whatever it needs to do(like get the user's profile from a different source or ignore the user's profile altogether, treating it as a new user or depending on the business use case).

Note that it is a good idea to log such timeouts and analyse the reasons if the timeouts occur very frequently.

:)

Thursday, 22 December 2016

Redis Cluster Monitoring - part 2, global cluster monitoring script.

In the first part of the series on how to effectively monitor redis cluster stacks here, we had seen the first of the four types of scripts that can be useful for us. It also mentioned about the monitoring shell script running on all the systems and monitoring the health of redis on that particular system.

So, lets say we have the redis instance monitoring script (described in part 1) running on a particular redis server and checking every 2 minutes by crontab, that the redis server process is up on that machine.
But what happens if the machine itself is down. If that is the case, the cron will not run and we will not get the notification about redis nodes on that machine. :(

In this part, we will see how we can globally maintain a script to fix the above problem.
In this script, we use the "cluster info" command to find out whether the cluster is in OK state or not.

#!/bin/sh

erroringStacks=''

# it expects five arguments, 1st being the cluster identifier, 
# 2nd n 3rd are IP port of a master in the cluster, 
# 4th n 5th are the IP port of the slave of the master.
checkCluster() {
   msg="Redis Cluster $1 $2 $3 $4 $5 "
   # try to get the cluster state using the first IP port combination. 
   val=`redis-cli -c -h $2 -p $3 cluster info 2> /dev/null | grep 'cluster_state'` 

   if [[ $val == *"ok"* ]]
   then
        msg="$msg OK"
   else
# if the first IP port did not give ok response, 
# try to get the cluster state from the second IP port combination.
val=`redis-cli -c -h $4 -p $5 cluster info 2> /dev/null | grep 'cluster_state'`

if [[ $val == *"ok"* ]]
        then
                msg="$msg OK"
        else
                msg="$msg down"
                erroringStacks="$erroringStacks $1"
        fi
   fi
}

# expects 3 args, 1st is stack identifier, 2nd,3rd are IP port
checkStandalone() {
   msg="Redis Standalone $1 $2 $3 "
   # try to set a simple value in the standalone instance, and get the output.
   val=`redis-cli -h $2 -p $3 set healthcheck:abc def 2> /dev/null`
   if [[ $val == *"OK"* ]]
   then
msg="$msg OK"
   else
msg="$msg down, not able to insert data"
erroringStacks="$erroringStacks $1"
   fi

}

checkCluster C1 127.0.0.1 7000 127.0.0.1 7001
checkStandalone S1 127.0.0.1 6380
checkCluster C2 127.0.0.1 7010 127.0.0.1 7011


if [[ "$erroringStacks" == "" ]]
then
    echo "all well" 
else
    echo "The following stacks are erroring out: $erroringStacks " 
    # send an email to the concerned team.
fi


The above script checks whether the various stacks are up or not. 

For clusters, it gets the info from redis-cli cluster info command. Further, if a particular node specified is not up, it checks its slave to see that it is up, and the cluster info is ok. If both the master and slave specified are not OK, then the cluster is bound to be down and an error is returned. Note that the above method will not work for cluster if cluster-require-full-coverage is set to 'no' in redis.conf file, which means that the cluster will still be up even if some of its slots are not being served properly.

For standalone instance, it can get to check the state by inserting a dummy key value pair in redis. If the key value pair cannot be inserted, then the error will be thrown.

The same code can be done for multiple stacks by specifying a stack name which is the identifier for the stack. Here the cluster stacks are mentioned as C1, and C2 and standalone stacks is S1.

In case, the error is thrown, an appropriate alert can be generated.

This script can be scheduled in crontab and could run every fixed interval like 5 minutes.

*/5 * * * * sh /redis/monitorGlobalRedisStacks.sh

Also, ideally this script should run on more than 2-3 machines, atleast 1 of those should be guaranteed to be up.

This concludes our second part of this series. In the third part, we will see how we can have an hourly checking script on the threshold of various important params in redis servers, and have a mailer in case the threshold in memory, connection, replication is reached.

:)

Wednesday, 21 December 2016

Redis Cluster monitoring - part 1 - node monitoring script

Redis is an in-memory database used for caching which provides very high performance and can run uninterrupted for months. Considering redis stores all the data in memory, if our data size is more than the memory for a single machine, we have to distribute the data on various machines. This is where Redis Cluster comes in, and it provides us a way to distribute the data on different machines, add/remove new machines etc.

However, once we have a large numbers of nodes in a redis cluster, it becomes imperative to continuously monitor the state and health of each redis cluster node/system because the chances of failure of one or more nodes increase.

Also, it helps to have automatic scripts which can monitor the redis nodes and alert in case it senses an error or that some memory/connection threshold has reached.

We should have the following monitoring in a cluster.
  1. monitoring individual nodes.
  2. monitoring overall cluster health.
  3. monitoring stats on redis nodes.
  4. viewing the redis stats over time on some graph

Monitoring individual nodes basically involves monitoring the redis process running on of each node and restarts it if it stops.

Monitoring overall cluster health is required because it is possible that one of the machine is down, so that its monitoring script running on it cannot send an alert. In this case, the global redis monitoring script should try to do a basic insert in each cluster stack, and if it fails, it should trigger the email alert.

Monitoring stats on redis nodes is required so that we don't have to wait for the things to go bad in redis, and we can identify whenever the threshold is reached for various indices in redis. This involves automatic monitoring of individual redis nodes, for the connections, memory, replication lag etc.

Finally, we need to have the stats of various redis nodes represented in terms of graphs. This is required to identify uneven patterns in the data usage/access and to have a global view of how the redis stack is used.

In this part, we will go through how we can monitor individual nodes.

Individual nodes can be monitored by a shell script, essentially a shell script will run every 30 seconds or so and will see if a redis server is running along with a port(s), if the redis server is not running on the ports defined, it restarts the redis server.

This can be achieved using a simple shell script as below.

#!/bin/bash

START_PORT=7000
END_PORT=7003

error=0
ports=''
checkRedis(){
        count=`ps -ef | grep "redis-server" | grep ":$1" | wc -l`
        if [[ $count -ne 1 ]]
        then
                error=1
                ports="$ports $1"
                echo "starting redis on port $1"
                # start redis either by redis-server or by service if redis is installed as a service
                # service redis-$1 start
                src/redis-server cluster-test/$1/redis.conf
        fi
}

for ((i=START_PORT;i<=END_PORT;i++)); do
    checkRedis $i
done

if [[ $error -eq 1 ]]
then
        echo "need to send mail that redis was started on ports $ports"
fi


The above script should run on each machine having one or many redis nodes. If a machine has redis running on different ports, they can be specified. In the above script, we specified that the redis will be running on ports 7000, 7001, 7002, 7003.

The above script can be saved in a file like 'monitorIndividualNodes.sh' and can be run every 2 minutes in crontab using

*/2 * * * * sh /redis/monitorIndividualNodes.sh

The script can be configured to run every interval, like every minute or so through crontab or any other trustworthy scheduling service, and will check whether the redis server is running on predefined ports on those machines. If it is not running, it will start the redis. Optionally, it should also send an email to alert the concerned.

Also, even in case of system restart, cron will run the script appropriately and all the redis instances will start.

Considering redis is very stable, and does not stop unless there is a machine restart, we don't have to worry about receiving too many emails. :)

In the next part, we will see the script to monitor the overall health of the cluster. This can be useful in case one or more machines are down as as result of which the monitoring script of individual nodes cannot run on them and no alert is generated by them.

Happy redising. :)

Tuesday, 6 December 2016

handling and killing idle clients in redis

Considering redis is single threaded, it is best if an application maintains a single connection and uses it to query redis, because redis will process all requests one by one, anyways.

Many times in redis, we have to kill the idle connections in redis. This is useful if there is an application which creates a number of connections but does not close them.

There are some ways to avoid the idle connections in redis.

One of way is to set an idle timeout in redis.
This is done by setting the "timeout" value in seconds in redis.conf which requires a redis restart. Below is the entry in redis.conf

#Close the connection after a client is idle for N seconds (0 to disable)
timeout 3600

This can also be done by setting the timeout through config without redis restart by executing the below command.

src/redis-cli CONFIG SET timeout 3600

Considering, the config value set using the method 2 above is lost if the redis restarts, it is important to set it in redis.conf(method1) and then set it using config(method 2) so that redis does not require a restart and also the value persists even after there is a restart.

Note that, by default the value of timeout is 0, ie idle connections are everlasting and are never killed by redis server.

The Catch
The catch here is that the timeout is only applicable for normal clients, not for pub-sub ones, so that a pubsub client will not timeout even though its idle time exceeds the idle timeout defined. This is because the default behavior of pub-sub clients is to wait for events.  This is mentioned here.


Killing clients of a particular type in redis

Sometimes if you want to kill all the clients of a particular type, the below command may come in handy.

# kills all normal clients
src/redis-cli CLIENT KILL TYPE normal

# kills all pub-sub clients
src/redis-cli CLIENT KILL TYPE pubsub

# kills all slave clients
src/redis-cli CLIENT KILL TYPE slave

It is important to note that the above commands should be executed after due consideration. If our redis clients(like Jedis/Lettuce clients in java) reconnect, the above command will kill all connections and valid connections will reconnect, so that the application will be ok.

However if our clients don't reconnect, then we need to manually identify the clients/ip addresses which are expired/need to be terminated, and we need to kill those clients only.


Killing redis connections from an ip address

If we want to kill connections of a particular type from a particular ip address, then we can identify those connections using the CLIENT LIST command in redis, and kill them using the CLIENT KILL command in redis.

The below command will kill all pub-sub connections of a given ip address '10.150.20.30'

redis-cli -h 127.0.0.1 -p 6379 CLIENT LIST | grep 'sub=1' | grep '10.150.20.30' | awk  {'print $2'} | awk -F "=" {'print "CLIENT KILL ADDR " $2'} |  redis-cli -h 127.0.0.1

In the above, "redis-cli -h 127.0.0.1 -p 6379 CLIENT LIST | grep 'sub=1' | grep '10.150.20.30' " will get the pub-sub connections from redis client list command, then get "id=10.150.20.30:port" using awk  {'print $2'} and print client kill ADDR <ip:port> and pass it to redis-cli command.

This will ensure that all those connections satusfying the criteria are killed.

Similar to above, the below command can also be used, but is a bit slower.

redis-cli -h 127.0.0.1 -p 6379 CLIENT LIST | grep '10.150.20.30' | awk  {'print $2'} | awk -F "=" {'print "CLIENT KILL TYPE pubsub addr " $2'} |  redis-cli -h 127.0.0.1

The complete details of the redis client kill command can be found here.

:)



Friday, 2 December 2016

How to get the rdb location, and config file location in redis.

Sometimes when we login to a redis server, we can see the running redis on it, but we don't know where are the configuration files and the rdb backups files for the redis. In this post, we will see how to find out the location of various configuration files in redis.

The below are the important files in redis.


  1. redis.conf          => main configuration file in redis
  2. dump.rdb          => backup file generated by background save in redis
  3. sentinel.conf     => sentinel  configuration file, required only while running sentinel
  4. nodes.conf        => cluster configuration file, auto-generated by running redis in cluster mode.

Location of redis.conf
redis.conf is the major redis configuration file which has lots of very useful info and lots of highly configurable parameters.
Whether you are a developer who wants to dive more in redis or a redis admin, it is very important to read the file and try to understand the configuration params. See the sample here.

For finding out the location of redis.conf, we will use the INFO command.
info command gives a lot of very useful information about the redis server, one of which is the location of config file.

so, below command gives us the location of redis.conf file.

src/redis-cli -p 6379 info | grep 'config_file'

Location of sentinel.conf file
sentinel.conf file is only required if you are using a sentinel in redis. See the sample sentinel.conf file here. It is normally present in the same directory as redis.conf.
For finding out the sentinel.conf, we can execute the command on sentinel(note that sentinel by default runs on 26379 port while redis runs on 6379 port)

src/redis-cli -p 26379 info | grep 'config_file'

Location of dump.rdb file
dump.rdb file is the default file in which redis will save the data to disk if you enable rdb based persistence in the redis.conf file.
The location of dump.rdb can be obtained in either of the two ways

  • reading the "dir" value in redis.conf file found from the location of redis.conf
           cat redis.conf | grep "dir "
  • getting it from redis config
            Sometimes if we get the "dir " from the redis.conf file, it just shows the current directory( "./")
            In this case, it is better to get it from the redis config by executing the command below.

           src/redis-cli -p 6379 CONFIG GET dir

Note that #2 will only work if CONFIG command is not renamed/disabled in redis. 
If it is you will have to find out the renamed command and then use the renamed command instead of CONFIG.

Location of nodes.conf file
In a redis cluster, each cluster node/instance automatically generates a nodes.conf file where the view of the cluster with respect to that node is stored. 
It stores the cluster related information like

For all the nodes in the cluster, it stores
  1. the current node, and whether it is a master or a slave.
  2. no of clients connected to it.
  3. slot ranges for which it holds data(if it is a master)
  4. the epoch
the nodes.conf file is also stored in the same location as dump.rdb.
 

Sunday, 27 November 2016

single command to kill all the redis processes running on a system

A running redis instance can be stopped by executing the command SHUTDOWN. This command will save the running redis into a backup RDB file and stop the redis server. It also has the option of not saving the RDB file and just stopping the redis server by executing "SHUTDOWN NOSAVE".

src/redis-cli SHUTDOWN NOSAVE



Sometimes in redis, particularly when testing redis cluster, we have a number of redis instances running on a system.



When we need to stop all the instances of the redis cluster, we can execute "SHUTDOWN NOSAVE" on all the instances one by one.

A better way is to have a single for loop which does the job

for port in {7000..7005} do ; src/redis-cli -p $port SHUTDOWN NOSAVE ; done ;



What if SHUTDOWN command is renamed or disabled.

Many times, we prefer to disable or rename the SHUTDOWN command as described here, so that it cannot be used by a rogue client. If that is the case, redis servers will need to be stopped by killing the redis processes. Again, the normal way for this is to find out the processes using ps -ef | grep "redis-server" and then manually passing the process ids to kill command. However this can be done by a single command like the below.

ps -ef | grep 'redis-server' | awk '{ print $2}' | xargs kill -9

The above command will find out all the processes having 'redis-server' and get their Ids and pass them to the kill command.

:)

Friday, 25 November 2016

redis data types and basic commands for each in redis

Sometime when working with redis, we need to know the data type of a particular key in redis.
This is useful to know what kind of commands should be executed against that key.
A key can have any one of the below data types.
  1. String
  2. Hash
  3. List
  4. Set
  5. Sorted Set
  6. HLL(they are stored as string types internally)
The "TYPE" command in redis can be used to find out type of a particular key.

Lets say we have a redis instance and we find out the keys by executing the "keys *" command. For simplicity, i have separate types of keys in redis.


For each of these keys, the 'type' command will give the type of the key.


For performing basic view operations on each of the types of redis keys, below are the types and commands for each.

String
          GET  command in redis can be used to get the data in a string key.

 src/redis-cli GET mystringkey

    


Hash

         HGETALL command can be used to view the data in a hash. It returns a list of values, which consists of the key followed by the value of that key.  Like in the below, in the key "myhashkey", field1 has value value1 while field2 has value value2. Hashes are normally stored to store an object which can have many fields, like a user can have various fields like name, age, gender, etc, so that the key can be user:<userid>, fields can be name, age, gender etc and values can be 'Jurgen', 50, 'm'

src/redis-cli HGETALL myhashkey



List
        Lists in redis are infact linked list, and data can be pushed or removed from either ends.    LRANGE command in redis can be used to view the data in a list, while LLEN can be used to get the length of the list.

src/redis-cli LLEN mylistkey
src/redis-cli LRANGE mylistkey -100 100

     


Set
          Set in redis consists of unique elements. The values in set can be viewed using the SMEMBERS command. 

src/redis-cli SMEMBERS mysetkey




Zset (Sorted Set)
          Sorted Set in redis contains the string data along with a score. Finding out any entry for a particular score takes log N time where N is the number of entries stored in a key. ZRANGEBYSCORE command can be used to view the data in a sorted set. Sorted sets are normally used for storing data involving some leaderboard where each user has a particular score. Another use case where sorted sets can be useful is in reporting where the data can be stored as value and date in YYMMDD format can be stored as a score. To get the data between two dates, ZRANGEBYSCORE query can be used.

src/redis-cli ZRANGEBYSCORE mysortedsetkey -INF +INF WITHSCORES
src/redis-cli ZRANGEBYSCORE mysortedsetkey 90 100 WITHSCORES



HLL
         HyperLogLog(HLL) in redis is used to calculate the estimate the number of unique values in a key without storing the actual values!!! Sounds too good to be true, go through this link to understand more. HLLs are stored as strings, so you can do the normal GET operation on it, but to approximately calculate the number of unique keys in a key, the command which is used is PFCOUNT

src/redis-cli PFCOUNT myhllkey


So what happens when we try to try to get a key using a separate command that what is expected for that type, say when we try to GET a hash key instead of HGETALL?
We get the error, "WRONGTYPE Operation against a key holding the wrong kind of value" which is expected since we should be using HGETALL command to get the data from a hash instead of GET.

:)