Shantanu's Blog

You can install and launch redis server in just one command...

 docker run -v /myredis/conf/redis.conf:/usr/local/etc/redis/redis.conf --name myredis redis redis-server /usr/local/etc/redis/redis.conf

This will use redis configuration file from base image file /myredis/conf

http://cassandra.apache.org

Entity–attribute–value model (EAV) also known as vertical database model is a data model to describe entities where the number of attributes are not known or are frequently changing. This is fundamentally different than SQL model and better than key-value store like Redis. Installation Cassandra is straight forward and commands are similar to redis set / get.

create keyspace abc;

use abc;

create column family users with comparator=AsciiType and column_metadata=[{column_name: full_name, validation_class: AsciiType}, {column_name: state, validation_class: AsciiType, index_type: KEYS}, {column_name: birth_date, validation_class: LongType, index_type: KEYS}];

assume users VALIDATOR as ascii;  
assume users COMPARATOR as ascii;  
assume users KEYS as ascii;  
assume users SUB_COMPARATOR as ascii;  

set users[bsanderson][full_name] = 'Brandon Sanderson';
set users[bsanderson][birth_date] = 1975;
set users[prothfuss][full_name] = 'Patrick Rothfuss';
set users[prothfuss][birth_date] = 1973;
set users[htayler][full_name] = 'Howard Tayler';
set users[htayler][birth_date] = 1968;

set users[bsanderson][state] = 'UT';
set users[prothfuss][state] = 'WI';
set users[htayler][state] = 'UT';

get users where birth_date = 1973;
get users where state = 'UT';
get users where state = 'UT' and birth_date > 1970;

Redis is an open source, advanced key-value store. Basically, if you can map a use case to Redis and discover you aren't at risk of running out of RAM by using Redis there is a good chance you should probably use Redis.

A few examples:

# You can sort by create timestamp if that is stored in epoch time.

> HMSET users:1 firstname 'john' lastname 'smith' created 1319729878
"OK"
> HMSET users:2 firstname 'Jane' lastname 'Forbes' created 1319729910
"OK"
> sadd users 1
true
> sadd users 2
true
> sort users get users:*->firstname by users:*->created
["john","Jane"]
> sort users get users:*->firstname by users:*->created desc
["Jane","john"]
_____

APPLE: RED ROUND FRUIT would map to the following inserts:

SADD RED:ROUND:FRUIT APPLE
SADD :ROUND:FRUIT APPLE
SADD RED::FRUIT APPLE
SADD RED:ROUND: APPLE
SADD RED:: APPLE
SADD :ROUND: APPLE
SADD ::FRUIT APPLE
SADD ::: APPLE

_____

redis 127.0.0.1:6379> SADD P1:YELLOW MANGO
(integer) 1
redis 127.0.0.1:6379> SADD P2:TASTE MANGO
(integer) 1
redis 127.0.0.1:6379> SADD P3:FRUIT MANGO
(integer) 1
redis 127.0.0.1:6379> SINTER P1:YELLOW P2:TASTE
1) "MANGO"

_____

redis 127.0.0.1:6379> HMSET id:4532143215432 username davejlong email dave@davejlong.com
OK
redis 127.0.0.1:6379> HMSET user:davejlong id 4532143215432 email dave@davejlong.com
OK
redis 127.0.0.1:6379> HGET id:4532143215432 username
"davejlong"
redis 127.0.0.1:6379> HGET user:davejlong id
"4532143215432"
redis 127.0.0.1:6379> HMGET user:davejlong email id
1) "dave@davejlong.com"
2) "4532143215432"
redis 127.0.0.1:6379> DEL user:davejlong
(integer) 1
redis 127.0.0.1:6379> DEL id:4532143215432
(integer) 1

_____


// set and expiry can be included in single command. The following key will expire after 48 hours
setex cb_num:r 172800 date_time

// Hash can be assgined to a key. In the following example, we can count zone + customerID
hincryby r:1633:1634 2012:03:15 12:55:00 3
hincryby r:1633:1634 noads 5

src/redis-cli hgetall r:1633:1634
1) "2012-03-15 12-50-00"
2) "188"
1) "2012-03-15 12-55-00"
4) "200"

// sorted sets allow to store and retrieve data more efficiently

zadd RequestSet 2012-03-15 r:zone:customerID

src/redis-cli ZRANGE RequestSet 0 1 WITHSCORES
1) "r:1008:0"
2) "20120310"
3) "r:1008:10422"
4) "20120310"
_____

Redis can be easily used in shell script or at command prompt.
It will be useful for storing the standard out data of other commands.

In the following example, you will get a break line (backslash+n) in the output as shown.

$ echo 'testme one more word new line' | ./src/redis-cli -x set mytest
OK
$ ./src/redis-cli get mytest
"testme one more word new line\"

That is added by the "echo" command. You can use echo -n to avoid that extra break line:
$ echo -n 'testme one more word new line' | ./src/redis-cli -x set mytest
_____

use the -n argument to choose DB number.

[root@server]# echo -n "testing" | /pat/to/redis/src/redis-cli -x -n 4 set my_pass > /dev/null 2>&1

[root@server]# /pat/to/redis/src/redis-cli --raw -n 4 get my_pass
testing

_____

We can write a shell script to import the data from mysql or use a tool like "awk"

#!/bin/sh
mysql pdb_name -Bse"select comments, concat(fee, status) as myfee from company limit 100000;" | while read -r account myfee
do
echo -n $myfee | /home/redis-2.2.12/src/redis-cli -h 10.10.10.100 -p 6379 -x set $account > /dev/null 2>&1

done
_____

mysql pdb_name -Bse"select comments, concat(fee, status) as myfee from company limit 100000;" | awk "{print \"set \" \$1 \" \" \$2}" | /home/redis-2.2.12/src/redis-cli -h 10.10.10.100 -p 6379 > /dev/null 2>&1

_____

# time mysql test -Bse"select concat(add_id,':', nd_id) from data_summary_hourly" | awk "{print \"incr \" \$1 }" | /home/shantanu/redis-2.4.8/src/redis-cli -h localhost -p 6379 > /dev/null 2>&1

real    23m0.207s
user    5m15.595s
sys    5m19.612s
_____

redisdump.sh is the script to dump all data from redis instance.

time sh redisdump.sh | sed 'N;s/\n/ /' | sed 's/KEY //' | sed 's/string //' > todump.txt

This will not work if you have tens of millions of records! but useful for small size databases.

#!/bin/sh
mypath="/home/shantanu/redis-2.4.8/src"
$mypath/redis-cli keys "*" > keys.txt
cat keys.txt | awk '{ printf "type %s\n", $1 }' | $mypath/redis-cli > types.txt

paste -d'|' keys.txt types.txt | awk -F\| '

$2 == "string" { printf "echo \"KEY %s %s\"\nget %s\n", $1, $2, $1 }

$2 == "list" || $2 == "set" { printf "echo \"KEY %s %s\"\nsort %s by nosort\n", $1, $2, $1 }

$2 == "hash" { printf "echo \"KEY %s %s\"\nhgetall %s\n", $1, $2, $1 }

$2 == "zset" { printf "echo \"KEY %s %s\"\nzrange %s 0 -1 withscores\n", $1, $2,$1 }

' | $mypath/redis-cli --raw

_____

redis> select 3
All subsequent commands will then use database 3, until you issue another SELECT

redis> flushdb
drop all the data in a single database

redis> FLUSHALL
drop all the data in all databases

in redis.conf — by default, it is set to 16. Simply set it to a higher number if you need more:
databases 42

_____

// cd to the redis installation
cd /home/shantanu/redis-2.4.8

// disable the save command, you will need to manually save the data to disk:
# sed -i 's/^save/#save/' redis.conf

// save the server log to a file instead of standard out
# sed -i 's/stdout/redis.log/' redis.conf

// increase the number of databases from default 16
sed -i 's/databases\ [0-9]*/databases\ 32/' redis.conf

// save data to disk, check redis info
# time src/redis-cli bgsave
# src/redis-cli info

// If you get a warning in redis log, you need to change the memory setting as shown below:
# echo 'vm.overcommit_memory = 1' >> /etc/sysctl.conf
# sysctl vm.overcommit_memory=1

// start redis server with the new config file:
# src/redis-server redis.conf &

// make current redis slave of master
// add slave from command line of redis config file
slaveof ec2-184-73-130-46.compute-1.amazonaws.com 6379

// make sure to add config file while stating redis server
/root/redis-2.6.12/src/redis-server /root/redis-2.6.12/redis.conf

// start redis on server startup
If there is an /etc/rc.local you can add an '/path/to/redis-server' line in there

// redis benchmark
./src/redis-benchmark -n 100 -r 100 -q -h 11.22.33.44 -p 6379

// general log that saves all commands received by redis
nohup ./src/redis-cli monitor > /mnt/bigDisk/nohup.out
time tail -100000 /mnt/todel/nohup.out | awk -F'.' '{print $1}' | sort | uniq -c

// If you have an error message saying "Error allocating resources for the client"
You will have to increase ulimit and also update ae.h file:
ulimit -n 10240 ; /usr/local/bin/redis-server

// If you get an error "gcc: Command not found" then install the required package.
yum install gcc

// If you get an error "Newer version of jemalloc required" you need to run...
make distclean

in file ae.h you have:
#define AE_SETSIZE (1024*10) /* Max number of fd supported */
You may want to try to increase this limit and recompile Redis.
_____

## Use docker to connect to any redis server:
docker run -it redis redis-cli -h  myredis.synfmnx.0001.use1.cache.amazonaws.com