RHEL 6.x on Rackspace Cloud and Dedicated hardware experiencing Redis Timeouts
- by zhallett
I just recently set up a mixture of RHEL 6.1 Rackspace cloud hosts and RHEL 6.2 dedicated hosts using Rackconnect. I am experiencing intermittent Redis timeouts from within our Rails 3.2.8 app with Redis 2.4.16 running on the RHEL 6.2 dedicated hosts. There is no network latency or packet loss. Also there are no errors on any interfaces on our cloud or dedicated servers or on the managed firewall from Rackspace.
When Redis timesout, there is nothing logged within redis even though it is set up to do debug logging. The only error we receive is from Airbrake saying there was a Redis timeout.
Network topology:
RHEL 6.1 cloud hosts <--> Alert logic IDS <--> Cisco ASA 5510 <--> RHEL 6.2 dedicated hosts
(web nodes) (two way NAT) (db hosts running redis)
Ping from db host to web host:
64 bytes from 10.181.230.180: icmp_seq=998 ttl=64 time=0.520 ms
64 bytes from 10.181.230.180: icmp_seq=999 ttl=64 time=0.579 ms
64 bytes from 10.181.230.180: icmp_seq=1000 ttl=64 time=0.482 ms
--- web1.xxxxxx.com ping statistics ---
1000 packets transmitted, 1000 received, 0% packet loss, time 999007ms
rtt min/avg/max/mdev = 0.359/0.535/5.684/0.200 ms
Ping from web host to db host:
64 bytes from 192.168.100.26: icmp_seq=998 ttl=64 time=0.544 ms
64 bytes from 192.168.100.26: icmp_seq=999 ttl=64 time=0.452 ms
64 bytes from 192.168.100.26: icmp_seq=1000 ttl=64 time=0.529 ms
--- data1.xxxxxx.com ping statistics ---
1000 packets transmitted, 1000 received, 0% packet loss, time 999017ms
rtt min/avg/max/mdev = 0.358/0.499/6.120/0.201 ms
Redis config:
daemonize yes
pidfile /var/run/redis/6379/redis_6379.pid
port 6379
timeout 0
loglevel debug
logfile /var/lib/redis/log
syslog-enabled yes
syslog-ident redis-6379
syslog-facility local0
databases 16
save 900 1
save 300 10
save 60 10000
rdbcompression yes
dbfilename dump-6379.rdb
dir /var/lib/redis
maxclients 10000
maxmemory-policy volatile-lru
maxmemory-samples 3
appendfilename appendonly-6379.aof
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
slowlog-log-slower-than 10000
slowlog-max-len 1024
vm-enabled no
vm-swap-file /tmp/redis.swap
vm-max-memory 0
vm-page-size 32
vm-pages 134217728
vm-max-threads 4
hash-max-zipmap-entries 512
hash-max-zipmap-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes
Redis-cli info:
redis-cli info
redis_version:2.4.16
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
gcc_version:4.4.6
process_id:4174
uptime_in_seconds:79346
uptime_in_days:0
lru_clock:1064644
used_cpu_sys:13.08
used_cpu_user:19.81
used_cpu_sys_children:1.56
used_cpu_user_children:7.69
connected_clients:167
connected_slaves:0
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:6
used_memory:15060312
used_memory_human:14.36M
used_memory_rss:22061056
used_memory_peak:15265928
used_memory_peak_human:14.56M
mem_fragmentation_ratio:1.46
mem_allocator:jemalloc-3.0.0
loading:0
aof_enabled:0
changes_since_last_save:166
bgsave_in_progress:0
last_save_time:1352823542
bgrewriteaof_in_progress:0
total_connections_received:286
total_commands_processed:507254
expired_keys:0
evicted_keys:0
keyspace_hits:1509
keyspace_misses:65167
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:690
vm_enabled:0
role:master
db0:keys=6,expires=0
edit 1: add redis-cli info output
