How to debug memcached "SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY" errors?
- by Jevgenij Evll
I have a two server memcached setup. When memcached write fails, I receive an email notification. About once per day "SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY" error comes and I have no idea how to find the reason.
I am using PHP Memcached client.
I am not using too long keys. I tried adding -v flag, but it does not help, the log remains empty.
If I include output of getStats to the error notification, I receive the following info:
Array
(
[192.168.0.3:11211] => Array
(
[pid] => 28167
[uptime] => 3671962
[threads] => 4
[time] => 1358714713
[pointer_size] => 64
[rusage_user_seconds] => 24516
[rusage_user_microseconds] => 130981
[rusage_system_seconds] => 86246
[rusage_system_microseconds] => 675512
[curr_items] => 1616352
[total_items] => 118339822
[limit_maxbytes] => 2684354560
[curr_connections] => 8
[total_connections] => 78108681
[connection_structures] => 356
[bytes] => 981522779
[cmd_get] => 1561752945
[cmd_set] => 158718324
[get_hits] => 1383072575
[get_misses] => 178680370
[evictions] => 0
[bytes_read] => 138113231690
[bytes_written] => 1091741700765
[version] => 1.4.15
)
[192.168.0.4:11211] => Array
(
[pid] => -1
[uptime] => 0
[threads] => 0
[time] => 0
[pointer_size] => 0
[rusage_user_seconds] => 0
[rusage_user_microseconds] => 0
[rusage_system_seconds] => 0
[rusage_system_microseconds] => 0
[curr_items] => 0
[total_items] => 0
[limit_maxbytes] => 0
[curr_connections] => 0
[total_connections] => 0
[connection_structures] => 0
[bytes] => 0
[cmd_get] => 0
[cmd_set] => 0
[get_hits] => 0
[get_misses] => 0
[evictions] => 0
[bytes_read] => 0
[bytes_written] => 0
[version] =>
)
)
