Permalink: 2014-07-09 10:04:00 by ning in redis tags: all

1   Redis latency problems troubleshooting

主要来自redis 作者的文章:

这篇文章介绍了很多工具, iostat, vmstat


redis-cli --latency -h `host` -p `port`


$ ./redis-cli --intrinsic-latency 100
Max latency so far: 573 microseconds.
Max latency so far: 695 microseconds.
Max latency so far: 919 microseconds.
Max latency so far: 1606 microseconds.
Max latency so far: 3191 microseconds.
Max latency so far: 9243 microseconds.
Max latency so far: 9671 microseconds.
Here we have an intrinsic latency of 9.7 milliseconds: this means that we can't ask better than that to Redis.



The typical latency of a 1 GBits/s network is about 200 us, while the latency with a Unix domain socket can be as low as 30 us.

  • 不要用虚拟机.
  • 长连接
  • use Unix domain sockets
  • MSET/MGET/pipeline
On Linux, some people can achieve better latencies by playing with :
process placement (taskset), cgroups, real-time priorities (chrt), NUMA configuration (numactl), or by using a low-latency kernel.

Please note vanilla Redis is not really suitable to be bound on a single CPU core. Redis can fork background tasks that can be extremely CPU consuming like bgsave or AOF rewrite. These tasks must never run on the same core as the main event loop.

from Redis 2.4 we use threads in Redis in order to perform some slow I/O operations in the background, mainly related to disk I/O,

1.2   2.slow commands


  • 避免keys, sort, lrem, sunion

1.3   3.fork

  • 内存page为4K, 24 GB Redis instance requires a page table of 24 GB / 4 KB * 8 = 48 MB
  • Xen 上更加糟糕.


  • 使用较小的实例.(<10G)

1.4   4.swapping


ning@ning-laptop ~/test$ cat /proc/18941/smaps  | grep Swap
Swap:                  0 kB
Swap:                  0 kB
Swap:                  0 kB
Swap:                  0 kB
Swap:                  0 kB
Swap:                156 kB

1.5   5.aof and disk I/O

write() + fdatasync()

Both the write(2) and fdatasync(2) calls can be source of latency. For instance write(2) can block both when there is a system wide sync in progress, or when the output buffers are full and the kernel requires to flush on disk in order to accept new writes.

close 也会导致flush.

fdatasync 可能从几个ms到几s. 所以redis2.4 尽可能在另一个线程里面做fdatasync.

When appendfsync is set to the value of no Redis performs no fsync. In this configuration the only source of latency can be write(2).


sudo strace -p $(pidof redis-server) -T -e trace=fdatasync -f

因为fdatasync 是在另一个线程, 所以需要加-f

However since write(2) is also used in order to write data to the client sockets this will likely show too many things unrelated to disk I/O. Apparently there is no way to tell strace to just show slow system calls so I use the following command:

strace -f -p $(pidof redis-server) -T -e trace=fdatasync,write 2>&1 | grep -v '0.0' | grep -v unfinished


  • appendfsync no
  • Using an SSD disk can help as well,

用这个, 我们在做copy 的时候, 可以观察到:

[pid 24734] write(42, "*4\r\n$5\r\nhmset\r\n$37\r\np-lc-d687791"..., 272475) = 272475 <0.036430>
[pid 24738] <... fdatasync resumed> )   = 0 <2.030435>
[pid 24738] <... fdatasync resumed> )   = 0 <0.012418>
[pid 24734] write(42, "*4\r\n$5\r\nHMSET\r\n$37\r\np-lc-6787211"..., 73) = 73 <0.125906>
[pid 24738] <... fdatasync resumed> )   = 0 <4.476948>
[pid 24734] <... write resumed> )       = 294594 <2.477184>   (2.47s)


$ ./_binaries/redis-cli --latency-history -h -p 2000
min: 0, max: 223, avg: 1.24 (1329 samples) -- 15.01 seconds range
min: 0, max: 2500, avg: 3.46 (1110 samples) -- 15.00 seconds range   (这里观察到2.5s)
min: 0, max: 5, avg: 1.01 (1355 samples) -- 15.01 seconds range

watchdog 输出:

[24734] 07 Jul 10:54:41.006 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
[24734 | signal handler] (1404701682)
bin/redis-server *:2000(logStackTrace+0x4b)[0x443bdb]
bin/redis-server *:2000(flushAppendOnlyFile+0x76)[0x43f616]
bin/redis-server *:2000(serverCron+0x325)[0x41b5b5]
bin/redis-server *:2000(aeProcessEvents+0x2b2)[0x416a22]
bin/redis-server *:2000(aeMain+0x3f)[0x416bbf]
bin/redis-server *:2000(main+0x1c8)[0x41dcd8]
bin/redis-server *:2000[0x415b1a]
[24734 | signal handler] (1404701682) --------

1.6   6.expires

if the database contains has many many keys expiring in the same second, and this keys are at least 25% of the current population of keys with an expire set,

1.7   工具

1.7.1   watchdog

redis2.6 自带. 通过时钟中断定时触发检查:

CONFIG SET watchdog-period 500

会在发现有大延迟时, 打印日志.

1.7.2   最新的latancy监控


latency-monitor-threshold 100

1.8   小结

2, 3, 5 我们都遇到了.