twemproxy代码分析

Redis 作者:Twemproxy, a Redis proxy from Twitter

http://antirez.com/news/44

• 计划中的redis-cluster: Multiple instances is a share-nothing architecture.
• 2.6 中实现了Redis Sentinel, 即将实现 partial resynchronization
• Twemproxy 是 single-threaded proxy
• What's awesome about Twemproxy is that it can be configured both to disable nodes on failure, and retry after some time
• 失败时, 可以disable 或者 retry - 作为data store : retry - 作为cache: disable(node-ejection)

同样的一组后端, 可以配成两种:

redis1:
  listen: 0.0.0.0:9999
  redis: true
  hash: fnv1a_64
  distribution: ketama
  auto_eject_hosts: true
  timeout: 400
  server_retry_timeout: 2000
  server_failure_limit: 1
  servers:
   - 127.0.0.1:6379:1
   - 127.0.0.1:6380:1
   - 127.0.0.1:6381:1
   - 127.0.0.1:6382:1

redis2:
  listen: 0.0.0.0:10000
  redis: true
  hash: fnv1a_64
  distribution: ketama
  auto_eject_hosts: false
  timeout: 400
  servers:
   - 127.0.0.1:6379:1
   - 127.0.0.1:6380:1
   - 127.0.0.1:6381:1
   - 127.0.0.1:6382:1

限制:

1. 不支持 mset, transaction:

   I think that twemproxy do it right, not supporting multiple keys commands nor transactions. Currently is AFAIK even more strict than Redis Cluster that instead allows MULTI/EXEC blocks if all the commands are about the same key.
   

2. 对mget, mdel 支持, 性能待测试:

   However there is some support for commands with multiple keys. MGET and DEL are handled correctly. Interestingly MGET will split the request among different servers and will return the reply as a single entity. This is pretty cool even if I don't get the right performance numbers with this feature (see later).
   
   So I expected to see almost the same numbers with an MGET as I see when I run the MGET against a single instance
   but I get only 50% of the operations per second. Maybe it's the time to reconstruct the reply, I'm not sure.
   
   mget 只有50%
   

3. 不支持EVAL,

4. 错误处理: (antirez 测试还挺认真)

   • non supported command: closes the connection.
   • sending just a "GET": hang

希望: 支持HA

• 能直接做主从切换.
• 配置热加载也可以啊~ (checking the Sentinel configuration regularly to upgrade the servers table if a failover happened.)

性能: 好!!!

• This Thing Is Fast. Really fast, it is almost as fast as talking directly with Redis. I would say you lose 20% of performances at worst.
• MGET可以优化 (目前性能降低50%)

结论: - I strongly suggest Redis users to give it a try.

人们的讨论:

• 对于MGET: The response time will then be at least as slow as the slowest node

存储分片和Twemproxy核心解读

http://www.wzxue.com/%E5%AD%98%E5%82%A8%E5%88%86%E7%89%87%E5%92%8Ctwemproxy%E6%A0%B8%E5%BF%83%E8%A7%A3%E8%AF%BB/

antirez(Redis作者)写过一篇对twemproxy的介绍http://antirez.com/news/44, 他认为twemproxy是目前Redis 分片管理的最好方案，虽然antirez的Redis cluster正在实现并且对其给予厚望

• 涉及到三个重要的结构:server, connection, message。

  • 每个server其实就是一个后端的缓存服务程序
  • connection在Twemproxy中非常重要，它分为三种类型的connection:proxy，client和server
  • struct msg是连接建立后的消息内容发送载体，这个复杂的msg结构很大程度是因为需要实现pipeline的效果，多个msg属于同一个conn，conn通过接收到内容解析来发现几个不同的msg。

• Twemproxy的架构比较清晰，对Twemproxy源码印象较深的是对logging的合理布局和错误处理的清晰，这是第一次看大公司开源出来的代码，非常重视logging和错误处理。

• 我的fork: 由于Twitter开源的Twemproxy直接使用epoll驱动，导致其他不支持epoll的系统无法使用，因此我fork了一个版本，加入了kqueue支持，让FreeBSD和Mac os x能够成功编译运行

这位同学2013.10 还是一个大四学生. github 上人气挺高: https://github.com/yuyuyu101

redis-twemproxy-benchmark

http://blog.jpush.cn/redis-twemproxy-benchmark/

• twemproxy 可以开http监控端口: http://ip:22222 json格式
• 性能基本和单台一样.
• 多个twemproxy 实例, 性能可以更好.
• 不支持除mget，del之外的redis批处理命令，如取多个集合交集等等
• 不支持脚本eval

这个文章总结不错

http://cloudaice.com/twemproxy-explore/

• redis-proxy

  • 使用node写的redis代理层。
  • 支持主从节点的失败处理（可以仔细研究）
  • 测试后发现性能为原生的1/3

• twemproxy

  • 支持失败节点自动删除

    • 可以设置重新连接该节点的时间
    • 可以设置连接多少次之后删除该节点
    • 该方式适合作为cache存储

  • 支持设置HashTag

    • 通过HashTag可以自己设定将两个KEYhash到同一个实例上去。

  • 减少与redis的直接连接数

    • 保持与redis的长连接
    • 可设置代理与后台每个redis连接的数目

  • 自动分片到后端多个redis实例上

    • 多种hash算法（部分还没有研究明白)
    • 可以设置后端实例的权重

  • 避免单点问题

    • 可以平行部署多个代理层.client自动选择可用的一个

  • 支持redis pipelining request

  • 支持状态监控

    • 可设置状态监控ip和端口，访问ip和端口可以得到一个json格式的状态信息串
    • 可设置监控信息刷新间隔时间

  • 高吞吐量

    • 连接复用，内存复用。
    • 将多个连接请求，组成reids pipelining统一向redis请求。

配置部署建议: 编译时候打开logging模块。

event

./event
./event/Makefile.am
./event/nc_epoll.c
./event/nc_event.h
./event/nc_evport.c
./event/nc_kqueue.c

请求处理

./nc_proxy.c
./nc_proxy.h

./nc_connection.c
./nc_connection.h
./nc_client.c
./nc_client.h
./nc_server.c
./nc_server.h

./nc_message.c
./nc_message.h
./nc_request.c
./nc_response.c

./nc_mbuf.c
./nc_mbuf.h

main

./nc.c, 处理daemon, pidfile, args, config, 最后:

nc_run(struct instance *nci)
{
    ctx = core_start(nci);
    for (;;) {
        status = core_loop(ctx);
        if (status != NC_OK) {
            break;
        }
    }
    core_stop(ctx);
}

os utils

./nc_conf.c
./nc_conf.h
./nc_core.c
./nc_core.h
./nc_log.c
./nc_log.h
./nc_signal.c
./nc_signal.h
./nc_stats.c
./nc_stats.h

ds utils

./nc_array.c
./nc_array.h
./nc_util.c
./nc_util.h
./nc_queue.h
./nc_rbtree.c
./nc_rbtree.h
./nc_string.c
./nc_string.h

协议

./proto/nc_proto.h
./proto/nc_memcache.c
./proto/nc_redis.c

hash

./hashkit/nc_crc16.c
./hashkit/nc_crc32.c
./hashkit/nc_fnv.c
./hashkit/nc_hashkit.h
./hashkit/nc_hsieh.c
./hashkit/nc_jenkins.c
./hashkit/nc_ketama.c
./hashkit/nc_md5.c
./hashkit/nc_modula.c
./hashkit/nc_murmur.c
./hashkit/nc_one_at_a_time.c
./hashkit/nc_random.c

epool 实现

event_base_create:

struct event_base *
event_base_create(int nevent, event_cb_t cb)
{
    struct event_base *evb = nc_alloc(sizeof(*evb));
    int ep = epoll_create(nevent);
    struct epoll_event *event = nc_calloc(nevent, sizeof(*event));

    evb->ep = ep;
    evb->event = event;
    evb->nevent = nevent;
    evb->cb = cb;

    return evb;
}

int
event_add_in(struct event_base *evb, struct conn *c)
{
    struct epoll_event event;
    event.events = (uint32_t)(EPOLLIN | EPOLLET);
    event.data.ptr = c;

    status = epoll_ctl(evb->ep, EPOLL_CTL_MOD, c->sd, &event);
}

int
event_wait(struct event_base *evb, int timeout)
{
    int ep = evb->ep;
    struct epoll_event *event = evb->event;
    int nevent = evb->nevent;

    for (;;) {
        int i, nsd;

        nsd = epoll_wait(ep, event, nevent, timeout);
        if (nsd > 0) {
            for (i = 0; i < nsd; i++) {
                struct epoll_event *ev = &evb->event[i];
                uint32_t events = 0;

                log_debug(LOG_VVERB, "epoll %04"PRIX32" triggered on conn %p",
                          ev->events, ev->data.ptr);

                if (ev->events & EPOLLERR) {
                    events |= EVENT_ERR;
                }

                if (ev->events & (EPOLLIN | EPOLLHUP)) {
                    events |= EVENT_READ;
                }

                if (ev->events & EPOLLOUT) {
                    events |= EVENT_WRITE;
                }

                if (evb->cb != NULL) {
                    evb->cb(ev->data.ptr, events);
                }
            }
            return nsd;
        }
    }
}

每次调用event_wait, 如果没有事件则一直等, 如果有事件, 调用回调，并返回.

注意: 这里用的都是边缘触发 EPOLLET (redis本身用的是水平触发)

上层对event_create 的使用

core_ctx_create(){
    /* initialize server pool from configuration */  pool 是一种隔离单位
    status = server_pool_init(&ctx->pool, &ctx->cf->pool, ctx);

    ctx->evb = event_base_create(EVENT_SIZE, &core_core); //这里有个EVENT_SIZE 是1024, core_core 是一个基础回调, 处理读写.
    status = server_pool_preconnect(ctx);
    status = proxy_init(ctx);
}

ctx->evb = event_base_create(EVENT_SIZE, &core_core); //这里有个EVENT_SIZE 是1024, 在epool里面没用, 设置的回调 core_core 是一个基础回调, 处理读写.

event_add_in里面, data.ptr 是一个conn数据结构,

所以，有事件的时候, 调用 core_core(conn, events), 这里events 说明是读还是写事件.

接下来的 proxy_init:

rstatus_t
proxy_each_init(void *elem, void *data) {
    struct server_pool *pool = elem;
    struct conn *p;

    p = conn_get_proxy(pool); //获得一个proxy类型的 conn, 它的回调设置: conn->recv = proxy_recv;
    status = proxy_listen(pool->ctx, p); //做bind/listen/set noblocking/加到epool
}

这里设置的回调 proxy_recv, 就是做accept, 建立与客户端的连接:

rstatus_t
proxy_recv(struct context *ctx, struct conn *conn)
{
    conn->recv_ready = 1;
    do {
        status = proxy_accept(ctx, conn);
        if (status != NC_OK) {
            return status;
        }
    } while (conn->recv_ready);
    return NC_OK;
}

用proxy_accept 与客户端建立连接, 获得client 类型的conn, 并且加入到事件队列:

static rstatus_t
proxy_accept(struct context *ctx, struct conn *p) {
    rstatus_t status;
    struct conn *c;
    int sd;

    for (;;) {//死循环去做. 不带sleep!!
        sd = accept(p->sd, NULL, NULL);
        if (sd < 0) {
            ...
            return NC_ERROR;
        }
        break;
    }

    c = conn_get(p->owner, true, p->redis); //获得一个client类型的连接
    c->sd = sd;

    status = nc_set_nonblocking(c->sd);
    status = event_add_conn(ctx->evb, c);

    return NC_OK;
}

上层对event_wait的使用

rstatus_t
core_loop(struct context *ctx)
{
    nsd = event_wait(ctx->evb, ctx->timeout);
    ...
}

超清晰的主循环:

static void
nc_run(struct instance *nci)
{
    rstatus_t status;
    struct context *ctx;

    ctx = core_start(nci);
    if (ctx == NULL) {
        return;
    }

    /* run rabbit run */
    for (;;) {
        status = core_loop(ctx);
        if (status != NC_OK) {
            break;
        }
    }

    core_stop(ctx);
}

和libevent 对比

从数据结构, 函数命令来看, 受到libevnet和nginx的强烈影响.

(redis是自己的事件框架, memcached 就是用libevent作为事件框架)

libevent 的结构大概是这样的(不准确)

event, 类似连接的概念:

struct event {
    TAILQ_ENTRY (event) ev_next;
    TAILQ_ENTRY (event) ev_active_next;
    TAILQ_ENTRY (event) ev_signal_next;
    unsigned int min_heap_idx;  /* for managing timeouts */

    struct event_base *ev_base;

    int ev_fd;
    short ev_events;

event_base是暴露给外面的统一接口:

struct event_base {
    const struct eventop *evsel;
    void *evbase;
    int event_count;        /* counts number of total events */
    int event_count_active; /* counts number of active events */
    ...
}

用法:

//获得listen_fd
listen_fd = network_server_socket(cfg->listen_host, cfg->listen_port);

struct event *ev_accept;
ev_accept = event_new(g_server.event_base, listen_fd, EV_READ | EV_PERSIST, on_accept, NULL);
event_add(ev_accept, NULL);

server_pool和server.

在 server_pool.h 中注释写的非常清楚, 还有示意图:

*  +-------------+
*  |             |<---------------------+
*  |             |<------------+        |
*  |             |     +-------+--+-----+----+--------------+
*  |   pool 0    |+--->|          |          |              |
*  |             |     | server 0 | server 1 | ...     ...  |
*  |             |     |          |          |              |--+
*  |             |     +----------+----------+--------------+  |
*  +-------------+                                             //
*  |             |
*  |             |
*  |             |
*  |   pool 1    |
*  |             |
*  |             |
*  |             |
*  +-------------+
*  |             |
*  |             |
*  .             .
*  .    ...      .
*  .             .
*  |             |
*  |             |
*  +-------------+
*            |
*            |
*            //

看看twemproxy的配置:

s1:
  listen: 127.0.0.1:22124
  servers:
   - 127.0.0.1:11214:1
   - 127.0.0.1:11215:1
   - 127.0.0.1:11216:1
   - 127.0.0.1:11217:1
s2:
  listen: 127.0.0.1:22125
   - 127.0.0.1:11218:1
   - 127.0.0.1:11219:1

这样的配置就对应着类似这样的结构:

conn, msg, mbuf

int event_add_in(struct event_base *evb, struct conn *c);

struct conn {
    TAILQ_ENTRY(conn)  conn_tqe;      /* link in server_pool / server / free q */
    void               *owner;        /* connection owner - server_pool / server */
                                     //对于client和proxy, conn->owner 是server_pool 对象
                                     //对于server, conn->owner 是一个server 对象
    ..
    struct msg_tqh     imsg_q;        /* incoming request Q */
    struct msg_tqh     omsg_q;        /* outstanding request Q */

    conn_recv_t        recv;          /* recv (read) handler */
    conn_recv_next_t   recv_next;     /* recv next message handler */
    conn_recv_done_t   recv_done;     /* read done handler */
    conn_send_t        send;          /* send (write) handler */
    conn_send_next_t   send_next;     /* write next message handler */
    conn_send_done_t   send_done;     /* write done handler */
    conn_close_t       close;         /* close handler */
    conn_active_t      active;        /* active? handler */
    ...
};

struct msg {
    TAILQ_ENTRY(msg)     c_tqe;           /* link in client q */
    TAILQ_ENTRY(msg)     s_tqe;           /* link in server q */
    TAILQ_ENTRY(msg)     m_tqe;           /* link in send q / free q */

    uint64_t             id;              /* message id */
    struct msg           *peer;           /* message peer */
    struct conn          *owner;          /* message owner - client | server */

    struct rbnode        tmo_rbe;         /* entry in rbtree */

    struct mhdr          mhdr;            /* message mbuf header */
    uint32_t             mlen;            /* message length */
    ...
};

小结

• server_pool里面保存该pool上面所有client_conn的链表,
• server里面保存该server上所有server_conn的链表.
• conn里面的msg会分为in_q, 和out_q 两个msg链表
• 每个msg会同时存在与一个client_conn的out_q 和server_conn的in_q里面
• msg中消息保存在mbuf链表中

conn和msg的结构:

他们的关系:

读写总控函数 core_core

rstatus_t
core_core(void *arg, uint32_t events)
{
    conn->events = events;
    if (events & EVENT_ERR) {
        core_error(ctx, conn);
        return NC_ERROR;
    }

    /* read takes precedence over write */
    if (events & EVENT_READ) {
        status = core_recv(ctx, conn); // //简单直接调用 conn->recv
    }
    if (events & EVENT_WRITE) {
        status = core_send(ctx, conn); // //简单直接调用 conn->send
    }

    return NC_OK;
}

核心就是 conn->recv , conn->send, 以及它们的变化.

accept连接

前面讲到proxy_init通过:

p = conn_get_proxy(pool); 获得一个proxy类型的 conn, 它的回调设置: conn->recv = proxy_recv;

在 proxy_accept 中 accetp到一个连接后, 又通过:

c = conn_get(p->owner, true, p->redis);

获得一个client类型的conn, 这里会把conn->recv设置为 conn->recv = msg_recv, 有数据发来时，就会调用 msg_recv

nc_connection.c

很赞的注释:

*                   nc_connection.[ch]
*                Connection (struct conn)
*                 +         +          +
*                 |         |          |
*                 |       Proxy        |
*                 |     nc_proxy.[ch]  |
*                 /                    \
*              Client                Server
*           nc_client.[ch]         nc_server.[ch]

messsage.c

*            nc_message.[ch]
*        _message (struct msg)
*            +        +            .
*            |        |            .
*            /        \            .
*         Request    Response      .../ nc_mbuf.[ch]  (mesage buffers)
*      nc_request.c  nc_response.c .../ nc_memcache.c; nc_redis.c (_message parser)

* Messages in nutcracker are manipulated by a chain of processing handlers,
* where each handler is responsible for taking the input and producing an
* output for the next handler in the chain. This mechanism of processing
* loosely conforms to the standard chain-of-responsibility design pattern

*             Client+             Proxy           Server+
*                              (nutcracker)
*                                   .
*       msg_recv {read event}       .       msg_recv {read event}
*         +                         .                         +
*         |                         .                         |
*         \                         .                         /
*         req_recv_next             .             rsp_recv_next
*           +                       .                       +
*           |                       .                       |       Rsp
*           req_recv_done           .           rsp_recv_done      <===
*             +                     .                     +
*             |                     .                     |
*    Req      \                     .                     /
*    ===>     req_filter*           .           *rsp_filter
*               +                   .                   +
*               |                   .                   |
*               \                   .                   /
*               req_forward-//  (1) . (3)  \\-rsp_forward
*                                   .
*                                   .
*       msg_send {write event}      .      msg_send {write event}
*         +                         .                         +
*         |                         .                         |
*    Rsp' \                         .                         /     Req'
*   <===  rsp_send_next             .             req_send_next     ===>
*           +                       .                       +
*           |                       .                       |
*           \                       .                       /
*           rsp_send_done-//    (4) . (2)    //-req_send_done
*
*
* (1) -> (2) -> (3) -> (4) is the normal flow of transaction consisting
* of a single request response, where (1) and (2) handle request from
* client, while (3) and (4) handle the corresponding response from the
* server.

好有爱的注释!!

对应这段注释的代码:

struct conn *
conn_get(void *owner, bool client, bool redis)
{
    struct conn *conn;

    conn = _conn_get();

    conn->client = client ? 1 : 0;

    if (conn->client) {
        /*
         * client receives a request, possibly parsing it, and sends a
         * response downstream.
         */
        conn->recv = msg_recv;
        conn->recv_next = req_recv_next;
        conn->recv_done = req_recv_done;

        conn->send = msg_send;
        conn->send_next = rsp_send_next;
        conn->send_done = rsp_send_done;

        conn->close = client_close;
        conn->active = client_active;

        conn->ref = client_ref;
        conn->unref = client_unref;

        conn->enqueue_inq = NULL;
        conn->dequeue_inq = NULL;
        conn->enqueue_outq = req_client_enqueue_omsgq;
        conn->dequeue_outq = req_client_dequeue_omsgq;
    } else {
        /*
         * server receives a response, possibly parsing it, and sends a
         * request upstream.
         */
        conn->recv = msg_recv;
        conn->recv_next = rsp_recv_next;
        conn->recv_done = rsp_recv_done;

        conn->send = msg_send;
        conn->send_next = req_send_next;
        conn->send_done = req_send_done;

        conn->close = server_close;
        conn->active = server_active;

        conn->ref = server_ref;
        conn->unref = server_unref;

        conn->enqueue_inq = req_server_enqueue_imsgq;
        conn->dequeue_inq = req_server_dequeue_imsgq;
        conn->enqueue_outq = req_server_enqueue_omsgq;
        conn->dequeue_outq = req_server_dequeue_omsgq;
    }

    conn->ref(conn, owner);

    return conn;
}

一个请求被处理的流程

我们按照 messsage.c 里面的4个步骤

后面的图都采用这样的表示方法:

初始状态

考察只有一个后端的情况, 假设有2个client要发送3个请求过来

1.读取请求

此时回调函数:

conn->recv = msg_recv;
conn->recv_next = req_recv_next;
conn->recv_done = req_recv_done;

函数调用栈:

每次发生 req_recv_done(req_forward), 就会调用 req_forward()

req_forward(struct context *ctx, struct conn *c_conn, struct msg *msg)
{
    if (!msg->noreply) {
        c_conn->enqueue_outq(ctx, c_conn, msg);
    }

    //获得到后端的连接. (可能是新建, 或者从pool里面获取)
    s_conn = server_pool_conn(ctx, c_conn->owner, key, keylen);

    s_conn->enqueue_inq(ctx, s_conn, msg);
    event_add_out(ctx->evb, s_conn);
}

就有一个msg就出现在client_conn->out_q, 同时出现在server_conn->in_q

req_forward用server_pool_conn获得一个server_conn.

2.转发到后端

对于server_conn来说, 因为挂了epoll_out事件, 很快就会调用 conn->send，也就是msg_send.

此时:

conn->send = msg_send;
conn->send_next = req_send_next;
conn->send_done = req_send_done;

调用栈:

#这时, 每次发生 req_send_done, 这个msg就被放到server_conn->out_q

注意, 此时两个msg依然在client_conn->in_q里面

3.接收后端响应

因为server_conn的 epoll_in是一直开着的, 响应很快回来后, 就到了server_conn的 msg_recv 这个过程和调用栈1类似,不过两个函数钩子不一样(图中灰色框)

conn->recv = msg_recv;(和<1>一样)
conn->recv_next = rsp_recv_next;
conn->recv_done = rsp_recv_done;

这里 rsp_recv_next 的作用是, 拿到下一个要接收的msg

rsp_forward 会把 msg从 server_conn->outq 里面摘掉, 同时设置req和resp之间的一一对应关系

//establish msg <-> pmsg (response <-> request) link
pmsg->peer = msg;
msg->peer = pmsg;

上面这个代码是整个过程的精华所在

这时候, client的q_out上排队的req, 就有了对应的response

这时也会设置:

event_add_out(ctx->evb, c_conn);

每收到一个rsp, 就从server_conn的out_q摘掉, 并设置一一对应关系, 如下:

4.把响应回给client

现在每个请求的msg都有了一个对应的response msg, client_conn的out事件也挂上了, 下面这个调用栈:

最终, 一切归于沉寂, 后端连接依然在:

具体hash函数

import binascii
word = 'hello'
crc32 =  binascii.crc32(word) & 0xffffffff
print '%08x' % crc32, crc32%4

uint32_t jenkins_one_at_a_time_hash(char *key, size_t len)
{
    uint32_t hash, i;
    for(hash = i = 0; i < len; ++i)
    {
        hash += key[i];
        hash += (hash << 10);
        hash ^= (hash >> 6);
    }
    hash += (hash << 3);
    hash ^= (hash >> 11);
    hash += (hash << 15);
    return hash;
}

hash = FNV_offset_basis
for each octet_of_data to be hashed
     hash = hash × FNV_prime
     hash = hash XOR octet_of_data
return hash

libstdc++ (ver 4.6), Perl,[24] nginx (ver 1.0.1),[25] Rubinius,[26] libmemcached (the C driver for Memcached),[27] maatkit,[28] Hadoop,[1] Kyoto Cabinet,[29], RaptorDB[30], and Cassandra.[31]

Murmur/Jenkins(2s) < FNV(4s) < CRC32(5s)

#/home/ning/idning/langtest/python/fnv
def hash_fnv1a_64(s):
    UINT32_MAX=2**32
    FNV_64_INIT = 0xcbf29ce484222325 % UINT32_MAX
    FNV_64_PRIME = 0x100000001b3 % UINT32_MAX

    hval = FNV_64_INIT
    for c in s:
        hval = hval ^ ord(c)
        hval = (hval * FNV_64_PRIME) % UINT32_MAX
    return hval

dispatch方法

• ketama
• modula
• random

注意

• 这个 pool->server_connections 一般都没有配置(默认值是1. 虽然一个连接可能可以应付绝大多数场景, 个人感觉这必须配置啊!!!)

  也就是说 不管前端多少个client, 到后端redis, 就只有1个连接. -- 这是这种proxy的特色

• 通过msg->peer 和msg->owner 耦合client_conn与server_conn

  • 非常巧妙的设计, 多个前端, 可以同时复用后端的 一个连接

    //下面这里建立的link是保证客户端收到的请求和响应一一对应的关键.
    //client_conn->out_q 里面还保存着按照发送顺序的请求msg. 这里收到了response 的msg之后.
    //一一对应起来, 向客户端返回response的时候, 按照 client_conn->out_q的顺序, 看相应的
    //msg 的peer是否被设置了, 如果设置了, 就可以返回.  //
    

  • 这也是与一般的proxy不一样的地方, 一般的proxy, 收到client_conn_1消息, 处理后找个后端连接发过去, 此时这个后端连接是不会被其它客户端连接复用的, 而这时收到的response也就必然是client_conn_1对应的response, 直接放到client_conn1的输出队列即可.:

    //
    /* establish msg <-> pmsg (response <-> request) link */
    pmsg->peer = msg;
    msg->peer = pmsg;
    

  这里proxy必须保存req msg的body和resp msg的body, 如果后端网络/响应慢, 则req msg撑爆内存, 如果前端不读取, 则rsp msg撑爆内存.

• 能用这种后端只开一个连接的proxy, 要求:

  1. 后端交互包不会被打散, 即一个连接上, 包一定是连续的发出, 或者连续的收到.
  2. 单连接上, 后端返回顺序和请求顺序一致. 如果服务器在一个连接上起多个线程来服务, 哪个线程先处理完，就 从这个连接 返回, 那proxy层就乱套了(不过这时候如果有个req/resp id, 也是可以的)
  3. 小包(req/resp都很小, 这样才容易满足1)
  4. 后端处理很快(否则都堆在proxy了)

debug级别

有两种debug: log_debug输出到日志和ASSERT:

flag                         log    ASSERT
--enable-debug=no       =>   0      啥也不做
--enable-debug=log      =>   1      啥也不做
--enable-debug=yes      =>   1      出错时打日志
--enable-debug=full     =>   1      出错时core

configure.ac:

AS_CASE([x$enable_debug],
  [xfull], [AC_DEFINE([HAVE_ASSERT_PANIC], [1],
                      [Define to 1 if panic on an assert is enabled])
            AC_DEFINE([HAVE_DEBUG_LOG], [1], [Define to 1 if debug log is enabled])
           ],
  [xyes], [AC_DEFINE([HAVE_ASSERT_LOG], [1],
                     [Define to 1 if log on an assert is enabled])
           AC_DEFINE([HAVE_DEBUG_LOG], [1], [Define to 1 if debug log is enabled])
          ],
  [xlog], [AC_DEFINE([HAVE_DEBUG_LOG], [1], [Define to 1 if debug log is enabled])],
  [xno], [],
  [AC_MSG_FAILURE([invalid value ${enable_debug} for --enable-debug])])
AC_MSG_RESULT($enable_debug)

core的问题

./configure --enable-debug=log

有些老的gcc可能会core

比如:

https://github.com/twitter/twemproxy/issues/115 https://github.com/twitter/twemproxy/issues/146

in my machine(centos6.3):

$ gcc -v
Using built-in specs.
Target: x86_64-redhat-linux
Thread model: posix
gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC)

debug=no:

./configure  && make clean && make -j 8                                         => core
./configure CFLAGS="-O1 -g3" && make clean && make -j 8                         => ok
./configure CFLAGS="-O2 -g3" && make clean && make -j 8                         => core
./configure CFLAGS="-O3 -g3" && make clean && make -j 8                         => core

debug=log:

./configure --enable-debug=log && make clean && make -j 8                       => core
./configure CFLAGS="-O1 -g3" --enable-debug=log && make clean && make -j 8      => ok
./configure CFLAGS="-O2 -g3" --enable-debug=log && make clean && make -j 8      => core
./configure CFLAGS="-O3 -g3" --enable-debug=log && make clean && make -j 8      => core

debug=full:

./configure --enable-debug=full && make clean && make -j 8                      => ok
./configure CFLAGS="-O1 -g3" --enable-debug=full && make clean && make -j 8     => ok
./configure CFLAGS="-O2 -g3" --enable-debug=full && make clean && make -j 8     => ok
./configure CFLAGS="-O3 -g3" --enable-debug=full && make clean && make -j 8     => ok

在configure里面写了, 默认是-O2(看直接./configure 之后make的输出, 也能看到-O2):

elif test $ac_cv_prog_cxx_g = yes; then
  if test "$GXX" = yes; then
    CXXFLAGS="-g -O2"
  else
    CXXFLAGS="-g"
  fi
else
  if test "$GXX" = yes; then
    CXXFLAGS="-O2"
  else
    CXXFLAGS=
  fi
fi

比较优化选项:

$ gcc -c -Q -O1 --help=optimizers > /tmp/o1
$ gcc -c -Q -O2 --help=optimizers > /tmp/o2

-finline-small-functions

分别用-O1 和-O2生成的二进制, 看某个函数的汇编, 差别太大了:

objdump -d src/nutcracker.o1

对函数禁止优化:

有时候，我们希望对某个函数或者某块代码添加自己的优化选项
这个可以通过下面的宏实现：
对代码块的优化/禁止优化：
#pragma GCC push_options
#pragma GCC optimize ("O0")

your code

#pragma GCC pop_options
to disable optimizations since GCC 4.4.

对函数的优化/禁止优化:

__attribute__((optimize("O0")))

写了个脚本逐一去掉 -O2的选项:

cd ~/twemproxy && ./configure CFLAGS="-O1  -falign-functions -falign-jumps -falign-labels -falign-loops -fcaller-saves -fcrossjumping -fcse-follow-jumps -fdelete-null-pointer-checks -fexpensive-optimizations -fforward-propagate -fgcse -finline-small-functions -fipa-cp -foptimize-register-move -foptimize-sibling-calls -fpeephole2 -fregmove -freorder-blocks -freorder-functions -frerun-cse-after-loop -fschedule-insns2 -fstrict-aliasing" && make clean && make -j 8

去掉 -fstrict-aliasing 就好.

带 -fstrict-aliasing 就会出问题.

下面的就ok:

./configure CFLAGS="-O3 -fno-strict-aliasing" && make clean && make

hash和distribution见前面分析

auto_eject_hosts相关

下面选项只在auto_eject_hosts时有用

• server_failure_limit(多少次开始弹出)

  • The number of conseutive failures on a server that would leads to it being temporarily ejected when auto_eject_host is set to true

• server_retry_timeout(每次弹出多长时间)

  • 是说在这段时间内, 这个host是被eject的(不在hash环中)

• 这一块的代码 modula_update 还没仔细看

timeout

默认情况下, twemproxy一直等着把请求传给后端, 但是如果等了timeout时间, proxy 就不会把这个请求传给后端, 而是向客户端回复一个

SERVER_ERROR Connection timed outrn is sent back to the client.

log

编译时加上log选项, 设为 LOG_INFO

mbuf size

• default mbuf-size of 16K

可以在启动时指定:

$ ./bin/nutcracker -h
This is nutcracker-0.2.4

Usage: nutcracker [-?hVdDt] [-v verbosity level] [-o output file]
                  [-c conf file] [-s stats port] [-a stats addr]
                  [-i stats interval] [-p pid file] [-m mbuf size]

Options:
  -h, --help             : this help
  -V, --version          : show version and exit
  -t, --test-conf        : test configuration for syntax errors and exit
  -d, --daemonize        : run as a daemon
  -D, --describe-stats   : print stats description and exit
  -v, --verbosity=N      : set logging level (default: 5, min: 0, max: 11)
  -o, --output=S         : set logging file (default: stderr)
  -c, --conf-file=S      : set configuration file (default: conf/nutcracker.yml)
  -s, --stats-port=N     : set stats monitoring port (default: 22222)
  -a, --stats-addr=S     : set stats monitoring ip (default: 0.0.0.0)
  -i, --stats-interval=N : set stats aggregation interval in msec (default: 30000 msec)
  -p, --pid-file=S       : set pid file (default: off)
  -m, --mbuf-size=N      : set size of mbuf chunk in bytes (default: 16384 bytes)

max key lenght

• 对memcache: ascii protocol key最多 250 characters. The key should not include whitespace, or 'r' or 'n' character.

• redis 没有这个限制

• nutcracker requires the key to be stored in a contiguous memory region.

  • 也就是说key必须小于mbuf size

server_connections: > 1

如果client想要在一个pipeline里面读到最新的写, 就要设置server_connections:1, 如果我们设置 server_connections:2

那么:

set foo bar
get foo

后面这个get可能在另一个连接里面发过去, 所以不一定能读到 set foo bar 的结果

监控

• server_err
• server_timedout
• server_eof

修改日志级别:

$ ./bin/nutcracker -h
This is nutcracker-0.2.4

Usage: nutcracker [-?hVdDt] [-v verbosity level] [-o output file]
                  [-c conf file] [-s stats port] [-a stats addr]
                  [-i stats interval] [-p pid file] [-m mbuf size]

Options:
  -h, --help             : this help
  -V, --version          : show version and exit
  -t, --test-conf        : test configuration for syntax errors and exit
  -d, --daemonize        : run as a daemon
  -D, --describe-stats   : print stats description and exit
  -v, --verbosity=N      : set logging level (default: 5, min: 0, max: 11)
  -o, --output=S         : set logging file (default: stderr)
  -c, --conf-file=S      : set configuration file (default: conf/nutcracker.yml)
  -s, --stats-port=N     : set stats monitoring port (default: 22222)
  -a, --stats-addr=S     : set stats monitoring ip (default: 0.0.0.0)
  -i, --stats-interval=N : set stats aggregation interval in msec (default: 30000 msec)
  -p, --pid-file=S       : set pid file (default: off)
  -m, --mbuf-size=N      : set size of mbuf chunk in bytes (default: 16384 bytes)

使用 nutcracker -v 0 即可

2.动态调整日志级别:

case SIGTTIN:
    actionstr = ", up logging level";
    action = log_level_up;
    break;

case SIGTTOU:
    actionstr = ", down logging level";
    action = log_level_down;
    break;

case SIGHUP:
    actionstr = ", reopening log file";
    action = log_reopen;
    break;

所以:

kill -s TTOU  23787

日志显示:

[Mon Jan 20 09:44:09 2014] nc_signal.c:122 signal 22 (SIGTTOU) received, down logging level
[Mon Jan 20 09:44:09 2014] nc_log.c:95 down log level to 4

此时就没有连接日志了.

如果想要多看日志:

kill -s TTIN 15797

3.切日志

如果我们需要用日志统计流量, 客户端信息, 可以切日志, 方法:

ning@ning-laptop /tmp/r/nutcracker-22000$ mv log/nutcracker.log log/nutcracker.log.20140120
ning@ning-laptop /tmp/r/nutcracker-22000$ ll log/
total 124K
813654 -rw-r--r-- 1 ning ning 114K 2014-01-20 09:44 nutcracker.log.20140120
813655 -rw-r--r-- 1 ning ning    5 2014-01-20 09:33 nutcracker.pid
ning@ning-laptop /tmp/r/nutcracker-22000$ cat log/nutcracker.pid
23787

#doit
ning@ning-laptop /tmp/r/nutcracker-22000$ cat log/nutcracker.pid | xargs kill -s HUP
ning@ning-laptop /tmp/r/nutcracker-22000$ ll log/
total 124K
813688 -rw-r--r-- 1 ning ning    0 2014-01-20 09:49 nutcracker.log
813654 -rw-r--r-- 1 ning ning 114K 2014-01-20 09:49 nutcracker.log.20140120
813655 -rw-r--r-- 1 ning ning    5 2014-01-20 09:33 nutcracker.pid