Introduction

Libevent is a library that lies hidden under the hood of much open-source software. It was developed by Niels Provos and is maintained by him and Nick Mathewson. You'll find it in memcached, node.js, tor, beanstalkd, gevent and twisted. I've installed these applications without ever precisely understanding what the "uses libevent" line in their documentation actually meant.

So I began playing with libevent and found that there is a dearth of introductory documentation about the project. Nick Mathewson has written an extremely useful book on the subject, but it's a reference more than a primer. Martin Brown wrote a helpful article on using libevent v1. My hope is that the article that follows will provide a primer for others like me who are putting this terrific piece of software to use for the first time.

What is libevent?

Libevent provides an easy way to do high-performance asynchronous IO. In real-world applications of libevent, this generally means that a server uses libevent to create thousands of concurrent client connections without waiting on any one of them to send data; rather the server dedicates "effort" to any one client only when that client's data has arrived. Done right and with a few cpus and some multi-threading, this can give amazing performance, such as 350k requests/second for some memcached installations.

Libevent provides this bounty through a mechanism for executing a callback function when an event occurs on a file descriptor. To somewhat over simplify, an "event on a file descriptor" usually means that someone has sent your application data and it is available for processing or your application has finished doing its processing and it has data to send. When that is the case, the libevent API allows your application to do something (via the "callback function"), such as read and process the client data coming in or ship a response out a particular client. Moreover -- and significantly -- libevent allows for your IO to be non-blocking, meaning that your application doesn't stop and wait for data to be available if doesn't have to. (Here's a quick overview of asynchronous programming.)

This non-blocking/event-driven pattern is not specific to libevent; it's found in any applications that uses poll, select, epoll, kqueue. What libevent has done is provide a generalized, portable interface to all of those underlying mechanisms.

A particularly nice discussion of these concepts can be found on Scot Doyle's website. The examples use Python and epoll, but the ideas are the same.

Buying Fish

I find I learn best when I see a concrete example of a concept and when I can analogize the concept(s) to something in the material world. With libevent, I tried to think of a metaphor that I could turn into a code example and I came up with the following whimsical (or perhaps silly) analogy about selling fish (though you could substitute aluminum cans, used books, diamonds and so forth).

So, leave networking programming aside for a moment and imagine then a cavernous warehouse of concrete and corrugated iron all stinking of fish. Behind a long table strewn with massive plastic buckets stands a fishmonger wearing a once white apron now spattered with blood and fish scales. The warehouse sits next to the piers where the fishing boats dock. When a boat comes in with its catch, the fishermen take their fish to the fishmonger who buys it from them in order to resell it later at a stall in a city market.

In our idealized fish market model, the actions and interactions of the actors would go something like this:

Fish market
The fishmonger opens the warehouse in the morning.
At some point after opening, a fisherman walks into the warehouse.
The fishmonger greets the fisherman and gives him bucket (or appropriate receptacle) for him to place his catch.
While the fisherman is doing that, the fishmonger waits for and perhaps attends to, other customers. So if the fisherman takes a long time to unload or if he gets a cell phone call while unloading, the fishmonger is not waiting for him and is free for other customers.
At the same time, the fishmonger keeps an eye on the fisherman and his bucket and when he sees that the fisherman is done, he goes back to him.
The fishmonger weighs the fish and pays the fisherman.
The fisherman leaves the warehouse.

A Simple Code Model

Surprising though it may seem, we can model those actions using libevent, sockets, and data as follows:

Fish market	Libevent
The fishmonger opens the warehouse in the morning.	The application opens a listening socket.
At some point after opening, a fisherman walks into the warehouse.	The application receives a connection on the listening socket.
The fishmonger greets the fisherman and gives him bucket (or appropriate receptacle) for him to place his catch.	The application calls the callback function associated with the listening socket and the callback sets up a bufferevent structure to receive the incoming data on the socket. The bufferevent structure is associated with its own callback when there is data to be read on the socket.
While the fisherman is doing that, the fishmonger waits for and perhaps attends to, other customers. So if the fisherman takes a long time to unload or if he gets a cell phone call while unloading, the fishmonger is not waiting for him and is free for other customers.	The application does not block while waiting for data to be read on the connection's socket.
At the same time, the fishmonger keeps an eye on the fisherman and his bucket and when he sees that the fisherman is done, he goes back to him.	The operating system's polling mechanism (epoll, select, kqueue, etc) polls to see if there is data to be read on the connection (more specifically: on the file descriptor associated with the connection). When there is data to be read on the connection, the "read" callback associated with the bufferevent structure is executed.
The fishmonger weighs the fish and pays the fisherman.	The read callback processes the incoming data and writes back data to the client.
The fisherman leaves the warehouse.	The socket connection with the client is closed.

I put together some simple code to model the above. I started with libevent author Nick Mathewson's echo server example (which can be found here) and modified it for my model of the market. In particular, libevent v2 has a number of methods that do several things at once -- I replaced those function calls with code to capture each of the constituent steps. I wrote a simple Python client to model the fishermen themselves.

The "fish exchange" protocol is extremely simple: the client sends a series of strings (separated by newlines) of the form n pounds of fish e.g. 3 pounds of trout. The server receives the lines, parses them for the type of fish and the quantity, tallies the result and returns it to the client. If the server doesn't recognize the fish type, it will value it at $0.

Obviously, this code isn't meant for anything other than pedagogical purposes, but please feel free to point out any errors either with my use of libevent or of C.

Quick overview of how libevent works

The basic libevent pattern (and there are certainly other) is as follows:

• Create a event_base structure, which acts as a container for the events you're interested in (e.g. an incoming socket connection).
• Create an event structure consisting of a file descriptor, a file descriptor event (e.g. data available for read on the fd), and a callback function to be executed when the file descriptor event actually happens; add that event structure to the event_base.
• Enable the event since events are "off" by default on creation.
• Use the event_base_dispatch function to start the event loop, which basically means your application sits back and waits for things to happen.
• When a registered event happens (e.g. socket connection), execute the callback function.
• Supposing that the event is an incoming connection, the callback function will accept the connection and setup a bufferevent structure into which incoming data will be placed and out of which data destined for the client will be written; associate with the bufferevent structure two callbacks - one to be executed when there is data available to be read from the bufferevent (the 'read' callback) and one to be executed when there is data to be written from the bufferevent to the client (the 'write' callback, which is often null).
• When data appears in the bufferevent structure, execute the read callback, which will normally be responsible for processing that incoming data.
• If the write callback is defined and there is data ready to be written from the bufferevent structure to the client, execute the write callback.

main()

The code begins with a simple main() routine that simply sets up a listener socket and sets up a callback function to be executed when a connection comes in. This is pretty generic C socket code.

int main(void)
{
  int socketlisten;
  struct sockaddr_in addrlisten;
  struct event *accept_event;
  struct event_base *accept_base;
  int reuse = 1;

  /* create the new event_base structure -- this will contain
       your events */
  accept_base=event_base_new();

  socketlisten = socket(AF_INET, SOCK_STREAM, 0);

  if (socketlisten < 0)
    {
      fprintf(stderr,"Failed to create listen socket");
      return 1;
    }

  memset(&addrlisten, 0, sizeof(addrlisten));

  addrlisten.sin_family = AF_INET;
  addrlisten.sin_addr.s_addr = INADDR_ANY;
  addrlisten.sin_port = htons(SERVER_PORT);

  if (bind(socketlisten,
           (struct sockaddr *)&addrlisten,
           sizeof(addrlisten)) < 0)
    {
      fprintf(stderr,"Failed to bind");
      return 1;
    }

  if (listen(socketlisten, 5) < 0)
    {
      fprintf(stderr,"Failed to listen to socket");
      return 1;
    }

  setsockopt(socketlisten,
             SOL_SOCKET,
             SO_REUSEADDR,
             &reuse,
             sizeof(reuse));

  setnonblock(socketlisten);

  /* now create an event -- this event will be fired if we get
      a connection on the listening socket, which is akin to
      a fishermen walking into the warehouse -- hence
      the name of the callback function: greet_fisherman */
  accept_event=event_new(accept_base,
            socketlisten,
            EV_READ|EV_PERSIST,
            greet_fisherman,
            accept_base);

   /* turn on the event */
  event_add(accept_event, NULL);

  /* sit back and wait for customers ... */
  event_base_dispatch(accept_base);

  close(socketlisten);

  return 0;
}

Listener socket callback

Thus, the callback function that we've set up above is called greet_fisherman. If a connection comes in, we call that function.

In our analogy, this function is the equivalent of the fishmonger greeting a fisherman who's walked in the door and telling him, "you can put your catch in bucket number 5 (or 6 or 18, whatever)."

static void greet_fisherman(int fd, short ev, void *ctx)
{
	/* A new socket connection -- like a fisherman coming into the
           warehouse */
	struct event_base *base = (struct event_base *)(ctx);
	struct bufferevent *bev;
        int client_fd;
        struct sockaddr_in client_addr;
        socklen_t client_len = sizeof(client_addr);

        /* accept the incoming connection */
        client_fd = accept(fd,
                     (struct sockaddr *)&client_addr,
                     &client_len);
        if (client_fd < 0)
        {
            warn("Client: accept() failed");
            return;
        }

        /* don't wait on the fisherman! */
        setnonblock(client_fd);

        /* set up the bufferevent structure -- give the fisherman
             a bin/bucket into which to put his fish */
        bev = bufferevent_socket_new(base, client_fd,
                        BEV_OPT_CLOSE_ON_FREE);

        /* set-up the callbacks on that buffer: the read callback
            (in this case: buyfish) is executed when the client has
            sent data which is available to be 'read' (hence the name
            read callback) on the file descriptor -- in our analogy
            the fishmonger will buy fish when there is fish in
            bucket ready to be bought. */
	bufferevent_setcb(bev, buyfish, NULL, cleanup_cb, NULL);
	bufferevent_enable(bev, EV_READ|EV_WRITE);
}

Setting up the bufferevent callback

The call bufferevent_setcb(bev, buyfish, NULL, cleanup_cb, NULL) is roughly analagous to giving the fisherman a bucket into which to put his catch -- the second argument is the callback function, buyfish. What we're saying here is that if there is data available to be read in the bufferevent structure, then we will call the buyfish function.

The real-world analogy of this would be this: if the fishmonger notices that a given fisherman has finished putting his catch into the bucket, then he goes over to him as soon as possible to buy his fish.

A note about asynchronous IO

As we leave exit the greet_fisherman function, we leave having set-up the bufferevent structure into which the now connected socket can write data. If no data is written for a while, then the program does nothing, but in particular, it doesn't block waiting for that IO. The real-world analogy for that point in the program's execution would be that the fishmonger has just finished setting-up a bucket into which the fisherman can dump his catch. If the fisherman takes a while to do that -- maybe has to go outside and bring his fish in or maybe his cell phone rings -- the fishmonger can still take care of other fisherman wanting to sell their catch. The fishmonger in the analogy is the single-thread of the application: he can't be two places at once in the same way that a single-threaded application can't be executing two functions at once.

The bufferevent callback: buyfish

Next, we have the function buyfish. In our analogy, the calling of this function is equivalent to the fishmonger taking the catch that's been put in the bucket, weighing it, and paying the fisherman.

static void buyfish(struct bufferevent *bev, void *ctx)
{
	/* This callback is invoked when there is data to read on bev. */
	struct evbuffer *bucket= bufferevent_get_input(bev);
	struct evbuffer *output = bufferevent_get_output(bev); 

        char *message;
        struct evbuffer *evreturn;
        evreturn = evbuffer_new();
        int tally=0;

        /* the incoming data will be a series of crlf separated lines
            of the form: "X pounds of Y", where X is a number and Y
            is a fish after each line, we parse the line to see how
            much that particular load of fish is worth -- we tally up
            the total and return that amount to the fisherman. */
        size_t n_read_out;
        do {
            message=evbuffer_readln(bucket, &n_read_out,
                             EVBUFFER_EOL_CRLF);
            if (message) {
               /* parse the message and update the tally */
               tally += getfishvalue(message);
               free(message);
            }
        } while (message);
        /* 'pay' the fisherman */
        evbuffer_add_printf(output,"%d", tally);
}

Other functions

The function cleanup_cb is called with things go wrong with the bufferevent or the client's connection gets closed. So, in our real-world analogy, this is probably like hosing out the bucket and putting the fish in the walk-in refrigerator.

static void cleanup_cb(struct bufferevent *bev, short events, void *ctx)
{
	if (events & BEV_EVENT_ERROR)
		perror("Error from bufferevent");
	if (events & (BEV_EVENT_EOF | BEV_EVENT_ERROR)) {
		bufferevent_free(bev);
	}
}

Modeling Fishermen in Python

To model the behavior of fisherman coming in to sell their daily catch, I wrote a simple Python script. The one point of note about the script is that it introduces a random delay in operation of "unloading the fish" (in other words: writing data to the socket), so that if you run the script, you will see that the first fisherman to walk into the fishmonger's warehouse won't necessarily be the first to be paid.

import socket
import sys
import random
import time
import threading

class market(object):

    def __init__(self,host,port):
        self.host=host
        self.port=port

class fisherman(object):

    def __init__(self,name):
        self.name=name
        self.pronoun=getPronoun(name)

    def createSocket(self,host,port):
        client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        client_socket.connect((host,port))
        return client_socket

    def fishmsg(self,lbs, fish):
        if lbs==1: x=''
        else: x='s'
        msg = "%s pound%s of %s\r\n" % (lbs, x, fish)
        return msg

    def sellfish(self, market, fishlist,n=32):
        try: s=self.createSocket(market.host,market.port)
        except:
            sys.stderr.write("The market appears to be closed.\n")
            return None
        messages=[]
        for lbs,fish in fishlist:
            msg=self.fishmsg(lbs,fish)
            messages.append(msg)
        msg=''.join(messages)
        has_delay = (random.randint(0,99) % 2)
        if has_delay:
            delay=random.randint(1,15)
            print "%s has been delayed for %d second(s)" % \
                    (self.name,delay)
            time.sleep(delay)
        s.send(msg)
        recv=s.recv(n)
        s.close()
        return recv

def run(fman, fishlist):
    mkt=market('localhost',8080)
    total=fman.sellfish(mkt,fishlist)
    if total is not None:
        print "%s is done - %s received $%s" % \
                (fman.name,fman.pronoun,total) 

def getPronoun(name):
    if name in ['fred','bob']: return 'he'
    if name in ['karen']: return 'she'
    else: return 'they'

if __name__=='__main__':
    fishermen={'fred':[(5,'haddock'),(3,'trout'),(4,'bream')],
               'bob':[(1,'haddock'),(2,'salmon'),],
               'karen':[(6,'trout')]}
    threads = []
    for name,fishlist in fishermen.items():
        print "%s walks into the fishmonger's warehouse" % name
        fm=fisherman(name)
        t = threading.Thread(target=run,args=(fm,fishlist,))
        threads.append(t)
        t.start()

Conclusion

In a future article, I hope to provide a slightly more sophisticated example as well as examining some of libevents other features, such as "watermarks".

Libevent is associated with what is known as the "c10k problem" (the idea that servers need to support 10,000 concurrent connections). This site has more on that.

Source code for the examples.