[comet] How does facebook, gmail send the real time notification?

I have read some posts about this topic and the answers are comet, reverse ajax, http streaming, server push, etc.

How does incoming mail notification on Gmail works?

How is GMail Chat able to make AJAX requests without client interaction?

I would like to know if there are any code references that I can follow to write a very simple example. Many posts or websites just talk about the technology. It is hard to find a complete sample code. Also, it seems many methods can be used to implement the comet, e.g. Hidden IFrame, XMLHttpRequest. In my opinion, using XMLHttpRequest is a better choice. What do you think of the pros and cons of different methods? Which one does Gmail use?

I know it needs to do it both in server side and client side. Is there any PHP and Javascript sample code?

This question is related to comet

The answer is


Update

As I continue to recieve upvotes on this, I think it is reasonable to remember that this answer is 4 years old. Web has grown in a really fast pace, so please be mindful about this answer.


I had the same issue recently and researched about the subject.

The solution given is called long polling, and to correctly use it you must be sure that your AJAX request has a "large" timeout and to always make this request after the current ends (timeout, error or success).

Long Polling - Client

Here, to keep code short, I will use jQuery:

function pollTask() { 

    $.ajax({

        url: '/api/Polling',
        async: true,            // by default, it's async, but...
        dataType: 'json',       // or the dataType you are working with
        timeout: 10000,          // IMPORTANT! this is a 10 seconds timeout
        cache: false

    }).done(function (eventList) {  

       // Handle your data here
       var data;
       for (var eventName in eventList) {

            data = eventList[eventName];
            dispatcher.handle(eventName, data); // handle the `eventName` with `data`

       }

    }).always(pollTask);

}

It is important to remember that (from jQuery docs):

In jQuery 1.4.x and below, the XMLHttpRequest object will be in an invalid state if the request times out; accessing any object members may throw an exception. In Firefox 3.0+ only, script and JSONP requests cannot be cancelled by a timeout; the script will run even if it arrives after the timeout period.

Long Polling - Server

It is not in any specific language, but it would be something like this:

function handleRequest () {  

     while (!anythingHappened() || hasTimedOut()) { sleep(2); }

     return events();

} 

Here, hasTimedOut will make sure your code does not wait forever, and anythingHappened, will check if any event happend. The sleep is for releasing your thread to do other stuff while nothing happens. The events will return a dictionary of events (or any other data structure you may prefer) in JSON format (or any other you prefer).

It surely solves the problem, but, if you are concerned about scalability and perfomance as I was when researching, you might consider another solution I found.

Solution

Use sockets!

On client side, to avoid any compatibility issues, use socket.io. It tries to use socket directly, and have fallbacks to other solutions when sockets are not available.

On server side, create a server using NodeJS (example here). The client will subscribe to this channel (observer) created with the server. Whenever a notification has to be sent, it is published in this channel and the subscriptor (client) gets notified.

If you don't like this solution, try APE (Ajax Push Engine).

Hope I helped.


Facebook uses MQTT instead of HTTP. Push is better than polling. Through HTTP we need to poll the server continuously but via MQTT server pushes the message to clients.

Comparision between MQTT and HTTP: http://www.youtube.com/watch?v=-KNPXPmx88E

Note: my answers best fits for mobile devices.


One important issue with long polling is error handling. There are two types of errors:

  1. The request might timeout in which case the client should reestablish the connection immediately. This is a normal event in long polling when no messages have arrived.

  2. A network error or an execution error. This is an actual error which the client should gracefully accept and wait for the server to come back on-line.

The main issue is that if your error handler reestablishes the connection immediately also for a type 2 error, the clients would DOS the server.

Both answers with code sample miss this.

function longPoll() { 
        var shouldDelay = false;

        $.ajax({
            url: 'poll.php',
            async: true,            // by default, it's async, but...
            dataType: 'json',       // or the dataType you are working with
            timeout: 10000,          // IMPORTANT! this is a 10 seconds timeout
            cache: false

        }).done(function (data, textStatus, jqXHR) {
             // do something with data...

        }).fail(function (jqXHR, textStatus, errorThrown ) {
            shouldDelay = textStatus !== "timeout";

        }).always(function() {
            // in case of network error. throttle otherwise we DOS ourselves. If it was a timeout, its normal operation. go again.
            var delay = shouldDelay ? 10000: 0;
            window.setTimeout(longPoll, delay);
        });
}
longPoll(); //fire first handler

According to a slideshow about Facebook's Messaging system, Facebook uses the comet technology to "push" message to web browsers. Facebook's comet server is built on the open sourced Erlang web server mochiweb.

In the picture below, the phrase "channel clusters" means "comet servers".

System overview

Many other big web sites build their own comet server, because there are differences between every company's need. But build your own comet server on a open source comet server is a good approach.

You can try icomet, a C1000K C++ comet server built with libevent. icomet also provides a JavaScript library, it is easy to use as simple as:

var comet = new iComet({
    sign_url: 'http://' + app_host + '/sign?obj=' + obj,
    sub_url: 'http://' + icomet_host + '/sub',
    callback: function(msg){
        // on server push
        alert(msg.content);
    }
});

icomet supports a wide range of Browsers and OSes, including Safari(iOS, Mac), IEs(Windows), Firefox, Chrome, etc.