Distributed scalable log aggreation with Scribe

Today I will explain how to configure scribe log server and how to aggregate log from different location to a central location.
Scribe uses boost library and Apache Thrift. Both scribe and Apache Thrift  were developed by Facebook and were open sourced. Latest release of scribe was 3 years ago and hence it may not build successfully with latest g++ compiler. I am using 4.4.4 version of g++ and I am building on Linux (CentOS 6.0 64 bit) platform and hence all my examples are for Linux platform only.
I am using boost library (Boost version 1.46.1). Problem is scribe won't build with this library also and hence we will need to change few cpp files in scribe to build it successfully.

We download and extract Boost archive and go to the directory where it was extracted. Then issue the below commands:

$ sh bootstrap.sh --prefix=/usr/local
$ ./bjam install

It will boost libraries and header files under /usr/local/include and /usr/local/lib.
Then download thrift and build it. Thrift will need libevent libarary.

$rpm -qa | grep libevent-devel

If the above command doesn't return anything, then we execute the command below:

$yum install libevent-devel

We have to do apt-get on debian/ubuntu.

Now extract the thrift archive and build it.
Cd to the directory where you extracted the thrift sources.
$ ./configure
$  make
$  make install

Generally, thrift will install in /usr/local directory. If you had installed maven , it will build and install JAVA jars as well in /usr/local/lib.

It is better to add the below headers in the /usr/local/incude/thrift/Thrift.h. Reason being, I was getting few compilation errors such as uint32_t/int32_t not being recognized; htons,ntohl etc. being reported as unknown functions etc. etc.

#include <stdint.h>
#include <inttypes.h>
#include <arpa/inet.h>

Now thrift is there for us. We built boost already.

Building scribe
Now, it's time to compile scribe. I downloaded scribe-2.2.tar.gz and  I explain building for this version only.
Extract the scribe archive and cd to the directory where you extracted the archive. Now issue the below commands:

$export LD_LIBRARY_PATH=

 In my case both of them were installed in /usr/local/lib. So, I issued the below command:

$export LD_LIBRARY_PATH=/usr/local/lib

$sh bootstrap.sh --with-boost=. In my case boost headers/libraries installed in /usr/local. So, I issued the below command:
$sh bootstrap.sh --with-boost=/usr/local
$make

I got compilation error for conflicting return type for virtual scribe::thrift::ResultCode scribeHandler::Log in scribe_server.h. I solved that by looking at the type declared in scribe.h and following the same for scribe_server.h , i.e., I changed return type of scribeHandler::Log to scribe::thrift::ResultCode::type.

Then I got the below error in another source file file.cpp.
file.cpp:203: error: ‘class boost::filesystem3::directory_entry’ has no member named ‘filename’
This was due to the higher version of boost library I am using. So, I made it compatible by replacing the below line in file.cpp 

 _return.push_back(dir_iter->filename());

with 

_return.push_back(dir_iter->path().filename().string()); 

Till there are more compilation errors :). This time the error is in conn_pool.cpp and it says TRY_LATER and OK are not declared (:
So, we replace occurrences of TRY_LATER with ResultCode::TRY_LATER and OK with ResultCode::OK. Also, we have to replace "ResulCode result" with "ResultCode::type result".

Now the compilation error scribe_server.cpp is resolved by replacing "scribe::thrift::ResultCode scribeHandler::Log" with scribe::thrift::ResultCode::type scribeHandler::Log. Fix for ResultCode related compilation issues are fixed in the same way we did for conn_pool.cpp.
Finally, everything compiled successfully and hence we can install scribe.

$make install

Now we can run scribe! Running scribe is simple because the required configuration file is not complex to understand and the scribe package already have some good examples.

In the directory "if" under the root scribe directory (where we extracted scribe.tar.gz) there is the thrift idl file scribe.thrift. This file describes the Log service and also the structure of the log messages that can be sent to scribe log server.
Below is the content of the scribe.thrift:

include "fb303/if/fb303.thrift"

namespace cpp scribe.thrift

enum ResultCode
{
  OK,
  TRY_LATER
}

struct LogEntry
{
  1:  string category,
  2:  string message
}

service scribe extends fb303.FacebookService
{
  ResultCode Log(1: list messages);
}

This shows that each messages are to be sent via the "Log" routine defined by scribe service and it takes a vector of LogEntry Messages as input. LogEntry message has two fields, category and message and both are strings.  By default scribe logger creates a separate directory for each category of messages (unless we explicitly configure scribe not to do so).  This file also include the file fb303/if/fb303.thrift. This was installed on /usr/local/share/fb303/if/fb303.thrift on my system. So, lets generate the cpp files by using thrift compiler.

$thrift -I /usr/local/share -gen cpp   scribe.thrift

This will create gen-cpp directory.

$ls gen-cpp
scribe_constants.cpp  scribe_constants.h  scribe.cpp  scribe.h  scribe_server.skeleton.cpp  scribe_types.cpp  scribe_types.h

We have all the files except the client executable which will call the Log routine and send the messages to the scribe server(s).
Below is the simple client code (save the code in client.cpp).

#include <stdio.h>            
#include <unistd.h>           
#include <sys/time.h>         

#include <iostream>
#include <sstream> 

#include <protocol/TBinaryProtocol.h>
#include <transport/TSocket.h>       
#include <transport/TTransportUtils.h>

#include "scribe.h"

using std::cout;
using std::endl;
using std::vector;
using boost::shared_ptr;
using std::stringstream;

using namespace apache::thrift;
using namespace apache::thrift::protocol;
using namespace apache::thrift::transport;
using namespace scribe;                   
using namespace scribe::thrift;           


int main(int argc, char** argv) {

    if (argc != 3) {
        cout << "Usage: " << argv[0] << "  host " <<  "port" << endl;
        exit(1);                                                                                   
    }                                                                                              

    shared_ptr<TTransport> socket(new TSocket(argv[1], atoi(argv[2])));
    shared_ptr<TTransport> transport(new TFramedTransport(socket));
    shared_ptr<TProtocol> protocol(new TBinaryProtocol(transport));

    // Cretae a scribe client
    scribeClient client(protocol);
    // Vector of Logentry
    vector<LogEntry> logdata;

    try {
        stringstream ss;
        try {
            transport->open();
            int i = 0;
            LogEntry a ;
            a.category = "MyCat";

            // We will send a 1000 messages to the scribe log server
            while ( i++ < 1000 ) {
                ss << "Hello " << i ;
                a.message = ss.str();
                ss.str("");
                ss.clear();
                logdata.push_back(a);
            }
            client.Log(logdata);
            logdata.clear();
        } catch (...) {
            cout << "Got some exception " << endl;
        }
        transport->close();
    } catch (TException &tex) {
        printf("Exception: %s\n", tex.what());
    }
}

Now we do the following:
1. Compile the client programs as shown below:

$g++ -g  -o scribe_client -I . -I /usr/local/include/thrift -I /usr/local/include/thrift/fb303 client.cpp scribe_types.cpp  scribe_constants.cpp  scribe.cpp     -L /usr/local/lib -lthriftnb -lthrift -levent  /usr/local/lib/libfb303.a
Note that we provided the include directory location, and library locations for thrift,boost and facebook interface library (fb303).
We will run all the central logger, client logger and the client program in the same machine for demonstration.
 
2. Scribe central logger example configuration is present in examples directory of scribe source (example2central.conf). I modified it to add a newline after every log messages by adding "add_newlines=1"  in the section of the file. Log files will be created under some sub-directory in /tmp/scribetest. For our example, the su-directory is /tmp/scribetest/MyCat. We run the scribe central logger as shown below:

$scribed -c example2central.conf

3.  example2client.conf is configured to send logs to the central logger. the client logger listens on 1464 port for log messages and it forwards the messages to the central logger. We run scribe client logger by issuing the below command:

$scribed -c example2client.conf

4. Now we run our client. It connects to the client logger and sends logs to the client logger who is listening on port 1464. We run the client program:
$./scribe_client 127.0.0.1 1464

Now we can see our log messages in files in /tmp/scribetest/MyCat directory.

How do we write a client in java? It is simple. Follow the steps given below:
$thrift -I /usr/local/share -gen java   scribe.thrift
This generates the below files under gen-java:

• LogEntry.java  
• ResultCode.java  
• scribe.java

We have to build libfb303-0.8.0.jar if it is not done already.  Cd to thrift-0.8.0/contrib/fb303/java and issue "ant "  build command. Basically we may need the below jars (which are part of thrift package or generated while building thrift).

• commons-codec-1.4.jar
• commons-lang-2.5.jar
• commons-logging-1.1.1.jar
• httpclient-4.1.2.jar
• httpcore-4.1.3.jar
• junit-4.4.jar
• libfb303-0.8.0.jar
• libthrift-0.8.0.jar
• log4j-1.2.14.jar
• servlet-api-2.5.jar
• slf4j-api-1.5.8.jar
• slf4j-log4j12-1.5.8.jar  

We need to write few lines of code to send messages to scribe logger though which will basically calls the "Log" routine defined in scribe.java.
Below is the code:

import org.apache.thrift.TException;
import org.apache.thrift.protocol.TBinaryProtocol;
import org.apache.thrift.protocol.TProtocol;      
import org.apache.thrift.transport.TSocket;       
import org.apache.thrift.transport.TTransport;    
import org.apache.thrift.transport.TFramedTransport;
import org.apache.thrift.transport.TTransportException;
import java.util.List;                                 
import java.util.ArrayList;                            

public class Main {

        public static void main(String[] args) {

                if (args.length != 2) {
                        System.out.println(" <Host> <Port> missing");
                        System.exit(1);
                }
                int port = -1;
                try {
                        port = Integer.parseInt(args[1]);
                } catch (NumberFormatException e) {
                        System.exit(1);
                }
                System.out.println(args[0]);
                System.out.println(args[1]);
                TTransport tr = new TFramedTransport(new TSocket(args[0], port));
                TProtocol proto = new TBinaryProtocol(tr);
                scribe.Client client = new scribe.Client(proto);

                int i = 0;
                List<LogEntry> list = new ArrayList<LogEntry>();
                LogEntry log = null;
                while (i < 100){
                        log = new LogEntry();
                        log.setCategory("javamessage");
                        log.setMessage("My Message " + i);
                        list.add(log);
                        i++;
                }
                try {
                        tr.open();
                        client.Log(list);
                } catch (org.apache.thrift.TException e){
                        e.printStackTrace();
                }
        }
}  

Save this in a file Main.java and compile this along with LogEntry.java, scribe.java,
ResultCode.java. While compiling and running, we have to put the jars listed above
in java classpath.