Apache Thrift File based transport

Today I will demonstrate how we can use Apache Thrift FileTransport. Thrift file based transport is useful when we want to store the messages in some local files if the remote server is down. We may replay the files when the remote server comes up again. If we want to implement a Thrift based messaging system, then also this feature will be useful for "store and forwarding message" approach.

exception BadOperation {
1: i32 what,
2: optional string reason
}
enum Operation {
CREATE,
DELETE,
FILECONTENT,
DIRCONTENT
}
struct Work {
1: Operation op
2: string filename
3: string data
4: string rootdir
}
service FileService {
i32 createFile(1:Work w) throws (1:BadOperation badop),
list<string> getFiles(1:Work w) throws (1:BadOperation badop)
}

Please refer my previous post or google for how to use the thrift compiler to generate code for handling thrift messages following the above definitions.My example will be in C++, but a PHP, Java or Python example would look very much similar.The generated code has a file FileService.cpp. Examining the code will show that createFileinterface calls two member functions of FileServiceClient class. They aresend_createFile and recv_createFile. send_createFile actually serialize our data,and send them over the socket in case of a typical client-server scenario.recv_createFile processes the response back from the server. In case we are using File based transport to "send" data, then there won't be any response. Hence, we don't need to call recv_createFile function. In stead we just call send_createFile to "send" or write the messages to the file. This is the trick.Same is the case with writing "getFiles" messages to the file as well.Actually we can make this much more efficient by serializing and batching the writes to the file ourselves. But here I am not showing that, as my purpose is to demonstrate the basic operations of File based transport.
Below I have pasted my code for "writing" messages to the file (file_writer.cpp)

#include <stdlib.h>
#include <time.h>  
#include <iostream>
#include <sstream> 
#include <protocol/TBinaryProtocol.h>
#include <transport/TSocket.h>       
#include <transport/TTransportUtils.h>

#include "FileService.h"

using std::cout;
using std::endl;
using std::stringstream;
using namespace apache::thrift;
using namespace apache::thrift::protocol;
using namespace apache::thrift::transport;

using namespace FileHandler;

using namespace boost;

int main(int argc, char** argv) {
    shared_ptr<TTransport> file(new TFileTransport("testfiletransport"));
    shared_ptr<TTransport> transport(new TBufferedTransport(file));      
    shared_ptr<TProtocol> protocol(new TBinaryProtocol(transport));      
    FileServiceClient client(protocol);                                  
    try {                                                                
        int i = 0;                                                       
        stringstream ss (stringstream::in | stringstream::out);          
        Work work;
        work.data = "mydata";
        work.rootdir = "/home/nipun/test";

        try {
            while (i++ < 100) {
                work.op = Operation::CREATE;
                ss << "filename" << i ;
                work.filename = ss.str();
                ss.clear();
                ss.str("");
                ss << "data" << rand() << "-" << rand() << time(0);
                work.data = ss.str();
                ss.clear();
                ss.str("");
                client.send_createFile(work);

                work.op = Operation::DIRCONTENT;
                ss << "filename" << i  << rand();
                work.filename = ss.str();
                ss.clear();
                ss.str("");
                ss << "data" << rand() << "-" << rand() << time(0) << rand();
                work.data = ss.str();
                ss.clear();
                ss.str("");
                client.send_getFiles(work);
            }
        } catch (BadOperation &op) {
            cout << "Exception "  <<  op.reason  << endl;
        }

    } catch (TException &tx) {
        cout <<  "ERROR: " << tx.what() << endl;
    }
}

Here we wrote "createFile" and "getFiles" messages 100 times each to the file "testfiletransport" in current directory. We used send_createFiles and send_getFiles routines for the same. Remember that we have to use exact similar type of transport while reading from the file also. TBinary protocol specifies how the data is serialized and TBuffered transport is used to buffer data before they are flushed to underlying transport which is the file "testfiletransport" here.

Now is the time to read the data from the file. Generally we write the data to a file and read it later when want to replay them later. In such cases the messages are read and send over another transport which may be over TSocket (socket based transport class defined in Thrift library). But in this example we will just read the messages and print them to standard out.
 

Code for file_reader.cpp

#include <stdlib.h>
#include <time.h>  
#include <iostream>
#include <sstream> 
#include <string>  
#include <protocol/TBinaryProtocol.h>
#include <transport/TSocket.h>       
#include <transport/TTransportUtils.h>

#include "FileService.h"

using std::cout;
using std::endl;
using std::stringstream;
using std::string;      
using namespace apache::thrift;
using namespace apache::thrift::protocol;
using namespace apache::thrift::transport;

using namespace FileHandler;

using namespace boost;

int main(int argc, char** argv) {
    shared_ptr<TTransport> file(new TFileTransport("testfiletransport", true));
    shared_ptr<TTransport> transport(new TBufferedTransport(file));
    shared_ptr<TProtocol> protocol(new TBinaryProtocol(transport));
    FileServiceClient client(protocol);
    string fname;
    TMessageType mtype;
    int32_t seqid;
    Work w;
    try {
        while (true) {
            protocol->readMessageBegin(fname, mtype, seqid);
            cout << fname << endl;
            if (fname == "createFile") {
                FileService_createFile_args args;
                args.read(protocol.get());
                protocol->readMessageEnd();
                w = args.w;
            } else if (fname == "getFiles") {
                FileService_getFiles_args args;
                args.read(protocol.get());
                protocol->readMessageEnd();
                w = args.w;
            }
            cout <<"\tData:\t" << w.data << endl;
            cout <<"\tFilename:\t" << w.filename << endl;
            cout <<"\tRootdir:\t" << w.rootdir << endl;
            cout <<"\tOperation:\t" << w.op << endl;

        }
    } catch (TTransportException& e) {
        if (e.getType() == TTransportException::END_OF_FILE) {
            cout << "\n\n\t\tRead All Data successfully\n";
        }
    } catch (TException &tx) {
        cout << "ERROR " <<  tx.what() << endl;
    }
}

In the file_reader.cpp example, we first read the message type using readMessageBegin interface of binary protocol.  Then we use "fname" (function name) field to decide if the message is a "createFile" or "getFiles" message. Then we use use appropriate FileService args class to read the actual message.
After that we just print the message on our terminal :)

Hope this will be useful for you and if so please don't forget to leave a comment here :)

In my next post, I will show how we may build a simple messaging system using Apache Thrift. Hopefully you will visit again.