Sunday, September 9, 2012

A simple REST framework on C/C++

REpresentational State Transfer (REST) is a software architecture pattern heavily used
on distributed systems, especially for web services.
It is really a client-server architecture. Client sends requests for resources and 
server responds with representation of the resources.In web-based REST system 
individual resources are exposed as URIs. The client and servers communicate using
HTTP protocol. All the HTTP verbs such as GET, DELTE,PUT,POST etc. can be used. The
input for the REST API may come from request headers or parameters. E.g. if an API
requires the client authentication, then it may look for the authentication token by
examining some header or parameter values.

So, for implementing a REST server we need the following bare minimum:
  • An HTTP server
  • A parser to parse the API parameters data
  • An executor which gets the representation of the resource
Generally you produce the REST response in XML or JSON format. But there is no restrictions regarding the formats. So,We need a HTTP library, a JSON/XML parser and of course we have to design and implement the back-end logic for our APIs. In this example I choose libmicrohttpd and Boost Property Tree to generate XML/JSON. Here is a samll example for how we can create a REST server using available http library in C/C++.We will create JSON or XML data as API responeses. All the example code is available on GitHub (click here) REST allows all HTTP methods such as GET,POST,PUT,DELETE etc. But for demonstration purpose we will only use GET method. Also, we will support only http and not https. In next post, I will demonstrate how to use https for secure data transfer for this example. ***PLEASE download and build libmicrohttpd, boost library as we need them to compile and run this example.*** The examples defines three simple APIs or resources. sysinfo diskinfo procinfo sysinfo returns few system related information of your Linux box. It optioanlly takes three flags "cpus", "memory", "os" as the value of parameter "fields", so that user can select a set of information. All the APIs take another parameter "type" to select the response format and valid values are JSON or XML. diskinfo returns few disks related information of your Linux box. It optioanlly takes two flags "totalparts", "spaceinfo" as the value of parameter "fields", so that user can select a set of information. procinfo returns few processes related information about the processes runnining on your Linux box. It also takes three flags "percentmemory", "percentcpu" as the values for parameter "fields", so that user can select a set of information. We need to compile the exmaples as shown below (assuming you built and installed libmicrohttpd in /usr/local)g** httphandler.cpp strutil.cpp api.cpp executor.cpp -I \ g++ -o example_rest_server httphandler.cpp strutil.cpp api.cpp executor.cpp -I\ /usr/local/lib -lmicrohttpd \ libboost_regex.a After we build the example, we run it as shown below $ exectuable_name_for_our_example port_number_where_it_listens E.g. $ ./example_rest_server 1234 We access the API as shown below: http://127.0.0.1:1234/procinfo?type=xml&fields=percentcpu,percentmemory http://127.0.0.1:1234/procinfo?type=json&fields=percentcpu http://127.0.0.1:1234/diskinfo http://127.0.0.1:1234/sysinfo?fields=memory Point your browser to any of the above locations and see the responses :) Below is the code for HTTP request handler (httphandler.cpp)

#include <signal.h>                 
#include <pthread.h>                
#include <platform.h>               
#include <microhttpd.h>             
#include <iostream>                 
#include <map>                      
#include <string>                   

#include <api.hpp>

using std::map;
using std::string;

#define PAGE "<html><head><title>Error</title></head><body>Bad data</body></html>"

static int shouldNotExit = 1;

static int send_bad_response( struct MHD_Connection *connection)
{                                                               
    static char *bad_response = (char *)PAGE;                   
    int bad_response_len = strlen(bad_response);                
    int ret;                                                    
    struct MHD_Response *response;                              

    response = MHD_create_response_from_buffer ( bad_response_len,
                bad_response,MHD_RESPMEM_PERSISTENT);             
    if (response == 0){                                           
        return MHD_NO;                                            
    }                                                             
    ret = MHD_queue_response (connection, MHD_HTTP_OK, response); 
    MHD_destroy_response (response);                              
    return ret;                                                   
}                                                                 


static int get_url_args(void *cls, MHD_ValueKind kind,
                    const char *key , const char* value)
{                                                       
    map<string, string> * url_args = static_cast<map<string, string> *>(cls);

    if (url_args->find(key) == url_args->end()) {
         if (!value)                                   
             (*url_args)[key] = "";                    
         else                                          
            (*url_args)[key] = value;                  
    }                                                  
    return MHD_YES;                                    

}
                
static int url_handler (void *cls,
    struct MHD_Connection *connection,
    const char *url,                  
    const char *method,               
    const char *version,              
    const char *upload_data, size_t *upload_data_size, void **ptr)
{                                                                 
    static int aptr;                                              
    const char *fmt = (const char *)cls;                          
    const char *val;                                              
    char *me;                                                     
    const char *typexml = "xml";                                  
    const char *typejson = "json";                                
    const char *type = typejson;                                  

    struct MHD_Response *response;
    int ret;                      
    map<string, string> url_args;
    map<string, string>:: iterator  it;
    ourapi::api callapi;                     
    string respdata;                         

    // Support only GET for demonstration
    if (0 != strcmp (method, "GET"))     
        return MHD_NO;                   


    if (&aptr != *ptr) {
        *ptr = &aptr;   
        return MHD_YES; 
    }                   


    if (MHD_get_connection_values (connection, MHD_GET_ARGUMENT_KIND, 
                           get_url_args, &url_args) < 0) {         
        return send_bad_response(connection);                         

    }

    callapi.executeAPI(url, url_args, respdata);

    *ptr = 0;                  /* reset when done */
    val = MHD_lookup_connection_value (connection, MHD_GET_ARGUMENT_KIND, "q");
    me = (char *)malloc (respdata.size() + 1);                                 
    if (me == 0)                                                               
        return MHD_NO;                                                         
    strncpy(me, respdata.c_str(), respdata.size() + 1);                        
    response = MHD_create_response_from_buffer (strlen (me), me,               
                                              MHD_RESPMEM_MUST_FREE);          
    if (response == 0){                                                        
        free (me);                                                             
        return MHD_NO;                                                         
    }                                                                          

    it = url_args.find("type");
    if (it != url_args.end() && strcasecmp(it->second.c_str(), "xml") == 0)
        type = typexml;                                                       

    MHD_add_response_header(response, "Content-Type", "text");
    MHD_add_response_header(response, "OurHeader", type);     

    ret = MHD_queue_response (connection, MHD_HTTP_OK, response);
    MHD_destroy_response (response);                             
    return ret;                                                  
}                                                                

void handle_term(int signo)
{                          
    shouldNotExit = 0;     
}                          

void* http(void *arg)
{
    int *port = (int *)arg;
    struct MHD_Daemon *d;

    d = MHD_start_daemon (MHD_USE_SELECT_INTERNALLY | MHD_USE_DEBUG | MHD_USE_POLL,
                        *port,
                        0, 0, &url_handler, (void *)PAGE, MHD_OPTION_END);
    if (d == 0){
        return 0;
    }
    while(shouldNotExit) {
        sleep(1);
    }
    MHD_stop_daemon (d);
    return 0;
}

int main (int argc, char *const *argv)
{

    if (argc != 2){
        printf ("%s PORT\n", argv[0]);
        exit(1);
    }
    daemon(0,0);
    signal(SIGTERM, handle_term);
    int port = atoi(argv[1]);
    pthread_t  thread;
    if ( 0 != pthread_create(&thread, 0 , http, &port)){
        exit(1);
    }
    pthread_join(thread, 0);
    return 0;
}

Below is the code for API parser (api.cpp). 
Here we implement the logic for parsing API parameters, and validating the request,
and calling the appropriate back-end executor routines. 

#include <string.h>         

#include <boost/foreach.hpp>

#include <api.hpp>
#include <strutil.hpp>

using namespace ourapi;

struct validate_data
{                   
    string api;     
    set <string>* params; 
};                              

api::api()
{         
    set<string> params;
    string sysinfoparams[] = {"cpus", "memory", "os"}; 
    string processinfoparams[] = {"percentmemory", "percentcpu" };
    string diskinfoparamas[] = {"totalparts", "spaceinfo" };      

    _apiparams["/sysinfo"] =  set<string>(sysinfoparams, sysinfoparams + 3);
    _apiparams["/procinfo"] = set<string>(processinfoparams, processinfoparams  + 2);
    _apiparams["/diskinfo"] = set<string>(diskinfoparamas, diskinfoparamas + 2);     
}                                                                                          

bool api::executeAPI(const string& url, const map<string, string>& argvals, string& response)
{                                                                                                  
    // Ignore all the args except the "fields" param                                               
    validate_data vdata ;                                                                          
    vdata.api = url;                                                                               
    Executor::outputType type = Executor::TYPE_JSON;                                               
    vector<string> params;                                                                   
    set<string> uniqueparams;                                                                
    map<string,string>::const_iterator it1 = argvals.find("fields");                         

    if (it1 != argvals.end()) {
        string prms = it1->second;
        StrUtil::eraseWhiteSpace(prms);
        StrUtil::splitString(prms, ",", params);   
    }                                              
    BOOST_FOREACH( string pr, params ) {           
        uniqueparams.insert(pr);                   
    }                                              
    vdata.params = &uniqueparams;                  

    if ( !_validate(&vdata)) {
        _getInvalidResponse(response);
        return false;                 
    }                                 

    it1 = argvals.find("type");
    if (it1 != argvals.end()){ 
        const string outputtype = it1->second;
        if (strcasecmp(outputtype.c_str(), "xml") == 0 ) {
            type = Executor::TYPE_XML;                    
        }                                                 
    }                                                     

    return _executeAPI(url, uniqueparams, type, response);
}                                                         

bool api::_executeAPI(const string& url, const set<string>& argvals, 
        Executor::outputType type, string& response)                       
{                                                                          
    bool ret = false;
    if (url == "/sysinfo")
        ret = _executor.sysinfo(argvals, type,  response);
    if (url == "/diskinfo")
        ret = _executor.diskinfo(argvals, type, response);
    if (url == "/procinfo")
        ret = _executor.procinfo(argvals, type, response);

    return ret;
}

bool api::_validate(const void *data)
{
    const validate_data *vdata = static_cast<const validate_data *>(data );
    map<string, set<string> > ::iterator it =  _apiparams.find(vdata->api);

    it = _apiparams.find(vdata->api);

    if ( it == _apiparams.end()){
        return false;
    }
    set<string>::iterator it2 = vdata->params->begin();
    while (it2 != vdata->params->end()) {
        if (it->second.find(*it2) == it->second.end())
            return false;
        ++it2;
    }

    return true;
}

void api::_getInvalidResponse(string& response)
{
    response = "Some error in your data ";
}

Below is the code for API back-end logic (executor.cpp) 
Here we generate the response, i.e., the representation of the resource. 

#include <stdio.h>               
#include <iostream>              
#include <vector>                
#include <sstream>               

#include <stdint.h>
#include <boost/regex.hpp>
#include <boost/format.hpp>
#include <boost/lexical_cast.hpp>
#include <boost/property_tree/ptree.hpp>
#include <boost/property_tree/json_parser.hpp>
#include <boost/property_tree/xml_parser.hpp> 

#include <executor.hpp>
#include <strutil.hpp> 

using namespace ourapi;
using std::vector;     
using boost::property_tree::ptree;
using std::make_pair;             
using boost::lexical_cast;        
using boost::bad_lexical_cast;    
using boost::format;              
using boost::regex_search;        
using boost::match_default;       
using boost::match_results;       
using boost::regex;               


Executor::Executor()
{                   
}                   

bool Executor::diskinfo(const set<string>& args, outputType type, 
        string& response)                                               
{                                                                       
    const char *command = "df | sed 's/ \\+/ /g'  | tail -n +2 ";       
    char line[255];                                                     
    vector<string> tokens;                                        
    int i = 0,j;                                                        
    bool spaceinfo = false;                                             
    bool totalparts = false;                                            
    uint64_t totalspace = 0;                                            
    uint64_t usedspace = 0;                                             
    int32_t partnum = 0;                                                

    FILE *fp = popen(command, "r");
    if (!fp){                      
        return false;              
    }                              
    while (fgets(line, 255, fp) != 0){
        response += string(line);     
    }                                 
    fclose(fp);                       

    if (args.find("spaceinfo") != args.end()) {
        spaceinfo = true;                      
    }                                          
    if (args.find("totalparts") != args.end()) {
        totalparts = true;                      
    }                                           


    StrUtil::splitString( response, " \t\n", tokens); 
                                                      
    j = tokens.size();                                
    ptree diskinforoot ;                              
    ptree diskinfo;                                   

    ptree::iterator  ptit = diskinforoot.push_back(make_pair("diskinfo", diskinfo ));
    ptree::iterator pit ;                                                            
    while (i < j) {                                                               
        {                                                                            
            ptree temp;                                                              
            pit = ptit->second.push_back(make_pair("FileSystem", temp));          
        }                                                                            
        pit->second.push_back(make_pair("Name", tokens[i++]));                    
        try {                                                                        
            if (spaceinfo) {                                                         
                totalspace += lexical_cast<uint64_t>(tokens[i]);               
            }                                                                        
            pit->second.push_back(make_pair("Size", tokens[i++]));                
            usedspace += lexical_cast<uint64_t>(tokens[i]);                    
            pit->second.push_back(make_pair("Used", tokens[i++]));                

        } catch ( bad_lexical_cast& e) {
        }                               
        pit->second.push_back(make_pair("Avail", tokens[i++]));
        pit->second.push_back(make_pair("PercentUse", tokens[i++]));
        pit->second.push_back(make_pair("MountedOn", tokens[i++])); 
        partnum++;                                                     
    }                                                                  

    if (spaceinfo) {
        ptree temp; 
        format fmter("%1%");
        pit = ptit->second.push_back(make_pair("SpaceInfo", temp));
        fmter % totalspace;                                           
        pit->second.push_back(make_pair("TotalSpace", fmter.str()));
        fmter.clear();                                                 
        fmter % usedspace;                                             
        pit->second.push_back(make_pair("UsedSpae", fmter.str()));  
        fmter.clear();                                                 

    }

    if (totalparts) {
        ptree temp;  
        format fmter("%1%");
        fmter % partnum;    
        ptit->second.push_back(make_pair("TotalParts", fmter.str()));
        fmter.clear();                                                  
    }                                                                   

    _generateOutput(&diskinforoot, type, response);
    std::cout << response << std::endl;
    return true;                                   
}                                                  

bool Executor::procinfo(const set<string>& args, outputType type, 
        string& response)                                               
{                                                                       
    const char *command = "ps auxef | tail -n +2 |awk ' { printf \"%s %s %s %s \", $1, $2, $3, $3 ; for (i = 11; i <= NF; i++) {printf \"%s \", $i }  print \"\" }  ' ";                                                                                                                                                                     
    char line[8096];                                                                                                                                                    
    FILE *fp = popen(command, "r");                                                                                                                                     

    if (!fp) {
        return false;
    }                

    string read_line;
    ptree prcinforoot ;
    ptree prcinfo;     
    string::const_iterator start, end;
    match_results<string::const_iterator > what;
    ptree::iterator  ptit = prcinforoot.push_back(make_pair("prcinfo", prcinfo ));
    ptree::iterator pit;                                                          
    regex expression("(.*?) (.*?) (.*?) (.*?) (.*)");                             
    ptree temp;                                                                   
    bool percentcpu = false;                                                      
    bool percentmemory = false;                                                   

    if (args.find("percentcpu") != args.end()) {
        percentcpu = true;                      
    }                                           
    if (args.find("percentmemory") != args.end()) {
        percentmemory = true;                      
    }                                              

    while (fgets(line, 8096, fp) != 0){
        read_line = line;              
        start = read_line.begin();     
        end = read_line.end();         
        if (!regex_search(start, end, what, expression, match_default)){
            continue;                                                   
        }                                                               
        if (what.size() != 6){                                          
            continue;                                                   
        }                                                               
        pit = ptit->second.push_back(make_pair("process", temp));    
        pit->second.push_back(make_pair("owner", string(what[1].first, what[1].second)));
        pit->second.push_back(make_pair("processid", string(what[2].first, what[2].second)));
        if (percentcpu)                                                                         
            pit->second.push_back(make_pair("percentcpu", string(what[3].first, what[3].second)));
        if (percentmemory)                                                                           
            pit->second.push_back(make_pair("percentmemory", string(what[4].first, what[4].second)));
        pit->second.push_back(make_pair("processcommand", string(what[5].first, what[5].second)));   
    }                                                                                                   
    fclose(fp);                                                                                         
    _generateOutput(&prcinforoot, type, response);                                                      
    std::cout << response << std::endl;                                                     
    return true;                                                                                        
}                                                                                                       

bool Executor::sysinfo(const set<string>& args, outputType type, 
        string& response)                                              
{                                                                      
    const char *commandcpu = "cat /proc/cpuinfo |  sed 's/\\s\\+: /:/g'";
    const char *commandmemory = "cat /proc/meminfo |  sed 's/:\\s\\+/:/g'";
    const char *commandos = "uname -a";                                    
    FILE *fp;                                                              
    char commandout[1048];                                                 
    string line;                                                           
    ptree sysinforoot ;                                                    
    ptree sysinfo;                                                         
    ptree::iterator  ptit = sysinforoot.push_back(make_pair("sysinfo", sysinfo ));

    while (args.empty() || args.find("cpus") != args.end()) {
        fp = popen(commandcpu, "r");                         
        if (!fp)                                             
            break;                                           
        ptree temp;                                          
        string field;                                        
        string value;                                        
        size_t index;                                        
        ptree::iterator pit;                                 
        while (fgets(commandout, 1048, fp) != 0){            
            line = commandout;                               
            StrUtil::eraseAllChars(line, ")( \r\n\t");       
            if (strncasecmp(line.c_str(),"processor:", 10) == 0) {
                pit = ptit->second.push_back(make_pair("cpus", temp));
            }                                                            
            index = line.find(":");                                      
            if (string::npos == index)                                   
                continue;                                                
            field = line.substr(0, index);                               
            value = line.substr(index + 1);                              
            pit->second.push_back(make_pair(field, value));           
        }                                                                
        fclose(fp);                                                      
        break;                                                           
    }                                                                    
                                                                         
    while (args.empty()  ||  args.find("memory") != args.end()) {        
        fp = popen(commandmemory, "r");                                  
        if (!fp)                                                         
            break;                                                       
        ptree temp;                                                      
        string field;                                                    
        string value;                                                    
        size_t index;                                                    
        ptree::iterator pit = ptit->second.push_back(make_pair("memory", temp));
        while (fgets(commandout, 1048, fp) != 0){                                  
            line = commandout;                                                     
            StrUtil::eraseAllChars(line, ")( \n\r\t");                             
            index = line.find(":");                                                
            if (string::npos == index)                                             
                continue;                                                          
            field = line.substr(0, index );                                        
            value = line.substr(index + 1);                                        
            pit->second.push_back(make_pair(field, value));                     
        }                                                                          
        fclose(fp);                                                                
        break;                                                                     
    }                                                                              
    while (args.empty() || args.find("os") != args.end()) {                        
        fp = popen(commandos, "r");
        if (!fp)
            break;
        if (fgets(commandout, 1048, fp) == 0) {
            fclose(fp);
            break;
        }
        line = commandout;
        ptree temp;
        string field;
        string value;
        size_t index;
        ptree::iterator pit = ptit->second.push_back(make_pair("os", temp));
        pit->second.push_back(make_pair("osdetails", line));
        fclose(fp);
        break;
    }

    _generateOutput(&sysinforoot, type, response);
    std::cout << response << std::endl;

    return true;
}

void Executor::_generateOutput(void *data, outputType type, string& output)
{
    std::ostringstream ostr;
    ptree *pt = (ptree *) data;
    if (TYPE_JSON == type)
        write_json(ostr, *pt);
    else if (TYPE_XML == type)
        write_xml(ostr, *pt);

    output = ostr.str();
}

22 comments:

  1. Replies
    1. This comment has been removed by the author.

      Delete
    2. No
      The command that are used to get the info are commands in Linux.Hence this will work only on Linux/UNIX system

      Delete
  2. I'd highly recommend taking a look at the restbed project over at https://bitbucket.org/Corvusoft/restbed.

    ReplyDelete
  3. Thanks, this is one of the better examples I've seen on how to incorporate a web server into an C++ API/library/service.

    ReplyDelete
  4. Just an update to my comment as Restbed has moved to github.

    https://github.com/Corvusoft/restbed

    ReplyDelete
  5. hi, is there a source code that create web service json using c, thank you :)

    ReplyDelete
  6. Nipun Da : Awesome, I will try it , as in my company , it is exactly the requirement.

    ReplyDelete
  7. Hi, I am trying to execute the above program, i have installed boost and microhttp in linux server. And facing some errors.

    is strutil.cpp missing in this article? what is ourapi and api.hpp

    ReplyDelete
    Replies
    1. What is the error you are getting?
      The strutil.cpp and all related files are available here https://github.com/nipuntalukdar/NipunTalukdarExamples/tree/master/cpp/restfrmwork

      Please note that you have to install devel package (libmicrohttpd-dev) of libmicrohttpd

      Delete
    2. sorry, missed out the git url mentioned in the article. once i included all the files, it works. Thanks for your response.

      Delete
  8. This comment has been removed by the author.

    ReplyDelete
  9. in the sample program url_handler send out the response. Is there a way to send out asynchronously?which api should be used?

    ReplyDelete
    Replies
    1. Currently it is done synchronously. For asynchronous handling, the response may have to have a parameter, something like requestid which may be sent with the response and the client may associate that requestid (hence the response) with the corresponding request.

      Delete
  10. Thanks for sharing Nipun. Will this work across network. I mean if server is running on linux machine, can i call APIs on an another windows/linux machine ?

    ReplyDelete
  11. This comment has been removed by the author.

    ReplyDelete
  12. Hi, am new Learner of REST Api/Framework. I give it a try today!

    As initial try with Clasablanka build in linux(as per the guidance) is failing. Also visual studio ide install taking too much time.

    Your post seems nice!
    -Thanks - Himadri

    ReplyDelete