Redis Performance Evaluation

Redis is an open source, BSD licensed, advanced key-value store. Recently I came across few projects that support redis as their storage. So I think lets explore it a bit more. 

Evalution of redis becomes more easier as it provides a support of redis-benchmark.

Lets see what was result for me. 

System Details : Core 2 Duo , 32-bit, Centos 6.5.

All operations are performed with no pipeline. 

• Set Operation Performance:

• Get Operation Performance

• LPush Operation Performance

• LPop Operation Performance

Redis perfomed quite well , but how much the performance will get affected on the use of jedis (java client for redis). That will be discussed in next blog.

Drools Performance Benchmark

Drools performing well in standalone mode, rate is ~ 114285 records/sec for state-full session and ~4760 records/sec for stateless session. Standalone mode refers to just triggering rules on records. There no integration with camel, spring or any other framework.

Have a look on graph for more details:

Statefull vs Stateless session over 1 rule

Statefull vs Stateless session over 10 rule

Performance comparison on 32 bytes, 64 bytes & 128 bytes java object

Note: Performance is almost same for larger java object the variation is due to limitation of java heap memory of VM

Dynamic Drool.

I recently came across a rule engine called drools . Drool 6.0 is quite different from its previous version. For more information you can have a look here.

I am currently working on a use case scenario in which I want the rules to be updated from my webapp (lets call it a drool-workbench) such that new rules can be applied on new tuple of an input stream without building the source again.

Consider this:

Suppose there is an input stream of tuple1,tuple2 & tuple3, and a rule that can be applied on these tuples.
Now, rule r1 is applied on tuple1 & tuple2. And then we need this rule to be updated with some new conditions. With the help of web-app (drools-wb) the rule r1 can be modified as per need. And then the latest artifact in maven repository can be deployed by "build and deploy" feature of drool-wb. As a new tuple (tuple3) is inserted into the stream, kieScanner will scan the maven repository for latest artifact and modified rule will be triggered on tuple3.

Sample Code:

package com.drool;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;

import org.kie.api.KieServices;
import org.kie.api.runtime.KieContainer;
import org.kie.api.runtime.KieSession;
import org.kie.api.builder.ReleaseId;
import org.kie.api.builder.KieScanner;
import sample.sample.Person;

public class Dynamicdrool{
 public static void main(String [] args){

  BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
  String s;

  KieServices ks = KieServices.Factory.get();
  ReleaseId releaseId = ks.newReleaseId("sample","sample","1.0-SNAPSHOT");
  KieContainer kc = ks.newKieContainer(releaseId);
  KieScanner kScan = ks.newKieScanner(kc);

  try
  {
   while((s = in.readLine())!=null && s.length()!=0)
   {
    Person p = new Person();
    p.setName(s);
    kScan.scanNow();
    KieSession ksess = kc.newKieSession("sampleKs");
    ksess.insert(p);
    ksess.fireAllRules();
    System.out.println(p.getName());
   }
  }catch(IOException e){
   e.printStackTrace();
  }
 }
}

Points to remember:

• Use maven build in your project, to be able to use maven repository.
• KieScanner plays an important role, and it generally scans for newer versions in repository.
  So use 1.0-SNAPSHOT as version or configure your maven for incremental version.
• There is a bug with kie-Scanner in 6.0.1-Final build. Either wait for latest release or build repos from master branch

Drop Table functionality in AlsoSQL(JsonServer plugin of Drizzle)

Do you want to drop table just with the help of json. Now you can do that , recently I have added drop table functionality with json_server plugin.

If you check with Drop table syntax in SQL. You can use drop table in various ways:

• Drop Table.
• Drop Table If_exists.
• Drop Temporary Table.

Json Server allows to drop table in first two ways only.

How you can drop a table?

Its really simple, just start drizzled server with json_server plugin enabled.And then send a curl request to drop a table.

Here is a demo version:

• Create a schema name as "json" and a table "test" in drizzle database.
• Now send a HTTP request to drizzle server.

curl -H "Content-Type: application/json" -H "Accept: application/json" -X POST -d '{"query":{"schema_name":"json","table_name":"test"}}' 'http://localhost:8086/json/ddl/table/drop'

• HTTP Response.

{
   "sql_state" : "00000"
}

• Send a HTTP request for "DROP TABLE IF_EXISTS [TABLE_NAME]"

curl -H "Content-Type: application/json" -H "Accept: application/json" -X POST -d '{"query":{"schema_name":"json","table_name":"test","if_exists":"true"}}' 'http://localhost:8086/json/ddl/table/drop'

• Response should be:

{
   "sql_state" : "00000"
}

Drizzle internals doesn't allow identifier::Table to drop a table, they use TableList. So Overload a function with identifier::Table as parameter works for me.

Branch related to this work: lp:~mohyt/drizzle/json_server_table

Beware to say "Treat warning as an error" in new GCC

Are you building your something using Gcc(>=4.6) ?
If yes, then you used be more careful to use "treat warning as an error".
Recently I tried to build zbase over Ubuntu 12.04 2 which uses Gcc (4.6.2) and I faced lots of problem.
And then I moved to Gcc(4.2), everything was too smooth. No error or warnings.

Then I looked into Gcc release details and came across with few important points:

• -Wunused-but-set-variable and -Wunused-but-set-parameter warnings were added. Also the -Wunused-but-set-variable warning is enabled by default by -Wall flag and -Wunused-but-set-parameter by -Wall -Wextra flags.
• Also you may face few linkers problem. I faced a one with jemalloc.

For more about changes have a look here :)

In drizzle , I have to include "config.h" in each file to get rid of this type of error.

Setting Up a Drizzle Development Ubuntu Box

Sometimes we just get bored to figure out dependencies for drizzle development.

Ubuntu have a drizzle-dev package for development purpose. But for me it's not sufficient.

I follow this particular steps for box setup:

• Get Ubuntu 12.04 or whatever you like, install it on virtual box.
• Now after setting up Ubuntu VM-Box. Open terminal and type:
  
  sudo apt-get install bzr drizzle-dev autopoint libboost1.48-all-dev libprotobuf-dev protobuf-compiler uuid-dev libcloog-ppl
• Also you can try apt-get build-dep drizzle
• Now compile source as mention here.

What's new in Json Server ??

As a part of Drizzle , I just started changing the internals of JSON Server with the help of Stewart.
So, what's new ? . That is a big question for this plugin. Last time I did some re-factoring and implemented some new functionalities. Right now I am not focusing on functionality part too much. But yeah, plugin will come up with some new functionalities soon.

Previous version is capable of basic functionality like Insertion, deletion , selection etc. For more details look here. In previous versions , we generate a sql query by parsing json request and then execute that query with the help of Execute API. But Execute API restricts some functionality.

So I just tried to perform sql execution without Execute API. Now we just parse json query and uses parameter with Storage Engine API.

Right now we have two functionalities with such implementation:

• Create Schema
• Drop Schema

If you like to play with this, try out this branch (lp:~mohyt/drizzle/drizzle-json_server) .

How to play with it? , its too simple :

• Start your drizzle server with json plugin enabled (./drizzled/drizzled --plugin-add=json_server)
• Now send a curl request to create schema. For example,

 --exec curl -H "Content-Type: application/json" -H "Accept: application/json" -X POST -d '{"query":{"name":"json"}}' 'http://localhost:8086/json/ddl/schema/create'

• And , to drop a schema , try this ,  

--exec curl -H "Content-Type: application/json" -H "Accept: application/json" -X POST -d '{"query":{"name":"json"}}' 'http://localhost:8086/json/ddl/schema/drop'

Comments are most welcome :)

Design of AlsoSQL: Drizzle JSON HTTP Server

This particular project was proposed by Henrik at Drizzle Day 2011. A week later, Stewart published version 0.1 of AlsoSQL. Later I and Henrik worked on version 0.2 and he talked about it at Drizzle Day 2012 .

AlsoSQL:Drizzle JSON HTTP Server, is capable of SQL-over-HTTP with key-value operation in pure json. Henrik explained about its working and functionality in this particular post. 

Here,I am going to talk about design of AlsoSQL and the various problems I faced.

Up-to version 0.2 , the API wasn't properly object oriented, so we planned to do re-factoring first.

After going through the code-base of AlsoSQL, I realized the need of design pattern. So we looked through various of them and chose Fascade design pattern for future development.

Here is the rough design of AlsoSQL (I called it rough because it might change in future).

Description:

HttpHandler is used to handle the http request ,parsing and validating json from that request and sending back response.

SQLGenerator simply generates the sql string corresponding to request type using input json.

SQLExecutor executes the sql and returns resultset.

SQLToJsonGenerator generates the output json corresponding to request type.

DBAccess is an interface to access generators and executor.

The main reason to design it in such a way is that it can be used over storage engine in future directly.

Problems Faced:

<config.h> , I always forgot to include this header file and it floods error on my terminal.

Another one , I need to get LAST_INSERT_ID() and I want to get it in a single query with REPLACE query. Still working on it.

Debug Drizzle Code with GDB

From last few days , I have been working on a bug in JSON Server of Drizzle. And the bug is of Segmentation fault , which is not easy to recognize by just reading the code-base. So , the best way to get rid of this problem is  to debug the code. But How ?
I used Visual Studio once during my internships to debug a project. So ,I tried to find out some IDEs that can be use with drizzle . But I figured it out as we can debug a C++ code-base easily and efficiently with GDB .

Here are few steps for debugging:

First of all build your server and enable debugging:

mohit@mohit-PC:~$ cd repos/drizzle/drizzle-json_server/

mohit@mohit-PC:~/repos/drizzle/drizzle-json_server$ ./config/autorun.sh && ./configure --with-debug && make install -j2

Default installation path is /usr/local/ but you can change it with --prefix = /installation_path/
Start server and debug the code:

mohit@mohit-PC:~/repos/drizzle/drizzle-json_server$ gdb /usr/local/sbin/drizzled >

Now set arguments needed to start server with specific plugin (In my case , plugin is json_server)

(gdb) set args --plugin-add=json_server

Set breakpoints,you can do that with this command:
 b <filename>:<line-number> or break <filename>:<line-number>

(gdb) break json_server.cc:1093

Since the json_server plugin is not loaded yet , hence it prompts with :

No source file named json_server.cc.
Make breakpoint pending on future shared library load? (y or [n])

Go with "y"
Run your server now :

(gdb) r

You will get something like this:

Starting program: /usr/local/sbin/drizzled --plugin-add=json_server
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".

Now you can use various commands of GDB to debug your code.
List of these commands are mentioned here.

Problem Faced :

Once I started drizzle server with plugin. I got a message:

Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
/usr/local/sbin/drizzled: relocation error: /usr/local/sbin/drizzled: symbol _ZN8drizzled7message6AccessC1Ev, version DRIZZLE7 not defined in file libdrizzledmessage.so.0 with link time reference
[Inferior 1 (process 23532) exited with code 0177]

I was unable to get a single line of this problem, Thank to Henrik for solution.

Solution:

mohit@mohit-PC:~$ sudo rm /etc/ld.so.cache
mohit@mohit-PC:~$ ldconfig

Toru and Padraig previously posted on this topic which may also be helpful.
Also You can find documentation on this at Drizzle wiki.

Calculating Value During Compile Time In C++

"If you think it's simple, then you have misunderstood the problem" - Bjarne Stroustrup

Recently one of my colleague asked me a question:

Print the value of (1 + 100 * 10 - 5 / 2 + 30) expression during compile time. And he allowed me to use any programming language.

For a while , I was not able to get the answer of this question. But after spending few minutes , I found out it's a good conceptual question.
The answer of this problem is simple, if you know about Templates in C++ a bit.

What are Templates?
Templates are a feature of the C++ programming language that allow functions and classes to operate with generic types. This allows a function or class to work on many different data types without being rewritten for each one.


The solution is based on Template Meta-Programming.
Template meta-programming is a technique in which templates are used by a compiler to generate temporary source code, which is merged by the compiler with the rest of the source code and then compiled. More on wiki.

#include <iostream>

using namespace std;
template<unsigned int n>
struct Expression {
   static const float result = n;
};

template<unsigned int exp>
struct _{ operator char() { return exp;} };

int main() {
        char(_<Expression<1 + 100 * 10 - 5 / 2 + 30>::result>());
        return 0;
}

If you want to explore more , try to print Factorial<n> ;)

"Undefined Reference" While Using Templates In C++

Did you get "collect2: ld returned 1 exit status" ??

If yes , then there is some linker's problem. Lets solve it.

I designed a template for one of my assignment , But I was unaware of "Declaring & Defining" fundamentals of templates in C++.

Here is some code :

//Add.h

#include "Basic.h"
using namespace std;
template <class T>
class Add{

        int x;
        int y;
        
public:
      Add(int x,int y);
      Dump();
};

Add.cpp

#include "Add.h"

template <class T>
Add<T>::Add(int x,int y):x(1),y(2){};


template <class T>
Add<T>::Dump(){
 cout<<x<<"  "<<y;
}

Output:
collect2: ld returned 1 exit status

Reason:
Templates are just a recipe , they are not code. In order to compile this code, it must be generated first, which needs to know templates definition & definition parameters which are passed to it. And compiler does not know about separate compilation (compiler doesn't know about anything about code outside of a file).

Solution:
Put the definitions in the header files itself :)

//Add.h

#include "Basic.h"
using namespace std;
template <class T>
class Add{

        int x;
        int y;
        
public:
      Add(int x,int y);
      Dump();
};

template <class T>
Add<T>::Add(int x,int y):x(1),y(2){};

template <class T>
Add<T>::Dump(){
 cout<<x<<"  "<<y;
}

"Bored Of Segmentation Fault"

Generally , the way to use "Pointers" defines , the way to tackle "Segmentation Fault".

Are you bored of segmentation fault, lets try to make it interesting. Without a fault (segmentation),we can't
learn it. :)
So,

Here is small conversation between me and my friend:


Me: (After spending almost a half day, over a silly mistake. I went to him) : Dude, I am bored of this segmentation fault ?  Help me to get rid of it.

Friend (After a cute smile) : Do you know what it is ??

Me: Yes , its a silly fault which can't be debug.

Friend: No man , You can debug it use GDB debbuger. And its just a  "attempt to access a memory that a processing unit cannot physically address".

Me(reminds me OS course): Ohoo.. Segmentation (approach to memory management and protection in the operating system). Got It. ....... But why it occurs.. even I didn't plan it to occur :)

Friend: Hmm .. Nice question .. reasons for it -
            - A buffer overflow.
            - use of unintialized pointers.
            - derefrencing Null pointers.
            - attempting a memory ehich is stranger for program.
            - exceeding the allowable stack size.

Me: Ohoo .. I think I figure out why it occurs in my code.

Here is that code(sample one):

Code #1:

                 
                     void func(string word,const int n)
                     {
                         int key[n];
                         for(int j=0;j<n*n;j++)
                         key[n]=word[j];
                     }

                     int main()
                      {
                         string word;
                         int len;
                         cin>>word;
                         len=word.length();
                         const int n=sqrt(len);
                        func(word,n);
                     }

I found segmentation fault in above code, but with small change I figure out a small lesson.

Code #2:

                    void func(string word,int n)
                     {
                         const int n = sqrt(len);
                         int key[n];
                         for(int j=0;j<n*n;j++)
                         key[n]=word[j];
                     }

Is this correct solution??-  No.
But it get you out of segmentation fault for a while.
Now question is how ??

As we use 'n' in parameter in Code #1, means it get place on stack. While in second case , on heap. 

Code # 3 (Correct solution):
               

                  void func(string word,const int n)
                     {
                         int key[n*n];
                         for(int j=0;j<n*n;j++)
                         key[n]=word[j];
                     }