Collect data in cache and writing to database

I need some help in Caching.

Here is my requirement :

I am having an application which has millio开发者_StackOverflow社区ns of hits per day. Currently i am logging the information like session and transaction logs in log tables using direct db inserts, which is slowing down the performance of the application.

I want something like : I use some caching mechanism which should collect data per hit. I will write it to a file and as soon as the file contains lets say 1000 records, these entries from cache should go to the database as a single batch.(write behind would be good).

Can some please help in solving this ?

---

Solution: Using log4j with an AsynAppender and a JDBCAppender.

You can configure the buffer size of your AsynAppender in the log4j.xml. Then when the buffer is full, it will use the JDBCAppender to flush everything in the Database.

ex:

    <appender name="DB" class="org.apache.log4j.jdbc.JDBCAppender">
     ....
     your DB CONFIG


    <appender name="PerfAppender" class="org.apache.log4j.AsyncAppender">
    <param name="BufferSize" value="5000" />
    <appender-ref ref="DB" />
</appender>

This way, it do what you describe, and you don't even have to write a line of code ;-) well almost...

More details:

You can persist the data using th MDC of log4J. MDC is basically a hashmap in which you can store the data you wanna log (key,value). Then in your log4j.xml you can access the data in the MDC with the %X.

ex:

<appender name="DB" class="org.apache.log4j.jdbc.JDBCAppender">
<param name="URL" value="yoururlconnection" />

<param name="Driver" value="com.ibm.db2.jcc.DB2Driver" />
<param name="User" value="myuser" />

<param name="Password" value="mypassord" />
<layout class="org.apache.log4j.PatternLayout"> 
     <param name="ConversionPattern"        
         value="INSERT INTO mytable (field1, field2, etc) VALUES
                    ('%X{value1}', '%X{value2}', etc)" />
        </layout>
</appender>

---

There is sure to be a Java interface to memcached that will solve this issue. In your logging code, just add the item to log to memcached, and then every X records write them all in one go.

In fact, you could do the batch writing in a separate process, so the user whose hit contains the cache write does not experience the additional delay.