Think that your applications need to process thousands of data simultaneously on SQL server to prepare and send them to another application domain.
And also think that your application works with more than 30 threads and 5 or more instances on one server.
So that means, your database operations are getting worse because of multiple access to tables, your indexes will be fragmented and so on.
We have solved this kind of performance issue with message queuing methods.
Message queuing is an old concept to send and receive data from a layer to another layer with almost no cost. There are two main points of queuing; Producer and Consumer.
What we have found !
After a little research we came up with lots of different message queue systems like MMSQ, Apache Kafka, Rabbit MQ and so on.
While deciding about what we need to choose, we compared their performance analysis, read some papers about them and searched for best practices.
We have decided on Apache Kafka.
What we have done !
First, we figured out, how could we integrate queuing structure to our application domain. We separated our applications as Producers and Consumers. We refactored our application code and we tried to implement Producer logic to domain.
Then, we have tried to implement Consumer logic to our code.
We have installed Kafka 0.11.x to our three Centos distribution as cluster. While installing Kafka we only followed instructions of documentation.
Kafka has a pool terminology which means, producers were going to send messages to those pools and consumers were going to listen those pools. After installation we have configured eight different pools with 32 partitions.
After we have started sending and receiving our data over message queue, our database operations and cpu usage decreased. Applications started to process data to with no loss of latency or database access.
It has been seven months and we have no problem about processing data or server issues, or JVM crash. Our applications are stable and we had no restart until now.
But it has recommended that JVM should be monitored, because in some time, it can cause trouble on cluster environment.