Sequential vs multi-threaded Accounting Events Processing

We are working on an event-driven accounting engine and so far we are doing everything in a batch/sequential manner.

Problem is there are thousands of events created per day and processing everything sequentially makes it slow.

Is there a safe way we can implement开发者_如何学Go a multi-thread event processing accounting engine without worrying about financial data integrity and consistency?

Or is it just better to play it safe and allows follow a batch/sequential approach?

---

Perhaps you might want to look into software transactional memory model. The concept has been discussed in this paper

---

You already have managed building an accounting engine satisfying your financial data integrity and consistency requirements. With this, it should be feasible to parallelize processing.

You would need

• a new dispatcher component which would be responsible for feeding multiple threads with to-be-processed event data while maintaining consistency (i.e. bill each event exactly once) and
• a new aggregator component for muxing the rated events on the accounts.

Typically the aggregator component will become the bottleneck in your parallel architecture.

Since you explicitly intend to build a multi-thread application: Depending on requirements and environment it might be feasible to build a multi-process design, i.e. have several independent processes running in parallel. You would gain a simpler concurrency model versus the evemtual need for interprocess communication.

Virtually all telecom billing systems scale by multi-threading or multi-processing so this is definitelly a sane way forward.

---

You can use multiple concurrent transactions if you follow the rules

check out
en.wikipedia.org/wiki/Atomicity_(database_systems)
en.wikipedia.org/wiki/ACID
en.wikipedia.org/wiki/Record_locking

or read all relevant articles in this section
http://en.wikipedia.org/wiki/Category:Transaction_processing

with the correct level of locking for what you are doing