开发者

Spring Batch Multithreaded Job with fixed order

开发者 https://www.devze.com 2023-03-29 00:45 出处:网络
I created a spring batch job that reads chunks (commit level = 10) of a flat CSV file and writ开发者_如何学运维es the output to another flat file. Plain and simple.

I created a spring batch job that reads chunks (commit level = 10) of a flat CSV file and writ开发者_如何学运维es the output to another flat file. Plain and simple. To test local scaling I also configured the tasklet with a TaskExecutor with a pool of 10 threads, thus introducing parallelism by using a multithreaded step pattern. As expected these threads concurrently read items until their chunk is filled and the chunk is written to the output file. Also as expected the order of the items has changed because of this concurrent reading. But is it possible to maintain the fixed order, preferably still leveraging the increased performance gained by using multiple threads?


I can't think of an easy way. A workaround would be to prefix all lines with an ID which is created sequentially while reading. After finishing the job, sort the lines by the id. Sounds hacky, but should work.


I don't think there is any easy solution, but only using one writer thread (that also performs a sort when writing) and multiple reading threads could work but it would not be as scalable..

0

精彩评论

暂无评论...
验证码 换一张
取 消

关注公众号