开发者

MapReduce shuffle/sort method

开发者 https://www.devze.com 2023-03-01 02:13 出处:网络
Somewhat of an odd question, but does anyone know what kind of sort MapReduce uses in the sort portion of shuffle/sort?I would think merge or insertion (in keeping with the whole MapReduce par开发者_如

Somewhat of an odd question, but does anyone know what kind of sort MapReduce uses in the sort portion of shuffle/sort? I would think merge or insertion (in keeping with the whole MapReduce par开发者_如何转开发adigm), but I'm not sure.


It's Quicksort, afterwards the sorted intermediate outputs get merged together. Quicksort checks the recursion depth and gives up when it is too deep. If this is the case, Heapsort is used.

Have a look at the Quicksort class:

org.apache.hadoop.util.QuickSort

You can change the algorithm used via the map.sort.class value in the hadoop-default.xml.

0

精彩评论

暂无评论...
验证码 换一张
取 消