开发者

Separating Hadoop Map and Reduce tasks

开发者 https://www.devze.com 2023-02-25 19:12 出处:网络
In a 3 node hadoop cluster. I would like the master to be 1 node. Map task taking place in one node and reduce tasks in 1 node. Map and reduce tasks should be separated. Is it possible? As far as i no

In a 3 node hadoop cluster. I would like the master to be 1 node. Map task taking place in one node and reduce tasks in 1 node. Map and reduce tasks should be separated. Is it possible? As far as i noticed both run together. It will开发者_运维技巧 be great if you can shed some light. Thank you !

-Sethu


This is everything else than optimal because the map output must ALWAYS be copied to another server.
But you can simply modify your mapred-site.xml on the servers.

<property>
    <name>mapred.tasktracker.map.tasks.maximum</name>
    <value>4</value>
    <description>The maximum number of map tasks that will be run simultaneously by a task tracker.</description>
</property>

<property>
    <name>mapred.tasktracker.reduce.tasks.maximum</name>
    <value>4</value>
    <description>The maximum number of reduce tasks that will be run simultaneously by a task tracker.</description>
</property>

On the server where no reducers should run you put into the reduce.task.maximum a zero. And vice versa for the other servers.

0

精彩评论

暂无评论...
验证码 换一张
取 消