开发者

Amazon MapReduce no reducer job

开发者 https://www.devze.com 2022-12-15 00:29 出处:网络
I am trying to create a mapper only job via AWS (a streaming job). The reducer field is required, so I am giving a dummy executable, and adding -jobconf mapred.map.tasks=0 to the Extra Args box. In th

I am trying to create a mapper only job via AWS (a streaming job). The reducer field is required, so I am giving a dummy executable, and adding -jobconf mapred.map.tasks=0 to the Extra Args box. In the hadoop environment (v开发者_StackOverflow中文版ersion 0.20) I've installed, no reducer jobs will launch, but in AWS the dummy executable launches and fails.

How can run a no reducer/ mapper only jobs in AWS?


you can also use cat or NONE as reducer argument.


There is no way to create a mapper only job in amazon M/R. I am using the Identity reducer (org.apache.hadoop.mapred.lib.IdentityReducer) instead.

0

精彩评论

暂无评论...
验证码 换一张
取 消