开发者

cygwin hadoop map-reduce problem

开发者 https://www.devze.com 2023-03-06 11:59 出处:网络
I am having problem getting map/reduce example to work on cygwin: http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/

I am having problem getting map/reduce example to work on cygwin: http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/

Under cygwin, passing -mapper=mapper.py result in "CreateProcess error=193, %1 is not a valid Win32 application"

I try to use -mappe开发者_开发知识库r="python mapper.py", it give error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1

Anyone has success running hadoop map/reduce using python under cygwin?

Thanks.


I have had success with that tutorial under Cygwin. I am using hadoop-0.20.2, under Cygwin 1.7.9-1 on WinXP. I haven't seen your exact message ... I'm answering, though, because I did have some trouble with the -mapper option and solved it by putting the python scripts in the /tmp directory. I saw some error messages that made me think there was some confusion about how the /home directory was named under Cygwin. I decided to avoid that by using /tmp and had success. I used single-quotes, too, BTW. Sometimes pasting double quotes in Windows gives you a character that a Unix process doesn't understand.

BTW, I also made use of the tutorial re. getting Hadoop going under Cygwin and Eclipse here: http://ebiquity.umbc.edu/Tutorials/Hadoop/ The Eclipse/java stuff near the end did not work for me and isn't how I was planning to work with Hadoop. The first few steps, though, were helpful in getting a pseudo-cluster going.

0

精彩评论

暂无评论...
验证码 换一张
取 消