As part of my Java mapper I have a command executes some code on the local node and copies a local output file to the hadoop fs. Unfortunately I'm getting the following output:
Error occurred during initialization of VM
Could not reserve enough space for object heap
I've tried adjusting mapred.map.child.java.opts to -Xmx512M, but unfortunately no luck.
Whe开发者_运维问答n I ssh into the node, I can run the -copyFromLocal command without any issues. The ouput files are also quite small like around 100kb.
Any help would be greatly appreciated!
An infinite loop in the mapper or reducer can cause Out of memory errors.
I ran into an OoM once when I had a while-loop with iterator.hasNext() as the condition, for the reducer values, and was not calling iterator.next() inside the loop.
精彩评论