开发者

What does Disco's "Could not parse worker event:" error mean?

开发者 https://www.devze.com 2023-03-19 15:01 出处:网络
I\'m trying to run a Disco job using map and reduce functions that are deserialized after being passed over a TCP socket using the mar开发者_如何学Goshal library. Specifically, I\'m unpacking them wit

I'm trying to run a Disco job using map and reduce functions that are deserialized after being passed over a TCP socket using the mar开发者_如何学Goshal library. Specifically, I'm unpacking them with

code = marshal.loads(data_from_tcp)
func = types.FunctionType(code, globals(), "func")

I've already tested plain Disco jobs (with locally defined functions) on the same system, and they work fine. However, when I run a Disco job with the new functions, the jobs keep failing and I keep getting the error message localhost WARNING: [map:0] Could not parse worker event: invalid_length

I've searched the documentation, and there is no mention that I could find of a "worker event", or of an invalid_length. After doing a grep on the source code, I find a single instance of the phrase "Could not parse worker event:", specifically in the file master/src/disco_worker.erl. I'm not familiar with Erlang, and have no idea how this works.

What is causing this problem? Should I do something else to circumvent it?

EDIT: After more debugging, I've realized that this error is tied to my use of the string.split() method inside my test-case function. Whenever it is used (even on strings that are not part of the input), this error is raised. I've verified that the method does exist on the object, but calling it seems to cause problems. Any thoughts?

EDIT 2: In addition, any use of the re.split function achieves the same effect.

EDIT 3: It appears that calling any string function on the input string in the map function creates this same error.


In my case this warning occured always when I printed something to sys.stderr in map function (and the job failed in the end). The documentation to worker protocol says: Workers should not write anything to stderr, except messages formatted as described below. stdout is also initially redirected to stderr.

0

精彩评论

暂无评论...
验证码 换一张
取 消

关注公众号