Python 2.6.4 marshal.load doesn't accept open file objects made with subprocess.Popen

This is a really unfortunate situation. Maya 2011 comes with 2.6.4 and this code does not work in it:

pipe = subprocess.Popen( 'p4 -G edit C:/somefile.txt', stdout=subprocess.PIPE).stdout

try:
    while 1:
        record = marshal.load( pipe )
        list.append( record )
except EOFError:
    pass

Various 2.5 versions work and the latest 2.6.5 also works, just not the version that ships with Maya, of course! It throws this error:

# Error: TypeError: file <maya console> line 3: marshal.load() arg must be file #

What is the best course of action?

Just to move on with life, the code was changed to just dump out a file so marshal.load could then load an actual file. While this works, it is lame.

Maya 2011's accesses python via a zip so, as a small test, I zipped up 2.6.5's Python26/Lib and swapped out the sys.path entries that pointed to the 2.6.4 zip to the 2.6.5 zip. While I haven't tested this extensively, this appears to work. I can't tell if this is a better or worse idea than the above one.

Ideally (I think) there is something within 2.6.4 I could do to make this work without mixing python versions or writing tempfiles to disk all the time. Any ideas?

Update regarding marshal.loads()

Trying marshal.loads(), works in that it doesn't error but it still doesn't work all the way. I'm definitely fumbling around in the dark with the stuff. The perforce stuff is unbearably slow if file operations are done individually, doing them in one query is a must. The original code did this:

files = [a,b,c...n] # Huge list of files
p = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
(fi, pipe) = (p.stdin, p.stdout)

# Fill the command up with all the files
for file in files:
    fi.write(file + '\n')
fi.close()

# Get all the results
try:
    while 1:
        record = marshal.load( pipe )
        listData.append( record )
except EOFError:
    pass

I don't know why this works (in 2.5.x/2.6.5) but it does; listData ends up as a list of all the file results. I can't figure out how to do this with marshal.loads(), I only get back the first result. This method is taken almost verbatim from perforce so I know it's the way things should be done. It is easy to imagine that I just don't know how to use subprocess and marshal properly.

Update 2 marshal.loads() totally does work!

After more testing, pipe.read() was supplying all the data but it contains some null characters or something else I don't fully underst开发者_StackOverflow中文版and so marshal.loads() would only read the first entry. In this particular case I can split the data on "{" and collect the data.

listData = []
results = pipe.read().split("{")
# Skip the first entry since it's empty
for result in results[1:]:
    listData.append( marshal.loads( "{" + result) )

Thanks Cristian for pointing me in the right direction and hopefully anyone upgrading to Maya 2011 using perforce will have things go a little smoother.

---

Encountered same problem with unmarshalling output of p4 with -G option. marshal.loads(str) reads only first record, marshal.load(StringIO(str)) fails with 'marshal.load() arg must be file'. Instead of splitting suggested I've used temporary file workaround:

import subprocess, marshal, tempfile

tempf = tempfile.TemporaryFile()
subprocess.Popen(cmd, stdout=tempf).communicate()
tempf.seek(0)
try:
    while 1:
        record = marshal.load(tempf)
        listData.append( record )
except EOFError:
    pass
tempf.close()

Note that python would delete temp file for you as soon as you close it.

---

This is an old question, but I thought I'd add what I've found for others coming across this qustion.

Unfortunately, senyacap's answer doesn't appear to work on Windows, marshal.load(tempf) fails stating that the temporary file is not a file object. It also doesn't work passing in tempf.file.

So, another (horrible) work around (and this is specifically for dealing with Perforce as there are no other options to using the marshal module), is this:

p = subprocess.Popen ( cmd, stdout = subprocess.PIPE ) 
( stdoutdata, stderrdata ) = p.communicate ( )

unmarshalled = []
while stdoutdata:
    record = marshal.loads ( stdoutdata )
    unmarshalled.append ( record )
    remarshalled = marshal.dumps ( record )
    stdoutdata = stdoutdata [ len ( remarshalled ) : ]

---

In the end I went for this.

import shutil
import os
import tempfile

def decode_list(outputstr):
  tmpfolder = tempfile.mkdtemp()
  try:
      binname = os.path.join(tmpfolder, "p4_decode.bin")
      with open(binname, "wb") as binfile:
          binfile.write(outputstr)

      with open(binname, "rb") as binfile:
          ret = []
          while True:
              try:
                  item = marshal.load(binfile)
                  ret.append(item)
              except EOFError:
                  return ret
  finally:
      if tmpfolder and os.path.isdir(tmpfolder):
          shutil.rmtree(tmpfolder)