ssh, nohup, then copy results back

I want a script which automates the running of big mathematical computing jobs on a remote machine. Currently I do this:

• ssh into remote machine; run matlab script
• periodically check in to see if job is done yet
• when job is done, I manually scp the files containing the results back to my home machine.

I tried to use these kind of lines in my script to do this job (note the script runs on my machine):

ssh nohup matlab -r theScript; exit;
scp remote@~/files ~/files

This doesn't work. After a while the ssh session ends and the script just proceeds to do the scp, even though the job is not finished yet and the files don't exist yet.

I think what I need to do is check periodically whether the job is done, perhaps by ssh'ing in periodically and reading the nohup.out file looking for a DONE! signal using grep. Then when I see that, copy the files back. But this seems com开发者_C百科plicated and I don't know how to get the DONE! message back to my computer to run a conditional on (if you see the DONE signal, do this stuff...) any ideas?

---

Yeah, the ssh could time out or something. So yeah, you best option is to poll. e.g.

RUN="$(date +%Y%m%d-%H%M%S)"
ssh remote " nohup bash -c \"( matlab -r theScript; echo \$? > $RUN.done ) >$RUN.log 2>&1 </dev/null &\" "
DONE=""
while [ -z "$DONE" ]
do
    sleep 60
    DONE="$(ssh cat $RUN.done 2>/dev/null)"
done
if [ $DONE -eq 0 ]
then
    scp ...
else
    # Optionally fetch logfile
    # scp $RUN.log@remote ...
    echo "ERROR in remote matlab...."
fi

---

There seems to be a way to configure this in your ssh_config.

This page explain how: http://nileshbansal.blogspot.com/2007/02/prevent-timeouts-in-ssh.html

---

Try setting the ServerAliveInterval in you ssh_config file. That way, your SSH session will not close until the remote command finishes.

See here.