Threading in Java for files

I am looking to read the contents of a file in Java. I have about 8000 files to read the contents and have it in HashMap like (path,contents). I think using Threads would be a option for doing this to speed up the process. From what I know having all 8000 files to read their contents in different threads is not possible(we may want to limit the threads),Any comments on it? Also I am new to threading in Java, can any one help on how to get started on this one?

so far I thought this pesudo code, :

    public class ThreadingTest extends Thread {

public HashMap<String, String > contents = new HashMap<String, String>();


public ThreadingTest(ArrayList<String> paths)
{
    for(String s : paths)
    {
      // paths is paths to files.
      // Have threading here for each path going to get contents from a        
      //  file  
        //Not sure how 开发者_如何学Goto limit and start threads here
        readFile(s);
        Thread t = new Thread();
        t.start();
    }
}


public String readFile(String path) throws IOException
{
    FileReader reader = new FileReader(path);
    StringBuilder sb = new StringBuilder();
    BufferedReader br = new BufferedReader(reader);
    String line;
    while ( (line=br.readLine()) != null) {
      sb.append(line);
    }

    return textOnly;

}

 }

Any help in completing the threading process. Thanks

---

Short answer: Read the files sequentially. Disk I/O doesn't parallelize well.

Long Answer: Threading might improve the read performance if the disks are good at random access (SSD disks are) or if the files are placed on several different disks, but if they're not you're just likely to end up with a lot of cache misses and waiting for the disks to seek the right read position. (You may still end up there even if your disks are good at random access.)

If you want to measure instead of guess, use Executors.newFixedThreadPool to create an ExecutorService which can read your files in parallell. Experiment with different thread counts, but don't be surprised if one reader thread per physical disk gives you the best performance.

---

This is a typical task for thread pool. See the tutorial here: http://download.oracle.com/javase/tutorial/essential/concurrency/pools.html

---

import java.io.*;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.*;

public class PooledFileProcessing {
    private Map<String, String> contents = Collections.synchronizedMap(new HashMap<String, String>());
    //  Integer.MAX_VALUE items max
    private LinkedBlockingQueue<Runnable> workQueue = new LinkedBlockingQueue<Runnable>();

    private ExecutorService executor = new ThreadPoolExecutor(
            5,  //  five workers by default
            20, //  up to twenty workers
            1, TimeUnit.MINUTES,    //  idle thread dies in one minute
            workQueue
    );

    public void process(final String basePath) {
        visit(new File(basePath));
        System.out.println(workQueue.size() + " jobs still in queue");
        executor.shutdown();
        try {
            executor.awaitTermination(5, TimeUnit.MINUTES);
        } catch (InterruptedException e) {
            System.out.println("interrupted while awaiting termination");
        }
        System.out.println(contents.size() + " files indexed");
    }

    public void visit(final File file) {
        if (!file.exists()) {
            return;
        }

        if (file.isFile()) {    //  skip the dirs
            executor.submit(new RunnablePullFile(file));
        }

        //  traverse children
        if (file.isDirectory()) {
            final File[] children = file.listFiles();
            if (children != null && children.length > 0) {
                for (File child : children) {
                    visit(child);
                }
            }
        }
    }

    public static void main(String[] args) {
        new PooledFileProcessing().process(args.length == 1 ? args[0] : System.getProperty("user.home"));
    }

    protected class RunnablePullFile implements Runnable {
        private final File file;

        public RunnablePullFile(File file) {
            this.file = file;
        }

        public void run() {
            BufferedReader reader = null;
            try {
                reader = new BufferedReader(new FileReader(file));
                StringBuilder sb = new StringBuilder();
                String line;
                while (
                        (line=reader.readLine()) != null &&
                        sb.length() < 8192 /* remove this check for a nice OOME or swap thrashing */
                ) {
                  sb.append(line);
                }

                contents.put(file.getPath(), sb.toString());
            } catch (IOException e) {
                System.err.println("failed on file: '" + file.getPath() + "': " + e.getMessage());
                if (reader != null) {
                    try {
                        reader.close();
                    } catch (IOException e1) {
                        //  ignore that one
                    }
                }
            }
        }
    }
}

---

From my experience, threading helps - use a thread pool and play with values around 1..2 threads per core.

Just take care with the hash map - consider putting data to the map via a synchronized method only. I remember I once had some ugly issues in similiar project and they were related to concurrent modifications of a central hash map.

---

just some quick tips.

First of all, to get you started on threads, you should just look at the Runnable interface, or the Thread class. To make a thread you either have to implement this interface with a class or extend this class with another class. You can also make anonymous threads too, but I dislike the readability of those unless its something SUPER simple.

Next, just some notes on processing text with multiple threads, because it just so happens I have some experience in exactly this! Keep in mind that if the files are large and take a noticeably long time to process a single file that you will want to monitor your CPU. In my experience I was doing lots of calculations and lookups when I was processing which added hugely to my load so in the end I found that I could only make as many threads as I had processors because each thread was so labor intensive. So keep that in mind, you want to monitor the effect each thread has on the processor.

---

I'm not sure having threads for this would really speed up the process if all the files are on the same physical disk. It could even slow things down because the disk would have to constantly switch from one location to the other.