BufferedReader no longer buffering after a while?_问答_开发者

Sorry I can't post code but I have a bufferedreader with 50000000 bytes set as the buffer size. It works as you would expect for half an hour, the HDD light flashing every two minutes or so, reading in the big chunk of data, and then going quiet again as the CPU processes it. But after about half an hour (this is a very big file), the HDD starts thrashing as if it is reading one byte at a time. It is still in the same loop and I think I checked free ram to rule out swapping (heap size is default).

Probably won't get any helpful answers, but worth a try.

OK I have changed heap size to 768mb and still nothing. There is plenty of free memory and java.exe is only using about 300mb.

Now I have profiled it and heap stays at about 200MB, well below what is available. CPU stays at 50%. Yet the HDD starts thrashing like crazy. I have.. no idea. I am going to rewrite the whole thing in c#, that is my solution.

Here is the code (it is just a throw-away script, not pretty):

    BufferedReader s = null;
    HashMap<String, Integer> allWords = new HashMap<String, Integer>();
    HashSet<String> pageWords = new HashSet<String>();
    long[] pageCount = new long[78592];
    long pages = 0;

    Scanner wordFile = new Scanner(new BufferedReader(new FileReader("allWords.txt")));
    while (wordFile.hasNext()) {
        allWords.put(wordFile.next(), Integer.parseInt(wordFile.next()));
    }
    s = new BufferedReader(new FileReader("wikipedia/enwiki-latest-pages-articles.xml"), 50000000);
    StringBuilder words = new StringBuilder();
    String nextLine = null;
    while ((nextLine = s.readLine()) != null) {
        if (a.matcher(nextLine).matches()) {
            continue;
        }
        else if (b.matcher(nextLine).matches()) {
            continue;
        }
        else if (c.matcher(nextLine).matches()) {
            continue;
        }
        else if (d.matcher(nextLine).matches()) {
            nextLine = s.readLine();
            if (e.matcher(nextLine).matches()) {
                if (f.matcher(s.readLine()).matches()) {
                    pageWords.addAll(Arrays.asList(words.toString().toLowerCase().split("[^a-zA-Z]")));
                    words.setLength(0);
                    pages++;
          开发者_运维技巧          for (String word : pageWords) {
                        if (allWords.containsKey(word)) {
                            pageCount[allWords.get(word)]++;
                        }
                        else if (!word.isEmpty() && allWords.containsKey(word.substring(0, word.length() - 1))) {
                            pageCount[allWords.get(word.substring(0, word.length() - 1))]++;
                        }
                    }
                    pageWords.clear();
                }
            }
        }
        else if (g.matcher(nextLine).matches()) {
            continue;
        }
        words.append(nextLine);
        words.append(" ");
    }

Have you tried removing the buffer size and trying it out with the defaults?

It may be not that the file buffering isn't working, but that your program is using up enough memory that your virtual memory system is page swapping to disk. What happens if you try with a smaller buffer size? What about larger?

I'd bet that you are running out of heap space and you are getting stuck doing back to back GC's. Have you profiled the app to see what is going on during that time? Also, try running with -verbose:gc to see garbage collection as it happens. You could also try starting with a larger heap like"

-Xms1000m -Xmx1000m

That will give you 1gb of heap so if you do use that all up, it should be much later than it is currently happening.

It appears to me that if the file you are reading is very large, then the following lines could result in a large portion of the file being copied to memory via a StringBuilder. If the process' memory footprint becomes too large, you will likely swap and/or throw your garbage collector into a spin.

...
words.append(nextLine);
words.append(" ");

Hopefully this may help: http://www.velocityreviews.com/forums/t131734-bufferedreader-and-buffer-size.html

Before you assume there is something wrong with Java and reading IO, I suggest you write a simple program which just reads the file as fast as it can. You should be able to read the file at 20 MB/s or more regardless of file size with default buffering. You should be able to do this by stripping down your application to just read the file. Then you can prove to yourself how long it takes to read the file.

You have used quite a lot of expensive operations. Perhaps you should look at how you can make your parser more efficient using a profiler. e.g.