I'm writing a genetic algorithm that needs to read/write lots of files. The fitness test for the GA is invoking a program called gradif, which takes a file as input and produces a file as output.

Everything is working except when I make the population size and/or the total number of generations of the genetic algorithm too large. Then, after so many generations, I start getting this: java.io.FileNotFoundException: testfiles/GradifOut29 (Too many open files). (I get it repeatedly for many different files, the index 29 was just the one that came up first last time I ran it). It's strange because I'm not getting the error after the first or second generation, but after a significant amount of generations, which would suggest that each generation opens up more files that it doesn't close. But as far as I can tell I'm closing all of the files.

The way the code is set up is the main() function is in the Population class, and the Population class contains an array of Individuals. Here's my code:

Initial creation of input files (they're random access so that I could reuse the same file across multiple generations)

files = new RandomAccessFile[popSize];

for(int i=0; i<popSize; i++){
    files[i] = new RandomAccessFile("testfiles/GradifIn"+i, "rw");
}

At the end of the entire program:

for(int i=0; i<individuals.length; i++){
    files[i].close();
}

Inside the Individual's fitness test:

FileInputStream fin = new FileInputStream("testfiles/GradifIn"+index);
FileOutputStream fout = new FileOutputStream("testfiles/GradifOut"+index);
Process process = Runtime.getRuntime().exec ("./gradif");
OutputStream stdin = process.getOutputStream();
InputStream stdout = process.getInputStream();

Then, later....

try{
      fin.close();
  fout.close();
  stdin.close();
  stdout.close();
      process.getErrorStream().close();
}catch (IOException ioe){
    ioe.printStackTrace();
}

Then, afterwards, I append an 'END' to the files to make parsing them easier.

FileWriter writer = new FileWriter("testfiles/GradifOut"+index, true);
writer.write("END");
try{
   writer.close();
}catch(IOException ioe){
   ioe.printStackTrace();
}

My redirection of stdin and stdout for gradif are from this answer. I tried using the try{close()}catch{} syntax to see if there was a problem with closing any of the files (there wasn't), and I got that from this answer.

It should also be noted that the Individuals' fitness tests run concurrently.

UPDATE: I've actually been able to narrow it down to the exec() call. In my most recent run, I first ran in to trouble at generation 733 (with a population size of 100). Why are the earlier generations fine? I don't understand why, if there's no leaking, the algorithm should be able to pass earlier generations but fail on later generations. And if there is leaking, then where is it coming from?

UPDATE2: In trying to figure out what's going on here, I would like to be able to see (preferably in real-time) how many files the JVM has open at any given point. Is there an easy way to do that?