How can I store large amount of data from a database to XML (memory problem)?

Posted by Andrija on Stack Overflow See other posts from Stack Overflow or by Andrija
Published on 2010-05-24T07:01:01Z Indexed on 2010/05/24 7:21 UTC
Read the original article Hit count: 238

Filed under:
|

First, I had a problem with getting the data from the Database, it took too much memory and failed. I've set -Xmx1500M and I'm using scrolling ResultSet so that was taken care of. Now I need to make an XML from the data, but I can't put it in one file. At the moment, I'm doing it like this:

while(rs.next()){
                i++;
                xmlStringBuilder.append("\n\t<row>");
                xmlStringBuilder.append("\n\t\t<ID>" + Util.transformToHTML(rs.getInt("id")) + "</ID>");
                xmlStringBuilder.append("\n\t\t<JED_ID>" + Util.transformToHTML(rs.getInt("jed_id")) + "</JED_ID>");
                xmlStringBuilder.append("\n\t\t<IME_PJ>" + Util.transformToHTML(rs.getString("ime_pj")) + "</IME_PJ>");
//etc.
                xmlStringBuilder.append("\n\t</row>");
                if (i%100000 == 0){
                                    //stores the data to a file with the name i.xml
                    storeKBR(xmlStringBuilder.toString(),i);
                    xmlStringBuilder= null;
                    xmlStringBuilder= new StringBuilder();  
                }

and it works; I get 12 100 MB files. Now, what I'd like to do is to do is have all that data in one file (which I then compress) but if just remove the if part, I go out of memory. I thought about trying to write to a file, closing it, then opening, but that wouldn't get me much since I'd have to load the file to memory when I open it.

P.S. If there's a better way to release the Builder, do let me know :)

© Stack Overflow or respective owner

Related posts about java

Related posts about Oracle