Huge file in Clojure and Java heap space error

Posted by trzewiczek on Stack Overflow See other posts from Stack Overflow or by trzewiczek
Published on 2012-04-02T21:18:40Z Indexed on 2012/04/02 23:30 UTC
Read the original article Hit count: 275

Filed under:

clojure

|

heap

|

inputstream

I posted before on a huge XML file - it's a 287GB XML with Wikipedia dump I want ot put into CSV file (revisions authors and timestamps). I managed to do that till some point. Before I got the StackOverflow Error, but now after solving the first problem I get: java.lang.OutOfMemoryError: Java heap space error.

My code (partly taken from Justin Kramer answer) looks like that:

(defn process-pages
  [page]
  (let [title     (article-title page)
        revisions (filter #(= :revision (:tag %)) (:content page))]
    (for [revision revisions]
      (let [user (revision-user revision)
            time (revision-timestamp revision)]
        (spit "files/data.csv"
              (str "\"" time "\";\"" user "\";\"" title "\"\n" )
              :append true)))))

(defn open-file
[file-name]
(let [rdr (BufferedReader. (FileReader. file-name))]
  (->> (:content (data.xml/parse rdr :coalescing false))
       (filter #(= :page (:tag %)))
       (map process-pages))))

I don't show article-title, revision-user and revision-title functions, because they just simply take data from a specific place in the page or revision hash. Anyone could help me with this - I'm really new in Clojure and don't get the problem.

© Stack Overflow or respective owner

Related posts about clojure

StackOverflow in compojure web project

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi I've been playing around with clojure and have been using it to build a simple little audio player. The strange thing is that sometimes, maybe one out of twenty, when contact the server I will get the following error: 2010-04-20 15:33:20.963::WARN: Error for /control java.lang.StackOverflowError … >>> More
Wrong number of args passed to: repl$repl

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I have a trouble with a compojure "Getting started" example that I do not seem to understand. When I run the example from http://weavejester.github.com/compojure/docs/getting-started.html ...I get the following error at the lein repl step: ~/hello-www> lein repl src/hello_www/core.clj Exception… >>> More
Setting the Classpath and Accessing code from book: Programming Clojure

as seen on Stack Overflow - Search for 'Stack Overflow'
(I posted this same question on the Clojure list but haven't got an answer yet. Is anyone here ready to help?) I am going through Programming Clojure and I recently downloaded the code from the books official website. For other utils I can do, for example, (require 'clojure.contrib.str-utils) and… >>> More
How can I define a clojure type that implements the servlet interface?

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm attempting to use deftype (from the bleeding-edge clojure 1.2 branch) to create a java class that implements the java Servlet interface. I would expect the code below to compile (even though it's not very useful). (ns foo [:import [javax.servlet Servlet ServletRequest ServletResponse]]) (deftype… >>> More
Stack overflow in compojure web project

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi I've been playing around with clojure and have been using it to build a simple little audio player. The strange thing is that sometimes, maybe one out of twenty, when contacting the server I will get the following error: 2010-04-20 15:33:20.963::WARN: Error for /control java.lang.StackOverflowError … >>> More

Related posts about heap

Know more about shared pool subpool

as seen on Oracle Blogs - Search for 'Oracle Blogs'
????T.askmaclean.com???Shared Pool?SubPool?????,????????_kghdsidx_count ? subpool ??subpool????( ???duration)???: SQL> select * from v$version; BANNER ---------------------------------------------------------------- Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bi PL/SQL… >>> More
(1 2 3 . #<void>)- heapsort

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello everybody: I tried to implement a "pairing heap" with all the regular operations (merge, delete-min etc.), then I've been requested to write a function that would sort a list using my newly constructed heap implementation. Unfortunately it seems that someting goes wrong... Here's the relevant… >>> More
JVM process resident set size "equals" max heap size, not current heap size

as seen on Server Fault - Search for 'Server Fault'
After a few reading about jvm memory (here, here, here, others I forgot...), I am expecting the resident set size of my java process to be roughly equal to the current heap space capacity. That's not what the numbers are saying, it seems to be roughly equal to the max heap space capacity: Resident… >>> More
Is the heap actually a heap?

as seen on Stack Overflow - Search for 'Stack Overflow'
In .NET (and Java as far as I know), the area where objects are dynamically allocated is referred to as the managed heap. However, most documentation that describes how the managed heap works depicts it as a linear data structure, such as a linked list or stack. So, is the managed heap actually… >>> More
Heap Dump Root Classes

as seen on Stack Overflow - Search for 'Stack Overflow'
We have production system going into infinite loop of full gc and memory drops form 8 gigs to like 1 MB in just 2 minutes. After taking heap dump it tells me there an is an array of java.lang.Object ([Ljava.lang.Object) with millions of java.lang.String objects having same String taking 99% of heap… >>> More