FO Processor is little more hungry for memory compare to other Java process. If XSLT scalable option is not set and the same time your RTF template is not well optimized definitely you are going to hit Out of memory exception while working with large volume of data. If the memory requirement is not too bad, you can set the OOP Heap size using following SQL queries.

Check the current OPP JVM Heap size using following SQL query

SQL> select DEVELOPER_PARAMETERS from FND_CP_SERVICES where SERVICE_ID = (select MANAGER_TYPE from FND_CONCURRENT_QUEUES where CONCURRENT_QUEUE_NAME = 'FNDCPOPP'

DEVELOPER_PARAMETERS
-----------------------------------------------------
J:oracle.apps.fnd.cp.gsf.GSMServiceController:-mx512m

Set the JVM Heap size using following SQL query

SQL> update FND_CP_SERVICES
set DEVELOPER_PARAMETERS = 'J:oracle.apps.fnd.cp.gsf.GSMServiceController:-mx2048m'
where SERVICE_ID =
(select MANAGER_TYPE
from FND_CONCURRENT_QUEUES
where CONCURRENT_QUEUE_NAME = 'FNDCPOPP');

SQL>Commit;
.
You need to restart the Concurrent Manager to make it effective.

If this does not resolve the issue, You need to optimize RTF template and set the XSLT scalable option true.