ColdFusion Server crash after thousands of HTTP requests
- by Jason Bristol
We are running ColdFusion 8 on a windows server 2003 VPS with an API that exposes student records to a partner API through a connector. Our API returns around 50k student records serialized in XML format pretty seamlessly. My question originates when something very frightening happened today when we tested our connector to our partners API. Our entire website and web host went down. We assumed that our host was just having some issues and after 4 hours with no resolution and no response from their customer service we finally got a response from them claiming that they had an "unauthorized user" in their network. After our server was back up we were unable to connect to our website as if the web service or coldfusion itself had froze. This is really where my concern comes from as I fear we may have overloaded the web service.
As I mentioned before we tried sending over 50k HTTP POST requests over to our partner's API, however everything stopped after around 1.6k
Is this bad practice or is there some sort of rate limiting I can relax somewhere in server configuration? We managed to find a workaround, but it bypasses our connector which is critical to our design.
This would have been a one time deal as the purpose of so many requests was to populate our partner's website with current data, after that hourly syncs will keep requests down to around 100 per hour.
UPDATE
Our partner API is owned and operated by Pardot. We are converting students to prospects by passing student data to their API which unfortunately only seems to accept one student at a time. For that reason we have to do all 50k requests individually.
Our server has 4GB of RAM, an Intel Core 2 Duo @ 2.8GHz running Windows Server 2003 SP2.
I monitored the server during a 100 student sync, a 400 student sync, and a 1.4k student sync with the following results:
100 students - 2.25GB of Memory, 30-40% CPU utilization, 0.2-0.3% network bandwidth
400 students - 2.30GB of Memory, 30-50% CPU utilization, 0.2-1.0% network bandwidth
1.4k students - 2.30GB of Memory, 30-70% CPU utilization, 0.2-1.0% network bandwidth
I know this is a far cry from 50k students, but I don't want to risk taking down our CMS system again assuming that was the cause.
To give you a look at our code:
<cfif (#getStudents.statusCode# eq "200 OK")>
<cftry>
<cfloop index="StudentXML" array="#XmlSearch(responseSTUD,'/students/student')#">
<cfset StudentXML = XmlParse(StudentXML)>
<cfhttp
url="#PARDOT_CMS_UPSERT#"
method="post"
timeout="10000" >
<cfhttpparam type="url" name="user_key" value="#PARDOT_CMS_USERKEY#">
<cfhttpparam type="url" name="api_key" value="#api_key#">
<cfhttpparam type="url" name="email" value="#StudentXML.student.email.XmlText#">
<cfhttpparam type="url" name="first_name" value="#StudentXML.student.first.XmlText#">
<cfhttpparam type="url" name="last_name" value="#StudentXML.student.last.XmlText#">
<cfhttpparam type="url" name="in_cms" value="#StudentXML.student.studentid.XmlText#">
<cfhttpparam type="url" name="company" value="#StudentXML.student.agencyname.XmlText#">
<cfhttpparam type="url" name="country" value="#StudentXML.student.countryname.XmlText#">
<cfhttpparam type="url" name="address_one" value="#StudentXML.student.address.XmlText#">
<cfhttpparam type="url" name="address_two" value="#StudentXML.student.address2.XmlText#">
<cfhttpparam type="url" name="city" value="#StudentXML.student.city.XmlText#">
<cfhttpparam type="url" name="state" value="#StudentXML.student.state_province.XmlText#">
<cfhttpparam type="url" name="zip" value="#StudentXML.student.postalcode.XmlText#">
<cfhttpparam type="url" name="phone" value="#StudentXML.student.phone.XmlText#">
<cfhttpparam type="url" name="fax" value="#StudentXML.student.fax.XmlText#">
<cfhttpparam type="url" name="output" value="simple">
</cfhttp>
</cfloop>
<cfcatch type="any">
<cfdump var="#cfcatch.Message#">
</cfcatch>
</cftry>
</cfif>
UPDATE 2
I checked the CF logs and found a couple of these:
"Error","jrpp-8","06/06/13","16:10:18","CMS-API","Java heap space The specific sequence of files included or processed is: D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm, line: 675 "
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.io.CharArrayWriter.write(CharArrayWriter.java:105)
at coldfusion.runtime.CharBuffer.replace(CharBuffer.java:37)
at coldfusion.runtime.CharBuffer.replace(CharBuffer.java:50)
at coldfusion.runtime.NeoBodyContent.write(NeoBodyContent.java:254)
at cfapi2ecfm292155732._factor30(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm:675)
at cfapi2ecfm292155732._factor31(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm:662)
at cfapi2ecfm292155732._factor36(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm:659)
at cfapi2ecfm292155732._factor42(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm:657)
at cfapi2ecfm292155732._factor37(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm)
at cfapi2ecfm292155732._factor44(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm:456)
at cfapi2ecfm292155732._factor38(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm)
at cfapi2ecfm292155732._factor46(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm:455)
at cfapi2ecfm292155732._factor39(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm)
at cfapi2ecfm292155732._factor47(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm:453)
at cfapi2ecfm292155732.runPage(D:\Clients\www.xxx.com\www\dev.cms\api\v1\api.cfm:1)
at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:192)
at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:366)
at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:279)
at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
at coldfusion.filter.PathFilter.invoke(PathFilter.java:86)
at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
at coldfusion.CfmServlet.service(CfmServlet.java:175)
at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
Looks like I might have crashed the JVM in CF, is there a better way to do this? We are thinking of just exporting all records initially as a CSV file and importing it into Pardot seeing as we will never have to do a request this large again.
