One of the projects I'm working on is online survey engine. It's my first big commercial project on Google App Engine.
I need your advice on how to collect stats and efficiently record them in DataStore without bankrupting me. Initial requirements are:
After user finishes survey client sends list of pairs [ID (int) + PercentHit (double)]. This list shows how close answers of this user match predefined answers of reference answerers (which identified by IDs). I call them "target IDs".
Creator of the survey wants to see aggregated % for given IDs for last hour, particular timeframe or from the beginning of the survey.
Some surveys may have thousands of target/reference answerers.
So I created entity
public class HitsStatsDO implements Serializable
{
@Id
transient private Long id;
transient private Long version = (long) 0;
transient private Long startDate;
@Parent transient private Key parent; // fake parent which contains target id
@Transient int targetId;
private double avgPercent;
private long hitCount;
}
But writing HitsStatsDO for each target from each user would give a lot of data. For instance I had a survey with 3000 targets which was answered by ~4 million people within one week with 300K people taking survey in first day. Even if we assume they were answering it evenly for 24 hours it would give us ~1040 writes/second. Obviously it hits concurrent writes limit of Datastore.
I decided I'll collect data for one hour and save that, that's why there are avgPercent and hitCount in HitsStatsDO. GAE instances are stateless so I had to use dynamic backend instance.
There I have something like this:
// Contains stats for one hour
private class Shard
{
ReadWriteLock lock = new ReentrantReadWriteLock();
Map<Integer, HitsStatsDO> map = new HashMap<Integer, HitsStatsDO>(); // Key is target ID
public void saveToDatastore();
public void updateStats(Long startDate, Map<Integer, Double> hits);
}
and map with shard for current hour and previous hour (which doesn't stay here for long)
private HashMap<Long, Shard> shards = new HashMap<Long, Shard>(); // Key is HitsStatsDO.startDate
So once per hour I dump Shard for previous hour to Datastore.
Plus I have class LifetimeStats which keeps Map<Integer, HitsStatsDO> in memcached where map-key is target ID.
Also in my backend shutdown hook method I dump stats for unfinished hour to Datastore.
There is only one major issue here - I have only ONE backend instance :) It raises following questions on which I'd like to hear your opinion:
Can I do this without using backend instance ?
What if one instance is not enough ?
How can I split data between multiple dynamic backend instances? It hard because I don't know how many I have because Google creates new one as load increases.
I know I can launch exact number of resident backend instances. But how many ? 2, 5, 10 ? What if I have no load at all for a week. Constantly running 10 backend instances is too expensive.
What do I do with data from clients while backend instance is dead/restarting?
