How to make ActiveRecord work with legacy partitioned/sharded databases/tables?

Posted by Utensil on Stack Overflow See other posts from Stack Overflow or by Utensil
Published on 2009-11-02T07:10:08Z Indexed on 2012/09/03 15:38 UTC
Read the original article Hit count: 174

thanks for your time first...after all the searching on google, github and here, and got more confused about the big words(partition/shard/fedorate),I figure that I have to describe the specific problem I met and ask around.

My company's databases deals with massive users and orders, so we split databases and tables in various ways, some are described below:

way             database and table name      shard by (maybe it's should be called partitioned by?)
YZ.X            db_YZ.tb_X                   order serial number last three digits
YYYYMMDD.       db_YYYYMMDD.tb               date
YYYYMM.DD       db_YYYYMM.tb_ DD             date too

The basic concept is that databases and tables are seperated acording to a field(not nessissarily the primary key), and there are too many databases and too many tables, so that writing or magically generate one database.yml config for each database and one model for each table isn't possible or at least not the best solution.

I looked into drnic's magic solutions, and datafabric, and even the source code of active record, maybe I could use ERB to generate database.yml and do database connection in around filter, and maybe I could use named_scope to dynamically decide the table name for find, but update/create opertions are bounded to "self.class.quoted_table_name" so that I couldn't easily get my problem solved. And even I could generate one model for each table, because its amount is up to 30 most.

But this is just not DRY!

What I need is a clean solution like the following DSL:

class Order < ActiveRecord::Base
   shard_by :order_serialno do |key|
      [get_db_config_by(key), #because some or all of the databaes might share the same machine in a regular way or can be configed by a hash of regex, and it can also be a const
       get_db_name_by(key), 
       get_tb_name_by(key),        
      ]
   end
end

Can anybody enlight me? Any help would be greatly appreciated~~~~

© Stack Overflow or respective owner

Related posts about ruby-on-rails

Related posts about activerecord