Storing a looong lookup table
        Posted  
        
            by 
                inquisitive
            
        on Programmers
        
        See other posts from Programmers
        
            or by inquisitive
        
        
        
        Published on 2014-06-03T06:30:35Z
        Indexed on 
            2014/06/03
            9:34 UTC
        
        
        Read the original article
        Hit count: 308
        
Background
The product i am working on has a very long lookup-table. the table contains static data and cannot be auto generated. there are about 500 rows and 10 columns. columns have mostly integers and strings. to complicate the matters, there are actually two such tables. every row in table-1 maps to zero-or-more rows in table-2.
we use an SQLite database with two tables. the product installer places the SQLite file in the installation directory. the application is written in dot-net and we use ADO to load the data once on startup.
now,
- the lookup table grows. in each release a month, we add about 10 new entries
- existing entries are adjusted. every release we fine tune existing entries.
The problem
a team of (10) developers work on the lookup table. Code goes in the SVN, but the little devil the SQLite does not. this prevents multiple developers to work on it. we do take regular backups of the file, but proper versioning is not possible. we never know who did the breaking change. the worse thing is we dont know if there is any change at all. diff'ing databases is tedious if not impossible.
the tables are expected to grow quite large in years to come and we would need developers to work in parallel on it. the data is business critical. we need to be able to audit changes made to it.
Question
What would be a solution for the problems outlines above? one idea was to transform the whole thing to XML and treat it like just another source file. that way SVN can do the versioning and we can work in parallel. but the data shows relational behavior. with XML we loose the unique and foreign-key constraints. also we cant query it with sql like ease.
any help here will be appreciated.
© Programmers or respective owner