What are your suggestions for best practises for regular data updates in a website database?
- by bboyle1234
My shared-hosting asp.net website must automatically run data update routines at regular times of day.
Once it has finished running certain update routines, it can run update routines that are dependent on the previous updates.
I have done this type of work before, using quite complicated setups. Some features of the framework I created are:
A cron job from another server makes a request which starts a data update routine on the main server
Each updater is loaded from web.config
Each updater overrides a "canRunUpdate" method that determines whether its dependencies have finished updating
Each updater overrides a "hasFinishedUpdate" method
Each updater overrides a "runUpdate" method
Updaters start and run in parallel threads
The initial request from the cron job server started each updater in its own thread and then ended.
As a result, the threads containing the updaters would be terminated before the updaters were finished. Therefore I had to give the updaters the ability to save partial results and continue the update job next time they are started up.
As a result, the cron server had to call the updater many times to ensure the job is done. Sometimes the cron server would continue making update requests long after all the updates were completed. Sometimes the cron server would finish calling the update requests and leave some updates uncompleted.
It's not the best system. I'm looking for inspiration.
Any ideas please? Thank you :)
