kick off a map reduce job from my java/mysql webapp

Posted by Brian on Stack Overflow See other posts from Stack Overflow or by Brian
Published on 2011-01-08T22:11:41Z Indexed on 2011/01/09 5:54 UTC
Read the original article Hit count: 350

Filed under:

mysql

|

architecture

|

jpa

|

hadoop

|

hbase

Hi guys,

I need a bit of archecture advice. I have a java based webapp, with a JPA based ORM backed onto a mysql relational database. Now, as part of the application I have a batch job that compares thousands of database records with each other. This job has become too time consuming and needs to be parallelized. I'm looking at using mapreduce and hadoop in order to do this. However, I'm not too sure about how to integrate this into my current architecture. I think the easiest initial solution is to find a way to push data from mysql into hadoop jobs. I have done some initial research on this and found the following relevant information and possibilities:

1) https://issues.apache.org/jira/browse/HADOOP-2536 this gives an interesting overview of some inbuilt JDBC support 2) This article http://architects.dzone.com/articles/tools-moving-sql-database describes some third party tools to move data from mysql to hadoop.

To be honest I'm just starting out with learning about hbase and hadoop but I really don't know how to integrate this into my webapp.

Any advice is greatly appreciated. cheers, Brian

© Stack Overflow or respective owner

Related posts about mysql

How to remove MySQL completely with config and library files on ubuntu 12.04 gnome 3.0

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I tried everything till now: sudo apt-get remove mysql-server mysql-client mysql-common sudo apt-get purge mysql-server mysql-client mysql-common sudo apt-get autoremove and even more commands... But whenever I am trying to locate mysql. I get a no. of files related to mysql command: shell>>… >>> More
mysql: Cannot load from mysql.proc. The table is probably corrupted

as seen on Server Fault - Search for 'Server Fault'
Mysql was started: /usr/bin/mysqld_safe --datadir=/srv/mysql/myDB --log-error=/srv/mysql/logs/mysqld-myDB.log --pid-file=/srv/mysql/pids/mysqld-myDB.pid --user=mysql --socket=/srv/mysql/sockets/mysql-myDB.sock --port=3700 but when I'm trying to do something: ERROR 1548 (HY000) at line 1: Cannot… >>> More
Why is there a /etc/init.d/mysql file on this Slackware machine? How could it have gotten there?

as seen on Server Fault - Search for 'Server Fault'
A client of my IT-consulting service owns a web-development shop. He's been having problems with a Slackware 12.0 server running MySQL 5.0.67. The machine was set up by the client's sysadmin, who left on bad terms. My client no longer employs a sysadmin. As far as I can tell, the only copy of MySQL… >>> More
mysql: Bind on unix socket: Permission denied

as seen on Server Fault - Search for 'Server Fault'
Can't start mysql with: sudo /usr/bin/mysqld_safe --datadir=/srv/mysql/myDB --log-error=/srv/mysql/logs/mysqld-myDB.log --pid-file=/srv/mysql/pids/mysqld-myDB.pid --user=mysql --socket=/srv/mysql/sockets/mysql-myDB.sock --port=3700 120222 13:40:48 mysqld_safe Starting mysqld daemon with databases… >>> More
MySQL – Learning MySQL Online in 6 Hours – MySQL Fundamentals in 320 Minutes

as seen on SQL Authority - Search for 'SQL Authority'
MySQL is one of the most popular database language and I have been recently working with it a lot. Data have no barrier and every database have their own place. I have been working with MySQL for quite a while and just like SQL Server, I often find lots of people asking me if I have a tutorial which… >>> More

Related posts about architecture

3-tier architecture v. 3-server architecture

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm building a traditional .NET MVC site, so I've got a natural 3-tier software architecture setup (presentation in the form of Views, business layer in the controller, and data layer in the models and data access layer). When I've deployed such sites, it usually goes either on one server (where… >>> More
"cloud architecture" concepts in a system architecture diagrams

as seen on Stack Overflow - Search for 'Stack Overflow'
If you design a distributed application for easy scale-out, or you just want to make use of any of the new “cloud computing” offerings by Amazon, Google or Microsoft, there are some typical concepts or components you usually end up using: distributed blob storage (aka S3) asynchronous, durable message… >>> More
Is the PowerPC (G4) architecture better than Intel x86 architecture for certain types of application

as seen on Super User - Search for 'Super User'
I have a PowerMac G4 from around the year 2000. It's a serviceable Unix machine but I don't do much with it since I have plenty of Pentium 4 machines around with Linux on them. I was wondering if the PowerPC based machine is capable of handling certain tasks faster or "better" than Intel based machines… >>> More
SAAS architecture and salesforce database architecture

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello all, I am architecting a software project and I want to create a SAAS (Software As a service) one. I want to model my application along the lines of Salesforce. I really like there customization features but I am not sure how they really go about it. I read that they create an ID for every… >>> More
Is there a Telecommunications Reference Architecture?

as seen on Oracle Blogs - Search for 'Oracle Blogs'
@font-face { font-family: "Arial"; }@font-face { font-family: "Courier New"; }@font-face { font-family: "Wingdings"; }@font-face { font-family: "Cambria"; }p.MsoNormal, li.MsoNormal, div.MsoNormal { margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman"; }p.MsoListParagraph… >>> More