Hadoop on windows server

Posted by Luca Martinetti on Stack Overflow See other posts from Stack Overflow or by Luca Martinetti
Published on 2009-01-22T02:58:02Z Indexed on 2010/06/06 13:42 UTC
Read the original article Hit count: 424

Filed under:

c#

|

Windows

|

clustering

|

hadoop

|

mapreduce

Hello,

I'm thinking about using hadoop to process large text files on my existing windows 2003 servers (about 10 quad core machines with 16gb of RAM)

The questions are:

Is there any good tutorial on how to configure an hadoop cluster on windows?
What are the requirements? java + cygwin + sshd ? Anything else?
HDFS, does it play nice on windows?
I'd like to use hadoop in streaming mode. Any advice, tool or trick to develop my own mapper / reducers in c#?
What do you use for submitting and monitoring the jobs?

Thanks

© Stack Overflow or respective owner

Related posts about c#

.NET WebRequest.PreAuthenticate not quite what it sounds like

as seen on West-Wind - Search for 'West-Wind'
I’ve run into the problem a few times now: How to pre-authenticate .NET WebRequest calls doing an HTTP call to the server – essentially send authentication credentials on the very first request instead of waiting for a server challenge first? At first glance this sound like it should be easy:… >>> More
HttpWebRequest and Ignoring SSL Certificate Errors

as seen on West-Wind - Search for 'West-Wind'
Man I can't believe this. I'm still mucking around with OFX servers and it drives me absolutely crazy how some these servers are just so unbelievably misconfigured. I've recently hit three different 3 major brokerages which fail HTTP validation with bad or corrupt certificates at least according to… >>> More
The dynamic Type in C# Simplifies COM Member Access from Visual FoxPro

as seen on West-Wind - Search for 'West-Wind'
I’ve written quite a bit about Visual FoxPro interoperating with .NET in the past both for ASP.NET interacting with Visual FoxPro COM objects as well as Visual FoxPro calling into .NET code via COM Interop. COM Interop with Visual FoxPro has a number of problems but one of them at least got a lot… >>> More
Dynamic Type to do away with Reflection

as seen on West-Wind - Search for 'West-Wind'
The dynamic type in C# 4.0 is a welcome addition to the language. One thing I’ve been doing a lot with it is to remove explicit Reflection code that’s often necessary when you ‘dynamically’ need to walk and object hierarchy. In the past I’ve had a number of ReflectionUtils that used string based expressions… >>> More
Finding a Relative Path in .NET

as seen on West-Wind - Search for 'West-Wind'
Here’s a nice and simple path utility that I’ve needed in a number of applications: I need to find a relative path based on a base path. So if I’m working in a folder called c:\temp\templates\ and I want to find a relative path for c:\temp\templates\subdir\test.txt I want to receive back subdir\test… >>> More

Related posts about Windows

Server 2012 DFS New Member Issue

as seen on Server Fault - Search for 'Server Fault'
I am trying to add a new member to our DFS topology. We have 3 DCs (VMs - VMware) running Windows server 2012, two servers are located in or Primary site and the third at our DR site. Currently the two servers at our primary site are currently replicating DFS (full mesh) and are working fine. I have… >>> More
error in IIS7 but not on IIS6

as seen on Server Fault - Search for 'Server Fault'
I have a website that is we are now deploying to windows 2008 servers that has worked in the past on IIS6 without a problem. It is using .net 2 framework. Most of the website works. Just when we create a screen report over a certain size on the server we get this error. Event code: 3005 Event… >>> More
error in IIS7 but not on IIS6

as seen on Server Fault - Search for 'Server Fault'
I have a website that is we are now deploying to windows 2008 servers that has worked in the past on IIS6 without a problem. It is using .net 2 framework. Most of the website works. Just when we create a screen report over a certain size on the server we get this error. Event code: 3005 Event… >>> More
Microsoft Management Console stops working when I add snap-in to it

as seen on Super User - Search for 'Super User'
I have Windows 7 Ultimate OS. I'm opening mmc.exe as administrator and trying add Certificates or any other snap-in, then while loading that snap-in MMC breaks and displays following message and after that it closes automatically once I click on close button on that message. What could be the problem… >>> More
Handling keyboard and mouse input (Win API)

as seen on Game Development - Search for 'Game Development'
There is a number of ways to catch mouse or keyboard under Windows. So I tried some of them, but every of them has some advantages and drawbacks. I want to ask you: Which method do use? I've tried these: WM_KEYDOWN/WM_KEYUP - Main disadvantage is that, I can't distinguish between left and right-handed… >>> More