Search Results

Search found 69357 results on 2775 pages for 'data oriented design'.

Page 56/2775 | < Previous Page | 52 53 54 55 56 57 58 59 60 61 62 63  | Next Page >

• Big Data – Interacting with Hadoop – What is PIG? – What is PIG Latin? – Day 16 of 21

  - by Pinal Dave

  In yesterday’s blog post we learned the importance of the HIVE in Big Data Story. In this article we will understand what is PIG and PIG Latin in Big Data Story. Yahoo started working on Pig for their application deployment on Hadoop. The goal of Yahoo to manage their unstructured data. What is Pig and What is Pig Latin? Pig is a high level platform for creating MapReduce programs used with Hadoop and the language we use for this platform is called PIG Latin. The pig was designed to make Hadoop more user-friendly and approachable by power-users and nondevelopers. PIG is an interactive execution environment supporting Pig Latin language. The language Pig Latin has supported loading and processing of input data with series of transforming to produce desired results. PIG has two different execution environments 1) Local Mode – In this case all the scripts run on a single machine. 2) Hadoop – In this case all the scripts run on Hadoop Cluster. Pig Latin vs SQL Pig essentially creates set of map and reduce jobs under the hoods. Due to same users does not have to now write, compile and build solution for Big Data. The pig is very similar to SQL in many ways. The Ping Latin language provide an abstraction layer over the data. It focuses on the data and not the structure under the hood. Pig Latin is a very powerful language and it can do various operations like loading and storing data, streaming data, filtering data as well various data operations related to strings. The major difference between SQL and Pig Latin is that PIG is procedural and SQL is declarative. In simpler words, Pig Latin is very similar to SQ Lexecution plan and that makes it much easier for programmers to build various processes. Whereas SQL handles trees naturally, Pig Latin follows directed acyclic graph (DAG). DAGs is used to model several different kinds of structures in mathematics and computer science. DAG Tomorrow In tomorrow’s blog post we will discuss about very important components of the Big Data Ecosystem – Zookeeper. Reference: Pinal Dave (http://blog.sqlauthority.com) Filed under: Big Data, PostADay, SQL, SQL Authority, SQL Query, SQL Server, SQL Tips and Tricks, T SQL

  Read the article

• What do you do to get your software design robust, flexible and clear?

  - by Oscar

  I am still getting mature as a software engineering/designer/architect, as you may want to call. At this point in time, I am getting small projects, private projects and so on. What I noticed is that even though I think about the SW structure, design some diagrams, have they really clear in my mind when I start coding, at the end, my software is not flexible and clear as I would like to. I would like to ask you what kind of approaches, mechanisms or even tricks do you use, to get your software (and SW design) flexible, robust and clear (easy to understand and use). So.... Any ideas to give to a beginner?

  Read the article

• Why is design by contract considered an alternative to the pseudo programming process?

  - by zoopp

  Right now I'm reading Code Complete by Steve McConnell and in chapter 9 he talks about the Pseudo Programming Process (PPP). From what I've understood, the PPP is a way of programming in which the programmer first writes the pseudo code for the routine he's working on, then refines it up to the point where pretty much each pseudo code line can be implemented in 1-3 lines of code, then writes the code in the designated programming language and finally the pseudo code is saved as comments for the purpose of documenting the routine. In chapter 9.4 the author mentions alternatives to the PPP, one of which is 'design by contract'. In design by contract you basically assert preconditions and postconditions of each routine. Now why would that be considered an alternative? To me it seems obvious that I should use both techniques at the same time and not chose one over the other.

  Read the article

• Raid 5 mdadm Problem - Help Please

  - by user66260

  My Raid 5 array (4 1tb Disks WD10EARS) had was showing as degraded. I looked and one of the disks wasnt installed, so i re-added it with the mdadm add command. the array is now showing as (null)Array , but cant be mounted if i run: root@warren-P5K-E:/home/warren# sudo mdadm --misc --detail /dev/md0 I get: mdadm: cannot open /dev/md0: No such file or directory and running: root@warren-P5K-E:/home/warren# cat /proc/mdstat gives me: Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] unused devices: < none > The data is very important root@warren-P5K-E:/home/warren# mdadm --examine /dev/sda /dev/sda: Magic : a92b4efc Version : 0.90.00 UUID : 00000000:00000000:00000000:00000000 Creation Time : Sat May 26 12:08:14 2012 Raid Level : -unknown- Raid Devices : 0 Total Devices : 4 Preferred Minor : 0 Update Time : Sat May 26 12:08:40 2012 State : active Active Devices : 0 Working Devices : 4 Failed Devices : 0 Spare Devices : 4 Checksum : 82d5b792 - correct Events : 1 Number Major Minor RaidDevice State this 1 8 0 1 spare /dev/sda 0 0 8 16 0 spare /dev/sdb 1 1 8 0 1 spare /dev/sda 2 2 8 32 2 spare /dev/sdc 3 3 8 48 3 spare /dev/sdd root@warren-P5K-E:/home/warren# mdadm --examine /dev/sdb /dev/sdb: Magic : a92b4efc Version : 0.90.00 UUID : 00000000:00000000:00000000:00000000 Creation Time : Sat May 26 12:08:14 2012 Raid Level : -unknown- Raid Devices : 0 Total Devices : 4 Preferred Minor : 0 Update Time : Sat May 26 12:08:40 2012 State : active Active Devices : 0 Working Devices : 4 Failed Devices : 0 Spare Devices : 4 Checksum : 82d5b7a0 - correct Events : 1 Number Major Minor RaidDevice State this 0 8 16 0 spare /dev/sdb 0 0 8 16 0 spare /dev/sdb 1 1 8 0 1 spare /dev/sda 2 2 8 32 2 spare /dev/sdc 3 3 8 48 3 spare /dev/sdd root@warren-P5K-E:/home/warren# oot@warren-P5K-E:/home/warren# mdadm --examine /dev/sdc /dev/sdc: Magic : a92b4efc Version : 0.90.00 UUID : 00000000:00000000:00000000:00000000 Creation Time : Sat May 26 12:08:14 2012 Raid Level : -unknown- Raid Devices : 0 Total Devices : 4 Preferred Minor : 0 Update Time : Sat May 26 12:08:40 2012 State : active Active Devices : 0 Working Devices : 4 Failed Devices : 0 Spare Devices : 4 Checksum : 82d5b7b4 - correct Events : 1 Number Major Minor RaidDevice State this 2 8 32 2 spare /dev/sdc 0 0 8 16 0 spare /dev/sdb 1 1 8 0 1 spare /dev/sda 2 2 8 32 2 spare /dev/sdc 3 3 8 48 3 spare /dev/sdd root@warren-P5K-E:/home/warren# mdadm --examine /dev/sdd /dev/sdd: Magic : a92b4efc Version : 0.90.00 UUID : 00000000:00000000:00000000:00000000 Creation Time : Sat May 26 12:08:14 2012 Raid Level : -unknown- Raid Devices : 0 Total Devices : 4 Preferred Minor : 0 Update Time : Sat May 26 12:08:40 2012 State : active Active Devices : 0 Working Devices : 4 Failed Devices : 0 Spare Devices : 4 Checksum : 82d5b7c6 - correct Events : 1 Number Major Minor RaidDevice State this 3 8 48 3 spare /dev/sdd 0 0 8 16 0 spare /dev/sdb 1 1 8 0 1 spare /dev/sda 2 2 8 32 2 spare /dev/sdc 3 3 8 48 3 spare /dev/sdd That on the 4 drives.

  Read the article

• What should be the minimal design/scope documentation before development begins?

  - by Oliver Hyde

  I am a junior developer working on my own in the programming aspect of projects. I am given a png file with 5-6 of the pages designed, most times in specific detail. From this I'm asked to develop the back end system needed to maintain the website, usually a cataloging system with products, tags and categories and match the front end to the design. I find myself in a pickle because when I make decisions based on assumptions about the flow of the website, due to a lack of outlining details, I get corrected and am required to rewrite the code to suit what was actually desired. This process happens multiple times throughout a project, often times on the same detail, until it's finally finished, with broken windows all through it. I understand that projects have scope creep, and can appreciate that I need to plan for this, but I feel that in this situation, I'm not receiving enough outlining details to effectively plan for the project, resulting in broken code and a stressed mind. What should be the minimal design/scope documentation I receive before I begin development?

  Read the article

• What benefits does a game design degree have for a hobby game programmer?

  - by sm4

  I am interested in studying game design, not because I want a job in the games industry, but because I am interested in the subject itself. I read the following questions, but they mostly deal with the effects on your career in game industry. Should I consider a graduate degree in game development? Game Development Degree vs Computer Science Degree First I thought a game development degree could be beneficial. But from the websites of colleges that offer such degrees, I feel like its more about basic programming with examples from games. This college offers game design degrees, for example. My question is, can I benefit from such a degree when I already have a degree in Computer Science, I already know programming, I'm already developing a game and finally, I have this site to help me when I get stuck?

  Read the article

• What's the difference been NoSQL and a Column-Oriented database?

  - by Tedk

  The more I read about NoSQL, the more it begins to sound like a column oriented database to me. What's the difference between NoSQL (e.g. CouchDB, Cassandra, MongoDB) and a column oriented database (e.g. Vertica, MonetDB)?

  Read the article

• Is this a well known design pattern? what is it's name

  - by GenEric35

  Hi I have seen this often in code, but when I speak of it i don't know the name of such 'pattern' I have a method with 2 arguments that calls an overloaded method that has 3 arguments and intentionality sets the 3rd one to empty string. public DoWork(string name, string phoneNumber) { CreateContact(name, phoneNumber, string.Empty) } public DoWork(string name, string phoneNumber, string emailAddress) { //do the work } The reason I'm doing this is I to not duplicate code, and allow the existing callers to still call the method that has only 2 parameters. I have associate a few tags to this question, but it probably fit in more categories of questions. Is this a pattern, and does it have a name?

  Read the article

• Level design - Are games development degrees worth it? [on hold]

  - by Tristan

  I want to go into Level design or Environment design and wondering if degrees are at all worth it for this area, as long as you have a good portfolio. I'm currently on a "Computing and Games Development" course and I feel like dropping out because I am not enjoying it. It's mostly computer based, which I'm not doing that great at, and little games Dev. But I don't know what do to... I do already have some high level education with a "Level 3 extended diploma". Thanks.

  Read the article

• Any technical references for game-oriented icons and symbols?

  - by willc2

  To make localizing easier, I'm using icons to show in-game information like achievements and bonuses. Coming up with good designs isn't easy, especially when it has to be integrated into the rest of the game's art style. Can I do better than looking at some random selection of existing games? Are there any reference books or sites that cover game graphics specifically? I'm looking for more theory and best-practices rather than pre-made graphics.

  Read the article

• What is a good design pattern / lib for iOS 5 to synchronize with a web service?

  - by Junto

  We are developing an iOS application that needs to synchronize with a remote server using web services. The existing web services have an "operations" style rather than REST (implemented in WCF but exposing JSON HTTP endpoints). We are unsure of how to structure the web services to best fit with iOS and would love some advice. We are also interested in how to manage the synchronization process within iOS. Without going into detailed specifics, the application allows the user to estimate repair costs at a remote site. These costs are broken down by room and item. If the user has an internet connection this data can be sent back to the server. Multiple photographs can be taken of each item, but they will be held in a separate queue, which sends when the connection is optimal (ideally wifi). Our backend application controls the unique ids for each room and item. Thus, each time we send these costs to the server, the server echoes the central database ids back, thus, that they can be synchronized in the mobile app. I have simplified this a little, since the operations contract is actually much larger, but I just want to illustrate the basic requirements without complicating matters. Firstly, the web service architecture: We currently have two operations: GetCosts and UpdateCosts. My assumption is that if we used a strict REST architecture we would need to break our single web service operations into multiple smaller services. This would make the services much more chatty and we would also have to guarantee a delivery order from the app. For example, we need to make sure that containing rooms are added before the item. Although this seems much more RESTful, our perception is that these extra calls are expensive connections (security checks, database calls, etc). Does the type of web api (operation over service focus) determine chunky vs chatty? Since this is mobile (3G), are we better handling lots of smaller messages, or a few large ones? Secondly, the iOS side. What is the current advice on how to manage data synchronization within the iOS (5) app itself. We need multiple queues and we need to guarantee delivery order in each queue (and technically, ordering between queues). The server needs to control unique ids and other properties and echo them back to the application. The application then needs to update an internal database and when re-updating, make sure the correct ids are available in the update message (essentially multiple inserts and updates in one call). Our backend has a ton of business logic operating on these cost estimates. We don't want any of this in the app itself. Currently the iOS app sends the cost data, and then the server echoes that data back with populated ids (and other data). The existing cost data is deleted and the echoed response data is added to the client database on the device. This is causing us problems, because any photos might not have been sent, but the original entity tree has been removed and replaced. Obviously updating the costs tree rather than replacing it would remove this problem, but I'm not sure if there are any nice xcode libraries out there to do such things. I welcome any advice you might have.

  Read the article

• Document-oriented database - What if the document definitions change?

  - by Sebastian Hoitz

  As I understand it, you can enter any non-structured information into a document-oriented database. Let's imagine a document like this: { name: 'John Blank', yearOfBirth: 1960 } Later, in a new version, this structure is refactored to { firstname: 'John', lastname: 'Blank', yearOfBirth: 1960 } How do you do this with Document-Oriented databases? Do you have to prepare merge-scripts, that alter all your entries in the database? Or are there better ways you can handle changes in the structure?

  Read the article

• Is there a good book or articles to learn about 2D Game Design and Effects?

  - by user28015

  I am not looking for a read how to develop games and how to implement one. I am looking for a general about possible effects in 2D Games and about general design of modern 2D gaming. I have programmed several smaller games over the years and also read books like "Golden Rules of Game Programming" by Martin Bronwlo. So I know how to implement games. What I am looking for are 2 things: Finishing touches such as effects like explosions, particles etc. Not how to make them, but how to design them so it looks right and cool. How to make a 2D game feel "more right" so that users get a satisfying gaming experience. I played a lot of 2D games but I could use some more advice.

  Read the article

• What is the best database design for managing historical information? [closed]

  - by Emmad Kareem

  Say you have a Person table with columns such as: ID, FirstName, LastName, BirthCountry, ...etc. And you want to keep track of changes on such a table. For example, the user may want to see previous names of a person or previous addresses, etc. The normalized way is to keep names in separate table, addresses in a separate table,...etc. and the main person table will contain only the information that you are not interested in monitoring changes for (such information will be updated in place). The problem I see here, aside form the coding hassle due to the extensive number of joins required in a real-life situation, is that I have never seen this type of design in any real application (maybe because most did not provide this feature!). So, is there a better way to design this? Thanks.

  Read the article

• How to determine the right amount of up front design?

  - by Gian

  Software developers occasionally are called upon to write fairly complex bits of software under tight deadlines. Often, it seems like the quickest thing to do is to simply start coding, and solve the problems as they arise. However, this approach can come back to bite you—often costing time or money in the long run! How do we determine the right amount of up front design work? If your work environment actively discourages you from thinking about things up front, how do you handle that? How can we manage risk if we eschew up-front thinking (by choice or under duress) and figure out the problems as they arise? Does the amount of up front design depend entirely on the size or complexity of the task, or is it based on something else?

  Read the article

• [Design Question] When to open a link on a new window?

  - by Ian

  Hi All, When designing a web application/web site, is there an accepted practice on when to open a link on a new window? Currently, if the site being linked to is outside the domain (say Google.com), I am always launching it on a new window. If the page being linked is within the same domain, I open it on the current active window. I've read somewhere the opening links on a new window explicitly is being frowned upon. Thanks!

  Read the article

• Design pattern: static function call with input/output containers?

  - by Pavlo Dyban

  I work for a company in software research department. We use algorithms from our real software and wrap them so that we can use them for prototyping. Every time an algorithm interface changes, we need to adapt our wrappers respectively. Recently all algorithms have been refactored in such a manner that instead of accepting many different inputs and returning outputs via referenced parameters, they now accept one input data container and one output data container (the latter is passed by reference). Algorithm interface is limited to a static function call like that: class MyAlgorithm{ static bool calculate(MyAlgorithmInput input, MyAlgorithmOutput &output); } This is actually a very powerful design, though I have never seen it in a C++ programming environment before. Changes in the number of parameters and their data types are now encapsulated and they don't change the algorithm callback. In the latest algorithm which I have developed I used the same scheme. Now I want to know if this is a popular design pattern and what it is called.

  Read the article

• Does my API design violate RESTful principles?

  - by peta

  Hello everybody, I'm currently (I try to) designing a RESTful API for a social network. But I'm not sure if my current approach does still accord to the RESTful principles. I'd be glad if some brighter heads could give me some tips. Suppose the following URI represents the name field of a user account: people/{UserID}/profile/fields/name But there are almost hundred possible fields. So I want the client to create its own field views or use predefined ones. Let's suppose that the following URI represents a predefined field view that includes the fields "name", "age", "gender": utils/views/field-views/myFieldView And because field views are kind of higher logic I don't want to mix support for field views into the "people/{UserID}/profile/fields" resource. Instead I want to do the following: utils/views/field-views/myFieldView/{UserID} Though Leonard Richardson & Sam Ruby state in their book "RESTful Web Services" that a RESTful design is somehow like an "extreme object oriented" approach, I think that my approach is object oriented and therefore accords to RESTful principles. Or am I wrong? When not: Are such "object oriented" approaches generally encouraged when used with care and in order to avoid query-based REST-RPC hybrids? Thanks for your feedback in advance, peta

  Read the article

• Deploying BAM Data Control Application to WLS server

  - by [email protected]

  var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); try { var pageTracker = _gat._getTracker("UA-15829414-1"); pageTracker._trackPageview(); } catch(err) {} Typically we would test our ADF pages that use BAM Data control using integrated wls server (ADRS). If we have to deploy this same application to a standalone WLS we have to make sure we have the BAM server connection created in WLS.unless we do that we may face runtime errors.In Development mode of WLS(Reference) For development-mode WebLogic Server, you can set the mode to OVERWRITE to test user names and passwords. You can set the mode by running setDomainEnv.cmd or setDomainEnv.sh with the following option added to the command. Add the following to the JAVA_PROPERTIES entry in the <FMW_HOME>/user_projects/domains/<yourdomain>/bin/setDomainEnv.sh file: -Djps.app.credential.overwrite.allowed=true In Production mode of WLS Enable MDS Create and/or Register your MDS repository. For more details refer this Edit adf-config.xml from your application and add the following tag <adf-mds-config xmlns="http://xmlns.oracle.com/adf/mds/config">     <mds-config version="11.1.1.000">     <persistence-config>   <metadata-store-usages>     <metadata-store-usage default-cust-store="true" deploy-target="true" id="myRepos">     </metadata-store-usage>   </metadata-store-usages>   </persistence-config>           </mds-config>  </adf-mds-config>Deploy the application to WLS server after picking the appropriate repository during deployment from the MDS Repository dialog that pops up Enterprise Manager (Use these steps if using a version prior to 11gR1 PS1 release of JDeveloper) Go to EM (http://<host>:<port>/EMIn the left pane, deployments select Application1(your application)In the right pane, top dropdown select "System Mbean Browser->oracle.adf.share.connections->Server: AdminServer->Server: AdminServer->Application:<Appname>->ADFConnections"Right pane click "Operations->CreateConnection"Enter Connection Type as "BAMConnection"Enter the connection name same as the one defined in JdevClick "Invoke"Click "Return"Click on Operation->SaveNow in the ADFConnections in the navigator, select the connection just created and enter all the configuration details.Save and run the page. Enterprise Manager (Use these steps or the steps above if using 11gR1 PS1 or newer) Go to EM (http://<host>:<port>/EMIn the left pane, deployments select Application1(your application)In the right pane, click on "Application Deployment" to invoke to dropdown. In that select "ADF -> Configure ADF Connections"Select Connection Type as "BAM" from the drop downEnter Connection Type as to be the same as the one defined in JDevClick on "Create Connection". This should add a new row below under "BAM Connections"Select the new connection and click on the "Edit" icon. This should bring up a dialogSpecific appropriate values for all connection parameters - Username, password, BAM Server Host, BAM Server Port, Webtier Server Host, Webtier Server Port and BAM Webtier Protocol - and then click on OK to dismiss the dialogClick on "Apply"Run the page page.

  Read the article

• UX Design Principles Pluralsight course review

  - by pluginbaby

  I've just finished the "Creating User Experiences: Fundamental Design Principles" course on Pluralsight, I am glad I took it, and here is why you should. The course is held by Billy Hollis, an internationally known author and speaker focused on user experience design. It was published in May 2012, so it is quite fresh (You’ll hear some reference to XAML, even if the content is not focused on any particular technology). I think what I liked the most about this course is the fact that Billy is not just imposing design ideas and pushing them in your throat (which would be too confronting for us developers, even if he was right), he spends a fair share amount of time explaining each topics, and illustrate them with great metaphors. If you are a minimum open minded you should get great value out of this course. Billy makes you think outside the box, he encourages you to use your right side brain, and understand design principles by simply looking at what’s around us (physical objects, nature, …). During the course he refers several time to "don't make me think" a book on UX design, which is about giving confidence to users, by making it easier for them to achieve their goals when using your app. Billy thinks that every developer can participate in elaborating good design when building software, not only designers should be involved. Get away of the easy path "let's build functional stuff for now and we will hire a designer later if we have time and budget". The course is also live and interactive as the author suggests that you do some live exercises during each module. He actually makes you realize and understand by yourself the need for change. We’re in a new era of software and devices, where grids and menus aren't enough. You can’t remain satisfied by just making things possible, you need to make them easier for your users. Understanding some fundamental design principles will help. This course can definitely be followed by any developers who wants to improve user experience of software they are working on, and I definitely recommend it.

  Read the article

• Folders in SQL Server Data Tools

  - by jamiet

  Recently I have begun a new project in which I am using SQL Server Data Tools (SSDT) and SQL Server Integration Services (SSIS) 2012. Although I have been using SSDT & SSIS fairly extensively while SQL Server 2012 was in the beta phase I usually find that you don’t learn about the capabilities and quirks of new products until you use them on a real project, hence I am hoping I’m going to have a lot of experiences to share on my blog over the coming few weeks. In this first such blog post I want to talk about file and folder organisation in SSDT. The predecessor to SSDT is Visual Studio Database Projects. When one created a new Visual Studio Database Project a folder structure was provided with “Schema Objects” and “Scripts” in the root and a series of subfolders for each schema: Apparently a few customers were not too happy with the tool arbitrarily creating lots of folders in Solution Explorer and hence SSDT has gone in completely the opposite direction; now no folders are created and new objects will get created in the root – it is at your discretion where they get moved to: After using SSDT for a few weeks I can safely say that I preferred the older way because I never used Solution Explorer to navigate my schema objects anyway so it didn’t bother me how many folders it created. Having said that the thought of a single long list of files in Solution Explorer without any folders makes me shudder so on this project I have been manually creating folders in which to organise files and I have tried to mimic the old way as much as possible by creating two folders in the root, one for all schema objects and another for Pre/Post deployment scripts: This works fine until different developers start to build their own different subfolder structures; if you are OCD-inclined like me this is going to grate on you eventually and hence you are going to want to move stuff around so that you have consistent folder structures for each schema and (if you have multiple databases) each project. Moreover new files get created with a filename of the object name + “.sql” and often people like to have an extra identifier in the filename to indicate the object type: The overall point is this – files and folders in your solution are going to change. Some version control systems (VCSs) don’t take kindly to files being moved around or renamed because they recognise the renamed/moved file simply as a new file and when they do that you lose the revision history which, to my mind, is one of the key benefits of using a VCS in the first place. On this project we have been using Team Foundation Server (TFS) and while it pains me to say it (as I am no great fan of TFS’s version control system) it has proved invaluable when dealing with the SSDT problems that I outlined above because it is integrated right into the Visual Studio IDE. Thus the advice from this blog post is: If you are using SSDT consider using an Visual-Studio-integrated VCS that can easily handle file renames and file moves I suspect that fans of other VCSs will counter by saying that their VCS weapon of choice can handle renames/file moves quite satisfactorily and if that’s the case…great…let me know about them in the comments. This blog post is not an attempt to make people use one particular VCS, only to make people aware of this issue that might rise when using SSDT. More to come in the coming few weeks! @jamiet

  Read the article

• Big Data – Interacting with Hadoop – What is Sqoop? – What is Zookeeper? – Day 17 of 21

  - by Pinal Dave

  In yesterday’s blog post we learned the importance of the Pig and Pig Latin in Big Data Story. In this article we will understand what is Sqoop and Zookeeper in Big Data Story. There are two most important components one should learn when learning about interacting with Hadoop – Sqoop and Zookper. What is Sqoop? Most of the business stores their data in RDBMS as well as other data warehouse solutions. They need a way to move data to the Hadoop system to do various processing and return it back to RDBMS from Hadoop system. The data movement can happen in real time or at various intervals in bulk. We need a tool which can help us move this data from SQL to Hadoop and from Hadoop to SQL. Sqoop (SQL to Hadoop) is such a tool which extract data from non-Hadoop data sources and transform them into the format which Hadoop can use it and later it loads them into HDFS. Essentially it is ETL tool where it Extracts, Transform and Load from SQL to Hadoop. The best part is that it also does extract data from Hadoop and loads them to Non-SQL (or RDBMS) data stores. Essentially, Sqoop is a command line tool which does SQL to Hadoop and Hadoop to SQL. It is a command line interpreter. It creates MapReduce job behinds the scene to import data from an external database to HDFS. It is very effective and easy to learn tool for nonprogrammers. What is Zookeeper? ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. In other words Zookeeper is a replicated synchronization service with eventual consistency. In simpler words – in Hadoop cluster there are many different nodes and one node is master. Let us assume that master node fails due to any reason. In this case, the role of the master node has to be transferred to a different node. The main role of the master node is managing the writers as that task requires persistence in order of writing. In this kind of scenario Zookeeper will assign new master node and make sure that Hadoop cluster performs without any glitch. Zookeeper is the Hadoop’s method of coordinating all the elements of these distributed systems. Here are few of the tasks which Zookeepr is responsible for. Zookeeper manages the entire workflow of starting and stopping various nodes in the Hadoop’s cluster. In Hadoop cluster when any processes need certain configuration to complete the task. Zookeeper makes sure that certain node gets necessary configuration consistently. In case of the master node fails, Zookeepr can assign new master node and make sure cluster works as expected. There many other tasks Zookeeper performance when it is about Hadoop cluster and communication. Basically without the help of Zookeeper it is not possible to design any new fault tolerant distributed application. Tomorrow In tomorrow’s blog post we will discuss about very important components of the Big Data Ecosystem – Big Data Analytics. Reference: Pinal Dave (http://blog.sqlauthority.com) Filed under: Big Data, PostADay, SQL, SQL Authority, SQL Query, SQL Server, SQL Tips and Tricks, T SQL

  Read the article

• SQL SERVER – Unable to DELETE Project in Data Quality Projects (DQS)

  - by pinaldave

  Here is the email which made me write this blog post. When I write a blog post I write keeping in mind that if the developer is not familiar with the concept he will attempt this on the development server. If due to any reason you attempt it on any other server than your personal server, developer should make sure to have complete confidence on his own expertise and understand the risk behind it.  Well, let us read the email which I received. I have modified it a bit to remove information related to organizational and individual. “I just read your blog post on Beginning DQS. I went ahead and followed every single screenshot and it worked fine. I was able to execute the DQS project successfully. However, the same blog post got me in trouble – a serious trouble. After first successful deployment I went ahead and created a few of my own knowledge base and projects. I played around a bit and then decided to get back to real work. Now we had deployed DQS on production server only, so experiment on production server. Now, when I got back to my work, I forgot to close all the windows. My manager found the window open and have seen my test projects. He has asked me to delete my experiments immediately and have said words which I cannot write to you. Here is the problem. I am not able to delete the project which I have created earlier. I am able to open it and play with it but the delete option is disabled and grayed out (see attached image). Now I believe there is nothing wrong with this project as it was just a test project. Would you please write to my manager that it is not harmful to leave that project there as it is? It is also not using any resources. I think he will believe you.” As I said this kind of email makes me uncomfortable. I do not want someone to execute anything on production server. I often write notes and disclaimer on my post when something is dangerous to execute on production server. However, if someone is not expert with SQL Server and attempts something new on production server, I think the major issue is here with the person (admin) who gave new developer permission to production server. This has to be carefully avoided. Here was my response to the individual. “I cannot write to your manager anything as he has not asked me anything. Honestly I believe he is correct in his behavior as you should have not executed anything on the production server without prior approval and testing on the development server. Any R&D must be done on local box or development box. I suggest you request your manager to prevent access to users who does not need access. If he is a good manager, he might have already implemented by now recent event. I also see your screenshot. Here is the issue: While you were playing with project, you might have closed the project half the way, without completing it. Due to the same reason it is locked. You can open and continue from the same place where you have left the project. If you do not need the project any more. Right click on it, click on unlock the project. This will enable the DELETE option and now you can delete the project. Next time, be safe out there. It may be dangerous to have admin access to production server when not needed.“ I have yet not heard from him but I believe he will take my words positively. Reference: Pinal Dave (http://blog.SQLAuthority.com) Filed under: PostADay, SQL, SQL Authority, SQL Query, SQL Server, SQL Tips and Tricks, T SQL, Technology Tagged: Data Quality Services, DQS

  Read the article

• Do your own design jobs and make it look professional

  - by Webgui

  Looks and design is becoming more and more important for customers and organizations event when we deal with internal enterprise applications. However,  many web developers who work on business apps end up not investing resources on the design. The reason may be that they ran out of time so with their client's pressure there was no choice but to skip past the design process. In some cases, especially in sall software houses, there are no trained professional designers and the developers have to do both jobs. Since designing web applications can be very complex and requires mastering several languages and concepts, unless a big budget was allocated to the project it is very hard to produce a professional custom design. For that exact reasons, Visual WebGui integrated Point & Click Design Tools within its Web/Cloud Development Platform. Those tools allow developers to customize the UI look of the applications they build in a visual way that is fairly simple and doesn't require coding or mastering HTML, CSS and JavaScript in order to design. The development tools also allow professional designers easier work interface with the developers and quicly create new skins. So if you are interested in getting your design job done much easier, you should probably tune in for about an hour and find out how. Click here to register: https://www1.gotomeeting.com/register/740450625

  Read the article

• Best strategy for synching data in iPhone app

  - by iamj4de

  I am working on a regular iPhone app which pulls data from a server (XML, JSON, etc...), and I'm wondering what is the best way to implement synching data. Criteria are speed (less network data exchange), robustness (data recovery in case update fails), offline access and flexibility (adaptable when the structure of the database changes slightly, like a new column). I know it varies from app to app, but can you guys share some of your strategy/experience? For me, I'm thinking of something like this: 1) Store Last Modified Date in iPhone 2) Upon launching, send a message like getNewData.php?lastModifiedDate=... 3) Server will process and send back only modified data from last time. 4) This data is formatted as so: <+><data id="..."></data></+> // add this to SQLite/CoreData <-><data id="..."></data></-> // remove this <%><data id="..."><attribute>newValue</attribute></data></%> // new modified value I don't want to make <+, <-, <%... for each attribute as well, because it would be too complicated, so probably when receive a <% field, I would just remove the data with the specified id and then add it again (assuming id here is not some automatically auto-incremented field). 5) Once everything is downloaded and updated, I will update the Last Modified Date field. The main problem with this strategy is: If the network goes down when I am updating something = the Last Modified Date is not yet updated = next time I relaunch the app, I will have to go through the same thing again. Not to mention potential inconsistent data. If I use a temporary table for update and make the whole thing atomic, it would work, but then again, if the update is too long (lots of data change), the user has to wait a long time until new data is available. Should I use Last-Modified-Date for each of the data field and update data gradually?

  Read the article

< Previous Page | 52 53 54 55 56 57 58 59 60 61 62 63  | Next Page >