Startup creates development framework for Hadoop
June 29, 2010 —
Building applications that run map/reduce jobs shouldn't be rocket science, says the founder of a startup company that has developed an application development framework for Apache Hadoop.
Hadoop is a Java-based framework that brings methods to large volumes of data on disk for map/reduce data analysis jobs. It is used by Internet search providers, social networking sites such as Facebook and Twitter, financial service providers, and in Amazon's Elastic Compute Cloud to process large data sets.
Karmasphere, a company founded by alumni of Aventail and Raytheon Systems, today released three new products based on its own framework. The products are Karmasphere Studio Analyst, Karmasphere Studio Professional Edition and Karmasphere Client.
The client sits at the lowest level of the Karmasphere framework, and handles provider interfaces and clusters created with the bigdata open-source computing fabric solution. Above that layer, infrastructure services manage clusters and file systems as well as debugging applications and reporting.
The framework also provides language support for Cascading, Hive QL, Java and Pig, in addition to an integrated graphical user environment.
Hadoop, an open-source map/reduce framework, was originally used by computer scientists and other "rocket scientist" developers, said Karmasphere founder and CEO Martin Hall. "Hadoop is great but not easy to use."
Hadoop requires developers to understand its file system, and there is no API compatibility between versions and distributions, Hall explained. Business users require an abstraction from that complexity, and analysts want something that looks like SQL, he said.
On the other hand, developers want to be able to see what is going on inside of a cluster, he said.
Karmasphere Studio Professional Edition provides developers with debugging capabilities and visualization tools to determine what occurred during a job. It also has the ability to export Hadoop jobs as binary applications that may be run on any supported operating system without any third-party software. The solutions run on Linux, Mac OS and Windows. Alternatively, a developer could create a Java API package to be used in other applications, said CTO and cofounder Ben Mankin.
Karmasphere Studio Analyst enables technical analysts to perform ad hoc analysis of bigdata hives (which are very large clusters), Mankin said. "It's SQL on map/reduce...the Holy Grail."
Differences in distributions and versions are handled by the SQL engine, said Mankin. "There's no infrastructure, no network tunnels, third-party servers, or funny error messages."
A standalone edition of Karmasphere Client is also available. It is a Java library that makes Hadoop applications built on top of it more fault-tolerant, and it saves bandwidth, Hall said. Event monitoring is also provided.
The company also ships a community edition of Karmasphere Studio that runs in Eclipse and NetBeans. It does not include all of the visualization and debugging tools that are found in the professional edition.
Related Search Term(s): Hadoop
Share this link: http://sdt.bz/34448
Most Read Latest News Blog Resources
Virtualization: Not just for machines anymore
Network virtualization allows multi-tiered applications to behave as though they were in a physical network
|
|
Zeichick’s Take: The handheld and the tablet, circa 1976
Texas Instruments' and Hewlett-Packard's calculators were doing things decades ago we take for granted today
|
|
Google talks tools at AnDevCon III
New 3D debugging tool and recent ADK changes are detailed by Google developers at the third Android Developer Conference
|
|
Android is the focus of two new design tools
Anywhere Software and Xamarin provide ways for developers to create and test their applications on PCs
|
Virtualization: Not just for machines anymore
Network virtualization allows multi-tiered applications to behave as though they were in a physical network
|
|
Achievements and learning: Gamification comes to businesses and schools
Startup takes page from gamers by offering achievement marks to get developers more engaged in their projects
|
|
Google talks tools at AnDevCon III
New 3D debugging tool and recent ADK changes are detailed by Google developers at the third Android Developer Conference
|
|
SmartBear rolls out new quality solution: API Complete
Software gives organizations ability to write test scripts and monitor APIs by bridging the DevOps divide
|
Creation
To write better software, cultivate your ability to be creative.
|
|
Slick...but who needs it?
compilr.com is a well-designed site and the folks behind it seem to have their heart in the right place. But...who needs it?
|
|
How to be a better software developer
Want to be a better developer? You won't get there by mastering an interesting language or learning a new set of APIs.
|
|
Wooing Galatea
Do yourself a favor and check out Galatea 2.2, a wonderful book by novelist Richard Powers.
|
Five SCM Best Practices
Two-thirds of all software projects fail, according to the Standish Group’s CHAOS study. Improper usage of software configuration management...
|
|
|
Best Practices for Branching and Merging Patterns
Development teams often create a branching pattern, usually drawn out on a white board or in a Visio document, that is used as a model to...
|
|
Automated Error Reporting
We invite you to read a short e-zine that tells you all about automated error reporting for .NET applications. This 8-page e-zine is packed...
|
|
The End of Application Redeploys
Imagine that every time you wanted to write, send or receive an email, you needed to restart your computer. How much time would this take, a...
|