Big Data winners and analytics on display at Big Data TechCon

Alex Handy
October 17, 2013 —  (Page 1 of 2)
Bringing together Big Data and big business is much more work for developers than they may anticipate. At the Big Data Technology Conference in San Francisco yesterday, representatives of both the Big Data world and the big business world mingled to discuss their successes and failures in the field.

One company on hand was MetaScale, a wholly owned subsidiary of Sears Holdings. The company began as Sears' internal Hadoop development and deployment team, but has now been unshackled from the retailer in order to help other businesses implement Big Data infrastructure and processes.

(Big Data means big money: Big Data spending to reach $114 billion in 2018)

Ankur Gupta, who heads sales and marketing at MetaScale, said that Sears began experimenting with Hadoop four years ago, giving the company a leg up on many other enterprises when it comes to Big Data processing maturity.

“We saw a business opportunity, and we thought we could provide an enterprise-based overview that's vendor neutral and platform agnostic,” he said. “So we formed MetaScale to help other companies accelerate their Big Data initiatives, so they don't make the same mistakes we made.

“We help companies get to production faster than they would on their own. We provide a vendor-neutral perspective. No matter if you're an IBM shop, and HP shop or a Teradata shop, we can provide from our experiences what may and may not work for you. Similarly, with Hadoop, we have experiences in all the different distribution providers.”

Andy McNalis, Hadoop infrastructure manager at MetaScale, explained some of the customizations to its Hadoop cluster architecture during his talk, titled “Running, Managing, and Operating Hadoop at Sears.”

He said that typically, within the Sears Hadoop cluster, each machine is a simple non-redundant machine, with a single power supply and a single 4TB hard drive. The cluster's Name Node, however, is a more robust and redundantly equipped box, capable of handling heavier workloads and remaining intact.

McNalis also said Sears uses a second, backup Name Node server, which is not necessarily hot-swappable. Instead, this second Name Node is purely used to back up the metadata of the cluster, allowing for that data to remain in place if the primary Name Node is lost (thus also losing the storage directory).

McNalis said the Sears project has grown significantly since it was started almost four years ago. “We started off with some really tiny clusters, just playing around. We built our first cluster with 50 data nodes. Today, I'm at 485 data nodes in that same cluster. You can just add the data nodes into the cluster. You don't have to take an outage,” he said.

“On Hadoop, I'm at a point now where we add entire racks of servers at a time, and there's no outage, and Hadoop starts using the new machines automatically.”

Related Search Term(s): Big Data, Big Data TechCon, Big Data Technology Conference, Datalog, Hadoop, MADlib, MetaScale, Name Node, Precog, Prolog, Sears, SQL

Pages 1 2 

Share this link:

Doug Cutting: Why Hadoop is still No. 1
How Hadoop become the de facto standard, and what it plans on doing next Read More...

News on Monday  more>>
Android Developer News  more>>
SharePoint Tech Report  more>>
Big Data TechReport  more>>



Download Current Issue

Need Back Issues?

Want to subscribe?