Talend adds enterprise features to data integration suite



Email    print   
May 19, 2009 —  Organizations can’t help but accumulate mountains of data. Yet that data only has value if you can find the value it contains. To do that requires heavy-duty data processing and crunching.

Open-source data integration company Talend has released Integration Suite MPX, which extends its commercial offering to address this need to process extremely large amounts of data. The solution, released today, now includes what the company is calling FileScale technology, and the ability to do massively parallelized processing.

The FileScale technology is based on the MapReduce technology created by Google, according to Yves de Montcheuil, Talend’s vice president of marketing. It is what lets Google process billions of Web pages for indexing and ranking purposes.

“The technology lets us break down complex problems into smaller problems. Then we can get multiple nodes process those problems,” and it all gets built back up automatically by a main node, de Montcheuil explained. FileScale has low-level components that are optimized for such tasks as sorting, filtering and merging, aggregating, and transforming. FileScale will distribute these tasks to where resources are available on the hardware, ensuring faster processing time, he said.

When the problems are broken down into smaller subsets, they can be processed in parallel, simultaneously, and then automatically synched back up after the processing is complete, de Montcheuil said. He cited a benchmark using industry-standard TPC-H data sets that was run on a Sun Blade X6270 server featuring two Intel Xeon 5520 quad-core processors—“admittedly a high-end system but a real one, not something cobbled together just for a benchmark,” he said.

In-memory data sorting reached one million records per second, while 3.3 billion records were sorted through at a speed of 200,000 to 400,000 per second. “That is a level of scalability that hasn’t been reached before,” de Montcheuil said.




Related Search Term(s): Talend


Share this link: http://sdt.bz/33492
 
Most Read Latest News Blog Resources

Add comment


Name*
Email*  
Country     


  • Comment
Loading




close
NEXT ARTICLE
Talend adds Apache-based ESB to product line
Enterprise Service Bus added to line-up of open source integration products Read More...
 
 
 
 
News on Monday
more>>
SharePoint Tech Report
more>>


   

 
 

Download Current Issue
FEBRUARY 2012 PDF ISSUE

Need Back Issues?
DOWNLOAD HERE

Want to subscribe?


 
blogs tab
Are you at risk for burnout?
Burnout is a severe problem and it can strike at any time. Here's how to tell if you are nearing the edge.
02/09/2012 02:16 PM EST

Agility, mom, and apple pie
If we're to evaluate the state-of-the-art in software development, we should start with the values espoused in the Agile Manifesto.
02/07/2012 11:57 AM EST

RIM woos developers with free tablet
How do you get more apps ported to the BlackBerry PlayBook? By giving every developer a free tablet, of course!
02/04/2012 01:57 PM EST

GitHire: Use Headhunters to Find Your Perfect Programmer
Are you a hiring manager tired of scouring the job boards? Check out this new service that will find 5 people interested in your jobs.
02/03/2012 12:17 PM EST

Facebook claims hacker cred
Facebook's SEC S-1 filing form includes a short essay on the Hacker Way by Mark Zuckerberg himself.
02/02/2012 08:26 AM EST

Ryan Dahl steps down
Ryan Dahl, creator of Node.js, steps back from his position as gatekeeper for the project.
02/01/2012 04:58 PM EST

 
Events calendar tab
2/13/2012 to 2/16/2012
Santa Clara
TechWeb

2/26/2012 to 2/29/2012
San Francisco
BZ Media

2/27/2012 to 3/2/2012
San Francisco
RSA

3/4/2012 to 3/7/2012
Las Vegas
IBM Tivoli

3/5/2012 to 3/9/2012
San Francisco
TechWeb