LOGIN | REGISTER NOW | SUBSCRIBE
 
print Printable version 
Most Read Latest News Blog Resources
Does Windows cost Microsoft opportunities?
Microsoft's decision to focus .NET on Windows is stifling innovation due to the specter of...

Mono brings .NET to Android
At Microsoft MIX, Mono will unveil the ability to port .NET applications to the Android op...

OSBC focus turns to best practices for open-source adoption
Businesses are no longer asking why they should adopt, but how do they use open source leg...

Digg!  Digg
Reddit  Reddit
Slashdot  Slashdot
Share on facebook  Facebook
Share on Twitter  Twitter

Talend adds enterprise features to data integration suite


By David Rubinstein
May 19, 2009 — 
Organizations can’t help but accumulate mountains of data. Yet that data only has value if you can find the value it contains. To do that requires heavy-duty data processing and crunching.

Open-source data integration company Talend has released Integration Suite MPX, which extends its commercial offering to address this need to process extremely large amounts of data. The solution, released today, now includes what the company is calling FileScale technology, and the ability to do massively parallelized processing.

The FileScale technology is based on the MapReduce technology created by Google, according to Yves de Montcheuil, Talend’s vice president of marketing. It is what lets Google process billions of Web pages for indexing and ranking purposes.

“The technology lets us break down complex problems into smaller problems. Then we can get multiple nodes process those problems,” and it all gets built back up automatically by a main node, de Montcheuil explained. FileScale has low-level components that are optimized for such tasks as sorting, filtering and merging, aggregating, and transforming. FileScale will distribute these tasks to where resources are available on the hardware, ensuring faster processing time, he said.

When the problems are broken down into smaller subsets, they can be processed in parallel, simultaneously, and then automatically synched back up after the processing is complete, de Montcheuil said. He cited a benchmark using industry-standard TPC-H data sets that was run on a Sun Blade X6270 server featuring two Intel Xeon 5520 quad-core processors—“admittedly a high-end system but a real one, not something cobbled together just for a benchmark,” he said.

In-memory data sorting reached one million records per second, while 3.3 billion records were sorted through at a speed of 200,000 to 400,000 per second. “That is a level of scalability that hasn’t been reached before,” de Montcheuil said.


Related Search Term(s): Talend


Share this link: http://www.sdtimes.com/link/33492
 

Add comment


Name*
Email*  
Country     


  • Comment
  • Preview
Loading



 
 
 
 
News on Monday
more>>
SharePoint Tech Report
more>>


   

 
 
Download Current Issue
ISSUE 3/15/2010 PDF

Need Back Issues?
DOWNLOAD HERE

Receive the print Edition?


 
blogs tab
Google Code turns 5
Google Code Turns 5, and adds a Paxos Algorithm to make the system more stable and reliable.
03/17/2010 11:16 AM EST

Test your Visual Studio 2010 know-how
Microsoft is offering free beta certification exams for Visual Studio 2010.
03/17/2010 11:08 AM EST

Microsoft lifts the hood on IE9
Microsoft is previewing IE9.
03/16/2010 01:10 PM EST

 

Events calendar tab
EclipseCon
3/22/2010 to 3/25/2010
Santa Clara, Calif.
The Eclipse Foundation

DevConnections
4/12/2010 to 4/14/2010
Las Vegas
Penton Media

MySQL Conf. and Expo
4/12/2010 to 4/15/2010
Santa Clara, Calif.
O'Reilly Media

High Performance Computing Linux Financial Markets
4/19/2010
New York City
Flagg Management

Informix User Conf.
4/25/2010 to 4/28/2010
Overland Park, Kans.
IIUG