Talend adds enterprise features to data integration suite
May 19, 2009 —
Organizations can’t help but accumulate mountains of data. Yet that data only has value if you can find the value it contains. To do that requires heavy-duty data processing and crunching.
Open-source data integration company Talend has released Integration Suite MPX, which extends its commercial offering to address this need to process extremely large amounts of data. The solution, released today, now includes what the company is calling FileScale technology, and the ability to do massively parallelized processing.
The FileScale technology is based on the MapReduce technology created by Google, according to Yves de Montcheuil, Talend’s vice president of marketing. It is what lets Google process billions of Web pages for indexing and ranking purposes.
“The technology lets us break down complex problems into smaller problems. Then we can get multiple nodes process those problems,” and it all gets built back up automatically by a main node, de Montcheuil explained. FileScale has low-level components that are optimized for such tasks as sorting, filtering and merging, aggregating, and transforming. FileScale will distribute these tasks to where resources are available on the hardware, ensuring faster processing time, he said.
When the problems are broken down into smaller subsets, they can be processed in parallel, simultaneously, and then automatically synched back up after the processing is complete, de Montcheuil said. He cited a benchmark using industry-standard TPC-H data sets that was run on a Sun Blade X6270 server featuring two Intel Xeon 5520 quad-core processors—“admittedly a high-end system but a real one, not something cobbled together just for a benchmark,” he said.
In-memory data sorting reached one million records per second, while 3.3 billion records were sorted through at a speed of 200,000 to 400,000 per second. “That is a level of scalability that hasn’t been reached before,” de Montcheuil said.
Related Search Term(s): Talend
Share this link: http://sdt.bz/33492
Most Read Latest News Blog Resources
Zeichick’s Take: Radio moves from analog waveforms to digital packets
Streaming radio highlights the need for streaming applications to be designed to take up as little bandwidth as possible
|
|
Taking enterprise architecture to the business side
Startup Corso is bringing out a cloud-based planning platform that ties into business plans
|
|
Appcelerator Acquires Cocoafish to Add Instant Mobile Cloud Capabilities to its Industry Leading Titanium Platform
Appcelerator Offers Messaging, Social, Location and Storage Mobile Cloud Services to All Mobile App Publishers
|
|
ComponentOne Releases a Collection of 40+ UI Widgets Powered by HTML5 and jQuery
ComponentOne has announced the 2012 release of Wijmo: a kit of UI widgets for HTML5 and jQuery development
|
Taking enterprise architecture to the business side
Startup Corso is bringing out a cloud-based planning platform that ties into business plans
|
|
Top five apps to manage your workload
Web applications offer new ways to track your “to-do” lists
|
|
Not so fast when it comes to testing in the cloud
Developers face outsourcing, virtual lab management and mobile devices as obstacles
|
|
Xceed releases UX-focused suite for Microsoft’s WPF
"Blendables" helps match user experiences to developer visions
|
Are you at risk for burnout?
Burnout is a severe problem and it can strike at any time. Here's how to tell if you are nearing the edge.
|
|
Agility, mom, and apple pie
If we're to evaluate the state-of-the-art in software development, we should start with the values espoused in the Agile Manifesto.
|
|
RIM woos developers with free tablet
How do you get more apps ported to the BlackBerry PlayBook? By giving every developer a free tablet, of course!
|
|
GitHire: Use Headhunters to Find Your Perfect Programmer
Are you a hiring manager tired of scouring the job boards? Check out this new service that will find 5 people interested in your jobs.
|
The Hidden Costs of Software Licensing
Moving beyond paper-based software licensing to more flexible, software-based licensing is a business decision. There is a growing trend tow...
|
|
Case Study: You May Need a Development Mechanic
As a contractor for a major financial player in Germany, SOBEGE, a German-based consultancy specializing in embedded IT and web services, wa...
|
|
Ensuring Software Quality at a Major International Bank
One of the world’s leading international banks has adopted AgitarOne technology for delivering generated unit tests for their Java software...
|
|
Load Testing Adobe Flex Applications
Adobe Flex applications may be different from applications you’ve worked with before. For classic HTML web applications, the server does all...
|