This post originated from an RSS feed registered with Java Buzz by News Manager. Original Post: Spark 1.6 feeds big data's hunger for memory Feed Title: JavaWorld Feed URL: http://www.javaworld.com/index.rss Feed Description: JavaWorld.com: Fueling Innovation Latest Java Buzz Posts Latest Java Buzz Posts by News Manager Latest Posts From JavaWorld

Those curious about what's coming in Apache Spark 1.6 can now get hands-on experience with the latest changes to the big data processing framework.

Databricks, a major commercial contributor to Spark and provider of a cloud-hosted platform for running Spark applications, is offering Spark 1.6 in preview on its services. Those who want to try out Spark on their own can obtain Spark 1.6 pre-release code directly from Apache.

More headroom

This new version packs major changes to the way Spark handles memory. Earlier editions of Spark used to subdivide available memory into two partitions, one for data and one for execution, and required users to figure out how much memory to split between the two. This might have been one of the reasons for InfoWorld contributor Ian Pointer's complaints about Spark's memory management.

To read this article in full or to leave a comment, please click here