/explore

Click through on any links that interest you or select the planets on the right to continue exploring the Outer Web.
You are here

 aneesh.mataroa.blog
The journey from Supercomputing to Spark - tech ramblings
| | timilearning.com
MIT 6.824: Lecture 15 - Spark
1.9 parsecs away

Travel
| | In the first lecture of this series, I wrote about MapReduce as a distributed computation framework. MapReduce partitions the input data across worker nodes, which process data in two stages: map and reduce. While MapReduce was innovative, it was inefficient for iterative and more complex computations. Researchers at UC Berkeley invented Spark to deal with these limitations.
| | www.altexsoft.com
Apache Hadoop vs Spark: Main Big Data Tools Explained
1.9 parsecs away

Travel
| | The article explains how the main Big Data tools, Hadoop and Spark, work, what benefits and limitations they have, and which one to choose for your project.
| | technicaldiscovery.blogspot.com
Technical Discovery: Anaconda and Hadoop --- a story of the journey and where we are now.
3.1 parsecs away

Travel
| | Early Experience with Clusters My first real experience with cluster computing came in 1999 during my graduate school days at the Mayo Cl...
| | tech.scribd.com
Kafka to Delta Lake, as fast as possible | Scribd Technology
6.8 parsecs away

Travel
| Streaming data from Apache Kafka into Delta Lake is an integral part of Scribd's data platform, but has been challenging to manage and scale. We use Spark Structured Streaming jobs to read data from Kafka topics and write that data into Delta Lake tables. This approach gets the job done but in production our experience has convinced us that a different approach is necessary to efficiently bring data from Kafka to Delta Lake. To serve this need, we created kafka-delta-ingest.