My Learning Space

A real-world Spark war story: how a simple groupBy(“city”) created a 1.2 GB skewed task, and how salting plus repartitioning took the job from ~25 minutes to ~6.

For any data operations in Apache spark, one of the first tasks is creating a DataFrame. This is done using the createDataFrame() API. There are two main things to consider: Data Source – Data can come from a list of dictionaries, a list of lists or tuples, or even...

PySpark in Apache Spark uses the DataFrameWriter to manage how data is saved. To write data in PySpark, you start with the .write attribute of a DataFrame, which gives you a DataFrameWriter to manage the save process. Basic Approach to...

Embedded: PySpark makes it easy to load data from different sources into DataFrames. At first, the process can seem a little overwhelming, but this guide is designed as a visual walkthrough to simplify the concepts. The data...

When a Spark application is submitted, it does not execute statements sequentially. Instead, Spark constructs a logical execution plan, represented as a Directed Acyclic Graph (DAG), which captures the computation flow and dependencies before...