I am a spark application with several points where I would like to persist the current state. This is usually after a large step, or caching a state that I would like to use multiple times. It appears that when I call cache on my dataframe a second time, a new copy is cached to memory. In my application, this leads to memory issues when scaling up. Even though, a given dataframe is a maximum of about 100 MB in my current tests, the cumulative size of the intermediate results grows beyond the alloted memory on the executor. See below for a small example that shows this behavior.
cache_test.py:
from pyspark import SparkContext, HiveContext
spark_context = SparkContext(appName='cache_test')
hive_context = HiveContext(spark_context)
df = (hive_context.read
.format('com.databricks.spark.csv')
.load('simple_data.csv')
)
df.cache()
df.show()
df = df.withColumn('C1+C2', df['C1'] + df['C2'])
df.cache()
df.show()
spark_context.stop()
simple_data.csv:
1,2,3
4,5,6
7,8,9
Looking at the application UI, there is a copy of the original dataframe, in adition to the one with the new column. I can remove the original copy by calling df.unpersist() before the withColumn line. Is this the recommended way to remove cached intermediate result (i.e. call unpersist before every cache()).
Also, is it possible to purge all cached objects. In my application, there are natural breakpoints where I can simply purge all memory, and move on to the next file. I would like to do this without creating a new spark application for each input file.
Thank you in advance!
Spark 2.x
You can use Catalog.clearCache:
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate
...
spark.catalog.clearCache()
Spark 1.x
You can use SQLContext.clearCache method which
Removes all cached tables from the in-memory cache.
from pyspark.sql import SQLContext
from pyspark import SparkContext
sqlContext = SQLContext.getOrCreate(SparkContext.getOrCreate())
...
sqlContext.clearCache()
We use this quite often
for (id, rdd) in sc._jsc.getPersistentRDDs().items():
rdd.unpersist()
print("Unpersisted {} rdd".format(id))
where sc is a sparkContext variable.
When you use cache on dataframe it is one of the transformation and gets evaluated lazily when you perform any action on it like count(),show() etc.
In your case after doing first cache you are calling show() that is the reason the dataframe is cached in memory. Now then you are again performing transformation on the dataframe to add additional column and again caching the new dataframe and then calling the action command show again and this would cache the second dataframe in memory. In case if size of your dataframe is big enough to just hold one dataframe then when you cache the second dataframe it would remove the first dataframe from the memory as it does not have enough space to hold the second dataframe.
Thing to keep in mind: You should not cache a dataframe unless you are using it in multiple actions otherwise it would be an overload in terms of performance as caching itself is costlier operation.
Related
I'm trying to understand how to leverage cache() to improve my performance. Since cache retains a DataFrame in memory "for reuse", it seems like i need to understand the conditions that eject the DataFrame from memory to better understand how to leverage it.
After defining transformations, I call an action, is the dataframe, after the action completes, gone from memory? This would imply that if I do execute an action on a dataframe, but I continue to do other stuff with the data, all the previous parts of the DAG, from the read to the action, will need to be re done.
Is this accurate?
The fact is, that after an action is executed another dag is created. You can check this via SparkUI
In code you can try to identify where your dag is done and new started by looking for actions
When looking at code you can use this simple rule:
When function is transforming one df into another df - its not action but only lazy evaluated transformation. Even if this is join or something else which requires shuffling
When fuction is returning value other dan df, then you are using and action (for example count which is returning Long)
Lets take a look at this code (Its Scala but api is similar). I created this example just to show you the mechanism, this could be done better of course :)
import org.apache.spark.sql.functions.{col, lit, format_number}
import org.apache.spark.sql.DataFrame
val input = spark.read.format("csv")
.option("header", "true")
.load("dbfs:/FileStore/shared_uploads/***#gmail.com/city_temperature.csv")
val dataWithTempInCelcius: DataFrame = input
.withColumn("avg_temp_celcius",format_number((col("AvgTemperature").cast("integer") - lit(32)) / lit(1.8), 2).cast("Double"))
val dfAfterGrouping: DataFrame = dataWithTempInCelcius
.filter(col("City") === "Warsaw")
.groupBy(col("Year"), col("Month"))
.max("avg_temp_celcius")//Not an action, we are still doing transofrmation from df to another df
val maxTemp: Row = dfAfterGrouping
.orderBy(col("Year"))
.first() // <--- first is our first action, output from first is a Row and not df
dataWithTempInCelcius
.filter(col("City") === "Warsaw")
.count() // <--- count is our second action
Here you may see what is the problem. It looks like between first and second action i am doing transformation which was already done in first dag. This intermediate results of calculation was not cached, so in second dag Spark is unable to get the dataframe after filter from the memory which leads us to recomputation. Spark is going to fetch the data again, apply our filter and then calculate the count.
In SparkUI u will find two separate dags and both of them are going to read the source csv
If you cache intermediate results after first .filter(col("City") === "Warsaw") and then use this cached DF to do grouping and count you will still find two separate dags (number of action has not changed) but this time in the plan for second dag you will find "In memory table scan" instead of read of a csv file - that means that Spark is reading data from cache
Now you can see in memory relation in plan. There is still read csv node in the dag but as you can see, for second action its skipped (0 bytes read)
** I am using Databrics cluster with Spark 3.2, SparkUI may look different on your env
Quote...
Using cache() and persist() methods, Spark provides an optimization mechanism to store the intermediate computation of a Spark DataFrame so they can be reused in subsequent actions.
See https://sparkbyexamples.com/spark/spark-dataframe-cache-and-persist-explained/
I'm new to Koalas (pyspark), and I was trying to utilize Koalas for parallel apply, but it seemed like it was using a single core for the whole operation (correct me if I'm wrong) and ended up using dask for parallel apply (using map_partition) which worked pretty well.
However, I would like to know if there's a way to utilize Koalas for parallel apply.
I used basic codes for operation like below.
import pandas as pd
import databricks.koalas as ks
my_big_data = ks.read_parquet('my_big_file') # file is single partitioned parquet file
my_big_data['new_column'] = my_big_data['string_column'].apply(my_prep) # my_prep does stirng operations
my_big_data.to_parquet('my_big_file_modified') # for Koalas does lazy evaluation
I found a link that discuss this problem. https://github.com/databricks/koalas/issues/1280
If the number of rows that are being applied by function is less than 1,000 (default value), then pandas dataframe will be called to do the operation.
The user defined function above my_prep is applied to each row, so single core pandas was being used.
In order to force it to work in pyspark (parallel) manner, user should modify the configuration as below.
import databricks.koalas as ks
ks.set_option('compute.default_index_type','distributed') # when .head() call is too slow
ks.set_option('compute.shortcut_limit',1) # Koalas will apply pyspark
Also, explicitly specifying type (type hint) in the user defined function will make Koalas not to go shortcut path and will make parallel.
def my_prep(row) -> string:
return row
kdf['my_column'].apply(my_prep)
I have a dataFrame of approx. 4 million rows and 35 columns as input.
All I do to this dataFrame is the following steps:
For a list of given columns, I calculate a sum for a given list of group features and joined it as new column to my input dataFrame
I drop each new column sum right after I joined it to the dataFrame.
Therefore we end up with the same dataFrame as we started from (in theory).
However, I noticed that if my list of given columns gets too big (from more than 6 columns), the output dataFrame becomes impossible to manipulate. Even a simple display takes 10 minutes.
Here is an example of my code (df is my input dataFrame):
for c in list_columns:
df = df.join(df.groupby(list_group_features).agg(sum(c).alias('sum_' + c)), list_group_features)
df = df.drop('sum_' + c)
This happens due to the inner workings of Spark and its lazy evaluation.
What Spark does when you call groupby, join, agg, it attaches these calls to the plan of the df object. So even though it is not executing anything on the data, you are creating a large execution plan which is internally stored in the Spark DataFrame object.
Only when you call an action (show, count, write, etc.), Spark optimizes the plan and executes it. If the plan is too large, the optimization step can take a while to perform. Also remember that the plan optimization is happening on the driver, not on the executors. So if your driver is busy or overloaded, it delays spark plan optimization step as well.
It is useful to remember that joins are expensive operations in Spark, both for optimization and execution. If you can, you should always avoid joins when operating on a single DataFrame and utilise the window functionality instead. Joins should only be used if you are joining different dataframes from different sources (different tables).
A way to optimize your code would be:
import pyspark
import pyspark.sql.functions as f
w = pyspark.sql.Window.partitionBy(list_group_features)
agg_sum_exprs = [f.sum(f.col(c)).alias("sum_" + c).over(w) for c in list_columns]
res_df = df.select(df.columns + agg_sum_exprs)
This should be scalable and fast for large list_group_features and list_columns lists.
I have been attempting to perform some iterative computation on pyspark dataframes. Columns are added to the df based on previous columns. However I am noting that the memory used keeps increasing. A simple example is shown below.
from pyspark import SparkContext, SparkConf
from pyspark.sql import SQLContext
from pyspark import Row
conf = SparkConf().setAppName("myFirstApp").setMaster("local")
sc = SparkContext(conf=conf)
sqlContext = SQLContext(sc)
df = [Row(Z_0=0.0, Z_1=0.0)]
df = sc.parallelize(df).toDF()
for each in range(0,400):
df = df.withColumn("Z_"+str(each+2), df['Z_'+str(each+1)]+1)
It is my understanding that I am in fact building an execution plan, not necessarily the data itself. However calling the execution of the df with collect(), count(), show() or conversion to rdd or even deletion of the df fails to release memory. I have been seeing 1.2GB of memory for the above task. It seems like garbage collection has no way of cleaning up the previous intermediate df objects, or perhaps that these objects are never de-referenced.
is there a better method of building out this type of iterative calculation, or is there any way to cleanup these intermediate df's? Note the simple +1 occurring here is just a minimum example mock of much more complex calculations.
I have been dealing with the same thing and havent come up with a good solution.
As a Temporary Solution :
Split The application to many .py files and execute them one by one ,
which will cause garbage collection to free all unnecessary cache.
I have found that you can call take() in order to remove the execution plan, leaving simply values. See the last line for the relevant call.
from pyspark import SparkContext, SparkConf
from pyspark.sql import SQLContext
from pyspark import Row
conf = SparkConf().setAppName("myFirstApp").setMaster("local")
sc = SparkContext(conf=conf)
sqlContext = SQLContext(sc)
df = [Row(Z_0=0.0, Z_1=0.0)]
df = sc.parallelize(df).toDF()
for each in range(0,1400):
df = df.withColumn("Z_"+str(each+2), df['Z_'+str(each+1)]+1)
df = sc.parallelize(df.take(df.count())).toDF()
My statement in the question about garbage collection being the issue is not quite correct. There is a difference between the heap size and the used heap. In investigation with visualVM, it was easy to see that the garbage collection is occurring, which reduces the used heap.
We see that problem that the jvm has in processing the code posted in the question. Towards the end, we have no space to move. Our heap size is maxed, and the used heap is too big at this point, with nothing to GC. This expansion was not due to the data, but instead the data lineage information being retained. What I needed to do was get rid of all of that lineage, which to be honest isn't all that useful in this problems context, and retain just the data.
The following is the profile of the answer code snippet above. Even with 1400 columns, we have little problem holding the data.
I have troubles designing a working udf for my task on PySpark (python=2.7, pyspark=1.6)
I have a data DataFrame which looks like this :
+-----------------+
| sequence|
+-----------------+
| idea can|
| fuel turn|
| found today|
| a cell|
|administration in|
+-----------------+
And for each row in data I'd like to lookup info in an other DataFrame called ggrams (based on the attribute sequence), compute aggregates and return that as a new column in data.
My feeling is that I should do it that way :
from pyspark.sql.types import IntegerType
from pyspark.sql.functions import udf
def compute_aggregates(x):
res = ggrams.filter((ggrams.ngram.startswith(x)) \
).groupby("ngram").sum("match_count")
return res.collect()[0]['sum(match_count)']
aggregate = udf(compute_aggregates, IntegerType())
result = data.withColumn('aggregate', compute_aggregates('sequence'))
But that returns a PicklingError.
PicklingError: Could not serialize object: Py4JError: An error occurred while calling o1759.__getstate__. Trace:
py4j.Py4JException: Method __getstate__([]) does not exist
The error is thrown because you cannot access a different dataframe in the udf of another. The simplest way to fix this is by collecting the dataframe you want to check to.
The other option is a cross-join, but I can say from experience that collecting the other dataframe is faster. (I have no math/statistics to back this up though)
so:
1. Collect dataframe you want to use in the udf
2. Call to this collected dataframe (which is now a list) in your udf, you can/must now use python logic since you are talking to a list of objects
Note: Try to limit the dataframe that you are collecting to a minimum, select only the columns you need
Update:
If you are dealing with a very big set that would make collecting impossible, cross-join will most likely now work (it didn't for me at least). The problem is then that the huge cross product of the two dataframes will take so much time to create that the connection to the worker node will time-out, this causes an broadcast error. So see if there is any way that you can limit the columns that you are using, or if there is a possibility to filter out rows of which you can know for sure that they will not be used.
If all this fails, see if you can create some batch approach*, so run only the first X rows with collected data, if this is done, load the next X rows. This will most likely be terrible slow, but atleast it won't time-out (I think, I have not tried this personally since the collection was possible in my case)
*batch both, the dataframe you are running the udf on and the other dataframe, since you still cannot collect inside the udf, because you cannot acces the dataframe from there