We Keep Coding

How to access dataframe returned by UDF inside column in Spark Streaming

I'm writing code to perform step detection on streaming sensor data. To do so I separate all incoming values as sensor parameters. After that I detect the steps with a UDF. Inside this UDF I create a dict to store two lists of important timestamps per determined step.
#udf(StructType())
def detect_steps(timestamp, x, y, z):
.....
d = dict()
d['timestamp_ic'] = times_ic
d['timestamp_to'] = timestamp_to
......
return d
In the Spark main function I created a dataframe that calculates all these steps in a sliding window like so:
stepData = LLLData \
.withWatermark("time", "10 seconds") \
.groupBy(
window("time", windowDuration="5 seconds",slideDuration="1 second"),
"sensor"
) \
.agg(
collect_list("time").alias("time_window"),
collect_list(sensorData.Acceleration_x).alias("Acceleration_x_window"),
collect_list(sensorData.Acceleration_y).alias("Acceleration_y_window"),
collect_list(sensorData.Acceleration_z).alias("Acceleration_z_window"),
) \
.select(
"window",
"sensor",
detect_steps("time_window", "Acceleration_x_window", "Acceleration_y_window", "Acceleration_z_window")
)
Now, when I print the df schema it looks like this:
|-- window: struct (nullable = true)
| |-- start: timestamp (nullable = true)
| |-- end: timestamp (nullable = true)
|-- sensor: string (nullable = true)
|-- detect_steps("time_window", "Acceleration_x_window", "Acceleration_y_window", "Acceleration_z_window"): string (nullable = true)
While I want this:
|-- window: struct (nullable = true)
| |-- start: timestamp (nullable = true)
| |-- end: timestamp (nullable = true)
|-- sensor: string (nullable = true)
|-- timestamp_ic: string (nullable = true)
|-- timestamp_to: string (nullable = true)
However I cannot perform a select statement on the UDF column in stepData, error: Column is not iterable.
When I try to alter the root dataframe afterwards, by for example parsing the ic column to a spark dataframe like so:
df_stepData = spark.createDataFrame(data=stepData.select("ic"))
it gives me TypeError: data is already a DataFrame.
Looking at the dataframe schema, however, ic is typed as string.
I've also tried to read ic as a json file, but that gives the following error:
TypeError: path can be only string, list or RDD
I could fix the problem by triggering the detect_steps UDF twice, the first one returning timestamp_ic and the second one returning timestamp_to, in order to get two columns but I'm sure there is a better, more efficient way.

Make VectorAssembler always choose DenseVector

This is the structure of my dataframe using df.columns.
['LastName',
'FirstName',
'Stud. ID',
'10 Relations',
'Related to Politics',
'3NF',
'Documentation & Scripts',
'SQL',
'Data (CSV, etc.)',
'20 Relations',
'Google News',
'Cheated',
'Sum',
'Delay Factor',
'Grade (out of 2)']
I have transformed this dataframe in pyspark using
assembler = VectorAssembler(inputCols=['10 Relations',
'Related to Politics',
'3NF'],outputCol='features')
and output = assembler.transform(df). Now it contains some Row objects. These objects have this architecture (This is what I get when I run output.printSchema())
root
|-- LastName: string (nullable = true)
|-- FirstName: string (nullable = true)
|-- Stud. ID: integer (nullable = true)
|-- 10 Relations: integer (nullable = true)
|-- Related to Politics: integer (nullable = true)
|-- 3NF: integer (nullable = true)
|-- Documentation & Scripts: integer (nullable = true)
|-- SQL: integer (nullable = true)
|-- Data (CSV, etc.): integer (nullable = true)
|-- 20 Relations: integer (nullable = true)
|-- Google News: integer (nullable = true)
|-- Cheated: integer (nullable = true)
|-- Sum: integer (nullable = true)
|-- Delay Factor: double (nullable = true)
|-- Grade (out of 2): double (nullable = true)
|-- features: vector (nullable = true)
For each row, the assembler chooses to make the features vector Sparse or Dense (For memory reasons). But this is a big problem. Because I want to use this transformed data for making a linear regression model. So, I'm searching for a way to make VectorAssembler always choose Dense Vector.
Any idea?
Note: I have read this post. But the problem is that since the Row class is a subclass of tuple, I cannot change a Row object after it is made.

Sparse and Dense vector are both inherited from pyspark.ml.linalg.Vector. So both vector types have .toarray() method in common. You can convert them into numpy array then Dense vetor with simple udf.
from pyspark.ml.linalg import DenseVector, SparseVector, Vectors, VectorUDT
from pyspark.sql import functions as F
from pyspark.sql.types import *
v = Vectors.dense([1,3]) # dense vector
u = SparseVector(2, {}) # sparse vector
# toDense function converts both vector type into Dense Vector
toDense = lambda v: Vectors.dense(v.toArray())
toDense(u), toDense(v)
Results:
DenseVector([0.0, 0.0]), DenseVector([1.0, 3.0])
Then You can create udf with this function.
df = sqlContext.createDataFrame([
((v,)),
((u,))
], ['feature'])
toDense = lambda v: Vectors.dense(v.toArray())
toDenseUdf = F.udf(toDense, VectorUDT())
df.withColumn('feature', toDenseUdf('feature')).show()
results:
+---------+
| feature|
+---------+
|[1.0,3.0]|
|[0.0,0.0]|
+---------+
You have single vectortype in column.

How to convert an array to string efficiently in PySpark / Python

I have a df with the following schema:
root
|-- col1: string (nullable = true)
|-- col2: array (nullable = true)
| |-- element: string (containsNull = true)
in which one of the columns, col2 is an array [1#b, 2#b, 3#c]. I want to convert this to the string format 1#b,2#b,3#c.
I am currently doing this through the following snippet
df2 = (df1.select("*", explode(col2)).drop('col2'))
df2.groupBy("col1").agg(concat_ws(",", collect_list('col')).alias("col2"))
While this gets the job done, it is taking time and also seems inefficient.
Is there a better alternative?

You can call concat_ws directly on a column, like this:
df1.withColumn('col2', concat_ws(',', 'col2'))

How to convert column with string type to int form in pyspark data frame?

I have dataframe in pyspark. Some of its numerical columns contain nan so when I am reading the data and checking for the schema of dataframe, those columns will have string type.
How I can change them to int type. I replaced the nan values with 0 and again checked the schema, but then also it's showing the string type for those columns.I am following the below code:
data_df = sqlContext.read.format("csv").load('data.csv',header=True, inferSchema="true")
data_df.printSchema()
data_df = data_df.fillna(0)
data_df.printSchema()
my data looks like this:
here columns Plays and drafts containing integer values but because of nan present in these columns, they are treated as string type.

from pyspark.sql.types import IntegerType
data_df = data_df.withColumn("Plays", data_df["Plays"].cast(IntegerType()))
data_df = data_df.withColumn("drafts", data_df["drafts"].cast(IntegerType()))
You can run loop for each column but this is the simplest way to convert string column into integer.

You could use cast(as int) after replacing NaN with 0,
data_df = df.withColumn("Plays", df.call_time.cast('float'))

Another way to do it is using the StructField if you have multiple fields that needs to be modified.
Ex:
from pyspark.sql.types import StructField,IntegerType, StructType,StringType
newDF=[StructField('CLICK_FLG',IntegerType(),True),
StructField('OPEN_FLG',IntegerType(),True),
StructField('I1_GNDR_CODE',StringType(),True),
StructField('TRW_INCOME_CD_V4',StringType(),True),
StructField('ASIAN_CD',IntegerType(),True),
StructField('I1_INDIV_HHLD_STATUS_CODE',IntegerType(),True)
]
finalStruct=StructType(fields=newDF)
df=spark.read.csv('ctor.csv',schema=finalStruct)
Output:
Before
root
|-- CLICK_FLG: string (nullable = true)
|-- OPEN_FLG: string (nullable = true)
|-- I1_GNDR_CODE: string (nullable = true)
|-- TRW_INCOME_CD_V4: string (nullable = true)
|-- ASIAN_CD: integer (nullable = true)
|-- I1_INDIV_HHLD_STATUS_CODE: string (nullable = true)
After:
root
|-- CLICK_FLG: integer (nullable = true)
|-- OPEN_FLG: integer (nullable = true)
|-- I1_GNDR_CODE: string (nullable = true)
|-- TRW_INCOME_CD_V4: string (nullable = true)
|-- ASIAN_CD: integer (nullable = true)
|-- I1_INDIV_HHLD_STATUS_CODE: integer (nullable = true)
This is slightly a long procedure to cast , but the advantage is that all the required fields can be done.
It is to be noted that if only the required fields are assigned the data type, then the resultant dataframe will contain only those fields which are changed.

Compare column names in two data frames pyspark

I have two data frames in pyspark df and data. The schema are like below
>>> df.printSchema()
root
|-- id: integer (nullable = false)
|-- name: string (nullable = true)
|-- address: string (nullable = true)
|-- nation: string (nullable = true)
|-- Date: timestamp (nullable = false)
|-- ZipCode: integer (nullable = true)
|-- car: string (nullable = true)
|-- van: string (nullable = true)
>>> data.printSchema()
root
|-- id: integer (nullable = true)
|-- name: string (nullable = true)
|-- address: string (nullable = true)
|-- nation: string (nullable = true)
|-- date: string (nullable = true)
|-- zipcode: integer (nullable = true)
Now I want to add columns car and van to my data data frame by comparing both the schema.
I would also want to compares two data frames if the columns are same do nothing, but if the columns are different then add the columns to the data frame that doesn't have the columns.
How can we achieve that in pyspark.
FYI I am using spark 1.6
once the columns are added to the data frame. The values for those columns in the newly added data frame Should be null.
for example here we are adding columns to data data frame so the columns car and van in data data frame should contain null values but the same columns in df data frame should have their original values
what happens if there are more than 2 new columns to be added

As the schema is not but StructType consisting of list of StructFields, we can retrieve the fields list, to compare and find the missing columns,
df_schema = df.schema.fields
data_schema = data.schema.fields
df_names = [x.name.lower() for x in df_scehma]
data_names = [x.name.lower() for x in data_schema]
if df_schema <> data_schema:
col_diff = set(df_names) ^ set(data_names)
col_list = [(x[0].name,x[0].dataType) for x in map(None,df_schema,data_schema) if ((x[0] is not None and x[0].name.lower() in col_diff) or x[1].name.lower() in col_diff)]
for i in col_list:
if i[0] in df_names:
data = data.withColumn("%s"%i[0],lit(None).cast(i[1]))
else:
df = df.withColumn("%s"%i[0],lit(None).cast(i[1]))
else:
print "Nothing to do"
You have mentioned to add the column if there is no null values, but your schema diference are nullable columns, so have not used that check. If you need it, then add check for nullable as below,
col_list = [(x[0].name,x[0].dataType) for x in map(None,df_schema,data_schema) if (x[0].name.lower() in col_diff or x[1].name.lower() in col_diff) and not x.nullable]
Please check the documentation for more about StructType and StructFields,
https://spark.apache.org/docs/1.6.2/api/python/pyspark.sql.html#pyspark.sql.types.StructType

If you have to do this to multiple tables, it might be worth it to generalize the code a bit. This code takes the first non-null value in the non-matching source column to create the new column in the target table.
from pyspark.sql.functions import lit, first
def first_non_null(f,t): # find the first non-null value of a column
return f.select(first(f[t], ignorenulls=True)).first()[0]
def match_type(f1,f2,miss): # add missing column to the target table
for i in miss:
try:
f1 = f1.withColumn(i, lit(first_non_null(f2,i)))
except:
pass
try:
f2 = f2.withColumn(i, lit(first_non_null(f1,i)))
except:
pass
return f1, f2
def column_sync_up(d1,d2): # test if the matching requirement is met
missing = list(set(d1.columns) ^ set(d2.columns))
if len(missing)>0:
return match_type(d1,d2,missing)
else:
print "Columns Match!"
df1, df2 = column_sync_up(df1,df2) # reuse as necessary