We Keep Coding

Pyspark external table compression does not work

I am trying to save an external table from PySpark in parquet format and I need
to compress it. The PySpark version I am using is 2.4.7. I am updating the table after the initial
creation and appending data in a loop manner.
So far I have set the following options:
.config("spark.sql.parquet.compression.codec", "snappy") df.write.mode("append").format("parquet").option("compression","snappy").saveAsTable(...) df.write.mode("overwrite").format("parquet").option("compression","snappy").saveAsTable(...)
Is there anything else that I need to set or am I doing something wrong?
Thank you

How to Use DataFrame Created in Scala in Databricks' PySpark

My Databricks notebook is on Python.
Some codes in the notebook are written in Scala (using the %scala) and one of them is for creating dataframe.
If I use Python/PySpark (the default mode) again, how can I use / access this dataframe that was created when it was on scala mode?
Is it even possible?
Thanks

You can access DataFrames created in one language with another language through temp tables in SparkSQL.
For instance, say you have a DataFarame in scala called scalaDF. You can create a temporary view of that and make it accessible to a Python cell, for instance:
scalaDF.createOrReplaceTempView("my_table")
Then in a Python cell you can run
pythonDF = spark.sql("select * from my_table")
pythonDF.show()
The same works for passing dataframes between those languages and R. The common construct is a SparkSQL table.

Using Table with Python

I need to incorporate a table into my python program so when I run my program it can read the information from the table as it is needed. What is the best way to incorporate the table into my program so that it is familiar with it when the program is ran?
This is the table that I will incorporate into my program:

You might use Excel or Google Sheets.
Paste the table into your spreadsheet
Export the sheet to CSV
Use Python's csv library to import the data
https://docs.python.org/3/library/csv.html

How can i use Hive on Cassandra with Python coding

I am using Python and my data is stored on Cassandra.But I want to write query with Hive(maybe it will faster than). But I can not found any API or library.Have you any suggestion for make this combination?

How to write a Python UDF for User Defined Aggregate Function in Hive

I would like to do some aggregation work on an aggregate column (after GROUP BY) in Hive using Python. I found there is UDAF for this purpose. All I can find is a Java example. Is there an example on writing in Python?
Or for python between UDF and UDAF, there is no difference? For UDAF, I just need to write it like a reducer? Please advise.

You can make use of Hive's streaming UDF functionality (TRANSFORM) to use a Python UDF which reads from stdin and outputs to stdout. You haven't found any Python "UDAF" examples because UDAF refers to the Hive Java class you extend so it would only be in Java.
When using a streaming UDF, Hive will choose whether to launch or a map or reduce job so there is no need to specify (for more on this functionality see this link: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Transform).
Basically, your implementation would be to write a python script which reads from stdin, calculates some aggregate number and outputs it to stdout. To implement in Hive do the following:
1) First add your python script to your resource library in Hive so that it gets distributed across your cluster:
add file script.py;
2) Then call your transform function and input the columns you want to aggregate. Here is an example:
select transform(input cols)
using 'python script.py' as (output cols)
from table
;
Depending on what you need to do, you may need a separate mapper and reducer script. If you need to aggregate based on column value, remember to use Hive's CLUSTER BY/DISTRIBUTE BY syntax in your mapper stage so that partitioned data gets sent to the reducer.
Let me know if this helps.