We Keep Coding

Snowpark Snowflake Python to run a sql statement and export to Excel

I'm creating a Snowflake procedure using Snowpark (python) package executing a query into a snowflake dataframe and I would like to export that into Excel, how can I accomplish that? Is it a better approach to do this? The end goal is to export it the query results into Excel. Needs to be in a Snowflake procedure since we already have others "parent" procedures. Thanks!
CREATE OR REPLACE PROCEDURE EXPORT_SP()
RETURNS string not null
LANGUAGE PYTHON
RUNTIME_VERSION = '3.8'
PACKAGES = ('snowflake-snowpark-python', 'pandas')
HANDLER = 'run'
AS
$$
import pandas
def run(snowpark_session):
## Execute the query into a Snowflake dataframe
results_df = snowpark_session.sql('''
SELECT * FROM
MY TABLES
;
''').collect()
return results_df
$$
;

In general, you can do this by:
"Unloading" the data from the table using the COPY INTO <location> command.
Using the GET command to copy the data to your local filesystem.
Open the file with Excel! If you used the CSV format and the appropriate format options in step 1, you should be able to easily open the resulting data with Excel.
Snowpark directly supports step 1 in the DataFrameWriter.copy_into_location method. An instance of DataFrameWriter contained in the DataFrame.write attribute, as described here.
Snowpark also directly supports step 2 in the FileOperation.get method. As per the example in that documentation page, you can access this method using the .file attribute of your Snowpark session object.
Putting this all together, you should be able to do something like this in Snowpark to save a single exported file into the current working directory:
source_table = "my_table"
unload_location = "#my_stage/export.csv"
def run(session):
df = session.table(source_table)
df.write.copy_into_location(
unload_location,
file_format_type="csv",
format_type_options=dict(
compression="none",
field_delimiter="\t",
),
single=True,
header=True,
)
session.file.get(unload_location, ".")
You can of course use session.sql() instead of session.table() as needed. You might also want to consider unloading data to the stage associated with the source data, instead of creating a separate stage, i.e. if the data is from table my_table then you would unload to the stage #%my_table.
For more details, refer to the documentation pages I linked, which contain important reference information as well as several examples.
Note that I am not sure if session.file is accessible from inside a stored procedure; you will have to experiment to see what works in your specific situation.
As always, remember that this is untested code written by an unpaid volunteer. Always triple-check and test any code that is provided here. Please do ask questions in the comments if anything is still unclear.

Declaring path to invoke a template file python27

I have tried using various declarations for trying to invoke the template file but for some reason the script is failing to pick the template from the location specified and loading the reporting contents to the template and then exporting it to a pdf format.
The code i have attached hereby :
`#Build the html report using the html template and save to the set location
output_from_parsed_template = buildTemplate()
with open(r"C:\python_report_scripts\anram_report.html","wb") as fh:
fh.write(output_from_parsed_template)
#Convert html to pdf
subprocess.call('C:\omniformat\html2pdf995.exe r"C:\python_report_scripts\anram_report.html" r"C:\ANRAM_Requests\Working_folder\\'+sectionName+'.pdf"')
#Run a summary report when there is at least 1 section in the section list
if len(sectionInfo)>1:
#Flag that this is a summary report
summary = 1
#Generate the Existing Map image
MapExisting(summary)
#Generate the Existing Risk Graph image
RiskgraphExisting(summary)
#Generate the New Map Image
MapNew(summary)
#Generate the New Risk Graph image
RiskgraphNew(summary)
#Generate the tabular results for the report
populateResults(summary)
#Indicate that the report is a summary, which then forms the report title
#global sectionName
sectionName = 'SUMMARY - '+sectionName
#Build the html report using the html template and save to the set location
output_from_parsed_template = buildTemplate()
with open(r"C:\python_report_scripts\anram_report.html","wb") as fh:
fh.write(output_from_parsed_template)
#Convert html to pdf
subprocess.call('C:\omniformat\html2pdf995.exe r"C:\python_report_scripts\anram_report.html" r"C:\ANRAM_Requests\Working_folder\\'+sectionName+'.pdf"')`
So am i declaring it correctly ?
Please advise,

This line looks weird:
#Convert html to pdf
subprocess.call('C:\omniformat\html2pdf995.exe r"C:\python_report_scripts\anram_report.html" r"C:\ANRAM_Requests\Working_folder\\'+sectionName+'.pdf"')`
You seem to be mixing up Python quoting, Python raw strings, with shell quoting and you still have some double backslashes around...
I suggest:
Use subprocess.check_call instead, that way Python will report if there are any errors running the command.
Pass it a list, instead of a string, that way it's more clear which argument is which and you don't depend so much on shell quoting and word splitting.
Use raw strings consistently (r'...' or r"...", either one is fine.)
Use os.path.join to join paths!
So, putting it all together, try this instead:
# Convert html to pdf
subprocess.check_call([
r'C:\omniformat\html2pdf995.exe',
r'C:\python_report_scripts\anram_report.html',
os.path.join(r'C:\ANRAM_Requests\Working_folder', sectionName + '.pdf')
])
I hope this solves the issue you're seeing... Or, if it doesn't, at least gives you a more meaningful error message that you can act on.

Python, Alembic.io, Cask: Properties of object do not save when using write_to_file()

I am often writing scripts for various 3d packages (3ds max, Maya, etc) and that is why I am interested with Alembic, a file format that is getting a lot of attention lately.
Quick explanation for anyone who does not know this project: alembic - www.alembic.io - is a file format created for containing 3d meshes and data connected with them. It is using a tree-like structure, as You may see below, with one root node and its childs, childs of childs etc. Objects of this node can have properties.
I am trying to learn how to use this Alembic with Python.
There are some tutorials on docks page of this project and I'm having some problems with this one:
http://docs.alembic.io/python/cask.html
It's about using cask module - a wrapper that should manipulating a content of files easier.
This part:
a = cask.Archive("animatedcube.abc")
r = cask.Xform()
x = a.top.children["cube1"]
a.top.children["root"] = r
r.children["cube1"] = x
a.write_to_file("/var/tmp/cask_insert_node.abc")
works well. Afther that there's new file "cask_insert_node.abc" and it has objects as expected.
But when I'm adding some properties to objects, like this:
a = cask.Archive("animatedcube.abc")
r = cask.Xform()
x = a.top.children["cube1"]
x.properties['new_property'] = cask.Property()
a.top.children["root"] = r
r.children["cube1"] = x
a.write_to_file("/var/tmp/cask_insert_node.abc")
the "cube1" object in a resulting file do not contain property "new_property".
The saving process is a problem, i know that the property has been added to "cube1" before saving, I've checked it another way, with a function that I wrote which creates graph of objects in archive.
The code for this module is there:
source
Does anyone know what I am doing wrong? How to save parameters? Some other way?

Sadly, cask doesn't support this. One cannot modify an archive and have the result saved (somehow related to how Alembic streams the data off of disk). What you'll want to do is create an output archive
oArchive = alembic.Abc.CreateArchiveWithInfo(...)
then copy all desired data from your input archive over to your output archive including time sampling (
.addTimeSampling(iArchive.getTimeSampling(i) for i in iArchive.getNumTimeSamplings()
, and objects, recursing through iArchive.getTop() and oArchive.getTop() defining output properties (alembic.Abc.OArrayProperty, or OScalarProperty) as you encounter them in the iArchive. When these are defined, you can interject your new values as samples to the property at that time.
It's a real beast, and something that cask really ought to support. In fact, someone in the Alembic community should just do everyone a favor and write a cask2 (casket?) which wraps all of this into simple calls like you instinctively tried to do.

Python - how to read Windows "Media Created" date (not file creation date)

I have several old video files that I'm converting to save space. Since these files are personal videos, I want the new files to have the old files' creation time.
Windows has an attribute called "Media created" which has the actual time recorded by the camera. The files' modification times are often incorrect so there are hundreds of files where this won't work.
How can I access this "Media created" date in Python? I've been googling like crazy and can't find it. Here's a sample of the code that works if the creation date and modify date match:
files = []
for file in glob.glob("*.AVI"):
files.append(file)
for orig in files:
origmtime = os.path.getmtime(orig)
origatime = os.path.getatime(orig)
mark = (origatime, origmtime)
for target in glob.glob("*.mp4"):
firstroot = target.split(".mp4")[0]
if firstroot in orig:
os.utime(target, mark)

As Borealid noted, the "Media created" value is not filesystem metadata. The Windows shell gets this value as metadata from within the file itself. It's accessible in the API as a Windows Property. You can easily access Windows shell properties if you're using Windows Vista or later and have the Python extensions for Windows installed. Just call SHGetPropertyStoreFromParsingName, which you'll find in the propsys module. It returns a PyIPropertyStore instance. The property that's labelled "Media created" is System.Media.DateEncoded. You can access this property using the property key PKEY_Media_DateEncoded, which you'll find in propsys.pscon. In Python 3 the returned value is a datetime.datetime subclass, with the time in UTC. In Python 2 the value is a custom time type that has a Format method that provides strftime style formatting. If you need to convert the value to local time, the pytz module has the IANA database of time zones.
For example:
import pytz
import datetime
from win32com.propsys import propsys, pscon
properties = propsys.SHGetPropertyStoreFromParsingName(filepath)
dt = properties.GetValue(pscon.PKEY_Media_DateEncoded).GetValue()
if not isinstance(dt, datetime.datetime):
# In Python 2, PyWin32 returns a custom time type instead of
# using a datetime subclass. It has a Format method for strftime
# style formatting, but let's just convert it to datetime:
dt = datetime.datetime.fromtimestamp(int(dt))
dt = dt.replace(tzinfo=pytz.timezone('UTC'))
dt_tokyo = dt.astimezone(pytz.timezone('Asia/Tokyo'))

If the attribute you're talking about came from the camera, it's not a filesystem permission: it's metadata inside the videos themselves which Windows is reading out and presenting to you.
An example of this type of metadata would be a JPEG image's EXIF data: what type of camera took the photo, what settings were used, and so forth.
You would need to open up the .mp4 files and parse the metadata, preferably using some existing library for doing that. You wouldn't be able to get the information from the filesystem because it's not there.
Now if, on the other hand, all you want is the file creation date (which didn't actually come from the camera, but was set when the file was first put onto the current computer, and might have been initialized to some value that was previously on the camera)... That can be gotten with os.path.getctime(orig).

Is there a simple way to write an ODT using Python?

My point is that using either pod (from appy framework, which is a pain to use for me) or the OpenOffice UNO bridge that seems soon to be deprecated, and that requires OOo.org to run while launching my script is not satisfactory at all.
Can anyone point me to a neat way to produce a simple yet clean ODT (tables are my priority) without having to code it myself all over again ?
edit: I'm giving a try to ODFpy that seems to do what I need, more on that later.

Your mileage with odfpy may vary. I didn't like it - I ended up using a template ODT, created in OpenOffice, oppening the contents.xml with ziplib and elementtree, and updating that. (In your case, it would create only the relevant table rows and table cell nodes), then recorded everything back.
It is actually straightforward, but for making ElementTree properly work with the XML namespaces. (it is badly documente) But it can be done. I don't have the example, sorry.

To edit odt files, my answer may not help, but if you want to create new odt files, you can use QTextDocument, QTextCursor and QTextDocumentWriter in PyQt4. A simple example to show how to write to an odt file:
>>>from pyqt4 import QtGui
# Create a document object
>>>doc = QtGui.QTextDocument()
# Create a cursor pointing to the beginning of the document
>>>cursor = QtGui.QTextCursor(doc)
# Insert some text
>>>cursor.insertText('Hello world')
# Create a writer to save the document
>>>writer = QtGui.QTextDocumentWriter()
>>>writer.supportedDocumentFormats()
[PyQt4.QtCore.QByteArray(b'HTML'), PyQt4.QtCore.QByteArray(b'ODF'), PyQt4.QtCore.QByteArray(b'plaintext')]
>>>odf_format = writer.supportedDocumentFormats()[1]
>>>writer.setFormat(odf_format)
>>>writer.setFileName('hello_world.odt')
>>>writer.write(doc) # Return True if successful
True
QTextCursor also can insert tables, frames, blocks, images. More information. More information at:
http://qt-project.org/doc/qt-4.8/qtextcursor.html
As a bonus, you also can print to a pdf file by using QPrinter.