We Keep Coding

Writing a dask dataframe to parquet using to_parquet() results "RuntimeError: file metadata is only available after writer close"

I am trying to use store Dask dataframe in parquet files. I have pyarrow library installed.
import numpy as np
import pandas as pd
import dask.dataframe as dd
df = pd.DataFrame(np.random.randint(100,size=(100000, 20)),columns=['A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T'])
ddf = dd.from_pandas(df, npartitions=10)
ddf.to_parquet('saved_data_prqt', compression='snappy')
However, I get this error as a result of my code
---------------------------------------------------------------------------
ArrowNotImplementedError Traceback (most recent call last)
~\anaconda3\lib\site-packages\pyarrow\parquet.py in write_table(table, where, row_group_size, version, use_dictionary, compression, write_statistics, use_deprecated_int96_timestamps, coerce_timestamps, allow_truncated_timestamps, data_page_size, flavor, filesystem, compression_level, use_byte_stream_split, data_page_version, use_compliant_nested_type, **kwargs)
~\anaconda3\lib\site-packages\pyarrow\parquet.py in close(self)
682 self.is_open = False
683 if self._metadata_collector is not None:
--> 684 self._metadata_collector.append(self.writer.metadata)
685 if self.file_handle is not None:
686 self.file_handle.close()
.................. it's a long error description which I shortened. If the whole error text required please let me know in the comments section and I'll try to add the full version.
~\anaconda3\lib\site-packages\pyarrow\_parquet.pyx in pyarrow._parquet.ParquetWriter.metadata.__get__()
RuntimeError: file metadata is only available after writer close
Does anybody know how to debug the error and what the reason of it?
Thank you!

I ran your exact code snippets and the Parquet files were written without any error. This code snippet also works:
ddf.to_parquet("saved_data_prqt", compression="snappy", engine="pyarrow")
I'm using Python 3.9.7, Dask 2021.8.1, and pyarrow 5.0.0. What versions are you using?
Here's the notebook I ran and here's the environment if you'd like to replicate my computations exactly.

I fixed the error by creating an isolated virtual environment with Python 3.9 and Pyarrow 5.0 in conda followed by installation of corresponding Python kernel in Jupyter Notebook.
It's important to activate the environment in conda followed by launching Jupyter Notebook from conda otherwise (for unknow reason) if I open Jupyter Notebook from windows start menu the error persists.

Python Pandas to convert CSV to Parquet using Fastparquet

I am using Python 3.6 interpreter in my PyCharm venv, and trying to convert a CSV to Parquet.
import pandas as pd
df = pd.read_csv('/parquet/drivers.csv')
df.to_parquet('output.parquet')
Error-1
ImportError: Unable to find a usable engine; tried using: 'pyarrow', 'fastparquet'.
pyarrow or fastparquet is required for parquet support
Solution-1
Installed fastparquet 0.2.1
Error-2
File "/Users/python parquet/venv/lib/python3.6/site-packages/fastparquet/compression.py", line 131, in compress_data
(algorithm, sorted(compressions)))
RuntimeError: Compression 'snappy' not available. Options: ['GZIP', 'UNCOMPRESSED']
I Installed python-snappy 0.5.3 but still getting the same error? Do I need to install any other library?
If I use PyArrow 0.12.0 engine, I don't experience the issue.

In fastparquet snappy compression is an optional feature.
To quickly check a conversion from csv to parquet, you can execute the following script (only requires pandas and fastparquet):
import pandas as pd
from fastparquet import write, ParquetFile
df = pd.DataFrame({"col1": [1,2,3,4], "col2": ["a","b","c","d"]})
# df.head() # Test your initial value
df.to_csv("/tmp/test_csv", index=False)
df_csv = pd.read_csv("/tmp/test_csv")
df_csv.head() # Test your intermediate value
df_csv.to_parquet("/tmp/test_parquet", compression="GZIP")
df_parquet = ParquetFile("/tmp/test_parquet").to_pandas()
df_parquet.head() # Test your final value
However, if you need to write or read using snappy compression you might follow this answer about installing snappy library on ubuntu.

I've used the following versions:
python 3.10.9 fastparquet==2022.12.0 pandas==1.5.2
This code works seemlessly for me
import pandas as pd
df = pd.read_csv('/parquet/drivers.csv')
df.to_parquet('output.parquet', engine="fastparquet")
I'd recommend you move away from python 3.6 as it has reached end of life and is no longer supported.

How can I open a .snappy.parquet file in python?

How can I open a .snappy.parquet file in python 3.5? So far, I used this code:
import numpy
import pyarrow
filename = "/Users/T/Desktop/data.snappy.parquet"
df = pyarrow.parquet.read_table(filename).to_pandas()
But, it gives this error:
AttributeError: module 'pyarrow' has no attribute 'compat'
P.S. I installed pyarrow this way:
pip install pyarrow

I have got the same issue and managed to solve it by following the solutio proposed in https://github.com/dask/fastparquet/issues/366 solution.
1) install python-snappy by using conda install (for some reason with pip install, I couldn't download it)
2) Add the snappy_decompress function.
from fastparquet import ParquetFile
import snappy
def snappy_decompress(data, uncompressed_size):
return snappy.decompress(data)
pf = ParquetFile('filename') # filename includes .snappy.parquet extension
dff=pf.to_pandas()

The error AttributeError: module 'pyarrow' has no attribute 'compat' is sadly a bit misleading. To execute the to_pandas() function on a pyarrow.Table instance you need pandas installed. The above error is a sympton of the missing requirement.
pandas is a not a hard requirement of pyarrow as most of its functionality is usable with just Python built-ins and NumPy. Thus users of pyarrow which include pandas can work with it without needing to have pandas pre-installed.

You can use pandas to read snppay.parquet files into a python pandas dataframe.
import pandas as pd
filename = "/Users/T/Desktop/data.snappy.parquet"
df = pd.read_parquet(filename)

Error with pulling data from Yahoo Finance

I am trying to pull data from Yahoo Finance via Pandas. I have used similar pulls before, but haven't faced any issue before this
import pandas as pd
import numpy as np
import datetime as dt
from dateutil import parser
from pandas_datareader import data
from dateutil.relativedelta import relativedelta
end_date=dt.datetime.today()
begdate = end_date + relativedelta(years=-10)
data1 = data.get_data_yahoo('^DJI',begdate,end_date,interval='m')
This is the error I am getting
RemoteDataError: Unable to read URL: http://ichart.finance.yahoo.com/table.csv
I am using Python 3.5

EDIT:
This issue has been fixed as of v0.5.0 of pandas-reader. The fix below no longer applies.
As pointed out by others, the API endpoint has changed and a patch has been made but hasn't been merged to the master branch of pandas-datareader yet (as of 2017-05-21 6:19 UTC). The fix is at this branch by Rob Kimball (Issue | PR). For a temporary fix (until the patch is merged into master), try:
$ pip install git+https://github.com/rgkimball/pandas-datareader#fix-yahoo --upgrade
Or, in case you want to tweak the source code:
$ git clone https://github.com/rgkimball/pandas-datareader
$ cd pandas-datareader
$ git checkout fix-yahoo
$ pip install -e .
On Python:
import pandas_datareader as pdr
print(pdr.__version__) # Make sure it is '0.4.1'.
pdr.get_data_yahoo('^DJI')

Import error: No module named gspread or csv?

I am working with a Google spreadsheet, and am trying to use Python 2.7 to convert the spreadsheet data to a CSV file.
When I attempt to run the script I receive:
Import error: No module named gspread.
When I take out the gspread portion, then I receive:
Import error: No module named csv.
Any suggestions would be greatly appreciated. Thank you in advance.
import csv
import gspread
g=gspread.login('skiesgoinggreen#gmail.com', 'e-mail_password')
docid = "0AgNp9UJ4CX93dHl3RW9GRXJDS3kxaXRJMGNqWmhQWVE"
spreadsheet = g.open_by_key(docid)
for i, worksheet in enumerate(spreadsheet.worksheets()):
filename = docid + '-worksheet' + str(i) + '.csv'
writer = csv.writer(open(filename, 'wb'))
writer.writerows(worksheet.get_all_values())

Try the following:
Check you don't have a file named csv.py
Check you have python version >= 2.3
Install gspread with:
pip install gspread

Let's get some clarification here... the "pip install gspread" command is only for linux.
If you are on Windows, you need to have these modules in C:\Python27\Lib\site-packages
I thought CSV module came built-in but I checked and I don't have gspread, you need to download it and place the gspread.py file in the aforementioned directory.