Working on a project where I am trying to query a SSAS data source we have at work through Python. The connection is presently within Excel files, but I am trying to reverse engineer the process with Python to automate part of the analysis I do on a day to day... I use the pyadomd library to connect to the data source, here`s my code:
clr.AddReference(r"C:\Program Files (x86)\Microsoft Office\root\vfs\ProgramFilesX86\Microsoft.NET\ADOMD.NET\130\Microsoft.AnalysisServices.AdomdClient.dll")
clr.AddReference('Microsoft.AnalysisServices.AdomdClient')
from Microsoft.AnalysisServices.AdomdClient import AdomdConnection , AdomdDataAdapter
from sys import path
path.append('C:\Program Files (x86)\Microsoft Office\root\vfs\ProgramFilesX86\Microsoft.NET\ADOMD.NET\130\Microsoft.AnalysisServices.AdomdClient.dll')
import pyadomd
from pyadomd import Pyadomd
from pyadomd._type_code import adomd_type_map, convert
constr= "connection string"
with Pyadomd(constr) as conn:
with conn.cursor().execute(query) as cur:
print(cur.fetchall())
Which works (in part), seemingly I am able to connect to the SSAS data source. Say I do conn = Pyadomd(constr), it returns no error (no more as it did before). The issue is when I try to execute the query with the cursor it returns an error saying:
File "C:\Users\User\Anaconda3\lib\site-packages\pyadomd\pyadomd.py", line 71, in execute
adomd_type_map[self._reader.GetFieldType(i).ToString()].type_name
KeyError: 'System.Object'
By doing a bit of research, I found that KeyError meant that the code was trying to access a key within a dictionary in which that key isn't present. By digging through my variables and going through the code, I realized that the line:
from pyadomd._type_code import adomd_type_map
Created this dictionary of keys:values:
See dictionary here
Containing these keys: System.Boolean, System.DateTime, System.Decimal, System.Double, System.Int64, System.String. I figured that the "KeyError: System.Object" was referring to that dictionary. My issue is how can I import this System.Object key to that dictionary? From which library/module/IronPython Clr reference can I get it from?
What I tried:
clr.AddReference("System.Object")
Gave me error message saying "Unable to find assembly 'System.Object'. at Python.Runtime.CLRModule.AddReference(String name)"
I also tried:
from System import Object #no error but didn't work
from System import System.Object #error saying invalid syntax
I think it has to do with some clr.AddReference IronPython thing that I am missing, but I've been looking everywhere and can't find it.
Thanks!
Glad that the newer version solved the problem.
A few comments to the code snippet above. It can be done a bit more concise 😊
Pyadomd will import the necessary classes from the AdomdClient, which means that the following lines can be left out.
clr.AddReference(r"C:\Program Files (x86)\MicrosoftOffice\root\vfs\ProgramFilesX86\Microsoft.NET\ADOMD.NET\130\Microsoft.AnalysisServices.AdomdClient.dll")
clr.AddReference('Microsoft.AnalysisServices.AdomdClient')
from Microsoft.AnalysisServices.AdomdClient import AdomdConnection , AdomdDataAdapter
Your code will then look like this:
import pandas as pd
from sys import path
path.append(r'C:\Program Files (x86)\MicrosoftOffice\root\vfs\ProgramFilesX86\Microsoft.NET\ADOMD.NET\130')
from pyadomd import Pyadomd
constr= "constring"
query = "query"
with Pyadomd(constr) as con:
with con.cursor().execute(query) as cur:
DF = pd.DataFrame(cur.fetchone(), columns = [i.name for i in cur.description])
The most important thing is to add the AdomdClient.dll to your path before importing the pyadomd package.
Furthermore, the package is mainly meant to be used with CPython version 3.6 and 3.7.
Well big problems require big solutions..
After endlessly searching the web, I went on https://pypi.org/project/pyadomd/ and directly contacted the author of the package (SCOUT). Emailed him the same question and apparently there was a bug within the code that he fixed overnight and produced a new version of the package, going from 0.0.5 to 0.0.6. In his words:
[Hi,
Thanks for writing me 😊
I investigated the error, and you are correct, the type map doesn’t support converting System.Object.
That is a bug!
I have uploaded a new version of the Pyadomd package to Pypi which should fix the bug – Pyadomd will now just pass a System.Object type through as a .net object. Because Pyadomd doesn’t know the specifics of the System.Object type at runtime, you will then be responsible yourself to convert to a python type if necessary.
Please install the new version using pip.]1
So after running a little pip install pyadomd --upgrade, I restarted Spyder and retried the code and it now works and I can query my SSAS cube !! So hopefully it can help others.
Snippet of the code:
import pandas as pd
import clr
clr.AddReference(r"C:\Program Files (x86)\MicrosoftOffice\root\vfs\ProgramFilesX86\Microsoft.NET\ADOMD.NET\130\Microsoft.AnalysisServices.AdomdClient.dll")
clr.AddReference('Microsoft.AnalysisServices.AdomdClient')
from Microsoft.AnalysisServices.AdomdClient import AdomdConnection , AdomdDataAdapter
from sys import path
path.append(r'C:\Program Files (x86)\MicrosoftOffice\root\vfs\ProgramFilesX86\Microsoft.NET\ADOMD.NET\130\Microsoft.AnalysisServices.Ado mdClient.dll')
import pyadomd
from pyadomd import Pyadomd
constr= "constring"
query = "query"
and then as indicated on his package website:
with Pyadomd(constr) as con:
with con.cursor().execute(query) as cur:
DF = pd.DataFrame(cur.fetchone(), columns = [i.name for i in cur.description])
and bam! 10795 rows by 39 columns DataFrame, I haven't precisely calculated time yet, but looking good so far considering the amount of data.
Related
I'm using Python 3.8.13 and Pandas 1.4.1 on Spyder 4.1.5.
I have no problem reading the dataframe into memory, but when I try to open the dataframe for viewing in the variable explorer I get the following:
"Spyder was unable to retrieve the value of this variable from the console. The error message was: Can't get attribute '_unpickle_block' on".
This is regardless of size (e.g. if I create a variable containing just the head of the dataset, the issue persists). Not sure why this specific error is showing as I don't believe I'm using a pickled object at any point.
I've confirmed that I can open the dataframe when using a Python 3.9 env with Spyder 5.1.5, but I need to use the former specs given they're locked in on a virtual machine. Here's my code as of now (very straightforward):
import pandas as pd
import numpy as np
filepath = "../../../projects2/"
df = pd.read_parquet(filepath + "hmda/lar/parquet/hmda_lar_2017.parquet")
I am trying to use the scikit-surprise module to build a recommender system however I am having an error in getting it to compile.
I am receiving the ImportError: Cannot import name "Reader" error
My class is as follows
import pandas as pd
from surprise import Reader, Dataset
userReviewsFilePath ="UserReviewsFirst5000WithHeadings.csv"
ratings = pd.read_csv(userReviewsFilePath) # reading data in pandas df
ratings_dict = {'recipeID': list(ratings.recipeID),
'rating': list(ratings.rating),
'userID': list(ratings.userID)}
df = pd.DataFrame(ratings_dict)
reader = Reader(rating_scale=(1, 5))
data = Dataset.load_from_df(df[['recipeID', 'rating', 'userID']], reader)
pip show says that version 1.0.6 is installed
I think your problem come from the installation... I installed "surprise" and past your code and it worked:
import pandas as pd
from surprise import Reader, Dataset
print(Reader) # or just print(surprise) if you import surprise
out:
<class 'surprise.reader.Reader'>
Start by re-install surprise and tell us.
If you have more than one version of python, do:
which pip
to see if you installed surprise on the used version of python
I think it's in surprise.reader: https://surprise.readthedocs.io/en/stable/reader.html
Your code should read:
from surprise.reader import Reader
from surprise.dataset import Dataset
Edit: I checked the instructions again which seem to contradict this, and give your original code as the correct example. https://surprise.readthedocs.io/en/stable/getting_started.html#getting-started
So maybe they add their own shortcuts? Either way, it seems like this isn't the correct solution, sorry. (Unless it works, in which case their instructions might be out of date.)
Edit 2: They do alias it, so "from surprise import Reader" should indeed have worked: https://github.com/NicolasHug/Surprise/blob/master/surprise/init.py#L19
I think you need to do
from surprise.reader import Reader
I'm experiencing a weird bug in a file conversion script I wrote in python. I'm using the comtypes library to convert different types of files into pdfs and for some reason, I get module 'comtypes.gen.Excel' has no attribute '_Application' whenever I try to create a client object for an excel application. I can't seem to find anything online specific to this issue. The script was working fine about a month or two ago, so I'm confused as to why it isn't working anymore - the only thing I could think of was excel updating or something (if that would even matter). I have the office 2016 if that's relevant. If anyone has experienced this bug or has any ideas, help would be greatly appreciated. Here's the script, for reference:
import comtypes.client
excel = comtypes.client.CreateObject("Excel.Application") # exception here
excel.Visible = False
in_file = "INPUT_FILE"
out_file = "OUT_FILE"
f = excel.Workbooks.Open(in_file)
f.ExportAsFixedFormat(0, out_file, 1, 0)
f.Close()
excel.Close()
Instead of using comtypes.client
Use pywin32
import win32com.client
#Replace excel=comtype.client.blablabla to
excel=win32com.client.DispatchEx("Excel.Application")
Keep remaining things as it is.
Ref. https://github.com/shardulind/pdfconv_python/blob/master/convertor.py
I intended
to write a code which helps me display Table / Dataframe on GUI (Kivy). To which I found the solution here. Apparently it uses a non-official package from a github repo which is dfgui.
The Problem
occurred to me when I executed as told on StackOverflow link. However returned Error that
wx._core.PyAssertionError: C++ assertion "!items.IsEmpty()" failed at
/usr/include/wx-3.0/wx/ctrlsub.h(154) in InsertItems(): need something
to insert
I Brokedown
the problem by selective execution in foll. way
import dfgui
import pandas as pd
xls = pd.read_excel('Res.xls')
df = pd.DataFrame(xls)
dfgui.show(df)
#dfgui.show(xls) Apparently the same as df
which then returned
TypeError: String or Unicode type required
and led me to this link, which I couldn't understand much.
Point me in North, or perhaps a different solution could be great too.
import json
from os import path, makedirs
_default_dir = path.expanduser('~/.config/gspread_pandas')
_default_file = 'google_secret.json'
def ensure_path(pth):
if not path.exists(pth):
makedirs(pth)
hi, I'm currently working on data collection via selenium and pandas to parse the data and edit it with pandas to send the data to google spread
however, while I'm working on gspread-pandas module, the module needs to put google_secret json file to '~/.config/gspread_pandas'. which is fixed location as described in the link below
https://pypi.python.org/pypi/gspread-pandas/0.15.1
I want to make some function to set the custom location to achieve independent working app environment.
for example, I want to locate the file to here
default_folder = os.getcwd()
the default_folder will be where my project is located(the same folder)
what can I do with it?
If you see the source https://github.com/aiguofer/gspread-pandas/blob/master/gspread_pandas/conf.py you can notice, that you can create your own config and pass it to Spread object constructor.
But yes, this part is really badly documented.
So, this code works well for me:
from gspread_pandas import Spread, conf
c = conf.get_config('[Your Path]', '[Your filename]')
spread = Spread('username', 'spreadname', config=c)
Thank you for this. It really should be documented better. I was getting so frustrated trying to get this to work with heroku, but it worked perfectly. I had to change to the following:
c = gspread_pandas.conf.get_config('/app/', 'google_secret.json')
spread = gspread_pandas.Spread('google_sheet_key_here_that_is_a_long_string', config=c)