I wanted to use this library to scrape data from otodom. I've read the docs however I'm stuck on a very basic level, trying to do basic import. The code I use is taken directly from the docs. On top of that I've tried to launch example.py from Github but I'm getting the same error.
Here's the excerpt from the code:
import otodom, os, logging
from otodom.category import get_category
from otodom.offer import get_offer_information
Error message I get:
Traceback (most recent call last):
File ".\otodom_import.py", line 2, in
from otodom.category import get_category
File "C:\Users\Dom\Anaconda3\lib\site-packages\otodom\category.py", line 9, in
from otodom.utils import get_response_for_url, get_url
File "C:\Users\Dom\Anaconda3\lib\site-packages\otodom\utils.py", line 14, in
from scrapper_helpers.utils import caching, normalize_text, key_sha1,
get_random_user_agent
File "C:\Users\Dom\Anaconda3\lib\site-packages\scrapper_helpers\utils.py",
line 22, in
MAX_FILENAME_LENGTH = subprocess.check_output("getconf NAME_MAX /",
shell=True).strip()
File "C:\Users\Dom\Anaconda3\lib\subprocess.py", line 336, in check_output
**kwargs).stdout
File "C:\Users\Dom\Anaconda3\lib\subprocess.py", line 418, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'getconf NAME_MAX /' returned non-zero
exit status 1.
Thanks in advance!
EDIT: Don't understand downvotes, I realize that this is a niche module but I did my homework. I've read the docs and tried to find help in other places. If there is a chance that as a noobie I'll find at least one person that can help me, why wouldn't I use this opportunity? Even info that module is buggy can help.
getconf, which the dependency scrapper_helpers module is trying to call, is a POSIX command not available on Windows.
You can track it down in C:\Users\Dom\Anaconda3\lib\site-packages\scrapper_helpers\utils.py and replace the line 22 with:
MAX_FILENAME_LENGTH = 255
But then the question is what other error might pop up given that the module obviously expects to run on a POSIX-compatible system - for example, it will attempt to use /var/tmp/scrapper-helpers/ as its temporary cache path which is also not available on Windows (but Python will interpret it as <DRIVE_LETTER>:\var\tmp\scrapper-helpers and attempt to create it).
Alternatively, looking at its code, you might get away with just declaring the following environment variables:
set MAX_FILENAME_LENGTH=255
set CACHE_DIR=%TEMP%\scapper-helpers\
Related
I am trying to make use of an open source Python program. I have a Python script configured exactly as per an example with all the pre-reqs installed, but I seem to be getting some kind of global Python error:
Traceback (most recent call last):
File "/etc/asterisk/scripts/detect/gong/gonglearn.py", line 15, in <module>
from pyAudioAnalysis.audioTrainTest import extract_features_and_train
File "/root/pyAudioAnalysis/pyAudioAnalysis/audioTrainTest.py", line 9, in <module>
from pyAudioAnalysis import MidTermFeatures as aF
File "/root/pyAudioAnalysis/pyAudioAnalysis/MidTermFeatures.py", line 8, in <module>
from pyAudioAnalysis import audioBasicIO
File "/root/pyAudioAnalysis/pyAudioAnalysis/audioBasicIO.py", line 6, in <module>
import eyed3
File "/usr/local/lib/python2.7/dist-packages/eyed3/__init__.py", line 31, in <module>
from .utils.log import log # noqa: E402
File "/usr/local/lib/python2.7/dist-packages/eyed3/utils/__init__.py", line 361
msg = f"invalid level choice: {level} (choose from {parser.log_levels})"
SyntaxError: invalid syntax
I went to look at the code for __init__.py and I found:
def __call__(self, parser, namespace, values, option_string=None):
values = values.split(':')
level, logger = values if len(values) > 1 else (values[0],
self.main_logger)
logger = logging.getLogger(logger)
try:
logger.setLevel(logging._nameToLevel[level.upper()])
except KeyError:
msg = f"invalid level choice: {level} (choose from {parser.log_levels})"
raise argparse.ArgumentError(self, msg)
super(LoggingAction, self).__call__(parser, namespace, values, option_string)
Since this appears to be system code, or something with eyed3, not specific to what I am trying to run, I am not really sure how to proceed. I can't find any references anywhere on how to debug this particular error.
What problem exactly is this indicating? I'm usually able to resolve syntax errors myself, but in this case the message seems a bit cryptic and I can't decipher it. I tried importing various other things into my script and doing import logging as log but neither of those has helped any. Do I have to somehow configure logging in Python before I can use it?
I apologize if this is a weird issue, but it seems fairly obscure, and it's been a while since I've used Python before, and that was on Windows.
I am running Python (not Python 3) on Debian 10.
The problem here is with the line suggested in the traceback:
msg = f"invalid level choice: {level} (choose from {parser.log_levels})"
The issue here is that this line is using f-strings, to do string interpolation. The reason this is a problem is that f-strings weren't introduced until python 3.6.
Unfortunately, because this is code from eyed3 you can't change it. You do have a few options though:
Upgrade your version of python to at least 3.6 (probably the best option)
fork the library, and edit that line so that you can use it with your version of python. Install your forked version. (This is probably a bad idea, this is unlikely to be the only place where there are incompatibilities. You will likely have to update quite a lot. It'll be easier to just update the version of python you're using).
If you look at the getting started section of the eyeD3 repo, it says:
Python >= 3.6 is required.
[autoreload of R17_Plot_profiles failed: Traceback (most recent call last):
File "/home/jdrevon/anaconda3/lib/python3.7/site-packages/IPython/extensions/autoreload.py", line 244, in check
superreload(m, reload, self.old_objects)
File "/home/jdrevon/anaconda3/lib/python3.7/site-packages/IPython/extensions/autoreload.py", line 378, in superreload
module = reload(module)
File "/home/jdrevon/anaconda3/lib/python3.7/imp.py", line 314, in reload
return importlib.reload(module)
File "/home/jdrevon/anaconda3/lib/python3.7/importlib/__init__.py", line 168, in reload
raise ModuleNotFoundError(f"spec not found for the module {name!r}", name=name)
ModuleNotFoundError: spec not found for the module 'R17_Plot_profiles'
]
This problem occurs in Python 3.7 on Spyder when I modify part of my code and I press F9 to execute some parts of it. Even if I execute my main python file which load every sub-python .py codes I have this error message. However, It does not stop the code from running but the file didn't save the modifications made. This is the first times it happens since I imported the 'R17_Plot_profiles.py' python file.
However in the case where I execute the main profile, I have the previous message but I have also the following one :
Reloaded modules: reconstruct, R2_Spice_Kernels, R4_Offset_Correction, R5_State_Vectors_twoway_v2, R6_Foot_Print, R7_Framework_Conversion,
R8_BendAng_ImpParam_up, R8_BendAng_ImpParam_dn, R9_Avg_BendAng_ImpParam, R10_Altitude_above_Ellipsoid, R11_Refractivity_and_Bending_Radius_v2,
R12a_Electron_Density_up, R12b_Electron_Density_dn, R12_Electron_Density_avg, R13_Neutral_Number_Density,
R14_Pressure_and_Temperature, R16_Errors, R17_Plot_profiles,
R3_Read_L2_Data_v2
So I don't know how to solve the autoreload problem. I've tried to downgrade my IPython version to 7.5 like some people tried and succeed but for me It doesn't change anything.
Edit : It's important to notice that if I restart the kernel, the first execution do not show the error message until I modify directly the python code.
I did few modifications yesterday and it seems to be solved. Since the error occurence started to be random at some points, i'm not sure for 100% but it can maybe solve the issue.
The problem could be came from the use of os.chdir function, I replaced it by specifying the entire path to save my figures until to go to the path using os.chdir and then save my figures in it.
It can maybe produce some conflicts of path and repository to find the python script which is no longer in the path where the script is currently working and therefore can't find it in order to save the modifications.
I keep you informed if at some point the error comes back but for now after some trials the error didn't show up anymore.
I'm trying to use the sutime python wrapper to make a date normalizer, that would convert any temporal information in strings into dates in the format YYYY-MM-DD. I've created a class, with rules over the sutime outputs to convert the sutime outputs into the standard format as mentioned above. The program is working properly on my local machine, but when i try to run it on a server I get the jpype._jclass.NoClassDefFoundError. The server is on ubuntu with python2, while my local has windows, with python3.
I've tried to implement the solutions to a similar problem on this https://sourceforge.net/p/jpype/discussion/379372/thread/689d7a9b/ forum, but i'm not sure if i was able to implement these soultions correctly. I've also checked that sutime supports both python3 and python2
I think the issue is with jpype or with the sutime library.
This is the traceback that i got
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "date_normalizer.py", line 38, in __init__
self.sutime = SUTime(jars=self.jar_files, mark_time_ranges=mark_time_ranges)
File "/home/bridgei2i/.local/lib/python2.7/site-packages/sutime/sutime.py", line 57, in __init__
'edu.stanford.nlp.python.SUTimeWrapper')
File "/home/bridgei2i/.local/lib/python2.7/site-packages/jpype/_jclass.py", line 130, in __new__
return _JClassNew(args[0], **kwargs)
File "/home/bridgei2i/.local/lib/python2.7/site-packages/jpype/_jclass.py", line 213, in _JClassNew
javaClass = _jpype.PyJPClass(arg)
jpype._jclass.NoClassDefFoundError: edu/stanford/nlp/python/SUTimeWrapper
Seems likely that the jar file holding edu/stanford/nlp/python/SUTimeWrapper was not found on the server. The specific code that failed was a call to JClass('edu.stanford.nlp.python.SUTimeWrapper') which is a request to load a class from a jar. I would recommend checking the classpath and configuration on the server.
Likely causes are (in order of likelihood)
jar file is not located in the classpath on the server.
The jar file is compiled with a JDK which is newer than runtime environment (though this should generate a different exception).
Some jar file that the class depends on is missing or has the wrong version. (this should produce a different classname in the exception, so it is unlikely.)
A dll for a native portion the jar file is missing or has an incorrect architecture. (rare)
Assuming the jar file is on the server, I would recommend checking the initialization in which the JPype startJVM call is made to see if the path to the jar was correct. It is also possible to examine the loaded classpath using print(jpype.java.lang.System.getProperty('java.class.path')) to see if there is a difference between your local and server machine.
thank you, as you said:Some jar file that the class depends on is missing or has the wrong version. (this should produce a different classname in the exception, so it is unlikely.)
I'm using check_output to do all my SSH and GitHub setup, and I'm attempting to execute eval $(ssh-agent), both to start the agent and to parse the output if I need the process id.
from subprocess import check_output
out = check_output(["eval", "$(ssh-agent)"])
print(out)
But regardless of how I escape things, I get the same error.
Traceback (most recent call last):
File "gitSetup.py", line 3, in <module>
out = check_output(["eval", "$(ssh-agent)"])
File "/usr/lib/python2.7/subprocess.py", line 216, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
I'm wondering if I'm not escaping or trimming my arguments properly; if anyone sees my error, I'd appreciate some insight!
Even if you managed to fix the superficial syntax problems, a subprocess which runs successfully simply then terminates without a trace; it doesn't attempt to (and couldn't, even if it wanted to) modify the environment of the parent Python process. This is a common FAQ.
You could run the eval in the parent process which starts your Python script, or perhaps figure out how to communicate with ssh-agent directly from Python. Its output is usually a simple sequence of variable assignments, which you can parse yourself.
from subprocess import check_output
from os import environ
eval_string = check_output(['ssh-agent'])
for line in eval_string.rstrip('\n').split('\n'):
for expr in line.rstrip(';').split(';'):
if expr.startswith((' export ', 'echo ')):
continue
var, value = expr.strip().split('=', 1)
environ[var] = value
If the output from ssh-agent contains shell quoting, you will need to perform additional parsing on it (basically, trim the quotes around the value string). But this is already rather clunky and brittle, so perhaps revert to setting up the environment before launching Python instead of trying to splash some sophistication onto this kludge.
In more detail, ssh-agent and a precious few other shell utilities have a very specific design in order for them to be able to communicate with their parent process. Precisely because a subprocess cannot make any changes in the environment of its parent process, it instead prints a piece of code for its parent process to execute. By default, it prints sh code like this:
SSH_AUTH_SOCK=/tmp/ssh-MUyniqn10506/agent.10506; export SSH_AUTH_SOCK;
SSH_AGENT_PID=10507; export SSH_AGENT_PID;
echo Agent pid 10507;
There is also an option for it to print similar code in csh syntax (this is a historical shell which thankfully isn't used much any longer) but, alas, no option for producing Python code. (It would not be hard to make ssh-agent do that, per se.)
(The above output sample copy/pasted from http://blog.joncairns.com/2013/12/understanding-ssh-agent-and-ssh-add/ which contains a fuller explanation.)
I am having trouble porting a working, Windows Python application to Linux. I am having some problems, because I did not write the code and am just learning Python. I am having trouble fixing the issues that it keeps throwing up. So here is a kind of error that right now I am stuck with
Traceback (most recent call last):
File "alpha_beta", line 237, in <module>
main()
File "alpha_beta", line 185, in main
ABCCmd()
File "alpha_beta.py", line 74, in ABCCmd
File "C:\softs\Python\Lib\shutil.py", line 80, in copy
File "C:\softs\Python\Lib\shutil.py", line 47, in copyfile
IOError: [Errno 13] Permission denied: '/myPath/XFiles.bin.addr_patched
Any pointers on how to fix it will be much appreciated
Edit:
1) What I mean by I am stuck is, the traceback of the error goes to C:\softs\Python\Lib but actually I am currently executing this code in Ubuntu. Why would the traceback reference to windows library
2) Another thing that bothers me is it says there is an IOError.But when I try to add permission for the denied one it gives me a chmod: changing permissions of /myPath/xFiles.bin.addr_patched': Operation not permitted
Edit 2:
I had commented out a module because I thought it wasn't very useful. Since Now I am anyway discussing the porting issues, I thought I can bring up this additional problem as well since I think the issue is the same and the fix should be similar. On including #pdb module in the python code, I get the following error
traceback (most recent call last):
File "alpha_beta", line 6, in <module>
import pdb
File "C:\softs\Python\Lib\pdb.py", line 14, in <module>
File "C:\softs\Python\Lib\pprint.py", line 39, in <module>
ImportError: No module named cStringIO
I looked at the importer_Cache and looks like this
'': None, '/usr/lib/python2.6/plat-linux2': None, '/usr/lib/python2.6/': None
'/usr/lib/pymodules/python2.6/gtk-2.0': None, '/usr/lib/python2.6/lib-tk': None,
'/usr/lib/python2.6/lib-old': <imp.NullImporter object at 0x7f1269048070>, '/usr/
/python2.6/dist-packages/gtk-2.0': None, '/usr/lib/python2.6/dist-packages/PIL': None,
'/usr/local/lib/python2.6/dist-packages': None, '/usr/lib/python2.6/dist-packages':
None
*SOLVED : There was a weird wrapper in the script which was causing the python script to execute from a different location. Still don't understand how though. After modifying my script to just directly execute as python myScript.py it runs fine. Thank you all anyway for the help
The mixture of Windows and Unix style paths in your error messages makes me think that you may have some filenames that are hard coded or manually created using os specific path separators ('\' or '/'). If you can figure out where the gOptions.inputTf and gWorkingTfFile values are assigned you should look into using os.path.join to help you use the correct paths.
from os.path import join
file = join('A', 'B')
# 'A/B' on unix systems
# 'A\B' on windows systems