** Importantly, this error does not occur all the time. It pops up every now and then, so I don't believe my code is at fault here.**
Hello, I am using threading along with Python 3.7 shelves. I have a dictionary object inside my shelf. Is this an error caused by threading/simultaneous access? (Using python's Threading library).
Relevant code is merely:
requestLibrary[requestID]
I am certain that this works, the key has been placed inside the library. But for some instances, the I get this key error. I suspect it has to do with the threading.
Exception in thread Thread-14:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/shelve.py", line 111, in __getitem__
value = self.cache[key]
KeyError: '4fGdb'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/lme/Workflow/Source/busDriver.py", line 189, in busDriver
while 'workflowEndingPoint'.lower() not in str(requestLibrary[requestID]).lower():
File "/usr/local/lib/python3.6/shelve.py", line 113, in __getitem__
f = BytesIO(self.dict[key.encode(self.keyencoding)])
KeyError: b'4fGdb'
Related
I'm running a pyspark job submit on a university server:
My configuration is :
--master yarn --deploy-mode cluster --num-executors 150 --executor-cores 4 --executor-memory 28g --driver-memory 28g
My first few steps runs correctly :
df = spark.read.format('csv') \
.option('header',True) \
.option('multiLine', True) \
.load(data_file)
df.show()
udf_function = udf(stamp, StringType())
new_df = df.withColumn("column_a", udf_function(struct([df[x] for x in df.columns])))
new_df.show()
When I try to run the following commands separately, I get two very similar errors:
Command 1:
new_df.select("column_a").distinct().show(100)
Error:
ERROR:root:Exception while sending command.
Traceback (most recent call last):
File "/hadoop4/yarn/nm/usercache/apps/appcache/application_1593105789029_2249545/container_e01_1593105789029_2249545_02_000002/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1159, in send_command
raise Py4JNetworkError("Answer from Java side is empty")
py4j.protocol.Py4JNetworkError: Answer from Java side is empty
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/hadoop4/yarn/nm/usercache/apps/appcache/application_1593105789029_2249545/container_e01_1593105789029_2249545_02_000002/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 985, in send_command
response = connection.send_command(command)
File "/hadoop4/yarn/nm/usercache/apps/appcache/application_1593105789029_2249545/container_e01_1593105789029_2249545_02_000002/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1164, in send_command
"Error while receiving", e, proto.ERROR_ON_RECEIVE)
py4j.protocol.Py4JNetworkError: Error while receiving
Traceback (most recent call last):
File "python_stamp.py", line 93, in <module>
main()
File "python_stamp.py", line 82, in main
new_df.select("planning_cluster_id").distinct().show(100)
File "/hadoop4/yarn/nm/usercache/apps/appcache/application_1593105789029_2249545/container_e01_1593105789029_2249545_02_000002/pyspark.zip/pyspark/sql/dataframe.py", line 380, in show
Command 2:
new_df.write.mode("overwrite").format("csv").option("delimiter", ",").option("header", "true").save(save_path)
Error:
ERROR:root:Exception while sending command.
Traceback (most recent call last):
File "/hadoop1/yarn/nm/usercache/apps/appcache/application_1593105789029_2249417/container_e01_1593105789029_2249417_02_000002/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1159, in send_command
raise Py4JNetworkError("Answer from Java side is empty")
py4j.protocol.Py4JNetworkError: Answer from Java side is empty
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/hadoop1/yarn/nm/usercache/apps/appcache/application_1593105789029_2249417/container_e01_1593105789029_2249417_02_000002/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 985, in send_command
response = connection.send_command(command)
File "/hadoop1/yarn/nm/usercache/apps/appcache/application_1593105789029_2249417/container_e01_1593105789029_2249417_02_000002/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1164, in send_command
"Error while receiving", e, proto.ERROR_ON_RECEIVE)
py4j.protocol.Py4JNetworkError: Error while receiving
Traceback (most recent call last):
File "python_stamp.py", line 91, in <module>
main()
File "python_stamp.py", line 83, in main
new_df.write.mode("overwrite").format("csv").option("delimiter", ",").option("header", "true").save(save_path)
File "/hadoop1/yarn/nm/usercache/apps/appcache/application_1593105789029_2249417/container_e01_1593105789029_2249417_02_000002/pyspark.zip/pyspark/sql/readwriter.py", line 738, in save
File "/hadoop1/yarn/nm/usercache/apps/appcache/application_1593105789029_2249417/container_e01_1593105789029_2249417_02_000002/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
File "/hadoop1/yarn/nm/usercache/apps/appcache/application_1593105789029_2249417/container_e01_1593105789029_2249417_02_000002/pyspark.zip/pyspark/sql/utils.py", line 63, in deco
File "/hadoop1/yarn/nm/usercache/apps/appcache/application_1593105789029_2249417/container_e01_1593105789029_2249417_02_000002/py4j-0.10.7-src.zip/py4j/protocol.py", line 336, in get_return_value
py4j.protocol.Py4JError: An error occurred while calling o112.save
Does anyone know the reason behind it? I'm pretty confident that it's not because of any memory error, as the previous steps which show the table, load the table all are running correctly.
Additional information: When I run all of these commands on pyspark shell, they run perfectly well.
I'm trying to get this, https://github.com/joaquinlpereyra/twitterImgBot, to work
and it works and it seems ok.
But after some hours, it stops working and this error comes up:
*python3 twitterbot.py
Traceback (most recent call last):
File "/home/user/.local/lib/python3.7/site-packages/tweepy/binder.py", line 118, in build_path
value = quote(self.session.params[name])
KeyError: 'id'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "twitterbot.py", line 209, in <module>
main()
File "twitterbot.py", line 200, in main
orders()
File "twitterbot.py", line 118, in orders
timeline.delete_tweet_by_id(tweet.in_reply_to_status_id, api)
File "/home/user/Skrivebord/twitterboot/lo/bot/timeline.py", line 12, in delete_tweet_by_id
api.destroy_status(id_to_delete)
File "/home/user/.local/lib/python3.7/site-packages/tweepy/binder.py", line 245, in _call
method = APIMethod(args, kwargs)
File "/home/user/.local/lib/python3.7/site-packages/tweepy/binder.py", line 71, in __init__
self.build_path()
File "/home/user/.local/lib/python3.7/site-packages/tweepy/binder.py", line 120, in build_path
raise TweepError('No parameter value found for path variable: %s' % name)
tweepy.error.TweepError: No parameter value found for path variable: id*
It seems like the Python has some problem because if I make a new install on a another PC it works for some hours and then stops.
Strange.
This is likely because tweet is not in reply to a status, so has an in_reply_to_status_id attribute that's None, so API.destroy_status is called with an id of None.
I am unable to access the disk drives on the Windows system using infi.manager package found on PyPi.
Tried the below :
from infi.devicemanager import DeviceManager
dm = DeviceManager()
dm.root.rescan()
disks = dm.disk_drives
names = [disk.friendly_name for disk in disks]
Error messages :
Traceback (most recent call last):
File "C:\Users\rsushmit\AppData\Local\Programs\Python\Python37-32\lib\site-packages\infi\devicemanager\setupapi\functions.py", line 56, in callee
yield decorated_func(*args, **kwargs)
File "C:\Users\rsushmit\AppData\Local\Programs\Python\Python37-32\lib\site-packages\infi\devicemanager\setupapi\functions.py", line 70, in SetupDiEnumDeviceInfo
interface(device_info_set, index, device_info_buffer)
File "C:\Users\rsushmit\AppData\Local\Programs\Python\Python37-32\lib\site-packages\infi\cwrap\__init__.py", line 138, in __new__
return_value = function(*args[1:], **kwargs)
File "C:\Users\rsushmit\AppData\Local\Programs\Python\Python37-32\lib\site-packages\infi\devicemanager\setupapi\__init__.py", line 35, in errcheck
raise WindowsException(GetLastError())
infi.devicemanager.setupapi.WindowsException: 259, No more data is available.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\rsushmit\AppData\Local\Programs\Python\Python37-32\lib\site-packages\infi\devicemanager\setupapi\functions.py", line 60, in callee
raise StopIteration
StopIteration
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\rsushmit\AppData\Local\Programs\Python\Python37-32\lib\site-packages\infi\devicemanager\__init__.py", line 215, in disk_drives
for controller in self.storage_controllers:
File "C:\Users\rsushmit\AppData\Local\Programs\Python\Python37-32\lib\site-packages\infi\devicemanager\__init__.py", line 227, in storage_controllers
return self.get_devices_from_handle(handle)
File "C:\Users\rsushmit\AppData\Local\Programs\Python\Python37-32\lib\site-packages\infi\devicemanager\__init__.py", line 198, in get_devices_from_handle
for devinfo in functions.SetupDiEnumDeviceInfo(handle):
RuntimeError: generator raised StopIteration
from infi.devicemanager import DeviceManager
dm = DeviceManager()
dm.root.rescan()
devices = dm.all_devices
for device in devices:
print(device)
I am running Dask Distributed on Linux CentOS 7, with a Python 3.6.2 installation. My computation seems to be getting fine (I am still improving my code, but I am able to have some results), but I keep getting some python errors apparently linked to tornado module. I am only launching a one node standalone Dask distributed cluster.
Here is the most common example:
Exception in thread Client loop:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/tornado/ioloop.py", line 832, in start
self._run_callback(self._callbacks.popleft())
AttributeError: 'NoneType' object has no attribute 'popleft'
And here is another one:
tornado.application - ERROR - Exception in callback <bound method WorkStealing.balance of <distributed.stealing.WorkStealing object at 0x7f752ce6d6a0>>
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/tornado/ioloop.py", line 1026, in _run
return self.callback()
File "/usr/local/lib/python3.6/site-packages/distributed/stealing.py", line 248, in balance
sat = s.rprocessing[key]
KeyError: 'read-block-9024000000-e3fefd2110094168cc0505db69b326e0'
Do you have any idea why? Should I close some connections or stop the standalone cluster?
Yes, if you don't close down the Tornado IOLoop before exiting the process then it can die in an unpleasant way. Fortunately this shouldn't affect your application, except by looking unpleasant.
You might submit a bug report about this, it's still something that we should fix.
When I start spyder the internal console pops up and gives the following "traceback":
>>> WARNING:traitlets:kernel died: 6.001837253570557
Traceback (most recent call last):
File "C:\...\lib\site-packages\traitlets\traitlets.py", line 528, in get
value = obj._trait_values[self.name]
KeyError: 'banner'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\...\lib\site-packages\qtconsole\base_frontend_mixin.py", line 163, in _dispatch
handler(msg)
File "C:\...\lib\site-packages\qtconsole\jupyter_widget.py", line 296, in _handle_kernel_info_reply
super(JupyterWidget, self)._started_channels()
File "C:\...\lib\site-packages\qtconsole\frontend_widget.py", line 617, in _started_channels
self.reset(clear=True)
File "C:\...\lib\site-packages\qtconsole\frontend_widget.py", line 661, in reset
self._append_plain_text(self.banner)
File "C:\...\lib\site-packages\traitlets\traitlets.py", line 556, in __get__
return self.get(obj, cls)
File "C:\...\lib\site-packages\traitlets\traitlets.py", line 535, in get
value = self._validate(obj, dynamic_default())
File "C:\...\lib\site-packages\spyder\widgets\ipythonconsole\shell.py", line 280, in _banner_default
return self.long_banner()
File "C:\...\lib\site-packages\spyder\widgets\ipythonconsole\shell.py", line 91, in long_banner
from IPython.core.usage import quick_guide
ImportError: cannot import name 'quick_guide'
The IPython console works without problems, it is just missing the "introduction text" (python version, help-commands).
Can I just ignore this exception or is there a way to fix this manually?
(Spyder developer here) This was fixed in Spyder 3.1.4 (just released). Please update to fix it.
To fix the exception manually one needs to change the last mentioned file in the traceback. In this case this one:
File "C:\...\lib\site-packages\spyder\widgets\ipythonconsole\shell.py", line 91, in long_banner
from IPython.core.usage import quick_guide
ImportError: cannot import name 'quick_guide'
and change quick_guide to quick_reference as quick_guide. One may need to restart Spyder but then the Exception disappeared.
I solved:
Only use conda install spyder=3.1.4 in your conda env.
Regards.