We Keep Coding

'unicode' object has no attribute 'saveAsTextFile'

I am trying to modify my file after fetching it from HDFS using pyspark and then i want to save it in HDFS for that i have written below code.
Code:
import subprocess
from subprocess import Popen, PIPE
from pyspark import SparkContext
cat = sc.textFile("/user/root/parsed.txt")
hrk = "#"
for line in cat.collect():
if (code == "ID"):
line =line.strip() + "|"+hrk
line.saveAsTextFile("/user/root/testsprk")
print(line)
But when i run the code i am getting below error.
Error:
Traceback (most recent call last):
File "<stdin>", line 30, in <module>
AttributeError: 'unicode' object has no attribute 'saveAsTextFile'
I know there is some issue with my line variable but i am not able to fix it.

It because you are collecting all data, it means that collection is not RDD, but normal list and line is just one string.
You shouldn't collect all data on driver. Instead, use RDD.map and then RDD.saveAsTextFile
def add_hrk_on_id(line):
if (code == "ID"):
return line.strip() + "|"+hrk
else
return line
cat.map(add_hrk_on_id).saveAsTextFile(path)

How do I save a file object to a shelve file?

When I run the following code:
import shelve
input = open("input.txt",)
shelveFile = shelve.open("myData")
shelveFile["inputFile"] = input
input.close()
shelveFile.close()
I expect the shelve file myData to hold the file object input. Instead, running the code produces the following error:
Traceback (most recent call last):
File "/Users/ashutoshmishra/Documents/Sandbox/Sandbox3.py", line 5, in <module>
shelveFile["inputFile"] = input
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/shelve.py", line 124, in __setitem__
p.dump(value)
TypeError: cannot serialize '_io.TextIOWrapper' object
I was wondering why I could not save the file object input to the shelve file myData?

The following answer is taken from #DanD.'s comment above.
Read the file: shelveFile["inputFile"] = input.read()

GetItem(),ItemCount(),SubElements() functions not working in pywinauto

I'm trying to automate tasks for teamcenter using pywinauto python module.But I was not able to work properly with TreeViews. When ever I'm trying to use GetItem(),SubElements(),Item() and many other function under _treeview_element giving error.Please look into below errors.
I'm using python 2.7 and i also have pywin32 module installed
for i in range(tr.ItemCount()):
print tr.GetItem(i)
Traceback (most recent call last):
File "<pyshell#61>", line 2, in <module>
print tr.GetItem(i) File "C:\Users\patibj\Desktop\pywinauto-
master\pywinauto\controls\common_controls.py", line 1010, in GetItem
if isinstance(path[0], int):
TypeError: 'int' object has no attribute '__getitem__'
This is when i'm trying to use GetChild()
a[0] is <pywinauto.controls.common_controls._treeview_element object at 0x02EC11B0> a[0].GetChild((1))
Traceback (most recent call last):
File "<pyshell#49>", line 1, in <module>
a[0].GetChild((1))
File "C:\Users\patibj\Desktop\pywinauto- master\pywinauto\controls\common_controls.py", line 840, in GetChild
return self.Children()[index]
File "C:\Users\patibj\Desktop\pywinauto-master\pywinauto\controls\common_controls.py", line 739, in Children
if self.Item().cChildren not in (0, 1):
File "C:\Users\patibj\Desktop\pywinauto-master\pywinauto\controls\common_controls.py", line 539, in Item
return self._readitem()[0]
File "C:\Users\patibj\Desktop\pywinauto-master\pywinauto\controls\common_controls.py", line 904, in _readitem
raise ctypes.WinError() WindowsError: [Error 0] The operation completed successfully.
Please help me out .Thanks in advance.

TreeViewWrapper.GetItem() method gets path argument which should be the string starting with back-slash \\ symbol. Example:
app.Dialog.TreeView.GetItem('\\1st_level_item_name\\2ndlevelitem').Click()
For access by index use a list of integers:
app.Dialog.TreeView.GetItem([1, 0])
GetChild() works with explorer.exe TreeView:
>>> tree.GetItem([1]).GetChild(2).Text()
u'Computer'

How can I recover data from a possibly corrupt, or too large Python Shelve db?

I've been using Shelve as a document store.
The key is a stringified integer and the value is just some html in a string. Unfortunately my script ended up putting so many entries in the db that errors occurred (I don't have the exact ones to hand). The db is about 36GB in size and now when I load it and then try and iterate on the keys or anything like that I get the following error...
import shelve
db = shelve.open("my.shelf")
ks = db.keys()
for k in ks: print(k)
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/_collections_abc.py", line 482, in __iter__
yield from self._mapping
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/shelve.py", line 95, in __iter__
for k in self.dict.keys():
SystemError: Negative size passed to PyBytes_FromStringAndSize
>>> list(ks.__dict__.values())[0].dict
<_dbm.dbm object at 0x10037ef90>
>>> help(list(ks.__dict__.values())[0].dict)
>>> list(ks.__dict__.values())[0].dict.keys()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
SystemError: Negative size passed to PyBytes_FromStringAndSize
This is on OSX Yosemite. Python 3.4
Any way to repair this or get the keys and values out for placement in a more appropriate store?

You must know key names whose corresponding objects were successfully stored.
Because a failure at saving data object would corrupt your db.
Then,
db = shelve.open("my.shelf")
for key in list_of_successfully_saved_obj's_name:
val = db.get(key)
...

Strange exception with python's cgitb and inspect.py

I have a function that decodes an exception and pushes the info to a file. Following is what I do basically:
exc_info = sys.exc_info
txt = cgitb.text(exc_info)
Using this, I got the following exception trace:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\site-packages\job_queue\utils\start_workers.py", line 40, in start_worker
worker_loop(r_jq, worktype, worker_id)
File "C:\Python27\lib\site-packages\job_queue\server\jq_worker.py", line 55, in worker_loop
_job_machine(*job)
File "C:\Python27\lib\site-packages\job_queue\server\jq_worker.py", line 34, in _job_machine
do_verbose_exception()
File "C:\Python27\lib\site-packages\job_queue\server\errors.py", line 23, in do_verbose_exception
txt = cgitb.text(exc_info)
File "C:\Python27\lib\cgitb.py", line 214, in text
formatvalue=lambda value: '=' + pydoc.text.repr(value))
File "C:\Python27\lib\inspect.py", line 885, in formatargvalues
specs.append(strseq(args[i], convert, join))
File "C:\Python27\lib\inspect.py", line 840, in strseq
return convert(object)
File "C:\Python27\lib\inspect.py", line 882, in convert
return formatarg(name) + formatvalue(locals[name])
KeyError: 'connection'
I ran the code multiple times after this exception, but couldn't reproduce it. However, I didn't find any reference in files cgitb.py or inspect.py to a dict with 'connection' key either.
Will anybody know if this is an issue with python's cgitb or inspect files? Any helpful inputs?

You passed a wrong type to text function
below is the correct way.
cgitb.text((sys.last_type, sys.last_value, sys.last_traceback))

Im not sure specifically why this exception is happening, but have you read the docs for cgitb module? It seems that since python 2.2 it has supported writing exceptions to a file:
http://docs.python.org/library/cgitb.html
Probably something like:
cgitb.enable(0, "/my/log/directory") # or 1 if you want to see it in browser
As far as your actual traceback, are you sure 'connection' isnt a name you are using in your own code? 'inspect' module is most likely trying to examine your own code to build the cgi traceback info and getting a bad key somewhere?