We Keep Coding

skip previously completed luigi task

I have inherited a luigi framework and I'm trying to debug some stuff and add features. The first script runs this command:
yield FinalizeData(environment=environment)
and then within a second script, we have:
#FinalizeData requires UploadData to have run
#UploadData requires ValidateData to have run
#ValidateData requires PullData to have run
#PullData is the first class that fires
I am trying to debug a few things that happening in the the ValidateData class - but in order to run it, I need to have PullData execute first, and it contains a SQL query that takes about an hour to run, and ultimately generates a .PKL file. Because I already have this .PKL file, I would like to "skip" this piece and go directly to the second UploadData class. I am not sure how to do that however.
Here is the first PullData class:
class PullData(luigi.Task):
environment = luigi.Parameter(default='dev')
def requires(self):
resp = list()
tests = {
'dev' : TestDevConnection(self.instance, role='user'),
}
resp.append(tests.get(self.environment))
return resp
def output(self):
return luigi.LocalTarget(f'.out/{CUR_DATE}_{self.environment}_{self.instance}_Data.pkl')
def run(self):
try:
mysql_dict = dict(environment=self.environment, role='user', instance=self.instance)
conn = get_conn(self.environment, role='user', instance=self.instance)
except Exception as e:
log.error_job(mysql_dict, self.environment, self.instance + '_details', e)
raise e
sql = "select * from foo.bar"
log.start_job(mysql_dict, self.environment, self.instance + '_details')
try:
data = pd.read_sql(sql, conn)
except Exception as e:
log.error_job(mysql_dict, self.environment, self.instance + '_details', e)
raise e
with open(self.output().path, 'wb') as out:
data.to_pickle(out, compression=None)
Moving on to the Validate class:
class ValidateData(luigi.Task):
environment = luigi.Parameter(default='dev')
def requires(self):
return { 'data' : PullData(self.environment, self.instance) }
def output(self):
return luigi.LocalTarget(f'.out/{CUR_DATE}_{self.environment}_{self.instance}_AGSCS_ValidateData.txt')
def run(self):
with open(self.input()['data'].path, 'rb') as base_data:
data = pd.read_pickle(base_data, compression=None)
try:
assert len(data) > 0, "SQL Pull contains no data"
except Exception as e:
log.complete_job(dict(environment=self.environment, role='user', instance=self.instance), self.environment, self.instance + '_details', e)
raise e
#### HERE IS WHERE I AM LOOKING TO ADD ADDITIONAL VALIDATIONS ####
#### HERE IS WHERE I AM LOOKING TO ADD ADDITIONAL VALIDATIONS ####
#### HERE IS WHERE I AM LOOKING TO ADD ADDITIONAL VALIDATIONS ####
with self.output().open('w') as out:
summary = f"""Validation Completed Successfully with {len(data)} records."""
out.write(summary)
Basically, I would like to know how to tell the ValidateData class that IF the .PKL file that PullData generates is already there, don't run it again and just proceed with the validation (or, tell PullData that if the PKL file already exists, don't attempt to re-pull it. Either is the same to me)

Navigating through pages using selenium python

def get_link(search_term,page):
try:
Pinbox = driver.find_element_by_xpath('//*[#id="container"]/div/div[1]/div[1]/div[2]/div[3]/div[2]/div/div[2]/div[2]/div/div/div[1]/form/input')
Pinbox.send_keys(Pincode)
Pinbox.submit()
except exceptions.NoSuchElementException:
return
try:
grocery="https://www.flipkart.com/search?q={}&otracker=search&otracker1=search&marketplace=GROCERY&as-show=on&as=off"
search_term = search_term.replace(' ', '+')
stem = grocery.format(search_term)
url_template = stem + '&as-pos=1&as-type=HISTORY&as-backfill=on&page='
next=url_template+str(page)
print(next)
if page > 1:
return next
else:
return Pinbox and stem
except exceptions.NoSuchElementException:
return
def PID():
for page in range(1,20):
path=get_link(term,page)
driver.get(path)
id=driver.find_elements_by_xpath('//div[#data-id]')
for i in id:
results=i.get_attribute('data-id')
print(results)
FSN_List.append(results)
PID()
unable to navigate through pages,get an error selenium.common.exceptions.InvalidArgumentException: Message: invalid argument: 'url' must be a string

Exception appears, because you return None, from get_link function. In get_link function first step is finding element, but you have not moved to the site yet:
try:
Pinbox = driver.find_element_by_xpath('//*[#id="container"]/div/div[1]/div[1]/div[2]/div[3]/div[2]/div/div[2]/div[2]/div/div/div[1]/form/input')
Pinbox.send_keys(Pincode)
Pinbox.submit()
except exceptions.NoSuchElementException:
return
you should first provide link to .get() driver funciton, thand then looking for elements

i have created a different function,
def pin(Pincode):
try:
Pinbox = driver.find_element_by_xpath('//*[#id="container"]/div/div[1]/div[1]/div[2]/div[3]/div[2]/div/div[2]/div[2]/div/div/div[1]/form/input')
Pinbox.send_keys(Pincode)
Pinbox.submit()
except exceptions.NoSuchElementException:
return
this resolved the issue

Keep getting this error not sure why "Can't convert 'set' object to str implicitly"

#admin.register(Book)
class BookAdmin(ImportExportActionModelAdmin):
resource_class = BookResource
def get_import_form(self):
return CustomImportForm
def get_resource_kwargs(self, request, *args, **kwargs):
rk = super().get_resource_kwargs(request, *args, **kwargs)
rk['input_author'] = None
if request.POST:
author = request.POST.get('input_author', None)
if author:
request.session['input_author'] = author
else:
try:
author = request.session['input_author']
except KeyError as e:
raise Exception("Context failure on row import" + {e})
rk['input_author'] = author
return rk
Have this code in django admin page, but getting an error during the export. Can anyone let me know where is the issue?

Your issue is on this line:
raise Exception("Context failure on row import" + {e})
The ‘{e}’ means that you create a set containing the error, and try to join it to the exception message string. You should be able to get rid of that error by replacing ‘{e}’ with just ‘e’.

Exception not raised assertRaises()?

I am giving a wrong input and I want an exception to be raised. Somehow this is not happening. This is my unit test code:
def test_invalid_bag_json_conversion_1(self):
file_name = "../test/test_files/wrong_bag.bag"
ru = RosbagUploader(file_name, self.json_output, "", "", "", "")
status = ru.start()
self.assertRaises(Exception, RosbagUploader, file_name, self.json_output, "", "", "", "")
self.assertEquals(ReturnCodes.FAIL, status)
and my code that I am testing:
class RosbagUploader(object):
"""
#brief Uploads deserialized input Rosbag file to ElasticSearch or
stores it locally
"""
def __init__(self, _rosbag_filepath, _json_filename, _es_addr, _es_index,
_es_type, _buffer_size):
self.overall_status = ReturnCodes.SUCCESS
self._rosbag_filepath = _rosbag_filepath
self._json_filename = _json_filename
self._buffer_size = _buffer_size if _buffer_size > 0 else 5000
self._es_type = _es_type
self._es_addr = _es_addr
self._es_index = _es_index
self._es_buff = []
try:
self._rosbag = rosbag.Bag(_rosbag_filepath, "r")
if self._es_addr:
self._es = Elasticsearch() if _es_addr == "" else \
Elasticsearch([_es_addr])
self._total_num_record = self._rosbag.get_message_count()
except:
print("[ERROR] {}".format(sys.exc_info()))
self.overall_status = ReturnCodes.FAIL
It shows the output that the exception is raised as below:
[ERROR] (<class 'rosbag.bag.ROSBagException'>, ROSBagException(), <traceback object at 0x7fdcb463e8c0>)
EException AttributeError: "'RosbagUploader' object has no attribute '_rosbag'" in <bound method RosbagUploader.__del__ of <rosbag_deserializer_core.RosbagUploader object at 0x7fdcb4899ad0>> ignored
Which is what it should do. But why doesnt it raise the exception?

You are essentially ignoring the exception since you are not re-raising after the print statement:
try:
[...]
except:
print("[ERROR] {}".format(sys.exc_info()))
self.overall_status = ReturnCodes.FAIL
You need to re-raise for callers to receive the exception:
try:
[...]
except:
print("[ERROR] {}".format(sys.exc_info()))
self.overall_status = ReturnCodes.FAIL
raise

Do you have a __del__ destruction method? Seems to be failing there.

Pickleing error: connot pickle Request object

I know That it is not possible to pickle a pyramid request object, but I cant seem to find where I am sending the Request object.
Consider the following:
#task
def do_consignment_task(store, agent):
print "GOTHERE IN TASK"
s = sqlahelper.get_session()
consign = store.gen_consignment()
ca = Agents.by_id(store.consignment_agents_id)
consign.consignment_agents_id = ca.id
consign.consignment_teamleader_id = ca.ou[0].lead_agents_id
consign.consignment_timestamp = func.now()
consign.created_by_agent_id = agent.id
consign.complete_stamp = func.now()
consign.sims = store.sims
consign.status = "SUCCESS"
print "GOT BEFORE LOOP "
for sim in store.sims:
if sim in consign.sims:
continue
else:
consign.sims.append(sim)
s.add(consign)
transaction.savepoint()
print "GOT AFTER SAVEPOINT"
for sim in consign.sims:
is_reconsign = sim.consignment_agent or sim.consignment_teamlead
if is_reconsign:
if not sim.consignment_history:
sim.consignment_history = []
sim.consignment_history.append(dict(
stamp=sim.consignment_timestamp,
consignment_agent_id=sim.consignment_agents_id,
consignment_teamleader_id=sim.consignment_teamleader_id,
by_agent_id=agent.id
))
s.query(
Sims
).filter(
Sims.iccid == sim.iccid
).update(
{
"consignment_agents_id": consign.consignment_agents_id,
"consignment_history": sim.consignment_history,
"consignment_teamleader_id": ca.ou[0].lead_agents_id,
"consignment_timestamp": func.now(),
"modify_stamp": func.now(),
"consignments_id": consign.id
},
synchronize_session=False
)
print "GOT BEFORE COMMIT"
transaction.savepoint()
print "THIS IS THE ID ID ID ID ID ID : ", consign.id
I call this function like:
if self.store.finalise:
try:
store = self.store
agent = self.agent
do_consignment_task.delay(store, agent)
transaction.commit()
self.check_and_purge()
return "Consignmnet is being processed"
except Exception, exc:
self.check_and_purge()
self.log.exception(exc)
exc_error = "CONSIGNERR:", exc.message
raise USSDFailure(exc_error)
else:
self.store.status = "CANCELLED"
if "fullconfirm" in self.session:
del self.session["fullconfirm"]
self.check_and_purge()
return "CONSIGNMENT Cancelled"
When I run this code I get the following error:
EncodeError: Can't pickle <class 'pyramid.util.Request'>: attribute lookup pyramid.util.Request failed
I am not sending self or request objects - at least not that I can see.
How can solve this problem? Am I sending a request object, because I can not see one?
The traceback can be seen here
EDIT:
okay So I have tried to change the data I send to the function - I am not passing a sqlalchemy object and I am making a copy of the store object, that changes my code to:
#task
def do_consignment_task(agent_id, **store):
print "GOTHERE IN TASK"
s = sqlahelper.get_session()
cObj = USSDConsignmentsObject()
consign = cObj.gen_consignment()
ca = Agents.by_id(store.consignment_agents_id)
consign.consignment_agents_id = ca.id
consign.consignment_teamleader_id = ca.ou[0].lead_agents_id
consign.consignment_timestamp = func.now()
consign.created_by_agent_id = agent_id
# etc
and:
if self.store.finalise:
try:
# del self.service
store = self.store.__dict__.copy()
agent_id = self.agent.id
print store
print agent_id
# print help(store)
do_consignment_task.delay(agent_id, **store)
transaction.commit()
#etc
This however still gives me the same error :|

Try not to serialise a Pyramid request object. When you interact with a celery task you should think of it as an independent process.
Provide it all the information it needs to do it's work. Be aware that you need to serialise that information.
So self.store possibly contains attribute references that may be unrealistic to serialise.
Perhaps create a method on the store object that returns a clean dictionary object.
def serialize(self):
data = {}
data["element1"] = self.element1
data["element2"] = self.element2
data["element3"] = self.element3
return data
Then when you want to call the delay method make sure to use store.serialize() instead of store or the dict.