This is difficult to describe or show much code for, but I'll try. Essentially I have a multi-threaded desktop app that will frequently handle the adding/removing/changing of tables in threads. From what I read, I should use scoped_session and pass that around to the various threads to do the work (I think?). Here're some basic code examples:
class SQL():
def __init__(self):
self.db = create_engine('mysql+mysqldb://thesqlserver')
self.metadata = MetaData(self.db)
self.SessionObj = scoped_session(sessionmaker(bind=self.db, autoflush=True))
db = SQL()
session = db.SessionObj()
someObj = Obj(val, val2)
session.add(someObj)
session.commit()
The above class is what I'm using as the general access of SQL stuff. After creating a new session, performing a query and update/add to it, upon the session.commit(), I get the following error:
Traceback (most recent call last):
File "core\taskHandler.pyc", line 42, in run
File "core\taskHandler.pyc", line 184, in addTasks
File "core\sqlHandler.pyc", line 35, in commit
File "sqlalchemy\orm\session.pyc", line 624, in rollback
File "sqlalchemy\orm\session.pyc", line 338, in rollback
File "sqlalchemy\orm\session.pyc", line 369, in _rollback_impl
File "sqlalchemy\orm\session.pyc", line 239, in _restore_snapshot
File "sqlalchemy\orm\state.pyc", line 252, in expire
AttributeError: 'NoneType' object has no attribute 'expire'
Then the next if another sql attempt goes through:
Traceback (most recent call last):
File "core\taskHandler.pyc", line 44, in run
File "core\taskHandler.pyc", line 196, in deleteTasks
File "sqlalchemy\orm\query.pyc", line 2164, in scalar
File "sqlalchemy\orm\query.pyc", line 2133, in one
File "sqlalchemy\orm\query.pyc", line 2176, in __iter__
File "sqlalchemy\orm\query.pyc", line 2189, in _execute_and_instances
File "sqlalchemy\orm\query.pyc", line 2180, in _connection_from_session
File "sqlalchemy\orm\session.pyc", line 729, in connection
File "sqlalchemy\orm\session.pyc", line 733, in _connection_for_bind
File "sqlalchemy\orm\session.pyc", line 249, in _connection_for_bind
File "sqlalchemy\orm\session.pyc", line 177, in _assert_is_active
sqlalchemy.exc.InvalidRequestError: This Session's transaction has been rolled back by a nested rollback() call. To begin a new transaction, issue Session.rollback() first.
That's about as much as I know and I think the best I can describe. Any ideas on what I'm supposed to be doing here? It's all mud to me. Thanks in advance!
The funny part is, you missed the most critical part of the answer you "ripped the code from", which is that there is a Python function in the middle, which is executing some abstract operation (it's labeled as func()). That code illustrates a transactional wrapper for a function, and in the above example you instead have an object method called commit() that isn't otherwise calling upon any additional operations with the Session.
Here you have kind of a session-holding object called SQL() that is not really adding any usefulness to your program and makes it needlessly complicated, and is probably also the source of the issue. Unless your application intends to connect to many different databases at different times, and use SQL() objects to represent that state, there's not much use in building a class called "SQL" that has an "engine" stuck onto it. Just stick the engine in a module somewhere, as well as your scoped_session().
The engine and scoped_session represent a pattern called the factory pattern - they are objects that create some other useful object, in this case scoped_session creates a Session, and the Engine is used internally by the Session to create a Connection with which to talk to the database. It doesn't make much sense to place the Session object as a sibling member along with Engine and scoped_session - you'd be carrying around either the factories (the Engine and scoped_session), or the object itself that they create (the Session), which all depends on what you're trying to do.
The Session itself, remember here we're talking about the thing the factories create (Session), not the factories themselves (Engine and scoped_session), is not in the least bit thread safe. It is something you usually create only local to a function - it shouldn't be global, and if you're in fact using a single SQL() object across threads that's probably the problem here. The actual error you're getting, I'm not really sure what that is and I could only have a better clue if I knew the exact version of SQLAlchemy in use here, though the randomness of the error suggests that you have some kind of threading issue where something is becoming None in one thread as another expects that same object to be present.
So what you need to establish in this program is when exactly a particular thread of execution begins, what it needs to do with the database as it proceeds, and then when it ends. When you can establish a consistent pattern for that, you would then link a single Session to this thread, which goes for the lifespan of that thread, and is never shared. All the objects which are produced by this session must also not be shared to other threads - they are extensions of the Session's state. If you have "worker threads" in use, those worker threads should load up their own data as needed, within their own Session. The Session represents a live database transaction and you generally want transactions local to a single thread.
As this is not a web application you might want to forego the usage of scoped_session, unless you do in fact have a place for a thread-local pattern to be used.
Related
This is my first post here. I'm building a Python window application with PyQt5 that implements interactions with the OpenAI completions endpoint. So far, any code that I've written myself has performed fine, and I was reaching the point where I wanted to start implementing long-term memory for conversational interactions. I started by just running my own chain of prompts for categorizing and writing topical subjects and summaries to text files, but I decided it best to try exploring open source options to see how the programming community is managing things. This led me to LangChain, which seems to have some popular support behind it and already implements many features that I intend.
However, I have not had even the tiniest bit of success with it yet. Even the most simple examples don't perform, regardless of what context I'm implementing it in (within a class, outside a class, in an asynchronous loop, to the console, to my text browsers within the main window, whatever) I always get the same error message.
The simplest possible example:
import os
from langchain.llms import OpenAI
from local import constants #For API key
os.environ["OPENAI_API_KEY"] = constants.OPENAI_API_KEY
davinci = OpenAI(model_name= 'text-davinci-003', verbose=True, temperature=0.6)
text = "Write me a story about a guy who is frustrated with Python."
print("Prompt: " + text)
print(davinci(text))
It capably instantiates the wrapper and prints the prompt to the console, but at any point a command is sent through the wrapper's functions to receive generated text, it encounters this AttributeError.
Here is the traceback:
Traceback (most recent call last):
File "D:\Dropbox\Pycharm Projects\workspace\main.py", line 16, in <module>
print(davinci(text))
File "D:\Dropbox\Pycharm Projects\workspace\venv\lib\site-packages\langchain\llms\base.py", line 255, in __call__
return self.generate([prompt], stop=stop).generations[0][0].text
File "D:\Dropbox\Pycharm Projects\workspace\venv\lib\site-packages\langchain\llms\base.py", line 128, in generate
raise e
File "D:\Dropbox\Pycharm Projects\workspace\venv\lib\site-packages\langchain\llms\base.py", line 125, in generate
output = self._generate(prompts, stop=stop)
File "D:\Dropbox\Pycharm Projects\workspace\venv\lib\site-packages\langchain\llms\openai.py", line 259, in _generate
response = self.completion_with_retry(prompt=_prompts, **params)
File "D:\Dropbox\Pycharm Projects\workspace\venv\lib\site-packages\langchain\llms\openai.py", line 200, in completion_with_retry
retry_decorator = self._create_retry_decorator()
File "D:\Dropbox\Pycharm Projects\workspace\venv\lib\site-packages\langchain\llms\openai.py", line 189, in _create_retry_decorator
retry_if_exception_type(openai.error.Timeout)
AttributeError: module 'openai.error' has no attribute 'Timeout'
I don't expect that there is a fault in the LangChain library, because it seems like nobody else has experienced this problem. I imagine I may have some dependency issue? Or I do notice that others using the LangChain library are doing so in a notebook development environment, and my lack of familiarity in that regard is making me overlook some fundamental expectation of the library's use?
Any advice is welcome! Thanks!
What I tried: I initially just replaced my own function for managing calls to the completion endpoint with one that issued the calls through LangChain's llm wrapper. I expected it to work as easily as my own code had, but I received that error. I then stripped everything apart layer by layer attempting to instantiate the wrapper at every scope of the program, then I attempted to make the calls in an asynchronous function through a loop that waited to completion, and no matter what, I always get that same error message.
I think it might be something about your current installed versions of Python, OpenAI, and/or LangChain. Maybe try using a newer version of Python and OpenAI. I'm new to Python and these things but hopefully I could help.
I'm having issues communicating with the Anritsu MG69377B microwave generators.
I have no problem going into a remote mode and identifying the instrument, but any other command which returns a value is timed out and the instrument does seem to respond to any write command. I'm using PyVISA to communicate to the instrument.
>>>import visa
>>> x = visa.connect('GPIB::5') #Address the instrument
>>> x.ask('*IDN?') # Works Fine
>>>'ANRITSU,69377B,010918,2.35'
>>>
>>> x.ask('*STB?')
Traceback (most recent call last):
File "<pyshell#37>", line 1, in <module>
x.ask('*STB?')
File "C:\Python27\lib\site-packages\pyvisa\visa.py", line 502, in ask
return self.read()
File "C:\Python27\lib\site-packages\pyvisa\visa.py", line 433, in read
return self._strip_term_chars(self.read_raw())
File "C:\Python27\lib\site-packages\pyvisa\visa.py", line 407, in read_raw
chunk = vpp43.read(self.vi, self.chunk_size)
File "C:\Python27\lib\site-packages\pyvisa\vpp43.py", line 840, in read
visa_library().viRead(vi, buffer, count, byref(return_count))
File "C:\Python27\lib\site-packages\pyvisa\vpp43.py", line 398, in check_status
raise visa_exceptions.VisaIOError, status
VisaIOError: VI_ERROR_TMO: Timeout expired before operation completed.
*STB? is a standard IEEE command and should work across any instrument which follows the standard (which is basically all). I've looked in the manual and I can't seem to find anything wrong with syntax for other code as well. The instrument just does not seem to work when connected to remotely with PyVISA. (Unsure if this is a PyVISA problem or an issue with my instrument).
The 693xxB supports two languages, referred to as Native and SCPI. In that instrument the SCPI language was an option (option 19). In a 693xxB with option 19 the language is an instrument setting.
Native mode only supports the common command *IDN?.
SCPI mode supports more common commands, like *STB? for example.
Your syntax and VISA connection seem fine. It's on the instrument setting/support side.
I've been developing a django app with south for a while, and doing a sort of loose continuous deployment. Very shortly after my initial migration, I did a couple data migrations that looked like this:
def forwards(self, orm):
from django.core.management import call_command
call_command("loaddata", "#######.json")
At the time, I didn't think anything of it. It had been easy enough to populate the database manually and then dump it all into a fixture. Then when I finally wrote some unit tests, I started getting errors like this:
Creating test database for alias 'default'...
Problem installing fixture '/home/axel/Workspace/02_ereader_blast/content/fixtures/99_deals.json': Traceback (most recent call last):
File "/home/axel/Workspace/02_ereader_blast/venv/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 196, in handle
obj.save(using=using)
File "/home/axel/Workspace/02_ereader_blast/venv/local/lib/python2.7/site-packages/django/core/serializers/base.py", line 165, in save
models.Model.save_base(self.object, using=using, raw=True)
File "/home/axel/Workspace/02_ereader_blast/venv/local/lib/python2.7/site-packages/django/db/models/base.py", line 551, in save_base
result = manager._insert([self], fields=fields, return_id=update_pk, using=using, raw=raw)
File "/home/axel/Workspace/02_ereader_blast/venv/local/lib/python2.7/site-packages/django/db/models/manager.py", line 203, in _insert
return insert_query(self.model, objs, fields, **kwargs)
File "/home/axel/Workspace/02_ereader_blast/venv/local/lib/python2.7/site-packages/django/db/models/query.py", line 1593, in insert_query
return query.get_compiler(using=using).execute_sql(return_id)
File "/home/axel/Workspace/02_ereader_blast/venv/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 910, in execute_sql
cursor.execute(sql, params)
File "/home/axel/Workspace/02_ereader_blast/venv/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 52, in execute
return self.cursor.execute(query, args)
DatabaseError: Could not load content.BookDeal(pk=1): column "entry_id" of relation "content_bookdeal" does not exist
LINE 1: INSERT INTO "content_bookdeal" ("id", "book_id", "entry_id",...
^
Installed 19 object(s) from 1 fixture(s)
Problem installing fixture '/home/axel/Workspace/02_ereader_blast/content/fixtures/99_deals_entries.json': Traceback (most recent call last):
File "/home/axel/Workspace/02_ereader_blast/venv/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 190, in handle
for obj in objects:
File "/home/axel/Workspace/02_ereader_blast/venv/local/lib/python2.7/site-packages/django/core/serializers/json.py", line 47, in Deserializer
raise DeserializationError(e)
DeserializationError: Entry has no field named 'book_deals'
As far as I can tell, the loaddata command is using my most recent models, rather than the state of south at the time, and because I have changed them significantly since then, the current models are interpreting the old data as invalid.
So my questions are:
What is the best way to set up future data migrations so that this doesn't happen?
How can I backpedal out of this situation, and take it to best-practices land?
I found the solution, courtesy of this stackoverflow question:
django loading data from fixture after backward migration / loaddata is using model schema not database schema
As noted in the top answer, I used a snippet that very elegantly patches where the loaddata command gets its model.
Note that I did have to broaden my freezes for those data migrations, so that they could access all the models they needed from the orm rather than directly.
This feels like the right way to solve the problem.
My approach is a highly likely to be a complete hack / abuse of south / worst kind of practice, I think. However... if you know that your django models conform to your data tables. I might go for a fresh start approach:
rename (or delete) the relevant migrations folders.
in the south_migrationhistory data table, remove all the associated entries for the apps you are trying to create a fresh start for.
python manage.py convert_to_south app-name.
everything should be good to go.
If the data tables are canonical, and your django models are out of line, the way I conform my django models to my data tables is to run:
python manage.py inspectdb > inspectdb.py
Now I can compare the two versions of the django code for my models to get them in alignment. This enables me to go through the fresh start sequence.
For a current assignment, I must import 2 .txt files into a MySQLdb with python. I'm having immense trouble. I have tried various methods and I simply can't do it.
I've searched through this site and many others over the past few days and I simply cannot get this to work. Whenever I've tried to adapt another person's solution to my own code, it fails - so I figure I should ask for help directly for my own code.
This is what I have so far:
import MySQLdb
# connect to database
mydb = MySQLdb.connect("localhost","root","0dy5seuS","cars_db")
# define the function
def data_entry(cars_for_sale):
# cursor creation
cursor = mydb.cursor()
#load the file 'cars_for_sale.txt' into the database under the table 'cars_for_sale'
sql = """LOAD DATA LOCAL INFILE 'cars_for_sale.TXT'
INTO TABLE cars_for_sale
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\r\n'"""
#execute the sql function above
cursor.execute(sql)
#commit to the database
mydb.commit()
#call data_entry(cars_for_sale) function
data_entry(cars_for_sale)
mydb.close()
I can hardly wrap my head around it, any help would be appreciated it.
I now get the following feedback from the testing function:
Trying:
data_entry("cars_for_sale") Expecting:
The number of rows inserted to cars_for_sale is 7049
**************************************** File "main", line 4, in main Failed example:
data_entry("cars_for_sale") Exception raised:
Traceback (most recent call last):
File "C:\Python27\lib\doctest.py", line 1289, in __run
compileflags, 1) in test.globs
File "", line 1, in
data_entry("cars_for_sale")
File "E:/Uni/104/Portfolio 2/MediumTask_DataStatistics/question/TEST2_data_statistics.py", line 270, in data_entry
data_entry(cars_for_sale) *it repeats this last portion several hundred/thousand times"
The following few lines are after the repeated error above.
File "C:\Python27\lib\site-packages\MySQLdb\connections.py", line
243, in cursor return (cursorclass or self.cursorclass)(self)
File "C:\Python27\lib\site-packages\MySQLdb\cursors.py", line 51, in
init from weakref import proxy RuntimeError: maximum recursion depth exceeded while calling a Python object
I'm aware that this is an infinite recursion although I have no idea how to stop it.
Thanks
The following code reproduces your error "RuntimeError: maximum recursion depth exceeded while calling a Python object":
def data_entry(cars_for_sale):
data_entry(cars_for_sale)
You don't need recursion here (and it is used incorrectly anyway).
I'm aware that this is an infinite recursion although I have no idea how to stop it.
Just remove the data_entry call inside the data_entry function.
My application allows users to create and delete Site objects. I have implemented this using session.add() and session.delete(). I then have 'Save' and 'Reset' buttons that call session.commit() and session.rollback().
If I add a new Site, then save/commit it, and then delete it, everything goes OK. However, if I try to remove an object from the session before it's been saved, I get a 'not persisted' error.
Code:
self.newSite = Site('foo')
self.session.add(self.newSite)
print self.session.new
self.session.delete(self.newSite)
Output:
IdentitySet([<Site('foo')>])
Traceback (most recent call last):
File "C:\Program Files\Eclipse\dropins\plugins\org.python.pydev.debug_2.2.1.2011071313\pysrc\pydevd_comm.py", line 744, in doIt
result = pydevd_vars.evaluateExpression(self.thread_id, self.frame_id, self.expression, self.doExec)
File "C:\Program Files\Eclipse\dropins\plugins\org.python.pydev.debug_2.2.1.2011071313\pysrc\pydevd_vars.py", line 375, in evaluateExpression
result = eval(compiled, updated_globals, frame.f_locals)
File "<string>", line 1, in <module>
File "C:\Python27\Lib\site-packages\sqlalchemy\orm\session.py", line 1245, in delete
mapperutil.state_str(state))
InvalidRequestError: Instance '<Site at 0x1ed5fb0>' is not persisted
I understand what's happening here, but I'm not sure what I should be doing instead.
Is there some other method of removing a not-yet-persisted object from a session? Or should I be calling session.flush() before attempting a deletion, in case the object I want to delete hasn't been flushed yet?
If it's the latter, how come session.query() auto-flushes (ensuring that pending objects show up in the query results), but session.delete() doesn't (which would ensure that pending objects can be deleted without error).
You can Session.expunge() it. I think the rationale with delete() being that way is, it worries you're not keeping track of things if you send it a pending. But I can see the other side of the story on that, I'll think about it. Basically the state implied by delete() includes some assumptions of persistence but they're probably not as significant as I'm thinking. An "expunge or delete" method then comes to mind, which is funny that's basically the "save or update" we originally copied from Hibernate, which just became "add". "add" can do the transitions of transient->pending as well as detached->persistent - would a potential "remove()" do both pending->transient and persistent->deleted ? too bad the scoped session already has "remove()"....
Session.query() autoflushes because it's about to go out to the database to emit some SQL to get some rows; so whatever you have locally needs to go out first. delete() just marks the state of an object so there's no need to invoke any SQL. If we wanted delete() to work on a pending, we'd just change that assertion.
Interestingly, if you rollback() the session, whatever you've add()'ed within that session, whether or not it got flushed, is expunged.