We Keep Coding

apscheduler: returned more than one DjangoJobExecution -- it returned 2

In my proyect scheduler return this error in the execute job, help me please
this is my error in cosole, then execute the program
Error notifying listener
Traceback (most recent call last):
File "C:\Users\angel\project\venv\lib\site-packages\apscheduler\schedulers\base.py", line 836, in _dispatch_event
cb(event)
File "C:\Users\angel\project\venv\lib\site-packages\django_apscheduler\jobstores.py", line 53, in handle_submission_event
DjangoJobExecution.SENT,
File "C:\Users\angel\project\venv\lib\site-packages\django_apscheduler\models.py", line 157, in atomic_update_or_create
job_id=job_id, run_time=run_time
File "C:\Users\angel\project\venv\lib\site-packages\django\db\models\query.py", line 412, in get
(self.model._meta.object_name, num)
django_apscheduler.models.DjangoJobExecution.MultipleObjectsReturned: get() returned more than one DjangoJobExecution -- it returned 2!
This is my code
class Command(BaseCommand):
help = "Runs apscheduler."
scheduler = BackgroundScheduler(timezone=settings.TIME_ZONE, daemon=True)
scheduler.add_jobstore(DjangoJobStore(), "default")
def handle(self, *args, **options):
self.scheduler.add_job(
delete_old_job_executions,
'interval', seconds=5,
id="delete_old_job_executions",
max_instances=1,
replace_existing=True
)
try:
logger.info("Starting scheduler...")
self.scheduler.start()
except KeyboardInterrupt:
logger.info("Stopping scheduler...")
self.scheduler.shutdown()
logger.info("Scheduler shut down successfully!")

Not sure if you're still having this issue. I have same error and found your question. Turned out this happens only in dev environment.
Because python3 manage.py runserver starts two processes by default, the code
seems to register two job records and find two entries at next run time.
With --noreload option, it starts only one scheduler thread and works well. As name implies, it won't reload changes you make automatically though.
python3 manage.py runserver --noreload

not sure if you're still having this issue. i think you can use socket , socket can use this issue.
look this enter image description here

Running asynchronous functions throws an error

Screenshot I am trying to write a script that runs a .exe file using a silent installer with python. My main problem is that when the installer finishes it request a user input - see screenshot attached and the script will not finish until the user hit any key to start the services
Therefore, I wrote a script that has two functions running asynchronous
As expected I am having errors
My expectations: Function click_window will wait for function run_installer to finish and then it will click enter in the cmd.exe window - I will add a screenshot
What is happening: The Script runs as expected but when it gets to os.system = (file + '/SILENT') base on the debugger - it dies out with an error message
import asyncio
import subprocess
import time
import pyautogui
import pywinauto
from pyautogui import getWindowsWithTitle
import pygetwindow
async def click_window():
# Wait for everything to load in, windows, apps etc.
# You may need to adjust this timing for your system.
time.sleep(10)
# We are waiting for the window to open up and then click ok
app = pywinauto.Application()
# Tell Pyautogui what the windows name is.
Win = "C:\\WINDOWS\\System32\\cmd.exe"
# wait for the window to appear
app = pywinauto.Application()
app.WindowSpecification.wait(Win)
app.window_(title=Win).SetFocus()
app.window_(title=Win).TypeKeys("Enter")
# keyboard shortcut R to hit 'Run Cleaner' button,
pyautogui.press('Enter')
# Wait 15 seconds to give time for the cleaning operation.
# You may need to adjust.
time.sleep(15)
# Move mouse to exit button and click.
# pyautogui.moveTo(1905, 10, 1)
pyautogui.click()
print("We did it")
async def run_installer():
import fnmatch
import glob
import os
import shutil
import time
import urllib.request
import zipfile
# Download the installer
url = 'http://build.****.com:8080/job/CI_build_***_grs_master_windows_mysql/lastBuild/artifact/com.****.sdm.****/target/release/*zip*/release.zip'
file_name = 'release.zip'
with urllib.request.urlopen(url) as response, open(file_name, 'wb') as out_file:
shutil.copyfileobj(response, out_file)
time.sleep(120)
print("Download is done")
file_name = "release.zip"
with zipfile.ZipFile(file_name, 'r') as zip:
# printing all the contents of the zip file
zip.printdir()
time.sleep(60)
# extracting all the files
print('Extracting all the files now...')
zip.extractall()
print('I am done extracting Done!')
print("Path at terminal when executing this file")
print(os.getcwd() + "\n")
os.chdir("C:\\Users\\****\\PycharmProjects\\Phyton-Project\\release")
print("Path at terminal when executing this file again")
print(os.getcwd() + "\n")
# Checking installer directory
for file in glob.glob("*.exe"):
print(file + " Here is the file found")
# file variable contains installer
time.sleep(120)
# Adding permissions to installer to run/execute
os.chmod("C:\\Users\\****\\PycharmProjects\\Phyton-Project\\release", 0o777) # read/write by everyone
for file in os.listdir('.'):
if fnmatch.fnmatch(file, '*.exe'):
print("here is the file" + file)
# file variable is the exe installer
time.sleep(120)
print("Installation starts")
os.system = (file + '/SILENT')
# time.sleep(360)
start = time.time()
loop = asyncio.get_event_loop()
tasks = [
asyncio.gather(run_installer()),
asyncio.gather(click_window()),
]
loop.run_until_complete(asyncio.wait(tasks))
loop.close()
end = time.time()
print("Total time: {}".format(end - start))
Expected Results:
Function run installer finish and the window opens up to start up the services and then the click_window function starts up and clicks on the window and after 4 clicks for every service the script finish
Actual Results:
Function run installer runs but it dies at os.system = (file + '/SILENT') because the installer never installs and then it throws the following error message:
Connected to pydev debugger (build 192.5728.105)
I am done waiting
File Name Modified Size
release/pa****_**_5.4.3_20190903_win_x64_db.exe 2019-09-03 22:29:42 757267728
Extracting all the files now...
Done!
Path at terminal when executing this file
C:\Users\j****\PycharmProjects\Phyton-Project
Path at terminal when executing this file again
C:\Users\j*****\PycharmProjects\Phyton-Project\release
pa***_***_5.4.3_20190903_win_x64_db.exe Here is the file found
here is the fileparasoft_dtp_5.4.3_20190903_win_x64_db.exe
Installation starts
Total time: 485.42616963386536
_GatheringFuture exception was never retrieved
future: <_GatheringFuture finished exception=AppNotConnected('Please use start or connect before trying anything else')>
Traceback (most recent call last):
File "C:/Users/****/PycharmProjects/Phyton-Project/DTP-Installation.py", line 27, in click_window
app.WindowSpecification.wait(Win)
File "C:\Users\****\PycharmProjects\Phyton-Project\venv\lib\site-packages\pywinauto\application.py", line 1234, in __getattribute__
return self[attr_name]
File "C:\Users\****\PycharmProjects\Phyton-Project\venv\lib\site-packages\pywinauto\application.py", line 1220, in __getitem__
return self.window(best_match=key)
File "C:\Users\*****\PycharmProjects\Phyton-Project\venv\lib\site-packages\pywinauto\application.py", line 1207, in window
raise AppNotConnected("Please use start or connect before trying "
pywinauto.application.AppNotConnected: Please use start or connect before trying anything else
Process finished with exit code

ok I decided to create a new function to click on the window - so far I can find the window, set the focus but I can't click enter or any key
import win32con
import win32gui
import pyautogui as pg
from pywinauto.win32structures import HANDLE
toplist = []
winlist = []
def enum_callback(hwnd, results): winlist.append((hwnd, win32gui.GetWindowText(hwnd)))
win32gui.EnumWindows(enum_callback, toplist)
services = [(hwnd, title) for hwnd, title in winlist if 'system32' in title.lower()]
# just grab the first window that matches
services = services[0]
# use the window handle to set focus
win32gui.SetForegroundWindow(services[0])
win32gui.GetForegroundWindow()
# print windows title
print("Windows Title")
print(services)
print('FOUND')
win32gui.EnableWindow(1, True)
win32api.SendMessage(services, win32con.WM_KEYDOWN, win32con.VK_F11, 0)
print('Window Close')
Response:
C:\Users\jromero\PycharmProjects\Phyton-Project\venv\Scripts\python.exe C:/Users/jromero/PycharmProjects/Phyton-Project/click-enter.py
Windows Title
Traceback (most recent call last):
(334514, 'C:\\WINDOWS\\system32\\cmd.exe')
File "C:/Users/jromero/PycharmProjects/Phyton-Project/click-enter.py", line 32, in <module>
FOUND
win32api.SendMessage(services, win32con.WM_KEYDOWN, win32con.VK_F11, 0)
TypeError: The object is not a PyHANDLE object
Process finished with exit code 1

Perforce (P4) Python API complains about too many locks

I wrote an application that opens several subprocesses, which initiate connections individually to a Perforce server. After a while I get this error message in almost all of these child-processes:
Traceback (most recent call last):
File "/Users/peter/Desktop/test_app/main.py", line 76, in p4_execute
p4.run_login()
File "/usr/local/lib/python3.7/site-packages/P4.py", line 665, in run_login
return self.run("login", *args, **kargs)
File "/usr/local/lib/python3.7/site-packages/P4.py", line 611, in run
raise e
File "/usr/local/lib/python3.7/site-packages/P4.py", line 605, in run
result = P4API.P4Adapter.run(self, *flatArgs)
P4.P4Exception: [P4#run] Errors during command execution( "p4 login" )
[Error]: "Fatal client error; disconnecting!
Operation 'client-SetPassword' failed.
Too many trys to get lock /Users/peter/.p4tickets.lck."
Does anyone have any idea what could cause this? I open my connections properly and all double checked on all source locations that I disconnect from the server properly via disconnect.
Only deleting the .p4tickets.lck manually works until the error comes back after a few seconds

The relevant code is here:
https://swarm.workshop.perforce.com/projects/perforce_software-p4/files/2018-1/support/ticket.cc#200
https://swarm.workshop.perforce.com/projects/perforce_software-p4/files/2018-1/sys/filetmp.cc#147
I can't see that there's any code path where the ticket.lck file would fail to get cleaned up without throwing some other error.
Is there anything unusual about the home directory where the tickets file lives? Like, say, it's on a network filer with some latency and some kind of backup process? Or maybe one that doesn't properly enforce file locks between all these subprocesses you're spawning?
How often are your scripts running "p4 login" to refresh and re-write the ticket? Many times a second? If you change them to not do that (e.g. only login if there's not already a ticket) does the problem persist?

Sequentially running an external independent process using Tkinter and python

BACKGROUND :
*I'm creating a batch simulation job chooser + scheduler using Tkinter (Portable PYscripter, python v2.7.3)
*This program will function as a front end, to a commercial solver program
*The program needs to allow the user to choose a bunch of files to simulate, sequentially, one after the other.
*It also needs to have the facility to modify (Add/delete) jobs from an existing/running job list.
*Each simulation will definitely run for several hours.
*The output of the simulation will be viewed on separate programs and I do not need any pipe to the output. The external viewer will be called from the GUI, when desired.
***I have a main GUI window, which allows the user to :
choose job files, submit jobs, view the submission log, stop running jobs(one by one)
The above works well.
PROBLEMS :
*If I use subprocess.Popen("command") : all the simulation input files are launched at the same time. It MUST be sequential (due to license and memory limitations)
*If I use subprocess.call(" ") or the wait() method, then the GUI hangs and there is no scope to stop/add/modify the job list. Even if the "job submit" command is on an independent window, both the parent windows hang untill the job completes.
QUESTION 1 :
*How do I launch the simulation jobs sequentially (like subprocess.call) AND allow the main GUI window to function for the purpose of job list modification or stopping a job ?
The jobs are in a list, taken using "askopenfilenames" and then run using a For loop.
Relevant parts of the Code :
cfx5solvepath=r"c:\XXXX"
def file_chooser_default():
global flist1
flist1=askopenfilename(parent = root2, filetypes =[('.def', '*.def'),('All', '*.*'),('.res', '*.res')], title ="Select Simulation run files...", multiple = True)[1:-1].split('} {')
def ext_process():
o=list(flist1)
p=list(flist1)
q=list(flist1)
i=0
while i < len(flist1):
p[i]='"%s" -def "%s"'%(cfx5solvepath,flist1[i])
i+=1
i=0
while i < len(p):
q[i]=subprocess.call(p[i])
i+=1
root2 = Tk()
root2.minsize(300,300)
root2.geometry("500x300")
root2.title("NEW WINDOW")
frame21=Frame(root2, borderwidth=3, relief="solid").pack()
w21= Button(root2,fg="blue", text="Choose files to submit",command=file_chooser_default).pack()
w2a1=Button(root2,fg="white", text= 'Display chosen file names and order', command=lambda:print_var(flist1)).pack()
w2b1= Button (root2,fg="white", bg="red", text="S U B M I T", command=ext_process).pack()
root2.mainloop()
Please let me know if you require anything else. Look forward to your help.
*EDIT *
On incorporating the changes suggested by #Tim , the GUI is left free. Since there is a specific sub-program associated with the main solver program to stop the job, I am able to stop the job using the right command.
Once the currently running job is stopped, the next job on the list starts up, automatically, as I was hoping.
This is the code used for stopping the job :
def stop_select(): #Choose the currently running files which are to be stopped
global flist3
flist3=askdirectory().split('} {')
def sim_stop(): #STOP the chosen simulation
st=list(flist3)
os.chdir("%s"%flist3[0])
st= subprocess.call('"%s" -directory "%s"'%(defcfx5stoppath,flist3[0]))
ret1=tkMessageBox.showinfo("INFO","Chosen simulation stopped successfully")
os.chdir("%s" %currentwd)
QUESTION 2 :
*Once the above jobs are Completed, using start_new_thread, the GUI doesn't respond. The GUI works while the jobs are running in the background. But the start_new_thread documentation says that the thread is supposed to exit silently when the function returns.
*Additionally, I have a HTML log file that is written into/updated as each job completes. When I use start_new_thread ,the log file content is visible only AFTER all the jobs complete. The contents, along with the time stamps are however correct. Without using start_new_thread, I was able to refresh the HTML file to get the updated submission log.
***On exiting the GUI program using the Task manager several times, I am suddenly unable to use the start_new_thread function !! I have tried reinstalling PYscripter and restarting the computer as well. I can't figure out anything sensible from the traceback, which is:
Traceback (most recent call last):
File "<string>", line 532, in write
File "C:\Portable Python 2.7.3.1\App\lib\site-packages\rpyc\core\protocol.py", line 439, in _async_request
seq = self._send_request(handler, args)
File "C:\Portable Python 2.7.3.1\App\lib\site-packages\rpyc\core\protocol.py", line 229, in _send_request
self._send(consts.MSG_REQUEST, seq, (handler, self._box(args)))
File "C:\Portable Python 2.7.3.1\App\lib\site-packages\rpyc\core\protocol.py", line 244, in _box
if brine.dumpable(obj):
File "C:\Portable Python 2.7.3.1\App\lib\site-packages\rpyc\core\brine.py", line 369, in dumpable
return all(dumpable(item) for item in obj)
File "C:\Portable Python 2.7.3.1\App\lib\site-packages\rpyc\core\brine.py", line 369, in <genexpr>
return all(dumpable(item) for item in obj)
File "C:\Portable Python 2.7.3.1\App\lib\site-packages\rpyc\core\brine.py", line 369, in dumpable
return all(dumpable(item) for item in obj)
File "C:\Portable Python 2.7.3.1\App\lib\site-packages\rpyc\core\brine.py", line 369, in <genexpr>
return all(dumpable(item) for item in obj)
File "C:\Portable Python 2.7.3.1\App\Python_Working_folder\v350.py", line 138, in ext_process
q[i]=subprocess.call(p[i])
File "C:\Portable Python 2.7.3.1\App\lib\subprocess.py", line 493, in call
return Popen(*popenargs, **kwargs).wait()
File "C:\Portable Python 2.7.3.1\App\lib\subprocess.py", line 679, in __init__
errread, errwrite)
File "C:\Portable Python 2.7.3.1\App\lib\subprocess.py", line 896, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified

I'd suggest using a separate thread for the job launching. The simplest way would be to use the start_new_thread method from the thread module.
Change the submit button's command to command=lambda:thread.start_new_thread(ext_process, ())
You will probably want to disable the button when it's clicked and enable it when the launching is complete. This can be done inside ext_process.
It becomes more complicated if you want to allow the user to cancel jobs. This solution won't handle that.

Python script freezes infinitely during a socket connection

I have a simple python script that updates that statuses of justin.tv streams in my database. It's a Django based web application. This script worked perfectly before I moved it to my production server, but now it has issues with timing out or freezing. I've solved the time out problem by adding try/except blocks and making the script retry, but I still can't figure out the freezing problem.
I know it freezes on the line streamOnline = manager.getStreamOnline(stream.name, LOG). That's the same point where the socket.timeout exception occurs. Some times however, it just locks up for ever. I just can't picture a scenario where python would freeze infinitely. Here is the code for the script that freezes. I'm linking website.networkmanagers below, as well as oauth and the justin.tv python library that I'm using.
import sys, os, socket
LOG = False
def updateStreamInfo():
# Set necessary paths
honstreams = os.path.realpath(os.path.dirname(__file__) + "../../../")
sys.path.append(honstreams)
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
# Import necessary moduels
from website.models import Stream, StreamInfo
from website.networkmanagers import get_manager, \
NetworkManagerReturnedErrorException
# Get all streams
streams = Stream.objects.all()
try:
# Loop through them
for stream in streams:
skipstream = False
print 'Checking %s...' % stream.name,
# Get the appropriate network manager and
manager = get_manager(stream.network.name)
# Try to get stream status up to 3 times
for i in xrange(3):
try:
streamOnline = manager.getStreamOnline(stream.name, LOG)
break
except socket.error as e:
code, message = e
# Retry up to 3 times
print 'Error: %s. Retrying...'
# If this stream should be skipped
if(skipstream):
print 'Can\'t connect! Skipping %s' % stream.name
continue
# Skip if status has not changed
if streamOnline == stream.online:
print 'Skipping %s because the status has not changed' % \
stream.name
continue
# Save status
stream.online = streamOnline
stream.save()
print 'Set %s to %s' % (stream.name, streamOnline)
except NetworkManagerReturnedErrorException as e:
print 'Stopped the status update loop:', e
if(__name__ == "__main__"):
if(len(sys.argv) > 1 and sys.argv[1] == "log"):
LOG = True
if(LOG): print "Logging enabled"
updateStreamInfo()
networkmanagers.py
oauth.py
JtvClient.py
Example of the script freezing
foo#bar:/.../honstreams/honstreams# python website/scripts/updateStreamStatus.py
Checking angrytestie... Skipping angrytestie because the status has not changed
Checking chustream... Skipping chustream because the status has not changed
Checking cilantrogamer... Skipping cilantrogamer because the status has not changed
| <- caret sits here blinking infinitely
Interesting update
Every time it freezes and I send a keyboard interrupt, it's on the same line in socket.py:
root#husta:/home/honstreams/honstreams# python website/scripts/updateStreamStatus.py
Checking angrytestie... Skipping angrytestie because the status has not changed
Checking chustream... Skipping chustream because the status has not changed
^CChecking cilantrogamer...
Traceback (most recent call last):
File "website/scripts/updateStreamStatus.py", line 64, in <module>
updateStreamInfo()
File "website/scripts/updateStreamStatus.py", line 31, in updateStreamInfo
streamOnline = manager.getStreamOnline(stream.name, LOG)
File "/home/honstreams/honstreams/website/networkmanagers.py", line 47, in getStreamOnline
return self.getChannelLive(channelName, log)
File "/home/honstreams/honstreams/website/networkmanagers.py", line 65, in getChannelLive
response = client.get('/stream/list.json?channel=%s' % channelName)
File "/home/honstreams/honstreams/website/JtvClient.py", line 51, in get
return self._send_request(request, token)
File "/home/honstreams/honstreams/website/JtvClient.py", line 90, in _send_request
return conn.getresponse()
File "/usr/lib/python2.6/httplib.py", line 986, in getresponse
response.begin()
File "/usr/lib/python2.6/httplib.py", line 391, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.6/httplib.py", line 349, in _read_status
line = self.fp.readline()
File "/usr/lib/python2.6/socket.py", line 397, in readline
data = recv(1)
KeyboardInterrupt
Any thoughts?

Have you tried using another application to open that connection? Given that it's an issue in production, perhaps you don't have some firewall issues.

Down in JtvClient.py it uses httplib to handle the connection. Have you tried changing this to use httplib2 instead?
Other than that stab in the dark, I would add a lot of logging statements to this code in order to track what actually happens and where it gets stuck. Then I would make sure that the point where it gets stuck can timeout on the socket (which usually involves either monkeypatching or forking the codebase) so that stuff fails instead of hanging.
You said:
I know it freezes on the line streamOnline = manager.getStreamOnline(stream.name, LOG). That's the same point where the socket.timeout exception occurs.
Wrong. It doesn't freeze on that line because that line is a function call which calls lots of other functions through several levels of other modules. So you do not yet know where the program freezes. Also, that line is NOT the point where the socket timeout occurs. The socket timeout will only occur on a low level socket operation like select or recv which is being called several times in the chain of activity triggered by getStreamOnline.
You need to trace your code in a debugger or add print statements to track down exactly where the hang occurs. It could possibly be an infinite loop in Python but is more likely to be a low-level call to an OS networking function. Until you find the source of the error, you can't do anything.
P.S. the keyboard interrupt is a reasonable clue that the problem is around line 90 in JtvClient.py, so put in some print statements and find out what happens. There may be a stupid loop in there that keeps calling getresponse, or you may be calling it with bad parameters or maybe the network server really is borked. Narrow it down to fewer possibilities.

It turns out this HTTP connection isn't passed a timeout in jtvClient.py
def _get_conn(self):
return httplib.HTTPConnection("%s:%d" % (self.host, self.port))
Changed the last line to
return httplib.HTTPConnection("%s:%d" % (self.host, self.port), timeout=10)
Which solved it