EDIT: I've since taken a different approach with my task such that the solution to this isn't necessary anymore, but I'm leaving the question up for posterity. Because of this post (Manual pyglet loop freezes after a few iterations) describing a similar issue, I believe something inherent to pyglet was the issue. The solution recommended in the cited thread did not work for me.
Brand new to Python and PsychoPy. I inherited a script that will run a task, but we're encountering a major issue. The overview is that participants see a set of instructions, and then begin a loop in which they see background information about a movie clip, then watch that movie clip (using MovieStim3 and pyglet) while we collect live, ongoing ratings from them using a joystick, and then that loop continues to reiterate through the 9 stimuli.
The problem is that the script freezes on the 6th iteration of the for loop every time. It will play the video through and freeze on the last frame. The first five iterations will work perfectly, though, it seems. No error messages are produced and I have to force quit in order to get psychopy to close. It also fails to produce a log or any data.
Here's what I've tried so far that hasn't worked:
Changing the order of the films
Reducing all films to a length of 5s
Changing the films from .mp4 to .mov
Changing which films we show
I think an additional piece of information that's helpful to know is that if I reduce my stimuli list to 5 films, the script executes perfectly from start to finish. Naturally, I assumed that the person who originally coded this must have limited the number of possible iterations to 5, but neither they nor I can find such a parameter. Knowing this, I thought a simple solution might be to simply make two separate loops and two separate stimuli lists (both under 6 items) and have it iterate through those sequentially. However, I run into the exact same issue. This makes me think it's got to be something outside of my stimuli presentation loop, however, I'm simply at a loss to figure out what it might be. If anyone could offer any assistance or guidance, it would be immensely appreciated.
Because I can't isolate the problem to one area and there's a character limit here, I'm going to share the whole script and the associated stimuli via my github repository, but if there's anything more targeted that would be helpful for me to share here, please don't hesitate to ask me. I'm sure it's also very bulky code, so apologies for that as well. If you've gotten this far, thanks so much for your time.
Related
I'm not asking for someone to write out specific codes for me, but rather just point me towards the general direction.
Sometimes after writing and pinning a comment under a Youtube video, it disappears!
It may happen hours later, it may happen a few minutes later, all randomly. Then re-appears hours later, not sure why.
Since I am currently learning programming (python), I might as well try to tackle the problem myself. I want to make a program that regularly checks if the pinned comment is still present, and if not, send me an alert in whatever way so I can go write and pin a new comment for the time being. (and maybe learn a few things during the process)
So far I learned some basic stuff like how to get source code from a webpage, output it to somewhere like a txt file, and also searching with Regex. However when I check out the source code of a Youtube video, the comments text aren't there. Where do I begin learning about the necessary things needed to make this program work?
I’ve been using BeautifulSoup to extract multiple pages of reviews from websites, and it’s worked wonders mostly but on large datasets has constantly been getting stuck at seemingly random points.
My code is always along the lines of the following.
for x in range(len(reviews)):
reviewsoups.append(BeautifulSoup(requests.get(reviews[x]).text, ‘html.parser))
I’ve never gotten any errors or anything (except the random ConnectionReset error), but it just seems as though the loop gets stuck randomly to the point where I consistently have to interrupt the kernel (which often takes 10+ minutes to actually work) and restart the process from the index where the loop got stuck.
It seems as though in some cases, if I try and use my laptop whatsoever while the code is running (like opening Chrome etc) that aggravates the situation.
Can anyone help? It’s just incredibly irritating having to sit by my laptop waiting just in case something like this happens.
Thanks in advance.
I think I’ve found a solution.
So I was trying to ‘soup’ 9000 URLs. What I did was iteratively created variables using the globals() function, with the idea of having each variable store 100 soups, so that would be 90 variables with 100 soups each rather than one list with 9000.
I noticed that the first few hundred were very quick and then slowed down, however running 100 at a time and not constantly elongating an already massive list made a difference.
I also got no crashes.
Bear in mind I only tried this with the last 1000 or so after I got stuck at the 8000 mark, but it was so much quicker and no technical issues.
Next time I will initialise a for loop that incorporates each variable and appends eg the 1056th soup to the 11th variable as the 56th element if that makes sense.
So I'm new to python and just finished my first application. (Giving random chords to be played on a midi piano and increasing the score if the right notes are hit in a graphical interface, nothing too fancy but also non-trivial.) And now I'm looking for a new challenge, this time I'm going to try and create a program that monitors a poker table and collects data on all the players. Though this is completely allowed on almost all poker rooms (example of the largest one) there is obviously no set and go API available. This probably makes the extraction of relevant data the most challenging part of the entire program. In my search for more information, I came across an undergraduate thesis that goes in to writing such a program using Java (Internet Poker: Data Collection and Analysis - Haruyoshi Sakai).
In this thesis, the author speaks of 3 data collection methods:
Sniffing packets
Hand history
Scraping the screen
Like the author, I've come to the conclusion that the third option is probably the best route, but unlike him I have no knowledge of how to start this.
What I do know is the following: Any table will look like the image below. Note how text, including numbers is written in the same font on the table. Additionally, all relevant information is also supplied in the chat box situated in the lower left corner of the window.
In some regards using the chat box sounds like the best way to go, seeing as all text is predictable and in the same font. The problem I see is computational speed: It will often occur that many actions get executed in rapid succession. Any program will have to be able to keep up with this.
On the other hand, using the table as reference means that you have to deal with unpredictable bet locations.
The plan: Taking this in to a count, I'd start by getting an index of all player's names and stacks from the table view and "initialising" the table that way, and continue to use their stacks to extrapolate the betting they do.
The Method: Of course, the method is the entire reason why I made this post. It seems to me like one would need some sort of OCR to achieve all this, but seeing as everything is in a known font, there may be some significant optimisations that can be made. I would love some input on resources to learn about solutions to similar problems. Or if you've got a better idea on how to tackle this problem, I'd love to hear that too!
Please do be sure to ask any questions you may have, I will be happy to answer them in as much detail as possible.
My company has slightly more than 300 vehicle based windows CE 5.0 mobile devices that all share the same software and usage model of Direct Store Delivery during the day then doing a Tcom at the home base every night. There is an unknown event(s) that results in the device freaking out and rebooting itself in the middle of the day. Frequency of this issue is ~10 times per week across the fleet of computers that all reboot daily, 6 days a week. The math is 300*6=1800 boots per week (at least) 10/1800= 0.5%. I realize that number is very low, but it is more than my boss wants to have.
My challenge, is to find a way to scan through several thousand logfille.txt files and try to find some sort of pattern. I KNOW there is a pattern here somewhere. I’ve got a couple ideas of where to start, but I wanted to throw this out to the community and see what suggestions you all might have.
A bit of background on this issue. The application starts a new log file at each boot. In an orderly (control) log file, you see the app startup, do its thing all day, and then start a shutdown process in a somewhat orderly fashion 8-10 hours later. In a problem log file, you see the device startup and then the log ends without any shutdown sequence at all in a time less than 8 hours. It then starts a new log file which shares the same date as the logfile1.old that it made in the rename process. The application that we have was home grown by windows developers that are no longer with the company. Even better, they don’t currently know who has the source at the moment.
I’m aware of the various CE tools that can be used to detect memory leaks (DevHealth, retail messages, etc..) and we are investigating that route as well, however I’m convinced that there is a pattern to be found, that I’m just not smart enough to find. There has to be a way to do this using Perl or Python that I’m just not seeing. Here are two ideas I have.
Idea 1 – Look for trends in word usage.
Create an array of every unique word used in the entire log file and output a count of each word. Once I had a count of the words that were being used, I could run some stats on them and look for the non-normal events. Perhaps the word “purple” is being used 500 times in a 1000 line log file ( there might be some math there?) on a control and only 4 times on a 500 line problem log? Perhaps there is a unique word that is only seen in the problem files. Maybe I could get a reverse “word cloud”?
Idea 2 – categorize lines into entry-type and then look for trends in the sequence of type of entry type?
The logfiles already have a predictable schema that looks like this = Level|date|time|system|source|message
I’m 99% sure there is a visible pattern here that I just can’t find. All of the logs got turned up to “super duper verbose” so there is a boatload of fluff (25 logs p/sec , 40k lines per file) that makes this even more challenging. If there isn’t a unique word, then this has almost got to be true. How do I do this?
Item 3 – Hire a windows CE platform developer
Yes, we are going down that path as well, but I KNOW there is a pattern I’m missing. They will use the tools that I don’t have) or make the tools that we need to figure out what’s up. I suspect that there might be a memory leak, radio event or other event that platform tools I’m sure will show.
Item 4 – Something I’m not even thinking of that you have used.
There have got to be tools out there that do this that aren’t as prestigious as a well-executed python script, and I’m willing to go down that path, I just don’t know what those tools are.
Oh yeah, I can’t post log files to the web, so don’t ask. The users are promising to report trends when they see them, but I’m not exactly hopeful on that front. All I need to find is either a pattern in the logs, or steps to duplicate
So there you have it. What tools or techniques can I use to even start on this?
was wondering if you'd looked at the ELK stack? It's an acronym for elasticsearch, kibana and log stash and fits your use case closely; it's often used for analysis of large numbers of log files.
Elasticsearch and kibana gives you a UI that lets you interactively explore and chart data for trends. Very powerful and quite straight forward to set up on a Linux platform and there's a Windows version too. (Took me a day or two of setup but you get a lot of functional power from it). Software is free to download and use. You could use this in a style similar to idea 1 / 2
https://www.elastic.co/webinars/introduction-elk-stack
http://logz.io/learn/complete-guide-elk-stack/
On the question of Python / idea 4 (which elk could be considered part of) I haven't done this for log files but I have used Regex to search and extract text patterns from documents using Python. That may also help you find patterns if you had some leads on the sorts of patterns you are looking for.
Just a couple of thoughts; hope they help.
There is no input data at all to this problem so this answer will be basically pure theory, a little collection of ideas you could consider.
To analize patterns out of a bunch of many logs you could definitely creating some graphs displaying relevant data which could help to narrow the problem, python is really very good for these kind of tasks.
You could also transform/insert the logs into databases, that way you'd be able to query the relevant suspicious events much faster and even compare massively all your logs.
A simpler approach could be just focusing on a simple log showing the crash, instead wasting a lot of efforts or resources trying to find some kind of generic pattern, start by reading through one simple log in order to catch suspicious "events" which could produce the crash.
My favourite approach for these type of tricky problems is different from the previous ones, instead of focusing on analizing or even parsing the logs I'd just try to reproduce the bug/s in a deterministic way locally (you don't even need to have the source code). Sometimes it's really difficult to replicate the production environment in your the dev environment but definitely is time well invested. All the effort you put into this process will help you to solve not only these bugs but improving your software much faster. Remember, the more times you're able to iterate the better.
Another approach could just be coding a little script which would allow you to replay logs which crashed, not sure if that'll be easy in your environment though. Usually this strategy works quite well with production software using web-services where there will be a lot of tuples with data-requests and data-retrieves.
In any case, without seeing the type of data from your logs I can't be more specific nor giving much more concrete details.
I have a program that I'm designing to teach myself Python but have gotten stuck. I've run across a way to condense python code using % and a list of arguments to have it run as code and looping through the list of arguments until it's done, but can't seem to find it in the documentation or with Google, mostly because I can't figure out how to search for it (sadly it's pretty vague I know).
Mostly what I'm trying to do is prevent having to write approx. 300 lines simply for GUI buttons, labels, frames, ect. with very few changes between them.
Obviously the code is huge so it's all available at Github, github.com/kari-fox/MixxMaster. I'm looking for any kind of recommendations for condensing all of the GUI elements, mostly because I find myself copy/pasting any time I need to make minor changes to any of the similar pieces.
Thanks to Steven Summers in the comments for mentioning classes. I had no idea they were so powerful! I found this guide (https://jeffknupp.com/blog/2014/06/18/improve-your-python-python-classes-and-object-oriented-programming/) and it made understanding and implementing classes work perfectly! That helped me cut out almost 100 lines of purely window-building code and let me do a few sneaky things besides. If anyone wants to see the full implementation check out the github link in the question.