after hunting through stackoverflow, I still can't find quite what I'm looking for.
Question:
I am looking to use (or create if this doesn't exist) an open source node graph, as a node-based approach to using Python code.
Clarification
Let me explain the concept of what I mean by "Node graph".
A node graph is a place to create "nodes", represented by little boxes with connections. The user can move select them, move them around, make and break connections based on appropriate data types.
Example
For example- and I think to myself, "surely this has been done before," but I'd like to write nodes in python script that load data, process that data in some way, etc.
For example, I want to write a python node that loads time based data from a file OR a real time source, then another node that processes it, then another node that maybe visualizes it, or writes out a custom format to file.
Background I come from a background in 3D animation where we use the software packages Maya and Nuke to process data in this way using node-based workflow- however the architectures are very high-level and specialized. What I want to do is reduce this idea of workflow to a very mid-low level. All I want is a node graph with custom python nodes.
So far I have tried:
Nodebox 3 - this is a high level software built on Java and compiled code. Seems to be too high level for me (with a fully functional gui) and too java based
Nodebox Windows - a nifty OpenGL derived library, but too low level, would have to build an interactive node graph from scratch
Coral - an interesting idea but too specialized in 3D I think, for doing generic data processing as I'd like. Perhaps I could modify it, but I'm afraid the dying community might hold me back.
appreciate any other suggestions you guys may have.
** EDIT **
StackOverflow won't let me post a screenshot example.. annoying.
A visual programming language- this general idea is what I'm looking for! But what I would prefer is the ability to write functions in Python and then graph them visually like those languages.
Try PyQtGraph
It has a node-graph among a lot of other visualization Widgets. It's called flowchart and lets you create and connect nodes. Each of these contains a python function to modify its input data.
As the name implies it relies on PyQt/PySide.
Related
TLDR: Is there a Python library that allows me to get a application window frame as an image and rewrite it to the said application?
So the whole story is that I want to write an application using Python that does something similar to Lossless Scaling and Magpie. I want to grab an application window (a videogame window, for example), get the current frame as an image, then use some Machine Learning/Deep Learning algorithm (like FSR or DLSS) to upscale said image, then rewrite the current frame from the application with said upscaled image.
So far, I have been playing around with some upscaling algorithms like the one from Real-ESRGAN, but now my main problem is how to upscale the video game images in real-time. The only thing I found that does something related to what I need to do is PyAutoGUI. But this package only allows you to take screenshots of an application but not rewrite the graphics of said application.
I hope I have clarified my problem; feel free to comment if you still have any questions.
Thank you for reading this post, and have a good day.
Doing this with Python is going to be very difficult. A lot of the performance involved in this sort of thing is in avoiding as many memory copies as possible, and Python's idiom for string and bytes processing unfortunately makes quite a few additional copies in the course of any idiomatic program. I say this as a die-hard Python fan who is constantly trying to cram Python in everywhere it doesn't belong: you'd be better off doing this in Rust.
Update: After receiving some feedback from some folks with more direct experience in this sort of thing, I may have overstated the difficulty here. Many ML tools in Python provide zero-copy access, you can easily access and manipulate memory-mapped data from numpy and there is even a CUDA protocol for doing this to data in GPU memory, so while it's not exactly easy, as long as your operations are implemented as numpy operations and not as pure-python pixel-by-pixel logic, it shouldn't be much harder than other python machine learning applications which require access to native APIs for accessing their source data.
However, there's no way to access framebuffer data directly from python, so step 1 is going to be writing your own bindings over the relevant DirectX APIs. Since Magpie is open source, you can see which APIs it's using, for example, in its various C++ "Frame Source" backends. For example, this looks relevant: https://github.com/Blinue/Magpie/blob/42cfcba1222b07e4cec282eaff639aead229f123/Runtime/GraphicsCaptureFrameSource.cpp#L87
You can then look those APIs up on MSDN; that one, for example, is here: https://learn.microsoft.com/en-us/uwp/api/windows.graphics.capture.direct3d11captureframepool.createfreethreaded?view=winrt-22621
CFFI is a good choice for writing native wrappers: https://cffi.readthedocs.io/en/latest/
Gluing these together appropriately is left as an exercise for the reader :).
I am currently starting a kind of larger project in python and I am unsure about how to best structure it. Or to put it in different terms, how to build it in the most "pythonic" way. Let me try to explain the main functionality:
It is supposed to be a tool or toolset by which to extract data from different sources, at the moment mainly SQL-databases, in the future maybe also data from files stored on some network locations. It will probably consist of three main parts:
A data model which will hold all the data extracted from files / SQL. This will be some combination of classes / instances thereof. No big deal here
One or more scripts, which will control everything (Should the data be displayed? Outputted in another file? Which data exactly needs to be fetched? etc) Also pretty straightforward
And some module/class (or multiple modules) which will handle the data extraction of data. This is where I struggle mainly
So for the actual questions:
Should I place the classes of the data model and the "extractor" into one folder/package and access them from outside the package via my "control script"? Or should I place everything together?
How should I build the "extractor"? I already tried three different approaches for a SqlReader module/class: I tried making it just a simple module, not a class, but I didn't really find a clean way on how and where to initialize it. (Sql-connection needs to be set up) I tried making it a class and creating one instance, but then I need to pass around this instance into the different classes of the data model, because each needs to be able to extract data. And I tried making it a static class (defining
everything as a#classmethod) but again, I didn't like setting it up and it also kind of felt wrong.
Should the main script "know" about the extractor-module? Or should it just interact with the data model itself? If not, again the question, where, when and how to initialize the SqlReader
And last but not least, how do I make sure, I close the SQL-connection whenever my script ends? Meaning, even if it ends through an error. I am using cx_oracle by the way
I am happy about any hints / suggestions / answers etc. :)
For this project you will need the basic Data Science Toolkit: Pandas, Matplotlib, and maybe numpy. Also you will need SQLite3(built-in) or another SQL module to work with the databases.
Pandas: Used to extract, manipulate, analyze data.
Matplotlib: Visualize data, make human readable graphs for further data analyzation.
Numpy: Build fast, stable arrays of data that work much faster than python's lists.
Now, this is just a guideline, you will need to dig deeper in their documentation, then use what you need in your project.
Hope that this is what you were looking for!
Cheers
I want to create a piece of software that allows me to create manipulable diagrams (state machine alike) in python. What library should I use (does at least one exist)?
I found this js library (https://gojs.net/latest/samples/stateChart.html), but I'd like to create the whole program in python.
I did not found something that lets you work with diagrams from the UI standpoint.
I would like to be able to click on a certain diagram and edit its contents and manipulate the contests of all diagram objects.
Use pysimpleautomata library pysimpleautomata
You might have to create your own api to create interface though. For that, I'd suggest using tkinter.
I'm making an application to process traffic data using python and postgresql. I need to create a GUI for users to select options such as dates, times, settings, etc. I've done this once before using QT Designer. This time, I need to create a map interface which allows users to select various points or segments on a map for processing. I would plot the locations of detectors, for example, and the user would select the end points of the segment they'd like to process and I would need to determine the points between as well. Potentially, I'd like to output some of the analysis to the map, but I'm mostly concerned about this base functionality as I've never done this before. I'm looking for a straightforward method with a low learning curve. I've looking into QML and creating a HTML/javascript based application, but wondering if there are other options or if one of these is the best? I would prefer not to mess with HTML/javascript too much unless it's clearly the best solution for my needs. I really appreciate any feedback.
I've got masses of network information held in a SQLite database. I want to draw a diagram based on this information in the style a network diagram.
I want it to be interactive, in a sense that at the highest level only network range communication can be seen, and as you move deeper into the diagram you begin to see individual nodes (switches, routers, firewalls, hosts, servers, etc) all linked together.
I'd like this process to be as smooth as possible, allowing you to zoom in using the scroll wheel in a location of the diagram and it expands as you do so. Allowing you to then click and drag around the map. However I'd like to get basics down first, so thinking that I should start with drawing the diagram in HTML and have hyper-links for nodes allowing the user to move around and deeper into the diagram using a browser.
It is also crucial that the user would be able to able to capture their view as a still image, which I'd guess would be far more easily done in HTML.
To get to the point I'm asking where I might start to do this. I've looked at PyQT, Graphviz, outputting to HTML etc. I'm just trying to decide what to use and generally how to go about doing this. I'm reasonably good with Python but I am open to suggestion of other languages.
If you guys think Python can do this, which Python? 2.7 or 3? I've been considering making a move to 3 for a while, is it time?
Thanks in advance!
The important thing here is that you want something dynamic and that can be captured by the user.
My answer will be fairly similar to this question.
Export your graph to a standard format such as GEXF and use a Javascript graph drawing library to make your graph interactive such as: SigmaJs, or VivaGraphJs.
The big advantage is that you can script your graph to respond to user event such as zoom, save as a picture or display information dynamically about nodes and edges, etc.
To resume:
First you a python graph library such as Networkx, then export your graph with its properties as JSON or GEXF.
Load the graph using a javascript graph library using the examples as baselines. List of examples using sigma.js, tutorial for VivaGraphJs.
Concerning the python version, it is really dependent on other libraries you may use. For scientific use, I wouldn't recommend to switch to Py3k but for anything else you're good to go.
I've faced in the past something similar to your problem. We have tons of routers documented in a MySQL database. We actually use Racktables and that tool stores all the information in such a way.
At one point we needed to plot networking topologies. If you want, please have a look at this:
https://notedisabbia.wordpress.com/2016/06/17/first-blog-post/
https://github.com/RackTables/racktables-contribs/tree/master/python-graph-topology
The first link is a blog I've written in order to explain what my python program (second link) does in terms of connecting to Racktables, gathering information and plotting network diagrams.
Hope this helps.
Cheers,
Lucas
I would recommend looking at library d3graph. It does not allow you to zoom using the scroll wheel but does have other features like breaking edges.
More information can be found in this blog.