We Keep Coding

Visualize data in Pycharm in the same way as in Jupyter?

Is it possible to visualize data in pycharm in the same way that you can do in jupyter?
For instance in jupyter you can run a single line at a time and see the output which can be helpful for working with data sets.
Of course you can just use a function like head() or show() to see what is going on but you have to run the whole file (as opposed to one line at a time in jupyter) and it can make working with data a bit harder to understand.
Does anyone have any recommendations for me in terms of pycharm as that is what I am most familar with, or do you think it is worth me learning something like jupyter?

Why don't you use Atom and install the package Hydrogen? it offers you the same possibilities as Jupyter while working in a script (not a notebook).
You can execute the code line by line like in Jupyter by clicking ctrl+Enter or run it as a script. Here's the documentation of the package.
Atom is a light IDE so it combines both the power of Jupyter and PyCharm. I have used it and it is great and has so many packages like hydrogen, pep8 (helps to write a code that is conform to the pep8) and code beautifiers (for Python, R, JSON,etc), and a lot of great features.

I recommend learning Jupyter. Although PyCharm is an excellent IDE, running Jupyter in it looks clunky and something like this:

Generate and execute python both on non-python machine

Short version:
Is it possible to create a standalone program that can be distributed to computers that don't have python installed, which writes a python script during runtime and executes it during or shortly after?
Long version:
A project I've been wanting to do for a while is to create a visual programming interface, that lets people use Machine Learning without needing to know python/keras/numpy syntax. Programs like lobe or rapidminer already do this, but they are all bound to their own interfaces and servers. What i would like to do is create a program that:
anyone can use without needing python/anaconda installed
allow the user to create visual scripts like in scratch or google-blocs
generates python code behind the scenes containing keras or tf code
is able to execute the script
is able to show the code to the user for:
educational purposes
allowing the user to use it as a base for a more complex program
Since the generation part would just need to create a text file almost any interface and language like python/java using Qt or C# / javascript using Unity, would qualify. I think that should definitely be doable, probably just very long, but that is not my largest concern.
The problem:
I have tried to search everywhere on the internet about things like standalone python programs which led me to for instance py2exe. Those kind of python-to-exe's work great, but they all require the script to be compiled on a machine containing a python environment. In my case i want to be able to generate a python script on the user's PC, and run it directly after from there.
The alternative
If this isn't possible i might just create the whole AI part myself, not using python nor libraries like keras or tensorflow, but in a unity game for instance. The downside to this would be that it exists already(like rapidminer), it would be less optimized/versatile/customizable and doesn't show what the "real" machine learning script would look.
If there are any other alternatives i would be happy to hear them
PS.
I have mostly Python & Keras, moderate Java and little Unity3D or web JS experience
I'm using Windows with anaconda
Distributing to linux/mac would be nice, but not required

maybe you should check the Orange Data Mining software, it's written in python and it has the same purpose as your project (https://orange.biolab.si/). On the other hand, some time ago I tried to compile my app that contains machine learning libraries but to date, I have not been able to achieve it. Instead, what I use is the WINPYTHON project, this is a portable Anaconda software that allows you to run projects on any PC without the need to install anything (https://winpython.github.io/).

Yes, it is possible!
I had the same requirement, so I wrote my visual programming language and IDE...
...that could generate an almost python-like script, and which is compiled natively inside the app, without the use of any external compilers or libraries.
My target architecture was mobile devices, but it also works on browsers via the unity plugin.
...and yes, that's correct, it runs natively on your phone or tablet in a simulated sandbox, with its own built-in IDE.
It's written in C#, and implemented in Unity3d
You can check it out at https://aiBoard.blog
..and see the videos at https://youtu.be/DIDgu9jrdLc

How to use both Jupyter Notebook and Sublime for Data Science projects?

I'm learning some Data Science and for that I'm using Python with Jupyter Notebook. Which I think it's great for data analysis, mainly because it’s super easy to run step-by-step code. You can see everything that is happening.
On the other hand, to do more complex projects, like a web crawler or an Object Oriented program to extract information from an API, I’m using Sublime Text3. IMO it’s simple, clean, light… perfect. Also I think that .py is better than .ipynb for that (I don't even know if it's possible to do OO with Jupyter).
My problem now is integrating these two tools. The best I can do now is convert the dictionnaires in some .csv file and read it manually in Jupyter notebook. Obviously it doesn't sounds very smart and it is like a temporary solution just for experimentation.
This is the first time I'm dealing with a project which I need to integrate more than one environment and not only working with the same languages with all the files in the same folder etc. so I'm not very familiar on how to approach that.
If someone could explain the right way of integrating these two IDEs, how to make all the process more 'automatic', if it's better to use some database and then extract with SQL or something like that I'd appreciate very much.
PS: Also, if you guys have any material on how should a Python Data Science project be organized it would be awesome. Thanks!

I use ipython magic commands to help me switch between a text editor and a ipython notebook.
Specifically, I like to experiment with the code in the Notebook for the reasons you mentioned, and then when I'm ready to integrate it as a class in a bigger system I use the %%writefile filename.py command which will export that cell into a .py file.
You can also use %load filename.py and %run myfile.py to bring .py files into the notebook.

Directly open Spyder or open it through Pythonxy?

Previously, I have been running Python in Spyder. I always started Spyder by directly double clicking its icon.
Today, I suddenly realize that I have another thing called Python(x,y). I notice that I can also start Spyder through it.
Do these two methods make any difference?
If not, what is the meaning of Python(x,y)? Does it have any other good features?

Python(x,y)
Python(x,y) is a scientific Python distribution. This means that it not only installs Python in your system but also the most important scientific packages needed for a scientist to be productive with the language.
These packages are difficult to install by the user and also are too many to be installed one by one, so that's why Python(x,y) offers the possibility to install them all at once.
Spyder
Spyder is one of the packages distributed by Python(x,y), which offers a simple and convenient graphical environment to use the other scientific packages (like numpy or matplotlib). It resembles Matlab in its design.
Perhaps a comparison to Matlab is worth here:
Matlab is a huge program (about 5 gigs) that it's developed by one company and comes with a lot of packages and a graphical interface to use it.
Python(x,y) is also a big program (about 1 gig) but it's developed by two individuals who collect a lot of open source programs developed by a lot of different groups and institutions. It also comes with a graphical interface (Spyder) but there are others that fulfill a similar purpose (like the IPython notebook).
So in the Python scientific world, every package is developed independently of each other and there is the need of a "meta" program (like Python(x,y)) that distributes them together.

Actually i did not use spyder alone, but when you open it through Python(x,y) which is a scientific oriented distribution, some packages that are used in scientific/numeric computing automatically loaded.
Python(x,y) is based on python, Qt, spyder and includes many tools for scientific/numeric computation, visualizing (2D-3D), etc.
https://code.google.com/p/pythonxy/

How do I prepare myself for a summer of working on Python using Linux environment?

I have used just Windows for programming so far. Now, I have an internship starting in two weeks and I will be using just Linux environment with Python programming language. I've installed Ubuntu on my system but have no exposure to shell scripting.
I need some advice on how I can quickly learn to use the Linux terminal quickly. Any books or web resources that you can suggest?
Also, is there a particular IDE that is generally preferred for Python programming on Linux, or is Vim preferred? How can I best prepare myself for the internship ahead?
Thanks for taking the time.

As an intern you'll want to use the tools your mentor is most comfortable with. If you get stuck you'll be able to ask for advice quickly.
Learning your way around either vi, vim, or emacs to start with will help. The basic concepts used in one will transfer to the other. You'll need to be able to open and read files, search through files, edit and save files, and learn how to apply any python formatting helpers correctly.
You should also familiarize yourself with version control if you haven't already. Again any one will do, you need to focus on concepts and etiquette rather than the specific tool.
The goal of the internship (and really your entire time at university) should be used to learn concepts rather than specific tools. If you learn the concepts you'll be well placed to apply those concepts using any tool. You will also "learn how to learn" a new tool, which is really valuable.

Your lack of shell scripting knowledge shouldn't matter in this case, although it won't be hard to learn. I read over some shell tutorials and put them into practice. Try doing everything from the command line, including find (grep), find/replace all (sed), finding files (find), automating things using python scripts etc. Basically, don't cheat. You'll pick up a lot this way. You'll also probably end up wondering how you ever managed with Windows.
What I use depends on the project. I really like Eclipse+PyDev but that's my personal preference, I also use Vim depending on where I am/what I'm doing. Remember you can just type python from the command line and it drops you into the python environment.

I recommend Eclipse + PyDev too. You can get started quickly with this develop environment. I also recommend the website Dive Into Python. It provides you a online free version of Dive Into Python book, which is very easy to read, easy to understand, and very suitable for Python beginners. If you really want a paper book at hand, Learning Python, a.k.a. The Animal Guide, is simply the best.

Learn to understand man(ual) pages.
For almost any old linux command/program there is a man page which usually explains the command in good detail.
So basics for filesystem navigation:
Show directory contents (list)
ls
Show hidden files
ls -a
Show details
ls -l
Change directory
cd /full/path/name
Print current directory
pwd
Delete a file
rm file
Delete a directory (recursive)
rm -r directoryName
Make a directory
mkdir directoryName
Move (or rename) a file
mv /path/to/file /new/path/to/file
Show the man page for mv
man mv
Learning vim might be necessary, depending on your intern environment. I do my Python (and everything that isn't simple text editing) in Eclipse. You should in any case learn enough to open a file, makes some changes and save the changes in Vim.
Keep in mind, Ubuntu is very easy. To make things harder on yourself, use the command line for every conceivable thing. Open programs by typing their names into a terminal. Browse your files with the terminal. Do simple editing with vim. That should provide good practice for the day you need to SSH into a computer in Neverland and download and install a local copy of your favorite interpreter from source in order to set up a cron job to run a script to play a clock noise.

In addition to the great advice already written, I'd suggest you install IPython (Open a terminal with Applications>Accessories>Terminal and type):
sudo apt-get install ipython
Also at the terminal, you can then type ipython to start the Python interpreter.
Unlike the built in python interpreter, ipython gives you tab completion.
For example, if you type the name of an object followed by a period and TAB (e.g. sys.[TAB]), ipython will show you (almost) all of object's attributes.
Type a question mark after an object name (e.g. sys?), and you get documentation on that object.
This is a great way to explore Python.

have no exposure to shell scripting
Good! You've got Python so hopefully there should be no need to resort to writing actual scripts with the shell. It may be more powerful than DOS batch files, but it's just as ugly.
I need some advice on how I can quickly learn to use the Linux terminal quickly.
Something like this?
As well as learning the commands, you'll want to get used to using tab-completion and arrow key command recall (if you don't already do that with the Windows Command Prompt), scrolling with shift-arrows, and so on. Also useful to know the & (perform in background) command suffix, ctrl-C-to-stop, ctrl-Z-to-pause, jobs, and screen.
Incidentally if you will be spending any amount of time in the interactive Python interpreter it is well worth adding tab completion there, too. (This is just as much the case on Windows, but on Win you tend not to get pyreadline by default.)
is there a particular IDE that is generally preferred for Python programming on Linux
Just like on Windows, there are IDEs available if you want them but many people just use a normal text editor. vim is fine if that's what you like. nano is another in-terminal text editor you usually get that's relatively simple. Ubuntu's default desktop-based editor gedit is also fine. It's a matter of personal taste.
(If you are interning at a particular company they might have their own development environment they'd prefer you to use.)

For a Python IDE, I recommend using either IDLE or Eclipse with PyDev.
Keep in mind you can also just use python on the linux command-line. It supports loading code from files, and if you use two command windows then one of them will be your "REPL" where you will be running python and dynamically loading code - and the other window can run your editor.
Regarding linux command line, I cannot recommend any great resources. However, you will be off to a great start if you immerse yourself in this environment and only use linux for the next 2 weeks. Just keep learning, and when you do not know how to do something, read a manpage or google it to find the answer.

for a very beginner intro to the command line, check out: http://en.flossmanuals.net/CommandLineIntro/GettingStarted

As far as a Python editor goes, I personally prefer to use SciTE. It's just a programmer's text editor with syntax highlighting for various languages. I prefer a lightweight editor over a more complicated environment, but if you want a full-fledged IDE you can always try out NetBeans, IDLE, or Komodo (all of which are available in both Windows and Linux).

as for terminall and quick way to understand it's and learn it there are a nice cheat sheets on net like this:
http://fosswire.com/post/2007/8/unixlinux-command-cheat-sheet/