We Keep Coding

Using import in Python

So I’m a new programmer and starting to use Python 3, and I see some videos of people teaching the language and use “import”. My question is how they know what to import and where you can see all the things you can import. I used import math in one example that I followed along with, but I see other videos of people using import JSON or import random, and I’m curious how they find what they can import and how they know what it will do.

Generally, you look in the python standard library reference, or on the Python Package Index for a module that contains methods you want to use. Then you import them.
A module in python is essentially a way of doing namespaces, same as most other languages. Generally, googling "how to do ______ in python" will provide some result of someone using the module you're aiming for. Then you can look up the documentation for that module to determine what functions and classes it provides (or alternatively, import modulename the module in a python console and then do help(modulename).

as a starting point, the list of all python's built-in modules can be viewed here, along with documentation for each:
https://docs.python.org/3/py-modindex.html
non built-ins are usually downloaded via pip and are available here:
https://pypi.org/
documentation is your friend
edit:
as stated in another comment, there is a vast amount of information at these resources..generally doing some kind of web search (google/stack/etc.) will point you towards a module you are looking for for your specific usage, then look at the examples given or check the docs

In all programming languages, whenever you actually need a library, you should import it. For example, if you need to generate a random number, search for this function in your chosen programming language, find the appropriate library, and import it into your code.

Python completely barebones graphics

I am looking for a program or simply documentation (if it's part of windows) of a library where you can edit an image directly, push it to the screen, and nothing else. Basically what could be a framework for a graphics library, but not giving any aid in terms of drawing things.

Pillow is a fork of the Python Imaging Library (PIL). Find its official docs here, and the very handy chapter on it in Automate the Boring Stuff With Python.
If you have specific needs as to what you mean by editing, please elaborate if I've missed the mark.

I would suggest looking into the PPM image filetype. This filetype, when provided a header, allows you to simply output "0 0 0" or any other values for one pixel. You print how many of these you want in your rows and columns, and it'll construct an image for you. This filetype is especially helpful in quickly implementing graphical representations of solutions
Refer to http://netpbm.sourceforge.net/doc/ppm.html for a quick guide!

Possible to autogenerate Cython bindings around a large, existing C library?

In otherwords: *.h/*.c --[??POSSIBLE??]--> *.pxd/*.pyx
OK. I’ve done (I hope) enough digging around the Internet - but I think this is a good question so I’ll ask it straight.
There are a few related questions (e.g. Generate python bindings, what methods/programs to use or Wrapping a C library in Python: C, Cython or ctypes? ) but which don't quite sum up the situation that I’m asking which is perhaps for a more “high-level” approach (and specifically for an existing library, not generating new C from python).
I’ve got a little bit of experience of this myself having wrapped a wee bit of code before using Cython. Cython gets the thumbs up for speed and maintainability. That’s OK in my book for small/single bits of code - but, this time I’ve got a bit more on my plate…
And following the first of the three great virtues of a programmer - I want to do this with as minimal effort as possible.
So the real question here is how can I ease the creation by automated means of the .pxd, and possibly .pyx, files (i.e. to save time and not slip up miss-typing something).
This here seems to be the only real hint/note about how to do this - but most of the projects on it are defunct, old or sourceforge. Many only seem to work for C++ (this is C I'm doing here).
Does anyone still use them? Recently? Has anyone got a workflow or best practice for doing this? Am I simply just better doing it by hand?
My library is well defined by a set of header files. One containing defs of all the C struct/types and another containing prototypes for all the functions. But it's loooonnnggg...
Thanks for any tips.
UPDATE (25th August, 2015):
Right, so over the last few months when I had a spare moment, I tried:
CFFI (thank for #David pointing that out) - has a noble aim of "to call C code from Python without learning a 3rd language: existing alternatives require users to learn domain specific language (Cython, SWIG) or API (ctypes)” - but it didn’t quite fit the bill as it involved a fair degree of embedded C code in the actual python files (or loading it in). This would be a pretty manual process to do for a large library. Maybe I missed something…
SWIG is the granddaddy of Python binding, and is pretty solid. Fundamentally though, it is not “hands off” as I understand it - i.e. you need a separate specification file. For example, you have to edit all your C header files to indicate building a python module with a #define SWIG_FILE_WITH_INIT or use other annotations. SIP has the same issue here. You don’t auto-generate from the headers, you modify them to include your own directives and annotations and create a complete specification file.
cwrap - I’m on a Mac so I used this version for clang. https://github.com/geggo/cwrap Really poor doc - but using the source I finally got it to run and it generated…. an empty .pyx file from a pretty simple header of structs. Not so good.
xdress - This showed promise. The website is down so the docs are actually seemingly here. There’s an impressive amount of work gone into it and it looks straightforward to use. But it needed all the llvm headers (and a correctly linked version of clang). I had to use brew install llvm —with-clang. There is a xdressclang-3.5 branch, but it doesn’t seem to have enough fixes done. I tried tapping homebrew/versions for an earlier version of clang (install llvm33 / llvm34) and that got it built. Anyway, I digress… it worked great for a simple example, but the resulting ctypes files for the full library was pretty garbled and refused to build. Something in the AST C->Python is a bit awry...
ctypesgen wasn’t one I had encountered in the original search. The documentation is pretty sparse - or you might call it concise. It hasn’t seemingly had much work done on it the last 4 years either (and people enquiring on the issues list if the developers are ever going to further the project). I’ve tried running it, but sadly it seems to fall over with what I suspect/seems like issues with the Clang compiler cdefs.h use of _attribute_. I’ve tried things like -std=c11 but to no avail.
In conclusion, out of all the ones I’ve looked at I think xdress came the closest to the fully automated generation of python bindings. It worked fine for the simple examples given, but couldn’t handle the more complex existing library headers, with all the complexities of forward declarations, enumerated types, void pointers… It seems a well designed and (for a while) well maintained project, so there is possibly some way to circumvent these issues if someone were to take it on again.
Still, the question remains, does anyone have a robust toolchain for generating python wrappers from C headers automatically? I think the reality is there always has to be a bit of manual work, and for that CFFI looks the most “modern” approach (one of the best overviews/comparisons I encountered is here) - yet it always involves a specially edited cdef() version of any header files (e.g. Using Python's CFFI and excluding system headers).

I find ctypesgen great for autogeneration. I'm only using it with one or two python modules that I hope to open source, and I've been happy so far. Here's a quick example using it with zlib, but I also just tried it successfully with a few other libraries:
(Edit: I know you mentioned ctypesgen has problems on a mac, so maybe it needs someone to tweak it to work on OSX - I don't have OSX at home or I'd try it.)
Get ctypesgen:
git clone https://github.com/davidjamesca/ctypesgen.git
Run short script to call ctypesgen (replace zlib info with another library):
import os
ZLIB_INC_DIR = "/usr/include"
ZLIB_LIB_DIR = "/usr/lib/x86_64-linux-gnu"
ZLIB_LIB = "libz.so"
ZLIB_HEADERS = "/usr/include/zlib.h"
# Set location of ctypesgen.py
ctypesgen_path = 'ctypesgen/ctypesgen.py'
wrapper_filename = 'zlib.py'
cmd = "LD_LIBRARY_PATH={} {} -I {} -L {} -l {} {} -o {}".format(
ZLIB_LIB_DIR, ctypesgen_path, ZLIB_INC_DIR, ZLIB_LIB_DIR, ZLIB_LIB,
ZLIB_HEADERS, wrapper_filename)
print(cmd)
os.system(cmd)
Usage example:
python
>>> import zlib
>>> zlib.compress("asdfasdfasdfasdfasdf")
'x\x9cK,NIKD\xc3\x00T\xfb\x08\x17'

Does one often use libraries outside the standard ones?

I am trying to learn Python and referencing the documentation for the standard Python library from the Python website, and I was wondering if this was really the only library and documentation I will need or is there more? I do not plan to program advanced 3d graphics or anything advanced at the moment.
Edit:
Thanks very much for the responses, they were very useful. My problem is where to start on a script I have been thinking of. I want to write a script that converts images into a web format but I am not completely sure where to begin. Thanks for any more help you can provide.

For the basics, yes, the standard Python library is probably all you'll need. But as you continue programming in Python, eventually you will need some other library for some task -- for instance, I recently needed to generate a tone at a specific, but differing, frequency for an application, and pyAudiere did the job just right.
A lot of the other libraries out there generate their documentation differently from the core Python style -- it's just visually different, the content is the same. Some only have docstrings, and you'll be best off reading them in a console, perhaps.
Regardless of how the other documentation is generated, get used to looking through the Python APIs to find the functions/classes/methods you need. When the time comes for you to use non-core libraries, you'll know what you want to do, but you'll have to find how to do it.
For the future, it wouldn't hurt to be familiar with C, either. There's a number of Python libraries that are actually just wrappers around C libraries, and the documentation for the Python libraries is just the same as the documentation for the C libraries. PyOpenGL comes to mind, but it's been a while since I've personally used it.

As others have said, it depends on what you're into. The package index at http://pypi.python.org/pypi/ has categories and summaries that are helpful in seeing what other libraries are available for different purposes. (Select "Browse packages" on the left to see the categories.)

One very common library, that should also fit your current needs, is the Python Image Library (PIL).
Note: the latest version is still in beta, and available only at Effbot site.

If you're just beginning, all you'll need to know is the stuff you can get from the Python website. Failing that a quick Google is the fastest way to get (most) Python answers these days.
As you develop your skills and become more advanced, you'll start looking for more exciting things to do, at which point you'll naturally start coming across other libraries (for example, pygame) that you can use for your more advanced projects.

It's very hard to answer this without knowing what you're planning on using Python for. I recommend Dive Into Python as a useful resource for learning Python.
In terms of popular third party frameworks, for web applications there's the Django framework and associated documentation, network stuff there's Twisted ... the list goes on. It really depends on what you're hoping to do!

Assuming that the standard library doesn't provide what we need and we don't have the time, or the knowledge, to implement the code we reuse 3rd party libraries.
This is a common attitude regardless of the programming language.

If there's a chance that someone else ever wanted to do what you want to do, there's a chance that someone created a library for it. A few minutes Googling something like "python image library" will find you what you need, or let you know that someone hasn't created a library for your purposes.

Sweave for python

I've recently started using Sweave* for creating reports of analyses run with R, and am now looking to do the same with my python scripts.
I've found references to embedding python in Sweave docs, but that seems like a bit of a hack. Has anyone worked out a better solution, or is there an equivalent for python I'm not aware of?
* Sweave is a tool that allows to embed the R code for complete data analyses in latex documents

I have written a Python implementation of Sweave called Pweave that implements basic functionality and some options of Sweave for Python code embedded in reST or Latex document. You can get it here: http://mpastell.com/pweave and see the original blog post here: http://mpastell.com/2010/03/03/pweave-sweave-for-python/

Some suggestions:
I have been using Pweave for several years now, and it is very similar to Sweave. Highly recommended.
The most popular tool for embedded reports in python at this stage is Jupyter notebooks, which allow you to embed markdown, and they are quite useful although I personally still like writing things in LaTeX...
You can also have a look at PyLit, which is intended for literate programming with Python, but not as well maintained as some of the alternatives.
Sphinx is great for documenting with python, and can output LaTex.
Here's a list of tools for literate programming. Some of these work with any programming language.

Dexy is a very similar product to Sweave. One advantage of Dexy is that it is not exclusive to one single language. You could create a Dexy document that included R code, Python code, or about anything else.

This is a bit late, but for future reference you might consider my PythonTeX package for LaTeX. PythonTeX allows you enter Python code in a LaTeX document, run it, and bring back the output. But unlike Sweave, the document you actually edit is a valid .tex document (not .Snw or .Rnw), so editing the non-code part of the document is fast and convenient.
PythonTeX provides many features, including the following:
The document can be compiled without running any Python code; code only needs to be executed when it is modified.
All Python output is saved or cached.
Code runs in user-defined sessions. If there are multiple sessions, sessions automatically run in parallel using all available cores.
Errors and warnings are synchronized with the line numbers of the .tex document, so you know exactly where they came from.
Code can be executed, typeset, or typeset and executed. Syntax highlighting is provided by Pygments.
Anything printed by Python is automatically brought into the .tex document.
You can customize when code is re-executed (modified, errors, warnings, etc.).
The PythonTeX utilities class is available in all code that is executed. It allows you to automatically track dependencies and specify created files that should be cleaned up. For example, you can set the document to detect when the data it depends on is modified, so that code will be re-executed.
A basic PythonTeX file looks like this:
\documentclass{article}
\usepackage{pythontex}
\begin{document}
\begin{pycode}
#Whatever you want here!
\end{pycode}
\end{document}

You might consider noweb, which is language independent and is the basis for Sweave. I've used it for Python and it works well.
http://www.cs.tufts.edu/~nr/noweb/

I've restructured Matti's Pweave a bit, so that it is possible to define arbitrary "chunk-processors" as plugin-modules. This makes it easy to extend for several chunk-based text-preprocessing applications. The restructured version is available at https://bitbucket.org/edgimar/pweave/src. As an example, you could write the following LaTeX-Pweave document (notice the "processor name" in this example is specified with the name 'mplfig'):
\documentclass[a4paper]{article}
\usepackage{graphicx}
\begin{document}
\title{Test document}
\maketitle
Don't miss the great information in Figure \ref{myfig}!
<<p=mplfig, label=myfig, caption = "Figure caption...">>=
import sys
import pylab as pl
pl.plot([1,2,3,4,5],['2,4,6,8,10'], 'b.', markersize=15)
pl.axis('scaled')
pl.axis([-3,3, -3,3]) # [xmin,xmax, ymin,ymax]
#
\end{document}

You could try SageTeX which implements Sweave-Like functionality for the SAGE mathematics platform. I haven't played around with it as much as I would like to, but SAGE is basically a python shell and evaluates python as it's native language.

I have also thought about the same thing many times. After reading your questions and looking into your link I made small modifications to the custom python Sweave driver, that you link to. I modified it to also keep the source code and produce the output as well the same way that Sweave does for R.
I posted the modified version and an example here: http://mpastell.com/2010/02/09/python-in-sweave-document/
Granted, it is not optimal but I'm quite happy with the output and I like the ability to include both R and Python in the same document.
Edit about PyLit:
I also like PyLit and contrary to my original answer you can catch ouput with it as well, although it not as elegant as Sweave! Here is a small example how to do it:
import sys
# Catch PyLit output
a = range(3)
sys.stdout = open('output.txt', 'w')
print a
sys.stdout = sys.__stdout__
# .. include:: output.txt

What you're looking for is achieved with GNU Emacs and org-mode*. org-mode does far more than can be detailed in a single response, but the relevant points are:
Support for literate programming with the ability to integrate multiple languages within the same document (including using one language's results as the input for another language).
Graphics integration.
Export to LaTeX, HTML, PDF, and a variety of other formats natively, automatically generating the markup (but you can do it manually, too).
Everything is 100% customizable, allowing you to adapt the editor to your needs.
I don't have Python installed on my system, but below is an example of two different languages being run within the same session. The excerpt is modified from the wonderful org-mode R tutorial by Erik Iverson which explains the set up and effective use of org-mode for literate programming tasks. This SciPy 2013 presentation demonstrates how org-mode can be integrated into a workflow (and happens to use Python).
Emacs may seem intimidating. But for statistics/data science, it offers tremendous capabilities that either aren't offered anywhere else or are spread across various systems. Emacs allows you to integrate them all into a single interface. I think Daniel Gopar says it best in his Emacs tutorial,
Are you guys that lazy? I mean, c'mon, just read the tutorial, man.
An hour or so with the Emacs tutorial opens the door to some extremely powerful tools.
* Emacs comes with org-mode. No separate install is required.

Well, with reticulate which is a recent best implementation of a Python interface in R you could continue using Sweave and call Python inline using the R interpreter. For example this now works in a .Rnw or .Rmd markdown file.
```{r example, include=FALSE}
library(reticulate)
use_python("./dir/python")
```
```{python}
import pandas
data = pandas.read_csv("./data.csv")
print(data.head())
```

I think that Jupyter-book may do what you want.