We Keep Coding

Group a collection of module files in a directory into a single import

I have a python application that has grown in size and complexity. I now have 2 folders - one that contain some utility classes and another that contains some other classes. It's quite a long list in each and the classes are referenced all over the place.
MyApp Folder
main_app.py
-- states
- Class1.py
- Class2.py
-- util
- Util1.py
- Util2.py
In main_ap.py is there a way I can just do import states and then reference any classes within that folder as states.Class1? I'd do the same for the util folder but the difference there is some of these classes reference each other.
I've tried __init__.py and some other things but i'm a legacy C++/C developer and relatively new to Python and i think this is handled much differently.

So __init__.py is a file that is executed the first time when you access something inside that package.
Saying that, import states is a way to execute content of states/__init__.py
# states/__init__.py
from states.Class1 import MyClass
# Or with relative path
from .Class2 import MyClass as MyClass2
# main_app.py
import states
states.MyClass() # It works
from states import MyClass2
MyClass2() # It works too.
In case you decide to use asterisk import (as Eldamir stated) you can be interested in __all__ keyword.

If states.__init__.py does from .Class1 import * and from .Class2 import *, then main.py can do import states and then states.SomeClassFromClass1Module

define a __init__.py file and then import the classes you want in the __init__.py file eg from class1 import class11 then at last define __all__ method and write all the class name which you want to be used as states.class11 as __all__ = ['class11', 'class12',.. ]

Avoiding huge files when following python google-style-guide

Google style guide requires to import modules. Say I have a non trivial class and several non-trivial derived classes. It seems that I have to put them all in the one file otherwise will have to use different prefixes when using them.
Personally I prefer one class per file. Is there a common idiom to avoid unnecessary large files?
E.g. can I rename several imports into one as long as there is no name conflicts?
import data_main as data
import data_derived_a as data
import data_derived_b as data

With modules, no, there's really no way of doing this that I know of. If you're happy to put your class in separate files within a package, though, you can import the individual classes in __init__.py, and your client modules can import them all at once, like so.
# mypackage/a.py
class A:
pass
# mypackage/b.py
class B:
pass
# mypackage/__init__.py
from .a import A
from .b import B
# mymodule.py
from mypackage import A, B
a = A()
b = B()
# Or mymodule.py
import mypackage as p
a = p.A()
b = p.B()

How to call a class outside of the file

I am an APCS student so my primary language is Java but am currently writing a program that requires python which I know somewhat. I assume there is a way to call classes / methods outside the file in which they were created like in Java because they are both OO, but was wondering how I would go about doing this. Thanks.

Depending on how you go about packaging your code (IE: you may end up using a python package or simply a module) you will likely use the import statement in a very similar way as you do java. Some Examples:
import my_module
my_object = my_module.MyClass()
or
from my_module import MyClass
my_object = MyClass()

You use the import statement, where you either import the whole file or just the required class.
For example, my directory has file1.py and file2.py.
file2.py has a class named "A", which has the method print_hi().
So, in file1.py I do this:
from file2 import A
A.print_hi()
And I get the expected output.
Or, I could do this:
import file2
file2.A.print_hi()
which gives me the same output.But, you can't just import a method from a class which is present in another file --- you either import the file or the required class.
Thus, doing things like:
from file2.A import print_hi
or
from file2 import A.print_hi
won't work.

avoid sub-modules and external packages in a module's namespace

I'm writing a module to load a dataset. I want to keep the interface/API as clean as possible - so I've made internal functions and variables hidden by prefacing their names with __. Awesome. My module, however, imports other packages (e.g. numpy) which still appear in my module's namespace, how can I avoid this?
i.e. my file looks something like:
Loader.py:
import numpy as np
__INTERNAL_VAR1 = True
EXTERNAL_VAR = True
def loadData():
data = __INTERNAL_FUNC1()
...
return data
def __INTERNAL_FUNC1():
...
return data
and when I import my module np is exposed:
> import Loader
> Loader.[TAB]
Loader.EXTERNAL_VAR Loader.loadData Loader.np

If the autocompletion you are using is correctly implemented, it should honour the __all__ attribute of modules, if set.
Add a list of all names your module exports in that name:
__all__ = ['loadData', 'EXTERNAL_VAR']
The __all__ variable is used to determine what names are imported if you use a from modulename import * wildcard import, as well as by the help() function when documenting your module.
There is no point in using double-underscore names as globals; it is a single underscore at the start that marks such names as 'internal' (by convention).

Another solution could be to create an __init__.py that contains a line that imports the functions and variables you need:
from Loader import EXTERNAL_VAR,loadData
This file should be placed inside a folder containing your module:
└── Loader
├── __init__.py
└── Loader.py
Now when you import Loader you can access directly only EXTERNAL_VAR and loadData while all the other classes and modules are in Loader.Loader.
This solution should be independent of your autocompletion system.

Python namespace elements visibility (vs proper package structure)

Python3
Tried to found an answer but failed. First I'll present the snippet, then I'll explain why I wanted to do it this way and what I wanted to achieve. Maybe it'll look like this approach is "the bad one".
Hence this semi-double topic, cause first I'd like to know why this snippet isn't working and second - I'd like to know if this approach is right.
So:
class Namespace:
def some_function():
pass
class SomeClass:
fcnt = some_function
This won't work due to:
NameError: name 'some_function' is not defined
What I want to achieve is code and file structure readability.
Above example is a snippet which I use (not this one, but it looks like this) in Pyramid project.
My project tree looks like this:
my_project
├── models
│   ├── __init__.py
│   └── some_model.py
├── schemas
│   ├── __init__.py
│   ├── some_schema.py
│   └── some_other_schema.py
...
├── views
│   ├── __init__.py
│   └── some_view.py
└── __init__.py
What I wanted to achieve is clean schema/model/view importing.
In some_schema.py file resides class SomeSchema, in some_other_schema.py class SomeOtherSchema.
With above snippet I can make:
from my_project.schemas.some_schema import Schema
and use it like Schema.SomeSchema()
I've got a little bit lost with packages and imports. How one could make a clean structure (one schema per file) and still be able to use Schema namespace? (In C++ I'd just put each of those classes in Schema namespace, that's why I did this in snippet above. But! What works in C++ maybe shouldn't be used in python, right?).
Thanks for answer in advance.
EDIT:
Ok, I've done some testing (I thought that I've done it, but looks like not..).
using from my_project.schemas.some_schema import Schema with another from my_project.schemas.some_other_schema import Schema causes in the second import shadowing first one. So if after first import I'd be able to use x = Schema.SomeSchema() than after second import I'd be unable to do this, because class Schema gets overriden. Right, so as Erik said - classes aren't namespaces. GOT IT!
in my very first snippet yes, I should've used fnct = Namespace.some_function. What's wierd - it works. I have the same statement in my pyramid code, with one difference. some_function has #colander.deferred decorator. In fact it looks like this:
class Schema:
#colander.deferred
def deferred_some_function(node, kw):
something = kw.get("something", [])
return deform.widget.SelectWidget(values=something,
multiple=True)
class SomeSchema(colander.MappingSchema):
somethings = colander.SchemaNode(colander.Set(),
widget=Schema.deferred_some_function)
And I get NameError: name 'Schema' is not defined
Getting back to package format. With this:
### another/file.py
from foo.bar.schema import SomeSchema
# do something with SomeSchema:
smth = SomeSchema()
smth.fcnt()
I have to make one module foo/bar/schema.py in which I'd have to put all my SomeXSchema classes. An if I have lots of them, then there's the unreadabilty glitch which I wanted to get rid off by splitting SomeXSchema - one per file. Can I accomplish this somehow? I want to call this class for example: User. And here's the THING. Maybe I do it wrong? I'd like to have class named User in schema namespace and class named User in model namespace. Shouldn't I? Maybe I ought to use prefix? Like class SchemaUser and class ModelUser ? I wanted to avoid it by the use of modules/packages.
If I'd use : import foo.bar.schema then I'd have to use it like x = foo.bar.schema.User() right? There is no way to use it like x = schema.User() ? Sorry, I just got stuck, my brain got fixed. Maybe I need a little break to take a fresh look?
ANOTHER EDIT (FOR POINT 3 ONLY)
I did some more research. The answer here would be to make it like this:
## file: myproject/schemas/__init__.py
from .some_schema import SomeSchema
from .some_other_schema import SomeOtherSchema
then usage would be like this:
## some file using it
import myproject.schemas as schema
s1 = schema.SomeSchema()
s2 = schema.SomeOtherSchema()
Would it be lege artis?
If anyone thinks that topic should be changed - go ahead, give me something more meaningful, I'd appreciate it.

Your are swimming upstream by trying to do what you are trying to do.
Classes are meant for defining new data types not as a means to group related parts of code together. Modules are perfectly suited for that, and I presume you know that well because of the "(vs proper package structure)" part in the question title.
Modules can also be imported as objects, so to achieve what you want:
### foo/bar/schema.py
def some_function():
pass
class SomeSchema:
fcnt = some_function
### another/file.py
from foo.bar import schema
# do something with SomeSchema:
smth = schema.SomeSchema()
smth.fcnt()
...although it's also typical to import classes directly into the scope like this (i.e. being able to refer to SomeSchema after the import as opposed to schema.SomeSchema):
### another/file.py
from foo.bar.schema import SomeSchema
# do something with SomeSchema:
smth = SomeSchema()
smth.fcnt()
(Also note that module names should be lowercase as suggested by PEP8 and only class names should use PascalCase)
This, by the way, applies to programming in general, not just Python. There are a few languages such as Java and C# which require that functions be declared inside of classes as statics because they disallow writing of code outside of classes for some weird reason, but even these languages have modules/proper namespaces for structuring your code; i.e. classes are not normally put inside other classes (they sometimes are, but for wholly different reasons/goals than yours).
So basically "class" means a "type" or "a set of objects having similar behavior"; once you ignore that principle/definition, you're writing bad code by definition.
PS. if you are using Python 2.x, you should be inheriting your classes from object so as to get new-style classes.
PPS. in any case, even technically speaking, what you are trying to do won't work cleanly in Python:
class fake_namespace:
def some_function():
pass
class RealClass:
some_function # <-- that name is not even visibile here;
# you'd have to use fake_namespace.some_function instead
...and this is the reason for the exception I reported I was getting: NameError: name 'some_function' is not defined.
EDIT AS PER YOUR EDITS:
I'm not really sure why you're making it so complicated; also some of your statements are false:
If I'd use : import foo.bar.schema then I'd have to use it like x = foo.bar.schema.User right?
No. Please learn how Python modules work.
I'd like to have class named User in Schema namespace and class named User in Model namespace. Shouldn't I? Maybe I ought to use prefix? Like class SchemaUser and class ModelUser
please note that namespaces a.k.a. modules should be lowercase not PascalCase.
An if I have lots of them, then there's the unreadabilty glitch which I wanted to get rid off by splitting SomeXSchema - one per file. Can I accomplish this somehow?
Yes; you can put your classes in individual submodules, e.g. schema1/class1.py, schema/class2.py etc; then you can "collect" them into schema/__init__.py so that you could import them directly from schema:
# schema/__init__.py
from .class1 import Class1
from .class2 import Class2
__all__ = [Class1, Class2] # optional
General note: you can name your schema modules differently, e.g. schema1, schema2, etc; then you could just use them like this:
from somewhere import schema1
from somewhere_else import schema2
s1_user = schema1.User()
s2_user = schema2.User()
# etc
For more information on how Python modules work, refer to http://docs.python.org/2/tutorial/modules.html

Name and binding
You can read Python naming and binding and understand how Python namespace works.
A scope defines the visibility of a name within a block. If a local variable is defined in a block, its scope includes that block. If the definition occurs in a function block, the scope extends to any blocks contained within the defining one, unless a contained block introduces a different binding for the name. The scope of names defined in a class block is limited to the class block; it does not extend to the code blocks of methods this includes generator expressions since they are implemented using a function scope.
BTW, use globals() and locals() can help debug for variable binding.
The User Problem
You can try this instead:
from model import User as modelUser
from foo.bar.schema import User as schemaUser