We Keep Coding

Partial Imports: Import everything up to a error

Is there a way to perform a python import which is not atomic?
For instance, I have a file as follows:
# Filename: a.py
myvariable = 1
mylist = [1, 2, 3]
raise ImportError
donotimportthis = 5
I then have a separate file which does the following:
import a
a.myvariable == 1 # This is okay as it imported it
a.donotimportthis # <-- raise an exception as this is not imported.
I have a file which contains some python code, this follows the format of:
...variables...
import X
I do not have X installed nor do I want it however I do want the variables.
Note: This file is autogenerated not by me but by a tool whose version is frozen.

Two choices, in descending order of preference:
Change the autogeneration process. Instead of invoking proprietary_autogen_process, invoke custom_autogen_wrapper. This wrapper in turn first invokes the proprietary third-party tool, and then modifies the produced module source code by searching for the code that imports module X, and deletes everything after it.
This is relatively straightforward. You just need to take some care to not introduce false positives or false negatives by performing too loose (or too strict) matching of the import code. Ideally you’d use an AST rewriter but that’s probably overkill; a regular expression search for import X might work, although it will yield wrong results if this text appears inside a comment, string literal or inside a method which isn’t executed.
Generate a stub module X in a location where it will be found by the autogenerated module when importing the latter. I don’t recommend this because it’s tedious: You probably can’t just generate an empty module, since the autogenerated module will want to use X. You need to generate meaningful method stubs.

You can do specific imports with
from a import myvariable
EDIT: The above won't work if anything that is flat in the file raises an error. If you have no way to edit the imported file then I don't know if there is a (resonable) solution to this. Sorry didn't realise.
(an unreasonable solution would be to read in the file as text, slice it, and then run eval on it).
Or, as mentioned in the comments, put the stuff you don't want under
if __name__=="__main__":
<here>
Then it will only be invoked if you run the file directly.

What you can do is removing the donotimportthis variable at the end of the module, as follows: del donotimportthis. I hope it helps

How can I modify modules and packages while keeping the original intact?

I have a program written in my python using the PyPDF2 package to scrape a batch of pdf files. These PDF's aren't in the greatest shape so in order for my program to run, I need to modify the pdf.py file located within the package library as recommended by this website:
https://cheonhyangzhang.wordpress.com/2015/03/31/python-pdffilereader-pdfreaderror-eof-marker-not-found/
Is there a way I can implement this change to the file while keeping the original file intact? I've tried creating a child class of PdfFileReader class and modifying the 'read' method as prescribed by my link above, however, I've found that that leads to several import dependency issues that I'd like to avoid.
Is there an easier way to do this?

I would recommend to copy the pdf.py file into our script directory and rename it to mypdf.py. You can then modify the copy as you please without affecting the original. You can import the package using
import mypdf
I have done something similar for shutil.py as the default buffer size is too small in Windows for transferring large files.

You can add (or redefine) a method of class using setattr() like this (where the class has been defined inline rather than being imported only for purposes of illustration):
class Class(object):
pass
def func(self, some_other_argument):
return some_other_argument
setattr(Class, 'func', func)
if __name__ == '__main__':
c = Class()
print(c.func(42)) # -> 42

How to get complete list of arguments for function from module in python

I am using iPython in command prompt, Windows 7.
I thought this would be easy to find, I searched and found directions on how to use the inspect package but it seems like the inspect package is meant to be used for functions that are created by the programmer rather than functions that are part of a package.
My main goal to to be able to use the help files from within command prompt of iPython, to be able to look up a function such as csv.reader() and figure out all the possible arguments for it AND all possible values for these arguements.
In R programming this would simply be args(csv.reader())
I have tried googling this but they all point me to the inspect package, perhaps I'm misunderstanding it's use?
For example,
If I wanted to see a list of all possible arguments and the corresponding possible values for these arguments for the csv.reader() function (from the import csv package), how would I go about doing that?
I've tried doing help(csv.reader) but this doesn't provide me a list of all possible arguments and their potential values. 'Dialect' shows up but it doesn't tell me the possible values of the dialect argument of the csv.reader function.
I can easily go to the site: https://docs.python.org/3/library/csv.html#csv-fmt-params and see that the dialect options are: delimiter, doublequote, escapechar, etc.. etc..but is there a way to see this in Python console?
I've also tried dir(csv.reader) but this isn't what I was looking for either.
Going bald trying to figure this out....

There is no way to do this generically, help(<function>) will at a minimum return you the function signature (including the argument names), Python is dynamically typed so you don't get any types and arguments by themselves don't tell you what the valid values are. This is where a good docstring comes in.
However, the csv module does have a specific function for listing the dialects:
>>> csv.list_dialects()
['excel', 'excel-tab', 'unix']
>>> help(csv.excel)
Help on class excel in module csv:
class excel(Dialect)
| Describe the usual properties of Excel-generated CSV files.
...

The inspect module is extremely powerful. To get a list of classes, for example in the csv module, you could go:
import inspect, csv
from pprint import pprint
module = csv
mod_string = 'csv'
module_classes = inspect.getmembers(module, inspect.isclass)
for i in range(len(module_classes)):
myclass = module_classes[i][0]
myclass = mod_string+'.'+myclass
myclass = eval(myclass)
# could construct whatever query you want about this class here...
# you'll need to play with this line to get what you want; it will failasis
#line = inspect.formatargspect(*inspect.getfullargspec(myclass))
pprint(myclass)
Hope this helps get you started!

python unittest faulty module

I have a module that I need to test in python.
I'm using the unittest framework but I ran into a problem.
The module has some method definitions, one of which is used when it's imported (readConfiguration) like so:
.
.
.
def readConfiguration(file = "default.xml"):
# do some reading from xml
readConfiguration()
This is a problem because when I try to import the module it also tries to run the "readConfiguration" method which fails the module and the program (a configuration file does not exist in the test environment).
I'd like to be able to test the module independent of any configuration files.
I didn't write the module and it cannot be re-factored.
I know I can include a dummy configuration file but I'm looking for a "cleaner", more elegant, solution.

As commenters have already pointed out, imports should never have side effects, so try to get the module changed if at all possible.
If you really, absolutely, cannot do this, there might be another way: let readConfiguration() be called, but stub out its dependencies. For instance, if it uses the builtin open() function, you could mock that, as demonstrated in the mock documentation:
>>> mock = MagicMock(return_value=sentinel.file_handle)
>>> with patch('builtins.open', mock):
... import the_broken_module
... # do your testing here
Replace sentinel.file_handle with StringIO("<contents of mock config file>") if you need to supply actual content.
It's brittle as it depends on the implementation of readConfiguration(), but if there really is no other way, it might be useful as a last resort.

Little confused with import python

I come from a PHP (as well as a bunch of other stuff) background and I am playing around with Python. In PHP when I want to include another file I just do include or require and everything in that file is included.
But it seems the recommended way to do stuff in python is from file import but that seems to be more for including libraries and stuff? How do you separate your code amongst several files? Is the only way to do it, to have a single file with a whole bunch of function calls and then import 15 other files?

Things are totally different between PHP and Python, and there are many reasons why.
But it seems the recommended way to do stuff in python is from file import but that seems to be more for including libraries and stuff?
Indeed, import statements are for importing objects from another module to current module. You can either import all the objects of the imported module to current module:
import foo
print foo.bar
or you can select what you want from that module:
from foo import bar
print bar
and even better, if you import a module twice, it will be only imported once:
>> import foo as foo1
>> import foo as foo2
>> foo1 is foo2
True
How do you separate your code amongst several files?
You have to think about your code... That's called software design, and here are a few rules:
you never write an algorithm at the module's level; instead make it a function, and call that function
you never instantiate an object at the module's level; you shall embed it in the function, and call that function
if you need an object in several different functions, create a class and encapsulate that object in that class, then use it in your functions bound to that class (so they now are called methods)
The only exception is when you want to launch a program from command line, you append:
if __name__ == "__main__":
at the end of the module. And my best advice would be to just call your first function afterwards:
if __name__ == "__main__":
main()
Is the only way to do it, to have a single file with a whole bunch of function calls and then import 15 other files?
It's not the only way to do it, but it's the best way to do it. You make all your algorithms into libraries of functions and objects, and then import exactly what you need in other libraries etc.. That's how you create a whole universe of reusable code and never have to reinvent the wheel! So forget about files, and think about modules that contains objects.
Finally, my best advice to you learning python is to unlearn every habit and usage you had while coding PHP, and learn those things again, differently. In the end, that can only make you a better software engineer.

I guess I understand what you are tring to say and to do.
Here is the random include example from PHP:
File #1 - vars.php
<?php
$color = 'green';
$fruit = 'apple';
?>
File #2 - main.php
<?php
echo "A $color $fruit"; // A
include 'vars.php';
echo "A $color $fruit"; // A green apple
?>
The fist echo command will print just "A" string, for it does not have any values assigned to the vars. The next echo will print a full string thanks to your include before it.
Python's "import", however, imports a module or it's part, so you could work with it in your current module.
Here is a python example:
File 1 - echo.py
apple = 'apple'
color = 'green'
File 2 - main.py
import echo
def func():
print "A "+echo.color+" "+echo.fruit
if __name__ == '__main__':
func()
In other words - you import some functionality from one module and then use it in your other module.
The example above is not really good from programming standarts or best practises, but I think it gives you some understanding.

Interesting question. As you know, in PHP, you can separate your code by using include, which literally takes all the code in the included file and puts it wherever you called include. This is convenient for writing web applications because you can easily divide a page into parts (such as header, navigation, footer, etc).
Python, on the other hand, is used for way more than just web applications. To reuse code, you must rely on functions or good old object-oriented programming. PHP also has functions and object-oriented programming FYI.
You write functions and classes in a file and import it in another file. This lets you access the functions or use the classes you defined in the other file.
Lets say you have a function called foo in file file1.py. From file2.py, you can write
import file1. Then, call foo with file1.foo(). Alternatively, write from file1 import foo and then you can call foo with foo(). Note that the from lets you call foo directly. For more info, look at the python docs.

On a technical level, a Python import is very similar to a PHP require, as it will execute the imported file. But since Python isn't designed to ultimately generate an HTML file, the way you use it is very different.
Typically a Python file will on the module level not include much executable code at all, but definitions of functions and classes. You them import them and use them as a library.
Hence having things like header() and footer() makes no sense in Python. Those are just functions. Call them like that, and the result they generate will be ignored.
So how do you split up your Python code? Well, you split it up into functions and classes, which you put into different files, and then import.

There's an execfile() function which does something vaguely comparable with PHP's include, here, but it's almost certainly something you don't want to do. As others have said, it's just a different model and a different programming need in Python. Your code is going to go from function to function, and it doesn't really make a difference in which order you put them, as long as they're in an order where you define things before you use them. You're just not trying to end up with some kind of ordered document like you typically are with PHP, so the need isn't there.