We Keep Coding

How to modify file content using python?

The following code reads a file, uses syntax tree to append fullstop to docstrings of that file. How can I save changes made in called file? I understand present code doesn't change content in original file but the local variables accessing it. Can you suggest changes, if possible provide learning resource as well?
astcode.py
import ast
import sys
import os
filename = sys.argv[1]
# """Getting all the functions"""
ast_filename = os.path.splitext(ast.__file__)[0] + '.py'
with open(filename) as fd:
file_contents = fd.read()
module = ast.parse(file_contents)
# with open(filename, 'w') as file:
# module level
if(isinstance(module.body[0], ast.Expr)):
docstr = module.body[0].value.s
if(module.body[0].value.s not in '.'):
docstr += '.'
# ast.dump(module, include_attributes=True)
print(docstr)
# function level
function_definitions = [node for node in module.body if isinstance(node, ast.FunctionDef)]
for function in function_definitions:
# next_node = function_definitions[idx].body
next_node = function.body
for new_node in next_node:
if(isinstance(new_node, ast.Expr)):
if(isinstance(new_node.value, ast.Str)):
# Docstring stored in docstr variable.
docstr = new_node.value.s
if(docstr[-1] not in '.'):
new_node.value.s += '.'
# astString = ast.dump(new_node, annotate_fields=True, include_attributes=True)
# print(astString)
# compile(astString, filename, 'eval')
# print(exec(astString))
print(new_node.value.s)
# for line in module:
# file.write(line)
Example
testfile.py
def readDictionaryFile(dictionary_filename):
"""readDictionaryfile doc string"""
return []
def readTextFile(text_filename):
"""readTextfile doc string"""
return []
$ python3 astcode.py testfile.py
Expected
testfile.py
def readDictionaryFile(dictionary_filename):
"""readDictionaryfile doc string."""
return []
def readTextFile(text_filename):
"""readTextfile doc string."""
return []
Note: Fullstop(.) appended.

Looking at the documentation link, I notice there's a NodeVisitor and NodeTransformer with a code example. I looked at how they unparse a function def, and it's basically the same as you've done in your original question, so I used that.
# https://docs.python.org/3/library/ast.html#ast.NodeTransformer
class MyDocstringTransformer(ast.NodeTransformer):
def visit_FunctionDef(self, node):
if len(node.body):
if isinstance(node.body[0], ast.Expr):
if isinstance(node.body[0].value, ast.Constant):
if isinstance(node.body[0].value.value, str):
docstring = node.body[0].value.value
node.body[0].value.value = docstring + '.'
return node
Using python 3.9's ast module gets us https://docs.python.org/3/library/ast.html#ast.unparse which is about as close as we can get to changing the ast node and then rewriting the original file.
tree = ast.parse(file_contents)
new_tree = MyDocstringTransformer().visit(tree)
print(ast.unparse(new_tree))
Instead of just overwriting to the same filename, you may want to write to a temp file, then let the OS attempt to delete the old file and rename the temp file to the old name, thus performing the replace in the OS.

read a dict into self

I have a class that is doing a lot of stuff. In the end, it saves everything into a pickle. When I rerun this class I want to read the pickle instead of doing everything again. Unfortunately it the variable is always empty if I unpickle. Why is that so?
import pandas as pd
class Test:
def __init__(path, value):
# path points to a .txt file but its in the same folder as the pickle
data_path, data = os.path.split(path)
pickle_path = os.path.join(data_path, name.split('.')[1] + '.pickle'
if os.path.isfile(pickle_path):
self = pd.read_pickle(path)
else:
# do a ton of stuff and safe it as pickle afterwards
variable = Test(path, value)
In this case variable is empty if I read from pickle but correct if I do all the stuff...

If I want to cache some calculation results I will load/dump the object outside the class, something like,
pickle_path = os.path.join(data_path, name.split('.')[1] + '.pickle'
if os.path.isfile(pickle_path):
with open(pickle_path, 'rb') as f:
variable = pickle.load(f) # use cached results
else:
variable = Test() # do all the calculations

PyTest setup or teardown a variable

How can I re-instantiate a variable for each of my PyTests?
Specifically, I want to create a new StringIO() object each time.
My current code is this:
output = StringIO()
def info_to_string(text):
output.write(text)
def write_to_file(some_text):
for row in some_text:
info_to_string(row)
I need output to be set up each time there is a new test fixture.
Copy and pastable code to test:
from io import StringIO
import pytest
output = StringIO()
def info_to_string(text):
output.write(text)
def write_to_file(some_text):
for row in some_text:
info_to_string(row)
def test_1():
write_to_file(['hello', 'there', 'what', 'is', 'up'])
print(output)
assert output.getvalue() == "hellotherewhatisup"
def test_2():
write_to_file(['nothing', 'much'])
assert output.getvalue() == "nothingmuch"
#This will error as the output is "hellotherewhatisupnothingmuch" if run after test_1
So I would need a new output = stringIO() for each test.

Incase anyone sees this in the future, the way I did this was by creating a class, re-initialising it each fixture
class WriteToString:
def __init__(self):
self.output = StringIO()
def info_to_string(self, text):
self.output.write(text)
#pytest.fixture
def write_to_string():
return WriteToString()
and changing the tests to:
def test_2(write_to_string):
write_to_file(['nothing', 'much'])
assert write_to_string.output.getvalue() == "nothingmuch"

I'm not sure I quite understand your questions, but you can do something like below, which will create a new StringIO instance every time you pass it into your test function. If you want to return a different string every time, I don't think you're looking for a fixture, but just a generic function call that does the work for you.
import pytest
from StringIO import StringIO
#pytest.fixture(scope='function')
def info_to_string():
output = StringIO()
output.write('blah')
return output
def test_something(info_to_string):
assert isinstance(info_to_string, StringIO)
assert info_to_string.getvalue() == 'blah'

I think the easiest way to test file writing operation is to use tempfile.
But once StringIO is mentioned as a part of testing strategy my suggestion would be to split file writing into two parts. This gives a clean entry point for StringIO buffer.
from pathlib import Path
from io import StringIO
import pytest
# Separate your data export to 2 functions, one operating
# with a filename, other with a file object.
def to_file(filename, content):
path = Path(filename)
if not path.exists():
path.touch()
with path.open('a') as f_obj:
write_to_file(f_obj, content)
# This is code under test. It must not be connected to your
# test code in any way, even if you have some version of "monkey-patching"
# in mind.
def write_to_file(file_object, content):
"""Dump *content* list of strings to *filename* with *writer*."""
for x in content:
file_object.write(str(x))
# This is test with StringIO buffer
def test_write_to_file_with_buffer():
buffer = StringIO()
# note you make several calls to write_to_file in one test, not across tests
# this helps keep tests isolated/independent
write_to_file(buffer, ['hello', 'there', 'what', 'is', 'up'])
write_to_file(buffer, ['nothing', 'much'])
assert buffer.getvalue() == 'hellotherewhatisupnothingmuch'

Best practice for keeping a config file in python

I have a python script in which I have a config file which looks like this:
PROD = 'production'
DEV = 'dev'
ENVIRONMENT = None
and I have a function which gets the wanted environment from a command argument and sets it like:
if sys.argv[1] in [config.PROD, config.DEV]:
config.ENVIRONMENT = sys.argv[1]
I understood it's not good practice when I started importing the config file in multiple files and ENV kept resetting back to None.
So, my question is: what is the best practice is this case

I'm not sure exactly what the best practice is but I like using JSON files. I use the following class as a layer of abstraction for interfacing with the config (properties) file. You can create one JSONPropertiesFile and pass it around your application.
import json
from collections import OrderedDict
import os
from stat import * # ST_SIZE etc
from datetime import datetime
from copy import deepcopy
class JSONPropertiesFileError(Exception):
pass
class JSONPropertiesFile(object):
def __init__(self, file_path, default={}):
self.file_path = file_path
self._default_properties = default
self._validate_file_path(file_path)
def _validate_file_path(self, file_path):
if not file_path.endswith(".json"):
raise JSONPropertiesFileError(f"Must be a JSON file: {file_path}")
if not os.path.exists(file_path):
self.set(self._default_properties)
def set(self, properties):
new_properties = deepcopy(self._default_properties)
new_properties.update(properties)
with open(self.file_path, 'w') as file:
json.dump(new_properties, file, indent=4)
def get(self):
properties = deepcopy(self._default_properties)
with open(self.file_path) as file:
properties.update(json.load(file, object_pairs_hook=OrderedDict))
return properties
def get_file_info(self):
st = os.stat(self.file_path)
res = {
'size':st[ST_SIZE],
'size_str':str(round(st[ST_SIZE]/1000,2)) + ' KB',
'last_mod': datetime.fromtimestamp(st[ST_MTIME]).strftime("%Y-%m-%d")
}
return res
In your case you might use it like this:
file_path = "path/to/your/config/file"
default_properties = {
'PROD': 'production',
'DEV': 'dev',
'ENVIRONMENT': ""
}
config_file = JSONPropertiesFile(file_path, default_properties)
config = config_file.get()
print(config["PROD"])
config["PROD"] = "something else"
config_file.set(config) # save new config

Mocking open(file_name) in unit tests [duplicate]

This question already has answers here:
How do I mock the filesystem in Python unit tests?
(5 answers)
Closed 1 year ago.
I have a source code that opens a csv file and sets up a header to
value association. The source code is given below:
def ParseCsvFile(source):
"""Parse the csv file.
Args:
source: file to be parsed
Returns: the list of dictionary entities; each dictionary contains
attribute to value mapping or its equivalent.
"""
global rack_file
rack_type_file = None
try:
rack_file = source
rack_type_file = open(rack_file) # Need to mock this line.
headers = rack_type_file.readline().split(',')
length = len(headers)
reader = csv.reader(rack_type_file, delimiter=',')
attributes_list=[] # list of dictionaries.
for line in reader:
# More process to happeng. Converting the rack name to sequence.
attributes_list.append(dict((headers[i],
line[i]) for i in range(length)))
return attributes_list
except IOError, (errno, strerror):
logging.error("I/O error(%s): %s" % (errno, strerror))
except IndexError, (errno, strerror):
logging.error('Index Error(%s), %s' %(errno, strerror))
finally:
rack_type_file.close()
I am trying to mock the following statement
rack_type_file = open(rack_file)
How do I mock open(...) function?

This is admittedly an old question, hence some of the answers are outdated.
In the current version of the mock library there is a convenience function designed for precisely this purpose. Here's how it works:
>>> from mock import mock_open
>>> m = mock_open()
>>> with patch('__main__.open', m, create=True):
... with open('foo', 'w') as h:
... h.write('some stuff')
...
>>> m.mock_calls
[call('foo', 'w'),
call().__enter__(),
call().write('some stuff'),
call().__exit__(None, None, None)]
>>> m.assert_called_once_with('foo', 'w')
>>> handle = m()
>>> handle.write.assert_called_once_with('some stuff')
Documentation is here.

To mock built-in function open with mox use __builtin__ module:
import __builtin__ # unlike __builtins__ this must be imported
m = mox.Mox()
m.StubOutWithMock(__builtin__, 'open')
open('ftphelp.yml', 'rb').AndReturn(StringIO("fake file content"))
m.ReplayAll()
# call the code you want to test that calls `open`
m.VerifyAll()
m.UnsetStubs()
Note that __builtins__ is not always a module, it can be of type dict, please use __builtin__ (with no "s") module to refer to system built-in methods.
More about __builtin__ module: http://docs.python.org/library/builtin.html

There are two ways that I like to do this, depending on the situation.
If your unit test is going to call ParseCsvFile directly I would add a new kwarg to ParseCsvFile:
def ParseCsvFile(source, open=open):
# ...
rack_type_file = open(rack_file) # Need to mock this line.
Then your unit test can pass a different open_func in order to accomplish the mocking.
If your unit test calls some other function that in turn calls ParseCsvFile then passing around open_func just for tests is ugly. In that case I would use the mock module. This lets you alter a function by name and replace it with a Mock object.
# code.py
def open_func(name):
return open(name)
def ParseCsvFile(source):
# ...
rack_type_file = open_func(rack_file) # Need to mock this line.
# test.py
import unittest
import mock
from StringIO import StringIO
#mock.patch('code.open_func')
class ParseCsvTest(unittest.TestCase):
def test_parse(self, open_mock):
open_mock.return_value = StringIO("my,example,input")
# ...

Is simple with decorator (Python3):
def my_method():
with open(file="/1.txt", mode='r', encoding='utf-8') as file:
return file.read().strip()
#mock.patch("builtins.open", create=True)
def test_my_method(mock_open):
mock_open.side_effect = [
mock.mock_open(read_data="A").return_value
]
resA = my_method()
assert resA == "A"
mock_open.mock_calls == [mock.call(file="/1.txt", mode='r', encoding='utf-8')]

I took the liberty of re-writing your sample function:
Assume your function is located in a file named code.py
# code.py
import csv
import logging
def ParseCsvFile(source):
"""Parse the csv file.
Args:
source: file to be parsed
Returns: the list of dictionary entities; each dictionary contains
attribute to value mapping or its equivalent.
"""
global rack_file
rack_file = source
attributes_list = []
try:
rack_type_file = open(rack_file)
except IOError, (errno, strerror):
logging.error("I/O error(%s): %s", errno, strerror)
else:
reader = csv.DictReader(rack_type_file, delimiter=',')
attributes_list = [line for line in reader] # list of dictionaries
rack_type_file.close()
return attributes_list
A simple test case would be:
# your test file
import __builtin__
import unittest
import contextlib
from StringIO import StringIO
import mox
import code
#contextlib.contextmanager
def mox_replayer(mox_instance):
mox_instance.ReplayAll()
yield
mox_instance.VerifyAll()
class TestParseCSVFile(unittest.TestCase):
def setUp(self):
self.mox = mox.Mox()
def tearDown(self):
self.mox.UnsetStubs()
def test_parse_csv_file_returns_list_of_dicts(self):
TEST_FILE_NAME = 'foo.csv'
self.mox.StubOutWithMock(__builtin__, 'open')
open(TEST_FILE_NAME).AndReturn(StringIO("name,age\nfoo,13"))
with mox_replayer(self.mox):
result = code.ParseCsvFile(TEST_FILE_NAME)
self.assertEqual(result, [{'age': '13', 'name': 'foo'}]) # works!
if __name__ == '__main__':
unittest.main()
EDIT:
% /usr/bin/python2.6
Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import __builtin__
>>> import mox
>>> mock = mox.Mox()
>>> mock.StubOutWithMock(__builtin__, 'open')
>>> mock.UnsetStubs()
Works fine on 2.6 using mox 0.53

Hi I was having a similar problem, and was tearing my hair out flipping between different mocking libraries. I finally found a solution that I am happy with, and maybe it might help you? In the end I went with the Mocker library http://labix.org/mocker and here is the code for mocking open:
from mocker import Mocker
from StringIO import StringIO
import __builtin__
mocker = Mocker()
sourceFile = 'myTestFile.txt'
__builtin__.open = mocker.mock()
__builtin__.open(sourceFile)
mocker.result(StringIO('this,is,a,test,file'))
<the rest of your test setup goes here>
mocker.replay()
ParseCsvFile(sourceFile)
mocker.restore()
mocker.verify()
Incidentaly the reason I went with Mocker is because I was testing a function which used open to read a file, and then used open again to overwrite the same file with new data. What I needed to be able to do was test the case where the initial file didn't exist, so set up a mock, that threw an IOError the first time, and then worked the second time. The setup for which looked like this:
from mocker import Mocker
import __builtin__
mocker = Mocker()
mockFileObject = mocker.mock()
__builtin__.open = mocker.mock()
__builtin__.open('previousState.pkl', 'r')
mocker.throw(IOError('Boom'))
__builtin__.open('previousState.pkl','w')
mocker.result(mockFileObject)
<rest of test setup >
mocker.replay()
<test>
mocker.restore() #required to restore the open method
mocker.verify()
Hope this helps!

#mock.patch decorator (2.7 example)
This is now much easier:
import your_script.py
import __builtin__
import mock
#mock.patch("__builtin__.open")
def test_example(self, mock_open):
your_script.your_method()
self.assertEqual(mock_open.call_count, 1)

>>> class A(object):
... def __init__(self):
... self.x = open('test.py')
...
>>> old_open = open
>>> def open(s):
... return "test\n"
...
>>> a = A()
>>> a.x
'test\n'
>>> open = old_open
>>> a = A()
>>> a.x
<open file 'test.py', mode 'r' at 0xb7736230>