I'm looking for the name for a procedure which handles output from one function in several others (trying to find better words for my problem). Some pseudo/actual code would be really helpful.
I have written the following code:
def read_data():
read data from a file
create df
return df
def parse_data():
sorted_df = read_data()
count lines
sort by date
return sorted_df
def add_new_column():
new_column_df = parse_data()
add new column
return new_column_df
def create_plot():
plot_data = add_new_column()
create a plot
display chart
What I'm trying to understand is how to skip a function, e.g. create following chain read_data() -> parse_data() -> create_plot().
As the code looks right now (due to all return values and how they are passed between functions) it requires me to change input data in the last function, create_plot().
I suspect that I'm creating logically incorrect code.
Any thoughts?
Original code:
import pandas as pd
import matplotlib.pyplot as plt
# Read csv files in to data frame
def read_data():
raw_data = pd.read_csv('C:/testdata.csv', sep=',', engine='python', encoding='utf-8-sig').replace({'{':'', '}':'', '"':'', ',':' '}, regex=True)
return raw_data
def parse_data(parsed_data):
...
# Convert CreationDate column into datetime
raw_data['CreationDate'] = pd.to_datetime(raw_data['CreationDate'], format='%Y-%m-%d %H:%M:%S', errors='coerce')
raw_data.sort_values(by=['CreationDate'], inplace=True, ascending=True)
parsed_data = raw_data
return parsed_data
raw_data = read_files()
parsed = parsed_data(raw_data)
Pass the data in instead of just effectively "nesting" everything. Any data that a function requires should ideally be passed in to the function as a parameter:
def read_data():
read data from a file
create df
return df
def parse_data(sorted_df):
count lines
sort by date
return sorted_df
def add_new_column(new_column_df):
add new column
return new_column_df
def create_plot(plot_data):
create a plot
display chart
df = read_data()
parsed = parse_data(df)
added = add_new_column(parsed)
create_plot(added)
Try to make sure functions are only handling what they're directly responsible for. It isn't parse_data's job to know where the data is coming from or to produce the data, so it shouldn't be worrying about that. Let the caller handle that.
The way I have things set up here is often referred to as "piping" or "threading". Information "flows" from one function into the next. In a language like Clojure, this could be written as:
(-> (read-data)
(parse-data)
(add-new-column)
(create-plot))
Using the threading macro -> which frees you up from manually needing to handle data passing. Unfortunately, Python doesn't have anything built in to do this, although it can be achieved using external modules.
Also note that since dataframes seem to be mutable, you don't actually need to return the altered ones them from the functions. If you're just mutating the argument directly, you could just pass the same data frame to each of the functions in order instead of placing it in intermediate variables like parsed and added. The way I'm showing here is a general way to set things up, but it can be altered depending on your exact use case.
Use class to contain your code
class DataManipulation:
def __init__(self, path):
self.df = pd.DataFrame()
self.read_data(path)
#staticmethod
def new(file_path):
return DataManipulation(path)
def read_data(self, path):
read data from a file
self.df = create df
def parse_data(self):
use self.df
count lines
sort by date
return self
def add_new_column(self):
use self.df
add new column
return self
def create_plot(self):
plot_data = add_new_column()
create a plot
display chart
return self
And then,
d = DataManipulation.new(filepath).parse_data().add_column().create_plot()
Related
I want to have a pandas DataFrame as a property of my class. I want users to be able to interact with the DataFrame in a normal way, but when it changes I want to run a few checks and balances over the resultant data to make sure it remains valid. I thought I would be able to achieve this using the #property decorator, however this doesn't appear to work when a column of the dataframe (as opposed to the entire dataframe) is changed. An example demonstrating what I mean is below.
My question: is there an alternative way to implement what I want? The only thing I can think of is creating a new class inheriting from DataFrame.... which I'd rather not do, and also the desire to monitor a dataframe whenever it gets updated seems like a broader issue than just my own case?
import pandas as pd
class DemoClass:
def __init__(self, data_frame):
self.data = data_frame
self._check_data_frame_is_valid()
#property
def data(self):
return self._data
#data.setter
def data(self, new_data):
self._data = new_data
self._check_data_frame_is_valid()
def _check_data_frame_is_valid(self):
# example check:
allowed_columns = ['one', 'two']
for column in self.data:
assert column in allowed_columns
assert self.data[column].sum() < 10
print('data passed checks')
if __name__ == '__main__':
example_data = pd.DataFrame({'one': [1,2,3], 'two': [1,2,3]})
class_instance = DemoClass(example_data)
DemoClass.data = example_data # this works; #data.setter is triggered
DemoClass.data['three'] = [4,5,6] #this does not work, and the data is not checked
I am trying to read 3 CSV files into 3 pandas DataFrame. But after executing the function the variable seems not available. Tries to create a blank data frame outside the function and read and set the frame in the function. But the frame is blank.
# Load data from the csv file
def LoadFiles():
x = pd.read_csv('columns_description.csv', index_col=None)
print("Columns Description")
print(f"Number of rows/records: {x.shape[0]}")
print(f"Number of columns/variables: {x.shape[1]}")
LoadFiles()
x.head()
Python Notebook for above code with Error
In the second approach, I am trying to create a new data frame with some consolidated information from the dataset. The issue reappears as the variable seems to be no longer available.
# Understand the variables
y = pd.read_csv('columns_description.csv', index_col=None)
def refresh_y():
var_y = pd.DataFrame(columns=['Variable','Number of unique values'])
for i, var in enumerate(y.columns):
var_y.loc[i] = [y, y[var].nunique()]
refresh_y()
Screenshot with error code and solution restructuring in the function
I am a bit new to Python, The code is a sample and does not represent actual data and in the function, an example is with a single column. I have multiple columns to refresh in this derived data set based on changes further hence the function approach.
When defining a function, if you want to use a variable that is defined in the function, you should end with return var. Check this: Function returns None without return statement and some tutorials on defining a function (https://learnpython.com/blog/define-function-python/).
A basic example to help you start with defining functions:
def sum_product(arg1,arg2): #your function takes 2 arguments
var1 = arg1 + arg2
var2 = arg1*arg2
return var1,var2 #returns two values
new_var1, new_var2 = sum_product(3,4)
For the first example try modifying it like:
def LoadFiles():
var = pd.read_csv('columns_description.csv', index_col=None)
print("Columns Description")
print(f"Number of rows/records: {var.shape[0]}")
print(f"Number of columns/variables: {var.shape[1]}")
return var
x = LoadFiles()
x.head()
try following code
# Load data from the csv file
def LoadFiles():
x = pd.read_csv('columns_description.csv', index_col=None)
print("Columns Description")
print(f"Number of rows/records: {x.shape[0]}")
print(f"Number of columns/variables: {x.shape[1]}")
return x
x2 = LoadFiles()
x2.head()
Variables in a function is only available inside function. You may need study about scope. I recommend the following simple site about scope in Python.
https://www.w3schools.com/python/python_scope.asp
I have a pandas DataFrame and I need to modify this based on some other data that I'm reading off of many files. To speed this up, I'm trying to do this in parallel using the built-in multiprocessing module.
I have a function that modifies the DataFrame. When called manually, this works fine and the DataFrame changes are preserved. When called in parallel, the changes are not preserved.
I know this is somehow related to the SettingWithCopyWarning, but I can't figure out how to work around the issue?
How can I preserve the changes when using the multiprocessing module?
I've created a MWE that illustrates the problem that I'm having.
import multiprocessing
import pandas as pd
class Data:
def __init__(self):
self.data = pd.DataFrame(
{
"ZAID": ["1001.00c", "92235.00nc"],
"ZA": [1001, 92235],
"T(K)": [293.6, 293.6]
}
)
# This is the data that I don't know when creating the DataFrame
# The content will be filled later using addData
columns = {"NE": int, "length": int}
for name, dtype in columns.items():
self.data[name] = pd.Series(dtype=dtype)
def addData(self, index):
"""
add data to an index of the data frame
"""
# read file and extract data
NE = 1200
length = 1234
self.data.loc[index, "NE"] = NE
self.data.loc[index, "length"] = length
if __name__ == "__main__":
print("Learning pandas")
dd = Data()
dd.addData(0) # This call *does* save the addition
with multiprocessing.Pool(2) as pool:
# This call *does not* save the additions
pool.map(dd.addData, dd.data.index)
I am trying to create a Pandas pipeline that creates dummy variables and append the column to the existing dataframe.
Unfortunately I can't get the appended columns to stick when the pipeline is finished.
Example:
def function(df):
pass
def create_dummy(df):
a = pd.get_dummy(df['col'])
b = df.append(a)
return b
def mah_pipe(df):
(df.pipe(function)
.pipe(create_dummy)
.pipe(print)
return df
print(mah_pipe(df))
First - I have no idea if this is good practice.
What's weird is that the .pipe(print) prints the dataframe with appended columns. Yay.
But the statement print(mah_pipe(df)) does not. I though they would behave the same way.
I have tried to read the documentation about pd.pipe but I couldn't figure it out.
Hoping someone could help shed some light on what's going on.
This is because print in Python returns None. Since you are not making a copy of df on your pipes, your df dies after print.
pipes in Pandas
Unless used as last pipe, in Pandas, we except (df) -> [pipe] -> (df_1)-> [pipe2] ->(df_2)-> [pipeN] -> df_N By having print as last pipe, the output is None.
Solution
...
def start_pipe(dataf):
# allows make a copy to avoid modifying original
dataf = dataf.copy()
def create_dummies(dataf, column_name):
dummies = pd.get_dummies(dataf[column_name])
dataf[dummies.columns] = dummies
return dataf
def print_dataf(dataf, n_rows=5):
print(dataf.head(n_rows))
return dataf # this is important
# usage
...
dt = (df
.pipe(start_pipe)
.pipe(create_dummies, column_name='a')
.pipe(print_dataf, n_rows=10)
)
def mah_pipe(df):
df = (df
.pipe(start_pipe)
.pipe(create_dummies, column_name='a')
.pipe(print_dataf, n_rows=10)
)
return df
print(mah_pipe(df))
In function, I can't use argument to define the name of the df in df.to_csv().
I have a long script to pull apart and understand. To do so I want to save the different dataframes it uses and store them in order. I created a function to do this and add the order number 01 (number_of_interim_exports) to the name (from argument).
My problem is that I need to use this for multiple dataframe names, but the df.to_csv part won't accept an argument in place of df...
def print_interim_results_any(name, num_exports, df_name):
global number_of_interim_exports
global print_interim_outputs
if print_interim_outputs == 1:
csvName = str(number_of_interim_exports).zfill(2) + "_" +name
interimFileName = "interim_export_"+csvName+".csv"
df.to_csv(interimFileName, sep=;, encoding='utf-8', index=False)
number_of_interim_exports += 1
I think i just screwed something else up: this works fine:
import pandas as pd
df = pd.DataFrame({1:[1,2,3]})
def f(frame):
frame.to_csv("interimFileName.csv")
f(df)