in the following piece of code I've extracted a window of data off of an audio sample(1000Hz signal). In the code, I've tried to obtain a wavelength of the signal.
https://paste.pound-python.org/show/HRVqQNy3w9Sr73q4oY8g/
sample = data[100:200]
x = 0
i = 1
num_occur = 0
while num_occur <2:
if sample[i] == sample[0]:
x = i
i += 1
num_occur += 1
else:
i += 1
wavelen = sample[:x]
But with less success...
the image of the sample : (https://pasteboard.co/HFFXGxW.png)
Well, I do understand what the problem is; even though matplotlib plots the wave as a continuous wave(due to the high sampling frequency), the wave is made up of discrete data, so there may or may not be a data value satisfying:
sample[i] == sample[0]
I'll greatly appreciate any help and advice on how to get around this problem.
Someone enlightened me on how to get to the answer. It's a simple but logical approach.
So I just needed to extract the points that would cut 0, and the wave between the 1st and 3rd such consecutive points would give me a wavelength.
Here's the code I wrote:
sample = data[100:200]
i = 1
num_occur = 0
cut_zero = []
x = 0
while num_occur < 3:
if sample[x] == abs(sample[x]):
if sample[i] == abs(sample[i]):
i +=1
else:
cut_zero.append(i)
num_occur += 1
i += 1
x = i
elif sample[x] != abs(sample[x]):
if sample[i] == abs(sample[i]):
cut_zero.append(i)
i += 1
num_occur += 1
x = i
else:
i += 1
print(cut_zero)
a = cut_zero[0]
b = cut_zero[2]
wavelen = sample[a:b]
Maybe I could do it more efficiently :), if so let me know.
here's the image of the wavelength https://pasteboard.co/HFLtOmr.png
Related
I am writing a code to read research data which have up to billion lines. I have to read data line by line because the data have multiple blocks. Each block has headers which are different from other block headers and datasets.
I hope to read those datasets into a Numpy matrix so I can perform matrix operations. Here are essential codes.
with open(datafile, "r") as dump:
i = 0 # block line number
line_no = 0 # total line number
block_size = 0
block_count = 0
for line in dump:
values = line.rstrip().rsplit()
i += 1
line_no += 1
if i <= self.head_line_no:
print(line) # for test
if self.tag_block in line or i == 1: # 1st line of a block
# save block size after reading 1st block
if block_size == 0 and block_count == 0:
block_size = line_no - 1
i = 1 # reset block line number
self.box = [] # reset box constant
print(self.matrix)
self.matrix = np.zeros((0, 0), dtype="float") # reset matrix
block_count += 1
elif i == 2:
self.timestamp.append(values[0])
elif i == 3 or i == 5:
continue
elif i == 4:
if self.atom_no != 0 and self.atom_no != values[0]:
self.warning_message = "atom number in timestep " + self.timestamp[-1] + "is inconsistent with" + self.timestamp[-2]
config.ConfigureUserEnv.log(self.warning_message)
else:
pass
self.atom_no = values[0]
elif i == 6 or i == 7 or i == 8:
self.box.append(values[0])
self.box.append(values[1])
elif i == self.head_line_no:
values = line.rstrip().rsplit(":")
for j in range(1,len(values)):
self.column_name.append(values[j])
else:
if self.matrix.size != 0:
np_array = np.array(values)
self.matrix = np.append(self.matrix, np.array(np.asarray(values)), 0)
else:
np_array = np.array(values)
self.matrix = np.zeros((1,len(values)), dtype="float")
self.matrix = np.asarray(values)
dump.close()
print(self.matrix) # for test
print(self.matrix.size) # for test
Original data like below:
ITEM: TIMESTEP
100
ITEM: NUMBER OF ATOMS
17587
ITEM: BOX BOUNDS pp pp pp
0.0000000000000000e+00 4.3491000000000000e+01
0.0000000000000000e+00 4.3491000000000000e+01
0.0000000000000000e+00 1.2994000000000000e+02
ITEM: ATOMS id type q xs ys zs
59 1 1.80278 0.110598 0.129682 0.0359397
297 1 1.14132 0.139569 0.0496654 0.00692627
315 1 1.17041 0.0832356 0.00620818 0.00507927
509 1 1.67165 0.0420777 0.113817 0.0313991
590 1 1.65209 0.114966 0.0630015 0.0447129
731 1 1.65143 0.0501253 0.13658 0.0108512
1333 2 1.049 0.00850751 0.0526546 0.0406341
......
I hope to add matrix data like below:
matrix = [[59 1 1.80278 0.110598 0.129682 0.0359397],
[297 1 1.14132 0.139569 0.0496654 0.00692627],
[315 1 1.17041 0.0832356 0.00620818 0.00507927],
...]
As mentioned above, there are very big size of datasets. I hope to use the fastest way to append array to the matrix. Any further help and advice would be highly appreciated.
Here are some important point to speed up the computation:
Do not use self.matrix = np.append(self.matrix, ...) in a loop, this is not efficient as it recreate a new growing array for each iteration (and copy the old one). This result in a quadratic run time. Use a pure-Python list instead with append and convert the list to a Numpy array in the end. This is the most critical performance-wise point.
Using self.box.extend((values[0], values[1])) should be significantly faster than performing two append.
Using dtype="float" is not very clear not very efficient, please consider using dtype=np.float64 instead (that do not need to be parsed by Numpy).
Using enumerate may be a bit faster than a manual increment in the loop.
Cython may help you to speed up this program if this is not fast enough for your input file. One should keep in mind that the standard Python interpreter (CPython) is not very fast to parse complex huge files compared to compiled native programs/modules written in languages like C or C++.
Note that values[i] are strings and so self.timestamp and self.box. Aren't they supposed to be integers/floats?
I've trying to implement transition from an amount of space to another which is similar to acceleration and deceleration, except i failed and the only thing that i got from this was this infinite stack of mess, here is a screenshot showing this in action:
you can see a very black circle here, which are in reality something like 100 or 200 circles stacked on top of each other
and i reached this result using this piece of code:
def Place_circles(curve, circle_space, cs, draw=True, screen=None):
curve_acceleration = []
if type(curve) == tuple:
curve_acceleration = curve[1][0]
curve_intensity = curve[1][1]
curve = curve[0]
#print(curve_intensity)
#print(curve_acceleration)
Circle_list = []
idx = [0,0]
for c in reversed(range(0,len(curve))):
for p in reversed(range(0,len(curve[c]))):
user_dist = circle_space[curve_intensity[c]] + curve_acceleration[c] * p
dist = math.sqrt(math.pow(curve[c][p][0] - curve[idx[0]][idx[1]][0],2)+math.pow(curve [c][p][1] - curve[idx[0]][idx[1]][1],2))
if dist > user_dist:
idx = [c,p]
Circle_list.append(circles.circles(round(curve[c][p][0]), round(curve[c][p][1]), cs, draw, screen))
This place circles depending on the intensity (a number between 0 and 2, random) of the current curve, which equal to an amount of space (let's say between 20 and 30 here, 20 being index 0, 30 being index 2 and a number between these 2 being index 1).
This create the stack you see above and isn't what i want, i also came to the conclusion that i cannot use acceleration since the amount of time to move between 2 points depend on the amount of circles i need to click on, knowing that there are multiple circles between each points, but not being able to determine how many lead to me being unable to the the classic acceleration formula.
So I'm running out of options here and ideas on how to transition from an amount of space to another.
any idea?
PS: i scrapped the idea above and switched back to my master branch but the code for this is still available in the branch i created here https://github.com/Mrcubix/Osu-StreamGenerator/tree/acceleration .
So now I'm back with my normal code that don't possess acceleration or deceleration.
TL:DR i can't use acceleration since i don't know the amount of circles that are going to be placed between the 2 points and make the time of travel vary (i need for exemple to click circles at 180 bpm of one circle every 0.333s) so I'm looking for another way to generate gradually changing space.
First, i took my function that was generating the intensity for each curves in [0 ; 2]
Then i scrapped the acceleration formula as it's unusable.
Now i'm using a basic algorithm to determine the maximum amount of circles i can place on a curve.
Now the way my script work is the following:
i first generate a stream (multiple circles that need to be clicked at high bpm)
this way i obtain the length of each curves (or segments) of the polyline.
i generate an intensity for each curve using the following function:
def generate_intensity(Circle_list: list = None, circle_space: int = None, Args: list = None):
curve_intensity = []
if not Args or Args[0] == "NewProfile":
prompt = True
while prompt:
max_duration_intensity = input("Choose the maximum amount of curve the change in intensity will occur for: ")
if max_duration_intensity.isdigit():
max_duration_intensity = int(max_duration_intensity)
prompt = False
prompt = True
while prompt:
intensity_change_odds = input("Choose the odds of occurence for changes in intensity (1-100): ")
if intensity_change_odds.isdigit():
intensity_change_odds = int(intensity_change_odds)
if 0 < intensity_change_odds <= 100:
prompt = False
prompt = True
while prompt:
min_intensity = input("Choose the lowest amount of spacing a circle will have: ")
if min_intensity.isdigit():
min_intensity = float(min_intensity)
if min_intensity < circle_space:
prompt = False
prompt = True
while prompt:
max_intensity = input("Choose the highest amount of spacing a circle will have: ")
if max_intensity.isdigit():
max_intensity = float(max_intensity)
if max_intensity > circle_space:
prompt = False
prompt = True
if Args:
if Args[0] == "NewProfile":
return [max_duration_intensity, intensity_change_odds, min_intensity, max_intensity]
elif Args[0] == "GenMap":
max_duration_intensity = Args[1]
intensity_change_odds = Args[2]
min_intensity = Args[3]
max_intensity = Args[4]
circle_space = ([min_intensity, circle_space, max_intensity] if not Args else [Args[0][3],circle_space,Args[0][4]])
count = 0
for idx, i in enumerate(Circle_list):
if idx == len(Circle_list) - 1:
if random.randint(0,100) < intensity_change_odds:
if random.randint(0,100) > 50:
curve_intensity.append(2)
else:
curve_intensity.append(0)
else:
curve_intensity.append(1)
if random.randint(0,100) < intensity_change_odds:
if random.randint(0,100) > 50:
curve_intensity.append(2)
count += 1
else:
curve_intensity.append(0)
count += 1
else:
if curve_intensity:
if curve_intensity[-1] == 2 and not count+1 > max_duration_intensity:
curve_intensity.append(2)
count += 1
continue
elif curve_intensity[-1] == 0 and not count+1 > max_duration_intensity:
curve_intensity.append(0)
count += 1
continue
elif count+1 > 2:
curve_intensity.append(1)
count = 0
continue
else:
curve_intensity.append(1)
else:
curve_intensity.append(1)
curve_intensity.reverse()
if curve_intensity.count(curve_intensity[0]) == len(curve_intensity):
print("Intensity didn't change")
return circle_space[1]
print("\n")
return [circle_space, curve_intensity]
with this, i obtain 2 list, one with the spacing i specified, and the second one is the list of randomly generated intensity.
from there i call another function taking into argument the polyline, the previously specified spacings and the generated intensity:
def acceleration_algorithm(polyline, circle_space, curve_intensity):
new_circle_spacing = []
for idx in range(len(polyline)): #repeat 4 times
spacing = []
Length = 0
best_spacing = 0
for p_idx in range(len(polyline[idx])-1): #repeat 1000 times / p_idx in [0 ; 1000]
# Create multiple list containing spacing going from circle_space[curve_intensity[idx-1]] to circle_space[curve_intensity[idx]]
spacing.append(np.linspace(circle_space[curve_intensity[idx]],circle_space[curve_intensity[idx+1]], p_idx).tolist())
# Sum distance to find length of curve
Length += abs(math.sqrt((polyline[idx][p_idx+1][0] - polyline[idx][p_idx][0]) ** 2 + (polyline [idx][p_idx+1][1] - polyline[idx][p_idx][1]) ** 2))
for s in range(len(spacing)): # probably has 1000 list in 1 list
length_left = Length # Make sure to reset length for each iteration
for dist in spacing[s]: # substract the specified int in spacing[s]
length_left -= dist
if length_left > 0:
best_spacing = s
else: # Since length < 0, use previous working index (best_spacing), could also jsut do `s-1`
if spacing[best_spacing] == []:
new_circle_spacing.append([circle_space[1]])
continue
new_circle_spacing.append(spacing[best_spacing])
break
return new_circle_spacing
with this, i obtain a list with the space between each circles that are going to be placed,
from there, i can Call Place_circles() again, and obtain the new stream:
def Place_circles(polyline, circle_space, cs, DoDrawCircle=True, surface=None):
Circle_list = []
curve = []
next_circle_space = None
dist = 0
for c in reversed(range(0, len(polyline))):
curve = []
if type(circle_space) == list:
iter_circle_space = iter(circle_space[c])
next_circle_space = next(iter_circle_space, circle_space[c][-1])
for p in reversed(range(len(polyline[c])-1)):
dist += math.sqrt((polyline[c][p+1][0] - polyline[c][p][0]) ** 2 + (polyline [c][p+1][1] - polyline[c][p][1]) ** 2)
if dist > (circle_space if type(circle_space) == int else next_circle_space):
dist = 0
curve.append(circles.circles(round(polyline[c][p][0]), round(polyline[c][p][1]), cs, DoDrawCircle, surface))
if type(circle_space) == list:
next_circle_space = next(iter_circle_space, circle_space[c][-1])
Circle_list.append(curve)
return Circle_list
the result is a stream with varying space between circles (so accelerating or decelerating), the only issue left to be fixed is pygame not updating the screen with the new set of circle after i call Place_circles(), but that's an issue i'm either going to try to fix myself or ask in another post
the final code for this feature can be found on my repo : https://github.com/Mrcubix/Osu-StreamGenerator/tree/Acceleration_v02
I am trying to align two videos using their utc timestamps.
for example:
video 1 timestamps = 1234.4321, 1234.4731, 1234.5432, 1234.5638, ...
video 2 timestamps = 1234.4843, 1234.5001, 1234.5632, 1234.5992, ...
I would like to align them so that the closest timestamps within a .0150s window are aligned without aligning two values from one array to one value in the second array.
example output:
video 1 timestamps = 1234.4321, 1234.4731, _________, 1234.5432, 1234.5638, _________, ...
video 2 timestamps = _________, 1234.4843, 1234.5001, _________, 1234.5632, 1234.5992, ...
Can someone help?
EDIT
There was a little confusion with the timestamps. The issue isn't that they simply need to be shifted once every two values. Hopefully this updated example will clear it up. Both examples are correct. A single solution should be able to solve both.
Example 2:
timestamp3 = 1590595834.6775, 1590595834.70479, 1590595834.73812, 1590595834.77163, 1590595834.80438
timestamp4 = 1590595835.70971, 1590595835.73674, 1590595835.7695, 1590595835.80338, 1590595835.83634
output:
timestamp3 = 1590595835.6775, 1590595835.70479, 1590595835.73812, 1590595835.77163, 1590595835.80438, _______________, ...
timestamp4 = _______________, 1590595835.70971, 1590595835.73674, 1590595835.7695, 1590595835.80338, 1590595835.83634, ...
Something like this:
timestamp3 = [1590595834.6775, 1590595834.70479, 1590595834.73812, 1590595834.77163, 1590595834.80438]
timestamp4 = [1590595834.70971, 1590595834.73674, 1590595834.7695, 1590595834.80338, 1590595834.83634]
len3 = len(timestamp3)
len4 = len(timestamp4)
ins = '_____________'
diff = 0.015
ii = jj = 0
while True:
if timestamp3[ii] < timestamp4[jj] - diff:
timestamp4.insert(jj, ins)
len4 += 1
elif timestamp4[ii] < timestamp3[jj] - diff:
timestamp3.insert(ii, ins)
len3 += 1
ii += 1
jj += 1
if ii == len3 or jj == len4:
if len3 > len4:
timestamp4.extend([ins]*(len3-len4))
elif len4 > len3:
timestamp3.extend([ins]*(len4-len3))
break
print(timestamp3)
print(timestamp4)
Gives:
[1590595834.6775, 1590595834.70479, 1590595834.73812, 1590595834.77163, 1590595834.80438, '_____________']
['_____________', 1590595834.70971, 1590595834.73674, 1590595834.7695, 1590595834.80338, 1590595834.83634]
I think this is what you mean:
timestamps1 = [1234.4321, 1234.4731, 1234.5432, 1234.5638]
timestamps2 = [1234.4843, 1234.5001, 1234.5632, 1234.5992]
index = len(timestamps1)
while index > 0:
timestamps1.insert(index,'_______')
index -= 2
timestamps2.insert(index,'_______')
print(timestamps1)
print(timestamps2)
Output:
[1234.4321, 1234.4731, '_______', 1234.5432, 1234.5638, '_______']
['_______', 1234.4843, 1234.5001, '_______', 1234.5632, 1234.5992]
I'm trading daily on Cryptocurrencies and would like to find which are the most desirable Cryptos for trading.
I have CSV file for every Crypto with the following fields:
Date Sell Buy
43051.23918 1925.16 1929.83
43051.23919 1925.12 1929.79
43051.23922 1925.12 1929.79
43051.23924 1926.16 1930.83
43051.23925 1926.12 1930.79
43051.23926 1926.12 1930.79
43051.23927 1950.96 1987.56
43051.23928 1190.90 1911.56
43051.23929 1926.12 1930.79
I would like to check:
How many quotes will end with profit:
for Buy positions - if one of the following Sells > current Buy.
for Sell positions - if one of the following Buys < current Sell.
How much time it would take to a theoretical position to become profitable.
What can be the profit potential.
I'm using the following code:
#converting from OLE to datetime
OLE_TIME_ZERO = dt.datetime(1899, 12, 30, 0, 0, 0)
def ole(oledt):
return OLE_TIME_ZERO + dt.timedelta(days=float(oledt))
#variables initialization
buy_time = ole(43031.57567) - ole(43031.57567)
sell_time = ole(43031.57567) - ole(43031.57567)
profit_buy_counter = 0
no_profit_buy_counter = 0
profit_sell_counter = 0
no_profit_sell_counter = 0
max_profit_buy_positions = 0
max_profit_buy_counter = 0
max_profit_sell_positions = 0
max_profit_sell_counter = 0
df = pd.read_csv("C:/P/Crypto/bitcoin_test_normal_276k.csv")
#comparing to max
for index, row in df.iterrows():
a = index + 1
df_slice = df[a:]
if df_slice["Sell"].max() - row["Buy"] > 0:
max_profit_buy_positions += df_slice["Sell"].max() - row["Buy"]
max_profit_buy_counter += 1
for index1, row1 in df_slice.iterrows():
if row["Buy"] < row1["Sell"] :
buy_time += ole(row1["Date"])- ole(row["Date"])
profit_buy_counter += 1
break
else:
no_profit_buy_counter += 1
#comparing to sell
for index, row in df.iterrows():
a = index + 1
df_slice = df[a:]
if row["Sell"] - df_slice["Buy"].min() > 0:
max_profit_sell_positions += row["Sell"] - df_slice["Buy"].min()
max_profit_sell_counter += 1
for index2, row2 in df_slice.iterrows():
if row["Sell"] > row2["Buy"] :
sell_time += ole(row2["Date"])- ole(row["Date"])
profit_sell_counter += 1
break
else:
no_profit_sell_counter += 1
num_rows = len(df.index)
buy_avg_time = buy_time/num_rows
sell_avg_time = sell_time/num_rows
if max_profit_buy_counter == 0:
avg_max_profit_buy = "There is no profitable buy positions"
else:
avg_max_profit_buy = max_profit_buy_positions/max_profit_buy_counter
if max_profit_sell_counter == 0:
avg_max_profit_sell = "There is no profitable sell positions"
else:
avg_max_profit_sell = max_profit_sell_positions/max_profit_sell_counter
The code works fine for 10K-20K lines but for a larger amount (276K) it take a long time (more than 10 hrs)
What can I do in order to improve it?
Is there any "Pythonic" way to compare each value in a data frame to all following values?
note - the dates in the CSV are in OLE so I need to convert it to Datetime.
File for testing:
Thanks for your comment.
Here you can find the file that I used:
First, I'd want to create the cumulative maximum/minimum values for Sell and Buy per row, so it's easy to compare to. pandas has cummax and cummin, but they go the wrong way. So we'll do:
df['Max Sell'] = df[::-1]['Sell'].cummax()[::-1]
df['Min Buy'] = df[::-1]['Buy'].cummin()[::-1]
Now, we can just compare each row:
df['Buy Profit'] = df['Max Sell'] - df['Buy']
df['Sell Profit'] = df['Sell'] - df['Min Buy']
I'm positive this isn't exactly what you want as I don't perfectly understand what you're trying to do, but hopefully it leads you in the right direction.
After comparing your function and mine, there is a slight difference, as your a is offset one off the index. Removing that offset, you'll see that my method produces the same results as yours, only in vastly shorter time:
for index, row in df.iterrows():
a = index
df_slice = df[a:]
assert (df_slice["Sell"].max() - row["Buy"]) == df['Max Sell'][a] - df['Buy'][a]
else:
print("All assertions passed!")
Note this will still take the very long time required by your function. Note that this can be fixed with shift, but I don't want to run your function for long enough to figure out what way to shift it.
I started learning Python < 2 weeks ago.
I'm trying to make a function to compute a 7 day moving average for data. Something wasn't going right so I tried it without the function.
moving_average = np.array([])
i = 0
for i in range(len(temp)-6):
sum_7 = np.array([])
avg_7 = 0
missing = 0
total = 7
j = 0
for j in range(i,i+7):
if pd.isnull(temp[j]):
total -= 1
missing += 1
if missing == 7:
moving_average = np.append(moving_average, np.nan)
break
if not pd.isnull(temp[j]):
sum_7 = np.append(sum_7, temp[j])
if j == (i+6):
avg_7 = sum(sum_7)/total
moving_average = np.append(moving_average, avg_7)
If I run this and look at the value of sum_7, it's just a single value in the numpy array which made all the moving_average values wrong. But if I remove the first for loop with the variable i and manually set i = 0 or any number in the range of the data set and run the exact same code from the inner for loop, sum_7 comes out as a length 7 numpy array. Originally, I just did sum += temp[j] but the same problem occurred, the total sum ended up as just the single value.
I've been staring at this trying to fix it for 3 hours and I'm clueless what's wrong. Originally I wrote the function in R so all I had to do was convert to python language and I don't know why sum_7 is coming up as a single value when there are two for loops. I tried to manually add an index variable to act as i to use it in the range(i, i+7) but got some weird error instead. I also don't know why that is.
https://gyazo.com/d900d1d7917074f336567b971c8a5cee
https://gyazo.com/132733df8bbdaf2847944d1be02e57d2
Hey you can using rolling() function and mean() function from pandas.
Link to the documentation :
https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.rolling.html
df['moving_avg'] = df['your_column'].rolling(7).mean()
This would give you some NaN values also, but that is a part of rolling mean because you don't have all past 7 data points for first 6 values.
Seems like you misindented the important line:
moving_average = np.array([])
i = 0
for i in range(len(temp)-6):
sum_7 = np.array([])
avg_7 = 0
missing = 0
total = 7
j = 0
for j in range(i,i+7):
if pd.isnull(temp[j]):
total -= 1
missing += 1
if missing == 7:
moving_average = np.append(moving_average, np.nan)
break
# The following condition should be indented one more level
if not pd.isnull(temp[j]):
sum_7 = np.append(sum_7, temp[j])
#^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
if j == (i+6):
# this ^ condition does not do what you meant
# you should use a flag instead
avg_7 = sum(sum_7)/total
moving_average = np.append(moving_average, avg_7)
Instead of a flag you can use a for-else construct, but this is not readable. Here's the relevant documentation.
Shorter way to do this:
moving_average = np.array([])
for i in range(len(temp)-6):
ngram_7 = [t for t in temp[i:i+7] if not pd.isnull(t)]
average = (sum(ngram_7) / len(ngram_7)) if ngram_7 else np.nan
moving_average = np.append(moving_average, average)
This could be refactored further:
def average(ngram):
valid = [t for t in temp[i:i+7] if not pd.isnull(t)]
if not valid:
return np.nan
return sum(valid) / len(valid)
def ngrams(seq, n):
for i in range(len(seq) - n):
yield seq[i:i+n]
moving_average = [average(k) for k in ngrams(temp, 7)]