Storing data in Redis then fetch that data [closed] - python

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I have an Excel file that contains four columns. I want to fetch this data and store it in MySQL. Later on I want fetch the data from here and store in Redis, then run a validation on it. I have already done the importing of data from Excel to Python.

You have to reshape your 4 column excel data to a 1 column data.
The redis client for Matlab/GNU Octave is doing this e.g.: https://github.com/markuman/go-redis/wiki/Data-Structure#arrays
Take care that in this example, Matlab/Octave are using Column-Major-Order.
Python is using Row-Major-Order: https://en.wikipedia.org/wiki/Row-major_order
So you have to save your 4 column X rows data as a row-major-order list in redis as a list (RPUSH).
example
given this excel sheet
using this python3 code
#!/usr/bin/python3
# -*- coding: utf-8 -*-
"""
Created on Tue Oct 20 23:02:53 2015
#author: markus
"""
import pandas as pd
import redis
# redis connection
r = redis.StrictRedis(host='localhost', port=6379, db=0)
# open the first worksheed
df = pd.read_excel('/home/markus/excel.xlsx',0)
# read in as a list
# [[1, 'two', 'python'], ['excel', 'redis', 'action']]
a = list(df.T.itertuples())
print("this is a, your excel list")
print(a)
for list in a:
for value in list:
r.rpush('myexceldata', str(value))
# read all back to python
b = r.lrange('myexceldata', '0', '-1')
print("A1 becomes 0, B1 becomes 3 ...")
print(b[3].decode('UTF-8'))
to save it serialized as a list in redis
127.0.0.1:6379> lrange myexceldata 0 -1
1) "1"
2) "two"
3) "python"
4) "excel"
5) "redis"
6) "action"
This is just one way to save a spreadsheet in redis. It always belong on your datastructure and what you're going to do with it.

Related

Convert a text file comprised of data separated by commas to an excel file using python [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 1 year ago.
Improve this question
I am pretty new to python and am working with some data from a weather station. The data is output to a text file where the data is separated by commas and new lines. I was wondering how I could write this data to a CSV file so I can process and graph the data in excel? I have never worked with text files before only CSV files so am unsure how to go about this.
Ideally, I would want to group the data by each day and calculate daily averages at the end of each set. I have included a sample of the dataset below. The data is in the following format: date, time, outside temp, outside humidity, dew point, wind speed, wind gust, avg wind bearing, rainfall rate, total rainfall, sea lvl pressure, rainfall counter, inside temp, inside humidity, latest gust, wind chill, heat index, uv index, solar rad, ET, annual et.
01/09/21,00:28,7.1,75,3.0,3.7,3.7,292,0.0,0.0,1025.8,81.9,17.1,44,3.7,4.6,7.1,0,0,0.00,0.00,3.0,0,0.0,292,0.0,0.0
01/09/21,00:58,7.0,75,2.9,5.1,5.1,248,0.0,0.0,1025.9,81.9,17.0,44,5.1,3.8,7.0,0,0,0.00,0.00,1.9,0,0.0,248,0.0,0.0
01/09/21,01:28,6.9,74,2.6,4.1,4.1,248,0.0,0.0,1025.8,81.9,17.0,44,4.1,4.1,6.9,0,0,0.00,0.00,2.5,0,0.0,248,0.0,0.0
01/09/21,01:58,6.9,74,2.6,4.8,4.8,292,0.0,0.0,1025.7,81.9,16.9,45,4.8,3.8,6.9,0,0,0.00,0.00,2.0,0,0.0,292,0.0,0.0
01/09/21,02:28,6.8,76,2.9,3.7,3.7,270,0.0,0.0,1025.5,81.9,16.9,45,3.7,4.2,6.8,0,0,0.00,0.00,2.7,0,0.0,270,0.0,0.0
01/09/21,02:58,6.7,76,2.8,3.1,3.1,270,0.0,0.0,1025.4,81.9,16.8,45,3.1,4.5,6.7,0,0,0.00,0.00,3.0,0,0.0,270,0.0,0.0
01/09/21,03:28,6.6,77,2.9,4.1,4.1,248,0.0,0.0,1025.5,81.9,16.8,45,4.1,3.8,6.6,0,0,0.00,0.00,2.2,0,0.0,248,0.0,0.0
01/09/21,03:58,6.3,77,2.6,2.7,2.7,292,0.0,0.0,1025.7,81.9,16.8,45,2.7,4.3,6.3,0,0,0.00,0.00,2.8,0,0.0,292,0.0,0.0
01/09/21,04:28,6.5,77,2.8,1.7,1.7,292,0.0,0.0,1025.7,81.9,16.7,45,1.7,5.4,6.5,0,0,0.00,0.00,3.8,0,0.0,292,0.0,0.0
01/09/21,04:58,6.4,77,2.7,1.7,1.7,292,0.0,0.0,1026.1,81.9,16.7,45,1.7,5.3,6.4,0,0,0.00,0.00,3.7,0,0.0,292,0.0,0.0
01/09/21,05:28,6.3,79,2.9,3.7,3.7,270,0.0,0.0,1026.5,81.9,16.7,45,3.7,3.6,6.3,0,0,0.00,0.00,2.2,0,0.0,270,0.0,0.0
01/09/21,05:58,6.3,80,3.1,3.7,3.7,292,0.0,0.0,1026.9,81.9,16.7,45,3.7,3.6,6.3,0,0,0.00,0.00,2.2,0,0.0,292,0.0,0.0
01/09/21,06:28,6.2,81,3.2,0.7,0.7,292,0.0,0.0,1027.3,81.9,16.6,45,0.7,6.2,6.2,0,0,0.00,0.00,4.2,0,0.0,292,0.0,0.0
01/09/21,06:58,6.4,80,3.2,1.7,1.7,270,0.0,0.0,1027.3,81.9,16.7,45,1.7,5.3,6.4,0,6,0.00,0.00,3.7,1,0.5,270,0.0,0.0
01/09/21,07:28,6.9,79,3.5,2.0,2.0,270,0.0,0.0,1027.5,81.9,16.7,45,2.0,5.6,6.9,0,19,0.00,0.00,4.1,40,0.5,270,0.0,0.0
01/09/21,07:58,7.5,77,3.7,2.7,2.7,270,0.0,0.0,1027.7,81.9,16.9,45,2.7,5.7,7.5,0,29,0.00,0.00,4.2,119,0.5,270,0.0,0.0
01/09/21,08:28,8.3,75,4.1,1.0,1.0,248,0.0,0.0,1027.6,81.9,17.3,43,1.0,8.3,8.3,0,36,0.00,0.00,6.3,212,0.5,248,0.0,0.0
01/09/21,08:58,9.1,74,4.7,2.0,2.0,270,0.0,0.0,1027.7,81.9,17.7,43,2.0,8.1,9.1,0,40,0.00,0.00,6.5,304,0.5,270,0.0,0.0
01/09/21,09:28,9.6,71,4.6,1.7,1.7,292,0.0,0.0,1027.7,81.9,18.0,43,1.7,9.0,9.6,0,44,0.00,0.00,7.2,391,0.5,292,0.0,0.0
01/09/21,09:58,10.8,67,4.9,6.8,6.8,248,0.0,0.0,1027.2,81.9,18.5,43,6.8,8.0,10.8,3,365,0.00,0.00,4.9,467,1.0,248,0.0,0.0
01/09/21,10:28,11.6,64,5.0,6.1,6.1,292,0.0,0.0,1027.0,81.9,18.8,43,6.1,9.2,11.6,3,402,0.00,0.00,6.2,531,1.5,292,0.0,0.0
01/09/21,10:58,12.1,63,5.3,7.1,7.1,314,0.0,0.0,1027.3,81.9,19.1,42,7.1,9.6,12.1,4,433,0.00,0.00,6.1,581,1.5,315,0.0,0.0
01/09/21,11:28,12.5,62,5.4,8.8,8.8,292,0.0,0.0,1027.1,81.9,19.4,42,8.8,9.7,12.5,4,492,0.00,0.00,5.3,616,2.0,292,0.0,0.0
01/09/21,11:58,12.6,62,5.5,8.2,8.2,270,0.0,0.0,1027.1,81.9,19.7,41,8.2,10.0,12.6,4,429,0.00,0.00,5.8,635,2.0,270,0.0,0.0
01/09/21,12:28,12.5,62,5.4,7.8,7.8,270,0.0,0.0,1026.8,81.9,20.0,40,7.8,10.0,12.5,1,124,0.00,0.00,6.0,637,2.0,270,0.0,0.0
01/09/21,12:58,13.0,60,5.4,4.8,4.8,292,0.0,0.0,1026.3,81.9,20.1,40,4.8,11.4,13.0,4,414,0.00,0.00,8.6,623,2.0,292,0.0,0.0
01/09/21,13:28,12.9,60,5.3,3.1,3.1,270,0.0,0.0,1025.9,81.9,20.2,40,3.1,11.9,12.9,0,59,0.00,0.00,9.7,592,2.0,270,0.0,0.0
01/09/21,13:58,12.8,60,5.2,3.1,3.1,270,0.0,0.0,1025.7,81.9,20.3,40,3.1,11.8,12.8,1,101,0.00,0.00,9.6,546,2.0,270,0.0,0.0
01/09/21,14:28,12.9,59,5.1,2.7,2.7,270,0.0,0.0,1025.5,81.9,20.3,40,2.7,12.1,12.9,0,47,0.00,0.00,9.9,485,2.0,270,0.0,0.0
01/09/21,14:58,12.8,59,5.0,3.1,3.1,270,0.0,0.0,1025.1,81.9,20.3,39,3.1,11.8,12.8,0,44,0.00,0.00,9.5,412,2.0,270,0.0,0.0
01/09/21,15:28,13.0,60,5.4,2.7,2.7,270,0.0,0.0,1025.2,81.9,20.5,39,2.7,12.3,13.0,0,40,0.00,0.00,10.1,328,2.0,270,0.0,0.0
01/09/21,15:58,12.5,60,4.9,5.1,5.1,270,0.0,0.0,1025.0,81.9,20.3,39,5.1,10.7,12.5,0,35,0.00,0.00,7.8,237,2.0,270,0.0,0.0
01/09/21,16:28,12.3,60,4.8,2.7,2.7,270,0.0,0.0,1025.2,81.9,19.9,40,2.7,11.4,12.3,0,29,0.00,0.00,9.2,143,2.0,270,0.0,0.0
01/09/21,16:58,12.0,62,4.9,1.7,1.7,292,0.0,0.0,1025.2,81.9,21.4,34,1.7,11.7,12.0,0,22,0.00,0.00,9.7,58,2.0,292,0.0,0.0
01/09/21,17:28,11.4,63,4.6,1.7,1.7,248,0.0,0.0,1025.4,81.9,19.7,40,1.7,11.1,11.4,0,11,0.00,0.00,9.0,5,2.5,248,0.0,0.0
01/09/21,17:58,10.8,67,4.9,2.7,2.7,292,0.0,0.0,1025.5,81.9,19.0,41,2.7,9.6,10.8,0,0,0.00,0.00,7.8,0,2.5,292,0.0,0.0
01/09/21,18:28,10.4,70,5.2,3.1,3.1,292,0.0,0.0,1025.9,81.9,18.7,43,3.1,8.9,10.4,0,0,0.00,0.00,7.1,0,2.5,292,0.0,0.0
01/09/21,18:58,10.0,73,5.4,1.7,1.7,270,0.0,0.0,1026.3,81.9,18.5,43,1.7,9.4,10.0,0,0,0.00,0.00,7.8,0,2.5,270,0.0,0.0
01/09/21,19:28,9.8,76,5.8,1.7,1.7,270,0.0,0.0,1026.6,81.9,18.3,44,1.7,9.2,9.8,0,0,0.00,0.00,7.6,0,2.5,270,0.0,0.0
01/09/21,19:58,9.4,78,5.8,1.7,1.7,292,0.0,0.0,1026.6,81.9,18.2,44,1.7,8.7,9.4,0,0,0.00,0.00,7.2,0,2.5,292,0.0,0.0
01/09/21,20:28,9.0,79,5.6,1.7,1.7,270,0.0,0.0,1026.8,81.9,18.1,44,1.7,8.3,9.0,0,0,0.00,0.00,6.8,0,2.5,270,0.0,0.0
01/09/21,20:58,8.8,79,5.4,2.0,2.0,270,0.0,0.0,1027.1,81.9,18.0,45,2.0,7.8,8.8,0,0,0.00,0.00,6.3,0,2.5,270,0.0,0.0
01/09/21,21:28,8.6,79,5.2,2.7,2.7,292,0.0,0.0,1027.2,81.9,17.9,45,2.7,7.0,8.6,0,0,0.00,0.00,5.6,0,2.5,292,0.0,0.0
01/09/21,21:58,8.3,79,4.9,2.0,2.0,270,0.0,0.0,1027.2,81.9,17.8,45,2.0,7.2,8.3,0,0,0.00,0.00,5.8,0,2.5,270,0.0,0.0
01/09/21,22:28,8.2,79,4.8,1.0,1.0,270,0.0,0.0,1027.3,81.9,17.7,46,1.0,8.2,8.2,0,0,0.00,0.00,6.3,0,2.5,270,0.0,0.0
01/09/21,22:58,7.6,80,4.4,0.0,0.0,0,0.0,0.0,1027.2,81.9,17.7,46,0.0,7.6,7.6,0,0,0.00,0.00,6.4,0,2.5,0,0.0,0.0
01/09/21,23:28,7.2,81,4.2,1.0,1.0,270,0.0,0.0,1027.5,81.9,17.5,46,1.0,7.2,7.2,0,0,0.00,0.00,5.2,0,2.5,270,0.0,0.0
01/09/21,23:58,6.0,84,3.5,0.0,0.0,0,0.0,0.0,1027.5,81.9,17.4,46,0.0,6.0,6.0,0,0,0.00,0.00,4.6,0,2.5,0,0.0,0.0
02/09/21,00:28,5.9,83,3.2,0.0,0.0,0,0.0,0.0,1027.7,81.9,17.4,46,0.0,5.9,5.9,0,0,0.00,0.00,4.4,0,0.0,0,0.0,0.0
02/09/21,00:58,5.1,86,3.0,0.0,0.0,0,0.0,0.0,1028.0,81.9,17.3,46,0.0,5.1,5.1,0,0,0.00,0.00,3.6,0,0.0,0,0.0,0.0
02/09/21,01:28,5.1,87,3.1,0.7,0.7,270,0.0,0.0,1027.9,81.9,17.2,46,0.7,5.1,5.1,0,0,0.00,0.00,3.1,0,0.0,270,0.0,0.0
02/09/21,01:58,4.9,86,2.8,0.0,0.0,0,0.0,0.0,1027.8,81.9,17.2,46,0.0,4.9,4.9,0,0,0.00,0.00,3.4,0,0.0,0,0.0,0.0
02/09/21,02:28,4.5,87,2.5,0.0,0.0,0,0.0,0.0,1027.7,81.9,17.1,46,0.0,4.5,4.5,0,0,0.00,0.00,2.9,0,0.0,0,0.0,0.0
02/09/21,02:58,3.4,90,1.9,0.0,0.0,0,0.0,0.0,1027.6,81.9,17.0,45,0.0,3.4,3.4,0,0,0.00,0.00,1.7,0,0.0,0,0.0,0.0
02/09/21,03:28,3.1,90,1.6,0.0,0.0,0,0.0,0.0,1028.0,81.9,16.9,46,0.0,3.1,3.1,0,0,0.00,0.00,1.4,0,0.0,0,0.0,0.0
02/09/21,03:58,3.4,89,1.8,0.7,0.7,270,0.0,0.0,1027.8,81.9,16.9,46,0.7,3.4,3.4,0,0,0.00,0.00,1.2,0,0.0,270,0.0,0.0
02/09/21,04:28,4.0,87,2.0,0.0,0.0,0,0.0,0.0,1027.8,81.9,16.9,46,0.0,4.0,4.0,0,0,0.00,0.00,2.3,0,0.0,0,0.0,0.0
02/09/21,04:58,3.1,89,1.5,0.7,0.7,270,0.0,0.0,1027.9,81.9,16.8,45,0.7,3.1,3.1,0,0,0.00,0.00,0.9,0,0.0,270,0.0,0.0
02/09/21,05:28,3.3,89,1.7,0.0,0.0,0,0.0,0.0,1028.0,81.9,16.8,45,0.0,3.3,3.3,0,0,0.00,0.00,1.6,0,0.0,0,0.0,0.0
02/09/21,05:58,3.3,90,1.8,0.7,0.7,270,0.0,0.0,1028.1,81.9,16.7,45,0.7,3.3,3.3,0,0,0.00,0.00,1.1,0,0.0,270,0.0,0.0
02/09/21,06:28,3.4,90,1.9,0.0,0.0,0,0.0,0.0,1028.4,81.9,16.7,45,0.0,3.4,3.4,0,0,0.00,0.00,1.7,0,0.0,0,0.0,0.0
02/09/21,06:58,3.3,90,1.8,0.0,0.0,0,0.0,0.0,1028.7,81.9,16.7,45,0.0,3.3,3.3,0,7,0.00,0.00,1.6,2,0.5,0,0.0,0.0
02/09/21,07:28,3.6,90,2.1,0.0,0.0,0,0.0,0.0,1028.7,81.9,16.8,45,0.0,3.6,3.6,0,19,0.00,0.00,1.9,44,0.5,0,0.0,0.0
02/09/21,07:58,5.0,87,3.0,0.0,0.0,0,0.0,0.0,1029.0,81.9,16.9,44,0.0,5.0,5.0,0,29,0.00,0.00,3.5,125,0.5,0,0.0,0.0
02/09/21,08:28,6.6,83,3.9,0.0,0.0,0,0.0,0.0,1029.4,81.9,17.2,43,0.0,6.6,6.6,0,38,0.00,0.00,5.3,218,0.5,0,0.0,0.0
02/09/21,08:58,8.4,76,4.4,0.0,0.0,0,0.0,0.0,1029.7,81.9,17.7,43,0.0,8.4,8.4,0,43,0.00,0.00,7.2,311,0.5,0,0.0,0.0
02/09/21,09:28,9.6,70,4.4,0.0,0.0,0,0.0,0.0,1029.8,81.9,18.1,43,0.0,9.6,9.6,0,48,0.00,0.00,8.4,397,0.5,0,0.0,0.0
02/09/21,09:58,12.3,65,5.9,0.7,0.7,270,0.0,0.0,1029.5,81.9,18.7,43,0.7,12.3,12.3,3,359,0.00,0.00,10.9,474,1.0,270,0.0,0.0
02/09/21,10:28,13.0,65,6.6,1.0,1.0,270,0.0,0.0,1029.9,81.9,19.0,42,1.0,13.0,13.0,4,411,0.00,0.00,11.5,538,1.5,270,0.0,0.0
02/09/21,10:58,14.6,63,7.6,0.0,0.0,0,0.0,0.0,1029.7,81.9,19.3,42,0.0,14.6,14.6,4,425,0.00,0.00,14.0,588,1.5,0,0.0,0.0
02/09/21,11:28,14.3,62,7.1,2.0,2.0,314,0.0,0.0,1029.9,81.9,19.7,41,2.0,14.2,14.3,4,480,0.00,0.00,12.2,623,2.0,315,0.0,0.0
02/09/21,11:58,14.5,62,7.3,1.7,1.7,314,0.0,0.0,1029.7,81.9,20.1,40,1.7,14.5,14.5,3,322,0.00,0.00,12.7,641,2.0,315,0.0,0.0
02/09/21,12:28,14.1,64,7.4,1.0,1.0,337,0.0,0.0,1029.9,81.9,20.3,40,1.0,14.1,14.1,1,164,0.00,0.00,12.8,643,2.0,338,0.0,0.0
02/09/21,12:58,14.6,60,6.9,1.0,1.0,337,0.0,0.0,1029.5,81.9,20.3,40,1.0,14.6,14.6,0,78,0.00,0.00,13.2,629,2.0,338,0.0,0.0
02/09/21,13:28,14.6,58,6.4,0.7,0.7,337,0.0,0.0,1029.3,81.9,20.5,40,0.7,14.6,14.6,1,101,0.00,0.00,13.3,598,2.0,338,0.0,0.0
02/09/21,13:58,15.1,57,6.6,1.7,1.7,337,0.0,0.0,1029.0,81.9,20.8,40,1.7,15.1,15.1,2,247,0.00,0.00,13.1,551,2.0,338,0.0,0.0
02/09/21,14:28,14.9,55,5.9,1.0,1.0,337,0.0,0.0,1029.1,81.9,20.8,40,1.0,14.9,14.9,1,143,0.00,0.00,13.3,491,2.0,338,0.0,0.0
02/09/21,14:58,14.5,60,6.8,1.0,1.0,337,0.0,0.0,1028.9,81.9,20.8,40,1.0,14.5,14.5,1,111,0.00,0.00,13.1,417,2.0,338,0.0,0.0
02/09/21,15:28,14.7,58,6.5,0.7,0.7,337,0.0,0.0,1029.0,81.9,21.0,39,0.7,14.7,14.7,0,55,0.00,0.00,13.4,333,2.0,338,0.0,0.0
02/09/21,15:58,13.9,67,7.9,0.7,0.7,337,0.0,0.0,1028.9,81.9,20.9,40,0.7,13.9,13.9,0,41,0.00,0.00,12.9,241,2.0,338,0.0,0.0
02/09/21,16:28,13.6,67,7.6,0.7,0.7,337,0.0,0.0,1029.3,81.9,20.5,40,0.7,13.6,13.6,0,32,0.00,0.00,12.5,147,2.0,338,0.0,0.0
02/09/21,16:58,13.2,68,7.4,1.0,1.0,337,0.0,0.0,1029.3,81.9,21.4,38,1.0,13.2,13.2,0,25,0.00,0.00,11.9,61,2.0,338,0.0,0.0
02/09/21,17:28,12.3,74,7.8,0.7,0.7,337,0.0,0.0,1029.7,81.9,20.3,40,0.7,12.3,12.3,0,12,0.00,0.00,11.3,6,2.5,338,0.0,0.0
02/09/21,17:58,12.0,73,7.3,0.7,0.7,337,0.0,0.0,1029.9,81.9,19.6,43,0.7,12.0,12.0,0,0,0.00,0.00,10.9,0,2.5,338,0.0,0.0
02/09/21,18:28,11.6,66,5.5,2.7,2.7,337,0.0,0.0,1030.2,81.9,19.2,44,2.7,10.6,11.6,0,0,0.00,0.00,8.7,0,2.5,338,0.0,0.0
02/09/21,18:58,11.0,68,5.3,1.7,1.7,292,0.0,0.0,1030.4,81.9,18.9,45,1.7,10.6,11.0,0,0,0.00,0.00,8.8,0,2.5,292,0.0,0.0
02/09/21,19:28,10.3,70,5.1,0.7,0.7,270,0.0,0.0,1031.1,81.9,18.8,46,0.7,10.3,10.3,0,0,0.00,0.00,8.7,0,2.5,270,0.0,0.0
02/09/21,19:58,10.5,73,5.9,0.7,0.7,225,0.0,0.0,1030.4,81.9,18.7,46,0.7,10.5,10.5,0,0,0.00,0.00,9.1,0,2.5,225,0.0,0.0
02/09/21,20:28,10.1,74,5.7,2.0,2.0,248,0.0,0.0,1031.0,81.9,18.5,46,2.0,9.3,10.1,0,0,0.00,0.00,7.7,0,2.5,248,0.0,0.0
02/09/21,20:58,10.0,75,5.8,2.0,2.0,292,0.0,0.0,1031.4,81.9,18.5,46,2.0,9.2,10.0,0,0,0.00,0.00,7.6,0,2.5,292,0.0,0.0
02/09/21,21:28,9.7,77,5.9,1.7,1.7,292,0.0,0.0,1031.7,81.9,18.3,46,1.7,9.1,9.7,0,0,0.00,0.00,7.6,0,2.5,292,0.0,0.0
02/09/21,21:58,9.4,79,5.9,1.0,1.0,292,0.0,0.0,1031.9,81.9,18.1,47,1.0,9.4,9.4,0,0,0.00,0.00,7.8,0,2.5,292,0.0,0.0
02/09/21,22:28,9.5,80,6.2,1.0,1.0,292,0.0,0.0,1031.8,81.9,18.1,47,1.0,9.5,9.5,0,0,0.00,0.00,7.9,0,2.5,292,0.0,0.0
02/09/21,22:58,9.4,80,6.1,2.0,2.0,292,0.0,0.0,1032.2,81.9,18.0,47,2.0,8.5,9.4,0,0,0.00,0.00,7.1,0,2.5,292,0.0,0.0
02/09/21,23:28,9.3,79,5.8,1.7,1.7,292,0.0,0.0,1032.2,81.9,18.0,47,1.7,8.6,9.3,0,0,0.00,0.00,7.2,0,2.5,292,0.0,0.0
02/09/21,23:58,8.9,80,5.6,0.0,0.0,0,0.0,0.0,1032.2,81.9,18.0,48,0.0,8.9,8.9,0,0,0.00,0.00,7.9,0,2.5,0,0.0,0.0
03/09/21,00:28,8.5,81,5.4,0.0,0.0,0,0.0,0.0,1032.5,81.9,17.9,48,0.0,8.5,8.5,0,0,0.00,0.00,7.5,0,0.0,0,0.0,0.0
03/09/21,00:58,7.6,85,5.2,0.0,0.0,0,0.0,0.0,1032.6,81.9,17.8,48,0.0,7.6,7.6,0,0,0.00,0.00,6.5,0,0.0,0,0.0,0.0
03/09/21,01:28,7.6,85,5.2,0.0,0.0,0,0.0,0.0,1032.6,81.9,17.7,48,0.0,7.6,7.6,0,0,0.00,0.00,6.5,0,0.0,0,0.0,0.0
03/09/21,01:58,7.6,85,5.2,0.0,0.0,0,0.0,0.0,1032.5,81.9,17.7,48,0.0,7.6,7.6,0,0,0.00,0.00,6.5,0,0.0,0,0.0,0.0
03/09/21,02:28,6.4,88,4.6,0.0,0.0,0,0.0,0.0,1032.4,81.9,17.6,48,0.0,6.4,6.4,0,0,0.00,0.00,5.2,0,0.0,0,0.0,0.0
03/09/21,02:58,5.8,89,4.1,0.0,0.0,0,0.0,0.0,1032.2,81.9,17.6,48,0.0,5.8,5.8,0,0,0.00,0.00,4.5,0,0.0,0,0.0,0.0
03/09/21,03:28,5.5,91,4.1,0.0,0.0,0,0.0,0.0,1032.4,81.9,17.4,48,0.0,5.5,5.5,0,0,0.00,0.00,4.2,0,0.0,0,0.0,0.0
03/09/21,03:58,5.4,90,3.9,0.0,0.0,0,0.0,0.0,1032.3,81.9,17.3,48,0.0,5.4,5.4,0,0,0.00,0.00,4.1,0,0.0,0,0.0,0.0
03/09/21,04:28,5.5,90,4.0,0.7,0.7,292,0.0,0.0,1032.5,81.9,17.3,48,0.7,5.5,5.5,0,0,0.00,0.00,3.7,0,0.0,292,0.0,0.0
03/09/21,04:58,5.1,90,3.6,0.0,0.0,0,0.0,0.0,1032.7,81.9,17.2,48,0.0,5.1,5.1,0,0,0.00,0.00,3.7,0,0.0,0,0.0,0.0
03/09/21,05:28,4.1,92,2.9,0.0,0.0,0,0.0,0.0,1032.9,81.9,17.2,48,0.0,4.1,4.1,0,0,0.00,0.00,2.6,0,0.0,0,0.0,0.0
03/09/21,05:58,4.0,93,3.0,0.0,0.0,0,0.0,0.0,1033.2,81.9,17.1,48,0.0,4.0,4.0,0,0,0.00,0.00,2.5,0,0.0,0,0.0,0.0
03/09/21,06:28,3.8,92,2.6,0.0,0.0,0,0.0,0.0,1033.1,81.9,16.9,49,0.0,3.8,3.8,0,0,0.00,0.00,2.2,0,0.0,0,0.0,0.0
03/09/21,06:58,4.3,91,3.0,1.0,1.0,292,0.0,0.0,1033.7,81.9,16.9,49,1.0,4.3,4.3,0,6,0.00,0.00,2.1,2,0.5,292,0.0,0.0
03/09/21,07:28,4.5,91,3.2,0.0,0.0,0,0.0,0.0,1033.5,81.9,17.0,48,0.0,4.5,4.5,0,17,0.00,0.00,3.0,47,0.5,0,0.0,0.0
03/09/21,07:58,5.6,87,3.6,1.0,1.0,292,0.0,0.0,1034.1,81.9,17.2,47,1.0,5.6,5.6,0,25,0.00,0.00,3.5,130,0.5,292,0.0,0.0
03/09/21,08:28,6.7,83,4.0,1.7,1.7,270,0.0,0.0,1034.2,81.9,17.6,46,1.7,5.6,6.7,0,31,0.00,0.00,4.2,224,0.5,270,0.0,0.0
03/09/21,08:58,7.9,80,4.7,0.7,0.7,225,0.0,0.0,1034.4,81.9,17.9,45,0.7,7.9,7.9,0,36,0.00,0.00,6.2,317,0.5,225,0.0,0.0
03/09/21,09:28,9.4,77,5.6,1.0,1.0,225,0.0,0.0,1034.2,81.9,18.5,45,1.0,9.4,9.4,0,45,0.00,0.00,7.7,404,0.5,225,0.0,0.0
03/09/21,09:58,12.5,72,7.6,0.0,0.0,0,0.0,0.0,1034.3,81.9,18.9,44,0.0,12.5,12.5,3,368,0.00,0.00,11.9,481,1.0,0,0.0,0.0
03/09/21,10:28,14.7,63,7.7,0.0,0.0,0,0.0,0.0,1034.2,81.9,19.3,43,0.0,14.7,14.7,4,432,0.00,0.00,14.2,545,1.5,0,0.0,0.0
03/09/21,10:58,15.2,53,5.7,3.1,3.1,248,0.0,0.0,1033.9,81.9,19.6,43,3.1,14.7,15.2,4,432,0.00,0.00,12.0,595,1.5,248,0.0,0.0
03/09/21,11:28,15.7,46,4.1,2.7,2.7,203,0.0,0.0,1033.8,81.9,20.0,41,2.7,15.5,15.7,5,515,0.00,0.00,12.5,629,2.0,202,0.0,0.0
03/09/21,11:58,15.9,45,4.0,3.7,3.7,203,0.0,0.0,1033.4,81.9,20.3,41,3.7,15.3,15.9,3,391,0.00,0.00,12.0,648,2.0,202,0.0,0.0
03/09/21,12:28,15.9,44,3.7,0.7,0.7,203,0.0,0.0,1032.8,81.9,20.7,40,0.7,15.9,15.9,1,114,0.00,0.00,14.0,650,2.0,202,0.0,0.0
03/09/21,12:58,16.1,47,4.8,1.0,1.0,270,0.0,0.0,1032.6,81.9,20.9,40,1.0,16.1,16.1,2,227,0.00,0.00,14.2,635,2.0,270,0.0,0.0
03/09/21,13:28,16.7,49,5.9,1.0,1.0,292,0.0,0.0,1032.4,81.9,21.0,39,1.0,16.7,16.7,1,159,0.00,0.00,15.1,604,2.0,292,0.0,0.0
03/09/21,13:58,15.9,48,4.9,2.0,2.0,292,0.0,0.0,1032.3,81.9,21.1,38,2.0,15.9,15.9,0,77,0.00,0.00,13.4,557,2.0,292,0.0,0.0
03/09/21,14:29,15.8,48,4.8,0.7,0.7,314,0.0,0.0,1032.0,81.9,21.1,38,0.7,15.8,15.8,0,55,0.00,0.00,14.1,494,2.0,315,0.0,0.0
03/09/21,14:59,15.9,49,5.2,1.7,1.7,314,0.0,0.0,1031.8,81.9,21.1,37,1.7,15.9,15.9,0,47,0.00,0.00,13.6,419,2.0,315,0.0,0.0
03/09/21,15:29,16.1,48,5.1,3.1,3.1,314,0.0,0.0,1031.8,81.9,21.4,37,3.1,15.8,16.1,1,143,0.00,0.00,12.8,334,2.0,315,0.0,0.0
03/09/21,15:59,15.4,52,5.6,0.0,0.0,0,0.0,0.0,1031.6,81.9,21.2,37,0.0,15.4,15.4,0,38,0.00,0.00,14.4,242,2.0,0,0.0,0.0
03/09/21,16:29,15.1,52,5.3,4.8,4.8,292,0.0,0.0,1031.9,81.9,20.9,39,4.8,14.0,15.1,0,31,0.00,0.00,10.7,148,2.0,292,0.0,0.0
03/09/21,16:59,14.2,57,5.8,3.7,3.7,314,0.0,0.0,1031.5,81.9,20.8,39,3.7,13.3,14.2,0,24,0.00,0.00,10.6,61,2.0,315,0.0,0.0
03/09/21,17:29,13.2,61,5.8,2.7,2.7,314,0.0,0.0,1032.0,81.9,20.3,40,2.7,12.5,13.2,0,11,0.00,0.00,10.4,6,2.5,315,0.0,0.0
03/09/21,17:59,12.0,66,5.8,2.7,2.7,314,0.0,0.0,1032.2,81.9,19.9,41,2.7,11.1,12.0,0,0,0.00,0.00,9.2,0,2.5,315,0.0,0.0
03/09/21,18:29,10.8,71,5.7,0.0,0.0,0,0.0,0.0,1032.4,81.9,19.4,42,0.0,10.8,10.8,0,0,0.00,0.00,9.8,0,2.5,0,0.0,0.0
03/09/21,18:59,9.4,76,5.4,0.0,0.0,0,0.0,0.0,1032.5,81.9,19.2,42,0.0,9.4,9.4,0,0,0.00,0.00,8.4,0,2.5,0,0.0,0.0
03/09/21,19:29,8.1,81,5.0,0.0,0.0,0,0.0,0.0,1032.8,81.9,18.9,44,0.0,8.1,8.1,0,0,0.00,0.00,7.0,0,2.5,0,0.0,0.0
03/09/21,19:59,7.5,83,4.8,0.0,0.0,0,0.0,0.0,1033.0,81.9,18.8,44,0.0,7.5,7.5,0,0,0.00,0.00,6.3,0,2.5,0,0.0,0.0
03/09/21,20:29,6.6,85,4.3,0.0,0.0,0,0.0,0.0,1033.0,81.9,18.6,45,0.0,6.6,6.6,0,0,0.00,0.00,5.3,0,2.5,0,0.0,0.0
03/09/21,20:59,6.0,86,3.8,0.0,0.0,0,0.0,0.0,1033.1,81.9,18.5,45,0.0,6.0,6.0,0,0,0.00,0.00,4.7,0,2.5,0,0.0,0.0
03/09/21,21:29,5.5,87,3.5,0.0,0.0,0,0.0,0.0,1033.2,81.9,18.3,45,0.0,5.5,5.5,0,0,0.00,0.00,4.1,0,2.5,0,0.0,0.0
03/09/21,21:59,5.4,87,3.4,0.0,0.0,0,0.0,0.0,1033.6,81.9,18.1,45,0.0,5.4,5.4,0,0,0.00,0.00,4.0,0,2.5,0,0.0,0.0
03/09/21,22:29,5.2,87,3.2,1.0,1.0,314,0.0,0.0,1033.4,81.9,18.0,45,1.0,5.2,5.2,0,0,0.00,0.00,3.0,0,2.5,315,0.0,0.0
03/09/21,22:59,5.9,86,3.7,0.7,0.7,314,0.0,0.0,1033.7,81.9,18.0,45,0.7,5.9,5.9,0,0,0.00,0.00,4.0,0,2.5,315,0.0,0.0
03/09/21,23:29,6.6,85,4.3,0.7,0.7,314,0.0,0.0,1033.6,81.9,18.0,45,0.7,6.6,6.6,0,0,0.00,0.00,4.8,0,2.5,315,0.0,0.0
03/09/21,23:59,6.8,85,4.5,0.0,0.0,0,0.0,0.0,1033.4,81.9,18.0,45,0.0,6.8,6.8,0,0,0.00,0.00,5.6,0,2.5,0,0.0,0.0
04/09/21,00:29,6.5,87,4.5,0.0,0.0,0,0.0,0.0,1033.6,81.9,18.0,45,0.0,6.5,6.5,0,0,0.00,0.00,5.3,0,0.0,0,0.0,0.0
04/09/21,00:59,6.6,87,4.6,0.0,0.0,0,0.0,0.0,1033.7,81.9,18.0,45,0.0,6.6,6.6,0,0,0.00,0.00,5.4,0,0.0,0,0.0,0.0
04/09/21,01:29,7.0,86,4.8,0.0,0.0,0,0.0,0.0,1033.6,81.9,18.0,45,0.0,7.0,7.0,0,0,0.00,0.00,5.8,0,0.0,0,0.0,0.0
04/09/21,01:59,7.6,84,5.1,0.0,0.0,0,0.0,0.0,1033.4,81.9,17.9,45,0.0,7.6,7.6,0,0,0.00,0.00,6.5,0,0.0,0,0.0,0.0
You can use a library called pandas for this:
import pandas as pd # library you need to download and install, e.g. anaconda or pip
df = pd.read_csv('path\to\your\file.csv')
You can use pandas:
import pandas as pd
info = [
["01/09/21","00:28",7.1,75,3.0,3.7,3.7,292,0.0,0.0,1025.8,81.9,17.1,44,3.7,4.6,7.1,0,0,0.00,0.00,3.0,0,0.0,292,0.0,0.0],
["01/09/21","00:58",7.0,75,2.9,5.1,5.1,248,0.0,0.0,1025.9,81.9,17.0,44,5.1,3.8,7.0,0,0,0.00,0.00,1.9,0,0.0,248,0.0,0.0],
["01/09/21","01:28",6.9,74,2.6,4.1,4.1,248,0.0,0.0,1025.8,81.9,17.0,44,4.1,4.1,6.9,0,0,0.00,0.00,2.5,0,0.0,248,0.0,0.0],
["01/09/21","01:58",6.9,74,2.6,4.8,4.8,292,0.0,0.0,1025.7,81.9,16.9,45,4.8,3.8,6.9,0,0,0.00,0.00,2.0,0,0.0,292,0.0,0.0],
["01/09/21","02:28",6.8,76,2.9,3.7,3.7,270,0.0,0.0,1025.5,81.9,16.9,45,3.7,4.2,6.8,0,0,0.00,0.00,2.7,0,0.0,270,0.0,0.0],
["01/09/21","02:58",6.7,76,2.8,3.1,3.1,270,0.0,0.0,1025.4,81.9,16.8,45,3.1,4.5,6.7,0,0,0.00,0.00,3.0,0,0.0,270,0.0,0.0],
]
df = pd.DataFrame(info)
print(df.head())
df.to_csv("example.csv")
This should give you the results you are looking for.

How to separate a CVS column by position in Python [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 1 year ago.
Improve this question
I have data that I want to separate into 3 columns form the one column in a CVS file.
The original file looks like this:
0400000006340000000000965871
0700000007850000000000336487
0100000003360000000000444444
I would like to separate the columns to resemble the list below, while still preserving the leading zeros:
04 0000000634 0000000000965871
07 0000000785 0000000000336487
01 0000000336 0000000000444444
I can upload the file onto Python, but I don't know which Delimiter or positioning I have to use. The code I have so far:
import pandas as pd
df = pd.read_cvs('new_numbers.txt', header=None)
Thank you for the help.
Use the pandas read_fwf() method - which stands for "fixed-width format":
pd.read_fwf('new_numbers.txt', widths=[2, 10, 16], header=None)
which will drop the leading zeroes:
0 1 2
0 4 634 965871
1 7 785 336487
2 1 336 444444
To keep them, specify the dtype as strings with object:
pd.read_fwf('new_numbers.txt', widths=[2, 10, 16], dtype=object, header=None)
Output:
0 1 2
0 04 0000000634 0000000000965871
1 07 0000000785 0000000000336487
2 01 0000000336 0000000000444444
It looks like there is no delimiter and you are using fixed lengths.
Access fixed lengths by their position in a list notation.
So for instance:
str1 = "0400000006340000000000965871"
str1A = str1[:2]
str1B = str1[3:14]
str1C = str1[14:]
I wouldn't particularly bother with pandas for it unless you need a dataframe out the far end.
You don't need pandas to load your text file and read its content (and also, you aren't loading a csv file).
with open("new_numbers.txt") as f:
lines = f.readlines()
What I suggest you is to use re module.
import re
PATTERN = re.compile(r"(0*[1-9]+)(0*[1-9]+)(0*[1-9]+)")
You can check here the result of this expression on your example.
Then you need to get matches from your lines, and join them with a space.
matches = []
for line in lines:
match = PATTERN.match(line)
first, second, third = match.group(1, 2, 3)
matches.append(" ".join([first, second, third]))
At the end, matches will be an array of space-separated numbers (with leading zeros).
At this point you can write them to another file, or do whatever you need to do with it.
towrite = "\n".join(matches)
with open("output.txt", "w") as f:
f.write(towrite)

How to create a nested dictionary in python [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
Hello all just learning dictionary in python. I have few data please let me know how to create a nested dictionary. Data are available below with duplicate values in excel file. Please do explain using for loop
Name Account Dept
John AC Lab1
Dev AC Lab1
Dilip AC Lab1,Lab2
Sat AC Lab1,Lab2
Dina AC Lab3
Surez AC Lab4
I need the result in below format:
{
'AC': {
'Lab1': ['John', 'Dev', 'Dilip', 'Sat'],
'Lab2': ['Dilip','Sat'],
'Lab3': ['Dina'],
'Lab4': ['Surez']
}
}
Something like this should get you closer to an answer but I'd need your input file to optimize it:
import xlrd
from collections import defaultdict
wb = xlrd.open_workbook("<your filename>")
sheet_names = wb.sheet_names()
sheet = wb.sheet_by_name(sheet_names[0])
d = defaultdict(defaultdict(list))
for row_idx in range(0, sheet.nrows):
cell_obj_0 = sheet.cell(row_idx, 0)
cell_obj_1 = sheet.cell(row_idx, 1)
cell_obj_2 = sheet.cell(row_idx, 2)
for lab in cell_obj_2.split(","):
d[cell_obj_1][lab].append(cell_obj_0)

Get stock data problems [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
I was making getting stock data file and output was just
In Progress
[]
what's the matter?
import quandl
from datetime import datetime as dt
def get_stock_data(stock_ticker):
print("In Progress")
start_date = dt(2019, 1, 1)
end_date = dt.now()
quandl_api_key = "tJDGptkdfqwjYi123RVV"
quandl.ApiConfig.api_key = quandl_api_key
source = "WIKI/" + stock_ticker
data = quandl.get(source, start_date=str(start_date), end_date=str(end_date))
data = data[["Open", "High", "Low", "Volume", "Close"]].values
print(data)
return data
get_stock_data("AAPL")
There's nothing wrong with your code. However recent stock data is a Premium product from Quandl and I presume you are just on the free subscription, hence your dataframe comes back empty. If you change the dates to 2017, you will get some results but that's as far as it goes on the free subscription it seems.

How can I organize my data [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
Let's say I have a list composed of a client number, number of a store, first name, last name and address which in this format:
11, 2, Lisa, Anderson, NewYork
13, 4, John, Smith, Alabama
54, 2, Lucy, Nicholsson, NewYork
etc.
What is the best way for me to organize this data within python so I can easily access it so I can do stuff like input a client number and have an output with the location, and other derivatives of stuff like that.
You can use pandas. It provides database-like (or spreadsheet-like) tables which can be used to store and query data. Like this:
import pandas as pd
df = pd.DataFrame([
[11, 2, 'Lisa', 'Anderson', 'NewYork'],
[13, 4, 'John', 'Smith', 'Alabama'],
[54, 2, 'Lucy', 'Nicholsson', 'NewYork']
], columns = ['cl_number', 'store', 'first_name', 'last_name','address'])
df.index=df["cl_number"]
# rows will be indexed with cl_number
df.loc[11]
# returns a record of client with number 11
df.loc[11, 'address']
# returns address of a client with number 11
df[df['address'] == 'NewYork']
# returns a list of all clients with address 'NewYork'
However, you may also need full-featured database (see SQLite, for example).
If your data is reasonably consistent and there isn't so much that you want a fully-fledged database, you can get quite far with a namedtuple:
from collections import namedtuple
Client = namedtuple('Client', ('id', 'storeno', 'first_name', 'last_name',
'location'))
# Read the data
with open('db.csv') as fi:
rows = fi.readlines()
db = {}
for row in rows:
f= row.split(',')
db[int(f[0])] = Client(int(f[0]), int(f[1]), f[2].strip(),
f[3].strip(), f[4].strip())
def find(**kwargs):
"""Return a list of all Clients matching a given set of criteria."""
matches = []
for client in db.values():
for k,v in kwargs.items():
if getattr(client, k) != v:
break
else:
matches.append(client)
return matches
# Client with ID 11
print db[11]
# All clients in New York
print find(location='NewYork')
# All clients called Lisa in New York
print find(location='NewYork', first_name='Lisa')

Categories