Changing list value inside of a json object - python

I find this a bit hard to explain, however.
I have a very big json object I got as a response from an API. This has about a years worth of data inside of each key. There are 3 keys prices, market_cap, total_volume.
Each key consists of the same; a list with N lists inside which store a UNIX timestamp and a price.
N = 24 per day (in this case my data is a years worth so 365 * 24)
Now I need to convert all of these UNIX timestamps back to readable datetime.
I want to do this for each key as each key has one large list with in there more lists which consists of a unix timestamp and another int.
I want the output to be either another dict or the same dict altered to then later use this data.
Json object
{'prices': [[1581984000000, 9723.605026422496], [1582070400000, 10133.817417084678], [1582156800000, 9618.409248250406], [1582243200000, 9608.16918128246], [1582329600000, 9673.011800270346], [1582416000000, 9658.606058375473], [1582502400000, 9946.716123467522], [1582588800000, 9670.9608064677], [1582675200000, 9345.342701734848], [1582761600000, 8801.657129472504], [1582848000000, 8781.175282202359], [1582934400000, 8717.331431359424], [1583020800000, 8552.989118581636], [1583107200000, 8567.802248679225], [1583193600000, 8905.876104262194], [1583280000000, 8756.718576742549], [1583366400000, 8758.646993191916], [1583452800000, 9038.870323233425], [1583539200000, 9135.84506603321], [1583625600000, 8902.20195010789], [1583712000000, 8041.365538071835], [1583798400000, 7921.33200691072], [1583884800000, 7906.731528510632], [1583971200000, 7935.522040170545], [1584057600000, 5142.990459018316], [1584144000000, 5542.819542373153], [1584230400000, 5214.189112383918], [1584316800000, 5397.93335743919], [1584403200000, 5032.501351487721], [1584489600000, 5389.41577503409], [1584576000000, 5376.280747845438], [1584662400000, 6170.200600237675], [1584748800000, 6195.603505257629], [1584835200000, 6145.770891252399], [1584921600000, 5859.647430299388], [1585008000000, 6456.347485963419], [1585094400000, 6730.173782371188], [1585180800000, 6695.9006183977235], [1585267200000, 6765.56207892991], [1585353600000, 6397.826328325604], [1585440000000, 6255.000398693634], [1585526400000, 5915.337154791903], [1585612800000, 6403.141235565223], [1585699200000, 6421.70541388854], [1585785600000, 6640.797666310131], [1585872000000, 6807.897017890566], [1585958400000, 6732.852018394612], [1586044800000, 6859.424923721944], [1586131200000, 6788.048272605917], [1586217600000, 7297.635558289496], [1586304000000, 7196.782202442051], [1586390400000, 7342.291601148024], [1586476800000, 7294.488875121554], [1586563200000, 6864.694257006497], [1586649600000, 6878.781212589853], [1586736000000, 6913.158787469097], [1586822400000, 6857.538537511484], [1586908800000, 6860.17853570111], [1586995200000, 6629.431738031291], [1587081600000, 7059.92622475854], [1587168000000, 7035.261503989225], [1587254400000, 7242.5109294929825], [1587340800000, 7127.511949689152], [1587427200000, 6856.456278354705], [1587513600000, 6842.038597634602], [1587600000000, 7109.995291181778], [1587686400000, 7382.793144116689], [1587772800000, 7495.393587498606], [1587859200000, 7538.557687279841], [1587945600000, 7683.867415083342], [1588032000000, 7774.281554448049], [1588118400000, 7758.230255185947], [1588204800000, 8744.430287016561], [1588291200000, 8610.63580374089], [1588377600000, 8824.818413551968], [1588464000000, 8966.307014689282], [1588550400000, 8888.671912686868], [1588636800000, 8884.407813577056], [1588723200000, 9003.240557621584], [1588809600000, 9144.68703972007], [1588896000000, 9959.166416261767], [1588982400000, 9821.81131529702], [1589068800000, 9566.777187205966], [1589155200000, 8752.617087745832], [1589241600000, 8604.75159101983], [1589328000000, 8788.466749414652], [1589414400000, 9283.08601265873], [1589500800000, 9796.494527024528], [1589587200000, 9309.29535940684], [1589673600000, 9375.29710843331], [1589760000000, 9666.32719340344], [1589846400000, 9708.439858793108], [1589932800000, 9760.198937162193], [1590019200000, 9526.50759300584], [1590105600000, 9059.962506871727], [1590192000000, 9131.767275081993], [1590278400000, 9170.361063506127], [1590364800000, 8731.848525870651], [1590451200000, 8883.691769863415], [1590537600000, 8839.130663273247], [1590624000000, 9174.118563996424], [1590710400000, 9546.04563503715], [1590796800000, 9427.120373393418], [1590883200000, 9662.70587254818], [1590969600000, 9466.961781429516], [1591056000000, 10167.93069332851], [1591142400000, 9515.243858655718], [1591228800000, 9645.227869360308], [1591315200000, 9776.20299178848], [1591401600000, 9636.965527050057], [1591488000000, 9662.858709002241], [1591574400000, 9738.603356828593], [1591660800000, 9773.02951309516], [1591747200000, 9767.00531665552], [1591833600000, 9874.898681832236], [1591920000000, 9325.996856202635], [1592006400000, 9469.533297509908], [1592092800000, 9469.473456163696], [1592179200000, 9345.960907722063], [1592265600000, 9431.719262201745], [1592352000000, 9524.92661691022], [1592438400000, 9463.361414311787], [1592524800000, 9399.767217129216], [1592611200000, 9312.780104497786], [1592697600000, 9360.247968201687], [1592784000000, 9298.360829121417], [1592870400000, 9678.683208975835], [1592956800000, 9624.684291831398], [1593043200000, 9288.061774486938], [1593129600000, 9258.667161007706], [1593216000000, 9166.486360416233], [1593302400000, 9013.90556467614], [1593388800000, 9139.903276297824], [1593475200000, 9185.166540651147], [1593561600000, 9149.721996758017], [1593648000000, 9230.672998590804], [1593734400000, 9094.318072166905], [1593820800000, 9071.3850427828], [1593907200000, 9132.908369533492], [1593993600000, 9087.407312582163], [1594080000000, 9342.376492626678], [1594166400000, 9253.630980242333], [1594252800000, 9432.172515827939], [1594339200000, 9235.716302064242], [1594425600000, 9282.913638839902], [1594512000000, 9234.314674712627], [1594598400000, 9297.479635872663], [1594684800000, 9240.76251972468], [1594771200000, 9247.060695963813], [1594857600000, 9203.371435179699], [1594944000000, 9136.483376363976], [1595030400000, 9156.276583115488], [1595116800000, 9168.402736564132], [1595203200000, 9202.615839500108], [1595289600000, 9163.159654576915], [1595376000000, 9384.379751903267], [1595462400000, 9514.304987626969], [1595548800000, 9589.81771944117], [1595635200000, 9535.93879573746], [1595721600000, 9691.825138917147], [1595808000000, 9925.751397476346], [1595894400000, 10962.258481207355], [1595980800000, 10904.916526918994], [1596067200000, 11093.612240442404], [1596153600000, 11116.307163685275], [1596240000000, 11325.5515272739], [1596326400000, 11812.094307268515], [1596412800000, 11066.306240590267], [1596499200000, 11230.907762749297], [1596585600000, 11181.917508034885], [1596672000000, 11719.26352395155], [1596758400000, 11768.127742240009], [1596844800000, 11571.487980683192], [1596931200000, 11739.131006414418], [1597017600000, 11682.851469154939], [1597104000000, 11862.938012702563], [1597190400000, 11398.671060896633], [1597276800000, 11579.867951602135], [1597363200000, 11817.164038803397], [1597449600000, 11777.391322489924], [1597536000000, 11864.905810156475], [1597622400000, 11901.776488302461], [1597708800000, 12272.465808160425], [1597795200000, 11949.610970628193], [1597881600000, 11733.278970862082], [1597968000000, 11861.83657727968], [1598054400000, 11515.124298729217], [1598140800000, 11676.385305081287], [1598227200000, 11647.928120934363], [1598313600000, 11758.828120368864], [1598400000000, 11350.753473213], [1598486400000, 11465.002564032086], [1598572800000, 11300.398363810944], [1598659200000, 11519.118388160729], [1598745600000, 11481.481823317012], [1598832000000, 11701.004008657852], [1598918400000, 11672.324104943627], [1599004800000, 11895.225345345636], [1599091200000, 11418.254756916149], [1599177600000, 10197.459822768922], [1599264000000, 10484.470392265588], [1599350400000, 10177.789718049991], [1599436800000, 10260.0177277544], [1599523200000, 10359.445216989981], [1599609600000, 10125.014956069688], [1599696000000, 10230.154699360752], [1599782400000, 10342.159391205681], [1599868800000, 10378.223044584596], [1599955200000, 10439.38467226404], [1600041600000, 10328.866065987393], [1600128000000, 10661.096235144483], [1600214400000, 10787.58020807624], [1600300800000, 10952.249969107099], [1600387200000, 10937.996396960929], [1600473600000, 10927.150310293275], [1600560000000, 11083.99836119821]]}
had to decrease the size of the object due to stackoverflow not allowing the amount of characters of my post, had to remove 2 keys from the object.
My for loop to just get the lists from the key prices
for c in obj['prices']:
print(c)
# [1581905057626, 9860.162323271938]
# [1581908672576, 9853.961230334764]
# [1581912178268, 9803.988325218852]
# etc..
Now how would I go about changing the first value and then storing this all again in a new object with the same structure.


you don't need new object
from datetime import datetime
obj = {
'prices': [[1581984000000, 9723.605026422496],
[1582070400000, 10133.817417084678]]
}
for c in obj['prices']:
ts = int(c[0]/1000) # convert milisecond to second
dt = datetime.utcfromtimestamp(ts).strftime('%Y-%m-%d %H:%M:%S')
c[0] = dt
print(obj)
result
{
"prices": [["2020-02-18 00:00:00", 9723.605026422496],
["2020-02-19 00:00:00", 10133.817417084678]]
}

Related

I want to add double quotes to each column name without changing the datatype of their values in pandas

I have a data frame with a lot of columns (around 600). I want specific columns from the 600 columns. I specify these columns in a
list [t01 .., t08, t09, t10, t11, t12, t13 ..... t600].
Right now I am just copying this from the csv file.
I want the list to be like this:
['t10', 't11', 't12', 't13' ..... 't400'].
In order to do this, I would have to manually put double quotes around each column name.
Is there a way to directly put double quotes around columns that start from t10 and end at t400? The columns are side by side and they make a range (t10-t401) .
I would store this in
features = df[col_names] and use it in something like :
scaler.fit_transform(features.values)

You can use list comprehension with format of strings:
['t{0}'.format(x) for x in range(10,401)]
#output:
['t10', 't11', 't12', 't13', 't14', 't15', 't16', 't17', 't18', 't19', 't20', 't21', 't22', 't23', 't24', 't25', 't26', 't27', 't28', 't29', 't30', 't31', 't32', 't33', 't34', 't35', 't36', 't37', 't38', 't39', 't40', 't41', 't42', 't43', 't44', 't45', 't46', 't47', 't48', 't49', 't50', 't51', 't52', 't53', 't54', 't55', 't56', 't57', 't58', 't59', 't60', 't61', 't62', 't63', 't64', 't65', 't66', 't67', 't68', 't69', 't70', 't71', 't72', 't73', 't74', 't75', 't76', 't77', 't78', 't79', 't80', 't81', 't82', 't83', 't84', 't85', 't86', 't87', 't88', 't89', 't90', 't91', 't92', 't93', 't94', 't95', 't96', 't97', 't98', 't99', 't100', 't101', 't102', 't103', 't104', 't105', 't106', 't107', 't108', 't109', 't110', 't111', 't112', 't113', 't114', 't115', 't116', 't117', 't118', 't119', 't120', 't121', 't122', 't123', 't124', 't125', 't126', 't127', 't128', 't129', 't130', 't131', 't132', 't133', 't134', 't135', 't136', 't137', 't138', 't139', 't140', 't141', 't142', 't143', 't144', 't145', 't146', 't147', 't148', 't149', 't150', 't151', 't152', 't153', 't154', 't155', 't156', 't157', 't158', 't159', 't160', 't161', 't162', 't163', 't164', 't165', 't166', 't167', 't168', 't169', 't170', 't171', 't172', 't173', 't174', 't175', 't176', 't177', 't178', 't179', 't180', 't181', 't182', 't183', 't184', 't185', 't186', 't187', 't188', 't189', 't190', 't191', 't192', 't193', 't194', 't195', 't196', 't197', 't198', 't199', 't200', 't201', 't202', 't203', 't204', 't205', 't206', 't207', 't208', 't209', 't210', 't211', 't212', 't213', 't214', 't215', 't216', 't217', 't218', 't219', 't220', 't221', 't222', 't223', 't224', 't225', 't226', 't227', 't228', 't229', 't230', 't231', 't232', 't233', 't234', 't235', 't236', 't237', 't238', 't239', 't240', 't241', 't242', 't243', 't244', 't245', 't246', 't247', 't248', 't249', 't250', 't251', 't252', 't253', 't254', 't255', 't256', 't257', 't258', 't259', 't260', 't261', 't262', 't263', 't264', 't265', 't266', 't267', 't268', 't269', 't270', 't271', 't272', 't273', 't274', 't275', 't276', 't277', 't278', 't279', 't280', 't281', 't282', 't283', 't284', 't285', 't286', 't287', 't288', 't289', 't290', 't291', 't292', 't293', 't294', 't295', 't296', 't297', 't298', 't299', 't300', 't301', 't302', 't303', 't304', 't305', 't306', 't307', 't308', 't309', 't310', 't311', 't312', 't313', 't314', 't315', 't316', 't317', 't318', 't319', 't320', 't321', 't322', 't323', 't324', 't325', 't326', 't327', 't328', 't329', 't330', 't331', 't332', 't333', 't334', 't335', 't336', 't337', 't338', 't339', 't340', 't341', 't342', 't343', 't344', 't345', 't346', 't347', 't348', 't349', 't350', 't351', 't352', 't353', 't354', 't355', 't356', 't357', 't358', 't359', 't360', 't361', 't362', 't363', 't364', 't365', 't366', 't367', 't368', 't369', 't370', 't371', 't372', 't373', 't374', 't375', 't376', 't377', 't378', 't379', 't380', 't381', 't382', 't383', 't384', 't385', 't386', 't387', 't388', 't389', 't390', 't391', 't392', 't393', 't394', 't395', 't396', 't397', 't398', 't399', 't400']

How to create a dataframe from a dictionary such that each entry is a list?

How do I create a dataframe from a dictionary where each entry is a list? For example, the script below
for k,v in pose.items():
print(k,v)
prints
p1_center_head [-0.91411722 -1.11261988 1.70305252]
p1_right_hand [-0.6696707 -1.45284259 0.85303462]
p1_front_chest [-0.90311265 -1.15465713 1.09575438]
p1_right_shoulder [-0.88741797 -1.37508869 1.49578547]
p1_right_elbow [-0.8592056 -1.48356283 1.14748859]
p1_right_head [-0.92714155 -1.27181339 1.6833576 ]
p1_left_shoulder [-1.14574909 -1.15120935 1.48121977]
p1_left_head [-1.06484675 -1.14306569 1.65506554]
p2_center_head [1.0070374 1.08997941 1.69709778]
p1_right_wrist [-0.72482365 -1.47362924 0.92447436]
p2_right_shoulder [0.95422077 1.31905603 1.48511732]
p2_right_hand [0.69995475 1.25936687 0.90006113]
p2_right_wrist [0.75100577 1.30776393 0.94339645]
p2_left_shoulder [1.24170136 1.11059046 1.47705841]
p2_back_chest [1.19639254 1.35239506 0.93713373]
p1_back_chest [-1.06148314 -1.38686275 1.03079486]
p2_front_chest [0.9896763 1.14398277 0.99924839]
p2_right_elbow [0.88454074 1.43418157 1.13609421]
p2_right_head [1.01738358 1.24647975 1.68129683]
p2_left_head [1.15873718 1.12672687 1.66546488]
When I create a dataframe and save it to a csv file however, each entry contains one float instead of a list of floats.
pose = pd.DataFrame.from_dict(pose)
pose.to_csv("pose.csv"),index=False)
I need each entry to be a list e.g. pose["p1_center_head"] = [-0.91411722 -1.11261988 1.70305252]

See if this is what you want:
df = pd.DataFrame.from_dict(pose, orient="index")
df['list_item'] = df.apply(list, axis=1)
df.drop(columns=[0, 1, 2], inplace=True)
print(df)
Output:
list_item
p1_center_head [-0.91411722, -1.11261988, 1.70305252]
p1_right_hand [-0.6696707, -1.45284259, 0.85303462]
p1_front_chest [-0.90311265, -1.15465713, 1.09575438]
p1_right_shoulder [-0.88741797, -1.37508869, 1.49578547]
p1_right_elbow [-0.8592056, -1.48356283, 1.14748859]
p1_right_head [-0.92714155, -1.27181339, 1.6833576]
p1_left_shoulder [-1.14574909, -1.15120935, 1.48121977]
p1_left_head [-1.06484675, -1.14306569, 1.65506554]
p2_center_head [1.0070374, 1.08997941, 1.69709778]
p1_right_wrist [-0.72482365, -1.47362924, 0.92447436]
p2_right_shoulder [0.95422077, 1.31905603, 1.48511732]
p2_right_hand [0.69995475, 1.25936687, 0.90006113]
p2_right_wrist [0.75100577, 1.30776393, 0.94339645]
p2_left_shoulder [1.24170136, 1.11059046, 1.47705841]
p2_back_chest [1.19639254, 1.35239506, 0.93713373]
p1_back_chest [-1.06148314, -1.38686275, 1.03079486]
p2_front_chest [0.9896763, 1.14398277, 0.99924839]
p2_right_elbow [0.88454074, 1.43418157, 1.13609421]
p2_right_head [1.01738358, 1.24647975, 1.68129683]
p2_left_head [1.15873718, 1.12672687, 1.66546488]
Then you can further query for an entry like this:
df.loc['p1_right_head']
Output:
list_item [-0.92714155, -1.27181339, 1.6833576]
Name: p1_right_head, dtype: object

Error in retrieving financial data for large list of tickers from yahoo finance into a dataframe using for loop

In this particular problem, I have a very long list of tickers for which I want to retrieve some of the financial information from yahoo finance website using python:
here is the list:
tickers = ["OMKAR.BO", "KCLINFRA.BO", "MERMETL.BO", "PRIMIND.BO", "VISIONCO.BO", "PANAFIC.BO", "KARANWO.BO", "SOURCEIND.BO", "WELCURE.BO", "NAVKETAN.BO", "CUBIFIN.BO", "IMPEXFERRO.BO", "MISHTANN.BO", "SUMERUIND.BO", "MISHTANN.BO", "MADHUVEER.BO", "TNTELE.BO", "JMGCORP.BO", "GSLSEC.BO", "DEVKI.BO", "MINAXI.BO", "INNOCORP.BO", "SURYACHAKRA.BO", "ANKITMETAL.BO", "HAVISHA.BO", "SHIVA.BO", "COMFINTE.BO", "KONNDOR.BO", "PAZEL.BO", "SHARPINV.BO", "MIDINFRA.BO", "UNIVPRIM.BO", "ATHARVENT.BO", "FGP.BO", "BKV.BO", "VIVIDHA.BO", "FISCHER.BO", "ADITRI.BO", "GLFL.BO", "RAJOIL.BO", "ALFL.BO", "PURITY.BO", "ARCEEIN.BO", "INTECH.BO", "MIDEASTP.BO", "STANCAP.BO", "OCTAVE.BO", "TRIJAL.BO", "SREEJAYA.BO", "4THGEN.BO", "RICHIRICH.BO", "VIRTUALS.BO", "SAVINFOCO.BO", "TTIENT.BO", "OONE.BO", "TILAK.BO", "XTGLOBAL.BO", "MANGIND.BO", "ROYALIND.BO", "ASHUTPM.BO", "SMPL.BO", "BGPL.BO", "NYSSACORP.BO", "BILENERGY.BO", "YOGISUNG.BO", "DOLPHMED.BO", "PRATIK.BO", "IPOWER.BO", "BIHSPONG.BO", "CAPFIN.BO", "MCLTD.BO", "KGL.BO", "OMNIAX.BO", "HEERAISP.BO", "VISIONCINE.BO", "SWORDEDGE.BO", "AARVINFRA.BO", "ADVENT.BO", "UVDRHOR.BO", "SUNGOLD.BO", "USHDI.BO", "HINDAPL.BO", "IMEC.BO", "ARAVALIS.BO", "SERVOTEACH.BO", "SCAGRO.BO", "UMESLTD.BO", "CHARMS.BO", "NCLRESE.BO", "SYMBIOX.BO", "PRADIP.BO", "INTEGFD.BO", "CLIOINFO.BO", "RRSECUR.BO", "MUKATPIP.BO", "SYNCOMF.BO", "DYNAMICP.BO", "TRABI.BO", "RADAAN.BO", "KIRANSY-B.BO", "RAMSARUP.BO", "UNIMOVR.BO", "MELSTAR.BO", "OMANSH.BO", "VERTEX.BO", "VENTURA.BO", "GEMSPIN.BO", "EXPLICITFIN.BO", "PASARI.BO", "BABA.BO", "MAHAVIRIND.BO", "BAMPSL.BO", "GAJRA.BO", "SUNRAJDI.BO", "ACCEL.BO", "SIMPLXPAP.BO", "PHARMAID.BO", "JATALIA.BO", "TWINSTAR.BO", "CINDRELL.BO", "SHRGLTR.BO", "EUROMULTI.BO", "CRESSAN.BO", "SEVENHILL.BO", "QUADRANT.BO", "PHTRADING.BO", "SIPTL.BO", "HOTELRUGBY.BO", "KAUSHALYA.BO", "YASHRAJC.BO", "ASHAI.BO", "BERYLSE.BO", "LLOYDSTEEL.BO", "SCANPRO.BO", "HBLEAS.BO", "ASHCAP.BO", "SUNSHINE.BO", "AREALTY.BO", "MSCTC.BO", "HARIAEXPO.BO", "CNIRESLTD.BO", "KABRADG.BO", "CLFL.BO", "TRANSASIA.BO", "KACL.BO", "JAIHINDS.BO", "SANBLUE.BO", "DHENUBUILD.BO", "DHENUBUILD.BO", "ODYCORP.BO", "SAWABUSI.BO", "KAKTEX.BO", "GANONPRO.BO", "GENUSPRIME.BO", "EUREKAI.BO", "CHROMATIC.BO", "ISHWATR.BO", "INTEGRA.BO", "KACL.BO", "SSLFINANCE.BO", "ORIENTTR.BO", "ZHINUDYP.BO", "SWADEIN.BO", "SHKALYN.BO", "BAPACK.BO", "MARUTISE.BO", "PMTELELIN.BO", "SPARCSYS.BO", "GOLKONDA.BO", "DECPO.BO", "NATHUEC.BO", "INDOCITY.BO", "IOSYSTEM.BO", "ADVIKCA.BO", "JRFOODS.BO", "INFOMEDIA.BO", "INDRANIB.BO", "REGTRUS.BO", "RAGHUNAT.BO", "DCMFINSERV.BO", "RRIL.BO", "FILATFASH.BO", "ISWL.BO", "ASINPET.BO", "KORE.BO", "UNIOFFICE.BO", "GUJINV.BO", "QUEST.BO", "GLITTEKG.BO", "AMFORG.BO", "LGBFORGE.BO", "MAL.BO", "CYBERMAT.BO", "AGRIMONY.BO", "METKORE.BO", "SKYLMILAR.BO", "KIRANPR.BO", "RAJSPTR.BO", "SHVFL.BO", "MPFSL.BO", "AMITINT.BO", "KREONFIN.BO", "GRAVITY.BO", "KACHCHH.BO", "STELLANT.BO", "DEVINE.BO", "ICSL.BO", "STELLAR.BO", "CORAGRO.BO", "ARCFIN.BO", "GAMMNINFRA.BO", "EMMSONS.BO", "OSCARGLO.BO", "HARIAAPL.BO", "CORNE.BO", "FACORALL.BO", "KANELIND.BO", "INDOASIAF.BO", "BHANDHOS.BO", "GAGANPO.BO", "SELMCL.BO", "VENLONENT.BO", "KBSINDIA.BO", "RAMAPETRO.BO", "UTIQUE.BO", "GUJSTATFIN.BO", "COUNCODOS.BO", "JDORGOCHEM.BO", "ANSHNCO.BO", "SILVERO.BO", "CONSTRONIC.BO", "SIPIND.BO", "ESARIND.BO", "GUJCOTEX.BO", "HILIKS.BO", "MINFY.BO", "LEENEE.BO", "DUGARHOU.BO", "JHACC.BO", "CINERAD.BO", "GCMCAPI.BO", "GCMCOMM.BO", "CHENFERRO.BO", "MANCREDIT.BO", "TRICOMFRU.BO", "VEGETABLE.BO", "JSHL.BO", "HATHWAYB.BO", "JAYIND.BO", "ROYALCU.BO", "DHANADACO.BO", "ELCIDIN.BO", "RAGHUTOB.BO", "GISOLUTION.BO", "RAGHUTOB.BO", "CONTICON.BO", "NETWORK.BO", "BANASFN.BO", "CRANESSOFT.BO", "RSCINT.BO", "JPTRLES.BO", "ALOKTEXT.BO", "PRAGBOS.BO", "WELTI.BO", "EKAMLEA.BO", "MASL.BO", "SAFFRON.BO", "SRDAPRT.BO", "FFPL.BO", "RITESHIN.BO", "BLOIN.BO", "YARNSYN.BO", "OISL.BO", "POLYTEX.BO", "SPSINT.BO", "GCMCOMM.BO", "FRONTCAP.BO", "SEZAL.BO", "CITYMAN.BO", "AJEL.BO", "ESCORTSFIN.BO", "ABHIINFRA.BO", "PRATIKSH.BO", "JCTLTD.BO", "GENESIS.BO", "HINDSECR.BO", "GKCONS.BO", "MODWOOL.BO", "ROHITFERRO.BO", "NMSRESRC.BO", "VARIMAN.BO", "WAGEND.BO", "INDLEASE.BO", "APOORVA.BO", "HITTCO.BO", "PREMPIPES.BO", "SRMENERGY.BO", "KEDIACN.BO", "TOYAMIND.BO", "EPSOMPRO.BO", "RICHUNV.BO", "CITYONLINE.BO", "ELANGO.BO", "AMITSEC.BO", "CTL.BO", "LPDC.BO", "CONTCHM.BO", "NTL.BO", "SYBLY.BO", "ELEFLOR.BO", "KMFBLDR.BO", "TRIVIKRAMA.BO", "RUCHINFRA.BO", "PROMACT.BO", "USHAKIRA.BO", "ARUNAHTEL.BO", "CIL.BO", "MOUNTSHIQ.BO", "SPTRSHI.BO", "SEATV.BO", "SWASTIVI.BO", "SUNDARAM.BO", "CREATIVEYE.BO", "EUROASIA.BO", "ANJANIFIN.BO", "ADARSH.BO", "GLOBALCA.BO", "INDERGR.BO", "USGTECH.BO", "RASIELEC.BO", "SHEETAL.BO", "SYLPH.BO", "GOYALASS.BO", "KANSAFB.BO", "ANERI.BO", "DRL.BO", "OSWALOR.BO", "SWAGRUHA.BO", "SARTHAKIND.BO", "GALADA.BO", "OSWAYRN.BO", "TRINITYLEA.BO", "GOLCA.BO", "SODFC.BO", "LEADFIN.BO", "KAYPOWR.BO", "PANELEC.BO", "TARAI.BO", "SANJIVIN.BO", "MKTCREAT.BO", "ECOBOAR.BO", "SUNRINV.BO", "MAYURFL.BO", "GARWAMAR.BO", "SURYAKR.BO", "BESTAGRO.BO", "INDCEMCAP.BO", "EASTSILK.BO", "MPAGI.BO", "HRMNYCP.BO", "RUBRAME.BO", "INCON.BO", "AMRAPLIN.BO", "RESPONSINF.BO", "BACPHAR.BO", "KRISHNACAP.BO", "SHBHAWPA.BO", "TOWASOK.BO", "PADMALAYAT.BO", "MHSGRMS.BO", "JMTAUTOLTD.BO", "WELCON.BO", "UNITEDTE.BO", "MNPLFIN.BO", "PARSHINV.BO", "UNISHIRE.BO", "RAJINFRA.BO", "MMLF.BO", "ALCHCORP.BO", "CHMBBRW.BO", "NOGMIND.BO", "SHRMFGC.BO", "SAMTEX.BO", "SUPERTEX.BO", "JAIHINDPRO.BO", "CENTEXT.BO", "BCG.BO", "GENNEX.BO", "EDUCOMP.BO", "SHIVAGR.BO", "ADINATH.BO", "MINID.BO", "SURANAT&P.BO", "GYANDEV.BO", "AVTIL.BO", "ZSWASTSA.BO", "JINDCAP.BO", "NBFOOT.BO", "SHESHAINDS.BO", "UTLINDS.BO", "MADHUSE.BO", "THAMBBI.BO", "KKPLASTICK.BO", "VAGHANI.BO", "SOLIDCO.BO", "HIMFIBP.BO", "KKFIN.BO", "CSL.BO", "GOPAIST.BO", "BALTE.BO", "ETIL.BO", "PAOS.BO", "RAINBOWDQ.BO", "JAGSONFI.BO", "REGENTRP.BO", "AFEL.BO", "BRIPORT.BO", "SURATEX.BO", "INFRAIND.BO", "SPENTEX.BO", "TITANSEC.BO", "ALPSINDUS.BO", "UNISTRMU.BO", "SPECMKT.BO", "SAENTER.BO", "TOKYOFIN.BO", "TRANSFD.BO", "BSELINFRA.BO", "WELSPLSOL.BO", "SONALAD.BO", "CRIMSON.BO", "UNITY.BO", "VIKASPROP.BO", "VELHO.BO", "SYNCOM.BO", "CYBELEIND.BO", "VANICOM.BO", "THAKRAL.BO", "INDOEURO.BO", "ALAN SCOTT.BO", "SALSTEEL.BO", "ADITYA.BO", "HASTIFIN.BO", "NIBE.BO", "JOINTECAED.BO", "GANGAPHARM.BO", "SBECSUG.BO", "EASTBUILD.BO", "LORDSHOTL.BO", "IYKOTHITE.BO", "URJAGLOBA.BO", "DHRUVCA.BO", "RAP.BO", "LAHL.BO", "MONNETIN.BO", "SETUINFRA.BO", "RRMETAL.BO", "GTLINFRA.BO", "ECOM.BO", "TTML.BO", "ARNOLD.BO", "FLORATX.BO", "GARODCH.BO", "PUROHITCON.BO", "KAMRLAB.BO", "MILESTONE.BO", "NETLINK.BO", "MARSONS.BO", "SESL.BO", "OBRSESY.BO", "VRWODAR.BO", "NUWAY.BO", "CJGEL.BO", "REDEXPR.BO", "AISHWARYA.BO", "PICTUREHS.BO", "BAGFILMS.BO", "WOODSVILA.BO", "MEHSECU.BO", "MBPARIKH.BO", "SICLTD.BO", "GITARENEW.BO", "DESHRAK.BO", "SENINFO.BO", "TELECANOR.BO", "STLSTRINF.BO", "JRELTD.BO", "OROSMITHS.BO", "MUNOTHFI.BO", "AVAILFC.BO", "NITINFIRE.BO", "PIFL.BO", "BLBLIMITED.BO", "SRECR.BO", "NAGTECH.BO", "ARISE.BO", "FRONTBUSS.BO", "PAEL.BO", "ROLLT.BO", "VALLABH.BO", "RANASUG.BO", "STRATMONT.BO", "SANTOSHF.BO", "SVAINDIA.BO", "PARKERAC.BO", "VSFPROJ.BO", "AUROCOK.BO", "HKG.BO", "CASTEXTECH.BO", "HOWARHO.BO", "RTNPOWER.BO", "SHRIBCL.BO", "GARWSYN.BO", "MEHSECU.BO", "PRAVEG.BO", "MEHTAHG.BO", "RTNINFRA.BO", "MMWL.BO", "GAGAN.BO", "WWALUM.BO", "HEMANG.BO", "DOLAT.BO", "SUPTANERY.BO", "EUROCERA.BO", "SURFI.BO", "TTIL.BO", "VARDHMAN.BO", "SUPERBAK.BO", "ESHAMEDIA.BO", "CONTILI.BO", "CESL.BO", "DAULAT.BO", "RAJATH.BO", "SURYVANSP.BO", "KUWERIN.BO", "SVARTCORP.BO", "SKRABUL.BO", "WSIND.BO", "DELTA.BO", "SIPL.BO", "RMCHEM.BO", "STDBAT.BO", "PICCASUG.BO", "AGIOPAPER.BO", "SHREYASI.BO", "CCCL.BO", "GAL.BO", "GOLECHA.BO", "RAAJMEDI.BO", "KINETRU.BO", "ZKHANDEN.BO", "LAKHOTIA.BO", "SANINFRA.BO", "KABSON.BO", "ENTRINT.BO", "SIROHIA.BO", "3IINFOTECH.BO", "MEHIF.BO", "BASANTGL.BO", "MAITRI.BO", "CEENIK.BO", "MAXIMAA.BO", "STANPACK.BO", "CRANEINFRA.BO", "CHITRTX.BO", "CAPRICORN.BO", "TAVERNIER.BO", "JPPOWER.BO", "PATIDAR.BO", "BANSTEA.BO", "NEWMKTADV.BO", "DANUBE.BO", "MAHALXSE.BO", "SARDAPPR.BO", "KZLFIN.BO", "ABHIFIN.BO", "AVI.BO", "GAYATRIBI.BO", "VXLINSTR.BO", "ADITYASP.BO", "OMKARPH.BO", "ESSARSEC.BO", "SALSAIN.BO", "NDASEC.BO", "PARABDRUGS.BO", "EPIC.BO", "HIGHSTREE.BO", "TRIMURTHI.BO", "DBSTOCKBRO.BO", "ADARSHPL.BO", "SONAL.BO", "FRASER.BO", "BRIDGESE.BO", "GBGLOBAL.BO", "UNRYLMA.BO", "ANNAINFRA.BO", "RTEXPO.BO", "FUNDVISER.BO", "LIBORD.BO", "HYPERSOFT.BO", "JTAPARIA.BO", "ANUBHAV.BO", "MEGFI.BO", "ACTIONFI.BO", "BCLENTERPR.BO", "RAMSONS.BO", "GUJARATPOLY.BO", "SBFL.BO", "CHDCHEM.BO", "MONEYBOXX.BO", "ALSL.BO", "DEVHARI.BO", "NARPROP.BO", "PIONAGR.BO", "JAYBHCR.BO", "QGO.BO", "KRIFILIND.BO", "GOLDCOINHF.BO", "GALLOPENT.BO", "MIC.BO", "INTELLCAP.BO", "ABIRAFN.BO", "OLPCL.BO", "ZSHERAPR.BO", "CELLA.BO", "ZSANMCOM.BO", "STEELCO.BO", "VFL.BO", "MODAIRY.BO", "ZSANMCOM.BO", "STEELCO.BO", "SHUKJEW.BO", "JAYKAY.BO", "MIC.BO", "MODAIRY.BO", "RGIL.BO", "GSBFIN.BO", "OLPCL.BO", "HINDMOTORS.BO", "GAJANANSEC.BO", "MKEXIM.BO", "BERLDRG.BO", "KUBERJI.BO", "ADDIND.BO", "INDOSOLAR.BO", "GOLDCOINHF.BO", "ACIIN.BO", "UNITINT.BO", "SDC.BO", "RAJKSYN.BO", "CHAMAK.BO", "BHILSPIN.BO", "PANORAMA.BO", "REGAL.BO", "KRRAIL.BO", "AMS.BO", "PARIKSHA.BO", "SURYAINDIA.BO", "ADHARSHILA.BO", "AMARNATH.BO", "JAYATMA.BO", "CANOPYFIN.BO", "FMEC.BO", "CITL.BO", "DAL.BO", "YORKEXP.BO", "MEWATZI.BO"]
and then what I am doing is as below in which I want to get Market Capitalization for each of the tickers in the above list:
from pandas_datareader import data
import pandas as pd
tickers = tick[0:30]
dat = data.get_quote_yahoo(tickers)['marketCap']
print(dat)
I am able to fetch 20-30 tickers using above code, but if I try to pull all, the code throws an error "request timed out" and "list out of range" etc.
Then I tried to fetch data one by one using for loop as below:
f = pd.DataFrame(columns=["Ticker", "MarketCapINR"])
columns = list(df)
for tick in ticks:
dat = data.get_quote_yahoo(tick)['marketCap']
zipped = zip(columns, dat)
a_dictionary = dict(zipped)
df = df.append(a_dictionary, ignore_index=True)
This returned me two errors, one of these is list out of bound and another (when I tried to shorten the length of list using 'slicing'), request timed out.
Is there a way out, to get all the data (ticker names in first column and MarketCap values in second column of a pandas dataframe) ??
..

Here's a solution using a package called yahooquery. Disclaimer: I am the author of the package.
from yahooquery import Ticker
import pandas as pd
tickers = [...] # Use your list above
t = Ticker(tickers)
data = t.quotes
df = pd.DataFrame(data).T
df['marketCap']
OMKAR.BO 4750000
KCLINFRA.BO 26331000
MERMETL.BO 11472136
PRIMIND.BO 22697430
VISIONCO.BO 14777874
...
CANOPYFIN.BO 93859304
FMEC.BO 10542380
CITL.BO NaN
YORKEXP.BO 57503880
MEWATZI.BO 51200000
Name: marketCap, Length: 632, dtype: object

Python convert a list of ips into a list of networks

I have a list of ips like this;
ban = ['162.210.197.58', '36.72.31.198', '163.172.221.63', '134.35.42.49', '176.65.117.81', '46.8.28.110', '110.139.16.72', '31.29.61.218', '38.95.109.67', '31.155.204.143', '5.254.65.190', '46.101.127.145', '111.15.193.84', '187.142.198.151', '188.227.45.110', '125.236.168.242', '5.254.65.122', '5.254.65.28', '185.182.81.93', '188.210.133.199', '151.241.125.237', '207.148.111.126', '145.239.70.231', '2.94.113.65', '204.85.191.30', '2.229.128.135', '92.127.255.182', '46.133.77.143', '118.69.63.15', '46.39.55.162', '185.182.81.14', '188.244.132.49', '138.197.160.121', '176.15.222.56', '82.178.233.175', '94.41.43.252', '78.137.73.171', '5.142.90.22', '212.2.212.151', '207.244.77.174', '176.59.46.108', '179.7.106.194', '154.118.17.57', '5.254.65.119', '176.33.236.247', '41.218.215.25', '199.249.224.47', '207.244.83.97', '84.241.13.93', '197.210.54.218', '93.35.166.148', '218.107.49.9', '39.55.130.71', '78.168.207.252', '213.141.227.238', '36.69.70.20', '5.254.65.218', '103.10.197.12', '207.244.83.194', '122.163.128.154', '93.185.16.82', '95.24.168.216', '103.16.26.148', '118.100.59.199', '209.59.188.147', '207.244.93.234', '88.81.46.212', '23.105.140.146', '89.104.104.225', '83.149.21.13', '41.34.51.195', '146.185.133.214', '37.204.108.198', '188.123.231.194', '85.140.3.172', '185.182.81.45', '185.182.81.61', '5.254.65.213', '185.182.81.136', '103.10.197.205', '185.182.81.86', '207.244.79.138', '116.240.84.42', '91.219.101.68', '80.191.221.23', '178.62.220.63', '209.58.148.85', '167.114.159.186', '108.59.8.246', '74.82.17.84', '178.129.172.216', '36.72.178.137', '41.254.7.157', '185.190.241.83', '185.182.81.58', '107.77.209.79', '207.244.77.137', '58.27.54.130', '69.15.118.106', '85.94.0.117', '41.215.173.28', '185.182.81.28', '46.101.18.209', '5.254.65.185', '195.123.217.178', '2.48.190.213', '188.247.190.79', '192.119.160.190', '95.72.13.156', '68.149.146.193', '197.157.244.206', '103.10.197.157', '176.195.155.114', '194.88.107.55', '83.220.237.120', '185.182.81.53', '82.102.21.114', '84.241.6.212', '207.244.78.48', '125.238.220.170', '159.8.170.20', '176.15.60.114', '46.138.146.207', '128.199.201.63', '5.254.65.179', '45.242.236.151', '188.162.177.199', '186.68.202.84', '5.202.179.40', '41.215.173.10', '103.255.6.80', '83.102.219.15', '188.32.127.17', '185.182.81.89', '207.244.86.235', '212.252.141.6', '37.147.79.146', '176.59.80.125', '212.71.255.8', '87.240.26.58', '91.79.215.243', '185.182.81.6', '207.244.83.102', '185.182.81.20', '109.133.114.88', '199.249.223.78', '37.110.94.227', '185.182.81.42', '109.252.76.194', '167.99.46.145', '95.221.186.30', '185.48.188.131', '2.191.41.213', '82.113.106.115', '178.62.102.23', '197.255.118.226', '185.182.81.12', '41.207.5.116', '68.133.80.56', '103.255.7.6', '162.210.197.53', '138.197.174.120', '185.182.81.40', '154.124.87.182', '73.218.73.13', '185.182.81.66', '139.5.231.112', '185.69.124.60', '93.190.177.36', '207.244.79.136', '66.160.199.70', '78.106.211.173', '121.52.156.4', '109.201.133.100', '78.85.40.39', '91.79.162.72', '83.242.238.138', '5.254.65.125', '5.254.65.175', '103.214.2.4', '37.232.192.102', '61.5.58.240', '41.190.14.40', '146.185.157.250', '195.123.225.47', '103.60.210.61', '85.102.111.101', '77.30.202.151', '194.146.149.46', '95.27.45.39', '146.185.158.24', '207.244.86.240', '195.175.45.154', '195.149.108.48', '95.12.119.124', '82.102.31.203', '156.221.118.172', '78.108.178.87', '185.182.81.37', '41.219.31.245', '191.43.27.200', '142.44.210.125', '5.254.65.169', '46.196.145.191', '68.133.80.55', '5.254.65.127', '68.133.80.58', '31.173.84.201', '39.50.173.251', '46.242.10.175', '201.208.20.239', '174.16.34.27', '65.19.167.132', '182.0.139.135', '128.68.185.234', '78.178.85.114', '182.182.91.44', '103.10.197.155', '199.249.223.77', '46.173.75.34', '197.32.29.188', '185.18.46.170', '107.181.182.227', '105.50.114.109', '199.87.154.255', '185.182.81.23', '5.115.181.178', '67.205.137.158', '46.73.40.13', '117.224.14.73', '155.239.150.15', '98.253.130.3', '122.129.77.114', '176.194.163.88', '109.63.235.0', '85.140.7.208', '207.244.82.229', '94.29.124.69', '188.245.245.146', '78.177.102.153', '197.210.173.174', '85.174.236.178', '82.247.106.95', '207.244.83.218', '162.210.197.55', '185.180.197.65', '31.8.130.8', '114.124.135.2', '46.172.203.108', '185.182.81.41', '199.115.116.37', '207.244.78.175', '207.244.77.139', '185.182.81.85', '172.56.30.213', '79.126.114.90', '5.254.65.114', '207.244.77.9', '103.10.197.202', '87.245.249.103', '185.182.81.48', '39.54.218.203', '109.169.172.138', '39.50.16.30', '89.104.104.227', '78.189.31.17', '207.244.77.19', '95.28.26.67', '39.54.6.166', '145.255.169.92', '176.43.253.71', '185.107.70.202', '65.19.167.131', '199.244.86.149', '197.156.241.249', '114.125.184.172', '89.178.171.173', '151.235.84.235', '185.104.184.120', '197.243.40.75', '88.253.119.59', '103.76.21.243', '207.244.78.50', '95.73.121.206', '185.182.81.60', '176.55.167.228', '179.54.147.13', '156.198.95.108', '199.249.223.63', '165.139.150.133', '185.182.81.16', '185.182.81.80', '162.210.197.54', '41.34.8.12', '86.161.166.126', '103.21.125.80', '5.254.65.15', '65.49.126.73', '69.117.214.158', '92.96.162.204', '64.62.232.66', '2.48.180.82', '213.87.163.155', '108.59.8.217', '91.151.188.194', '197.229.0.32', '76.218.238.42', '178.62.6.233', '159.65.117.243', '217.118.78.80', '109.252.25.1', '96.80.89.69', '89.40.118.87', '185.182.81.25', '73.43.73.161', '108.59.10.184', '5.155.214.63', '103.10.197.195', '95.71.21.143', '113.21.99.50', '198.96.155.3', '203.156.158.201', '5.254.65.19', '103.10.197.133', '207.244.94.5', '190.39.1.149', '103.10.197.61', '213.87.139.175', '91.244.39.229', '136.0.99.204', '77.34.114.167', '185.76.248.253', '185.182.81.139', '207.244.82.154', '199.115.115.194', '139.59.84.23', '209.126.90.78', '156.219.221.191', '109.252.99.3', '67.252.57.173', '222.70.120.196', '207.244.78.47', '156.197.120.27', '185.182.81.76', '78.95.166.34', '103.10.197.131', '37.21.231.95', '87.225.68.121', '46.40.227.14', '77.43.173.56', '114.124.181.244', '37.1.129.188', '128.72.83.20', '5.254.65.85', '31.129.204.120', '5.158.237.18', '151.243.199.70', '185.182.81.77', '103.228.157.94', '69.202.142.68', '94.25.180.120', '185.182.81.91', '103.10.197.154', '202.22.227.26', '49.145.151.70', '207.244.86.198', '185.182.81.19', '185.182.81.69', '5.254.65.123', '65.49.68.203', '103.79.155.58', '216.185.36.99', '80.191.221.21', '5.254.65.220', '91.215.69.212', '5.254.65.24', '103.10.197.204', '105.67.7.89', '195.112.116.163', '46.42.167.129', '190.142.101.84', '207.244.79.152', '5.59.53.95', '14.139.125.241', '197.211.63.74', '195.190.107.166', '185.182.81.135', '194.146.181.131', '83.220.237.252', '41.215.173.33', '142.44.210.206', '62.213.109.218', '199.168.151.35', '5.254.65.217', '45.242.236.74', '80.253.23.149', '185.182.81.11', '93.168.185.76', '185.182.81.82', '77.106.62.60', '185.220.101.20', '185.182.81.47', '185.182.81.56', '5.41.65.151', '66.160.188.43', '74.82.35.71', '79.139.176.23', '88.203.110.34', '64.147.94.201', '213.87.132.49', '91.216.114.201', '179.53.157.148', '185.182.81.4', '196.74.129.52', '95.72.247.249', '185.182.81.71', '185.182.81.138', '154.68.60.100', '81.214.142.169', '147.110.59.244', '200.2.187.239', '68.133.80.57', '5.44.174.110', '142.234.201.129', '89.238.177.226', '120.188.86.112', '207.244.78.173', '173.208.82.162', '115.178.200.218', '139.162.233.190', '43.245.8.67', '103.10.197.58', '207.244.83.219', '148.0.219.242', '5.126.66.228', '37.190.37.137', '182.179.179.160', '103.10.197.220', '41.66.255.100', '37.144.44.196', '207.244.77.18', '5.254.65.102', '213.33.193.34', '185.182.81.51', '154.121.5.242', '92.246.191.120', '186.167.242.158', '217.117.66.54', '85.88.176.70', '114.124.182.15', '146.185.177.103', '179.180.67.5', '207.154.232.214', '91.192.70.25', '156.38.35.15', '147.135.254.228', '103.10.197.130', '195.214.140.20', '31.29.217.147', '73.170.68.18', '142.234.201.141', '2.191.76.162', '151.80.36.117', '5.254.65.195', '213.87.225.138', '5.0.249.218', '27.68.68.184', '178.59.189.246', '95.30.31.106', '213.87.126.137', '104.45.14.157', '196.6.216.5', '185.182.81.131', '5.254.65.177', '103.255.7.21', '212.75.159.112', '103.10.197.196', '217.55.47.233', '109.252.76.112', '36.84.62.52', '185.253.97.52', '31.148.253.33', '185.182.81.78', '159.224.41.242', '46.101.16.232', '5.254.65.97', '89.160.136.97', '207.244.78.8', '5.254.65.82', '185.182.81.92', '95.29.19.127', '185.182.81.22', '66.160.201.58', '93.90.204.172', '2.93.201.96', '182.1.178.122', '148.0.142.65', '94.233.224.122', '163.232.200.115', '200.236.75.164', '106.204.119.26', '168.8.192.22', '185.182.81.83', '107.77.209.106', '207.244.78.17', '89.104.104.224', '106.51.234.22', '71.12.167.41', '188.247.109.2', '213.87.156.209', '192.240.119.59', '78.180.74.219', '37.107.111.54', '103.10.197.132', '5.254.65.8', '105.167.52.122', '146.185.157.173', '95.184.110.95', '23.101.10.145', '5.254.65.124', '151.231.40.224', '178.129.222.227', '185.182.81.68', '185.107.47.215', '146.185.156.139', '120.188.39.56', '176.14.163.226', '178.62.113.166', '95.153.134.122', '185.182.81.75', '156.194.101.98', '185.182.81.72', '182.187.97.214', '41.72.194.58', '196.201.199.210', '185.182.81.79', '51.39.237.115', '202.146.235.80', '176.123.126.254', '86.149.9.202', '41.254.4.43', '91.240.73.248', '185.88.157.231', '171.229.95.88', '5.254.65.129', '139.59.225.74', '95.24.175.100', '5.254.65.182', '185.182.81.50', '89.31.57.5', '129.56.11.18', '207.244.83.104', '199.115.118.85', '89.235.123.202', '154.69.67.5', '36.84.62.49', '146.185.157.38', '31.29.248.93', '109.252.20.82', '207.244.94.36', '66.160.188.67', '5.254.65.6', '38.95.109.110', '178.175.132.67', '103.214.2.35', '196.24.72.24', '185.182.81.9', '109.252.177.102', '136.252.163.99', '185.104.184.116', '162.247.72.199', '213.55.105.3', '5.79.68.161', '188.170.103.189', '5.254.65.21', '88.228.167.196', '41.242.138.49', '103.10.197.59', '178.73.220.45', '200.82.249.248', '5.254.65.107', '207.244.77.172', '5.254.65.84', '37.190.37.197', '213.87.131.60', '37.112.229.42', '95.24.91.228', '185.182.81.84', '77.247.181.162', '78.81.231.143', '185.182.81.46', '207.244.83.114', '41.253.39.240', '92.50.223.44', '173.208.82.164', '41.85.161.178', '213.87.122.199', '37.52.183.88', '207.244.72.201', '109.252.53.106', '108.59.8.208', '104.129.18.10', '122.57.37.39', '176.59.48.37', '125.178.76.189', '154.130.48.206', '113.169.71.251', '91.215.69.23', '54.36.105.8', '95.10.207.189', '128.31.0.13', '81.211.123.74', '103.10.197.13', '46.242.10.245', '81.131.164.246', '156.194.244.214', '178.62.106.169', '204.85.191.31', '207.244.86.200', '77.27.69.166', '192.96.205.131', '139.59.3.30', '80.252.16.20', '185.182.81.137', '150.100.253.6', '103.25.72.234', '207.244.86.195', '154.160.16.103', '78.95.6.2', '185.182.81.44', '197.248.213.18', '202.142.79.94', '38.95.108.247', '2.135.158.83', '37.73.82.56', '103.255.6.75', '5.254.65.110', '64.62.175.68', '207.244.83.209', '172.241.114.173', '207.244.79.137', '207.244.82.251', '207.244.86.232', '94.233.224.26', '174.141.115.146', '109.174.112.128', '93.91.80.6', '41.190.3.77', '159.89.115.182', '188.163.3.226', '95.191.226.57', '139.5.154.162', '193.0.219.72', '185.182.81.70', '138.197.174.51', '41.217.109.25', '39.50.91.154', '37.204.198.46', '130.180.218.30', '5.254.65.17', '138.197.160.42', '5.254.65.187', '37.195.213.64', '188.170.73.65', '197.48.231.108', '185.189.113.42', '185.182.81.74', '41.66.203.243', '163.21.7.42', '5.254.65.83', '157.50.236.233', '94.233.224.7', '128.71.22.17', '130.34.246.190', '185.52.141.42', '139.59.84.2', '84.241.202.178', '123.242.160.17', '74.82.17.52', '182.0.205.139', '203.202.229.194', '89.238.191.150', '79.165.225.194', '73.61.8.75', '41.254.8.167', '185.182.81.5', '94.47.16.26', '182.186.238.234', '156.216.204.96', '109.252.57.81', '180.252.104.52', '79.148.109.242', '66.160.201.47', '2.191.110.236', '207.244.77.13', '5.254.65.98', '145.255.166.214', '159.203.130.12', '197.234.221.77', '159.89.206.211', '85.141.69.27', '185.182.81.90', '77.31.101.152', '202.22.237.40', '95.221.205.237', '207.244.77.55', '104.236.233.75', '5.254.65.104', '104.236.233.182', '103.10.197.60', '103.10.197.222', '182.54.141.154', '138.197.174.79', '176.33.136.50', '185.182.81.10', '185.220.101.30', '94.138.40.2', '41.44.178.2', '196.207.121.147', '213.138.86.113', '31.173.83.65', '199.115.115.197', '46.72.89.106', '103.240.170.209', '5.254.65.215', '146.185.171.157', '185.182.81.57', '188.119.59.75', '46.164.104.176', '176.52.96.95', '207.244.83.99', '31.43.31.116', '178.62.208.24', '5.254.65.167', '185.182.81.73', '104.194.24.250', '79.200.158.58', '79.111.52.85', '103.10.197.91', '199.249.224.65', '5.136.175.48', '111.119.165.253', '216.218.222.12', '191.248.209.198', '5.254.65.81', '178.46.97.50', '27.97.248.46', '190.72.126.220', '178.140.204.131', '5.114.82.81', '118.175.154.120', '207.244.90.69', '185.182.81.7', '103.17.88.247', '108.248.141.54', '130.34.246.189', '207.244.79.154', '41.66.255.247', '128.69.14.62', '87.107.208.172', '199.115.116.35', '46.191.213.221', '199.115.114.220', '172.56.44.234', '69.12.94.91', '5.254.65.99', '37.203.243.97', '185.182.81.29', '37.210.142.211', '207.244.78.9', '206.189.56.39', '193.104.27.86', '37.190.37.236', '112.197.34.56', '189.124.82.209', '99.66.132.171', '5.254.65.16', '46.201.114.83', '81.31.178.23', '94.47.97.98', '82.102.31.251', '103.10.197.203', '94.98.252.208', '14.139.160.236', '185.182.81.130', '184.43.6.218', '92.37.143.180', '5.254.65.12', '83.142.185.126', '5.254.65.178', '213.87.123.37', '36.67.214.132', '185.182.81.21', '205.164.32.216', '207.244.83.198', '5.254.65.11', '197.210.221.42', '207.244.83.206', '5.254.65.130', '176.14.156.125', '94.138.40.7', '182.75.59.110', '162.210.197.57', '41.35.98.90', '95.139.11.104', '103.10.197.93', '39.40.168.95', '213.87.127.74', '2.94.25.98', '216.218.222.14', '207.244.82.155', '138.197.154.118', '199.249.223.69', '5.254.65.112', '62.120.19.197', '185.182.81.17', '148.0.140.9', '103.10.197.197', '185.182.81.15', '217.114.239.64', '185.182.81.134', '130.180.217.17', '111.88.82.111', '112.210.165.48', '81.92.200.142', '185.182.81.43', '207.244.77.3', '169.159.85.133', '5.254.65.20', '197.234.221.34', '192.162.242.11', '185.182.81.67', '103.247.48.9', '109.173.73.174', '171.25.193.20', '207.244.77.17', '176.59.44.162', '95.28.48.169', '5.254.65.113', '62.112.114.74', '5.254.65.174', '179.96.223.197', '149.202.202.78', '146.185.157.243', '110.93.216.238', '217.148.214.162', '23.100.15.206', '24.133.136.29', '59.189.57.53', '185.182.81.62', '31.181.90.52', '5.254.65.117', '94.99.170.144', '146.185.157.164', '178.62.230.201', '41.223.117.58', '197.54.177.15', '212.252.67.30', '109.252.86.175', '194.146.149.39', '207.244.83.197', '190.94.3.236', '207.244.94.9', '5.254.65.193', '5.254.65.194', '207.244.77.10', '146.185.158.9', '5.254.65.172', '37.156.22.79', '209.190.17.22', '103.10.197.156', '5.254.65.9', '178.140.62.96', '207.244.77.16', '109.161.121.117', '37.78.31.88', '185.220.101.13', '185.182.81.87', '87.107.165.5', '185.135.233.14', '209.58.148.100', '83.110.13.200', '5.254.65.132', '93.80.234.243', '190.39.198.140', '95.221.183.242', '185.222.209.32', '154.160.16.167', '80.71.249.236', '185.182.81.39', '41.79.197.12', '91.76.89.93', '45.247.206.86', '185.182.81.88', '192.81.222.95', '207.244.86.236', '197.210.29.171', '213.61.95.74', '41.254.7.18', '31.9.55.66', '154.112.8.183', '5.254.65.180', '37.187.158.97', '195.149.108.45', '94.179.147.128', '94.130.206.211', '31.173.85.237', '103.10.197.221', '46.39.83.173', '5.254.65.170', '5.254.65.18', '95.167.31.202', '207.244.83.212', '138.197.174.132', '185.182.81.35', '192.96.205.135', '130.88.240.92', '185.182.81.94', '90.147.2.94', '209.126.90.97', '128.75.107.209', '2.94.139.161', '103.106.174.102', '186.4.212.26', '37.145.142.85', '207.244.86.194', '72.211.60.177', '95.30.45.134', '80.71.253.200', '5.254.65.109', '185.182.81.38', '5.254.65.214', '178.62.221.111', '81.92.200.204', '89.104.104.226', '95.73.236.188', '207.244.83.113', '185.182.81.24', '103.247.48.111', '176.59.43.149', '109.233.172.188', '207.244.70.35', '84.17.230.179', '83.123.148.16', '122.171.168.163', '213.87.121.217', '110.54.232.113', '206.189.56.10', '154.160.30.129', '207.244.93.227', '178.46.96.130', '87.116.196.230', '181.208.250.104', '43.252.233.33', '207.244.82.249', '154.68.5.60', '37.27.72.154', '178.68.165.3', '194.186.9.228', '207.244.79.134', '41.34.209.13', '192.119.160.242', '37.187.129.166', '41.254.7.152', '46.38.23.226', '185.182.81.52', '27.34.104.88', '31.29.253.114', '200.17.141.3', '86.62.75.82', '93.183.219.225', '94.143.40.157', '103.10.197.194', '188.166.212.116', '185.182.81.2', '212.98.135.186', '103.10.197.92', '207.244.83.195', '217.64.17.124', '46.242.10.244', '209.58.136.236', '156.203.226.215', '103.10.197.218', '199.115.114.218', '185.182.81.8', '207.244.82.231', '95.8.0.20', '207.244.82.133', '128.74.124.28', '111.95.14.210', '213.87.126.35', '5.254.65.134', '91.240.210.185', '185.182.81.132', '95.32.139.134', '213.55.110.157', '185.182.81.26', '176.47.67.182', '91.65.54.25', '217.10.34.243', '185.182.81.133', '182.1.191.230', '103.10.197.90', '213.87.225.153', '207.244.77.184', '185.182.81.81', '5.254.65.88', '123.176.6.58', '213.55.110.206', '93.157.169.12', '103.10.197.219', '109.252.81.16', '78.111.187.63', '82.196.1.179', '185.182.81.59', '148.0.193.199', '14.202.100.66', '5.250.37.110', '156.219.245.39', '31.9.112.197', '185.112.37.25', '128.70.176.120', '185.182.81.27', '188.226.226.40', '109.174.113.45', '5.246.33.162', '5.254.65.189', '46.151.246.57', '77.71.24.206', '178.62.36.218', '185.182.81.36', '196.188.112.153', '199.249.223.49', '80.252.153.74', '87.109.169.79', '207.244.83.200', '197.210.54.144', '95.57.231.156', '207.244.94.33', '103.16.25.2', '217.174.233.46', '178.184.98.116', '185.182.81.55', '107.77.207.197', '86.21.86.205', '5.254.65.192', '38.95.108.230', '94.13.131.173', '107.77.209.181', '176.59.54.71', '128.179.252.54', '31.215.58.51', '207.244.83.123', '85.106.69.110', '192.160.102.170', '37.146.34.208', '46.39.228.226', '87.225.75.186', '159.89.123.60', '176.59.68.249', '37.143.20.94', '105.112.29.37', '89.146.74.31', '90.147.2.86', '217.144.175.138', '212.252.57.203', '131.117.162.145', '109.63.245.141', '92.97.199.104', '176.123.121.221', '213.142.154.3', '180.160.55.195', '176.195.133.16', '172.241.112.3', '207.244.83.217', '125.167.120.44', '109.252.85.9', '46.39.53.115', '206.189.201.4', '31.173.83.78', '199.115.115.215', '178.62.153.200']
How do I convert a list of ips like above into a list of subnets?
For example;
185.182.0.0/y, 207.244.0.0/x
Edit
I'm getting a lot of malicious traffic from several ips that look like they're in the same subnet, however there are a few thousand ips, how do I convert a list like this into subnets so that I can block the subnet?

Im guessing subnets are 000.000.XXX.XXX where the X's can be a single 0?
Lets assume you have:
ban = ['162.210.197.58/32', '36.72.31.198/34', ..., '134.35.42.49/20', '176.65.117.81/32']
Use a set comprehension to ensure you get only unique elements:
subnets = {'.'.join(addr.split('.')[:2]) + '.0.0/' + addr.split('/')[-1] for addr in ban}
And you get:
>>> subnets
{'162.210.0.0/32', '36.72.0.0/34', ..., '134.35.0.0/20'}

Note that an IPv4 subnet may be many different sizes, from /32 for a single IP address to /0 for the whole internet. You don't have enough information to work out how a network administrator has subnetted their network if the only thing you have is an IP address. Assuming a /24 or a /16 will often be wrong.
That said, you could use whois to lookup the net ranges associated with the ASNs which contain particular IP addresses. You could use whois from python via a tool like ipwhois to build up a list of net ranges that you consider "bad" based on the fact that you've had a contact from a bad IP in that ASN.
from ipwhois import IPWhois
bad_ips = ["185.182.81.12",]
bad_asn_ranges = set()
for ip in bad_ips:
obj = IPWhois(ip)
ret = obj.lookup_whois()
print(ret)
bad_asn_ranges.add(ret['asn_cidr'])
print(bad_asn_ranges)
This would have unintended consequences if good people and bad people both coexist on the same ASN as one another.

If IP addresses are stored as a list and are in a string type, here is one very simple solution to reseting all subnets and then at the same time leave duplicates in another condition. Maybe slow code, but functional code.
data = ['162.210.197.58', '36.72.121.146', '36.72.121.4', '162.210.108.4', '190.5.140.90']
tmp = []
for ip in data:
ip1, ip2, ip3, ip4 = ip.split('.')
if (ip1 + '.' + ip2 + '.0.0') not in tmp:
tmp.append(ip1 + '.' + ip2 + '.0.0')
data = tmp
print(data)
Another example:
data = ['162.210.197.58', '36.72.121.146', '36.72.121.4', '162.210.108.4', '190.5.140.90']
msk = 1 # range 1-3, how much bytes do you want set to zero, from right side (in IP address)
tmp = []
for ip in data:
s = '.'.join(ip.split('.')[:-msk]) + ('.0'*msk) # split the string by '.' delimiter, then combine all to back but without msk entries from the right side, then add string '.0' repeated msk times
if s not in tmp:
tmp.append(s)
data = tmp
print(data)
A note:
It is better to keep IP addresses as an integer data type (as Python does not know the Byte data type). Then any IP address operations will be really simple and fast. Logical operations NOR, XOR, OR, AND, etc. (as needed), you could do everything you need in a professional area of computer networks using a mask. Of course, it depends on the length of the addressable IP address field - as the colleagues mentioned above. If the length of the standard most frequently used is 16 bits (2 bytes) or 24 bits (3 bytes), then my code is sufficient. However, if the length in the bits is non-standard (26 bits, 12 bits, 10 bits, etc.), then it is better to set the integer type from the string IP address and further process the IP addresses already as IPv4 length estimates, which are 4 bytes operations XOR, AND, OR, ...).

This will give you the 24 bit prefixes and number of occurrences. defaultdict is used to automatically insert a 0 in the dictionary when the key is missing, making the code more simple.
data = ['162.210.197.58', '36.72.31.198'] #etc....
import collections
subnets = collections.defaultdict(int)
for ip in data:
subnets[".".join(ip.split('.')[0:3])] += 1
print(subnets)

This answer by Wang is the correct way to do this: https://stackoverflow.com/a/45503187/3176550
list_of_ips = ['10.0.0.0', '10.0.0.1', '10.0.0.2', '10.0.0.3', '10.0.0.5']
import ipaddress
nets = [ipaddress.ip_network(_ip) for _ip in list_of_ips]
cidrs = ipaddress.collapse_addresses(nets)
list(cidrs)
Out[6]: [IPv4Network('10.0.0.0/30'), IPv4Network('10.0.0.5/32')]

Convert Unicode Output of Tuple into string Python

I have the following response I am getting when I for loop through the following:
for col in ids:
print (format(col)),
print ("\n")
print type(ids[0])
I have a tuple that has element of different type: unicode, and int.
(u'0021401225', u'0021401203', u'0021401167', u'0021401148', u'0021401124', u'0021401092', u'0021401078', u'0021401066', u'0021401037', u'0021401023', u'0021401005', u'0021400995', u'0021400985', u'0021400964', u'0021400948', u'0021400924', u'0021400915', u'0021400900', u'0021400891', u'0021400854', u'0021400839', u'0021400821', u'0021400809', u'0021400800', u'0021400796', u'0021400766', u'0021400747', u'0021400743', u'0021400718', u'0021400711', u'0021400700', u'0021400681', u'0021400676', u'0021400658', u'0021400643', u'0021400626', u'0021400620', u'0021400599', u'0021400586', u'0021400570', u'0021400561', u'0021400547', u'0021400523', u'0021400503', u'0021400484', u'0021400479', u'0021400466', u'0021400429', u'0021400416', u'0021400401', u'0021400386', u'0021400370', u'0021400355', u'0021400336', u'0021400326', u'0021400308', u'0021400303', u'0021400281', u'0021400273', u'0021400253', u'0021400238', u'0021400211', u'0021400199', u'0021400186', u'0021400179', u'0021400159', u'0021400147', u'0021400133', u'0021400125', u'0021400098', u'0021400081', u'0021400066', u'0021400055', u'0021400022', u'0021400018')
I'm trying to store these values without the
u'
Because I can key off of these id's later in my program. But I just want to store them as integers which might look like the below:
[('0021401225','0021401203', '0021401167', '0021401148','0021401124',
'0021401092', '0021401078', '0021401066', '0021401037', '0021401023')]
I would like to for-loop through them.

lt = (u'0021401225', u'0021401203', u'0021401167', u'0021401148', u'0021401124', u'0021401092', u'0021401078', u'0021401066', u'0021401037', u'0021401023', u'0021401005', u'0021400995', u'0021400985', u'0021400964', u'0021400948', u'0021400924', u'0021400915', u'0021400900', u'0021400891', u'0021400854', u'0021400839', u'0021400821', u'0021400809', u'0021400800', u'0021400796', u'0021400766', u'0021400747', u'0021400743', u'0021400718', u'0021400711', u'0021400700', u'0021400681', u'0021400676', u'0021400658', u'0021400643', u'0021400626', u'0021400620', u'0021400599', u'0021400586', u'0021400570', u'0021400561', u'0021400547', u'0021400523', u'0021400503', u'0021400484', u'0021400479', u'0021400466', u'0021400429', u'0021400416', u'0021400401', u'0021400386', u'0021400370', u'0021400355', u'0021400336', u'0021400326', u'0021400308', u'0021400303', u'0021400281', u'0021400273', u'0021400253', u'0021400238', u'0021400211', u'0021400199', u'0021400186', u'0021400179', u'0021400159', u'0021400147', u'0021400133', u'0021400125', u'0021400098', u'0021400081', u'0021400066', u'0021400055', u'0021400022', u'0021400018')
lt = list(map(str,lt) )
print(lt)
['0021401225', '0021401203', '0021401167', '0021401148', '0021401124', '0021401092', '0021401078', '0021401066', '0021401037', '0021401023', '0021401005', '0021400995', '0021400985', '0021400964', '0021400948', '0021400924', '0021400915', '0021400900', '0021400891', '0021400854', '0021400839', '0021400821', '0021400809', '0021400800', '0021400796', '0021400766', '0021400747', '0021400743', '0021400718', '0021400711', '0021400700', '0021400681', '0021400676', '0021400658', '0021400643', '0021400626', '0021400620', '0021400599', '0021400586', '0021400570', '0021400561', '0021400547', '0021400523', '0021400503', '0021400484', '0021400479', '0021400466', '0021400429', '0021400416', '0021400401', '0021400386', '0021400370', '0021400355', '0021400336', '0021400326', '0021400308', '0021400303', '0021400281', '0021400273', '0021400253', '0021400238', '0021400211', '0021400199', '0021400186', '0021400179', '0021400159', '0021400147', '0021400133', '0021400125', '0021400098', '0021400081', '0021400066', '0021400055', '0021400022', '0021400018']

Categories