pandas: merge on column of ByteArray - python

Any ideas on how I can join two pandas arrays on a commonly named bytearray field? The field in the source (Teradata) is an actual ByteArray, and from the Teradata Side, this cannot be forced to character or something usable outside of Teradata)
The Teradata Export reads into a Panda's array beautifully. But I can't merge two tables with a commonly named field (DatabaseId) where that field is a bytearray.
(importing both pandas as pd and itertools)
When I try a simple merge of:
merge1 = pd.merge(tvm, dbase, on="DatabaseId")
I get the error of:
TypeError: type object argument after * must be a sequence, not itertools.imap
I searched StackOverflow and found a similar problem for joining on a cell containing a collection
dbase['DBID'] = dbase.DatabaseId.apply(lambda r: type(sorted(r.iteritems())))
But I get the Error:
AttributeError: 'bytearray' object has no attribute 'iteritems'
UPDATE
Example of Data
Data gathered through pandas using
dbase = pd.read_sql('select databaseid, databasename from ud812.dbase sample 10', conn)
conn is a connection to a teradata database
Data Types coming out of Teradata are Varchar for all columns EXCEPT:
DatabaseID = bytearray (Byte(4))
TVMID = bytearray (Byte(4))
>>> dbase.dtypes
DatabaseId object
DatabaseName object
dtype: object
>>> dbase
DatabaseId DatabaseName
0 [2, 0, 243, 185] PCDW_CRS_BBCONV3_TB
1 [2, 0, 168, 114] PAMLIF_TB
2 [2, 0, 133, 153] PADW_PRESN_TB
3 [2, 0, 29, 184] CEDW_MOBILE_TB
4 [2, 0, 190, 183] CEDW_MODEL_SCORE_TB
5 [2, 0, 71, 55] PBBBAM_TB
6 [2, 0, 169, 183] CEDW_OCC_TB
7 [2, 0, 201, 183] CCDW_DGTL_DEAL_TB
8 [0, 0, 139, 8] PRECDSS_TB
9 [2, 0, 142, 203] CDBDW_TB
>>>
>>>
>>> tvm.dtypes
TVMId object
DatabaseId object
TVMName object
TableKind object
CreateText object
dtype: object
>>> tvm
TVMId DatabaseId TVMName \
0 [230, 1, 41, 11, 0, 0] [2, 0, 67, 183] JCP_03538_112002
1 [214, 1, 60, 133, 0, 0] [2, 0, 186, 52] STL_AUTHNCTD_RULE_EXECN
2 [193, 2, 59, 48, 0, 0] [2, 0, 225, 150] uye177_Xsell_EM_OPCL_TB2
3 [0, 2, 235, 154, 0, 0] [2, 0, 244, 181] PL_CALCD_INVSTR_MTHLY_HIST_ST
4 [255, 1, 131, 76, 0, 0] [2, 0, 110, 63] IMH867_AVA0803_SNAP
5 [125, 1, 217, 138, 0, 0] [2, 0, 237, 153] FD_ACCT_STMT_ADR_ST
6 [224, 0, 80, 233, 0, 0] [2, 0, 243, 127] EXP_SRCH_RSLT_DESC
7 [208, 1, 72, 15, 0, 0] [2, 0, 8, 57] SGI_PAY_DENIED_SEP_112012
8 [246, 0, 27, 61, 0, 0] [2, 0, 143, 130] CR_INDIVD
9 [186, 1, 242, 167, 0, 0] [0, 0, 244, 18] wzu448_sb_apps
TableKind CreateText
0 T None
1 V CREATE VIEW ... ... ... ... ... ... ... ... ...
2 T None
3 V CREATE VIEW ... ... ... ... ... ... ... ... ...
4 T None
5 V CREATE VIEW ... ... ... ... ... ... ... ... ...
6 V CREATE VIEW ... ... ... ... ... ... ... ... ...
7 V CREATE VIEW ... ... ... ... ... ... ... ... ...
8 V CREATE VIEW ... ... ... ... ... ... ... ... ...
9 T None

Convert your bytearrays to their immutable cousin bytes.
import pandas as pd
# Create your example `dbase`
DatabaseId_dbase = list(map(bytearray, [[2, 0, 243, 185], [2, 0, 168, 114],
[2, 0, 133, 153], [2, 0, 29, 184], [2, 0, 190, 183], [2, 0, 71, 55],
[2, 0, 169, 183], [2, 0, 201, 183], [0, 0, 139, 8], [2, 0, 142, 203]]))
DatabaseName = ['PCDW_CRS_BBCONV3_TB', 'PAMLIF_TB', 'PADW_PRESN_TB',
'CEDW_MOBILE_TB', 'CEDW_MODEL_SCORE_TB', 'PBBBAM_TB', 'CEDW_OCC_TB',
'CCDW_DGTL_DEAL_TB', 'PRECDSS_TB', 'CDBDW_TB']
dbase = pd.DataFrame({'DatabaseId': DatabaseId_dbase,
'DatabaseName': DatabaseName})
# Create your example `tvm`
DatabaseId_tvm = list(map(bytearray, [[2, 0, 67, 183], [2, 0, 186, 52],
[2, 0, 225, 150], [2, 0, 244, 181], [2, 0, 110, 63], [2, 0, 237, 153],
[2, 0, 243, 127], [2, 0, 243, 185], [2, 0, 143, 130], [0, 0, 244, 18]]))
TVMId = list(map(bytearray, [[230, 1, 41, 11, 0, 0], [214, 1, 60, 133, 0, 0],
[193, 2, 59, 48, 0, 0], [0, 2, 235, 154, 0, 0], [255, 1, 131, 76, 0, 0],
[125, 1, 217, 138, 0, 0], [224, 0, 80, 233, 0, 0], [208, 1, 72, 15, 0, 0],
[246, 0, 27, 61, 0, 0], [186, 1, 242, 167, 0, 0]]))
TVMName = ['JCP_03538_112002', 'STL_AUTHNCTD_RULE_EXECN',
'uye177_Xsell_EM_OPCL_TB2', 'PL_CALCD_INVSTR_MTHLY_HIST_ST',
'IMH867_AVA0803_SNAP', 'FD_ACCT_STMT_ADR_ST', 'EXP_SRCH_RSLT_DESC',
'SGI_PAY_DENIED_SEP_112012', 'CR_INDIVD', 'wzu448_sb_apps']
TableKind = ['T', 'V', 'T', 'V', 'T', 'V', 'V', 'V', 'V', 'T']
tvm = pd.DataFrame({'DatabaseId': DatabaseId_tvm, 'TVMId': TVMId,
'TVMName': TVMName, 'TableKind': TableKind})
# This line would fail with the following error
# TypeError: type object argument after * must be a sequence, not map
# merge = pd.merge(tvm, dbase, on='DatabaseId')
# Apply the `bytes` constructor to the `bytearray` columns
dbase['DatabaseId'] = dbase['DatabaseId'].apply(bytes)
tvm['DatabaseId'] = tvm['DatabaseId'].apply(bytes)
tvm['TVMId'] = tvm['TVMId'].apply(bytes)
# Now it works!
merge = pd.merge(tvm, dbase, on='DatabaseId')
The resulting merge is
DatabaseId TVMId TVMName \
0 b'\x02\x00\xf3\xb9' b'\xd0\x01H\x0f\x00\x00' SGI_PAY_DENIED_SEP_112012
TableKind DatabaseName
0 V PCDW_CRS_BBCONV3_TB
(I had to change the DatabaseId field of one of the rows in your tvm, since otherwise merge would have been empty. I also didn't include the CreateText column — too awkward for SO)

Related

Matrix row masking at different indices

I have an 2-D array that I want to find the max value per row and then find the next max-value that is not within +/- n of the previous value. For example I have the following matrix:
results =
array([[ 33, 108, 208, 96, 96, 112, 18, 208, 33, 323, 60, 42],
[ 51, 6, 39, 112, 160, 144, 342, 195, 27, 136, 42, 54],
[ 12, 176, 266, 162, 45, 70, 156, 198, 143, 56, 342, 130],
[ 22, 288, 304, 162, 21, 238, 156, 126, 165, 91, 144, 130],
[342, 120, 36, 51, 10, 128, 156, 272, 32, 98, 192, 288]])
row_max_index = results.argmax(1)
row_max_index #show max index
array([ 9, 6, 10, 2, 0])
Now I'd like to get the next max value not within say +/- 2 of the current max.
Here is what I have but it feels sloppy:
maskIndx = np.c_[row_max_index-2, row_max_index-1, row_max_index, row_max_index+1, row_max_index+2,]%12
maskIndx #show windowed index
array([[ 7, 8, 9, 10, 11],
[ 4, 5, 6, 7, 8],
[ 8, 9, 10, 11, 0],
[ 0, 1, 2, 3, 4],
[10, 11, 0, 1, 2]])
results[np.meshgrid(np.arange(5), np.arange(5))[1], maskIndx] = 0 #uses array indexing
results #show results
array([[ 33, 108, 208, 96, 96, 112, 18, 0, 0, 0, 0, 0],
[ 51, 6, 39, 112, 0, 0, 0, 0, 0, 136, 42, 54],
[ 0, 176, 266, 162, 45, 70, 156, 198, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 238, 156, 126, 165, 91, 144, 130],
[ 0, 0, 0, 51, 10, 128, 156, 272, 32, 98, 0, 0]])
next_max_index = results.argmax(1)
array([2, 9, 2, 5, 7])
Any ideas on doing this faster through indexing/windowing?
You can create a mask around the indices you compute for the max by taking an array of indices and subtracting the relevant max_indices, and then use the masked api to recompute the argmax:
import numpy as np
result = ... # Result here
row_max_index = results.argmax(axis=1, keepdims=True)
indices = np.arange(results.shape[1])
mask = np.abs(indices - row_max_index) <= 2
out = np.ma.array(results, mask=mask).argmax(axis=1)

(34, 'Numerical result out of range') calculating Annual Percentage Rate with scipy

I am trying to solve Annual Percentage Rate in Python using scipy library and newton method. At some point Python reaches max Pythonic float range solving this equation:
Here is example code with data that fails. Is there anything that I can use (interval etc.) that is able to solve this equation?
from scipy.optimize import newton
class ARP:
def __init__(self):
self.year_days = 365
self.table = [[0, 416.0, '2022-04-01', 0, 0, 0, 0, 0], [1, 75.92, '2022-04-02', 1, 1, 0, 0, 0], [2, 75.92, '2022-05-02', 30, 1, 0, 1, 0], [3, 90.92, '2022-06-02', 31, 1, 0, 2, 0], [4, 75.92, '2022-07-02', 30, 1, 0, 3, 0], [5, 75.92, '2022-08-02', 31, 1, 0, 4, 0], [6, 90.92, '2022-09-02', 31, 1, 0, 5, 0], [7, 75.92, '2022-10-02', 30, 1, 0, 6, 0], [8, 90.92, '2022-11-02', 31, 1, 0, 7, 0], [9, 75.92, '2022-12-02', 30, 1, 0, 8, 0], [10, 75.92, '2023-01-02', 31, 1, 0, 9, 0], [11, 90.92, '2023-02-02', 31, 1, 0, 10, 0], [12, 75.92, '2023-03-02', 28, 1, 0, 11, 0], [13, 90.92, '2023-04-02', 31, 1, 0, 12, 0], [14, 782.9, '2023-05-02', 30, 1, 0, 13, 0], [15, 732.9, '2023-06-02', 31, 1, 0, 14, 0], [16, 732.9, '2023-07-02', 30, 1, 0, 15, 0], [17, 732.9, '2023-08-02', 31, 1, 0, 16, 0], [18, 732.9, '2023-09-02', 31, 1, 0, 17, 0], [19, 732.9, '2023-10-02', 30, 1, 0, 18, 0], [20, 732.9, '2023-11-02', 31, 1, 0, 19, 0], [21, 732.9, '2023-12-02', 30, 1, 0, 20, 0], [22, 732.9, '2024-01-02', 31, 1, 0, 21, 0], [23, 732.9, '2024-02-02', 31, 1, 0, 22, 0], [24, 732.9, '2024-03-02', 29, 1, 0, 23, 0], [25, 732.9, '2024-04-02', 31, 1, 0, 24, 0], [26, 782.9, '2024-05-02', 30, 1, 0, 25, 0], [27, 732.9, '2024-06-02', 31, 1, 0, 26, 0], [28, 732.9, '2024-07-02', 30, 1, 0, 27, 0], [29, 732.9, '2024-08-02', 31, 1, 0, 28, 0], [30, 732.9, '2024-09-02', 31, 1, 0, 29, 0], [31, 732.9, '2024-10-02', 30, 1, 0, 30, 0], [32, 732.9, '2024-11-02', 31, 1, 0, 31, 0], [33, 732.9, '2024-12-02', 30, 1, 0, 32, 0], [34, 732.9, '2025-01-02', 31, 1, 0, 33, 0], [35, 732.9, '2025-02-02', 31, 1, 0, 34, 0], [36, 732.9, '2025-03-02', 28, 1, 0, 35, 0], [37, 732.9, '2025-04-02', 31, 1, 0, 36, 0], [38, 782.9, '2025-05-02', 30, 1, 0, 37, 0], [39, 732.9, '2025-06-02', 31, 1, 0, 38, 0], [40, 732.9, '2025-07-02', 30, 1, 0, 39, 0], [41, 732.9, '2025-08-02', 31, 1, 0, 40, 0], [42, 732.9, '2025-09-02', 31, 1, 0, 41, 0], [43, 732.9, '2025-10-02', 30, 1, 0, 42, 0], [44, 732.9, '2025-11-02', 31, 1, 0, 43, 0], [45, 732.9, '2025-12-02', 30, 1, 0, 44, 0], [46, 732.9, '2026-01-02', 31, 1, 0, 45, 0], [47, 732.9, '2026-02-02', 31, 1, 0, 46, 0], [48, 732.9, '2026-03-02', 28, 1, 0, 47, 0], [49, 732.9, '2026-04-02', 31, 1, 0, 48, 0], [50, 782.9, '2026-05-02', 30, 1, 0, 49, 0], [51, 732.9, '2026-06-02', 31, 1, 0, 50, 0], [52, 732.9, '2026-07-02', 30, 1, 0, 51, 0], [53, 732.9, '2026-08-02', 31, 1, 0, 52, 0], [54, 732.9, '2026-09-02', 31, 1, 0, 53, 0], [55, 732.9, '2026-10-02', 30, 1, 0, 54, 0], [56, 732.9, '2026-11-02', 31, 1, 0, 55, 0], [57, 732.9, '2026-12-02', 30, 1, 0, 56, 0], [58, 732.9, '2027-01-02', 31, 1, 0, 57, 0], [59, 732.9, '2027-02-02', 31, 1, 0, 58, 0], [60, 732.9, '2027-03-02', 28, 1, 0, 59, 0], [61, 732.9, '2027-04-02', 31, 1, 0, 60, 0], [62, 782.9, '2027-05-02', 30, 1, 0, 61, 0], [63, 732.9, '2027-06-02', 31, 1, 0, 62, 0], [64, 732.9, '2027-07-02', 30, 1, 0, 63, 0], [65, 732.9, '2027-08-02', 31, 1, 0, 64, 0], [66, 732.9, '2027-09-02', 31, 1, 0, 65, 0], [67, 732.9, '2027-10-02', 30, 1, 0, 66, 0], [68, 732.9, '2027-11-02', 31, 1, 0, 67, 0], [69, 732.9, '2027-12-02', 30, 1, 0, 68, 0], [70, 732.9, '2028-01-02', 31, 1, 0, 69, 0], [71, 732.9, '2028-02-02', 31, 1, 0, 70, 0], [72, 732.9, '2028-03-02', 29, 1, 0, 71, 0], [73, 732.9, '2028-04-02', 31, 1, 0, 72, 0], [74, 782.9, '2028-05-02', 30, 1, 0, 73, 0], [75, 732.9, '2028-06-02', 31, 1, 0, 74, 0], [76, 732.9, '2028-07-02', 30, 1, 0, 75, 0], [77, 732.9, '2028-08-02', 31, 1, 0, 76, 0], [78, 732.9, '2028-09-02', 31, 1, 0, 77, 0], [79, 732.9, '2028-10-02', 30, 1, 0, 78, 0], [80, 732.9, '2028-11-02', 31, 1, 0, 79, 0], [81, 732.9, '2028-12-02', 30, 1, 0, 80, 0], [82, 732.9, '2029-01-02', 31, 1, 0, 81, 0], [83, 732.9, '2029-02-02', 31, 1, 0, 82, 0], [84, 732.9, '2029-03-02', 28, 1, 0, 83, 0], [85, 732.9, '2029-04-02', 31, 1, 0, 84, 0], [86, 782.9, '2029-05-02', 30, 1, 0, 85, 0], [87, 732.9, '2029-06-02', 31, 1, 0, 86, 0], [88, 732.9, '2029-07-02', 30, 1, 0, 87, 0], [89, 732.9, '2029-08-02', 31, 1, 0, 88, 0], [90, 732.9, '2029-09-02', 31, 1, 0, 89, 0], [91, 732.9, '2029-10-02', 30, 1, 0, 90, 0], [92, 732.9, '2029-11-02', 31, 1, 0, 91, 0], [93, 732.9, '2029-12-02', 30, 1, 0, 92, 0], [94, 732.9, '2030-01-02', 31, 1, 0, 93, 0], [95, 732.9, '2030-02-02', 31, 1, 0, 94, 0], [96, 732.9, '2030-03-02', 28, 1, 0, 95, 0], [97, 732.9, '2030-04-02', 31, 1, 0, 96, 0], [98, 782.9, '2030-05-02', 30, 1, 0, 97, 0], [99, 732.9, '2030-06-02', 31, 1, 0, 98, 0], [100, 732.9, '2030-07-02', 30, 1, 0, 99, 0], [101, 732.9, '2030-08-02', 31, 1, 0, 100, 0], [102, 732.9, '2030-09-02', 31, 1, 0, 101, 0], [103, 732.9, '2030-10-02', 30, 1, 0, 102, 0], [104, 732.9, '2030-11-02', 31, 1, 0, 103, 0], [105, 732.9, '2030-12-02', 30, 1, 0, 104, 0], [106, 732.9, '2031-01-02', 31, 1, 0, 105, 0], [107, 732.9, '2031-02-02', 31, 1, 0, 106, 0], [108, 732.9, '2031-03-02', 28, 1, 0, 107, 0], [109, 732.9, '2031-04-02', 31, 1, 0, 108, 0], [110, 782.9, '2031-05-02', 30, 1, 0, 109, 0], [111, 732.9, '2031-06-02', 31, 1, 0, 110, 0], [112, 732.9, '2031-07-02', 30, 1, 0, 111, 0], [113, 732.9, '2031-08-02', 31, 1, 0, 112, 0], [114, 732.9, '2031-09-02', 31, 1, 0, 113, 0], [115, 732.9, '2031-10-02', 30, 1, 0, 114, 0], [116, 732.9, '2031-11-02', 31, 1, 0, 115, 0], [117, 732.9, '2031-12-02', 30, 1, 0, 116, 0], [118, 732.9, '2032-01-02', 31, 1, 0, 117, 0], [119, 732.9, '2032-02-02', 31, 1, 0, 118, 0], [120, 732.9, '2032-03-02', 29, 1, 0, 119, 0], [121, 732.9, '2032-04-02', 31, 1, 0, 120, 0], [122, 782.9, '2032-05-02', 30, 1, 0, 121, 0], [123, 732.9, '2032-06-02', 31, 1, 0, 122, 0], [124, 732.9, '2032-07-02', 30, 1, 0, 123, 0], [125, 732.9, '2032-08-02', 31, 1, 0, 124, 0], [126, 732.9, '2032-09-02', 31, 1, 0, 125, 0], [127, 732.9, '2032-10-02', 30, 1, 0, 126, 0], [128, 732.9, '2032-11-02', 31, 1, 0, 127, 0], [129, 732.9, '2032-12-02', 30, 1, 0, 128, 0], [130, 732.9, '2033-01-02', 31, 1, 0, 129, 0], [131, 732.9, '2033-02-02', 31, 1, 0, 130, 0], [132, 732.9, '2033-03-02', 28, 1, 0, 131, 0], [133, 732.9, '2033-04-02', 31, 1, 0, 132, 0], [134, 782.9, '2033-05-02', 30, 1, 0, 133, 0], [135, 732.9, '2033-06-02', 31, 1, 0, 134, 0], [136, 732.9, '2033-07-02', 30, 1, 0, 135, 0], [137, 732.9, '2033-08-02', 31, 1, 0, 136, 0], [138, 732.9, '2033-09-02', 31, 1, 0, 137, 0], [139, 732.9, '2033-10-02', 30, 1, 0, 138, 0], [140, 732.9, '2033-11-02', 31, 1, 0, 139, 0], [141, 732.9, '2033-12-02', 30, 1, 0, 140, 0], [142, 732.9, '2034-01-02', 31, 1, 0, 141, 0], [143, 732.9, '2034-02-02', 31, 1, 0, 142, 0], [144, 732.9, '2034-03-02', 28, 1, 0, 143, 0], [145, 732.9, '2034-04-02', 31, 1, 0, 144, 0], [146, 782.9, '2034-05-02', 30, 1, 0, 145, 0], [147, 732.9, '2034-06-02', 31, 1, 0, 146, 0], [148, 732.9, '2034-07-02', 30, 1, 0, 147, 0], [149, 732.9, '2034-08-02', 31, 1, 0, 148, 0], [150, 732.9, '2034-09-02', 31, 1, 0, 149, 0], [151, 732.9, '2034-10-02', 30, 1, 0, 150, 0], [152, 732.9, '2034-11-02', 31, 1, 0, 151, 0], [153, 732.9, '2034-12-02', 30, 1, 0, 152, 0], [154, 732.9, '2035-01-02', 31, 1, 0, 153, 0], [155, 732.9, '2035-02-02', 31, 1, 0, 154, 0], [156, 732.9, '2035-03-02', 28, 1, 0, 155, 0], [157, 732.9, '2035-04-02', 31, 1, 0, 156, 0], [158, 782.9, '2035-05-02', 30, 1, 0, 157, 0], [159, 732.9, '2035-06-02', 31, 1, 0, 158, 0], [160, 732.9, '2035-07-02', 30, 1, 0, 159, 0], [161, 732.9, '2035-08-02', 31, 1, 0, 160, 0], [162, 732.9, '2035-09-02', 31, 1, 0, 161, 0], [163, 732.9, '2035-10-02', 30, 1, 0, 162, 0], [164, 732.9, '2035-11-02', 31, 1, 0, 163, 0], [165, 732.9, '2035-12-02', 30, 1, 0, 164, 0], [166, 732.9, '2036-01-02', 31, 1, 0, 165, 0], [167, 732.9, '2036-02-02', 31, 1, 0, 166, 0], [168, 732.9, '2036-03-02', 29, 1, 0, 167, 0], [169, 732.9, '2036-04-02', 31, 1, 0, 168, 0], [170, 782.9, '2036-05-02', 30, 1, 0, 169, 0], [171, 732.9, '2036-06-02', 31, 1, 0, 170, 0], [172, 732.9, '2036-07-02', 30, 1, 0, 171, 0], [173, 732.9, '2036-08-02', 31, 1, 0, 172, 0], [174, 732.9, '2036-09-02', 31, 1, 0, 173, 0], [175, 732.9, '2036-10-02', 30, 1, 0, 174, 0], [176, 732.9, '2036-11-02', 31, 1, 0, 175, 0], [177, 732.9, '2036-12-02', 30, 1, 0, 176, 0], [178, 732.9, '2037-01-02', 31, 1, 0, 177, 0], [179, 732.9, '2037-02-02', 31, 1, 0, 178, 0], [180, 732.9, '2037-03-02', 28, 1, 0, 179, 0], [181, 732.9, '2037-04-02', 31, 1, 0, 180, 0], [182, 782.9, '2037-05-02', 30, 1, 0, 181, 0], [183, 732.9, '2037-06-02', 31, 1, 0, 182, 0], [184, 732.9, '2037-07-02', 30, 1, 0, 183, 0], [185, 732.9, '2037-08-02', 31, 1, 0, 184, 0], [186, 732.9, '2037-09-02', 31, 1, 0, 185, 0], [187, 732.9, '2037-10-02', 30, 1, 0, 186, 0], [188, 732.9, '2037-11-02', 31, 1, 0, 187, 0], [189, 732.9, '2037-12-02', 30, 1, 0, 188, 0], [190, 732.9, '2038-01-02', 31, 1, 0, 189, 0], [191, 732.9, '2038-02-02', 31, 1, 0, 190, 0], [192, 732.9, '2038-03-02', 28, 1, 0, 191, 0], [193, 732.9, '2038-04-02', 31, 1, 0, 192, 0], [194, 782.9, '2038-05-02', 30, 1, 0, 193, 0], [195, 732.9, '2038-06-02', 31, 1, 0, 194, 0], [196, 732.9, '2038-07-02', 30, 1, 0, 195, 0], [197, 732.9, '2038-08-02', 31, 1, 0, 196, 0], [198, 732.9, '2038-09-02', 31, 1, 0, 197, 0], [199, 732.9, '2038-10-02', 30, 1, 0, 198, 0], [200, 732.9, '2038-11-02', 31, 1, 0, 199, 0], [201, 732.9, '2038-12-02', 30, 1, 0, 200, 0], [202, 732.9, '2039-01-02', 31, 1, 0, 201, 0], [203, 732.9, '2039-02-02', 31, 1, 0, 202, 0], [204, 732.9, '2039-03-02', 28, 1, 0, 203, 0], [205, 732.9, '2039-04-02', 31, 1, 0, 204, 0], [206, 782.9, '2039-05-02', 30, 1, 0, 205, 0], [207, 732.9, '2039-06-02', 31, 1, 0, 206, 0], [208, 732.9, '2039-07-02', 30, 1, 0, 207, 0], [209, 732.9, '2039-08-02', 31, 1, 0, 208, 0], [210, 732.9, '2039-09-02', 31, 1, 0, 209, 0], [211, 732.9, '2039-10-02', 30, 1, 0, 210, 0], [212, 732.9, '2039-11-02', 31, 1, 0, 211, 0], [213, 732.9, '2039-12-02', 30, 1, 0, 212, 0], [214, 732.9, '2040-01-02', 31, 1, 0, 213, 0], [215, 732.9, '2040-02-02', 31, 1, 0, 214, 0], [216, 732.9, '2040-03-02', 29, 1, 0, 215, 0], [217, 732.9, '2040-04-02', 31, 1, 0, 216, 0], [218, 782.9, '2040-05-02', 30, 1, 0, 217, 0], [219, 732.9, '2040-06-02', 31, 1, 0, 218, 0], [220, 732.9, '2040-07-02', 30, 1, 0, 219, 0], [221, 732.9, '2040-08-02', 31, 1, 0, 220, 0], [222, 732.9, '2040-09-02', 31, 1, 0, 221, 0], [223, 732.9, '2040-10-02', 30, 1, 0, 222, 0], [224, 732.9, '2040-11-02', 31, 1, 0, 223, 0], [225, 732.9, '2040-12-02', 30, 1, 0, 224, 0], [226, 732.9, '2041-01-02', 31, 1, 0, 225, 0], [227, 732.9, '2041-02-02', 31, 1, 0, 226, 0], [228, 732.9, '2041-03-02', 28, 1, 0, 227, 0], [229, 732.9, '2041-04-02', 31, 1, 0, 228, 0], [230, 782.9, '2041-05-02', 30, 1, 0, 229, 0], [231, 732.9, '2041-06-02', 31, 1, 0, 230, 0], [232, 732.9, '2041-07-02', 30, 1, 0, 231, 0], [233, 732.9, '2041-08-02', 31, 1, 0, 232, 0], [234, 732.9, '2041-09-02', 31, 1, 0, 233, 0], [235, 732.9, '2041-10-02', 30, 1, 0, 234, 0], [236, 732.9, '2041-11-02', 31, 1, 0, 235, 0], [237, 732.9, '2041-12-02', 30, 1, 0, 236, 0], [238, 732.9, '2042-01-02', 31, 1, 0, 237, 0], [239, 732.9, '2042-02-02', 31, 1, 0, 238, 0], [240, 732.9, '2042-03-02', 28, 1, 0, 239, 0], [241, 732.9, '2042-04-02', 31, 1, 0, 240, 0], [242, 782.9, '2042-05-02', 30, 1, 0, 241, 0], [243, 732.9, '2042-06-02', 31, 1, 0, 242, 0], [244, 732.9, '2042-07-02', 30, 1, 0, 243, 0], [245, 732.9, '2042-08-02', 31, 1, 0, 244, 0], [246, 732.9, '2042-09-02', 31, 1, 0, 245, 0], [247, 732.9, '2042-10-02', 30, 1, 0, 246, 0], [248, 732.9, '2042-11-02', 31, 1, 0, 247, 0], [249, 732.9, '2042-12-02', 30, 1, 0, 248, 0], [250, 732.9, '2043-01-02', 31, 1, 0, 249, 0], [251, 732.9, '2043-02-02', 31, 1, 0, 250, 0], [252, 732.9, '2043-03-02', 28, 1, 0, 251, 0], [253, 732.9, '2043-04-02', 31, 1, 0, 252, 0], [254, 782.9, '2043-05-02', 30, 1, 0, 253, 0], [255, 732.9, '2043-06-02', 31, 1, 0, 254, 0], [256, 732.9, '2043-07-02', 30, 1, 0, 255, 0], [257, 732.9, '2043-08-02', 31, 1, 0, 256, 0], [258, 732.9, '2043-09-02', 31, 1, 0, 257, 0], [259, 732.9, '2043-10-02', 30, 1, 0, 258, 0], [260, 732.9, '2043-11-02', 31, 1, 0, 259, 0], [261, 732.9, '2043-12-02', 30, 1, 0, 260, 0], [262, 732.9, '2044-01-02', 31, 1, 0, 261, 0], [263, 732.9, '2044-02-02', 31, 1, 0, 262, 0], [264, 732.9, '2044-03-02', 29, 1, 0, 263, 0], [265, 732.9, '2044-04-02', 31, 1, 0, 264, 0], [266, 782.9, '2044-05-02', 30, 1, 0, 265, 0], [267, 732.9, '2044-06-02', 31, 1, 0, 266, 0], [268, 732.9, '2044-07-02', 30, 1, 0, 267, 0], [269, 732.9, '2044-08-02', 31, 1, 0, 268, 0], [270, 732.9, '2044-09-02', 31, 1, 0, 269, 0], [271, 732.9, '2044-10-02', 30, 1, 0, 270, 0], [272, 732.9, '2044-11-02', 31, 1, 0, 271, 0], [273, 732.9, '2044-12-02', 30, 1, 0, 272, 0], [274, 732.9, '2045-01-02', 31, 1, 0, 273, 0], [275, 732.9, '2045-02-02', 31, 1, 0, 274, 0], [276, 732.9, '2045-03-02', 28, 1, 0, 275, 0], [277, 732.9, '2045-04-02', 31, 1, 0, 276, 0], [278, 782.9, '2045-05-02', 30, 1, 0, 277, 0], [279, 732.9, '2045-06-02', 31, 1, 0, 278, 0], [280, 732.9, '2045-07-02', 30, 1, 0, 279, 0], [281, 732.9, '2045-08-02', 31, 1, 0, 280, 0], [282, 732.9, '2045-09-02', 31, 1, 0, 281, 0], [283, 732.9, '2045-10-02', 30, 1, 0, 282, 0], [284, 732.9, '2045-11-02', 31, 1, 0, 283, 0], [285, 732.9, '2045-12-02', 30, 1, 0, 284, 0], [286, 732.9, '2046-01-02', 31, 1, 0, 285, 0], [287, 732.9, '2046-02-02', 31, 1, 0, 286, 0], [288, 732.9, '2046-03-02', 28, 1, 0, 287, 0], [289, 732.9, '2046-04-02', 31, 1, 0, 288, 0], [290, 782.9, '2046-05-02', 30, 1, 0, 289, 0], [291, 732.9, '2046-06-02', 31, 1, 0, 290, 0], [292, 732.9, '2046-07-02', 30, 1, 0, 291, 0], [293, 732.9, '2046-08-02', 31, 1, 0, 292, 0], [294, 732.9, '2046-09-02', 31, 1, 0, 293, 0], [295, 732.9, '2046-10-02', 30, 1, 0, 294, 0], [296, 732.9, '2046-11-02', 31, 1, 0, 295, 0], [297, 732.9, '2046-12-02', 30, 1, 0, 296, 0], [298, 732.9, '2047-01-02', 31, 1, 0, 297, 0], [299, 732.9, '2047-02-02', 31, 1, 0, 298, 0], [300, 732.9, '2047-03-02', 28, 1, 0, 299, 0], [301, 732.9, '2047-04-02', 31, 1, 0, 300, 0], [302, 782.9, '2047-05-02', 30, 1, 0, 301, 0], [303, 732.9, '2047-06-02', 31, 1, 0, 302, 0], [304, 732.9, '2047-07-02', 30, 1, 0, 303, 0], [305, 732.9, '2047-08-02', 31, 1, 0, 304, 0], [306, 732.9, '2047-09-02', 31, 1, 0, 305, 0], [307, 732.9, '2047-10-02', 30, 1, 0, 306, 0], [308, 732.9, '2047-11-02', 31, 1, 0, 307, 0], [309, 732.9, '2047-12-02', 30, 1, 0, 308, 0], [310, 732.9, '2048-01-02', 31, 1, 0, 309, 0], [311, 732.9, '2048-02-02', 31, 1, 0, 310, 0], [312, 732.9, '2048-03-02', 29, 1, 0, 311, 0], [313, 732.9, '2048-04-02', 31, 1, 0, 312, 0], [314, 782.9, '2048-05-02', 30, 1, 0, 313, 0], [315, 732.9, '2048-06-02', 31, 1, 0, 314, 0], [316, 732.9, '2048-07-02', 30, 1, 0, 315, 0], [317, 732.9, '2048-08-02', 31, 1, 0, 316, 0], [318, 732.9, '2048-09-02', 31, 1, 0, 317, 0], [319, 732.9, '2048-10-02', 30, 1, 0, 318, 0], [320, 732.9, '2048-11-02', 31, 1, 0, 319, 0], [321, 732.9, '2048-12-02', 30, 1, 0, 320, 0], [322, 732.9, '2049-01-02', 31, 1, 0, 321, 0], [323, 732.9, '2049-02-02', 31, 1, 0, 322, 0], [324, 732.9, '2049-03-02', 28, 1, 0, 323, 0], [325, 732.9, '2049-04-02', 31, 1, 0, 324, 0], [326, 782.9, '2049-05-02', 30, 1, 0, 325, 0], [327, 732.9, '2049-06-02', 31, 1, 0, 326, 0], [328, 732.9, '2049-07-02', 30, 1, 0, 327, 0], [329, 732.9, '2049-08-02', 31, 1, 0, 328, 0], [330, 732.9, '2049-09-02', 31, 1, 0, 329, 0], [331, 732.9, '2049-10-02', 30, 1, 0, 330, 0], [332, 732.9, '2049-11-02', 31, 1, 0, 331, 0], [333, 732.9, '2049-12-02', 30, 1, 0, 332, 0], [334, 732.9, '2050-01-02', 31, 1, 0, 333, 0], [335, 732.9, '2050-02-02', 31, 1, 0, 334, 0], [336, 732.9, '2050-03-02', 28, 1, 0, 335, 0], [337, 732.9, '2050-04-02', 31, 1, 0, 336, 0], [338, 782.9, '2050-05-02', 30, 1, 0, 337, 0], [339, 732.9, '2050-06-02', 31, 1, 0, 338, 0], [340, 732.9, '2050-07-02', 30, 1, 0, 339, 0], [341, 732.9, '2050-08-02', 31, 1, 0, 340, 0], [342, 732.9, '2050-09-02', 31, 1, 0, 341, 0], [343, 732.9, '2050-10-02', 30, 1, 0, 342, 0], [344, 732.9, '2050-11-02', 31, 1, 0, 343, 0], [345, 732.9, '2050-12-02', 30, 1, 0, 344, 0], [346, 732.9, '2051-01-02', 31, 1, 0, 345, 0], [347, 732.9, '2051-02-02', 31, 1, 0, 346, 0], [348, 732.9, '2051-03-02', 28, 1, 0, 347, 0], [349, 732.9, '2051-04-02', 31, 1, 0, 348, 0], [350, 782.9, '2051-05-02', 30, 1, 0, 349, 0], [351, 732.9, '2051-06-02', 31, 1, 0, 350, 0], [352, 732.9, '2051-07-02', 30, 1, 0, 351, 0], [353, 732.9, '2051-08-02', 31, 1, 0, 352, 0], [354, 732.9, '2051-09-02', 31, 1, 0, 353, 0], [355, 732.9, '2051-10-02', 30, 1, 0, 354, 0], [356, 732.9, '2051-11-02', 31, 1, 0, 355, 0], [357, 732.9, '2051-12-02', 30, 1, 0, 356, 0], [358, 732.9, '2052-01-02', 31, 1, 0, 357, 0], [359, 732.9, '2052-02-02', 31, 1, 0, 358, 0], [360, 732.9, '2052-03-02', 29, 1, 0, 359, 0], [361, 732.9, '2052-04-02', 31, 1, 0, 360, 0], [362, 782.9, '2052-05-02', 30, 1, 0, 361, 0], [363, 732.9, '2052-06-02', 31, 1, 0, 362, 0], [364, 732.9, '2052-07-02', 30, 1, 0, 363, 0], [365, 732.9, '2052-08-02', 31, 1, 0, 364, 0], [366, 732.9, '2052-09-02', 31, 1, 0, 365, 0], [367, 732.9, '2052-10-02', 30, 1, 0, 366, 0], [368, 732.9, '2052-11-02', 31, 1, 0, 367, 0], [369, 732.9, '2052-12-02', 30, 1, 0, 368, 0], [370, 732.9, '2053-01-02', 31, 1, 0, 369, 0], [371, 732.9, '2053-02-02', 31, 1, 0, 370, 0], [372, 732.9, '2053-03-02', 28, 1, 0, 371, 0], [373, 732.9, '2053-04-02', 31, 1, 0, 372, 0], [374, 726.66, '2053-05-02', 30, 1, 0, 373, 0]]
self.drawing_calendar = [[0, '50000', '2022-04-01', 0, 0, 0, 0, 0], [1, '45000', '2022-06-13', 73, 30, 0, 5, 0], [2, '45000', '2022-08-25', 73, 72, 0, 6, 0], [3, '40000', '2022-11-06', 73, 72, 0, 7, 0], [4, '40000', '2023-01-18', 73, 72, 0, 8, 0], [5, '20000', '2023-04-01', 73, 72, 0, 9, 0]]
def calculate_arp(self):
try:
result = newton(self.arp_general, 0)
print("ARP result: ", result)
except Exception as e:
print(f"Calculation failed! Exception: {e}")
def arp_general(self, x):
left_sum = 0
right_sum = 0
for i in range(len(self.drawing_calendar)):
amount = float(self.table[i][1])
day_part = float(self.table[i][4] / self.year_days)
week_part = float(self.table[i][5] / 52)
month_part = float((self.table[i][6]) / 12)
year_part = self.table[i][7]
coef_pow = year_part + month_part + week_part + day_part
left_sum += amount / ((1 + x) ** coef_pow)
for i in range(len(self.table)):
amount = float(self.table[i][1])
day_part = float(self.table[i][4] / self.year_days)
week_part = float(self.table[i][5] / 52)
month_part = float((self.table[i][6]) / 12)
year_part = self.table[i][7]
coef_pow = year_part + month_part + week_part + day_part
right_sum += amount / ((1 + x) ** coef_pow)
return left_sum - right_sum
a = ARP()
a.calculate_arp()
I found a bug with help of my friend. The problem was that left_sum used wrong data. It should use data from self.drawing_calendar instead of self.table.

Convert Bytes into BufferedReader object in Python?

The title of this question is the same as this one, and I have voted to reopoen the question.
I want to convert a byte object into a BufferedReader one, and here is my attempts(after referring to many articles):
import numpy as np
from PIL import Image as PILImage
from io import BytesIO
img_np = np.asarray([[[16, 16, 16], [2, 2, 2], [0, 0, 0], [6, 6, 6], [8, 8, 8], [0, 0, 0], [21, 21, 21], [3, 3, 3], [0, 0, 0], [62, 62, 62]], [[0, 0, 0], [71, 71, 71], [142, 142, 142], [107, 107, 107], [99, 99, 99], [101, 101, 101], [4, 4, 4], [86, 86, 86], [99, 99, 99], [146, 146, 146]], [[162, 162, 162], [203, 203, 203], [192, 192, 192], [228, 228, 228], [191, 191, 191], [178, 178, 178], [222, 222, 222], [200, 200, 200], [198, 198, 198], [182, 182, 182]], [[117, 117, 117], [178, 178, 178], [199, 199, 199], [214, 214, 214], [222, 222, 222], [208, 208, 208], [255, 255, 255], [251, 251, 251], [219, 219, 219], [255, 255, 255]], [[0, 0, 0], [0, 0, 0], [80, 80, 80], [169, 169, 169], [193, 193, 193], [238, 238, 238], [239, 239, 239], [243, 243, 243], [254, 254, 254], [230, 230, 230]], [[20, 20, 20], [20, 20, 20], [9, 9, 9], [1, 1, 1], [130, 130, 130], [194, 194, 194], [216, 216, 216], [255, 255, 255], [252, 252, 252], [255, 255, 255]], [[9, 9, 9], [0, 0, 0], [0, 0, 0], [0, 0, 0], [3, 3, 3], [44, 44, 44], [191, 191, 191], [217, 217, 217], [248, 248, 248], [225, 225, 225]], [[0, 0, 0], [11, 11, 11], [3, 3, 3], [11, 11, 11], [6, 6, 6], [15, 15, 15], [0, 0, 0], [153, 153, 153], [255, 255, 255], [253, 253, 253]], [[0, 0, 0], [5, 5, 5], [1, 1, 1], [4, 4, 4], [8, 8, 8], [4, 4, 4], [3, 3, 3], [0, 0, 0], [159, 159, 159], [241, 241, 241]], [[10, 10, 10], [9, 9, 9], [6, 6, 6], [2, 2, 2], [0, 0, 0], [0, 0, 0], [3, 3, 3], [20, 20, 20], [0, 0, 0], [185, 185, 185]]])
im = PILImage.fromarray(img_np.astype(np.uint8))
# im.save('./temp.jpeg', "JPEG")
# f = open('./temp.jpeg', 'rb')
# print(type(f))
#
b_handle = io.BytesIO()
im.save(b_handle, format="JPEG")
# b = im.tobytes()
print(type(b_handle))
b = b_handle.read()
print(type(b))
print(b)
im.save(b_handle, format="JPEG")
b_br = io.BufferedReader(b_handle)
print(type(b_br))
b = b_br.read()
print(type(b))
print(b)
The output is as below:
<class '_io.BytesIO'>
<class 'bytes'>
b''
<class '_io.BufferedReader'>
<class 'bytes'>
b''
It seems that the file like objects are empty. I know that for the b_handle I can get the value by b_handle.getvalue() but for the bufferedreader it doesn't work as a file object.
How can I convert a byte string into a bufferedreader object, the same as I open a file?
You are almost there. Once you save the image bytes into the buffer you need to seek(Change stream position) to byte offset 0 prior to the read call.
b_handle = io.BytesIO()
im.save(b_handle, format="JPEG")
b_handle.seek(0)
b_handle.name = "temp.jpeg"
b_br = io.BufferedReader(b_handle)
b = b_br.read()
Example,
>>> from io import BytesIO, BufferedReader
>>>
>>> b_handle = BytesIO()
>>> b_handle.write(b"Hello World")
11
>>> b_handle.seek(0) # This is important.
0
>>> br = BufferedReader(b_handle)
>>> br
<_io.BufferedReader>
>>> br.read()
b'Hello World'

Reduce image dimension with mapping

I have a .png image with four colors in it. If I convert the image to a numpy array I get an array with the following dimensions: [length X height X 3], with length == height.
How can I reduce the dimension with mapping the colors?
This is the current structure:
array([[[ 0, 65, 101],
[ 0, 65, 101],
[ 0, 65, 101],
...,
[ 0, 65, 101],
[ 0, 65, 101],
[ 0, 65, 101]],
[[ 0, 65, 101],
[163, 219, 232],
[163, 219, 232],
...,
[ 0, 65, 101],
[163, 219, 232],
[ 0, 65, 101]],
[[ 0, 65, 101],
[163, 219, 232],
[ 0, 65, 101],
...,
[ 0, 65, 101],
[163, 219, 232],
[ 0, 65, 101]],
...,
[[ 0, 65, 101],
[163, 219, 232],
[ 0, 65, 101],
...,
[ 0, 65, 101],
[ 0, 65, 101],
[ 0, 65, 101]],
[[ 0, 65, 101],
[163, 219, 232],
[163, 219, 232],
...,
[163, 219, 232],
[163, 219, 232],
[ 0, 65, 101]],
[[ 0, 65, 101],
[ 0, 65, 101],
[ 0, 65, 101],
...,
[ 0, 65, 101],
[ 0, 65, 101],
[ 0, 65, 101]]], dtype=uint8)
And I would like an array with two dimensions, and every value in the i'th row and j'th column would correspond to the color it had in the third dimension. So if the original image had 7 X 7 X 3 dimension with four colors, the output would be something like this:
array([[0, 1, 1, 3, 3, 3, 0],
[0, 2, 1, 1, 1, 1, 0],
[0, 2, 0, 1, 2, 1, 0],
[0, 3, 1, 1, 3, 1, 0],
[0, 1, 0, 0, 3, 0, 0],
[0, 1, 1, 1, 1, 1, 0],
[0, 0, 0, 0, 0, 0, 0]])
The values in the forementioned arrays are all made up, so they don't correspond to each other, I have just tried to represent the concept.
I read the picture as:
from PIL import Image
import numpy as np
img = Image.open('image.png')
imgarray = np.asarray(img)
print(imgarray)
You can use numpy.unique for this. For example, here's a 3x5 image that has just three colors:
In [105]: img
Out[105]:
array([[[10, 20, 30],
[ 5, 5, 0],
[ 5, 5, 0],
[ 5, 5, 0],
[ 0, 0, 0]],
[[ 5, 5, 0],
[ 5, 5, 0],
[ 0, 0, 0],
[ 0, 0, 0],
[ 0, 0, 0]],
[[10, 20, 30],
[10, 20, 30],
[10, 20, 30],
[10, 20, 30],
[ 5, 5, 0]]])
Call numpy.unique on the reshaped image. The first two dimensions are flattened into a single dimension, and then axis=0 is used so we get the unique colors. inv will holds the array of "inverses", i.e. the indices into colors of the original values.
In [106]: colors, inv = np.unique(img.reshape(-1, 3), axis=0, return_inverse=True)
In [107]: colors
Out[107]:
array([[ 0, 0, 0],
[ 5, 5, 0],
[10, 20, 30]])
In [108]: inv
Out[108]: array([2, 1, 1, 1, 0, 1, 1, 0, 0, 0, 2, 2, 2, 2, 1])
Reshape inv to get the array of indices into colors with the same shape as the original image:
In [109]: inv.reshape(img.shape[:2])
Out[109]:
array([[2, 1, 1, 1, 0],
[1, 1, 0, 0, 0],
[2, 2, 2, 2, 1]])

Why do values in my 3D numpy array change when I write it to file?

strange problem in that I have a 3D array full of labels (lets say from 1-36) called labelled_stack. These are just areas where the value in the array is equal to the label given. A quick 2D example with 5 labels would be something like:
labelled_stack = (0 0 0 0 0 0 0 0 0 0)
(0 1 1 0 0 0 2 2 2 0)
(0 1 0 0 3 0 0 2 2 0)
(0 0 0 3 3 0 0 0 0 0)
(0 4 0 0 3 0 0 0 5 0)
(0 4 0 0 0 0 5 5 5 0)
(0 0 0 0 0 0 0 0 0 0)
But imagine its a numpy array...
I have tried using cv2.imwrite and scipy.misc.imsave to save the stack, but when i do this and then open them, their values have changed so that the values = 5, now equal 255, and the values equal to 1, equal 51 etc. This is not acceptable. I need the values to stay as they are.. increasing values of integers in steps of 1. i know cv2 and scipy.misc both write as 8bit images but that shouldnt mean I get this error.
I have even re-read the images back into python to check this is going on and it is.
My labelled_stack is of type np.uint32
Edit to include save commands:
for j in np.arange(0,l,1):
misc.imsave(save_path+Folder+'/Labelled/slice_{:04d}of{:}.tif'.format(j+1,l),labelled_stack[:,:,j])
or...
cv2.imwrite(save_path+Folder+'/Labelled/slice_{:04d}of{:}.tif'.format(j+1,l),labelled_stack[:,:,j])
The function imwrite saves the data in an 8 bit format. When the function converts your 32 bit data to 8 bit, it scales the data to use the full bit depth of 8 bits, so for example the maximum of your data is scaled up to 255. See scipy imsave saves wrong values for another example.
To avoid this, convert your data to numpy.uint8 before saving it. In the following example, a has data type numpy.uint32.
In [98]: from scipy.misc import imsave, imread
In [99]: a
Out[99]:
array([[0, 3, 2, 2, 4, 0, 3, 0],
[2, 0, 0, 3, 3, 1, 3, 0],
[2, 4, 4, 0, 2, 3, 1, 3],
[0, 1, 3, 1, 0, 0, 0, 4],
[2, 1, 1, 2, 1, 1, 3, 1],
[0, 4, 0, 1, 0, 0, 2, 3],
[3, 1, 3, 3, 3, 2, 3, 4],
[0, 4, 1, 4, 2, 2, 0, 2]], dtype=uint32)
The astype method is used to convert the array to numpy.uint8 before giving it to the imsave function.
In [100]: imsave("a.tif", a.astype(np.uint8)) # Convert to 8 bit before saving
Read it back with scipy.misc.imread:
In [101]: b = imread("a.tif")
In [102]: b
Out[102]:
array([[0, 3, 2, 2, 4, 0, 3, 0],
[2, 0, 0, 3, 3, 1, 3, 0],
[2, 4, 4, 0, 2, 3, 1, 3],
[0, 1, 3, 1, 0, 0, 0, 4],
[2, 1, 1, 2, 1, 1, 3, 1],
[0, 4, 0, 1, 0, 0, 2, 3],
[3, 1, 3, 3, 3, 2, 3, 4],
[0, 4, 1, 4, 2, 2, 0, 2]], dtype=uint8)
Note that when the data is read with scipy.misc.imread, the result has data type numpy.uint8.
Another recommended option, especially if you are trying to preserve data that requires 16 bits or 32 bits, is to use tifffile.
In this example, a has data type numpy.uint32, with some values that are larger than can fit in an unsigned 8 bit number.
In [152]: from tifffile import imsave, imread
In [153]: a
Out[153]:
array([[ 0, 5, 10, 15, 20, 25, 30, 35, 40, 45],
[ 50, 55, 60, 65, 70, 75, 80, 85, 90, 95],
[100, 105, 110, 115, 120, 125, 130, 135, 140, 145],
[150, 155, 160, 165, 170, 175, 180, 185, 190, 195],
[200, 205, 210, 215, 220, 225, 230, 235, 240, 245],
[250, 255, 260, 265, 270, 275, 280, 285, 290, 295],
[300, 305, 310, 315, 320, 325, 330, 335, 340, 345],
[350, 355, 360, 365, 370, 375, 380, 385, 390, 395],
[400, 405, 410, 415, 420, 425, 430, 435, 440, 445],
[450, 455, 460, 465, 470, 475, 480, 485, 490, 495]], dtype=uint32)
In [154]: imsave("a.tif", a)
In [155]: b = imread("a.tif")
In [156]: b
Out[156]:
array([[ 0, 5, 10, 15, 20, 25, 30, 35, 40, 45],
[ 50, 55, 60, 65, 70, 75, 80, 85, 90, 95],
[100, 105, 110, 115, 120, 125, 130, 135, 140, 145],
[150, 155, 160, 165, 170, 175, 180, 185, 190, 195],
[200, 205, 210, 215, 220, 225, 230, 235, 240, 245],
[250, 255, 260, 265, 270, 275, 280, 285, 290, 295],
[300, 305, 310, 315, 320, 325, 330, 335, 340, 345],
[350, 355, 360, 365, 370, 375, 380, 385, 390, 395],
[400, 405, 410, 415, 420, 425, 430, 435, 440, 445],
[450, 455, 460, 465, 470, 475, 480, 485, 490, 495]], dtype=uint32)
As you can read in the documentation:
The function imwrite saves the image to the specified file. The image format is chosen based on the filename extension (see imread() for the list of extensions). Only 8-bit (or 16-bit unsigned (CV_16U) in case of PNG, JPEG 2000, and TIFF) single-channel or 3-channel (with ‘BGR’ channel order) images can be saved using this function.
You said that "labelled_stack is of type np.uint32" (32-bit unsigned int). That is the problem. You need to handle the type convertion yourself.

Categories