We Keep Coding

Create a raster stack in from xarray dataset in Python

I am trying to make a raster stack from an xarray dataset which I obtained from multiple netCDF files. There are 365 netCDF files each containing a 2D data of Sea Surface Temperature (SST) having height and width 3600 and 7200 respectively. To perform further operations I need to prepare a raster stack.
import netCDF4 as nc
import rasterio as rio
import numpy as np
import xarray as xr
import os
fpath = '/home/sst/2015'
pattern = '*.nc'
filelist = []
for root, dirs, files in os.walk(fpath):
for name in files:
filelist.append(os.path.join(fpath,name))
ds = xr.open_mfdataset(eflist, concat_dim='time', parallel = True) # netCDF data
ds_data = ds.sel(time='2015')['SST'] # xarray dataset with dimension 365x3600x7200.
Raster stack of this xarray will be used to extract data values at point locations. I am currently using numpy and rasterio as mentioned in rasterio documentation. By iterating over the 3D xarray following code writes 365 files to HDD and later I can read and stack those.
from rasterio.transform import from_origin
transform = from_origin(-180,90, 0.05, 0.05)
fpath = '/home/sst/sst_tif'
fname = 'sst_array_'
extname = '.tiff'
timedf = ds.time # time dimension to loop over
for i in range(len(timedf)):
np_array = np.array(ds_data[i])
iname = str(i)
fwname = fpath + fname + iname + extname
sst_tif = rio.open(fwname,
'w',
driver = 'GTiff',
height = 3600,
width = 7200,
dtype = np_array.dtype,
count = 1,
crs = 'EPSG:4326',
transform = transform)
sst_tif.write(np_array, 1)
sst_tif.close()
However this takes a very long time to process entire dataset. I also attempted converting entire xarray to a numpy array and writing all 365 layers in a single file but it freezes the Python kernel.
Is there any way I can create this raster stack in memory and do further processing without having to write it to file/s on HDD. I am trying to obtain functionality similar to that of stack function available in raster package of R.

cant save an 4d array int .txt file

I am using the sliding window technic to an image and i am extracting the mean values of pixels of each one window. So the results are someting like this [[[[215.015625][123.55036272][111.66057478]]]].now the question is how could i save all these values for every one window into a txt file or at a CSV because i want to use them for further compare similarities? whatever i tried the error is same..that it is a 4D array and not an 1D or 2D. I ll appreciate any help really.! Thank you in advance
import cv2
import matplotlib.pyplot as plt
import numpy as np
# read the image and define the stepSize and window size
# (width,height)
image2 = cv2.imread("bird.jpg")# your image path
image = cv2.resize(image2, (224, 224))
tmp = image # for drawing a rectangle
stepSize = 10
(w_width, w_height) = (60, 60 ) # window size
for x in range(0, image.shape[1] - w_width, stepSize):
for y in range(0, image.shape[0] - w_height, stepSize):
window = image[x:x + w_width, y:y + w_height, :]
# classify content of the window with your classifier and
# determine if the window includes an object (cell) or not
# draw window on image
cv2.rectangle(tmp, (x, y), (x + w_width, y + w_height), (255, 0, 0), 2) # draw rectangle on image
plt.imshow(np.array(tmp).astype('uint8'))
# show all windows
plt.show()
mean_values=[]
mean_val, std_dev = cv2.meanStdDev(image)
mean_val = mean_val[:3]
mean_values.append([mean_val])
mean_values = np.asarray(mean_values)
print(mean_values)

Human Readable Option
Assuming that you want the data to be human readable, saving the data takes a little bit more work. My search showed me that there's this solution for saving 3D data to a text file. However, it's pretty simple to extend this example to 4D for your use case. This code is taken and adapted from that post, thank you Joe Kington and David Cheung.
import numpy as np
data = np.arange(2*3*4*5).reshape((2,3,4,5))
with open('test.csv', 'w') as outfile:
# We write this header for readable, the pound symbol
# will cause numpy to ignore it
outfile.write('# Array shape: {0}\n'.format(data.shape))
# Iterating through a ndimensional array produces slices along
# the last axis. This is equivalent to data[i,:,:] in this case.
# Because we are dealing with 4D data instead of 3D data,
# we need to add another for loop that's nested inside of the
# previous one.
for threeD_data_slice in data:
for twoD_data_slice in threeD_data_slice:
# The formatting string indicates that I'm writing out
# the values in left-justified columns 7 characters in width
# with 2 decimal places.
np.savetxt(outfile, twoD_data_slice, fmt='%-7.2f')
# Writing out a break to indicate different slices...
outfile.write('# New slice\n')
And then once the data has been saved all you need to do is load it and reshape it (np.load()) will default to reading in the data as a 2D array but np.reshape() will allow us to recover the structure. Again, this code is adapted from the previous post.
new_data = np.loadtxt('test.csv')
# Note that this returned a 2D array!
print(new_data.shape)
# However, going back to 3D is easy if we know the
# original shape of the array
new_data = new_data.reshape((2,3,4,5))
# Just to check that they're the same...
assert np.all(new_data == data)
Binary Option
Assuming that human readability is not necessary, I would recommend using the built-in *.npy format which is described here. This stores the data in a binary format.
You can save the array by doing np.save('NAME_OF_ARRAY.npy', ARRAY_TO_BE_SAVED) and then load it with SAVED_ARRAY = np.load('NAME_OF_ARRAY.npy').
You can also save several numpy array in a single zip file with the np.savez() function like so np.savez('MANY_ARRAYS.npz', ARRAY_ONE, ARRAY_TWO). And you load the zipped arrays in a similar fashion SEVERAL_ARRAYS = np.load('MANY_ARRAYS.npz').

Comparing two lists of stellar x-y coordinates to find matching objects

I have two .txt files that contain the x and y pixel coordinates of thousands of stars in an image. These two different coordinate lists were the products of different data processing methods, which result in slightly different x and y values for the same object.
File 1 *the id_out is arbitrary
id_out x_out y_out m_out
0 803.6550 907.0910 -8.301
1 700.4570 246.7670 -8.333
2 802.2900 894.2130 -8.344
3 894.6710 780.0040 -8.387
File 2
xcen ycen mag merr
31.662 37.089 22.759 0.387
355.899 37.465 19.969 0.550
103.079 37.000 20.839 0.847
113.500 38.628 20.966 0.796
The objects listed in the .txt files are not organized in a way that allows me to identify the same object in both files. So, I thought for every object in file 1, which has fewer objects than file 2, I would impose a test to find the star match between file 1 and file 2. For every star in file 1, I want to find the star in file 2 that has the closest match to x-y coordinates using the distance formula:
distance= sqrt((x1-x2)^2 + (y1-y2)^2) within some tolerance of distance that I can change. Then print onto a master list the x1, y1, x2, y2, m_out, mag, and merr parameters in the file.
Here is the code I have so far, but I am not sure how to arrive at a working solution.
#/usr/bin/python
import pandas
import numpy as np
xcen_1 = np.genfromtxt('file1.txt', dtype=float, usecols=1)
ycen_1 = np.genfromtxt('file1.txt', dtype=float, usecols=2)
mag1 = np.genfromtxt('file1.txt', dtype=float, usecols=3)
xcen_2 = np.genfromtxt('file2.txt', dtype=float, usecols=0)
ycen_2 = np.genfromtxt('file2.txt', dtype=float, usecols=1)
mag2 = np.genfromtxt('file2.txt', dtype=float, usecols=2)
merr2 = np.genfromtxt('file2.txt', dtype=float, usecols=3)
tolerance=10.0
i=0
file=open('results.txt', 'w+')
file.write("column names")
for i in len(xcen_1):
dist=np.sqrt((xcen_1[i]-xcen_2[*])^2+(ycen_1[i]-ycen_2[*]^2))
if dist < tolerance:
f.write(i, xcen_1, ycen_1, xcen_2, ycen_2, mag1, mag2, merr2)
else:
pass
i=i+1
file.close
The code doesn't work as I don't know how to implement that every star in file 2 must be run through the test, as indicated by the * index (which comes from idl, in which I am more versed). Is there a solution for this logic, as opposed to the thinking in this case:
To compare two independent image coordinate lists with same scale but coordinate-grid having some rotation and shift
Thanks in advance!

You can use pandas Dataframes. Here's how:
import pandas as pd
# files containing the x and y pixel coordinates and other information
df_file1=pd.read_csv('file1.txt',sep='\s+')
df_file2=pd.read_csv('file2.txt',sep='\s+')
join=[]
for i in range(len(df_file1)):
for j in range(len(df_file2)):
dis=((df_file1['x_out'][i]-df_file2['xcen'][j])**2+(df_file1['y_out'][i]-df_file2['ycen'][j])**2)**0.5
if dis<10:
join.append({'id_out': df_file1['id_out'][i], 'x_out': df_file1['x_out'][i], 'y_out':df_file1['y_out'][i],
'm_out':df_file1['m_out'][i],'xcen':df_file2['xcen'][j],'ycen':df_file2['ycen'][j],
'mag':df_file2['mag'][j],'merr':df_file2['merr'][j]})
df_join=pd.DataFrame(join)
df_join.to_csv('results.txt', sep='\t')

How to rotate coordinates (x,y) of an image at a specific angle

To best understand, please reproduce the code in a Jupyternotebook:
I have two files: img.jpg and img.txt. Img.jpg is the image and img.txt is the face landmarks....If you plot them both, it will look like this:
I rotated the image by 24.5 degree....but how to do I also rotate the coordinates?
import cv2
img = cv2.imread('img.jpg')
plt.imshow(img)
plt.show()
# In[130]:
landmarks = []
with open('img.txt') as f:
for line in f:
landmarks.extend([float(number) for number in line.split()])
landmarks.pop(0) #Remove first line.
#Store all points inside the variable.
landmarkPoints = [] #Store the points in this
for j in range(int(len(landmarks))):
if j%2 == 1:
continue
landmarkPoints.append([int(landmarks[j]),int(landmarks[j+1])])
# In[ ]:
def rotate_bound(image, angle):
# grab the dimensions of the image and then determine the
# center
(h, w) = image.shape[:2]
(cX, cY) = (w // 2, h // 2)
# grab the rotation matrix (applying the negative of the
# angle to rotate clockwise), then grab the sine and cosine
# (i.e., the rotation components of the matrix)
M = cv2.getRotationMatrix2D((cX, cY), -angle, 1.0)
cos = np.abs(M[0, 0])
sin = np.abs(M[0, 1])
# compute the new bounding dimensions of the image
nW = int((h * sin) + (w * cos))
nH = int((h * cos) + (w * sin))
# adjust the rotation matrix to take into account translation
M[0, 2] += (nW / 2) - cX
M[1, 2] += (nH / 2) - cY
# perform the actual rotation and return the image
return cv2.warpAffine(image, M, (nW, nH))
# In[131]:
imgcopy = img.copy()
for i in range(len(landmarkPoints)):
cv2.circle(imgcopy, (landmarkPoints[i][0], landmarkPoints[i][1]), 5, (0, 255, 0), -1)
plt.imshow(imgcopy)
plt.show()
landmarkPoints
# In[146]:
print(img.shape)
print(rotatedImage.shape)
# In[153]:
face_angle = 24.5
rotatedImage = rotate_bound(img, -face_angle)
for i in range(len(landmarkPoints)):
x,y = (landmarkPoints[i][0], landmarkPoints[i][1])
cv2.circle(rotatedImage, (int(x),int(y)), 5, (0, 255, 0), -1)
plt.imshow(rotatedImage)
plt.show()
Please download img.jpg and img.txt for reproducing this: https://drive.google.com/file/d/1FhQUFvoKi3t7TrIepx2Es0mBGAfT755w/view?usp=sharing
I tried this function, but y-axis is wrong
def rotatePoint(angle, pt):
a = np.radians(angle)
cosa = np.cos(a)
sina = np.sin(a)
return pt[0]*cosa - pt[1]*sina, pt[0] * sina + pt[1] * cosa
Edit: The above function gives me this result:

Although it has been long time since the question was asked. But I have decided to answer it as it has no accepted answer yet, even if it is a well accepted question. I have added a lot of comments to make the implementation clear. So, the code is hopefully self-explanatory. But I am also describing the ImageAugmentation's parameters for further clarification:
Here, original_data_dir is the directory to the parent folder, where all of the image's folders exists (yes it can read from multiple image folders). This parameter is compulsory.
augmentation_data_dir is the folder directory where you want to save the outputs. The program will automatically create all sub-folders inside of the output directory just like they appear in input directory. It is totally optional, it can generate the output directory by mimicking the input directory by appending the string _augmentation after the input folder name.
keep_original is another optional parameter. In many cases you may want to keep the original image with the augmented images in the output folder. If you want so, make it True (default).
num_of_augmentations_per_image is the total number of augmented images to be generated from each image. Although you wanted only rotation, but this program is designed to do other augmentations as well, change them, add or remove them as you need. I have also added a link to documentation where you will find other augmentations which can be introduced here in this code. It is defaulted to 3, if you keep the original image, there will be 3 + 1 = 4 images will be generated in the output.
discard_overflow_and_underflow is for handling the case where due to spatial transformation, the augmented points along with the image underneath can go outside of image's resolution, you can optionally keep them. But it is discarded here by default. Again, it will also discard images having width or height values <= 0. Defaulted to True.
put_landmarks means if you want the landmarks to be shown in the output. Make it True or False as required. It is False by default.
Hope you like it!
import logging
import imgaug as ia
import imgaug.augmenters as iaa
from imgaug.augmentables import Keypoint
from imgaug.augmentables import KeypointsOnImage
import os
import cv2
import re
SEED = 31 # To reproduce the result
class ImageAugmentation:
def __init__(self, original_data_dir, augmentation_data_dir = None, keep_original = True, num_of_augmentations_per_image = 3, discard_overflow_and_underflow = True, put_landmarks = False):
self.original_data_dir = original_data_dir
if augmentation_data_dir != None:
self.augmentation_data_dir = augmentation_data_dir
else:
self.augmentation_data_dir = self.original_data_dir + '_augmentation'
# Most of the time you will want to keep the original images along with the augmented images
self.keep_original = keep_original
# For example for self.num_of_augmentations_per_image = 3, from 1 image we will get 3 more images, totaling 4 images.
self.num_of_augmentations_per_image = num_of_augmentations_per_image
# if discard_overflow_and_underflow is True, the program will discard all augmentation where landmark (and image underneath) goes outside of image resolution
self.discard_overflow_and_underflow = discard_overflow_and_underflow
# Optionally put landmarks on output images
self.put_landmarks = put_landmarks
def get_base_annotations(self):
"""This method reads all the annotation files (.txt) and make a list
of annotations to be used by other methods.
"""
# base_annotations are the annotations which has come with the original images.
base_annotations = []
def get_info(content):
"""This utility function reads the content of a single annotation
file and returns the count of total number of points and a list of coordinates
of the points inside a dictionary.
As you have provided in your question, the annotation file looks like the following:
106
282.000000 292.000000
270.000000 311.000000
259.000000 330.000000
.....
.....
Here, the first line is the number of points.
The second and the following lines gives their coordinates.
"""
# As all the lines newline separated, hence splitting them
# accordingly first
lines = content.split('\n')
# The first line is the total count of the point, we can easily get it just by counting the points
# so we are not taking this information.
# From the second line to the end all lines are basically the coordinate values
# of each point (in each line). So, going to each of the lines (from the second line)
# and taking the coordinates as tuples.
# We will end up with a list of tuples and which will be inserted to the dict "info"
# under the key "point_coordinates"
points = []
for line in lines[1:]:
# Now each of the line can be splitted into two numbers representing coordinates
try:
# Keeping inside try block, as some of the lines might be accidentally contain
# a single number, or it can be the case that there might be some extra newlines
# where there is no number.
col, row = line.split(' ')
points.append((float(col), float(row)))
except:
pass
# Returns: List of tuples
return points
for subdir, dirs, files in os.walk(self.original_data_dir):
for file in files:
ext = os.path.splitext(file)[-1].lower()
# Looping through image files (instead of annotation files which are in '.txt' format)
# because image files can have very different extensions and we have to preserve them.
# Whereas, all the annotation files are assumed to be in '.txt' format.
# Annotation file's (.txt) directory will be generated from here.
if ext not in ['.txt']:
input_image_file_dir = os.path.join(subdir, file)
# As the image filenames and associated annotation text filenames are the same,
# so getting the common portion of them, it will be used to generate the annotation
# file's directory.
# Also assuming, there are no dots (.) in the input_annotation_file_dir except before the file extension.
image_annotation_base_dir = self.split_extension(input_image_file_dir)[0]
# Generating annotation file's directory
input_annotation_file_dir = image_annotation_base_dir + '.txt'
try:
with open(input_annotation_file_dir, 'r') as f:
content = f.read()
image_annotation_base_dir = os.path.splitext(input_annotation_file_dir)[0]
if os.path.isfile(input_image_file_dir):
image = cv2.imread(input_image_file_dir)
# Taking image's shape is basically surving dual purposes.
# First of all, we will need the image's shape for sanity checking after augmentation
# Again, if any of the input image is corrupt this following line will through exception
# and we will be able to skip that corrput image.
image_shape = image.shape # height (y), width (x), channels (depth)
# Collecting the directories of original annotation files and their contents.
# The same folder structure will be used to save the augmented data.
# As the image filenames and associated annotation text filenames are the same, so
base_annotations.append({'image_file_dir': input_image_file_dir,
'annotation_data': get_info(content = content),
'image_resolution': image_shape})
except:
logging.error(f"Unable to read the file: {input_annotation_file_dir}...SKIPPED")
return base_annotations
def get_augmentation(self, base_annotation, seed):
image_file_dir = base_annotation['image_file_dir']
image_resolution = base_annotation['image_resolution']
list_of_coordinates = base_annotation['annotation_data']
ia.seed(seed)
# We have to provide the landmarks in specific format as imgaug requires
landmarks = []
for coordinate in list_of_coordinates:
# coordinate[0] is along x axis (horizontal axis) and coordinate[1] is along y axis (vertical axis) and (left, top) corner is (0, 0)
landmarks.append(Keypoint(x = coordinate[0], y = coordinate[1]))
landmarks_on_original_img = KeypointsOnImage(landmarks, shape = image_resolution)
original_image = cv2.imread(image_file_dir)
"""
Here the magic happens. If you only want rotation then remove other transformations from here.
You can even add other various types of augmentation, see documentation here:
# Documentation for image augmentation with keypoints
https://imgaug.readthedocs.io/en/latest/source/examples_keypoints.html
# Here you will find other possible transformations
https://imgaug.readthedocs.io/en/latest/source/examples_basics.html
"""
seq = iaa.Sequential([
iaa.Affine(
scale={"x": (0.8, 1.2), "y": (0.8, 1.2)}, # scale images to 80-120% of their size, individually per axis
translate_percent={"x": (-0.2, 0.2), "y": (-0.2, 0.2)}, # translate by -20 to +20 percent (per axis)
rotate=(-90, 90), # rotate by -90 to +90 degrees; for specific angle (say 30 degree) use rotate = (30)
shear=(-16, 16), # shear by -16 to +16 degrees
)
], random_order=True) # Apply augmentations in random order
augmented_image, _landmarks_on_augmented_img = seq(image = original_image, keypoints = landmarks_on_original_img)
# Now for maintaining consistency, making the augmented landmarks to maintain same data structure like base_annotation
# i.e, making it a list of tuples.
landmarks_on_augmented_img = []
for index in range(len(landmarks_on_original_img)):
landmarks_on_augmented_img.append((_landmarks_on_augmented_img[index].x,
_landmarks_on_augmented_img[index].y))
return augmented_image, landmarks_on_augmented_img
def split_extension(self, path):
# Assuming there is no dots (.) except just before extension
# Returns [directory_of_file_without_extension, extension]
return os.path.splitext(path)
def sanity_check(self, landmarks_aug, image_resolution):
# Returns false if the landmark is outside of image resolution.
# Or, if the resolution is faulty.
for index in range(len(landmarks_aug)):
if landmarks_aug[index][0] < 0 or landmarks_aug[index][1] < 0:
return False
if landmarks_aug[index][0] >= image_resolution[1] or landmarks_aug[index][1] >= image_resolution[0]:
return False
if image_resolution[0] <= 0:
return False
if image_resolution[1] <= 0:
return False
return True
def serialize(self, serialization_data, image):
"""This method to write the annotation file and the corresponding image.
"""
# Now it is time to actually writing the image file and the annotation file!
# We have to make sure the output folder exists
# and "head" is the folder's directory here.
image_file_dir = serialization_data['image_file_dir']
annotation_file_dir = self.split_extension(image_file_dir)[0] + '.txt'
point_coordinates = serialization_data['annotation_data'] # List of tuples
total_points = len(point_coordinates)
# Getting the corresponding output folder for current image
head, tail = os.path.split(image_file_dir)
# Creating the folder if it doesn't exist
if not os.path.isdir(head):
os.makedirs(head)
# Writing annotation file
with open(annotation_file_dir, 'w') as f:
s = ""
s += str(total_points)
s += '\n'
for point in point_coordinates:
s += "{:.6f}".format(point[0]) + ' ' + "{:6f}".format(point[1]) + '\n'
f.write(s)
if self.put_landmarks:
# Optionally put landmarks in the output images.
for index in range(total_points):
cv2.circle(image, (int(point_coordinates[index][0]), int(point_coordinates[index][1])), 2, (255, 255, 0), 2)
cv2.imwrite(image_file_dir, image)
def augmentat_with_landmarks(self):
base_annotations = self.get_base_annotations()
for base_annotation in base_annotations:
if self.keep_original == True:
# As we are basically copying the same original data in new directory, changing the original image's directory with the new one with re.sub()
base_data = {'image_file_dir': re.sub(self.original_data_dir, self.augmentation_data_dir, base_annotation['image_file_dir']),
'annotation_data': base_annotation['annotation_data']}
self.serialize(serialization_data = base_data, image = cv2.imread(base_annotation['image_file_dir']))
for index in range(self.num_of_augmentations_per_image):
# Getting a new augmented image in each iteration from the same base image.
# Seeding (SEED) for reproducing same result across all execution in the future.
# Also seed must be different for each iteration, otherwise same looking augmentation will be generated.
image_aug, landmarks_aug = self.get_augmentation(base_annotation, seed = SEED + index)
# As for spatial transformations for some images, the landmarks can go outside of the image.
# So, we have to discard those cases (optionally).
if self.sanity_check(landmarks_aug, base_annotation['image_resolution']) or not self.discard_overflow_and_underflow:
# Getting the filename without extension to insert an index number in between to generate a new filename for augmented image
filepath_without_ext, ext = self.split_extension(base_annotation['image_file_dir'])
# As we are writing newly generated images to similar sub folders (just in different base directory)
# that is replacing original_data_dir with augmentation_data_dir.
# So, to do this we are using, re.sub(what_to_replace, with_which_to_replace, from_where_to_replace)
filepath_for_aug_img_without_ext = re.sub(self.original_data_dir, self.augmentation_data_dir, filepath_without_ext)
new_filepath_wo_ext = filepath_for_aug_img_without_ext + '_' + str(index)
augmentation_data = {
'image_file_dir': new_filepath_wo_ext + ext,
'annotation_data': landmarks_aug
}
self.serialize(serialization_data = augmentation_data, image = image_aug)
# Make put_landmarks = False if you do not want landmarks to be shown in output
# original_data_dir is the single parent folder directory inside of which all image folder(s) exist.
img_aug = ImageAugmentation(original_data_dir = 'parent/folder/directory/of/img/folder', put_landmarks = True)
img_aug.augmentat_with_landmarks()
Following is a snapshot of sample output of the code:
Please note that, I have used a package imgaug. I will suggest you to install the 0.4.0 version, as I have found it to be working. See the reason here and it's accepted answer.

When you try things like that it's very important to choose the proper coordinate system. In your case you have to put the origin (0,0) point in the center of the image.
Once you apply the rotation to the coordinates with the origin point in the center, the face points will be properly aligned on the new image.

Convert Longitude, latitude to Pixel values using GDAL

I have a satellite GeoTIFF Image and a corresponding OSM file with only the highways. I want to convert the longitude latitude value in the OSM file to the pixels and want to highlight highway on the satellite image.
I have tried several methods that are explained on StackExchange. But I get the negative and same pixel value for every longitude and latitude values. Could somebody explain, what am I missing?
Here is the information of the image that I have gathered using OTB application.
Here is the code that i am using.
from osgeo import gdal, osr
import numpy as np
import xml.etree.ElementTree as xml
src_filename = 'image.tif'
dst_filename = 'foo.tiff'
def readLongLat(path):
lonlatList = []
latlongtuple = ()
root = xml.parse(path).getroot()
for i in root:
if i.tag == "node":
latlong = []
lat = float(i.attrib["lat"])
long = float(i.attrib["lon"])
latlong.append(lat)
latlong.append(long)
lonlatList.append(latlong)
return lonlatList
# Opens source dataset
src_ds = gdal.Open(src_filename)
format = "GTiff"
driver = gdal.GetDriverByName(format)
# Open destination dataset
dst_ds = driver.CreateCopy(dst_filename, src_ds, 0)
# Get raster projection
epsg = 4269 # http://spatialreference.org/ref/sr-org/lambert_conformal_conic_2sp/
srs = osr.SpatialReference()
srs.ImportFromEPSG(epsg)
# Make WGS84 lon lat coordinate system
world_sr = osr.SpatialReference()
world_sr.SetWellKnownGeogCS('WGS84')
transform = src_ds.GetGeoTransform()
gt = [transform[0],transform[1],0,transform[3],0,-transform[5]]
#Reading the osm file
lonlat = readLongLat("highways.osm")
# Transform lon lats into XY
coord_transform = osr.CoordinateTransformation(world_sr, srs)
newpoints = coord_transform.TransformPoints(lonlat) # list of XYZ tuples
# Make Inverse Geotransform (try:except due to gdal version differences)
try:
success, inverse_gt = gdal.InvGeoTransform(gt)
except:
inverse_gt = gdal.InvGeoTransform(gt)
# [Note 1] Set pixel values
marker_array_r = np.array([[255]], dtype=np.uint8)
marker_array_g = np.array([[0]], dtype=np.uint8)
marker_array_b = np.array([[0]], dtype=np.uint8)
for x,y,z in newpoints:
pix_x = int(inverse_gt[0] + inverse_gt[1] * x + inverse_gt[2] * y)
pix_y = int(inverse_gt[3] + inverse_gt[4] * x + inverse_gt[5] * y)
dst_ds.GetRasterBand(1).WriteArray(marker_array_r, pix_x, pix_y)
dst_ds.GetRasterBand(2).WriteArray(marker_array_g, pix_x, pix_y)
dst_ds.GetRasterBand(3).WriteArray(marker_array_b, pix_x, pix_y)
# Close files
dst_ds = None
src_ds = None

Something I have tried recently is using the xarray module. I think of xarray as a hybrid between pandas and numpy that allows you to store information as an array but access it using simply .sel requests. Docs here.
UPDATE: Seems as if rasterio and xarray are required to be installed for the below method to work. See link.
It is a much simpler way of translating a GeoTiff file to a user-friendly array. See my example below:
import xarray as xr
ds = xr.open_rasterio("/path/to/image.tif")
# Insert your lat/lon/band below to extract corresponding pixel value
ds.sel(band=2, lat=19.9, lon=39.5, method='nearest').values
>>> [10.3]
This does not answer your question directly, but may help you identify a different (and probably simpler) approach that I've recently switched to.
Note: obviously care needs to be taken to ensure that your lat/lon pairs are in the same coordinate system as the GeoTiff file, but I think you're handling that anyway.

I was able to do that using the library geoio.
import geoio
img = geoio.GeoImage(src_filename)
pix_x, pix_y = img.proj_to_raster(lon,lat)