We Keep Coding

Color coding cell error containing specific values

I'm currently stuck when it comes to color coding cells with specific values containing =="PPI". I'm using openpyxl to save an excel sheet that will state the color code RedFill if these cells in column B contain the value "PPI". I have a feeling I'm missing something specific in my code.
from openpyxl import load_workbook
from openpyxl.styles import Font, Color, Alignment, Border, Side, PatternFill
filename='DB_1.xlsx'
workbook = load_workbook(filename='DB_1.xlsx')
sheet = workbook.active
bold_font = Font(bold=True)
big_blue_text = Font(color="000066CC", size=11)
center_aligned_text = Alignment(horizontal="center")
redFill = PatternFill(start_color='FFFF0000',end_color='FFFF0000',fill_type='solid')
cell_1= sheet["G1"]
cell_1
cell_1.value = " Notes "
cell_1.value
cell_2 = sheet["A1"]
cell_2
cell_2.value = " Primary Key"
cell_2.value
sheet.auto_filter.ref = "A1:G1"
for c in sheet["A1:G1"][0]:
c.font = bold_font
for a in sheet["A1:G1"][0]:
a.font = big_blue_text
for b in sheet["A1:G1"][0]:
b.alignment = center_aligned_text
for d in sheet["B:B"]:
if d == "PPI":
d.PatternFill = RedFill
More specifically:
for d in sheet["B:B"]:
if d == "PPI":
d.PatternFill = RedFill
Below are the samples data that I need to color code that contains "PPI":
enter image description here

There are few changes required...
The d == "PPI" will return false as it is not searching for a
pattern. You need to search for the substring using something like
if "PPI" in d.value
If true, you need to use d.fill = redFill, not d.PatternFill
If not already doing it, do save the file
I was able to make this work updating the code like this. Hope this helps.
for d in sheet["B:B"]:
if "PPI" in d.value:
d.fill = redFill
workbook.save('new.xlsx') ## Or whatever name you want the updated file to be in

Camelot switches characters around

I'm trying to parse tables in a PDF using Camelot. The cells have multiple lines of texts in them, and some have an empty line separating portions of the text:
First line
Second line
Third line
I would expect this to be parsed as First line\nSecond line\n\nThird line (notice the double line breaks), but I get this instead: T\nFirst line\nSecond line\nhird line. The first character after a double-line-break moves to the beginning of the text, and I only get a single line-break instead.
I also tried using tabula, but that one messes up de entire table (data-frame actually) when there is an empty row in the table, and also in case of some words it puts a space between the characters.
EDIT:
My main issue is the removal of multiple line-breaks. The other one I could fix from code if I knew where the empty lines were.

my friend, can you check the example here
https://camelot-py.readthedocs.io/en/master/user/advanced.html#improve-guessed-table-rows
tables = camelot.read_pdf('group_rows.pdf', flavor='stream', row_tol=10)
tables[0].df
I solved the same problem with the code below
tables = camelot.read_pdf(file, flavor = 'stream', table_areas=['24,618,579,93'], columns=['67,315,369,483,571'], row_tol=10,strip_text='t\r\n\v')

I also encountered the same problem in case of a double line break. It was Switching Characters around as its doing in your case. I Spent some time looking at the code and i did some changes and fixed the issue. You can use the below code.
After Adding the below code, instead of using camelot.read_pdf, use the custom method i made read_pdf_custom()
And for a better experience, i suggest you using camelot v==0.8.2
import sys
import warnings
from camelot import read_pdf
from camelot import handlers
from camelot.core import TableList
from camelot.parsers import Lattice
from camelot.parsers.base import BaseParser
from camelot.core import Table
import camelot
from camelot.utils import validate_input, remove_extra,TemporaryDirectory,get_page_layout,get_text_objects,get_rotation,is_url,download_url,scale_image,scale_pdf,segments_in_bbox,text_in_bbox,merge_close_lines,get_table_index,compute_accuracy,compute_whitespace
from camelot.image_processing import (
adaptive_threshold,
find_lines,
find_contours,
find_joints,
)
class custom_lattice(Lattice):
def _generate_columns_and_rows(self, table_idx, tk):
# select elements which lie within table_bbox
t_bbox = {}
v_s, h_s = segments_in_bbox(
tk, self.vertical_segments, self.horizontal_segments
)
custom_horizontal_indexes=[]
custom_vertical_indexes=[]
for zzz in self.horizontal_text:
try:
h_extracted_text=self.find_between(str(zzz),"'","'").strip()
h_text_index=self.find_between(str(zzz),"LTTextLineHorizontal","'").strip().split(",")
custom_horizontal_indexes.append(h_text_index[1])
except:
pass
inserted=0
for xxx in self.vertical_text:
v_extracted_text=self.find_between(str(xxx),"'","'").strip()
v_text_index=self.find_between(str(xxx),"LTTextLineVertical","'").strip().split(",")
custom_vertical_indexes.append(v_text_index[1])
vertical_second_index=v_text_index[1]
try:
horizontal_index=custom_horizontal_indexes.index(vertical_second_index)
self.horizontal_text.insert(horizontal_index,xxx)
except Exception as exxx:
pass
self.vertical_text=[]
t_bbox["horizontal"] = text_in_bbox(tk, self.horizontal_text)
t_bbox["vertical"] = text_in_bbox(tk, self.vertical_text)
t_bbox["horizontal"].sort(key=lambda x: (-x.y0, x.x0))
t_bbox["vertical"].sort(key=lambda x: (x.x0, -x.y0))
self.t_bbox = t_bbox
cols, rows = zip(*self.table_bbox[tk])
cols, rows = list(cols), list(rows)
cols.extend([tk[0], tk[2]])
rows.extend([tk[1], tk[3]])
cols = merge_close_lines(sorted(cols), line_tol=self.line_tol)
rows = merge_close_lines(sorted(rows, reverse=True), line_tol=self.line_tol)
cols = [(cols[i], cols[i + 1]) for i in range(0, len(cols) - 1)]
rows = [(rows[i], rows[i + 1]) for i in range(0, len(rows) - 1)]
return cols, rows, v_s, h_s
def _generate_table(self, table_idx, cols, rows, **kwargs):
print("\n")
v_s = kwargs.get("v_s")
h_s = kwargs.get("h_s")
if v_s is None or h_s is None:
raise ValueError("No segments found on {}".format(self.rootname))
table = Table(cols, rows)
table = table.set_edges(v_s, h_s, joint_tol=self.joint_tol)
table = table.set_border()
table = table.set_span()
pos_errors = []
for direction in ["vertical", "horizontal"]:
for t in self.t_bbox[direction]:
indices, error = get_table_index(
table,
t,
direction,
split_text=self.split_text,
flag_size=self.flag_size,
strip_text=self.strip_text,
)
if indices[:2] != (-1, -1):
pos_errors.append(error)
indices = Lattice._reduce_index(
table, indices, shift_text=self.shift_text
)
for r_idx, c_idx, text in indices:
temp_text=text.strip().replace("\n","")
if len(temp_text)==1:
text=temp_text
table.cells[r_idx][c_idx].text = text
accuracy = compute_accuracy([[100, pos_errors]])
if self.copy_text is not None:
table = Lattice._copy_spanning_text(table, copy_text=self.copy_text)
data = table.data
table.df = pd.DataFrame(data)
table.shape = table.df.shape
whitespace = compute_whitespace(data)
table.flavor = "lattice"
table.accuracy = accuracy
table.whitespace = whitespace
table.order = table_idx + 1
table.page = int(os.path.basename(self.rootname).replace("page-", ""))
# for plotting
_text = []
_text.extend([(t.x0, t.y0, t.x1, t.y1) for t in self.horizontal_text])
_text.extend([(t.x0, t.y0, t.x1, t.y1) for t in self.vertical_text])
table._text = _text
table._image = (self.image, self.table_bbox_unscaled)
table._segments = (self.vertical_segments, self.horizontal_segments)
table._textedges = None
return table
class PDFHandler(handlers.PDFHandler):
def parse(
self, flavor="lattice", suppress_stdout=False, layout_kwargs={}, **kwargs
):
tables = []
with TemporaryDirectory() as tempdir:
for p in self.pages:
self._save_page(self.filepath, p, tempdir)
pages = [
os.path.join(tempdir, f"page-{p}.pdf") for p in self.pages
]
parser = custom_lattice(**kwargs) if flavor == "lattice" else Stream(**kwargs)
for p in pages:
t = parser.extract_tables(
p, suppress_stdout=suppress_stdout, layout_kwargs=layout_kwargs
)
tables.extend(t)
return TableList(sorted(tables))
def read_pdf_custom(
filepath,
pages="1",
password=None,
flavor="lattice",
suppress_stdout=False,
layout_kwargs={},
**kwargs
):
if flavor not in ["lattice", "stream"]:
raise NotImplementedError(
"Unknown flavor specified." " Use either 'lattice' or 'stream'"
)
with warnings.catch_warnings():
if suppress_stdout:
warnings.simplefilter("ignore")
validate_input(kwargs, flavor=flavor)
p = PDFHandler(filepath, pages=pages, password=password)
kwargs = remove_extra(kwargs, flavor=flavor)
tables = p.parse(
flavor=flavor,
suppress_stdout=suppress_stdout,
layout_kwargs=layout_kwargs,
**kwargs
)
return tables

How to convert a CSV table into COCO format in python?

I have a CSV table with the following columns:
column_names = ['image_id', 'xmin', 'ymin', 'width', 'height', 'xmax','ymax']
where xmin, ymin, xmax and ymax represent the bounding box that encloses some object; width and height, the image dimensions; and image_id, the file name (.JPG file). Since I want to do object detection, I need to convert this table into COCO format. Amazingly enough, I can't find any answer to this inquiry in the internet.

I had the same issue before, then I found this code it is very helpful
you will need to change the column names to this columns and update the csv file
column_names =['filename','class','width', 'height','xmin','ymin','xmax','ymax']
then try this code
import numpy as np
import json
import pandas as pd
path = 'annotations.csv' # the path to the CSV file
save_json_path = 'traincoco.json'
data = pd.read_csv(path)
images = []
categories = []
annotations = []
category = {}
category["supercategory"] = 'none'
category["id"] = 0
category["name"] = 'None'
categories.append(category)
data['fileid'] = data['filename'].astype('category').cat.codes
data['categoryid']= pd.Categorical(data['class'],ordered= True).codes
data['categoryid'] = data['categoryid']+1
data['annid'] = data.index
def image(row):
image = {}
image["height"] = row.height
image["width"] = row.width
image["id"] = row.fileid
image["file_name"] = row.filename
return image
def category(row):
category = {}
category["supercategory"] = 'None'
category["id"] = row.categoryid
category["name"] = row[2]
return category
def annotation(row):
annotation = {}
area = (row.xmax -row.xmin)*(row.ymax - row.ymin)
annotation["segmentation"] = []
annotation["iscrowd"] = 0
annotation["area"] = area
annotation["image_id"] = row.fileid
annotation["bbox"] = [row.xmin, row.ymin, row.xmax -row.xmin,row.ymax-row.ymin ]
annotation["category_id"] = row.categoryid
annotation["id"] = row.annid
return annotation
for row in data.itertuples():
annotations.append(annotation(row))
imagedf = data.drop_duplicates(subset=['fileid']).sort_values(by='fileid')
for row in imagedf.itertuples():
images.append(image(row))
catdf = data.drop_duplicates(subset=['categoryid']).sort_values(by='categoryid')
for row in catdf.itertuples():
categories.append(category(row))
data_coco = {}
data_coco["images"] = images
data_coco["categories"] = categories
data_coco["annotations"] = annotations
json.dump(data_coco, open(save_json_path, "w"), indent=4)

(Python) Change pagesize and format of PDF file generated with xtopdf

I want to convert an xlsx with Python. I used the modules tablib and xtopdf to build a well structured table. Works excellent! Unfortunately the content does not fit on one pdf page. So I wanted to change the pagesize and format to horizontal A3. But I don't know how that could work. My code:
import random
import tablib
from openpyxl import load_workbook
from xtopdf import PDFWriter
from pyPdf import PdfFileWriter, PdfFileReader
workbook = load_workbook('C:/Users/user1/Testexcel.xlsx', guess_types=True, data_only=True)
worksheet = workbook.get_sheet_by_name('Testsheet')
ws_range = worksheet.iter_rows('A4:H6')
# Helper function to output a string to both screen and PDF.
def print_and_write(pw, strng):
print strng
pw.writeLine(strng)
# Create an empty Dataset and set its headers.
data = tablib.Dataset()
data.headers = ['col1', 'col2', 'col3', 'col4']
widths = [30, 20, 10, 20] # Display widths for columns.
for row in ws_range:
col1 = str(row[0].value)
col2 = str(row[1].value)
col3 = str(row[2].value)
col4 = str(row[3].value)
columns = [col1, col2, col3, col4]
row = [ str(col).center(widths[idx]) for idx, col in enumerate(columns) ]
data.append(row)
# Set up the PDFWriter.
pw = PDFWriter('C:/Users/user1/Test.pdf')
pw.setFont('Courier', 10)
pw.setHeader('Test')
pw.setFooter('Test')
# Generate header and data rows as strings; output them to screen and PDF.
separator = '-' * sum(widths)
print_and_write(pw, separator)
# Output headers
header_strs = [ header.center(widths[idx]) for idx, header in enumerate(data.headers) ]
print_and_write(pw, ''.join(header_strs))
print_and_write(pw, separator)
# Output data
for row in data:
print_and_write(pw, ''.join(row))
print_and_write(pw, separator)
pw.close()
Found out that the PDFWriter from xtopdf itself instanciates an canvas object of the reportlab library. In the canvas class an attribute pagesize is declared which is setted by default to 'A4'. But if I change the entry to 'A3' the result pdf still is in 'A4'.
class Canvas(textobject._PDFColorSetter):
from reportlab.pdfgen import canvas
c = canvas.Canvas("hello.pdf")
from reportlab.lib.units import inch
# move the origin up and to the left
c.translate(inch,inch)
# define a large font
c.setFont("Helvetica", 80)
# choose some colors
c.setStrokeColorRGB(0.2,0.5,0.3)
c.setFillColorRGB(1,0,1)
# draw a rectangle
c.rect(inch,inch,6*inch,9*inch, fill=1)
# make text go straight up
c.rotate(90)
# change color
c.setFillColorRGB(0,0,0.77)
# say hello (note after rotate the y coord needs to be negative!)
c.drawString(3*inch, -3*inch, "Hello World")
c.showPage()
c.save()
"""
def __init__(self,filename,
pagesize='A3',
bottomup = 1,
pageCompression=None,
encoding = None,
invariant = None,
verbosity=0):
"""Create a canvas of a given size. etc.
You may pass a file-like object to filename as an alternative to
a string.
Most of the attributes are private - we will use set/get methods
as the preferred interface. Default page size is A4."""
if pagesize is None: pagesize = 'A3'
if encoding is None: encoding = rl_config.defaultEncoding
if invariant is None: invariant = rl_config.invariant
self._filename = filename
self._encodingName = encoding
self._doc = pdfdoc.PDFDocument(encoding,
compression=pageCompression,
invariant=invariant, filename=filename)
#this only controls whether it prints 'saved ...' - 0 disables
self._verbosity = verbosity
#this is called each time a page is output if non-null
self._onPage = None
self._pagesize = pagesize
self._pageRotation = 0
#self._currentPageHasImages = 0
self._pageTransition = None
self._pageDuration = None
self._destinations = {} # dictionary of destinations for cross indexing.
self.setPageCompression(pageCompression)
self._pageNumber = 1 # keep a count
#self3 = [] #where the current page's marking operators accumulate
# when we create a form we need to save operations not in the form
self._codeStack = []
self._restartAccumulators() # restart all accumulation state (generalized, arw)
self._annotationCount = 0
self._outlines = [] # list for a name tree
self._psCommandsBeforePage = [] #for postscript tray/font commands
self._psCommandsAfterPage = [] #for postscript tray/font commands
#PostScript has the origin at bottom left. It is easy to achieve a top-
#down coord system by translating to the top of the page and setting y
#scale to -1, but then text is inverted. So self.bottomup is used
#to also set the text matrix accordingly. You can now choose your
#drawing coordinates.
self.bottomup = bottomup
self.imageCaching = rl_config.defaultImageCaching
self._make_preamble()
self.init_graphics_state()
self.state_stack = []
edit: I think the changes in the reportlab module are not accepted by the system. Tried to remove the dictionary reportlab and tried to import it then in the commandline. Ironically it works ylthough python should not find that module anymore.

try this
from reportlab.pdfgen import canvas
from reportlab.lib.units import mm
c = canvas.Canvas("hello.pdf", pagesize = (297 * mm, 420 * mm))
# or (420 * mm, 297 * mm) if you want it in portrait format
# values for inch: 11.69 * inch , 16.53 * inch
#the following would create an empty page
c.showPage()
c.save()

Just forked a project named xtopdf at bitbucket and made the following change:
##------------------------ PDFWriter.__init__ ----------------------------
- def __init__(self, pdf_fn):
+ def __init__(self, pdf_fn, pagesize='A4'):
'''
Constructor.
"pdf_fn" arg is the name of the PDF file to be created.
'''
self.__pdf_fn = pdf_fn # file name of PDF file
- self.__canv = canvas.Canvas(pdf_fn) # canvas to write on
+ self.__canv = canvas.Canvas(pdf_fn, pagesize) # canvas to write on
self.__font_name = None # font name
self.__font_size = None # font size
self.__header_str = None # header string (partial)
Can you try it? use pw = PDFWriter('C:/Users/user1/Test.pdf', 'A3').

Python pptx module

I have to edit a pptx template which contain tables like this:
How to append values stored in my python dict to the empty fields?
I'm using a pptx module, but I couldn't find any example of doing this.
from pptx import Presentation
prs = Presentation('template.pptx')
slide = prs.slides[2] #<- This is the slide that contains the table
shape = slide.shapes #<- As I understood This gives access to the shapes
textframe=shape.textframe
textframe.clear()
prs.save('test.pptx') #<- Saves the new file
pptx module link

quote from the dev-group of python-ppty developer
-if you know its index, something like table = slide.shapes[2] would do the trick.
Then you'll need to navigate the cells before you can change their contents:
for idx, row in enumerate(table.rows):
if idx = 0: # skip header row
continue
name_cell = row.cells[0]
name_cell.text = 'foobar'
corners_cell = row.cells[1]

---in main----
table_data = [['ID', 'Name', 'Age', 'Second name'], ['1', 'Petro', 22, 'Petrovich'], ['2', 'Ivan', 32, 'Ivanovich'], ['3', 'Oles', 23, 'Marko']]
prs = Presentation(template_filepath)
slide_1 = slide_build(prs, 5)
table_draw(table_data, slide_1.shapes)
prs.save(result_filepath)
def slide_build(prs, layout):
slide = prs.slides.add_slide(prs.slide_layouts[layout])
return slide
def table_draw(table_data, shapes):
rows_number = 0
columns_number = 0
# get table size
rows_number = len(table_data)
for i, item in enumerate(table_data):
columns_number += 1
table = table_build(rows_number, columns_number, shapes)
column_coord = 0
row_coord = 0
for row_count, row in enumerate(table_data):
for item_count, row_item in enumerate(row):
table.cell(row_count + row_coord, item_count + column_coord).text = str(row_item)
def table_build(rows, cols, shapes):
left = (0.1)
top = Inches(0.7)
width = Inches(6.0)
height = Inches(0.8)
table = shapes.add_table(rows, cols, left, top, width, height).table
# set column widths
i = 0
while i
Some thing like this