We Keep Coding

Extract zip file inline in Oracle OCI - Object Storage without downloading to save time

Is it possible to extract a zip file 'inline' which is in cloud say Oracle cloud, Object storage. Meaning, without downloading it and extracting it in the o/s and re-uploading it to object storage, because the file size is big and we need to save time on upload/download.. Any sample code, with Oracle Functions, or python, java etc. ? Is it possible ? I tried with S3 browser/explorer or other similar tools, but that basically at the background, downloads and extract on local computer.

If I understand the question correctly, your use case would be that you have a compressed value on the server and want to extract it on the server and keep it there.
This is possible and mostly depends on how the values has been compressed.
If you use the Lempel-Ziv-Welch algorithm used in the UTL_COMPRESS package, you can extract it directly in PL/SQL.
For other formats like zip, you will need to use some custom Java code like the following example:
CREATE OR REPLACE
JAVA SOURCE NAMED ZIP_Java
AS
import java.io.*;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
import java.util.zip.ZipInputStream;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.BufferedOutputStream;
import java.sql.Blob;
public class Java_Class {
public static int ZipBlob(Blob inLob, Blob[] outLob, String filename) {
try {
// create the zipoutputstream from the end of the outLob
Blob zipLob = outLob[0];
OutputStream os = zipLob.setBinaryStream(1);
ZipOutputStream zos = new ZipOutputStream(os);
// add one zip entry
ZipEntry entry = new ZipEntry(filename);
zos.putNextEntry(entry);
// write data to the zip lob
long len = inLob.length();
long offset = 0;
byte[] buffer;
int chunksize = 32768;
while (offset < len) {
buffer = inLob.getBytes(offset + 1, chunksize);
if (buffer == null)
break;
zos.write(buffer, 0, buffer.length);
offset += buffer.length;
}
zos.closeEntry();
zos.close();
outLob[0] = zipLob;
} catch (Exception e) {
System.out.println("Exception: " + e.toString());
e.printStackTrace(System.out);
return 0;
}
return 1;
}
public static int UnzipBlob(Blob inLob, Blob[] outLob, String filename) {
try {
final int kBUFFER = 2048;
InputStream inps = inLob.getBinaryStream();
ZipInputStream zis = new ZipInputStream(inps);
ZipEntry entry;
Blob fileLob = outLob[0];
OutputStream os = fileLob.setBinaryStream(1);
while((entry = zis.getNextEntry()) != null) {
if (entry.getName().equalsIgnoreCase(filename)) {
byte data[] = new byte[kBUFFER];
BufferedOutputStream dest = new BufferedOutputStream(os, kBUFFER);
int count;
while ((count = zis.read(data, 0, kBUFFER)) != -1) {
dest.write(data, 0, count);
}
dest.flush();
dest.close();
}
}
zis.close();
return 1;
} catch (Exception e) {
System.out.println("Exception: " + e.toString());
e.printStackTrace();
return 0;
}
}
}
/
CREATE OR REPLACE
FUNCTION ZipBlobJava(theSource IN BLOB, theDestination IN OUT NOCOPY BLOB, theFilename IN VARCHAR2) RETURN NUMBER
AS LANGUAGE JAVA NAME 'Java_Class.ZipBlob(java.sql.Blob, java.sql.Blob[], java.lang.String) return int';
/
CREATE OR REPLACE
FUNCTION UnzipBlobJava(theSource IN BLOB, theDestination IN OUT NOCOPY BLOB, theFilename IN VARCHAR2) RETURN NUMBER
AS LANGUAGE JAVA NAME 'Java_Class.UnzipBlob(java.sql.Blob, java.sql.Blob[], java.lang.String) return int';
/

Python InDesign scripting: Get overflowing textbox from preflight for automatic resizing

Thanks to this great answer I was able to figure out how to run a preflight check for my documents using Python and the InDesign script API. Now I wanted to work on automatically adjusting the text size of the overflowing text boxes, but was unable to figure out how to retrieve a TextBox object from the Preflight object.
I referred to the API specification, but all the properties only seem to yield strings which do not uniquely define the TextBoxes, like in this example:
Errors Found (1):
Text Frame (R=2)
Is there any way to retrieve the violating objects from the Preflight, in order to operate on them later on? I'd be very thankful for additional input on this matter, as I am stuck!

If all you need is to find and to fix the overset errors I'd propose this solution:
Here is the simple Extendscript to fix the text overset error. It decreases the font size in the all overflowed text frames in active document:
var doc = app.activeDocument;
var frames = doc.textFrames.everyItem().getElements();
var f = frames.length
while(f--) {
var frame = frames[f];
if (frame.overflows) resize_font(frame)
}
function resize_font(frame) {
app.scriptPreferences.enableRedraw = false;
while (frame.overflows) {
var texts = frame.parentStory.texts.everyItem().getElements();
var t = texts.length;
while(t--) {
var characters = texts[t].characters.everyItem().getElements();
var c = characters.length;
while (c--) characters[c].pointSize = characters[c].pointSize * .99;
}
}
app.scriptPreferences.enableRedraw = true;
}
You can save it in any folder and run it by the Python script:
import win32com.client
app = win32com.client.Dispatch('InDesign.Application.CS6')
doc = app.Open(r'd:\temp\test.indd')
profile = app.PreflightProfiles.Item('Stackoverflow Profile')
print('Profile name:', profile.name)
process = app.PreflightProcesses.Add(doc, profile)
process.WaitForProcess()
errors = process.processResults
print('Errors:', errors)
if errors[:4] != 'None':
script = r'd:\temp\fix_overset.jsx' # <-- here is the script to fix overset
print('Run script', script)
app.DoScript(script, 1246973031) # run the jsx script
# 1246973031 --> ScriptLanguage.JAVASCRIPT
# https://www.indesignjs.de/extendscriptAPI/indesign-latest/#ScriptLanguage.html
process = app.PreflightProcesses.Add(doc, profile)
process.WaitForProcess()
errors = process.processResults
print('Errors:', errors) # it should print 'None'
if errors[:4] == 'None':
doc.Save()
doc.Close()
input('\nDone... Press <ENTER> to close the window')

Thanks to the exellent answer of Yuri I was able solve my problem, although there are still some shortcomings.
In Python, I load my documents and check if there are any problems detected during the preflight. If so, I move on to adjusting the text frames.
myDoc = app.Open(input_file_path)
profile = app.PreflightProfiles.Item(1)
process = app.PreflightProcesses.Add(myDoc, profile)
process.WaitForProcess()
results = process.processResults
if "None" not in results:
# Fix errors
script = open("data/script.jsx")
app.DoScript(script.read(), 1246973031, variables.resize_array)
process.WaitForProcess()
results = process.processResults
# Check if problems were resolved
if "None" not in results:
info_fail(card.name, "Error while running preflight")
myDoc.Close(1852776480)
return FLAG_PREFLIGHT_FAIL
I load the JavaScript file stored in script.jsx, that consists of several components. I start by extracting the arguments and loading all the pages, since I want to handle them individually. I then collect all text frames on the page in an array.
var doc = app.activeDocument;
var pages = doc.pages;
var resizeGroup = arguments[0];
var condenseGroup = arguments[1];
// Loop over all available pages separately
for (var pageIndex = 0; pageIndex < pages.length; pageIndex++) {
var page = pages[pageIndex];
var pageItems = page.allPageItems;
var textFrames = [];
// Collect all TextFrames in an array
for (var pageItemIndex = 0; pageItemIndex < pageItems.length; pageItemIndex++) {
var candidate = pageItems[pageItemIndex];
if (candidate instanceof TextFrame) {
textFrames.push(candidate);
}
}
What I wanted to achieve was a setting where if one of a group of text frames was overflowing, the text size of all the text frames in this group are adjusted as well. E.g. text frame 1 overflows when set to size 8, no longer when set to size 6. Since text frame 1 is in the same group as text frame 2, both of them will be adjusted to size 6 (assuming the second frame does not overflow at this size).
In order to handle this, I pass an array containing the groups. I now check if the text frame is contained in one of these groups (which is rather tedious, I had to write my own methods since InDesign does not support modern functions like filter() as far as I am concerned...).
// Check if TextFrame overflows, if so add all TextFrames that should be the same size
for (var textFrameIndex = 0; textFrameIndex < textFrames.length; textFrameIndex++) {
var textFrame = textFrames[textFrameIndex];
// If text frame overflows, adjust it and all the frames that are supposed to be of the same size
if (textFrame.overflows) {
var foundResizeGroup = filterArrayWithString(resizeGroup, textFrame.name);
var foundCondenseGroup = filterArrayWithString(condenseGroup, textFrame.name);
var process = false;
var chosenGroup, type;
if (foundResizeGroup.length > 0) {
chosenGroup = foundResizeGroup;
type = "resize";
process = true;
} else if (foundCondenseGroup.length > 0) {
chosenGroup = foundCondenseGroup;
type = "condense";
process = true;
}
if (process) {
var foundFrames = findTextFramesFromNames(textFrames, chosenGroup);
adjustTextFrameGroup(foundFrames, type);
}
}
}
If this is the case, I adjust either the text size or the second axis of the text (which condenses the text for my variable font). This is done using the following functions:
function adjustTextFrameGroup(resizeGroup, type) {
// Check if some overflowing textboxes
if (!someOverflowing(resizeGroup)) {
return;
}
app.scriptPreferences.enableRedraw = false;
while (someOverflowing(resizeGroup)) {
for (var textFrameIndex = 0; textFrameIndex < resizeGroup.length; textFrameIndex++) {
var textFrame = resizeGroup[textFrameIndex];
if (type === "resize") decreaseFontSize(textFrame);
else if (type === "condense") condenseFont(textFrame);
else alert("Unknown operation");
}
}
app.scriptPreferences.enableRedraw = true;
}
function someOverflowing(textFrames) {
for (var textFrameIndex = 0; textFrameIndex < textFrames.length; textFrameIndex++) {
var textFrame = textFrames[textFrameIndex];
if (textFrame.overflows) {
return true;
}
}
return false;
}
function decreaseFontSize(frame) {
var texts = frame.parentStory.texts.everyItem().getElements();
for (var textIndex = 0; textIndex < texts.length; textIndex++) {
var characters = texts[textIndex].characters.everyItem().getElements();
for (var characterIndex = 0; characterIndex < characters.length; characterIndex++) {
characters[characterIndex].pointSize = characters[characterIndex].pointSize - 0.25;
}
}
}
function condenseFont(frame) {
var texts = frame.parentStory.texts.everyItem().getElements();
for (var textIndex = 0; textIndex < texts.length; textIndex++) {
var characters = texts[textIndex].characters.everyItem().getElements();
for (var characterIndex = 0; characterIndex < characters.length; characterIndex++) {
characters[characterIndex].setNthDesignAxis(1, characters[characterIndex].designAxes[1] - 5)
}
}
}
I know that this code can be improved upon (and am open to feedback), for example if a group consists of multiple text frames, the procedure will run for all of them, even though it need only be run once. I was getting pretty frustrated with the old JavaScript, and the impact is negligible. The rest of the functions are also only helper functions, which I'd like to replace with more modern version. Sadly and as already stated, I think that they are simply not available.
Thanks once again to Yuri, who helped me immensely!

Hashing algorithm Node js vs Python

I was trying to convert a hash algorithm which is written on Python to node.js
The python code looks something as
import uuid
import hashlib
import struct
CLIENT_ID = uuid.UUID('c5f92e0d-e762-32cd-98cb-8c546c410dbe')
SECRET = uuid.UUID('2cf26ff5-bd06-3245-becf-4d5a3baa704f')
data = CLIENT_ID.bytes_le + SECRET.bytes_le + struct.pack("I", 2017) + struct.pack("I", 9) + struct.pack("I", 2)
token = str(uuid.UUID(bytes_le=hashlib.sha256(data).digest()[0:16]))
The token generated is 32d86f00-eb49-2739-e957-91513d2b9969
Here the date values struct.pack values are generated using datetime but for convenient I have hard coded here.
I tried to convert the same by looking at the python doc for the respective libraries and did so far as
let CLIENT_ID = new Buffer('c5f92e0d-e762-32cd-98cb-8c546c410dbe');
let SECRET = new Buffer('2cf26ff5-bd06-3245-becf-4d5a3baa704f');
let d = new Buffer(2);
let m = new Buffer(9);
let y = new Buffer(2017);
let data = CLIENT_ID+SECRET+y+m+d;
const uuidv4 = require('uuid/v4');
const hash = crypto.createHash('sha256');
let token = uuidv4({random: hash.update(data, 'utf8').digest().slice(0, 16)}, 0);
And the hash it generates is b7b82474-eab4-4295-8318-cc258577ff9b
So, basically I am miserably missing something for the nodejs part.
Could you please guide me on where what went wrong. Thanks for the help

There's a lot of missed parts actually it tuned out.
###node parts:
new Buffer('c5')
does not represent <Buffer c5>, but <Buffer 63 35>.
To write c5 you would need to use Buffer.from([0xc5]) or Buffer.from([197]) (dec).
new Buffer(2)
does not represent <Buffer 02>, it just allocates 2 bytes.
CLIENT_ID+SECRET+y+m+d
concatenation of buffers does not work that way.
Use array of buffers and Buffer.concat([buffers]) to concatenate buffers.
###uuid parts:
it turned out that uuid operates modified version of buffers (bytes_le part in python code)
#the most interesting part:
in the python version of uuid, if no version argument is passed to uuid.UUID(...), uuid would generate an ID without fixing bits
According to the RFC-4122 4.4 uuid should fix that bits.
uuid.py skips RFC-4122 4.4
node-uuid/v4.js fixes required bits
that way even with the same results for sha256 hashing, the results between python and node implementation still would differ
python: 32d86f00-eb49-2739-e957-91513d2b9969
node: 32d86f00-eb49-4739-a957-91513d2b9969
^ ^
So, I see here 2 options
to pass version to python uuid (only for the last uuid call uuid.UUID(bytes_le=..., version=4)), that way python would return 32d86f00-eb49-4739-a957-91513d2b9969
if there's no way to change source code in python project, I guess there's an option to fork uuid and remove two lines of code in node-uuid/v4.js?
##See node version of your code below:
const uuidv4 = require('uuid/v4');
const crypto = require('crypto');
const hash = crypto.createHash('sha256');
const client_id_hex_str = "c5f92e0d-e762-32cd-98cb-8c546c410dbe".replace(/-/g, "");
const secret_hex_str = "2cf26ff5-bd06-3245-becf-4d5a3baa704f".replace(/-/g, "");
let CLIENT_ID = Buffer.from(to_bytes_le(to_bytes(client_id_hex_str, null, 16, 'big')));
let SECRET = Buffer.from(to_bytes_le(to_bytes(secret_hex_str, null, 16, 'big')));
let d = Buffer.from(to_bytes(null, 2, 4));
let m = Buffer.from(to_bytes(null, 9, 4));
let y = Buffer.from(to_bytes(null, 2017, 4));
let data = Buffer.concat([CLIENT_ID, SECRET, y, m, d]);
let hashBytes = hash.update(data, 'utf8').digest().slice(0, 16);
hashBytes = [].slice.call(hashBytes, 0);
hashBytes = Buffer.from(to_bytes_le(hashBytes));
let token = uuidv4({random: hashBytes});
console.log(token);
// https://stackoverflow.com/questions/16022556/has-python-3-to-bytes-been-back-ported-to-python-2-7
function to_bytes(hexString, number, length, endianess) {
if (hexString == null && number == null) {
throw new Error("Missing hex string or number.");
}
if (!length || isNaN(length)) {
throw new Error("Missing or invalid bytes array length number.");
}
if (hexString && typeof hexString != "string") {
throw new Error("Invalid format for hex value.");
}
if (hexString == null) {
if (isNaN(number)) {
throw new Error("Invalid number.");
}
hexString = number.toString(16);
}
let byteArray = [];
if (hexString.length % 2 !== 0) {
hexString = '0' + hexString;
}
const bitsLength = length * 2
hexString = ("0".repeat(bitsLength) + hexString).slice(-1 * bitsLength);
for (let i = 0; i < hexString.length; i += 2) {
const byte = hexString[i] + hexString[i + 1];
byteArray.push(parseInt(byte, 16));
}
if (endianess !== "big") {
byteArray = byteArray.reverse();
}
return byteArray;
}
// https://github.com/python/cpython/blob/master/Lib/uuid.py#L258
function to_bytes_le(bytes) {
const p1 = bytes.slice(0, 4).reverse();
const p2 = bytes.slice(4, 6).reverse();
const p3 = bytes.slice(6, 8).reverse();
const p4 = bytes.slice(8);
const bytes_le = [].concat.apply([], [p1, p2, p3, p4]);
return bytes_le;
}

Do you want the hashing of the data to be the same as the Python code above?
If not, you can take a look at the the Sha256 module below in NodeJS
https://www.npmjs.com/package/sha256

extract data from gmail add to spreadsheet- Google apps script

I have searched, copied and modified code, and tried to break down what others have done and I still can't get this right.
I have email receipts for an ecommerce webiste, where I am trying to harvest particular details from each email and save to a spreadsheet with a script.
Here is the entire script as I have now.
function menu(e) {
var ui = SpreadsheetApp.getUi();
ui.createMenu('programs')
.addItem('parse mail', 'grabreceipt')
.addToUi();
}
function grabreceipt() {
var ss = SpreadsheetApp.getActiveSheet();
var ss = SpreadsheetApp.getActiveSpreadsheet();
var s = ss.getSheetByName("Sheet1");
var threads = GmailApp.search("(subject:order receipt) and (after:2016/12/01)");
var a=[];
for (var i = 0; i<threads.length; i++)
{
var messages = threads[i].getMessages();
for (var j=0; j<messages.length; j++)
{
var messages = GmailApp.getMessagesForThread(threads[i]);
for (var j = 0; j < messages.length; j++) {
a[j]=parseMail(messages[j].getPlainBody());
}
}
var nextRow=s.getDataRange().getLastRow()+1;
var numRows=a.length;
var numCols=a[0].length;
s.getRange(nextRow,1,numRows,numCols).setValues(a);
}
function parseMail(body) {
var a=[];
var keystr="Order #,Subtotal:,Shipping:,Total:";
var keys=keystr.split(",");
var i,p,r;
for (i in keys) {
//p=keys[i]+(/-?\d+(,\d+)*(\.\d+(e\d+)?)?/);
p=keys[i]+"[\r\n]*([^\r^\n]*)[\r\n]";
//p=keys[i]+"[\$]?[\d]+[\.]?[\d]+$";
r=new RegExp(p,"m");
try {a[i]=body.match(p)[1];}
catch (err) {a[i]="no match";}
}
return a;
}
}
So the email data to pluck from comes as text only like this:
Order #89076
(body content, omitted)
Subtotal: $528.31
Shipping: $42.66 via Priority Mail®
Payment Method: Check Payment- Money order
Total: $570.97
Note: mywebsite order 456. Customer asked about this and that... etc.
The original code regex was designed to grab content, following the keystr values which were easily found on their own line. So this made sense:
p=keys[i]+"[\r\n]*([^\r^\n]*)[\r\n]";
This works okay, but results where the lines include more data that follows as in line Shipping: $42.66 via Priority Mail®.
My data is more blended, where I only wish to take numbers, or numbers and decimals. So I have this instead which validates on regex101.com
p=keys[i]+"[\$]?[\d]+[\.]?\d+$";
The expression only, [\$]?[\d]+[.]?\d+$ works great but I still get "no match" for each row.
Additionally, within this search there are 22 threads returned, and it populates 39 rows in the spreadsheet. I can not figure out why 39?

The reason for your regex not working like it should is because you are not escaping the "\" in the string you use to create the regex
So a regex like this
"\s?\$?(\d+\.?\d+)"
needs to be escaped like so:
"\\s?\\$?(\\d+\\.?\\d+)"
The below code is just modified from your parseEmail() to work as a snippet here. If you copy this to your app script code delete document.getElementById() lines.
Your can try your example in the snippet below it will only give you the numbers.
function parseMail(body) {
if(body == "" || body == undefined){
var body = document.getElementById("input").value
}
var a=[];
var keystr="Order #,Subtotal:,Shipping:,Total:";
var keys=keystr.split(",");
var i,p,r;
for (i in keys) {
p=keys[i]+"\\s?\\$?(\\d+\\.?\\d+)";
r=new RegExp(p,"m");
try {a[i]=body.match(p)[1];}
catch (err) {a[i]="no match";}
}
document.getElementById("output").innerHTML = a.join(";")
return a;
}
<textarea id ="input"></textarea>
<div id= "output"></div>
<input type = "button" value = "Parse" onclick = "parseMail()">
Hope that helps

How do I access fields in an active Revit schedule via RevitPythonShell/IronPython?

I'm working on an IronPython script for Revit 2016. For starters, I'm trying to access values (as text) in an active Revit schedule, and load them into a variable. This works well enough for non-calculated values.
However, some of my schedule fields are calculated. Here's a sample schedule (all values here are calculated):
Schedule Snippet
The Revit API shows 2 methods, called TableView.GetCalculatedValueName()and TableView.GetCalculatedValueText(), which I'd like to use, but don't seem to work as advertised.
doc = __revit__.ActiveUIDocument.Document
uidoc = __revit__.ActiveUIDocument
schedule = doc.ActiveView
tableData = schedule.GetTableData()
print(tableData)
tableName = schedule.GetCellText(SectionType.Header,0,0)
qty = schedule.GetCalculatedValueText(SectionType.Body,4,1)
calcValName = schedule.GetCalculatedValueName(SectionType.Body,4,1)
print(tableName)
print("Calculated Qty is: " + qty)
print("Calculated Value Name is: " + calcValName)
Running this code (in Revit) produces the following output:
88-06134-01
Calculated Qty is:
Calculated Value Name is:
I'd like to point out that using TableView.GetCellText() actually works on calculated values, but it's the GetCalculatedValueName() that I'd really like to make work here.

I have done the same thing but in c# for Revit 2019. I hope you will understand it.
You can access the values of schedule data without exporting. Firstly, get all the schedules and read the data cell by cell. Secondly, create dictionary and store data in form of key, value pairs. Now you can use the schedule data as you want. I have tried this in Revit 2019.
Here is the implementation.
public void getScheduleData(Document doc)
{
FilteredElementCollector collector = new FilteredElementCollector(doc);
IList<Element> collection = collector.OfClass(typeof(ViewSchedule)).ToElements();
foreach (Element e in collection)
{
ViewSchedule viewSchedule = e as ViewSchedule;
TableData table = viewSchedule.GetTableData();
TableSectionData section = table.GetSectionData(SectionType.Body);
int nRows = section.NumberOfRows;
int nColumns = section.NumberOfColumns;
if (nRows > 1)
{
List<List<string>> scheduleData = new List<List<string>>();
for (int i = 0; i < nRows; i++)
{
List<string> rowData = new List<string>();
for (int j = 0; j < nColumns; j++)
{
rowData.Add(viewSchedule.GetCellText(SectionType.Body, i, j));
}
scheduleData.Add(rowData);
}
List<string> columnData = scheduleData[0];
scheduleData.RemoveAt(0);
DataMapping(columnData, scheduleData);
}
}
}
public static void DataMapping(List<string> keyData, List<List<string>>valueData)
{
List<Dictionary<string, string>> items= new List<Dictionary<string, string>>();
string prompt = "Key/Value";
prompt += Environment.NewLine;
foreach (List<string> list in valueData)
{
for (int key=0, value =0 ; key< keyData.Count && value< list.Count; key++,value++)
{
Dictionary<string, string> newItem = new Dictionary<string, string>();
string k = keyData[key];
string v = list[value];
newItem.Add(k, v);
items.Add(newItem);
}
}
foreach (Dictionary<string, string> item in items)
{
foreach (KeyValuePair<string, string> kvp in item)
{
prompt += "Key: " + kvp.Key + ",Value: " + kvp.Value;
prompt += Environment.NewLine;
}
}
Autodesk.Revit.UI.TaskDialog.Show("Revit", prompt);
}