We Keep Coding

Extract zip file inline in Oracle OCI - Object Storage without downloading to save time

Is it possible to extract a zip file 'inline' which is in cloud say Oracle cloud, Object storage. Meaning, without downloading it and extracting it in the o/s and re-uploading it to object storage, because the file size is big and we need to save time on upload/download.. Any sample code, with Oracle Functions, or python, java etc. ? Is it possible ? I tried with S3 browser/explorer or other similar tools, but that basically at the background, downloads and extract on local computer.

If I understand the question correctly, your use case would be that you have a compressed value on the server and want to extract it on the server and keep it there.
This is possible and mostly depends on how the values has been compressed.
If you use the Lempel-Ziv-Welch algorithm used in the UTL_COMPRESS package, you can extract it directly in PL/SQL.
For other formats like zip, you will need to use some custom Java code like the following example:
CREATE OR REPLACE
JAVA SOURCE NAMED ZIP_Java
AS
import java.io.*;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
import java.util.zip.ZipInputStream;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.BufferedOutputStream;
import java.sql.Blob;
public class Java_Class {
public static int ZipBlob(Blob inLob, Blob[] outLob, String filename) {
try {
// create the zipoutputstream from the end of the outLob
Blob zipLob = outLob[0];
OutputStream os = zipLob.setBinaryStream(1);
ZipOutputStream zos = new ZipOutputStream(os);
// add one zip entry
ZipEntry entry = new ZipEntry(filename);
zos.putNextEntry(entry);
// write data to the zip lob
long len = inLob.length();
long offset = 0;
byte[] buffer;
int chunksize = 32768;
while (offset < len) {
buffer = inLob.getBytes(offset + 1, chunksize);
if (buffer == null)
break;
zos.write(buffer, 0, buffer.length);
offset += buffer.length;
}
zos.closeEntry();
zos.close();
outLob[0] = zipLob;
} catch (Exception e) {
System.out.println("Exception: " + e.toString());
e.printStackTrace(System.out);
return 0;
}
return 1;
}
public static int UnzipBlob(Blob inLob, Blob[] outLob, String filename) {
try {
final int kBUFFER = 2048;
InputStream inps = inLob.getBinaryStream();
ZipInputStream zis = new ZipInputStream(inps);
ZipEntry entry;
Blob fileLob = outLob[0];
OutputStream os = fileLob.setBinaryStream(1);
while((entry = zis.getNextEntry()) != null) {
if (entry.getName().equalsIgnoreCase(filename)) {
byte data[] = new byte[kBUFFER];
BufferedOutputStream dest = new BufferedOutputStream(os, kBUFFER);
int count;
while ((count = zis.read(data, 0, kBUFFER)) != -1) {
dest.write(data, 0, count);
}
dest.flush();
dest.close();
}
}
zis.close();
return 1;
} catch (Exception e) {
System.out.println("Exception: " + e.toString());
e.printStackTrace();
return 0;
}
}
}
/
CREATE OR REPLACE
FUNCTION ZipBlobJava(theSource IN BLOB, theDestination IN OUT NOCOPY BLOB, theFilename IN VARCHAR2) RETURN NUMBER
AS LANGUAGE JAVA NAME 'Java_Class.ZipBlob(java.sql.Blob, java.sql.Blob[], java.lang.String) return int';
/
CREATE OR REPLACE
FUNCTION UnzipBlobJava(theSource IN BLOB, theDestination IN OUT NOCOPY BLOB, theFilename IN VARCHAR2) RETURN NUMBER
AS LANGUAGE JAVA NAME 'Java_Class.UnzipBlob(java.sql.Blob, java.sql.Blob[], java.lang.String) return int';
/

Crypto++ Signing with PKCS1v15 padding with different algorithms

I am wondering if Crypto++ only signs files using SHA (RSASSA_PKCS1v15_SHA_Signer)?
I have been using pyCryptodome to do the signing and verifying, but I want to make a C++ application that does the same. In Python, I can sign the files with any of the supported hashing (SHA3/BLAKE2B etc..) algorithms. At the very least I want to support signing using SHA256 in C++.
std::string Hasher::sign(const std::string& message)
{
RSASSA_PKCS1v15_SHA_Signer signer(m_privateKey);
size_t length = signer.MaxSignatureLength();
SecByteBlock signature(length);
length = signer.SignMessage(rng, (CryptoPP::byte*)message.c_str(), message.size(), signature);
signature.resize(length);
//Want the signature as a hex
return toHex(signature, signature.size());
}
However, I want to be able to do something similar I do in Python:
def(message, key, passphrase, hashType):
rsakey = RSA.importKey(key,passphrase)
signer = PKCS1_v1_5.new(rsakey)
# Get the hash class based os user given hashType ex ("SHA256" returns "SHA256.New()")
digest = getHash(hashType)
digest.update(message.encode("ascii"))
return signer.sign(digest).hex()
If I chose the same private key, and use the hashType "SHA" I get the same signature result as my C++ code would.

So I found an answer to it. So partially following the tutorial found here
and from RSASS<PSS, SHA256>::Signer signer(privateKey); you could add the PKCS1v15 padding as such RSASS<PKCS1v15, SHA256>::Signer signer(privateKey);
So the final code would look something like this:
std::string Hasher::sign(const std::string& message)
{
CryptoPP::RSASS<CryptoPP::PKCS1v15, CryptoPP::SHA256>::Signer signer(m_privateKey);
size_t length = signer.MaxSignatureLength();
SecByteBlock signature(length);
length = signer.SignMessage(rng, (CryptoPP::byte*)message.c_str(), message.size(), signature);
signature.resize(length);
//Want the signature as a hex
return toHex(signature, signature.size());
}
So super similar, I still don't know why RSASSA_PKCS1v15_SHA_Signer class still exists. To throw in something extra my final goal, as I could find nothing on it except a random 13-year-old question on a mailing list :
How to sign and verify a file with Crypto++ (Like you sign files in pyCrypto)
Signing file:
std::string Hasher::sign()
{
CryptoPP::RSASS<CryptoPP::PKCS1v15, CryptoPP::SHA256>::Signer signer(m_privateKey);
size_t length = signer.MaxSignatureLength();
SecByteBlock signature(length);
// Get bytes of the file
const auto& data = getFileBytes();
//AFAIK you need to create a signer PK_MessageAccumulator object to acheive this
CryptoPP::PK_MessageAccumulator* pSigMsgAcc = signer.NewSignatureAccumulator(rng);
//Update it with your data, toBinary_const is just a short for (const CryptoPP::byte*)data.data()
pSigMsgAcc->Update(toBinary_const(data), data.size());
length = signer.Sign(rng, pSigMsgAcc, signature);
signature.resize(length);
//I return it as hex as people can actually read the signature and verify it themself
return toHex(signature, signature.size());
}
Verifying a file
bool Hasher::verify(const std::string& signature, const std::string& type)
{
CryptoPP::RSASS<CryptoPP::PKCS1v15, CryptoPP::SHA256>::Verifier verifier(m_publicKey);
const auto& data = getFileBytes();
//Create a verifyer PK_MessageAccumulator object
CryptoPP::PK_MessageAccumulator* pVrfyMsgAcc = verifier.NewVerificationAccumulator();
pVrfyMsgAcc->Update(toBinary_const(data), data.size());
//Here comes the down side of using hex as signatrue, I convert it to a binary string
auto toVerify = toBinaryString(signature);
//Then I have to convert the binary string to binary
// (It does not like me to convert the hex directly to binary)
verifier.InputSignature(*pVrfyMsgAcc, toBinary_const(toVerify), toVerify.size());
return verifier.Verify(pVrfyMsgAcc);
}

Hashing algorithm Node js vs Python

I was trying to convert a hash algorithm which is written on Python to node.js
The python code looks something as
import uuid
import hashlib
import struct
CLIENT_ID = uuid.UUID('c5f92e0d-e762-32cd-98cb-8c546c410dbe')
SECRET = uuid.UUID('2cf26ff5-bd06-3245-becf-4d5a3baa704f')
data = CLIENT_ID.bytes_le + SECRET.bytes_le + struct.pack("I", 2017) + struct.pack("I", 9) + struct.pack("I", 2)
token = str(uuid.UUID(bytes_le=hashlib.sha256(data).digest()[0:16]))
The token generated is 32d86f00-eb49-2739-e957-91513d2b9969
Here the date values struct.pack values are generated using datetime but for convenient I have hard coded here.
I tried to convert the same by looking at the python doc for the respective libraries and did so far as
let CLIENT_ID = new Buffer('c5f92e0d-e762-32cd-98cb-8c546c410dbe');
let SECRET = new Buffer('2cf26ff5-bd06-3245-becf-4d5a3baa704f');
let d = new Buffer(2);
let m = new Buffer(9);
let y = new Buffer(2017);
let data = CLIENT_ID+SECRET+y+m+d;
const uuidv4 = require('uuid/v4');
const hash = crypto.createHash('sha256');
let token = uuidv4({random: hash.update(data, 'utf8').digest().slice(0, 16)}, 0);
And the hash it generates is b7b82474-eab4-4295-8318-cc258577ff9b
So, basically I am miserably missing something for the nodejs part.
Could you please guide me on where what went wrong. Thanks for the help

There's a lot of missed parts actually it tuned out.
###node parts:
new Buffer('c5')
does not represent <Buffer c5>, but <Buffer 63 35>.
To write c5 you would need to use Buffer.from([0xc5]) or Buffer.from([197]) (dec).
new Buffer(2)
does not represent <Buffer 02>, it just allocates 2 bytes.
CLIENT_ID+SECRET+y+m+d
concatenation of buffers does not work that way.
Use array of buffers and Buffer.concat([buffers]) to concatenate buffers.
###uuid parts:
it turned out that uuid operates modified version of buffers (bytes_le part in python code)
#the most interesting part:
in the python version of uuid, if no version argument is passed to uuid.UUID(...), uuid would generate an ID without fixing bits
According to the RFC-4122 4.4 uuid should fix that bits.
uuid.py skips RFC-4122 4.4
node-uuid/v4.js fixes required bits
that way even with the same results for sha256 hashing, the results between python and node implementation still would differ
python: 32d86f00-eb49-2739-e957-91513d2b9969
node: 32d86f00-eb49-4739-a957-91513d2b9969
^ ^
So, I see here 2 options
to pass version to python uuid (only for the last uuid call uuid.UUID(bytes_le=..., version=4)), that way python would return 32d86f00-eb49-4739-a957-91513d2b9969
if there's no way to change source code in python project, I guess there's an option to fork uuid and remove two lines of code in node-uuid/v4.js?
##See node version of your code below:
const uuidv4 = require('uuid/v4');
const crypto = require('crypto');
const hash = crypto.createHash('sha256');
const client_id_hex_str = "c5f92e0d-e762-32cd-98cb-8c546c410dbe".replace(/-/g, "");
const secret_hex_str = "2cf26ff5-bd06-3245-becf-4d5a3baa704f".replace(/-/g, "");
let CLIENT_ID = Buffer.from(to_bytes_le(to_bytes(client_id_hex_str, null, 16, 'big')));
let SECRET = Buffer.from(to_bytes_le(to_bytes(secret_hex_str, null, 16, 'big')));
let d = Buffer.from(to_bytes(null, 2, 4));
let m = Buffer.from(to_bytes(null, 9, 4));
let y = Buffer.from(to_bytes(null, 2017, 4));
let data = Buffer.concat([CLIENT_ID, SECRET, y, m, d]);
let hashBytes = hash.update(data, 'utf8').digest().slice(0, 16);
hashBytes = [].slice.call(hashBytes, 0);
hashBytes = Buffer.from(to_bytes_le(hashBytes));
let token = uuidv4({random: hashBytes});
console.log(token);
// https://stackoverflow.com/questions/16022556/has-python-3-to-bytes-been-back-ported-to-python-2-7
function to_bytes(hexString, number, length, endianess) {
if (hexString == null && number == null) {
throw new Error("Missing hex string or number.");
}
if (!length || isNaN(length)) {
throw new Error("Missing or invalid bytes array length number.");
}
if (hexString && typeof hexString != "string") {
throw new Error("Invalid format for hex value.");
}
if (hexString == null) {
if (isNaN(number)) {
throw new Error("Invalid number.");
}
hexString = number.toString(16);
}
let byteArray = [];
if (hexString.length % 2 !== 0) {
hexString = '0' + hexString;
}
const bitsLength = length * 2
hexString = ("0".repeat(bitsLength) + hexString).slice(-1 * bitsLength);
for (let i = 0; i < hexString.length; i += 2) {
const byte = hexString[i] + hexString[i + 1];
byteArray.push(parseInt(byte, 16));
}
if (endianess !== "big") {
byteArray = byteArray.reverse();
}
return byteArray;
}
// https://github.com/python/cpython/blob/master/Lib/uuid.py#L258
function to_bytes_le(bytes) {
const p1 = bytes.slice(0, 4).reverse();
const p2 = bytes.slice(4, 6).reverse();
const p3 = bytes.slice(6, 8).reverse();
const p4 = bytes.slice(8);
const bytes_le = [].concat.apply([], [p1, p2, p3, p4]);
return bytes_le;
}

Do you want the hashing of the data to be the same as the Python code above?
If not, you can take a look at the the Sha256 module below in NodeJS
https://www.npmjs.com/package/sha256

How do I access fields in an active Revit schedule via RevitPythonShell/IronPython?

I'm working on an IronPython script for Revit 2016. For starters, I'm trying to access values (as text) in an active Revit schedule, and load them into a variable. This works well enough for non-calculated values.
However, some of my schedule fields are calculated. Here's a sample schedule (all values here are calculated):
Schedule Snippet
The Revit API shows 2 methods, called TableView.GetCalculatedValueName()and TableView.GetCalculatedValueText(), which I'd like to use, but don't seem to work as advertised.
doc = __revit__.ActiveUIDocument.Document
uidoc = __revit__.ActiveUIDocument
schedule = doc.ActiveView
tableData = schedule.GetTableData()
print(tableData)
tableName = schedule.GetCellText(SectionType.Header,0,0)
qty = schedule.GetCalculatedValueText(SectionType.Body,4,1)
calcValName = schedule.GetCalculatedValueName(SectionType.Body,4,1)
print(tableName)
print("Calculated Qty is: " + qty)
print("Calculated Value Name is: " + calcValName)
Running this code (in Revit) produces the following output:
88-06134-01
Calculated Qty is:
Calculated Value Name is:
I'd like to point out that using TableView.GetCellText() actually works on calculated values, but it's the GetCalculatedValueName() that I'd really like to make work here.

I have done the same thing but in c# for Revit 2019. I hope you will understand it.
You can access the values of schedule data without exporting. Firstly, get all the schedules and read the data cell by cell. Secondly, create dictionary and store data in form of key, value pairs. Now you can use the schedule data as you want. I have tried this in Revit 2019.
Here is the implementation.
public void getScheduleData(Document doc)
{
FilteredElementCollector collector = new FilteredElementCollector(doc);
IList<Element> collection = collector.OfClass(typeof(ViewSchedule)).ToElements();
foreach (Element e in collection)
{
ViewSchedule viewSchedule = e as ViewSchedule;
TableData table = viewSchedule.GetTableData();
TableSectionData section = table.GetSectionData(SectionType.Body);
int nRows = section.NumberOfRows;
int nColumns = section.NumberOfColumns;
if (nRows > 1)
{
List<List<string>> scheduleData = new List<List<string>>();
for (int i = 0; i < nRows; i++)
{
List<string> rowData = new List<string>();
for (int j = 0; j < nColumns; j++)
{
rowData.Add(viewSchedule.GetCellText(SectionType.Body, i, j));
}
scheduleData.Add(rowData);
}
List<string> columnData = scheduleData[0];
scheduleData.RemoveAt(0);
DataMapping(columnData, scheduleData);
}
}
}
public static void DataMapping(List<string> keyData, List<List<string>>valueData)
{
List<Dictionary<string, string>> items= new List<Dictionary<string, string>>();
string prompt = "Key/Value";
prompt += Environment.NewLine;
foreach (List<string> list in valueData)
{
for (int key=0, value =0 ; key< keyData.Count && value< list.Count; key++,value++)
{
Dictionary<string, string> newItem = new Dictionary<string, string>();
string k = keyData[key];
string v = list[value];
newItem.Add(k, v);
items.Add(newItem);
}
}
foreach (Dictionary<string, string> item in items)
{
foreach (KeyValuePair<string, string> kvp in item)
{
prompt += "Key: " + kvp.Key + ",Value: " + kvp.Value;
prompt += Environment.NewLine;
}
}
Autodesk.Revit.UI.TaskDialog.Show("Revit", prompt);
}

Need to insert row into Google spreadhseet at specific location

I am using python to update a google spreadsheet. The google python library for spreadsheets allows you to insert a row to a spreasheet using the InsertRow api.
Here is an example:
gd_client.InsertRow(myDict, spreadSheetKey, workSheetKey)
myDictionary is the dictionary you are inserting, spreadsheetKey is the key for your spreadsheet and worksheetKey is the key for your worksheet.
However, I'd like to insert into a specific row in the middle of my spreadsheet. This API only inserts at the end.
Anyone know, if there is any way to do this?
Thanks.

An easy way is to have two sheets in the same spreadsheet. One is the raw data as it is inserted from the outside, and the second is the calculated one (sorting, functions...) that is based on that raw data.
For example if you want the data to be sorted by the second column, you can put in the second sheet:
=EXPAND(SORT(Sheet1!A:D,2,1))
This separation is also good as if you want to update the data from the outside. It will conflict with operations that you might have on your data, as adding calculated columns to the data rows.

How to * insert a row * into the middle of a google spreadsheet via the API/OAuth ..... the hard way. (I assume there is no easy way, because there is a feature request to add this to the API).
Pattern cellRefPattern = Pattern.compile("R(\\[?)([-0-9]+)\\]?C(\\[?)([-0-9]*)\\]?");
worksheet.setRowCount(worksheet.getRowCount()+1);
worksheet.update();
CellFeed batchRequest = new CellFeed();
for (AppCell cellAddr : cellAddresses.values()) {
// create a copy of the cell to replace
CellEntry batchEntry = new CellEntry(cellAddr.row, cellAddr.col, cellAddr.reference);
String updateReference = cellAddr.inputValue;
if(updateReference.startsWith("=")) {
String removeReferenceBug = updateReference.replace( (CharSequence) "C:R", (CharSequence) "C[0]:R");
Matcher referenceMatcher = cellRefPattern.matcher(removeReferenceBug);
StringBuffer restultBuffer = new StringBuffer();
while (referenceMatcher.find()) {
try {
if(referenceMatcher.group(1).equals("[")) {
int rowOffset = Integer.parseInt(referenceMatcher.group(2));
int topRowOfSpan;
int bottomRowOfSpan;
int incSize = 1;
if(rowOffset > 0) {
topRowOfSpan = cellAddr.row;
bottomRowOfSpan = cellAddr.row + rowOffset;
} else {
topRowOfSpan = cellAddr.row + rowOffset;
bottomRowOfSpan = cellAddr.row ;
incSize = -1;
}
//System.out.println("move down: reference:"+cellAddr.reference+" topRowOfSpan:"+topRowOfSpan+
// " insertLocationRow:"+insertLocationRow+" bottomRowOfSpan:"+bottomRowOfSpan);
if(topRowOfSpan <= insertLocationRow && bottomRowOfSpan > insertLocationRow) rowOffset += incSize;
if(referenceMatcher.group(3).equals("[")) {
referenceMatcher.appendReplacement(restultBuffer, "R["+rowOffset+"]C["+referenceMatcher.group(4)+"]");
} else {
int colOffset = 0;
String col = referenceMatcher.group(4);
if(col != null && "".equals(col) == false) {
colOffset = Integer.parseInt(col) - cellAddr.col;
}
referenceMatcher.appendReplacement(restultBuffer, "R["+rowOffset+"]C["+colOffset+"]");
}
} else {
int absoluteRow = Integer.parseInt(referenceMatcher.group(2));
if(absoluteRow >= insertLocationRow ) absoluteRow ++;
if(referenceMatcher.group(3).equals("[")) {
referenceMatcher.appendReplacement(restultBuffer, "R"+absoluteRow+"C["+referenceMatcher.group(4)+"]");
} else {
referenceMatcher.appendReplacement(restultBuffer, "R"+absoluteRow+"C"+referenceMatcher.group(4));
}
}
} catch(NumberFormatException nfe) {}
}
referenceMatcher.appendTail(restultBuffer);
updateReference = restultBuffer.toString();
}
batchEntry.setId(String.format("%s/%s", worksheet.getCellFeedUrl().toString(), cellAddr.reference));
batchEntry.changeInputValueLocal(updateReference);
BatchUtils.setBatchId(batchEntry, cellAddr.reference);
BatchUtils.setBatchOperationType(batchEntry, BatchOperationType.UPDATE);
// add the copy to the batch list
batchRequest.getEntries().add(batchEntry);
}
// Submit the update
Link batchLink = cellFeed.getLink(Link.Rel.FEED_BATCH, Link.Type.ATOM);
service.setHeader("If-Match", "*");
CellFeed batchResponse = service.batch(new URL(batchLink.getHref()), batchRequest);
service.setHeader("If-Match", null);
Modified version of https://developers.google.com/google-apps/spreadsheets/#updating_multiple_cells_with_a_batch_request
Issue: Does not check for strings in the formulas
AppCell contains: row, col & reference