I was wondering if there was a way to connect to my sqlite DB in Python and push that data to the cloud in DBHub.io.
The reason for this is that I already have my DB in DBBrowser locally and now I want to push it into the cloud.
You might try the library pydbhub, published by the DBHub.io team at their section Libraries for accessing DBHub.io via API.
The library contains a section called "Upload database" at their Further examples, that points to this main.py file.
For historical researching in StackOverflow, this is how the example looks like today:
import sys
import os
import datetime
from platform import python_version
# https://github.com/willmcgugan/rich
from rich.console import Console
from rich.theme import Theme
import pydbhub.dbhub as dbhub
if __name__ == '__main__':
custom_theme = Theme({
"info": "green",
"warning": "yellow",
"error": "bold red"
})
console = Console(theme=custom_theme)
if python_version()[0:3] < '3.7':
console.print(
"[ERROR] Make sure you have Python 3.7+ installed, quitting.\n\n", style="error")
sys.exit(1)
# Create a new DBHub.io API object
db = dbhub.Dbhub(config_file=f"{os.path.join(os.path.dirname(__file__), '..', 'config.ini')}")
# Prepare any information you want to include with the upload (eg a commit message, etc)
info = dbhub.UploadInformation(
commitmsg="An example upload",
committimestamp=datetime.datetime.fromisoformat("2021-06-01 10:00:00"),
)
try:
# Read the database file
myDB = os.path.join(os.getcwd(), "examples", "upload", 'example.db')
f = open(myDB, 'rb')
except FileNotFoundError:
print(f"File {myDB} not found. Aborting")
sys.exit(1)
except OSError:
print(f"OS error occurred trying to open {myDB}")
sys.exit(1)
except Exception as err:
print(f"Unexpected error opening {myDB} is", repr(err))
sys.exit(1)
else:
with f:
# Upload the database
res, err = db.Upload(db_name='somedb.sqlite', info=info, db_bytes=f)
if err is not None:
console.print(f"[ERROR] {err}", style="error")
sys.exit(1)
console.print(f"Database uploaded, commit: {res['commit']}", style="info")
HTH
Related
I am building a Python app using FBS, but part of it relies on an SQLite3 database. I have code to create this database if it doesn't find this, using a try-catch block.
When I try to run it after compiling, it not only can not find the preexisting SQLite3 file, but also won't create it. It does not display any error messages.
I have tried creating the file if it doesn't exist using this code:
try:
self.connection = sqlite3.connect(path)
self.cursor = self.connection.cursor()
except:
if not os.path.exists(path):
# Try and make the .config directory
try:
os.makedirs(".config")
except OSError as e:
if e.errno != errno.EEXIST:
raise
# Create the datastore, and close it
f = open(path, "w+")
f.close()
# And try connect to database again
return self.__connect(path)
else:
print(f"No database exists, and could not create one.\nPlease create file in app directory called: {path}\nThen restart application.")
raise
The code works find in dev, but as soon as I compile it to a Mac app, it refuses to find or create the database.
Fixed. If anyone has a similar issue please use the builtin appctxt.get_resource(file_path) method.
I have the following line code in a Python script:
sql_create_table(total_stores, "total_stores")
This is a function I created to upload tables to an Oracle database. I want to do something like this in order to record what tables were not created because the line failed to run:
Try:
sql_create_table(total_stores, "total_stores")
except:
print in a log.txt "table x could not be created in the database"
Any suggestions?
Thanks in advance!
There is the python logging module, which has a good tutorial, which even includes how to log to a file.
Very basic example:
import logging
logging.basicConfig(filename="program.log", level=logging.INFO)
…
try:
sql_create_table(total_stores, "total_stores")
except:
logging.warning("table x could not be created in the database")
You can write the log to a txt file by doing the following:
Try:
sql_create_table(total_stores, "total_stores")
except:
with open('log.txt', 'a') as log:
log.write("table x could not be created in the database")
Note, that by using 'a', we are appending to the txt file and won't be overwriting old logs.
Below are issues I am facing
I am not able to install dataextract package properly using below command and hence facing issues while importing the package.
"python -m pip install dada extract"
Also facing issues in the below command in tableau_rest_api package
TableauRestApi(server, username, password)
Can any one tell me alternative way to perform same task or help me correct the existing code
#######convert .csv file to .tde file#########################
import os,csv,datetime
import dataextract as tde
# INSERT EXCEPT CONDITION
# EXTRACT FILE AND OPEN .CSV
try:
tdefile = tde.Extract('trial1.tde')
except:
os.remove('trial1.tde')``
tdefile = tde.Extract('trial1.tde')
csvReader= csv.reader(open('test1.csv','rb'),delimiter=',',quotechar='"')
# CREATE THE TABLE DEF
tableDef= tde.TableDefinition()
tableDef.addColumn('Country',tde.Type.CHAR_STRING)
tableDef.addColumn('sales',tde.Type.INTEGER)
tableDef.addColumn('units',tde.Type.INTEGER)
# CREATE TABLE
table=tdefile.addTable('C:\Users\Mahita.Gm\Desktop\Extract',tableDef)
# DATA TO TDE
newrow= tde.Row(tableDef)
csvReader.next()
for line in csvReader:
newrow.setCharString(0,str(line[0]))
newrow.setInteger(1,int(line[1]))
newrow.setInteger(2,int(line[2]))
table.insert(newrow)
# closing tde
tdefile.close()
#############code to publish tableau extract in to server #################
import tableau_rest_api
from tableau_rest_api.tableau_rest_api import * import urllib2
username = '<user name>'
password = '<password>'
server = '<server address>'
logger = Logger('publish.log')
test = TableauRestApi(server, username, password)
test.enable_logging(logger)
test.signin()
test_ds_proj_luid = test.query_project_luid_by_name('default')
#test_ds_wb_luid = test.query_workbook_luid_by_name('test_1_c_CF')
#test_ds_p_luid = test.query_datasource_luid_by_name_in_project('trial1', 'test_1_c_CF')
# Publishing up my first data source to Test, from disk
new_ds_luid = test.publish_datasource('trial1.tde', 'trial1', test_ds_proj_luid, overwrite=True)
I am running into a problem when running the follow python script on the server looking for commit information for the push making sure it follows a particular syntax, I am unable to get input from the user which is why the username and password are hard coded. I am now also unable to get the list of commit message that occurred before this particular push.
#!/usr/bin/python
import SOAPpy
import getpass
import datetime
import sys
import re
import logging
import os
def login(x,y):
try:
auth = soap.login(x, y)
return auth
except:
sys.exit( "Invalid username or password")
def getIssue(auth,issue):
try:
issue = soap.getIssue(auth, issue)
except:
sys.exit("No issue of that type found : Make sure all PRs are vaild jira PRs")
def git_get_commit_msg(commit_id):
return get_shell_cmd_output("git rev-list --pretty --max-count=1 " + commit_id)
def git_get_last_commit_id():
return get_shell_cmd_output("git log --pretty=format:%H -1")
def getCommitText():
commit_msg_filename = sys.argv[1]
try:
commit_msg_text = open(commit_msg_filename).read()
return commit_msg_text
except:
sys.exit("Could not read commit message")
def git_get_array_of_commit_ids(start_id, end_id):
output = get_shell_cmd_output("git rev-list " + start_id + ".." + end_id)
if output == "":
return None
commit_id_array = string.split(output, '\n')
return commit_id_array
def get_shell_cmd_output(cmd):
try:
proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
return proc.stdout.read().rstrip('\n')
except KeyboardInterrupt:
logging.info("... interrupted")
except Exception, e:
logging.error("Failed trying to execute '%s'", cmd)
def findpattern(commit_msg):
pattern = re.compile("\w\w*-\d\d*")
group = pattern.findall(commit_msg)
print group
found = len(group)
found =0
issues = 0
for match in group:
auth = soap.login(jirauser,passwd)
getIssue(auth,match)
issues = issues + 1
found+=1
if found ==0:
sys.exit("No issue patterns found.")
print "Retrieved issues: " + str(issues)
def update():
print sys.argv[2]
print sys.argv[3]
old_commit_id = sys.argv[2]
new_commit_id = sys.argv[3]
commit_id_array = git_get_array_of_commit_ids(old_commit_id, new_commit_id)
for commit_id in commit_id_array:
commit_text = git_get_commit_msg(commit_id)
findpattern(commit_text)
soap = SOAPpy.WSDL.Proxy('some url')
# this line if for repointing the input from dev/null
#sys.stdin = open('/dev/tty', 'r') # this fails horribly.
#ask user for input
#jirauser = raw_inp
#("Username for jira: ")
jirauser = "username"
passwd = "987654321"
#passwd = getpass.getpass("Password for %s: " % jirauser)
login(jirauser,passwd)
#commit_msg = getCommitText()
#findpattern(commit_msg)
update()
The intended goal of this code is to check the commits made locally, and to parse through them for the intended pattern, as well as checking the in jira if that PR exists. it is a server side hook that get activated on a push to the repository.
Any tips on writing python hooks would be appreciated. Please and thank you.
I suggest that you have a look at gitorious (http://gitorious.org/gitorious).
They use ssh to handle authentication and rights management (getting the username given by ssh).
They also have some hooks on git repositories. I guess it could help to see how they are processing git hooks using ruby.
By the time your update hook fires, the server has the new commits: the question is whether your hook will allow the ref in question to move. What information from the local (sending) repository do you want?
For the credentials issue, funnel everyone through a single user. For example, GitHub does it with the git user, which is why their SSH URLs begin with git#github.com:.... Then in ~git/.ssh/authorized_keys, associate a username with each key. Note that the following should be on a single line but is wrapped for presentation purposes.
no-agent-forwarding,no-port-forwarding,no-pty,no-X11-forwarding,
command="env myuser=gbgcoll /usr/bin/git-shell -c \"${SSH_ORIGINAL_COMMAND:-}\""
ssh-rsa AAAAB...
Now to see who's trying to do the update, your hook examines the $myuser environment variable.
This doesn't give you each user's Jira credentials. To solve that issue, create a dummy Jira account that has read-only access to everything, and hardcode that Jira account's credentials in your hook. This allows you to verify that a given PR exists.
I have the following PyObjC script:
from Foundation import NSObject
import QTKit
error = None
capture_session = QTKit.QTCaptureSession.alloc().init()
print 'capture_session', capture_session
device = QTKit.QTCaptureDevice.defaultInputDeviceWithMediaType_(QTKit.QTMediaTypeVideo)
print 'device', device, type(device)
success = device.open_(error)
print 'device open success', success, error
if not success:
raise Exception(error)
capture_device_input = QTKit.QTCaptureDeviceInput.alloc().initWithDevice_(device)
print 'capture_device_input', capture_device_input, capture_device_input.device()
success = capture_session.addInput_error_(capture_device_input, error)
print 'session add input success', success, error
if not success:
raise Exception(error)
capture_decompressed_video_output = QTKit.QTCaptureDecompressedVideoOutput.alloc().init()
print 'capture_decompressed_video_output', capture_decompressed_video_output
class Delegate(NSObject):
def captureOutput_didOutputVideoFrame_withSampleBuffer_fromConnection_(self, captureOutput, videoFrame, sampleBuffer, connection):
print videoFrame, sampleBuffer, connection
delegate = Delegate.alloc().init()
print 'delegate', delegate
capture_decompressed_video_output.setDelegate_(delegate)
print 'output delegate:', capture_decompressed_video_output.delegate()
success = capture_session.addOutput_error_(capture_decompressed_video_output, error)
print 'capture session add output success', success, error
if not success:
raise Exception(error)
print 'about to run session', capture_session, 'with inputs', capture_session.inputs(), 'and outputs', capture_session.outputs()
capture_session.startRunning()
print 'capture session is running?', capture_session.isRunning()
import time
time.sleep(10)
The program reports no errors, but iSight's green light is never activated and the delegate's frame capture callback is never called. Here's the output I get:
$ python prueba.py
capture_session <QTCaptureSession: 0x1006c16f0>
device Built-in iSight <objective-c class QTCaptureDALDevice at 0x7fff70366aa8>
device open success (True, None) None
capture_device_input <QTCaptureDeviceInput: 0x1002ae010> Built-in iSight
session add input success (True, None) None
capture_decompressed_video_output <QTCaptureDecompressedVideoOutput: 0x104239f10>
delegate <Delegate: 0x10423af50>
output delegate: <Delegate: 0x10423af50>
capture session add output success (True, None) None
about to run session <QTCaptureSession: 0x1006c16f0> with inputs (
"<QTCaptureDeviceInput: 0x1002ae010>"
) and outputs (
"<QTCaptureDecompressedVideoOutput: 0x104239f10>"
)
capture session is running? True
PS: Please don't answer I should try PySight, I have but it won't work because Xcode can't compile CocoaSequenceGrabber in 64bit.
Your problem here is that you don't have an event loop. If you want to do this as a standalone script, you'll have to figure out how to create one. The PyObjC XCode templates automatically set that up for you with:
from PyObjCTools import AppHelper
AppHelper.runEventLoop()
Trying to insert that at the top of your script, however, shows that something inside AppHelper (probably NSApplicationMain) expects a plist file to extract the main class from. You can get that by creating a setup.py file and using py2app, something like this example from a PyObjc talk:
from distutils.core import setup
import py2app
plist = dict(
NSPrincipalClass='SillyBalls',
)
setup(
plugin=['SillyBalls.py'],
data_files=['English.lproj'],
options=dict(py2app=dict(
extension='.saver',
plist=plist,
)),
)
You should give a try to motmot's camiface library from Andrew Straw. It also works with firewire cameras, but it works also with the isight, which is what you are looking for.
From the tutorial:
import motmot.cam_iface.cam_iface_ctypes as cam_iface
import numpy as np
mode_num = 0
device_num = 0
num_buffers = 32
cam = cam_iface.Camera(device_num,num_buffers,mode_num)
cam.start_camera()
frame = np.asarray(cam.grab_next_frame_blocking())
print 'grabbed frame with shape %s'%(frame.shape,)