I want to create a python notebook in databricks that will do the following -
Connect to Azure Devops Git repo
Make couple of changes in a yaml file
Commit the changes in the master branch
Push the changes back to repo
I tried the below code to achieve step 1:
import git
repo_url = <repo url>
pat_token = <token>
dbfs_path ='<dbfs_path>'
repo = git.Repo.clone_from(repo_url, dbfs_path, branch='master')
But I got the below error:
GitCommandError: Cmd('git') failed due to: exit code(128)
cmdline: git clone --branch=master -v <repo_url> <dbfs_path>
stderr: 'Cloning into <dbfs_path>...
fatal: could not read Username for <repo_url>: No such device or address
I am not sure where to provide the username.
Related
An error I met when I use git API
from git import Repo
from github import Github
Connecting to git.xxxxx.com (port 9418) ... fatal: unable to connect to git.xxxxx.com:
git.xxxxx.com[0: 141.113.0.105]: errno=Connection refused
I can clone code from Enterprise github by command
git clone https://[name]:[token]#git.git.xxxx.com/xxx.git
But I'm confusing the way of clone in git API
def walk_githubprojects(token, organization):
client = Github(base_url='https://git.xxx.com/api/v3', login_or_token=token)
user=client.get_user().get_repos()
for repo in client.get_organization(organization).get_repos():
print(repo.name)
print(repo.git_url)
Repo.clone_from(repo.git_url, 'my_path')
I can get repo names under organization, the script should clone repos to my local dir
The solution is：
from git import Repo
from github import Github
def walk_githubprojects(token, organization):
client = Github(base_url='https://git.xxxxx.com/api/v3', login_or_token=token)
user=client.get_user().get_repos()
for repo in client.get_organization(organization).get_repos():
HTTPS_REMOTE_URL = f'https://[name]:[token]#git.xxxxx.com/{repo.full_name}'
Repo.clone_from(HTTPS_REMOTE_URL, f'/xxxx/{repo.name}')
I am currently trying to use GitLab to run a CI/CD job that runs a Python file that makes changes to a particular repository and then commits and pushes those changes to master. I also have a role of Master in the repository. It appears that all git functions run fine except for the git push, which leads to fatal: You are not currently on a branch. and with using git push origin HEAD:master --force, that leads to fatal: unable to access 'https://gitlab-ci-token:xxx#xxx/project.git/': The requested URL returned error: 403. I've been looking over solutions online, one being this one, and another being unprotecting it, and couldn't quite find what I was looking for just yet. This is also a sub-project within the GitLab repository.
Right now, this is pretty much what my .gitlab-ci.yml looks like.
before_script:
- apt-get update -y
- apt-get install git -y
- apt-get install python -y
- apt-get python-pip -y
main:
script:
- git config --global user.email "xxx#xxx"
- git config --global user.name "xxx xxx"
- git config --global push.default simple
- python main.py
My main.py file essentially has a function that creates a new file within an internal directory provided that it doesn't already exist. It has a looks similar to the following:
import os
import json
def createFile(strings):
print ">>> Pushing to repo...";
if not os.path.exists('files'):
os.system('mkdir files');
for s in strings:
title = ("files/"+str(s['title'])+".json").encode('utf-8').strip();
with open(title, 'w') as filedata:
json.dump(s, filedata, indent=4);
os.system('git add files/');
os.system('git commit -m "Added a directory with a JSON file in it..."');
os.system('git push origin HEAD:master --force');
createFile([{"title":"A"}, {"title":"B"}]);
I'm not entirely sure why this keeps happening, but I have even tried to modify the repository settings to change from protected pull and push access, but when I hit Save, it doesn't actually save. Nonetheless, this is my overall output. I would really appreciate any guidance any can offer.
Running with gitlab-runner 10.4.0 (00000000)
on cicd-shared-gitlab-runner (00000000)
Using Kubernetes namespace: cicd-shared-gitlab-runner
Using Kubernetes executor with image ubuntu:16.04 ...
Waiting for pod cicd-shared-gitlab-runner/runner-00000000-project-00000-concurrent-000000 to be running, status is Pending
Waiting for pod cicd-shared-gitlab-runner/runner-00000000-project-00000-concurrent-000000 to be running, status is Pending
Running on runner-00000000-project-00000-concurrent-000000 via cicd-shared-gitlab-runner-0000000000-00000...
Cloning repository...
Cloning into 'project'...
Checking out 00000000 as master...
Skipping Git submodules setup
$ apt-get update -y >& /dev/null
$ apt-get install git -y >& /dev/null
$ apt-get install python -y >& /dev/null
$ apt-get install python-pip -y >& /dev/null
$ git config --global user.email "xxx#xxx" >& /dev/null
$ git config --global user.name "xxx xxx" >& /dev/null
$ git config --global push.default simple >& /dev/null
$ python main.py
[detached HEAD 0000000] Added a directory with a JSON file in it...
2 files changed, 76 insertions(+)
create mode 100644 files/A.json
create mode 100644 files/B.json
remote: You are not allowed to upload code.
fatal: unable to access 'https://gitlab-ci-token:xxx#xxx/project.git/': The requested URL returned error: 403
HEAD detached from 000000
Changes not staged for commit:
modified: otherfiles/otherstuff.txt
no changes added to commit
remote: You are not allowed to upload code.
fatal: unable to access 'https://gitlab-ci-token:xxx#xxx/project.git/': The requested URL returned error: 403
>>> Pushing to repo...
Job succeeded
Here is a resource from Gitlab that describes how to make commits to the repository within the CI pipeline: https://gitlab.com/guided-explorations/gitlab-ci-yml-tips-tricks-and-hacks/commit-to-repos-during-ci/commit-to-repos-during-ci
Try configuring your gitlab-ci.yml file to push the changes rather than trying to do it from the python file.
I managed to do this via ssh on a runner by making sure the ssh key is added, and then using the full git url:
task_name:
stage: some_stage
script:
- ssh-add -K ~/.ssh/[ssh key]
- git push -o ci-skip git#gitlab.com:[path to repo].git HEAD:[branch name]
If it is the same repo that triggered the job, the url could also be written as:
git#$CI_SERVER_HOST:$CI_PROJECT_PATH.git
This method can be used to commit tags or files. You may also wish to consider using the CI CD
variable API to store cross-build persistent data if it does not have to be committed to the repo
https://docs.gitlab.com/ee/api/project_level_variables.html
https://docs.gitlab.com/ee/api/group_level_variables.html
ACCESS_TOKEN below is a variable at the repo or an upbound group level that contains a token that
can write to the target repos. Since maintainer can see these, it is best practice to
create tokens on special API users who are least privileged for just what they need to do.
write_to_another_repo:
before_script:
- git config --global user.name "${GITLAB_USER_NAME}"
- git config --global user.email "${GITLAB_USER_EMAIL}"
script:
- |
echo "This CI job demonstrates writing files and tags back to a different repository than this .gitlab-ci.yml is stored in."
OTHERREPOPATH="guided-explorations/gitlab-ci-yml-tips-tricks-and-hacks/commit-to-repos-during-ci/pushed-to-from-another-repo-ci.git"
git clone https://gitlab-ci-token:${CI_JOB_TOKEN}#$CI_SERVER_HOST/$OTHERREPOPATH
cd pushed-to-from-another-repo-ci
CURRENTDATE="$(date)"
echo "$CURRENTDATE added a line" | tee -a timelog.log
git status
git add timelog.log
# "[ci skip]" and "-o ci-skip" prevent a CI trigger loop
git commit -m "[ci skip] updated timelog.log at $CURRENTDATE"
git push -o ci-skip http://root:$ACCESS_TOKEN#$CI_SERVER_HOST/$OTHERREPOPATH HEAD:master
#Tag commit (can be used without commiting files)
git tag "v$(date +%s)"
git tag
git push --tags http://root:$ACCESS_TOKEN#$CI_SERVER_HOST/$OTHERREPOPATH HEAD:master
The requested URL returned error: 403
The HTTP 403 Forbidden client error status response code indicates that the server understood the request but refuses to authorize it.
The problem is we cannot provide a valid authentication to git and hence our request is forbidden.
Try this:Control Panel => User Accounts => Manage your credentials => Windows Credentials
It worked for me.However I'm not quite sure if it will work for you.
Maybe you can need to generate access token on profile, edit profile - then access tokens for 'read_repository' or 'write_repository'
profile => edit profile => access tokens
I cloned the application commcare-hq after installing python and django in my cpanel. here's the link: https://github.com/dimagi/commcare-hq but whenever i enter the following command
git submodule update --init --recursive
i get the following error
fatal: clone of 'git://github.com/dimagi/xml2json.git' into submodule path
'/home/hcdcnetl/myProject/commcare-hq/submodules/xml2json' failed
Failed to clone 'submodules/xml2json'. Retry scheduled
Cloning into '/home/hcdcnetl/myProject/commcare-
hq/corehq/apps/hqmedia/static/hqmedia/MediaUploader'...
fatal: unable to connect to github.com:
github.com[0: 192.30.253.113]: errno=Connection refused
github.com[1: 192.30.253.112]: errno=Connection refused
git://github.com is not an SSH URL, it is a Git-protocol URL.
Try
git config --global url."git#github.com/".insteadOf git://github.com/
Any git://github.com/ will be replaced by the SSH URL git#github.com/...
I have two repositories on github, using gitpython I'm trying to push a file from one repository to another remote repository. I've managed to do it using git but struggling with the gitpython code.
git remote add remote_to_push git#bitbucket...
git fetch remote_to_push
git checkout remote_to_push/master
git add file_to_push
git commit -m "pushing file"
git push remote_to_push HEAD:master
I've managed to create a repo object of the remote I think with the following
from git import Repo
repo = Repo('path/to/other/git/repo')
remote = repo.remotes.origin
I can't figure out how to add something to then push, if i call
remote.add("file_to_push")
Then I get errors about the create() function
TypeError: create() takes exactly 4 arguments (2 given)
Trying to follow what they have done in How to push to remote repo with GitPython with
remote.push(refspec='{}:{}'.format(local_branch, remote_branch))
I assume it should work with using master and master as the remote branches as they both must exist but it's giving me the error
stderr: 'error: src refspec master does not match any.'
Thanks
Solved it.
First created a remote of the other repo
git remote add remote_to_push git#bitbucket...
Then the gitpython code
from git import Repo
repo = Repo('path/to/other/git/repo') #create repo object of the other repository
repo.git.checkout('remote_to_push/master') #checkout to a branch linked to the other repo
file = 'path/to/file' #path to file to push
repo.git.add(file) # same as git add file
repo.git.commit(m = "commit message") # same as git commit -m "commit message"
repo.git.push('remote_to_push', 'HEAD:master') # git push remote_to_push HEAD:master
Aside from the docs, I found the following to be quite helpful if anyone's struggling with gitpython as I found it quite a pain
Python Git Module experiences?
http://sandlininc.com/?p=801
Git push via GitPython
How to push to remote repo with GitPython
I am following the instructions on Push to deploy to use Jenkins to test and deploy a Google App Engine app written in python and Flask.
the test is located in the root folder of the app in a file called tests.py
The command in the execute shell step is
nosetests tests.py
I get the following error and I am not sure how to troubleshoot this as I am fairly new to Jenkins.
Started by user User Name
Building remotely on cloud-dev-php in workspace /var/jenkins/workspace/CFC Melbourne production pipeline
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url https://source.developers.google.com/p/cfc-melbourne-website/ # timeout=10
Fetching upstream changes from https://source.developers.google.com/p/cfc-melbourne-website/
> git --version # timeout=10
using .gitcredentials to set credentials
> git config --local credential.helper store --file=/tmp/git7069316934747655973.credentials # timeout=10
> git -c core.askpass=true fetch --tags --progress https://source.developers.google.com/p/cfc-melbourne-website/ +refs/heads/*:refs/remotes/origin/*
> git config --local --remove-section credential # timeout=10
> git rev-parse refs/remotes/origin/master^{commit} # timeout=10
> git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 3a8caffa38303b3ae4741aac83e6ac807077b5be (refs/remotes/origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 3a8caffa38303b3ae4741aac83e6ac807077b5be
> git rev-list 3a8caffa38303b3ae4741aac83e6ac807077b5be # timeout=10
[CFC Melbourne production pipeline] $ /bin/sh -xe /tmp/hudson3364335209750264714.sh
+ nosetests tests.py
/tmp/hudson3364335209750264714.sh: 2: /tmp/hudson3364335209750264714.sh: nosetests: not found
Build step 'Execute shell' marked build as failure
Finished: FAILURE
This isn't really a Jenkins problem — as the build output indicates, your shell script is failing because it cannot find the nosetests executable:
nosetests: not found
Have you made sure that nose is installed on the cloud-dev-php Jenkins build machine?
Supposedly it should already be installed if you're using that push-to-deploy image — but as your build is running on the PHP build machine rather than the Python machine, perhaps that's not the case.
You should double-check that you've followed the instructions to ensure that your Python Jenkins job runs on a Python build machine.
If it is installed, perhaps it's not on the default PATH, in which case you can change the usage of nosetests to /usr/local/bin/nosetests (or whatever the path is).