We Keep Coding

Unstable connection to remote server in Docker

I have created a docker container to run my python program inside.
My program requires retrieving the known_host under my .ssh folder:
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.load_host_keys(os.path.expanduser(os.path.join("~", ".ssh", "known_hosts")))
ssh.connect(server, username=username, password=password)
I have mounted it into the docker container using:
docker run --name test_cntr --rm \
-v $SCRIPT_DIR:/home/ \
-v $DATA_DIR:/home/data \
-v $HOME/.ssh/known_hosts:/root/.ssh/known_hosts \
-e PYTHONPATH=/home/sciprt_dir:/home/sciprt_dir/lib \
-e INDEX=0 \
dummy_image python /home/run.py
Found that my program can successfully get the known_hosts file sometimes, but sometimes not, below error is shown:
Exception is [Errno -2] Name or service not known
I didn't re-run the container during the run.py execution. Suppose the known_hosts mounted to the container at the beginning and run.py should be able to use it throughout whole running.

At the end I found that, one of the servers using for this program, did not register on the domain server, so that sometimes my program works when using server that is registered, and sometimes it does not work when the server is not registered.. Thanks all for help!

How to solve the "remote: You are not allowed to upload code." error on GitLab CI/CD job?

I am currently trying to use GitLab to run a CI/CD job that runs a Python file that makes changes to a particular repository and then commits and pushes those changes to master. I also have a role of Master in the repository. It appears that all git functions run fine except for the git push, which leads to fatal: You are not currently on a branch. and with using git push origin HEAD:master --force, that leads to fatal: unable to access 'https://gitlab-ci-token:xxx#xxx/project.git/': The requested URL returned error: 403. I've been looking over solutions online, one being this one, and another being unprotecting it, and couldn't quite find what I was looking for just yet. This is also a sub-project within the GitLab repository.
Right now, this is pretty much what my .gitlab-ci.yml looks like.
before_script:
- apt-get update -y
- apt-get install git -y
- apt-get install python -y
- apt-get python-pip -y
main:
script:
- git config --global user.email "xxx#xxx"
- git config --global user.name "xxx xxx"
- git config --global push.default simple
- python main.py
My main.py file essentially has a function that creates a new file within an internal directory provided that it doesn't already exist. It has a looks similar to the following:
import os
import json
def createFile(strings):
print ">>> Pushing to repo...";
if not os.path.exists('files'):
os.system('mkdir files');
for s in strings:
title = ("files/"+str(s['title'])+".json").encode('utf-8').strip();
with open(title, 'w') as filedata:
json.dump(s, filedata, indent=4);
os.system('git add files/');
os.system('git commit -m "Added a directory with a JSON file in it..."');
os.system('git push origin HEAD:master --force');
createFile([{"title":"A"}, {"title":"B"}]);
I'm not entirely sure why this keeps happening, but I have even tried to modify the repository settings to change from protected pull and push access, but when I hit Save, it doesn't actually save. Nonetheless, this is my overall output. I would really appreciate any guidance any can offer.
Running with gitlab-runner 10.4.0 (00000000)
on cicd-shared-gitlab-runner (00000000)
Using Kubernetes namespace: cicd-shared-gitlab-runner
Using Kubernetes executor with image ubuntu:16.04 ...
Waiting for pod cicd-shared-gitlab-runner/runner-00000000-project-00000-concurrent-000000 to be running, status is Pending
Waiting for pod cicd-shared-gitlab-runner/runner-00000000-project-00000-concurrent-000000 to be running, status is Pending
Running on runner-00000000-project-00000-concurrent-000000 via cicd-shared-gitlab-runner-0000000000-00000...
Cloning repository...
Cloning into 'project'...
Checking out 00000000 as master...
Skipping Git submodules setup
$ apt-get update -y >& /dev/null
$ apt-get install git -y >& /dev/null
$ apt-get install python -y >& /dev/null
$ apt-get install python-pip -y >& /dev/null
$ git config --global user.email "xxx#xxx" >& /dev/null
$ git config --global user.name "xxx xxx" >& /dev/null
$ git config --global push.default simple >& /dev/null
$ python main.py
[detached HEAD 0000000] Added a directory with a JSON file in it...
2 files changed, 76 insertions(+)
create mode 100644 files/A.json
create mode 100644 files/B.json
remote: You are not allowed to upload code.
fatal: unable to access 'https://gitlab-ci-token:xxx#xxx/project.git/': The requested URL returned error: 403
HEAD detached from 000000
Changes not staged for commit:
modified: otherfiles/otherstuff.txt
no changes added to commit
remote: You are not allowed to upload code.
fatal: unable to access 'https://gitlab-ci-token:xxx#xxx/project.git/': The requested URL returned error: 403
>>> Pushing to repo...
Job succeeded

Here is a resource from Gitlab that describes how to make commits to the repository within the CI pipeline: https://gitlab.com/guided-explorations/gitlab-ci-yml-tips-tricks-and-hacks/commit-to-repos-during-ci/commit-to-repos-during-ci
Try configuring your gitlab-ci.yml file to push the changes rather than trying to do it from the python file.

I managed to do this via ssh on a runner by making sure the ssh key is added, and then using the full git url:
task_name:
stage: some_stage
script:
- ssh-add -K ~/.ssh/[ssh key]
- git push -o ci-skip git#gitlab.com:[path to repo].git HEAD:[branch name]
If it is the same repo that triggered the job, the url could also be written as:
git#$CI_SERVER_HOST:$CI_PROJECT_PATH.git

This method can be used to commit tags or files. You may also wish to consider using the CI CD
variable API to store cross-build persistent data if it does not have to be committed to the repo
https://docs.gitlab.com/ee/api/project_level_variables.html
https://docs.gitlab.com/ee/api/group_level_variables.html
ACCESS_TOKEN below is a variable at the repo or an upbound group level that contains a token that
can write to the target repos. Since maintainer can see these, it is best practice to
create tokens on special API users who are least privileged for just what they need to do.
write_to_another_repo:
before_script:
- git config --global user.name "${GITLAB_USER_NAME}"
- git config --global user.email "${GITLAB_USER_EMAIL}"
script:
- |
echo "This CI job demonstrates writing files and tags back to a different repository than this .gitlab-ci.yml is stored in."
OTHERREPOPATH="guided-explorations/gitlab-ci-yml-tips-tricks-and-hacks/commit-to-repos-during-ci/pushed-to-from-another-repo-ci.git"
git clone https://gitlab-ci-token:${CI_JOB_TOKEN}#$CI_SERVER_HOST/$OTHERREPOPATH
cd pushed-to-from-another-repo-ci
CURRENTDATE="$(date)"
echo "$CURRENTDATE added a line" | tee -a timelog.log
git status
git add timelog.log
# "[ci skip]" and "-o ci-skip" prevent a CI trigger loop
git commit -m "[ci skip] updated timelog.log at $CURRENTDATE"
git push -o ci-skip http://root:$ACCESS_TOKEN#$CI_SERVER_HOST/$OTHERREPOPATH HEAD:master
#Tag commit (can be used without commiting files)
git tag "v$(date +%s)"
git tag
git push --tags http://root:$ACCESS_TOKEN#$CI_SERVER_HOST/$OTHERREPOPATH HEAD:master

The requested URL returned error: 403
The HTTP 403 Forbidden client error status response code indicates that the server understood the request but refuses to authorize it.
The problem is we cannot provide a valid authentication to git and hence our request is forbidden.
Try this:Control Panel => User Accounts => Manage your credentials => Windows Credentials
It worked for me.However I'm not quite sure if it will work for you.

Maybe you can need to generate access token on profile, edit profile - then access tokens for 'read_repository' or 'write_repository'
profile => edit profile => access tokens

Local pypi server tells me 403 Forbidden when registering packages

I am trying to deploy a pypi-server to host our internal packages. I've been pulling my hair out over the behavior of pypi inside this docker container.
Dockerfile
FROM python:3.5
RUN apt-get update
RUN pip install --upgrade pip
RUN pip install -U passlib pypiserver[cache]==1.2.0
RUN mkdir -p /src/pypi/packages
EXPOSE 8080
ADD ./htpasswd /src/pypi/htpasswd
CMD ["pypi-server", "-p 8080", "-P", "/src/pypi/htpasswd","/src/pypi/packages"]
Pretty straightforward, right? Sister to the dockerfile there is of course a file called htpasswd containing a username/password pair.
If I got through the steps defined in the docker file locally (in my environment outside docker) and then execute the command defined above, it works! I can register packages against it.
Pypi running outside Dockerfile:
python setup.py register -r local
running register
running egg_info
writing top-level names to ah_model.egg-info/top_level.txt
writing dependency_links to ah_model.egg-info/dependency_links.txt
writing ah_model.egg-info/PKG-INFO
file foobar_utils.py (for module foobar_utils) not found
reading manifest file 'ah_model.egg-info/SOURCES.txt'
writing manifest file 'ah_model.egg-info/SOURCES.txt'
running check
Registering ah_model to http://localhost:8081
Server response (200): OK
However, if I build and run the dockerfile, then try to register inside that, it does NOT work:
Pypi running inside docker:
python setup.py register -r local
running register
running egg_info
writing top-level names to ah_model.egg-info/top_level.txt
writing ah_model.egg-info/PKG-INFO
writing dependency_links to ah_model.egg-info/dependency_links.txt
file foobar_utils.py (for module foobar_utils) not found
reading manifest file 'ah_model.egg-info/SOURCES.txt'
writing manifest file 'ah_model.egg-info/SOURCES.txt'
running check
Registering ah_model to http://localhost:8080
Server response (403): Forbidden
Is there something about the way pypi works that requires special tweaking for being inside a docker container? Has anyone tried this themselves?
UPDATE
Looks like the docker container is listening on port 8080:
root#a0ec19c89a9d:/src/pypi# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 1/python3.5

Expose does not expose the ports to the host machine. To do that you need to use -p or -P or create a docker-compose.yml file with ports definitions.
See https://docs.docker.com/engine/reference/builder/#expose and https://docs.docker.com/compose/compose-file/#ports

Fabric command not working with specific ssh key

I'm working with fabric3 (https://pypi.python.org/pypi/Fabric3) , a python 3 port of fabric.
I have the following function wchich I'm running locally in win7 using git-bash:
#roles('production')
def dir():
env.key_filename = '~/.ssh/deploy'
local("git push mysite master")
run('pwd')
run('ls')
code_dir = '/home/deploy/mysite'
with cd(code_dir):
run('pwd')
run('git reset --hard master')
run('ls -la')
output:
$ fab dir
[deploy#198.x.x.x] Executing task 'dir'
[localhost] local: git push mysite master
deploy#198.x.x.x's password:
When I run the function, I get asked for the password. It seems to be ignoring the key. How can I get the function to use the prescribed key?

I added git as a user to my .ssh/config file and it now appears to work.
Host deploy
HostName 198.x.x.x
User deploy
PreferredAuthentications publickey
IdentityFile ~/.ssh/deploy
IdentitiesOnly yes
Host 198.x.x.x
HostName 198.x.x.x
User git
IdentityFile ~/.ssh/deploy

Deploying Code to Heroku (Issues?)

https://devcenter.heroku.com/articles/python
When i'm trying to deploy my code through $ git push heroku master i get an error:
Warning: Permanently added the RSA host key for IP address '50.19.85.132' to the list of known hosts.
Permission denied (publickey).
fatal: The remote end hung up unexpectedly
Next, I was able to add the RSA host key for that IP address by $ heroku keys:add but I still get this error:
Found existing public key: /Users/opalkale/.ssh/github_rsa.pub
Uploading SSH public key /Users/opalkale/.ssh/github_rsa.pub... done
(venv)Opal-Kales-MacBook-Pro:helloflask opalkale$ git push heroku master
Permission denied (publickey).
What am I doing wrong...?

Add your id_rsa.pub SSH key to your Heroku account online: https://dashboard.heroku.com/account
And make sure that the corresponding private key has been added to your identity
$ ssh-add ~/.ssh/id_rsa
E.g.
$ ssh-add ~/.ssh/github_rsa