gitlab has this functionality that you can use pipelines that will execute code whenever you push code to your project.
this is done through their .gitlab-ci.yml file format
i am trying to somehow make the pipeline to merge all branches with prefix "ready/"
i have written a python program to do it locally, but it wont execute on the gitlab docker remote machine. this is due to the fact that it only lists "* and master" as branches with "git branch -a".
i have tried to checkout to master but that dosent work.
is this even possible on the gitlab pipeline? how would i go forward?
There are a couple of ways to achieve this depending what credentials you want to use, what you prefer, and what is better suited to your use case.
Use SSH in CI/CD (with SSH keys) to use your standard git commands to pull, do whatever, then push to the repo as part of a pipeline job.
Use the merge requests API which requires a personal access token. The API allows you to create, accept, and merge a merge request.
If you have a lot of branches, then you may want to use the first method.
Related
I use python to develop code on my work laptop and then deploy to our server for automation purposes.
I just recently started using git and github through PyCharm in hopes of making the deployment process smoother.
My issue is that I have a config file (YAML) that uses different parameters respective to my build environment (laptop) and production (server). For example, the file path changes.
Is there a git practice that I could implement that when either pushing from my laptop or pulling from the server it will excluded changes to specific parts of a file?
I use .gitignore for files such as pyvenv.cfg but is there a way to do this within a file?
Another approach I thought of would be to utilize different branches for local and remote specific parameters...
For Example:
Local branch would contain local parameters and production branch would contain production parameters. In this case I would push 1st from my local to the local branch. Next I would make the necessary changes to the parameters for production, in my situation it is much easier to work on my laptop than through the server, then push to the production branch. However, I have a feeling this is against good practice or simply changes the use of branches.
Thank you.
Config files are also a common place to store credentials (eg : a login/pwd for the database, an API key for a web service ...) and it is generally a good idea to not store those in the repository.
A common practice is to store template files in the repo (eg : config.yml.sample), to not store the actual config file along with the code (even add it in .gitignore, if it is in a versioned directory), and add steps at deployment time to either set up the initial config file or update the existing one - those steps can be manual, or scripted. You can backup and version the config separately, if needed.
Another possibility is to take the elements that should be adapted from somewhere else (the environment for instance), and have some user: $APP_DB_USER entries in your config file. You should provision these entries on both your servers - eg : have an env.txt file on your local machine and a different one on your prod server.
If we maintain our code/scripts in github repository account, is there any way to copy these scripts from Github repository and execute on some other cluster ( which can be Hadoop or Spark).
Does airflow provides any operator to connect to Github for fetching such files ?
Maintaining scripts in Github will provide more flexibility as every change in the code will be reflected and used directly from there.
Any idea on this scenario will really help.
You can use GitPython as part of a PythonOperator task to run the pull as per a specified schedule.
import git
g = git.cmd.Git( git_dir )
g.pull()
Don't forget to make sure that you have added the relevant keys so that the airflow workers have permission to pull the data.
I have a Django 1.6 project (stored in a Bitbucket Git repo) that I wish to host on a VPS.
The idea is that when someone purchases a copy of the software I have written, I can type in a few simple commands that will take a designated copy of the code from Git, create a new instance of the project with its own subdomain (e.g. <customer_name>.example.com), and create a new Postgres database (on the same server).
I should hopefully be able to create and remove these 'instances' easily.
What's the best way of doing this?
I've looked into writing scripts using some sort of combination of Supervisor/Gnunicorn/Nginx/Fabric etc. Other options could be something more serious like using Docker or Vagrant. I've also looked into various PaaS options too.
Thanks in advance.
(EDIT: I have looked at the following services/things: Dokku (can't use Heroku due to data constraints), Vagrant (inc Puppet), Docker, Fabfile, Deis, Cherokee, Flynn (under dev))
If I was doing it (and I did a similar thing with a PHP application I inherited), I'd have a fabric command that allows me to provision a new instance.
This could be broken up into the requisite steps (check-out code, create database, syncdb/migrate, create DNS entry, start web server).
I'd probably do something sane like use the DNS entry as the database name: or at least use a reversible function to do that.
You could then string these together to easily create a new instance.
You will also need a way to tell the newly created instance which database and domain name they needed to use. You could have the provisioning script write some data to a file in the checked out repository that is then used by Django in it's initialisation phase.
I have many application servers running on the same host. Every application server is installed in a different directory.
How should I tackle deployments on the servers, using Fabric? I would like to be able to perform deployments on each server separately, and on subsets of servers. Clearly the env.hosts parameter has no use here, since all servers are on the same host. Same goes for the env.roledefs parameter. These come in handy when every server is installed on a different host.
How should I deal with grouping of the servers, and setting separate environment parameters for each one of them which the fab tool can read and apply.
It's just python so do what you need to do to keep them seperate. You can define the dir differences in a dictionary or some yaml file that's read into the script. There isn't anything made in fabric to make you do it one way nor provide any specific way to do this.
But essentially just keep in mind that it's not a DSL, it's a full python file, and you'll stumble onto what works best for you and your environment.
How would I go about setting up one github user and ssh key and then replicating that to several other laptops so they can all use the same account? It would be optimal if I could copy a configuration file so I wouldn't have to apply it one laptop at a time - I could apply it through server administration.
This isn't a typical github setup so don't worry about this being the correct way to set it up.
Setup the project the way github describes.
Create your ssh keys.
Tar or Zip everything up.
Distribute and Untar/Unzip.
Done.