I have a repo "A" with shared python build scripts which I today run in various "Execute shell" build steps in Jenkins. I seed this steps/scripts from job-dsl groovy code.
Using the newer Jenkins 2 Pipeline-concept in a repo "B" (where my app source code resides) what must my Jenkinsfile in this repo look like to keep it DRY and reuse my existing python build scripts?
I have studied the plugin 'workflow-cps-global-lib' and I have tried to setup "Pipeline Libraries" on my Jenkins master but since this setup groovy-oriented it does not just feel like the right way to go or I just does not get hang of the correct syntax. I cannot find any examples on this specific use case.
Basically I just want to to this in my Jenkinsfile:
Clone my source repo ('B') for my app
Make my shared python build scripts from my repo "A" available
Execute the python build scripts from various "execute shell" steps
Etcetera...
workflow-cps-global-lib is the way to go. Install it and setup in 'Manage Jenkins -> Configure System -> Global Pipeline Libraries to use your repository.
If you decided to use python scripts and not groovy, put all your python scripts in (root)/resources dir.
in your Jenkinsfile - load the script with libraryResource
script = libraryResource 'my_script.py'
and use it
sh script
(not enough reputation to add comment to accepted answer above)
Given a python script in /resources/myscript.py like this:
#!/usr/bin/env python3
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--echo")
args = parser.parse_args()
print(args.echo)
Use a Jenkins function like this:
def runPy(String scriptPath, def args) {
String script = libraryResource(scriptPath)
String argsString = args.join(' ')
sh "python3 -c '${script}' ${argsString}"
}
runPy('myscript.py', ['--echo', 'foo'])
Related
I source a dotcshrc file in my python script with :os.system(‘/bin/csh dotcshrc’) and it works,but when I want to use the command I have just put into the env by the source command,like os.system(‘ikvalidate mycase ‘),linux complaints:command not found.
But when I do it all by hand,everything go well.
Where is problem?
If you have a command in linux like ls and you want to use it in your python code do like this:
import os
ls = lambda : os.system('ls')
# This effectively turns that command into a python function.
ls() # skadoosh!
Output is :
FileManip.py Oscar
MySafety PROJECT DOCS
GooSpace Pg Admin
l1_2014 PlatformMavenRepo
l1_2015 R
l1_201617 R64
l2_2014 Resources
os.system runs each command in its own isolated environment. If you are sourcing something in an os.system call, subsequent calls will not see that because they are starting with a fresh shell environment. If you have dependencies like the above, you might be able to combine it into one call:
os.system(‘/bin/csh "dotcshrc; ikvalidate mycase"’)
Is there a NODE_ENV equivalent in Python?
I want to dynamically load JSON configurations to the python application based on the executing environment. In nodeJs, I do this by using the process.env.NODE_ENV.
For example,
I start the app like this,
NODE_ENV=production node server.js
And use the variable in the application like this,
if(process.env.NODE_ENV == "production") {
// Load the production config file here (eg: database.json)
} else {
// Load the development config file here (located in a different directory)
}
How can I achieve the same in Python? Or can I make use of python virtualenv or python setuptools to have a workaround?
For starters, you could do something like this.
import os
env = os.environ.get("PYTHON_ENV")
if(env =="production"):
//
else :
The script can be run with PYTHON_ENV="production" python myscript.py. Or you could use some library like dotenv(https://github.com/theskumar/python-dotenv).
you can try environs library in python
pip install environs
I was doing a "Hello World" exercise with Twitter's Pants build tool. After cloned the "pants" repo - source, I successfully configured pants on my local.
First, I created an nested dir in the repo as:
$ mkdir -p mark/python/project_test
Then, I created two files in that dir to specify my app and BUILD as:
$ touch mark/python/project_test/Hello_world.py
$ touch mark/python/project_test/BUILD
Hello_World.py:
print "Hello World!"
BUILD:
python_binary(name="myapp",
source="Hello_world.py"
)
It ran perfectly when i run it with ./pants like:
$ ./pants run mark/python/project_test:myapp
$ Hello World!
Then, I was trying to add dependencies by change the "Hello_world.py" as:
import utility
print "Hello World!", utility.user(), "!"
I also created the utility.py in the same dir as:
import os
def user():
return os.environ['USER']
As I add dependencies to my original app, I also modified the BUILD as:
python_library(name="app-lib",
source=globs("*py")
)
python_binary(name="myapp",
source="hello_world.py",
dependencies=[pants(':app-lib')]
)
However, when I called the ./pants with the same command, it returned error:
$ ./pants run mark/python/project_test:myapp
Exception caught: (<class 'pants.base.cmd_line_spec_parser.BadSpecError'>)
Exception message: name 'pants' is not defined
while executing BUILD file BuildFile(mark/python/project_test/BUILD,
FileSystemProjectTree(/Users/mli/workspace/source))
Loading addresses from 'mark/python/project_test' failed.
when translating spec mark/python/project_test:myapp
There are currently three files on my dir:
$ ls mark/python/project_test
$ BUILD Hello_world.py utility.py
Why my app can't load the library from utility.py and what is the right way to arrange folder tree and BUILD files?
I am sort of new to build tool and would really appreciate if somebody could provide a bit context of using pants when answering the question. Thanks!!:)
I was able to make your project run with a few small adjustments. Your issues were:
There used to be a pants() wrapper for pants shortcuts but it doesn't exist anymore. I think you have the syntax slightly wrong even if it did.
You used source and sources interchangeably when they are in fact distinct.
For number 2, it is perhaps a subtle distinction:
python_binary has one source - the entrypoint for the created binary.
python_library has sources - an arbitrary number of files to be imported into other projects.
If you change your BUILD file to match the definition below, you should have success rerunning your invocation. Good luck!
python_library(
name='app-lib',
sources=globs('*.py'),
)
python_binary(
name="myapp",
source="hello_world.py",
dependencies=[':app-lib']
)
I'm using waf as the build system for my project and I need to execute two consecutive shell commands during build process, in which output file from first command should be given as an input file at command-line to second command. According to waf book, general template for executing OS commands looks like this:
bld(rule='cp ${SRC} ${TGT}', source='input.txt', target='output.txt')
Using this template, the target directory will be automatically prepended to target file. But it's not clear how to refer to that file as an input file in later commands.
Linux OS, Python version 2.7, waf version 1.8.9
How can this be done?
Usually you just have to use the target file. Most WAF tools try to first find a file in the build directory and in the source directory. If not found it's something to build. So you can do:
rule = 'cp ${SRC} ${TGT}'
bld(rule=rule, source='input.txt', target='output.txt')
bld(rule=rule, source='output.txt', target='output2.txt')
And you will get something like:
[1/2] output.txt: input.txt -> build/output.txt
[2/2] output2.txt: build/output.txt -> build/output2.txt
WAF looks for relative paths from build and source directory.
I use nunjucks for templating the frontend in a python project. Nunjucks templates must be precompiled in production. I don't use extensions or asynchronous filters in the nunjucks templates. Rather than use grunt-task to listen for changes to my templates, I prefer to use the nunjucks-precompile command (offered via npm) to sweep the entire templates directory into templates.js.
The idea is to have the nunjucks-precompile --include ["\\.tmpl$"] path/to/templates > templates.js command execute within setup.py so I can simply piggyback our deployer scripts' regular execution.
I found a setuptools override and a distutils scripts argument might serve the right purpose, but I'm not so sure either is the simplest approach to execution.
Another approach is to use subprocess to execute the command directly within setup.py, but I've been cautioned against this (rather preemptively IMHO). I don't really deeply understand why not.
Any ideas? Affirmations? Confirmations?
Update (04/2015): - If you don't have the nunjucks-precompile command available, simply use Node Package Manager to install nunjucks like so:
$ npm install nunjucks
Pardon the quick self-answer. I hope this helps someone out there in the ether. I want to share this now that I've worked out a solution I'm satisfied with.
Here's a solution that's safe and based on Peter Lamut's write-up. Note that this does not use shell=True in the subprocess invocation. You may bypass grunt-task requirements on your python deployment system and also use this for obfuscation and JS packaging all the same.
from setuptools import setup
from setuptools.command.install import install
import subprocess
import os
class CustomInstallCommand(install):
"""Custom install setup to help run shell commands (outside shell) before installation"""
def run(self):
dir_path = os.path.dirname(os.path.realpath(__file__))
template_path = os.path.join(dir_path, 'src/path/to/templates')
templatejs_path = os.path.join(dir_path, 'src/path/to/templates.js')
templatejs = subprocess.check_output([
'nunjucks-precompile',
'--include',
'["\\.tmpl$"]',
template_path
])
f = open(templatejs_path, 'w')
f.write(templatejs)
f.close()
install.run(self)
setup(cmdclass={'install': CustomInstallCommand},
...
)
I think that the link here encapsulates what you are trying to achieve.