I wrote 2-3 Plugins for pyload.
Sometimes they change and i let users know over forum that theres a new version.
To avoid that i'd like to give my scripts an auto selfupdate function.
https://github.com/Gutz-Pilz/pyLoad-stuff/blob/master/FileBot.py
Something like that easy to setup ?
Or someone can point me in a direction ?
Thanks in advance!
It is possible, with some caveats. But it can easily become very complicated. Before you know it, your auto-update "feature" will be bigger than the original code!
First you need to have an URL that always contains the latest version. Since you are using github, using raw.githubusercontent might do very well.
Have your code download the latest version from that URL (e.g. using requests), and compare the version with that in the current code. For this purpose I would recommend a simple integer version number, so you don't need any complicated parsing logic.
However, you might want to consider only running that check once per day, or once per week. If you do it every time your file is run, the server might get hammered! So now you have to save a file with the date when the check was last done, and read that to see if it is time to run the check again. This file will need to be saved in a location that you can access on every platform your code is liable to run on. That in itself can be a challenge.
If it is just a single python file, which is installed as the user that is running it, updating is relatively easy. But if the original was installed as root in the global Python directory and your script is running as a nonprivileged user it will be difficult. Especially if it is running as a plugin and cannot ask the user for (temporary) root credentials to install the file.
And what are you going to do if a newer version has more dependencies outside the standard library?
Last but not least, as a sysadmin I don't really like auto-updating software. Especially for critical system infrstructure I like to be able to estimate the consequences before an update.
Related
I've started python a few months ago, for school projects and I was pretty surprised at the file edition power it had without asking for any permissions. My question is, how far can python go? Can it delete System files? Can it delete normal files? I also saw a video that I didn't click that said python malware was really easy to make... So I am just really curious to how far it goes, mostly because my IDE didn't even need admin permissions to be installed...
P-S: not sure if this is appropriate to stack overflow, kinda new here :)
Python can go just as far as the user running Python can. If you have the right to delete a file, then if you start Python and run a script or issue a command that deletes a file, Python will be allowed to. Python will be acting under your user account.
Having said that, it's not always obvious what user is running Python exactly. Normally, if you start Python yourself and pass it a script, or run some commands interactively, it'll be you.
But if, for example, you start Python from a scheduled task, the user running Python won't be you by default, but some sort of system account which may have more restricted rights.
On the other hand, if you're not allowed to do something (say access a folder that has restricted access for other users only), you can still write a Python script that tries to perform the actions. If you were to run that script, it would fail, but if one of those other users logs on and runs the same script, it will succeed.
Python is restricted in that it doesn't contain libraries for every function imaginable (although you could probably write them yourself in Python, given enough time). To get around that, you typically install third party packages, created by people that have already written that code, further extending what Python can do. But no package should be able to get around the restrictions the OS imposes on the user running Python.
To get a sense of how complete Python is, even without third party packages, have a look at the Python Standard Library. All those things can be done with standard Python, provided the user running it is allowed to.
I have the following issue:
I have some software that installs a particular version of IronPython to GAC. But I need to install a newer version of IronPython without affecting GAC. Hence the need for somehow using pyenv on windows with IronPython.
I am not from a programming background, more of a brick and mortar background, so please bear with me here.
[pyenv-win][1] doesn't support IronPython yet, and given my background, I have no idea how to modify a GIT repository and then install it (I'm trying to learn all that, but first I need to set this environment up, so that I don't mess things up, its a vicious cycle :P).
I downloaded a copy of the code and I was looking into how it is addressing different python versions.
It seems in [this file][2]
there are variables:
mirror, mirrorEnvPath, listEnv
that point to location of the exe that pyenv-win is using to install and maintain python version
So I somehow need to add an iron python mirror location and another array to the list that reads something like this
ironmirror = "https://github.com/IronLanguages/ironpython2/releases/download"
and add a line to listEnv
Array("ipy-2.7.9", ironmirror&"/ipy-2.7.9/", "IronPython-2.7.9.msi", "x64")_
That's how far I can get. If someone could help me put all this together, that would be nice. I have no idea how to run this modified code from my local hard drive. I would also like to somehow add this functionality to the package on GitHub as well so that others can use the same. Also, I am not sure if I am allowed to look into the code that others have shared and modify it. This is a new world for me. Apologies if that's the case.
Any help is appreciated.
[1]: https://github.com/pyenv-win/pyenv-win
[2]: https://github.com/pyenv-win/pyenv-win/blob/master/pyenv-win/libexec/pyenv-install.vbs
Up to now we keep the version number of our python source code in setup.py.
This version gets increased after every successful ci run.
This means the version of central libraries get increased several times per day.
Since the version number is stored in a file in the git repo, every increase of the version number is a new commit.
This means roughly 50% of all commits are not made by humans, but by CI.
I have got the feeling, that we are on the wrong track. Maybe it is no good solution to keep the version number in ci.
How could we avoid the "useless" CI commits which just increase the version number?
How to avoid keeping version number in source code?
Update
We live without manual release since several years. We do not have a versioning scheme like MAJOR.MINOR. And we have not missed this in the past. I know that this does not work for all environments. But it works for my current environment.
We have a version number which looks like this: YEAR.MONTH.X
This means every commit which passes CI is a new release.
After reading the answers I realize: I need to asks myself: Do I have a version number at all? I think no. I have a build number. More is not needed in this context.
(thank you for the up-votes. Before asking this question I was sure that this question will get closed because people will think it is "unclear" or "too broad")
It is a common practice to keep a version number in the source code, there is nothing wrong in that.
You need to separate CI procedures to regular builds, release publishing and release deployment.
Regular builds: run daily or even after each commit, can include static code analysis and automatic tests, check if the code can be built at all. Regular builds should not change the version number.
Release publishing: can only be triggered by explicit manual action by release manager.
The trigger action could be tagging a commit with a new version number, new merge into the release branch, or just a commit that changes version number kept in a special file (e.g. pom.xml). Refer to git flow for example.
Release publishing assigns a new version number (either automatically or manually), commits it into the source code if necessary, builds a binary package with a new version and uploads it to the binary package repository (e.g. Nexus, devpi, local APT repository, Docker registry and so on).
Release deployment: another manually triggered action that takes a ready binary package from a package repository and installs it to the target environment (dev, QA / UAT / staging, part of production for canary deployments or to the whole production environment).
Premises:
I assume these are the premises under which the solution is discussed.
Currently version number is kept in a git-tracked source file, but you are OK to get rid of it.
No one manually manages version number, nor triggers a release procedure, which includes: (a) increase version number, (b) build from source and (c) store the built result somewhere. These are taken cared by CI, and SHOULD remain that way.
Solution:
Instead of writing to a source file and create new commit, CI simply tag the specific commit that passed CI check, then push the tag to remote repo.
The build script should read the tag of current HEAD commit, and use it as the version number for publishing a release.
Optionally, you might want to use git filter-branch to rewrite your existing git repo history, tag previous release commits for consistency, remove and stop tracking the version number source cile, then get rid of those CI commits.
I think you should use git flow. And create a master branch and a develop branch. Every time the CI checks the develop the version number remains the same. Everytime you create a release e.g. merge develop into master, you can increase the version number by CI.
Or have i missing something, but in my Opinion there is no reason that the version number is increased everytime ci runs.
So all in all you better should think about when to "release" changes to a new version!!
If the project's kept in a git repo for production use, just use whichever variant of git describe floats your boat, no need to store it in a tracked file because the result identifies the particular history, and you've got that history right there.
If the source is shipped separately, you can use git archive and the export-subst attribute to embed pretty much anything you want in the exported source.
PS : Being a new user cannot add comment.
support and expand on this answer by #VibrantVivek.
For Continuous-Integration , tagging of repository back is very important and whether you keep it in your code or simply by any other git way , after every successful CI there must be corresponding tag/version.
And if you're having CI tags/version which are not against commit , then something really wrong is at work here.
And +1 for Martin Fowler , here another link for a more detailed article (more or less by same person) https://www.thoughtworks.com/continuous-integration (recommend to read please)
On your first question :
How could we avoid the "useless" CI commits which just increase the
version number?
Please se Continuous Integration (CI) is a development practice which verifies each check-in by an automated build , allowing teams to detect problems early.
Having said that, would like to articulate here :
From practice : Every commit should build on an integration
machine
Under how to do it : The CI server monitors the repository and checks out changes when they occur
In simple words, the CI server should enhance the version only and only when there is a new commit and thus making sure every code commit is releasable.
Looks like from OP , that in your area there more (as you said) "useless" commits from CI server.
Based on your CI mechanism, I hope you should/must be able to control it , almost there are ways to handle in every tool we use. (Eg: webhooks in bitbucket, version plugin etc).
So, making sure only after a new commit we have a new version.
Now if you're thinking about those regular nightly integration builds , then read below :
Many organizations do regular builds on a timed schedule, such as every night. This is not the same thing as a continuous build and isn't enough for continuous integration. The whole point of continuous integration is to find problems as soon as you can. Nightly builds mean that bugs lie undetected for a whole day before anyone discovers them. Once they are in the system that long, it takes a long time to find and remove them.
Also you have mentioned : Every commit which passes CI is a new release, thus in a way you're already on true CI.
Despite this, if you're still unable to figure out how you can avoid "useless" commits of version number, then I would suggest to add another question with detail on how your CI mecahnism works and why it is difficult with given conditions.
I bet there must be a solution. Also have look on GithubFlowVsGitFlow.
source : Martin fowler's white paper on CI
How to avoid keeping version number in source code?
On this, would like to expand on #void answer as it is said there there:
It is a common practice to keep a version number in the source code, there is nothing wrong in that.
There are projects which have to know the exact version deployed (for some important xy reasons) in such scenarios they keep version in source and HTTP GET API to fetch from deployed code (one way of doing it) to know the version currently deployed on X server.
However it is more on the requirement, suppose for another project there is no such situation then recommended way to keep version is using commit hash / tagging each successful CI build.
You can have more details here :
Hope this helps.
I just installed git and GitHub and the reason is that it's a version control system (that's the only thing I know about it). I am starting a project (a game in Python) and it's going to stretch a long time as I am going to learn Python in deep while working on this. I want to be organized while working on this project and that's why I need a version control system.
So, the thing is:
1. I am the only programmer.
2. I don't want to share the code anywhere.
3. I want to work locally only on my system.
4. It's not open source (for the beginning)
So I want to know if GitHub functions according to my needs and how should I create local code repositories and manage my code?
P.S. I am sorry if it looks all messed up, but I couldn't find any better way to put forward.
Yes, using GitHub is a great idea.
Even if you are working alone, git will help you backup your code and will let you go to older versions in case your latest code stops working.
If you are a student, then you can get a plan to have private repositories for a couple of years for free. If you are not a student and privacy is very important for you, then you can buy private repositories in GitHub.
Virtually all of the information that you might need will be here: http://git-scm.com/doc
Is it possible to deploy python applications such that you don't release the source code and you don't have to be sure the customer has python installed?
I'm thinking maybe there is some installation process that can run a python app from just the .pyc files and a shared library containing the interpreter or something like that?
Basically I'm keen to get the development benefits of a language like Python - high productivity etc. but can't quite see how you could deploy it professionally to a customer where you don't know how there machine is set up and you definitely can't deliver the source.
How do professional software houses developing in python do it (or maybe the answer is that they don't) ?
You protect your source code legally, not technologically. Distributing py files really isn't a big deal. The only technological solution here is not to ship your program (which is really becoming more popular these days, as software is provided over the internet rather than fully installed locally more often.)
If you don't want the user to have to have Python installed but want to run Python programs, you'll have to bundle Python. Your resistance to doing so seems quite odd to me. Java programs have to either bundle or anticipate the JVM's presence. C programs have to either bundle or anticipate libc's presence (usually the latter), etc. There's nothing hacky about using what you need.
Professional Python desktop software bundles Python, either through something like py2exe/cx_Freeze/some in-house thing that does the same thing or through embedding Python (in which case Python comes along as a library rather than an executable). The former approach is usually a lot more powerful and robust.
Yes, it is possible to make installation packages. Look for py2exe, cx_freeze and others.
No, it is not possible to keep the source code completely safe. There are always ways to decompile.
Original source code can trivially be obtained from .pyc files if someone wants to do it. Code obfuscation would make it more difficult to do something with the code.
I am surprised no one mentioned this before now, but Cython seems like a viable solution to this problem. It will take your Python code and transpile it into CPython compatible C code. You also get a small speed boost (~25% last I checked) since it will be compiled to native machine code instead of just Python byte code. You still need to be sure the user has Python installed (either by making it a pre-requisite pushed off onto the user to deal with, or bundling it as part of the installer process). Also, you do need to have at least one small part of your application in pure Python: the hook into the main function.
So you would need something basic like this:
import cython_compiled_module
if __name__ == '__main__':
cython_compiled_module.main()
But this effectively leaks no implementation details. I think using Cython should meet the criteria in the question, but it also introduces the added complexity of compiling in C, which loses some of Python's easy cross-platform nature. Whether that is worth it or not is up to you.
As others stated, even the resulting compiled C code could be decompiled with a little effort, but it is likely much more close to the type of obfuscation you were initially hoping for.
Well, it depends what you want to do. If by "not releasing the source code" you mean "the customer should not be able to access the source code in any way", well, you're fighting a losing battle. Even programs written in C can be reverse engineered, after all. If you're afraid someone will steal from you, make them sign a contract and sue them if there's trouble.
But if you mean "the customer should not care about python files, and not be able to casually access them", you can use a solution like cx_Freeze to turn your Python application into an executable.
Build a web application in python. Then the world can use it via a browser with zero install.