We Keep Coding

How can I check poetry lockfile freshness without importing poetry

We've been using pipenv for dependency management for a while, and using micropipenv's protected functionality to check lock freshness - the idea here being that micropipenv is lightweight, so this is a cheap and cheerful way of ensuring that our dependencies haven't drifted during CI or during a docker build.
Alas, micropipenv has no such feature for poetry (it skips the hash check completely), and I am therefore left to "reverse-engineer" the feature on my own. Ostensibly this should be super easy - I've assembled the code posted later from what I traced through the poetry and poetry-core repos (Locker, Factory, core.Factory, and PyProjectTOML, primarily). This absolutely does not do the trick, and I'm at a loss as to why.
_relevant_keys = ["dependencies", "group", "source", "extras"]
def _get_content_hash(pyproject):
content = pyproject["tool"]["poetry"]
print(content)
relevant_content = {}
for key in _relevant_keys:
relevant_content[key] = content.get(key)
print(json.dumps(relevant_content, sort_keys=True).encode())
content_hash = sha256(
json.dumps(relevant_content, sort_keys=True).encode()
).hexdigest()
print(f"Calculated: {content_hash}")
return content_hash
def is_fresh(lockfile, pyproject):
metadata = lockfile.get("metadata", {})
print(f"From file: {lockfile['metadata']['content-hash']}")
if "content-hash" in metadata:
return _get_content_hash(pyproject) == lockfile["metadata"]["content-hash"]
return False
Would love to figure out what exactly the heck I'm missing here - i'm guessing that the poetry locker _local_config gets changed at some point and I've failed to notice it.
References:
Locker: https://github.com/python-poetry/poetry/blob/a1a5bce96d85bdc0fdc60b8abf644615647f969e/poetry/packages/locker.py#L454
core.Factory: https://github.com/python-poetry/poetry-core/blob/afaa6903f654b695d9411fb548ad10630287c19f/poetry/core/factory.py#L24

Naturally, this ended up being a PEBKAC error. I was using the hash generation function from the master branch but using an earlier version of poetry on the command line. Once I used the function from the correct code version, everything was hunky dory.
I think this functionality actually exists in micropipenv now anyways lol

Git merge fails unexpected

I've implemented automated merging script in Python. It should merge branches automatically and create pull requests if there are some merging conflicts.
Part of my script looks like:
from subprocess import check_call, CalledProcessError
# E.g. destination_branch = "master", source_branch = "release"
try:
check_call(['git', 'checkout', '%s' % destination_branch])
check_call(['git', 'merge', '--no-ff', '%s' % source_branch])
except CalledProcessError:
# Creating pull request.
It looks like everything is good, but there are some issues here.
After some automated merges I've got following errors:
error: you need to resolve your current index first
Dockerfile: needs merge. Also I'm printing status code of these two step. Status code is 1, which is not good.
As a result, I can see too many pull requests, most of them do not have any merging conflicts.
What is wrong here?
UPDATE:
Before merging something I also have a stuff for branch updating (to keep up-to-date version).
It looks like:
try:
check_call(['git', 'checkout', str(source_branch)])
# branch successfully checked out
check_call(['git', 'pull', 'origin', str(source_branch)])
except CalledProcessError:
# Logging an errors.
One important thing to add:
As guys: #torek, #MarkAdelsberger mentioned in their comments, I've tried also their solution with adding git merge --abort command after failed merging for some reason.
So, it doesn't help. It fails with another error:
check_call([GIT_CMD, 'merge', '--abort'])
File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'merge', '--abort']' returned non-zero exit status 128
So, I'm looking for some solution again....
Ideas?

An error message of the form you need to resolve your current index first means you, earlier, started a regular (non-fast-forward) merge and it failed with a merge conflict.
When git merge fails with a merge conflict, Git produces a status 1 exit. The check_call code will raise CalledProcessError if that particular failed merge occurred as part of this code, but all we know for sure is that there was such a failed merge earlier, not that it happened right here:
subprocess.CalledProcessError: Command '...' returned non-zero exit status 1
Once this happens—whether as part of your Python code, or by some other means—the repository's work-tree and index are left in this "merge failed, hand treatment required" state. If you don't know the repository's state in advance, you should find it out before you start, e.g., using git status --porcelain or git status --porcelain=v2 (see the documentation for git status; note that this web version does not show the new v2 format, which was first release in Git 2.11).
If you have put the work-tree and index into this partially merged, intermediate, requires-hand-treatment state and wish to restore it to the pre-merge state, simply run git merge --abort. Do not attempt a commit or a pull request from this half-vast1 state: the merge must be either finished, or aborted entirely.
Aside: '%s' % expr is silly, just use str(expr), or expr itself if it is already known to be a string.
1Say "half-vast" aloud, three times, fast. :-)

git merge --abort command after failed merging for some reason
Git 2.38 (Q3 2022) hould help: when "git merge"(man) finds that it cannot perform a merge, it should restore the working tree to the state before the command was initiated, but in some corner cases it didn't (before 2.38).
See commit c23fc07, commit 034195e, commit aa77ce8, commit 1369f14, commit 8f240b8, commit e4cdfe8, commit 24ba8b7, commit 11f4290 (23 Jul 2022) by Elijah Newren (newren).
(Merged by Junio C Hamano -- gitster -- in commit 966ff64, 03 Aug 2022)
merge: do not exit restore_state() prematurely
Reported-by: ZheNing Hu
Signed-off-by: Elijah Newren
Previously, if the user:
Had no local changes before starting the merge
A merge strategy makes changes to the working tree/index but returns with exit status 2
Then we'd call restore_state() to clean up the changes and either let the next merge strategy run (if there is one), or exit telling the user that no merge strategy could handle the merge.
Unfortunately, restore_state() did not clean up the changes as expected; that function was a no-op if the stash was a null, and the stash would be null if there were no local changes before starting the merge.
So, instead of "Rewinding the tree to pristine..." as the code claimed, restore_state() would leave garbage around in the index and working tree (possibly including conflicts) for either the next merge strategy or for the user after aborting the merge.
And in the case of aborting the merge, the user would be unable to run "git merge --abort"(man) to get rid of the unintended leftover conflicts, because the merge control files were not written as it was presumed that we had restored to a clean state already.
Fix the main problem by making sure that restore_state() only skips the stash application if the stash is null rather than skipping the whole function.
However, there is a secondary problem -- since merge.c forks subprocesses to do the cleanup, the in-memory index is left out-of-sync.
While there was a refresh_cache(REFRESH_QUIET) call that attempted to correct that, that function would not handle cases where the previous merge strategy added conflicted entries.
We need to drop the index and re-read it to handle such cases.
(Alternatively, we could stop forking subprocesses and instead call some appropriate function to do the work which would update the in-memory index automatically.
For now, just do the simple fix.)
Also, add a testcase checking this, one for which the octopus strategy fails on the first commit it attempts to merge, and thus which it cannot handle at all and must completely bail on (as per the "exit 2" code path of commit 98efc8f ("octopus: allow manual resolve on the last round.", 2006-01-13, Git v1.2.0 -- merge)).

Diff commit messages of two branches with gitpython

At work, we have a workflow where each branch is "named" by date. During the week, at least once, the latest branch gets pushed to production. What we require now is the summary/commit messages of the changes between the latest branch in production vs the new branch via gitpython.
What I have tried to do:
import git
g = git.Git("pathToRepo")
r = git.Repo("pathToRepo")
g.pull() # get latest
b1commits = r.git.log("branch1")
b2commits = r.git.log("branch2")
This give me all of the commit history from both branches but I can't figure out how to compare them to just get the newest commit messages.
Is this possible to do in gitPython? Or is there a better solution?

I figured it out:
import git
g = git.Git(repoPath+repoName)
g.pull()
commitMessages = g.log('%s..%s' % (oldBranch, newBranch), '--pretty=format:%ad %an - %s', '--abbrev-commit')
Reading through the Git documentation I found that I can compare two branches with this syntax B1..B2. I tried the same with gitpython and it worked, the other parameters are there for a custom format.

This solution uses GitPython
import git
def get_commit_from_range(start_commit, end_commit):
repo = git.Repo('path')
commit_range = f"{start_commit}...{end_commit}"
result = repo.iter_commits(commit_range)
for commit in result:
print(commit.message)

How do I perform a "yum update" using the Yumbase Python module?

Edit: So apparantly my install wasn't working. This pointed me to a mailing list Here where I figured out which commands I was missing. I have the answer for the update below. Now that I think about it, it does make sense. I just wish they'd put this somewhere simple on the dev pages.
yb = yum.YumBase()
yb.conf.assumeyes = True
yb.update(name='aws-cli')
yb.buildTransaction()
yb.processTransaction()
I'm trying to perform an update using yumbase when a server first boots with my kickstart script. At the moment I have a rather crude python subprocess to do "yum update" and would like to make this better.
I'm trying to hook into Yumbase, but the documentation is quite scarce. I have had a look at both the source code and documentation on this page: http://yum.baseurl.org/wiki/5MinuteExamples
I've figured out how to list all packages but not the ones that need updating using an SO answer from 2008: Given an rpm package name, query the yum database for updates
I've also figured out it's a very simple 3-line process to install a new package:
yb = yum.YumBase()
yb.conf.assumeyes = True
yb.install(name='aws-cli')
However the following doesn't work to "update" the package:
yb = yum.YumBase()
yb.conf.assumeyes = True
yb.update(name='aws-cli')
So what I need is:
1: A way to list the packages that need updating, much like "yum check-update"
2: Install the packages above using "yum update"

From what I can see in the yum code, it doesn't seem to be written to be used as a library. The code you gave is not the right way to do it, there's much else happening behind the scenes.
Basically, as of yum-3.4.3, the process looks like this:
->yummain.__main__
<trap KeyboardInterrupt>
->yummain.user_main(sys.argv[1:], exit_code=True)
<check YUM_PROF,YUM_PDB envvars, wrap the following into debugger/profiler if set>
->yummain.main(args)
<set up locale, set up logging>
-><create a YumBaseCli (child of YumBase & YumOutput)>
<incl. fill a list field with YumCommand instances of known commands>
->cli.YumBaseCli.getOptionsConfig()
<parse args into the YumBaseCli instance, includes initializing plugins>
<obtain global yum lock>
<check write permissions for current dir>
->cli.YumBaseCli.doCommands()
<select a YumCommand from the list>
->YumCommand.needTs/needTsRemove if needed
->YumCommand.doCommand(self, self.basecmd, self.extcmds)
<handle errors & set error code if any>
'Resolving Dependencies'
->cli.YumBaseCli.buildTransaction()
<check for an unfinished transaction>
<resolve deps using the info written by the YumCommand into the object>
<honor clean_requirements_on_remove, protected_packages,
protected_multilib, perform some checks>
<handle errors & set error code if any>
'Dependencies Resolved'
->cli.YumBaseCli.doTransaction()
<download, transaction check, transaction test, transaction
using the info in the object>
<handle errors & set error code if any>
'Complete!'
<release global yum lock>
sys.exit(error_code)
As you can see, the main working sequence is embedded directly into main so you can only replicate this logic in-process by running it directly:
yummain.main(<sequence of cmdline arguments>)
Which is just the same as running a separate process minus process isolation.

How to get count of unpublished commit with GitPython?

With git status I can get information about count of unpublished commits:
» git status
# On branch master
# Your branch is ahead of 'origin/master' by 2 commits.
# (use "git push" to publish your local commits)
#
nothing to commit, working directory clean
I want to get unpublished commits (or count) with GitPython. I docs I found repo.git.status(), but this is not what I want.

The command you are looking for is:
repo.iter_commits('BRANCH#{u}..BRANCH')
or if you want this as a list:
list(repo.iter_commits('BRANCH#{u}..BRANCH'))
The BRANCH#{u} syntax refers to the upstream branch of BRANCH.

Thanks to #Chronial and #Clare-Macrae for your feedback
Using gitPython ==3.1.11, I did it like that:
branch = self.repo.active_branch
unpushed_symbol = '⇡' if list(self.repo.iter_commits(f'{branch}#{{u}}..{branch}')) else constants.NOTHING
unpulled_symbol = '⇣' if list(self.repo.iter_commits(f'{branch}..{branch}#{{u}}')) else constants.NOTHING