I've implemented automated merging script in Python. It should merge branches automatically and create pull requests if there are some merging conflicts.
Part of my script looks like:
from subprocess import check_call, CalledProcessError
# E.g. destination_branch = "master", source_branch = "release"
try:
check_call(['git', 'checkout', '%s' % destination_branch])
check_call(['git', 'merge', '--no-ff', '%s' % source_branch])
except CalledProcessError:
# Creating pull request.
It looks like everything is good, but there are some issues here.
After some automated merges I've got following errors:
error: you need to resolve your current index first
Dockerfile: needs merge. Also I'm printing status code of these two step. Status code is 1, which is not good.
As a result, I can see too many pull requests, most of them do not have any merging conflicts.
What is wrong here?
UPDATE:
Before merging something I also have a stuff for branch updating (to keep up-to-date version).
It looks like:
try:
check_call(['git', 'checkout', str(source_branch)])
# branch successfully checked out
check_call(['git', 'pull', 'origin', str(source_branch)])
except CalledProcessError:
# Logging an errors.
One important thing to add:
As guys: #torek, #MarkAdelsberger mentioned in their comments, I've tried also their solution with adding git merge --abort command after failed merging for some reason.
So, it doesn't help. It fails with another error:
check_call([GIT_CMD, 'merge', '--abort'])
File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'merge', '--abort']' returned non-zero exit status 128
So, I'm looking for some solution again....
Ideas?
An error message of the form you need to resolve your current index first means you, earlier, started a regular (non-fast-forward) merge and it failed with a merge conflict.
When git merge fails with a merge conflict, Git produces a status 1 exit. The check_call code will raise CalledProcessError if that particular failed merge occurred as part of this code, but all we know for sure is that there was such a failed merge earlier, not that it happened right here:
subprocess.CalledProcessError: Command '...' returned non-zero exit status 1
Once this happens—whether as part of your Python code, or by some other means—the repository's work-tree and index are left in this "merge failed, hand treatment required" state. If you don't know the repository's state in advance, you should find it out before you start, e.g., using git status --porcelain or git status --porcelain=v2 (see the documentation for git status; note that this web version does not show the new v2 format, which was first release in Git 2.11).
If you have put the work-tree and index into this partially merged, intermediate, requires-hand-treatment state and wish to restore it to the pre-merge state, simply run git merge --abort. Do not attempt a commit or a pull request from this half-vast1 state: the merge must be either finished, or aborted entirely.
Aside: '%s' % expr is silly, just use str(expr), or expr itself if it is already known to be a string.
1Say "half-vast" aloud, three times, fast. :-)
git merge --abort command after failed merging for some reason
Git 2.38 (Q3 2022) hould help: when "git merge"(man) finds that it cannot perform a merge, it should restore the working tree to the state before the command was initiated, but in some corner cases it didn't (before 2.38).
See commit c23fc07, commit 034195e, commit aa77ce8, commit 1369f14, commit 8f240b8, commit e4cdfe8, commit 24ba8b7, commit 11f4290 (23 Jul 2022) by Elijah Newren (newren).
(Merged by Junio C Hamano -- gitster -- in commit 966ff64, 03 Aug 2022)
merge: do not exit restore_state() prematurely
Reported-by: ZheNing Hu
Signed-off-by: Elijah Newren
Previously, if the user:
Had no local changes before starting the merge
A merge strategy makes changes to the working tree/index but returns with exit status 2
Then we'd call restore_state() to clean up the changes and either let the next merge strategy run (if there is one), or exit telling the user that no merge strategy could handle the merge.
Unfortunately, restore_state() did not clean up the changes as expected; that function was a no-op if the stash was a null, and the stash would be null if there were no local changes before starting the merge.
So, instead of "Rewinding the tree to pristine..." as the code claimed, restore_state() would leave garbage around in the index and working tree (possibly including conflicts) for either the next merge strategy or for the user after aborting the merge.
And in the case of aborting the merge, the user would be unable to run "git merge --abort"(man) to get rid of the unintended leftover conflicts, because the merge control files were not written as it was presumed that we had restored to a clean state already.
Fix the main problem by making sure that restore_state() only skips the stash application if the stash is null rather than skipping the whole function.
However, there is a secondary problem -- since merge.c forks subprocesses to do the cleanup, the in-memory index is left out-of-sync.
While there was a refresh_cache(REFRESH_QUIET) call that attempted to correct that, that function would not handle cases where the previous merge strategy added conflicted entries.
We need to drop the index and re-read it to handle such cases.
(Alternatively, we could stop forking subprocesses and instead call some appropriate function to do the work which would update the in-memory index automatically.
For now, just do the simple fix.)
Also, add a testcase checking this, one for which the octopus strategy fails on the first commit it attempts to merge, and thus which it cannot handle at all and must completely bail on (as per the "exit 2" code path of commit 98efc8f ("octopus: allow manual resolve on the last round.", 2006-01-13, Git v1.2.0 -- merge)).
Related
Is it possible to have a muti-line status spinner for python with something like yaspin or equiv?
For example, I would like to use multiprocessing and show the output of n number of items at one time:
⌛ Loading Version 1
✔️ Completed Version 8
⌛ Starting Version 3
✘ Task Version 77 failed
⌛ Running Something Different
In the sample above, i am just showing an hourglass, checkmark, etc. to indicate running/success/error.
Halo is going to support this exact use case as soon as this PR gets approved.
EDIT
If you want to use this feature straight away, (as of today) this is the fork you are looking for: frostming/halo:multiple-spinners
For instance, to use it as a dependency with poetry:
halo = { git = "https://github.com/frostming/halo.git", branch = "multiple-spinners" }
I'm using rebot tool to merge outputs after re-execution of failed tests.
robot --output original.xml /path/to/dir/.
robot --rerunfailed original.xml --output rerun.xml /path/to/dir/.
rebot -o machine1.xml -l machine1.html --merge original.xml rerun.xml
The same action is provided on few test machines.
Test suite is identical and is executed against each VM. All VMs are considered as identical, however, they are not stable and I receive different results on each machine.
I want to merge all results from all machines and retrieve the maximum number of Passed tests, to understand whether the test is REALLY failing or it's just unstable environment and test is OK by itself.
Other words, if test passed at least on 1 machine, but failed on other 3 machines, it needs to be considered as Passed in the final report.
However I receive False for such case.
Is it possible to change the behavior somehow?
Example from the final report:
Status: FAIL (critical)
Message: Re-executed test has been merged.
New status: FAIL
New message: Re-executed test has been merged.
New status: FAIL
New message: IndexError: Given index 0 is out of the range 0--1.
Old status: FAIL
Old message: IndexError: Given index 0 is out of the range 0--1.
Old status: PASS
Old message: Re-executed test has been merged.
New status: PASS
New message:
Old status: PASS
Old message: Re-executed test has been merged.
New status: PASS
New message:
Old status: PASS
Old message:
Resolved by using simple script for prerebotmodifier written by myself. Maybe it will be helpful for somebody else.
So, I created TestStatusChecker.py which helps to merge reports with the following logic: test status will be failed ONLY if it was failing in ALL reports.
from robot.api import SuiteVisitor
class TestStatusChecker(SuiteVisitor):
def __init__(self, *args):
pass
def visit_test(self, test):
if 'PASS' in test.message and 'Re-executed test has been merged' in test.message:
test.status = 'PASS'
test.message = 'Test passed because it passed at least once.'
CLI command for merging the results:
rebot -l final_log.html --prerebotmodifier TestStatusChecker.py --merge 1.xml 2.xml 3.xml 4.xml
report
I'm running iPython (Jupyter) through Anaconda, on a Mac Sierra, through iTerm, with $SHELL=bash - if I've missed any helpful set up details, just let me know.
I love the $HISTCONTROL aspect of bash, mentioned here. To sum that answer up: when traversing history (aka hitting the up arrow), it's helpful to remove duplicate entries so you don't scroll past the same command multiple times, and this is accomplished with $HISTCONTROL=ignoreboth:erasedups.
Is there any equivalent for this inside the Python interpreter (or iPython, specifically)? I have readline installed and feel like that's a good place to start, but nothing jumped out as obviously solving the problem, and I would've thought this was built in somewhere.
Through some deep-diving into IPython, sifting through poorly-explained and/or deprecated documentation, I've pieced together a solution that seems to work fine, though I'm sure it's not optimal for a number of reasons, namely:
it runs a GROUP BY query on the history database every time I run a line in IPython
it doesn't take care to clean up/coordinate the database tables - I only modify history, but ignore output_history and sessions tables
I put the following in a file (I named it dedupe_history.py, but name is irrelevant) inside $HOME/.ipython/profile_default/startup:
import IPython
import IPython.core.history as H
## spews a UserWarning about locate_profile() ... seems safe to ignore
HISTORY = H.HistoryAccessor()
def dedupe_history():
query = ("DELETE FROM history WHERE rowid NOT IN "
"(SELECT MAX(rowid) FROM history GROUP BY source)")
db = HISTORY.db
db.execute(query)
db.commit()
def set_pre_run_cell_event():
IPython.get_ipython().events.register("pre_run_cell", dedupe_history)
## dedupe history at start of new session - maybe that's sufficient, YMMV
dedupe_history()
## run dedupe history every time you run a command
set_pre_run_cell_event()
Edit: So apparantly my install wasn't working. This pointed me to a mailing list Here where I figured out which commands I was missing. I have the answer for the update below. Now that I think about it, it does make sense. I just wish they'd put this somewhere simple on the dev pages.
yb = yum.YumBase()
yb.conf.assumeyes = True
yb.update(name='aws-cli')
yb.buildTransaction()
yb.processTransaction()
I'm trying to perform an update using yumbase when a server first boots with my kickstart script. At the moment I have a rather crude python subprocess to do "yum update" and would like to make this better.
I'm trying to hook into Yumbase, but the documentation is quite scarce. I have had a look at both the source code and documentation on this page: http://yum.baseurl.org/wiki/5MinuteExamples
I've figured out how to list all packages but not the ones that need updating using an SO answer from 2008: Given an rpm package name, query the yum database for updates
I've also figured out it's a very simple 3-line process to install a new package:
yb = yum.YumBase()
yb.conf.assumeyes = True
yb.install(name='aws-cli')
However the following doesn't work to "update" the package:
yb = yum.YumBase()
yb.conf.assumeyes = True
yb.update(name='aws-cli')
So what I need is:
1: A way to list the packages that need updating, much like "yum check-update"
2: Install the packages above using "yum update"
From what I can see in the yum code, it doesn't seem to be written to be used as a library. The code you gave is not the right way to do it, there's much else happening behind the scenes.
Basically, as of yum-3.4.3, the process looks like this:
->yummain.__main__
<trap KeyboardInterrupt>
->yummain.user_main(sys.argv[1:], exit_code=True)
<check YUM_PROF,YUM_PDB envvars, wrap the following into debugger/profiler if set>
->yummain.main(args)
<set up locale, set up logging>
-><create a YumBaseCli (child of YumBase & YumOutput)>
<incl. fill a list field with YumCommand instances of known commands>
->cli.YumBaseCli.getOptionsConfig()
<parse args into the YumBaseCli instance, includes initializing plugins>
<obtain global yum lock>
<check write permissions for current dir>
->cli.YumBaseCli.doCommands()
<select a YumCommand from the list>
->YumCommand.needTs/needTsRemove if needed
->YumCommand.doCommand(self, self.basecmd, self.extcmds)
<handle errors & set error code if any>
'Resolving Dependencies'
->cli.YumBaseCli.buildTransaction()
<check for an unfinished transaction>
<resolve deps using the info written by the YumCommand into the object>
<honor clean_requirements_on_remove, protected_packages,
protected_multilib, perform some checks>
<handle errors & set error code if any>
'Dependencies Resolved'
->cli.YumBaseCli.doTransaction()
<download, transaction check, transaction test, transaction
using the info in the object>
<handle errors & set error code if any>
'Complete!'
<release global yum lock>
sys.exit(error_code)
As you can see, the main working sequence is embedded directly into main so you can only replicate this logic in-process by running it directly:
yummain.main(<sequence of cmdline arguments>)
Which is just the same as running a separate process minus process isolation.
Well the title is self explanatory. What will be the python code equivalent to running git reset --hard (on terminal) using GitPython module?
You can use:
repo = git.Repo('c:/SomeRepo')
repo.git.reset('--hard')
Or if you need to reset to a specific branch:
repo.git.reset('--hard','origin/master')
Or in my case, if you want to just hard update a repo to origin/master (warning, this will nuke your current changes):
# blast any current changes
repo.git.reset('--hard')
# ensure master is checked out
repo.heads.master.checkout()
# blast any changes there (only if it wasn't checked out)
repo.git.reset('--hard')
# remove any extra non-tracked files (.pyc, etc)
repo.git.clean('-xdf')
# pull in the changes from from the remote
repo.remotes.origin.pull()
I searched for reset in the documentation and found this:
class git.refs.head.HEAD(repo, path='HEAD')
reset(commit='HEAD', index=True, working_tree=False, paths=None, **kwargs)
Reset our HEAD to the given commit optionally synchronizing the index and working tree. The reference we refer to will be set to commit as well.
You can use:
repo = git.Repo('repo')
# ...
# Remove last commit
repo.head.reset('HEAD~1', index=True, working_tree=True)