I have the edx-platform, ecommerce, ecommerce-themes, credentials and edx-theme directories. I have installed successfully tutor and devstack but I didn't find the way to replace these custom directories. So, what is the correct way to replace them ?
After devstack runned successfully, I tried replacing the default directories with the custom ones but when I runned make dev.provision I get this output
+ docker-compose exec -T lms bash -e -c 'source /edx/app/edxapp/edxapp_env && cd /edx/app/edxapp/edx-platform && NO_PYTHON_UNINSTALL=1 paver install_prereqs'
/edx/app/edxapp/edxapp_env: line 13: manpath: command not found
---> pavelib.prereqs.install_prereqs
---> pavelib.prereqs.install_node_prereqs
npm install error detected. Retrying...
Captured Task Output:
---------------------
---> pavelib.prereqs.install_prereqs
---> pavelib.prereqs.install_node_prereqs
Traceback (most recent call last):
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/paver/tasks.py", line 201, in _run_task
return do_task()
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/paver/tasks.py", line 198, in do_task
return func(**kw)
File "/edx/app/edxapp/edx-platform/pavelib/utils/timer.py", line 40, in timed
return wrapped(*args, **kwargs)
File "/edx/app/edxapp/edx-platform/pavelib/prereqs.py", line 332, in install_prereqs
install_node_prereqs()
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/paver/tasks.py", line 333, in __call__
retval = environment._run_task(self.name, self.needs, self.func)
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/paver/tasks.py", line 219, in _run_task
return do_task()
File "/edx/app/edxapp/venvs/edxapp/lib/python3.8/site-packages/paver/tasks.py", line 198, in do_task
return func(**kw)
File "/edx/app/edxapp/edx-platform/pavelib/utils/timer.py", line 40, in timed
return wrapped(*args, **kwargs)
File "/edx/app/edxapp/edx-platform/pavelib/prereqs.py", line 184, in install_node_prereqs
prereq_cache("Node prereqs", ["package.json"], node_prereqs_installation)
File "/edx/app/edxapp/edx-platform/pavelib/prereqs.py", line 111, in prereq_cache
install_func()
File "/edx/app/edxapp/edx-platform/pavelib/prereqs.py", line 154, in node_prereqs_installation
raise Exception("npm install failed: See {}".format(npm_log_file_path))
Exception: npm install failed: See /edx/app/edxapp/edx-platform/test_root/log/npm-install.log
make[1]: *** [Makefile:217: impl-dev.provision] Error 1
make[1]: Leaving directory '/home/pablo/Documents/prueba/devstack'
Would you like to assist devstack development by sending anonymous usage metrics to edX? Run `make metrics-opt-in` to learn more!
make: *** [Makefile:221: dev.provision] Error 2
EDIT
The directories that I have after run make dev.provision and make dev.up with the default project of devstack, are the following ones:
Directories of default project devstack openedx
The thing that I tried was replace the default directories with the custom directories (open-edx-platform, ecommerce, ..., etc).
If you look at your log, you'll see the following error:
/edx/app/edxapp/edxapp_env: line 13: manpath: command not found
This suggests (guessing here!) you're missing man. So try installing it, e.g.:
apt install man
Others have reported similar kinds of errors when using nvm, e.g. see here
It seems the npm install is failing.
I don't know if you already have node & npm installed but you could try installing it, and see what the result is afterwards.
sudo apt update
sudo apt install nodejs npm
If you posted the npm log file, it would help a lot. You should be able to find it, per the source code, in the directory specified by the environment variable GEN_LOG_DIR:
{Env.GEN_LOG_DIR}/npm-install.log'
Related
I have an analysis pipeline using snakemake where one rule requires a singularity container, and several others require conda environments. I specify the "container:" and "conda:" directives where needed within the Snakefile.
The way snakemake is invoked is a little complex. For various reasons, I invoke the snakemake API via a custom script within a conda environment, with both "use_conda=True" and "use_singularity=True" specified in the config. The pipeline has completed with no issues in the past, but someone recently reinstalled the pipeline, and now I get the following error:
/bin/sh: 1: conda: not found
Traceback (most recent call last):
File "/home/cfos/miniconda3/envs/CIS/lib/python3.10/site-packages/snakemake/__init__.py", line 726, in snakemake
success = workflow.execute(
File "/home/cfos/miniconda3/envs/CIS/lib/python3.10/site-packages/snakemake/workflow.py", line 978, in execute
dag.create_conda_envs(
File "/home/cfos/miniconda3/envs/CIS/lib/python3.10/site-packages/snakemake/dag.py", line 313, in create_conda_envs
env.create(dryrun)
File "/home/cfos/miniconda3/envs/CIS/lib/python3.10/site-packages/snakemake/deployment/conda.py", line 391, in create
pin_file = self.pin_file
File "/home/cfos/miniconda3/envs/CIS/lib/python3.10/site-packages/snakemake/common/__init__.py", line 192, in __get__
value = self.method(instance)
File "/home/cfos/miniconda3/envs/CIS/lib/python3.10/site-packages/snakemake/deployment/conda.py", line 107, in pin_file
f".{self.conda.platform}.pin.txt"
File "/home/cfos/miniconda3/envs/CIS/lib/python3.10/site-packages/snakemake/common/__init__.py", line 192, in __get__
value = self.method(instance)
File "/home/cfos/miniconda3/envs/CIS/lib/python3.10/site-packages/snakemake/deployment/conda.py", line 102, in conda
return Conda(self._container_img)
File "/home/cfos/miniconda3/envs/CIS/lib/python3.10/site-packages/snakemake/deployment/conda.py", line 648, in __init__
shell.check_output(
File "/home/cfos/miniconda3/envs/CIS/lib/python3.10/site-packages/snakemake/shell.py", line 63, in check_output
return sp.check_output(cmd, shell=True, executable=executable, **kwargs)
File "/home/cfos/miniconda3/envs/CIS/lib/python3.10/subprocess.py", line 420, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/home/cfos/miniconda3/envs/CIS/lib/python3.10/subprocess.py", line 524, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command ' singularity --quiet --silent exec --home /home/cfos/Programs/COVID_Illumina_Snakemake /home/cfos/Programs/COVID_Illumina_Snakemake/.snakemake/singularity/1ad1f7122bf7704a6b02b6359ede7533.simg sh -c 'conda info --json'' returned non-zero exit status 127.
I get the same error if I run the snakemake command 'manually' (outside of my script) like so:
snakemake -j20 --use-conda --use-singularity -s /path/to/Snakefile --config [OPTIONS]
It appears like snakemake is checking for the presence of conda within a singularity container that I have only specified for one rule. This is not the desired behaviour, since conda is not within that container. Instead, snakemake should be detecting and using the system installation of miniconda, and the the installation of mamba within the conda environment from which I run the pipeline. This is the way the pipeline used to run with no problems.
Does anyone have a solution to this problem? The confusing thing is that I've tried various versions of snakemake, including ones that worked well previously, and I still get the error.
Details
snakemake v7.8.5
conda 4.11.0
If it helps to understand the way I invoke the pipeline: https://github.com/charlesfoster/covid-illumina-snakemake
Thanks!
I am trying to install sagemath package in my macbook. I entered
./configure followed by make. Here are the last few
lines of output. Couldn't debug the cause.
[sagelib-8.7] Finished cleaning, time: 0.40 seconds.
[sagelib-8.7] if [ "$UNAME" = "CYGWIN" ]; then \
[sagelib-8.7] sage-rebase.sh "$SAGE_LOCAL" 2>/dev/null; \
[sagelib-8.7] fi
[sagelib-8.7]
[sagelib-8.7] real 19m37.851s
[sagelib-8.7] user 59m16.416s
[sagelib-8.7] sys 2m58.759s
cp /Users/satya/Downloads/SageMath/src/bin/sage-env-config /Users/satya/Downloads/SageMath/local/bin/sage-env-config
cd ../.. && sage-logger -p './sage --docbuild --no-pdf-links all html ' logs/dochtml.log
[dochtml] Traceback (most recent call last):
[dochtml] File "/Users/satya/Downloads/SageMath/local/lib/python2.7/runpy.py", line 163, in _run_module_as_main
[dochtml] mod_name, _Error)
[dochtml] File "/Users/satya/Downloads/SageMath/local/lib/python2.7/runpy.py", line 111, in _get_module_details
[dochtml] __import__(mod_name) # Do not catch exceptions initializing package
[dochtml] File "/Users/satya/Downloads/SageMath/local/lib/python2.7/site-packages/sage_setup/docbuild/__init__.py", line 60, in <module>
[dochtml] import sage.all
[dochtml] File "/Users/satya/Downloads/SageMath/local/lib/python2.7/site-packages/sage/all.py", line 98, in <module>
[dochtml] from sage.symbolic.all import *
[dochtml] File "/Users/satya/Downloads/SageMath/local/lib/python2.7/site-packages/sage/symbolic/all.py", line 3, in <module>
[dochtml] from sage.libs.pynac.pynac import I
[dochtml] File "sage/symbolic/expression.pxd", line 4, in init sage.libs.pynac.pynac (build/cythonized/sage/libs/pynac/pynac.cpp:30147)
[dochtml] File "sage/symbolic/expression.pyx", line 161, in init sage.symbolic.expression (build/cythonized/sage/symbolic/expression.cpp:74075)
[dochtml] ImportError: dlopen(/Users/satya/Downloads/SageMath/local/lib/python2.7/site-packages/sage/symbolic/ring.so, 2): Symbol not found: __ZNSt3__16vectorIN5GiNaC2exENS_9allocatorIS2_EEE11__vallocateEm
[dochtml] Referenced from: /Users/satya/Downloads/SageMath/local/lib/python2.7/site-packages/sage/symbolic/ring.so
[dochtml] Expected in: flat namespace
[dochtml] in /Users/satya/Downloads/SageMath/local/lib/python2.7/site-packages/sage/symbolic/ring.so
make[3]: *** [doc-html] Error 1
make[2]: *** [all-start] Error 2
real 19m41.468s
user 59m18.862s
sys 3m0.256s
***************************************************************
Error building Sage.
The following package(s) may have failed to build (not necessarily
during this run of 'make all-start'):
The build directory may contain configuration files and other potentially
helpful information. WARNING: if you now run 'make' again, the build
directory will, by default, be deleted. Set the environment variable
SAGE_KEEP_BUILT_SPKGS to 'yes' to prevent this.
make[1]: *** [all-start] Error 1
make: *** [all] Error 2
Any help would be appreciated.
Running make for SageMath proceeds in two steps: make build
followed by make doc.
In your case, the make build phase seems to have completed,
and there was an issue only in the make doc phase.
This means you probably already have a functional SageMath,
with the drawback that the documentation failed to build.
So you might decide to try and use it as is.
Following #John Palmieri, I would recommend posting to sage-devel
or sage-support to get help debugging the make failure. It will
be useful to specify the operating system on which it occurred
(what version of macOS), if the buid was started from a directory
obtained by git clone or from a source tarball, what exact
sequence of operations were run.
Also, on your next attempt, you might want to move the SageMath
folder out of the Downloads folder before you run make, since
a Sage installation cannot be moved after it has been started once
(if Sage is moved, one has to run make again before it can be used).
Building off Samuel's answer, a different thing you might want to try is simply to run make doc-clean and then make doc again. I have found that (on Mac) there is sometimes a weird burp I can never track down, and this usually gets past it.
I have a small rest api built in python using Falcon Framework and it runs in a virtual environment with gunicorn + gevent and pypy 2.3 (I successfully installed the pip packages from gevent#pypy-hacks and gevent-on-pypy/pypycore). I wanted to add a "supervisor" for my application so I first installed supervisord and followed the steps to get it up and running.
The first thing I did in the server was to test the command to start the api; it ran successfully and I could use the api without any issues. The command is:
/bin/bash -c 'cd /path/to/project/api && /path/to/project/env/bin/gunicorn -c settings.py my-api:my_api'
Note: in settings.py I set the parameter daemon=False since I saw that supervisors require that the commands are run in the "foreground" and not daemonized.
Once that was working, I went ahead and created the app's /etc/supervisor/conf.d/my_api.conf conf file with the "command" parameter set as the command above I ran successfully. When I tried to start the api through supervisord, it failed, and in the log of the api, the error was:
Traceback (most recent call last):
File "/path/to/project/env/site-packages/gunicorn/arbiter.py", line 507, in spawn_worker
worker.init_process()
File "/path/to/project/env/site-packages/gunicorn/workers/ggevent.py", line 185, in init_process
self.patch()
File "/path/to/project/env/site-packages/gunicorn/workers/ggevent.py", line 74, in patch
_sock=s))
File "/path/to/project/env/site-packages/gevent/socket.py", line 240, in __init__
self.hub = get_hub()
File "/path/to/project/env/site-packages/gevent/hub.py", line 169, in get_hub
hub = _threadlocal.hub = hubtype(*args, **kwargs)
File "/path/to/project/env/site-packages/gevent/hub.py", line 268, in __init__
loop_class = _import(self.loop_class)
File "/path/to/project/env/site-packages/gevent/hub.py", line 198, in _import
return _import(path[-1])
File "/path/to/project/env/site-packages/gevent/hub.py", line 210, in _import
x = __import__(module)
ImportError: No module named gevent.core
So, I was surprised that running the command manually did work, but when supervisor tried to run it, the above error was thrown.
After lots of trial and errors, I decided to install monit to see if it could monitor my api, and I did set it up successfully specifying the "start program" parameter as the aforementioned command.
I was surprised that monit throwed the exact same error. So, does this mean that the nature of my api (pypy + gunicorn + gevent) prevents itself from being monitored by any monitor software around there?
Is there something I'm doing wrong when configuring supervisord/monit?
Any help is greatly appreciated.
Turns out that I have the environment var GEVENT_LOOP already exported in my /etc/environment as follows:
export GEVENT_LOOP=pypycore.loop
Since for running gevent on pypy some hacks are needed including the above export.
But I needed to export it manually in the 'start_program' command entry of Monit (at the end I sticked with Monit), so the resulting command is:
/bin/bash -c 'cd /path/to/project/api && export GEVENT_LOOP=pypycore.loop && /path/to/project/env/bin/gunicorn -c settings.py my-api:my_api'
nosetests --with-gae fails when running under the new cloud sdk. It's looking for dev_appserver.py but looks like it's been renamed to old_dev_appserver.py. I can hack a fix but is there an existing solution? Don't see one on the internets...
$ nosetests --with-gae --gae-lib-root=~/google-cloud-sdk/platform/google_appengine
Traceback (most recent call last):
File "/Users/edahl/src/gmj/bin/nosetests", line 9, in <module>
load_entry_point('nose==1.3.4', 'console_scripts', 'nosetests')()
File "/Users/edahl/src/gmj/lib/python2.7/site-packages/nose/core.py", line 121, in __init__
**extra_args)
File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/main.py", line 94, in __init__
self.parseArgs(argv)
File "/Users/edahl/src/gmj/lib/python2.7/site-packages/nose/core.py", line 145, in parseArgs
self.config.configure(argv, doc=self.usage())
File "/Users/edahl/src/gmj/lib/python2.7/site-packages/nose/config.py", line 346, in configure
self.plugins.configure(options, self)
File "/Users/edahl/src/gmj/lib/python2.7/site-packages/nose/plugins/manager.py", line 284, in configure
cfg(options, config)
File "/Users/edahl/src/gmj/lib/python2.7/site-packages/nose/plugins/manager.py", line 99, in __call__
return self.call(*arg, **kw)
File "/Users/edahl/src/gmj/lib/python2.7/site-packages/nose/plugins/manager.py", line 167, in simple
result = meth(*arg, **kw)
File "/Users/edahl/src/gmj/lib/python2.7/site-packages/nosegae.py", line 80, in configure
from dev_appserver import fix_sys_path
ImportError: No module named dev_appserver
Here's what I see in my google_appengine directory...
$ ls ~/google-cloud-sdk/platform/google_appengine/
BUGS api_server.py godoc php
LICENSE backends_conversion.py gofmt php_cli.py
README bulkload_client.py google remote_api_shell.py
RELEASE_NOTES bulkloader.py google_sql.py run_tests.py
RELEASE_NOTES.go_sdk demos goroot tools
VERSION download_appstats.py lib wrapper_util.py
_php_runtime.py gen_protorpc.py new_project_template wrapper_util.pyc
_python_runtime.py goapp old_dev_appserver.py
The old /usr/local/google_appengine directory seems to be deprecated... but if I use it I can get things working.
[update]
a manual unittest setup using old_dev_appserver like this does work:
def main(sdk_path, test_path):
sys.path.insert(0, sdk_path)
import old_dev_appserver
old_dev_appserver.fix_sys_path()
suite = unittest.loader.TestLoader().discover(test_path)
unittest.TextTestRunner(verbosity=2).run(suite)
In the end this was pretty silly... needed to expand the user path that's passed to --gae-lib-root
nosetests --with-gae --gae-lib-root=/Users/USERNAME/google-cloud-sdk/platform/google_appengine
On my mac, I get this same error:
After an app engine update
But before I've run the GoogleAppEngineLauncher app since the update
The easy solution is just launch GoogleAppEngineLauncher. This updates the sym links that nosetests needs.
Hopefully that works for you.
You might need to set your path to the lib folder under that directory:
--gae-lib-root=~/google-cloud-sdk/platform/google_appengine/lib
But you might also need to set your PATH variable so it includes the /bin as well:
export PATH=$PATH:~/google-cloud-sdk/platform/google_appengine/bin
If neither of those work for you, try adding the without sandbox switch:
nosetests -s -v --with-gae --without-sandbox
According to the NoseGAE docs the default location it checks for the google-cloud-sdk is in /usr/local/google_appengine.
Moving the google-cloud-sdk folder to /usr/local/google_appengine/google-cloud-sdk/ fixed this issue for me.
http://farmdev.com/projects/nosegae/
I have a project folder that is a git repository (has a .git folder). When I use the command pip freeze, pip tries to use git. However, I don't have git installed on my system so this causes an error:
(env) PS C:\Users\eclaird\work\myproject> pip freeze
Cannot find command 'git'
Storing complete log in C:\Users\eclaird\pip\pip.log
(env) PS C:\Users\eclaird\work\myproject>
pip.log:
------------------------------------------------------------
C:\Users\eclaird\work\env\Scripts\pip-script.py run on 01/09/14 11:54:42
Cannot find command 'git'
Exception information:
Traceback (most recent call last):
File "C:\Users\eclaird\work\env\lib\site-packages\pip\basecommand.py", line 134, in main
status = self.run(options, args)
File "C:\Users\eclaird\work\env\lib\site-packages\pip\commands\freeze.py", line 73, in run
req = pip.FrozenRequirement.from_dist(dist, dependency_links, find_tags=find_tags)
File "C:\Users\eclaird\work\env\lib\site-packages\pip\__init__.py", line 180, in from_dist
req = get_src_requirement(dist, location, find_tags)
File "C:\Users\eclaird\work\env\lib\site-packages\pip\vcs\__init__.py", line 249, in get_src_requirement
return version_control().get_src_requirement(dist, location, find_tags)
File "C:\Users\eclaird\work\env\lib\site-packages\pip\vcs\git.py", line 151, in get_src_requirement
repo = self.get_url(location)
File "C:\Users\eclaird\work\env\lib\site-packages\pip\vcs\git.py", line 122, in get_url
[self.cmd, 'config', 'remote.origin.url'],
File "C:\Users\eclaird\work\env\lib\site-packages\pip\vcs\__init__.py", line 110, in cmd
command = find_command(self.name)
File "C:\Users\eclaird\work\env\lib\site-packages\pip\util.py", line 108, in find_command
raise BadCommand('Cannot find command %r' % cmd)
BadCommand: Cannot find command 'git'
Is there a way to disable the git integration in pip?
(pip 1.4.1,
Python 2.7.6)
Unfortunately, no. There is no config option to enable/disable backends.
Details, found by digging in the code:
Git module is always registered: In pip/install.py, the git module is imported. At the end of it, it registers itself, and will thus be queried whenever one of the schemes declared in ´git.Git´ matches the url of the dependency.
schemes = ('git', 'git+http', 'git+https', 'git+ssh', 'git+git', 'git+file')
Exception is not handled Funnily, freezing anticipates that an error might occur when determining the dependency url to be frozen. Excerpt from pip.FrozenRequirement:
try:
req = get_src_requirement(dist, location, find_tags)
except InstallationError:
logger.warn("Error when trying to get requirement for VCS system %s, falling back to uneditable format" % ex)
InstallationError inherits from PipError. Unfortunately, an exception of type ´BadCommand´ is raised, which inherits from ´PipError´.
So, aside from hacking the source: Nothing you can do. If you need this to work, you need to install git, hack the source, or simulate a git executable. If you go for the latter, for starters you need to fake ´git config remote.origin.url´, which is called (and fails) in pip.vcs.git.Git.get_url.
Hope that helps, even though it's not a yes. ;)