We Keep Coding

Elastalert deployment failed

I have installed elastalert on Centos 7.6 and while starting the elastalert receiving the following error.
[root#e2e-27-36 elastalert]# python -m elastalert.elastalert --verbose --rule example_rules/example_frequency.yaml
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/root/elastalert/elastalert/elastalert.py", line 29, in <module>
from . import kibana
File "elastalert/kibana.py", line 4, in <module>
import urllib.error
ImportError: No module named error
How should I go about fixing this?

You can try to check if urllib3 is installed by running pip freeze or try to reinstall it with pip install urllib3.
You maybe need to correctly activate your environment variable like this : source [env]/bin/activate.

Setup conda environment
conda create -n elastalert python=3.6 anaconda
Activate conda env
conda activate elastalert
Install all the requirements
pip install -r requirements-dev.txt
pip install -r requirements.txt

I have found my fix by own.
1.On python2.7 the issue still persist
2.Install python3.6 version to fix the issue.
yum install python3 python3-devel python3-urllib3
3.Run the elastalert command
python3 -m elastalert.elastalert --config /root/elastalert/config.yaml --verbose --rule /root/elastalert/example_rules/example_frequency.yaml
4.If you received issue with the modules (ModuleNotFoundError: No module named 'pytz')
5.Install the modules as per the requirement.
pip3 install -r /root/elastalert/requirements.txt
6.Let's run the command "python3 -m elastalert.elastalert --config /root/elastalert/config.yaml --verbose --rule /root/elastalert/example_rules/example_frequency.yaml" and got error
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='elasticsearch.example.com', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known',))
7.Above error due to not valid hostname on config.yaml file. Edit the config.yaml file and change the hostname to server hostname at es.hosts field
Make sure you had an entry for the same on the /etc/hosts file.
8.Ok the issue got fixed and run the command "python3 -m elastalert.elastalert --config /root/elastalert/config.yaml --verbose --rule /root/elastalert/example_rules/example_frequency.yaml" and one more error
pkg_resources.DistributionNotFound: The 'jira>=2.0.0'
9.We need to install the jira by using below command
pip3 install jira==2.0.0
10.Now let's run the command "python3 -m elastalert.elastalert --config /root/elastalert/config.yaml --verbose --rule /root/elastalert/example_rules/example_frequency.yaml" and again another error OMG.
elasticsearch.exceptions.TransportError: TransportError(429, 'circuit_breaking_exception', '[parent] Data too large, data for [] would be [994793504/948.7mb], which is larger than the limit of [986061209/940.3mb], real usage: [994793056/948.7mb], new bytes reserved: [448/448b]')
11.You need to fix the same by changing the heap value on following /etc/elasticsearch/jvm.options
Xms-1g to Xms-2g
Xmx-1g to Xms-2g
and restart elasticsearch service "service elasticsearch restart"
12.Everything set again run the command "python3 -m elastalert.elastalert --config /root/elastalert/config.yaml --verbose --rule /root/elastalert/example_rules/example_frequency.yaml" and ended up receiving another error.
ERROR:root:Error finding recent pending alerts: NotFoundError(404, 'index_not_found_exception', 'no such index [elastalert_status]', elastalert_status, index_or_alias) {'query': {'bool': {'must': {'query_string': {'query': '!exists:aggregate_id AND alert_sent:false'}}, 'filter': {'range': {'alert_time': {'from': '2019-12-04T19:45:09.635478Z', 'to': '2019-12-06T19:45:09.635529Z'}}}}}, 'sort': {'alert_time': {'order': 'asc'}}}
13.Fix the issue by running the below command
elastalert-create-index
14.Finally everything done and run the below command
python3 -m elastalert.elastalert --config /root/elastalert/config.yaml --verbose --rule /root/elastalert/example_rules/example_frequency.yaml
Now cancelled the command and ran the same on background
python3 -m elastalert.elastalert --config /root/elastalert/config.yaml --verbose --rule /root/elastalert/example_rules/example_frequency.yaml &

Error creating virtualenv with Python3

I'm working on Linux Mint 17 and I'm trying to create a new virtualenv with Python3 like this:
python3.6 -m venv env
And this is the error that I get:
Error: Command '['/home/ric/myprojs/django-example-channels/env/bin/python3.6', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.
I've googled this error message but haven't managed to find anything too informative.
This is my pip version, in case it make any difference:
pip --version
pip 9.0.1 from /usr/local/lib/python2.7/dist-packages (python 2.7)
I've been using Python2 for some time, but I'm new to Python3. I don't know what I may be missing.
UPDATE 1:
Answering #cezar's question, when I type  which python3 this is what I get:
$ which python3
/usr/bin/python3
UPDATE 2:
Answering #Chłop Z Lasu:
$ virtualenv -p python3.6 env
Running virtualenv with interpreter /usr/bin/python3.6
Using base prefix '/usr'
New python executable in /home/ric/myprojs/django-example-channels/example_channels/env/bin/python3.6
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 2328, in <module>
main()
File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 713, in main
symlink=options.symlink)
File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 925, in create_environment
site_packages=site_packages, clear=clear, symlink=symlink))
File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 1231, in install_python
shutil.copyfile(executable, py_executable)
File "/usr/lib/python3.6/shutil.py", line 104, in copyfile
raise SameFileError("{!r} and {!r} are the same file".format(src, dst))

The error indicates that virtualenv is trying to make an environment in your python path. therefore, you have to specify your virtualenv destination
virtualenv -p python3.6 /path/to/yourenv

For venv and python3.6 installing venv one can use below to fix the issue
sudo apt install python3.6-venv

You gotta install the venv via code bellow:
sudo apt install python3.6-venv

Expanding on the Answer Above:
The error indicates that virtualenv is trying to make an environment in your python path. therefore, you have to specify your virtualenv destination
Since -p is a flag equivalent to --python, you can also use.
$ virtualenv --python=python3.6 /path/to/yourenv
Or
$ virtualenv -p python3.6 /path/to/yourenv

How to resolve this error while installing devstack kilo on ubuntu

This is what I did:
1) Cloned devstack kilo using the command:
git clone https://github.com/openstack-dev/devstack.git -b stable/kilo
2) cd devstack
3) ./stack.sh
I get the following error while installing devstack kilo on ubuntu.
You are using pip version 7.1.2, however version 8.0.2 is available. You should consider upgrading via the 'pip install --upgrade pip' command.
+ local 'test_req=testtools>=0.9.36,!=1.2.0,<2.0.0/test-requirements.txt'
+ [[ -e testtools>=0.9.36,!=1.2.0,<2.0.0/test-requirements.txt ]]
+ pip_install_gr os-testr
+ local name=os-testr
++ get_from_global_requirements os-testr
++ local package=os-testr
+++ grep -i -h '^os-testr' /opt/stack/requirements/global-requirements.txt
+++ cut -d# -f1 grep: /opt/stack/requirements/global-requirements.txt: No such file or directory
++ local required_pkg=
++ [[ '' == '' ]]
++ die 142 'Can'\''t find package os-testr in requirements'
++ local exitcode=0
++ set +o xtrace [ERROR] /home/nikhil/devstack/inc/python:142 Can't find package os-testr in requirements
+ local 'clean_name=[Call Trace] ./stack.sh:715:pip_install_gr /home/nikhil/devstack/inc/python:63:get_from_global_requirements /home/nikhil/devstack/inc/python:142:die'
+ pip_install '[Call' 'Trace]' ./stack.sh:715:pip_install_gr /home/nikhil/devstack/inc/python:63:get_from_global_requirements /home/nikhil/devstack/inc/python:142:die
++ set +o
++ grep xtrace
+ local 'xtrace=set -o xtrace'
+ set +o xtrace
+ sudo -H http_proxy= https_proxy= no_proxy= PIP_FIND_LINKS=file:///opt/stack/.wheelhouse /usr/local/bin/pip install '[Call' 'Trace]' ./stack.sh:715:pip_install_gr /home/nikhil/devstack/inc/python:63:get_from_global_requirements /home/nikhil/devstack/inc/python:142:die Exception: Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 211, in main
status = self.run(options, args) File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 282, in run
wheel_cache File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 272, in populate_requirement_set
wheel_cache=wheel_cache File "/usr/local/lib/python2.7/dist-packages/pip/req/req_install.py", line 213, in from_line
wheel_cache=wheel_cache, constraint=constraint) File "/usr/local/lib/python2.7/dist-packages/pip/req/req_install.py", line 67, in __init__
req = pkg_resources.Requirement.parse(req) File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 2980, in parse
reqs = list(parse_requirements(s)) File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 2911, in parse_requirements
raise RequirementParseError("Missing distribution spec", line) RequirementParseError: Missing distribution spec [Call You are using pip version 7.1.2, however version 8.0.2 is available. You should consider upgrading via the 'pip install --upgrade pip' command.
+ exit_trap
+ local r=2
++ jobs -p
+ jobs=
+ [[ -n '' ]]
+ kill_spinner
+ '[' '!' -z '' ']'
+ [[ 2 -ne 0 ]]
+ echo 'Error on exit' Error on exit
+ generate-subunit 1455806136 50 fail ./stack.sh: line 463: generate-subunit: command not found
Here is the screenshot:
I think that the error is caused by the python package. I updated python and tried again but no avail.
What is the error and how can I solve it please?

In your command line:
sudo apt-get install python-pip
sudo pip install --upgrade pip
sudo pip install -U os-testr
This worked fine for me...

This worked for me.
sudo apt-get install python-pip
sudo pip install --upgrade pip
Looks like I've installed and upgraded python.
I remember trying this before but it didn't work. I don't know why, but it worked now.
It would be helpful if anyone can state the reason behind the error and delineate the solution.

Have you tried this?
sudo pip install -U os-testr
Source: https://ask.openstack.org/en/question/88189/open-stack-installation-i-got-an-error/?answer=88228#post-id-88228

pip broken, reinstall doesn't work. EC2

I am using pip on EC2 now, python version is 2.7. 'sudo pip' suddenly doesn't work anymore.
[ec2-user#ip-172-31-17-194 ~]$ sudo pip install validate_email
Traceback (most recent call last):
File "/usr/bin/pip", line 5, in <module>
from pkg_resources import load_entry_point
File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3138, in <module>
#_call_aside
File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3124, in _call_aside
f(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3151, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 663, in _build_master
return cls._build_from_requirements(__requires__)
File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 676, in _build_from_requirements
dists = ws.resolve(reqs, Environment())
File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 849, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'pip==6.1.1' distribution was not found and is required by the application
[ec2-user#ip-172-31-17-194 ~]$ which pip
/usr/local/bin/pip

first, which pip is not going to return the same result as sudo which pip, so you should check that out first.
you may also consider not running pip as sudo at all.
Is it acceptable & safe to run pip install under sudo?
second, can you try this:
easy_install --upgrade pip
if you get an error here (regarding pip's wheel support), try this, then run the above command again:
easy_install -U setuptools

I fixed the same error ("The 'pip==6.1.1' distribution was not found") by using the tip of Wesm :
$> which pip && sudo which pip
/usr/local/bin/pip
/usr/bin/pip
So, it seels that "pip" of average user and of root are not the same. Will fix it later.
Then I ran "sudo easy_install --upgrade pip" => succeed
Then I used "sudo /usr/local/bin/pip install " and it works.

Some additional information for anyone who is also stuck on the same issue:-
Running commands with sudo searches for the command in usr/bin directory. One way to solve this issue is to specify the complete path to the command while using sudo as commented by #Cissoid in the question's comment section
Or
...what you can do is create a symbolic link(sym link) to that command in the usr/bin directory using ln command.
$> ln -s /usr/local/bin/pip /usr/bin/pip
The syntax is:-
$> ln -s /path/to/file /path/to/link

I tried a few of these solutions without much success. In the end I just created a new instance using Ubuntu as the operating system. It was already setup properly for using Python properly.
If that is not possible then you can try linking the user pip into a folder on root's (sudo) path.

python pip error pkg_resources.DistributionNotFound

I'm trying to install a new module, but I found that pip doesn't work - it shows
pkg_resources.DistributionNotFound: The 'pip==1.5.6' distribution was not found and is required by the application
for each call I performed. Also, for pip --help. I found some advises to reinstall pip using pip or easy_install, but easy_install doesn't works too:
pkg_resources.DistributionNotFound: The 'setuptools==5.7' distribution was not found and is required by the application
Is there a way to fix this issue without reinstalling python and all its modules?
I have Linux Ubuntu 14.04.
apt-get install python-pip suggests installing pip (and a heap of another following software) like I don't have it in my system. I afraid to do it, would be a conflict here?
Upd.
apt-get says that it want to install the following: build-essential dpkg-dev fakeroot g++ g++-4.8 libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl libfakeroot libstdc++-4.8-dev python-chardet-whl python-colorama python-colorama-whl python-distlib python-distlib-whl python-html5lib python-html5lib-whl python-pip python-pip-whl python-requests-whl python-setuptools python-setuptools-whl python-six-whl python-urllib3-whl python-wheel. But as I know pip worked for me without these packages.
Also I performed
which -a python: /usr/bin/python
which -a pip: /usr/local/bin/pip
python --version: Python 2.7.6

It would be useful to have the full backtrace of the error. One scenario that can lead to this issue, is having multiple versions of Python installed.
For example, on my Ubuntu 20.04.1 LTS I have installed Python 3.6 from source, in addition to Python 3.8 distributed with Ubuntu 20. This is what I get with pip3 --version after installing Python 3.6:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 581, in _build_master
ws.require(__requires__)
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 898, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 789, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (pip 20.2.3 (/usr/local/lib/python3.6/site-packages), Requirement.parse('pip==20.0.2'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/pip3", line 6, in <module>
from pkg_resources import load_entry_point
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3126, in <module>
#_call_aside
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3110, in _call_aside
f(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3139, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 583, in _build_master
return cls._build_from_requirements(__requires__)
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 596, in _build_from_requirements
dists = ws.resolve(reqs, Environment())
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 784, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'pip==20.0.2' distribution was not found and is required by the application
The last line is the error in question, but notice an important detail in the traceback: raise VersionConflict.
This can be solved by explicitly using the required version of pip:
$ python3.6 -m pip --version
pip 20.2.3 from /usr/local/lib/python3.6/site-packages/pip (python 3.6)
Or simply:
$ pip3.6 --version
pip 20.2.3 from /usr/local/lib/python3.6/site-packages/pip (python 3.6)
Another issue one may stumble upon, is the missing lsb_release issue when doing pip3.6 install:
File "/usr/local/lib/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '('lsb_release', '-a')' returned non-zero exit status 1.
This can be solved by locating lsb_release.py and creating a symlink, for example:
sudo ln -s /usr/lib/python3/dist-packages/lsb_release.py /usr/local/lib/python3.6/site-packages/lsb_release.py
Now it's possible to upgrade pip3.6 and install new packages:
$ pip3.6 install --upgrade pip
$ pip3.6 install mypy
There are different methods for managing Python versions. Beware of using something like sudo update-alternatives --config python. Changing the system's default Python, will likely cause problems, including symptoms like the terminal not opening. It's good to have a backup terminal available, such as the one in Visual Studio Code.