We Keep Coding

Elastalert deployment failed

I have installed elastalert on Centos 7.6 and while starting the elastalert receiving the following error.
[root#e2e-27-36 elastalert]# python -m elastalert.elastalert --verbose --rule example_rules/example_frequency.yaml
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/root/elastalert/elastalert/elastalert.py", line 29, in <module>
from . import kibana
File "elastalert/kibana.py", line 4, in <module>
import urllib.error
ImportError: No module named error
How should I go about fixing this?

You can try to check if urllib3 is installed by running pip freeze or try to reinstall it with pip install urllib3.
You maybe need to correctly activate your environment variable like this : source [env]/bin/activate.

Setup conda environment
conda create -n elastalert python=3.6 anaconda
Activate conda env
conda activate elastalert
Install all the requirements
pip install -r requirements-dev.txt
pip install -r requirements.txt

I have found my fix by own.
1.On python2.7 the issue still persist
2.Install python3.6 version to fix the issue.
yum install python3 python3-devel python3-urllib3
3.Run the elastalert command
python3 -m elastalert.elastalert --config /root/elastalert/config.yaml --verbose --rule /root/elastalert/example_rules/example_frequency.yaml
4.If you received issue with the modules (ModuleNotFoundError: No module named 'pytz')
5.Install the modules as per the requirement.
pip3 install -r /root/elastalert/requirements.txt
6.Let's run the command "python3 -m elastalert.elastalert --config /root/elastalert/config.yaml --verbose --rule /root/elastalert/example_rules/example_frequency.yaml" and got error
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='elasticsearch.example.com', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known',))
7.Above error due to not valid hostname on config.yaml file. Edit the config.yaml file and change the hostname to server hostname at es.hosts field
Make sure you had an entry for the same on the /etc/hosts file.
8.Ok the issue got fixed and run the command "python3 -m elastalert.elastalert --config /root/elastalert/config.yaml --verbose --rule /root/elastalert/example_rules/example_frequency.yaml" and one more error
pkg_resources.DistributionNotFound: The 'jira>=2.0.0'
9.We need to install the jira by using below command
pip3 install jira==2.0.0
10.Now let's run the command "python3 -m elastalert.elastalert --config /root/elastalert/config.yaml --verbose --rule /root/elastalert/example_rules/example_frequency.yaml" and again another error OMG.
elasticsearch.exceptions.TransportError: TransportError(429, 'circuit_breaking_exception', '[parent] Data too large, data for [] would be [994793504/948.7mb], which is larger than the limit of [986061209/940.3mb], real usage: [994793056/948.7mb], new bytes reserved: [448/448b]')
11.You need to fix the same by changing the heap value on following /etc/elasticsearch/jvm.options
Xms-1g to Xms-2g
Xmx-1g to Xms-2g
and restart elasticsearch service "service elasticsearch restart"
12.Everything set again run the command "python3 -m elastalert.elastalert --config /root/elastalert/config.yaml --verbose --rule /root/elastalert/example_rules/example_frequency.yaml" and ended up receiving another error.
ERROR:root:Error finding recent pending alerts: NotFoundError(404, 'index_not_found_exception', 'no such index [elastalert_status]', elastalert_status, index_or_alias) {'query': {'bool': {'must': {'query_string': {'query': '!exists:aggregate_id AND alert_sent:false'}}, 'filter': {'range': {'alert_time': {'from': '2019-12-04T19:45:09.635478Z', 'to': '2019-12-06T19:45:09.635529Z'}}}}}, 'sort': {'alert_time': {'order': 'asc'}}}
13.Fix the issue by running the below command
elastalert-create-index
14.Finally everything done and run the below command
python3 -m elastalert.elastalert --config /root/elastalert/config.yaml --verbose --rule /root/elastalert/example_rules/example_frequency.yaml
Now cancelled the command and ran the same on background
python3 -m elastalert.elastalert --config /root/elastalert/config.yaml --verbose --rule /root/elastalert/example_rules/example_frequency.yaml &

Error creating virtualenv with Python3

I'm working on Linux Mint 17 and I'm trying to create a new virtualenv with Python3 like this:
python3.6 -m venv env
And this is the error that I get:
Error: Command '['/home/ric/myprojs/django-example-channels/env/bin/python3.6', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.
I've googled this error message but haven't managed to find anything too informative.
This is my pip version, in case it make any difference:
pip --version
pip 9.0.1 from /usr/local/lib/python2.7/dist-packages (python 2.7)
I've been using Python2 for some time, but I'm new to Python3. I don't know what I may be missing.
UPDATE 1:
Answering #cezar's question, when I type  which python3 this is what I get:
$ which python3
/usr/bin/python3
UPDATE 2:
Answering #Chłop Z Lasu:
$ virtualenv -p python3.6 env
Running virtualenv with interpreter /usr/bin/python3.6
Using base prefix '/usr'
New python executable in /home/ric/myprojs/django-example-channels/example_channels/env/bin/python3.6
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 2328, in <module>
main()
File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 713, in main
symlink=options.symlink)
File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 925, in create_environment
site_packages=site_packages, clear=clear, symlink=symlink))
File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 1231, in install_python
shutil.copyfile(executable, py_executable)
File "/usr/lib/python3.6/shutil.py", line 104, in copyfile
raise SameFileError("{!r} and {!r} are the same file".format(src, dst))

The error indicates that virtualenv is trying to make an environment in your python path. therefore, you have to specify your virtualenv destination
virtualenv -p python3.6 /path/to/yourenv

For venv and python3.6 installing venv one can use below to fix the issue
sudo apt install python3.6-venv

You gotta install the venv via code bellow:
sudo apt install python3.6-venv

Expanding on the Answer Above:
The error indicates that virtualenv is trying to make an environment in your python path. therefore, you have to specify your virtualenv destination
Since -p is a flag equivalent to --python, you can also use.
$ virtualenv --python=python3.6 /path/to/yourenv
Or
$ virtualenv -p python3.6 /path/to/yourenv

Pydoop Installation on CentOS 6

I am trying to install Pydoop (https://crs4.github.io/pydoop/installation.html) on CentOS 6 (Hortonworks HDP physical cluster, 4 nodes). I have Python 3.4 and Python 2.6 installed on my master. When I install it using Python 3 it shows syntax error like below:
[root#abc]# python3 setup.py build
File "setup.py", line 45
print 'using setuptools version', setuptools.__version__
^
SyntaxError: Missing parentheses in call to 'print'
When I install it using Python 2 it shows following error
[root#abc]# python setup.py build
using setuptools version 35.0.2
Traceback (most recent call last):
File "setup.py", line 65, in <module>
import pydoop
File "/root/pydoop/pydoop/__init__.py", line 42, in <module>
_HADOOP_INFO = _PATH_FINDER.find() # fill the cache ASAP
File "/root/pydoop/pydoop/hadoop_utils.py", line 624, in find
info[a] = getattr(self, a)()
File "/root/pydoop/pydoop/hadoop_utils.py", line 433, in hadoop_home
_hadoop_home_from_version_cmd() or
File "/root/pydoop/pydoop/hadoop_utils.py", line 394, in
_hadoop_home_from_version_cmd
output = sp.check_output([hadoop_exec, 'version'])
AttributeError: 'module' object has no attribute 'check_output'
I don't want to upgrade default python from 2.6 to 2.7 as it may break yum which uses python2.6 and upgrading it may cause problems. Any suggestions please?

You can install Python 2.7 and pip on CentOS 6 without breaking yum:
# yum update
# yum install centos-release-SCL epel-release
# yum install python27 python27-devel
# echo ". /opt/rh/python27/enable" >/etc/profile.d/python.sh
# source /etc/profile
# curl -o get-pip.py https://bootstrap.pypa.io/get-pip.py
# python get-pip.py
# python --version
Python 2.7.13
# pip --version
pip 9.0.1 from /opt/rh/python27/root/usr/lib/python2.7/site-packages (python 2.7)

error occurs when installing pip

I don't know what to do next to fix? All the things are in red!
this is what shows when I conducted the final step using : python get-pip.py
I just follow the instructions getting from
Exception:
Traceback (most recent call last):
.....(too long)
conn = self.get_connection(request.url, proxies)
File "/tmp/tmpGK7Hjj/pip.zip/pip/_vendor/requests/adapters.py", line 290, in get_connection
proxy_manager = self.proxy_manager_for(proxy)
File "/tmp/tmpGK7Hjj/pip.zip/pip/_vendor/requests/adapters.py", line 184, in proxy_manager_for
**proxy_kwargs
File "/tmp/tmpGK7Hjj/pip.zip/pip/_vendor/requests/packages/urllib3/contrib/socks.py", line 154, in __init__
"Unable to determine SOCKS version from %s" % proxy_url
ValueError: Unable to determine SOCKS version from socks://10.77.8.70:8080/
This is what shows when enter
whereis python?
python: /usr/bin/python2.7 /usr/bin/python3.5m /usr/bin/python /usr/bin /python2.7-config /usr/bin/python3.5 /usr/lib/python2.7 /usr/lib/python3.5 /etc/python2.7 /etc/python /etc/python3.5 /usr/local/lib/python2.7 /usr/local/lib/python3.5 /usr/include/python2.7 /usr/share/python /usr/share/man/man1/python.1.gz
whereis pip?
pip: /usr/local/bin/pip3.5 /usr/local/bin/pip /usr/local/bin/pip2.7
Thank you very much!

To intsall pip for Python2 use the following command:
sudo apt-get install python-pip
For Python3 use
sudo apt-get install python3-pip

'EntryPoint' object has no attribute 'resolve' when using Google Compute Engine

I have an issue related to Cryptography package in Python. Can you please help in resolving these, if possible ? (tried a lot, but couldnt figure out the exact solution)
The python code which initiates this error:
print("Salt: %s" % salt)
server_key = pyelliptic.ECC(curve="prime256v1") # ----->> Line2
print("Server_key: %s" % server_key) # ----->> Line3
server_key_id = base64.urlsafe_b64encode(server_key.get_pubkey()[1:])
http_ece.keys[server_key_id] = server_key
http_ece.labels[server_key_id] = "P-256"
encrypted = http_ece.encrypt(data, salt=salt, keyid=server_key_id,
dh=self.receiver_key, authSecret=self.auth_key) # ----->> Line8
Value of "Salt" is getting displayed in 100% of the cases.
If Line3 gets executed successfully, I see the the following EntryPoint Error because of http_ece.encrypt() call (Line8):
AttributeError("'EntryPoint' object has no attribute 'resolve'",)
(Ref. File Link: https://github.com/martinthomson/encrypted-content-encoding/blob/master/python/http_ece/init.py#L128 )
Requirements.txt(partial):
cryptography==1.5
pyelliptic==1.5.7
pyOpenSSL==16.1.0
On Running the command: sudo pip freeze --all |grep setuptools, I get:
setuptools==27.1.2
Please let me know if any more detail is required.
This problem seems to be basically due to some Old/Incompatible packages(related to PyElliptic, Cryptography, PyOpenSSL and/or setuptools) installed on the VM. For Reference: https://github.com/pyca/cryptography/issues/3149
Can someone please suggest a good solution to resolve this issue completely ?
Thanks,

The issue referenced in c66303382 has this traceback (you never gave your traceback so I have to assume yours ends the same way):
File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/__init__.py", line 35, in default_backend
_default_backend = MultiBackend(_available_backends())
File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/__init__.py", line 22, in _available_backends
"cryptography.backends"
The full line that triggers the error looks like this:
_available_backends_list = [
ep.resolve()
for ep in pkg_resources.iter_entry_points(
"cryptography.backends"
)
]
Searching the repository for EntryPoint definition, then blaming pkg_resources/__init__.py where it is reveals that pkg_resources.EntryPoint.resolve() was added in commit 92a553d3adeb431cdf92b136ac9ccc3f2ef98bf1 (2015-01-05) that went into setuptools v11.3.
Thus you'll see this error if you use an older version.

Ran Following Commands from the project path /opt/projects/myproject-google/myproject and it resolved the Attribute EntryPoint Error Issue:
(Assuming project virtual env path as: /opt/projects/myproject-google/venv)
Command: (from path: /opt/projects/myproject-google/myproject)
export PYTHONPATH= # [Blank]
sudo pip install --upgrade virtualenv setuptools
sudo rm -rf ../venv
sudo virtualenv ../venv
source ../venv/bin/activate
sudo pip install --upgrade -r requirements.txt
deactivate
Running the above commands upgraded the virtual environment & the setuptools version inside the virtual Env. located at path: /opt/projects/myproject-google/venv/lib/python2.7/site-packages. To test if setuptools have upgraded successfully, try some of these commands:
Command: sudo virtualenv --version
Output: 15.0.3
Command: echo $PYTHONPATH
Output: [blank]
Command: python -c 'import pkg_resources; print(pkg_resources.__file__)'
Output: ~/.local/lib/python2.7/site-packages/pkg_resources/__init__.pyc
Command: python -c 'import sys; print(sys.path)'
Output: ['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '~/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/dist-packages', '/opt/projects/myproject-google/myproject', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PILcompat']
Command: ls /opt/projects/myproject-google/venv/lib/python2.7/site-packages
Output:
easy_install.py pip pkg_resources setuptools-27.2.0.dist-info wheel-0.30.0a0.dist-info
easy_install.pyc pip-8.1.2.dist-info setuptools wheel
Command: python -c 'from cryptography.hazmat.backends import default_backend; print(default_backend())'
Output: <cryptography.hazmat.backends.multibackend.MultiBackend object at 0x7ff83a838d50>
Command /opt/projects/myproject-google/venv/bin/python -c 'from cryptography.hazmat.backends import default_backend; print(default_backend())'
Output
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named cryptography.hazmat.backends
Command: /opt/projects/myproject-google/venv/bin/python -c "import pkg_resources; print(pkg_resources.__file__)"
Output: /opt/projects/myproject-google/venv/local/lib/python2.7/site-packages/pkg_resources/__init__.pyc
Ref Link: https://github.com/pyca/cryptography/issues/3149
These Steps resolved the Attribute EntryPoint Issue completely with an updated version of cryptography package & the setuptools.
Update As on 15 September 2016, The Cryptography Team has again added the workaround for supporting old packages too.
(Ref. Link: https://github.com/pyca/cryptography/issues/3150 )