We Keep Coding

How to integrate and mocking Redshift and S3 locally using redshift-fake-driver

I would like to build redshift and s3 locally, and then use them for tasks that may run from airflow, tools ... to reduce CI/CD code when have to deploy them to dev, also want to avoid conflict about resources, files, ...
Currently can use LocalStack's S3, but for Redshift, jusr looking for solutions but only get combination using redshift-fake-driver along with package JayDeBeApi in python, but it seems not working properly
import jpype # JPype1==1.4.1
import jaydebeapi # JayDeBeApi==1.2.3
jars = "/Users/trancongminh/Downloads/jars/*"
jpype.startJVM(classpath=jars)
driverName = "jp.ne.opt.redshiftfake.postgres.FakePostgresqlDriver"
print(jpype.JClass(driverName))
# as I spin up a docker container for postgresQL
connectionString = "jdbc:postgresqlredshift://localhost:5432/docker"
uid = "docker"
pwd = "docker"
driverFileName = "/Users/trancongminh/Downloads/jars/redshift-fake-driver_2.12-1.0.15.jar"
conn = jaydebeapi.connect(
jclassname=driverName,
url=connectionString,
driver_args={'user': uid, 'password': pwd},
jars=driverFileName
)
curs = conn.cursor()
curs.execute("SELECT * FROM pg_catalog.pg_tables limit 10;")
curs.fetchall()
curs.execute("copy db_table_name_v2 from 'http://localhost:4566/events-streaming/traveller/v2/ym_202210/d_04/hm_131901.parquet' CREDENTIALS 'aws_access_key_id=test;aws_secret_access_key=test' ")
But get errors like No such file or directory, or smth like this
Traceback (most recent call last):
File "FakeConnection.scala", line 31, in jp.ne.opt.redshiftfake.FakeConnection.prepareStatement
Exception: Java Exception
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/trancongminh/Pelago/pelago-ds-env/lib/python3.9/site-packages/jaydebeapi/__init__.py", line 531, in execute
self._prep = self._connection.jconn.prepareStatement(operation)
java.lang.NoSuchMethodError: java.lang.NoSuchMethodError: 'void scala.util.parsing.combinator.Parsers.$init$(scala.util.parsing.combinator.Parsers)'
or may be like this:
Traceback (most recent call last):
File "FakePreparedStatement.scala", line 138, in jp.ne.opt.redshiftfake.FakePreparedStatement$FakeAsIsPreparedStatement.execute
Exception: Java Exception
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/trancongminh/Pelago/pelago-ds-env/lib/python3.9/site-packages/jaydebeapi/__init__.py", line 534, in execute
is_rs = self._prep.execute()
org.postgresql.util.PSQLException: org.postgresql.util.PSQLException: ERROR: could not open file "s3://events-streaming/traveller/v2/ym_202210/d_04/hm_131901.parquet" for reading: No such file or directory
Hint: COPY FROM instructs the PostgreSQL server process to read a file. You may want a client-side facility such as psql's \copy.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/trancongminh/Pelago/pelago-ds-env/lib/python3.9/site-packages/jaydebeapi/__init__.py", line 536, in execute
_handle_sql_exception()
File "/Users/trancongminh/Pelago/pelago-ds-env/lib/python3.9/site-packages/jaydebeapi/__init__.py", line 165, in _handle_sql_exception_jpype
reraise(exc_type, exc_info[1], exc_info[2])
File "/Users/trancongminh/Pelago/pelago-ds-env/lib/python3.9/site-packages/jaydebeapi/__init__.py", line 57, in reraise
raise value.with_traceback(tb)
File "/Users/trancongminh/Pelago/pelago-ds-env/lib/python3.9/site-packages/jaydebeapi/__init__.py", line 534, in execute
is_rs = self._prep.execute()
jaydebeapi.DatabaseError: org.postgresql.util.PSQLException: ERROR: could not open file "s3://events-streaming/traveller/v2/ym_202210/d_04/hm_131901.parquet" for reading: No such file or directory
Hint: COPY FROM instructs the PostgreSQL server process to read a file. You may want a client-side facility such as psql's \copy
Anyy body has exp with this pattern just help, thanks
Solutions or keywords that helpful for further investigation

How to fix Octave symbolic package error in Ubuntu 20.04?

I have installed the symbolic package. There are some python related errors in the command window of Octave when I try to use symbolic.Below is the output:
> pkg load symbolic
>> syms a
Symbolic pkg v2.9.0: Traceback (most recent call last):
File "<stdin>", line 28, in <module>
AttributeError: '_PrintFunction' object has no attribute '__globals__'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 12, in octoutput_drv
File "<stdin>", line 54, in octoutput
File "<stdin>", line 55, in octoutput
AttributeError: module 'sympy' has no attribute 'compatibility'
Closing the Python communications link.
error: Python exception: AttributeError: '_PrintFunction' object has no attribute '__globals__'
occurred in python_header import block.
Try "sympref reset" and repeat your command?
(consider filing an issue at https://github.com/cbm755/octsympy/issues)
error: called from
pycall_sympy__ at line 191 column 5
valid_sym_assumptions at line 38 column 10
assumptions at line 82 column 7
syms at line 97 column 13
>> Opening in existing browser session.
Any idea how i can troubleshoot this?
Thanks.

The error is a known bug (https://github.com/cbm755/octsympy/issues/1035)
You need to use an older version of sympy sunch as version 1.5.1

Using language check module for python - "can't find Java" error

I'm planning to use the Language-check module for python 3 in my poem making code. I was just testing it with
import language_tool
lang_tool = language_tool.LanguageTool("en-US")
text = "A sentence with a error in the Hitchhiker’s Guide tot he Galaxy"
matches = lang_tool.check(text)
print(len(matches))
but it gave me this error.
Traceback (most recent call last):
File "/home/runner/.site-packages/language_tool/__init__.py", line 548, in get_server_cmd
cmd = cache["server_cmd"]
KeyError: 'server_cmd'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/runner/.site-packages/language_tool/__init__.py", line 559, in get_jar_info
java_path, jar_path = cache["jar_info"]
KeyError: 'jar_info'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 10, in <module>
lang_tool = language_tool.LanguageTool("en-US")
File "/home/runner/.site-packages/language_tool/__init__.py", line 188, in __init__
self._start_server_on_free_port()
File "/home/runner/.site-packages/language_tool/__init__.py", line 323, in _start_server_on_free_port
cls._start_server()
File "/home/runner/.site-packages/language_tool/__init__.py", line 335, in _start_server
server_cmd = get_server_cmd(cls._port)
File "/home/runner/.site-packages/language_tool/__init__.py", line 550, in get_server_cmd
java_path, jar_path = get_jar_info()
File "/home/runner/.site-packages/language_tool/__init__.py", line 563, in get_jar_info
raise JavaError("can’t find Java")
language_tool.JavaError: can’t find Java
I've looked at some similar questions and I know I probably need to put something in 'Path' but I don't know what specifically to enter. Thanks for your help!
Also I'm on doing this on repl.it, not IDLE.

Note: I'm adding this as an actual answer because the question was resolved in the comments.
Your question says:
Also I'm on doing this on repl.it, not IDLE.
That means that you need to install this module and all its prerequisites into your repl.it environment, not into your local machine.
On the BitBucket link you provided, under "Prerequisites", the page says:
LanguageTool requires Java 6 or later.
So you will need to ensure Java is installed into your repl.it environment somehow. I don't know whether that is possible or not. Alternatively, you can simply install this package and all its prerequisites locally instead.

wxWidgets AttributeError on editable listctrl

I have copied an example of an editable ListCtrl from http://www.blog.pythonlibrary.org/2011/01/04/wxpython-wx-listctrl-tips-and-tricks/ . I have changed InsertStringItem and SetStringItem to InsertItem and SetItem. When I try to run this I get the following error
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\wx\lib\mixins\listctrl.py", line 560, in OnLeftDown
self.OpenEditor(col, row)
File "C:\Python27\lib\site-packages\wx\lib\mixins\listctrl.py", line 568, in OpenEditor
evt.Index = row
AttributeError: can't set attribute
I am using Python 2.7.10 on Windows 10.
How do I fix this?

Memory error in iGraph Python when adding large number of edges

Regarding this thread : http://goo.gl/uEyFua
I am experiencing a quite similar issue with the following code, trying to import a large Graph :
for (edge_id) in cursorSQL:
L.add((edge_id[2], str(edge_id[1])))
g = igraph.Graph.TupleList(L)
I get the following errors :
Traceback (most recent call last):
File "C.py", line 707, in __getitem__ return self._ids[item]
KeyError: '184840900'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
g = igraph.Graph.TupleList(L)
File "C:\Python34\lib\site-packages\igraph\__init__.py", line 2473, in TupleList
edge_list.append((idgen[item[0]], idgen[item[1]]))
File "C:\Python34\lib\site-packages\igraph\datatypes.py", line 709, in __getitem__
self._ids[item] = next(self._generator)
MemoryError
Just to make it clear. This code works perfectly until number of edges is too large (~4millions.)
Thanks.

Switched to python-64bits, and problem is now solved.
Problem was the 2GB-per-process limit in win7 for python-32bits.