We Keep Coding

How to integrate and mocking Redshift and S3 locally using redshift-fake-driver

I would like to build redshift and s3 locally, and then use them for tasks that may run from airflow, tools ... to reduce CI/CD code when have to deploy them to dev, also want to avoid conflict about resources, files, ...
Currently can use LocalStack's S3, but for Redshift, jusr looking for solutions but only get combination using redshift-fake-driver along with package JayDeBeApi in python, but it seems not working properly
import jpype # JPype1==1.4.1
import jaydebeapi # JayDeBeApi==1.2.3
jars = "/Users/trancongminh/Downloads/jars/*"
jpype.startJVM(classpath=jars)
driverName = "jp.ne.opt.redshiftfake.postgres.FakePostgresqlDriver"
print(jpype.JClass(driverName))
# as I spin up a docker container for postgresQL
connectionString = "jdbc:postgresqlredshift://localhost:5432/docker"
uid = "docker"
pwd = "docker"
driverFileName = "/Users/trancongminh/Downloads/jars/redshift-fake-driver_2.12-1.0.15.jar"
conn = jaydebeapi.connect(
jclassname=driverName,
url=connectionString,
driver_args={'user': uid, 'password': pwd},
jars=driverFileName
)
curs = conn.cursor()
curs.execute("SELECT * FROM pg_catalog.pg_tables limit 10;")
curs.fetchall()
curs.execute("copy db_table_name_v2 from 'http://localhost:4566/events-streaming/traveller/v2/ym_202210/d_04/hm_131901.parquet' CREDENTIALS 'aws_access_key_id=test;aws_secret_access_key=test' ")
But get errors like No such file or directory, or smth like this
Traceback (most recent call last):
File "FakeConnection.scala", line 31, in jp.ne.opt.redshiftfake.FakeConnection.prepareStatement
Exception: Java Exception
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/trancongminh/Pelago/pelago-ds-env/lib/python3.9/site-packages/jaydebeapi/__init__.py", line 531, in execute
self._prep = self._connection.jconn.prepareStatement(operation)
java.lang.NoSuchMethodError: java.lang.NoSuchMethodError: 'void scala.util.parsing.combinator.Parsers.$init$(scala.util.parsing.combinator.Parsers)'
or may be like this:
Traceback (most recent call last):
File "FakePreparedStatement.scala", line 138, in jp.ne.opt.redshiftfake.FakePreparedStatement$FakeAsIsPreparedStatement.execute
Exception: Java Exception
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/trancongminh/Pelago/pelago-ds-env/lib/python3.9/site-packages/jaydebeapi/__init__.py", line 534, in execute
is_rs = self._prep.execute()
org.postgresql.util.PSQLException: org.postgresql.util.PSQLException: ERROR: could not open file "s3://events-streaming/traveller/v2/ym_202210/d_04/hm_131901.parquet" for reading: No such file or directory
Hint: COPY FROM instructs the PostgreSQL server process to read a file. You may want a client-side facility such as psql's \copy.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/trancongminh/Pelago/pelago-ds-env/lib/python3.9/site-packages/jaydebeapi/__init__.py", line 536, in execute
_handle_sql_exception()
File "/Users/trancongminh/Pelago/pelago-ds-env/lib/python3.9/site-packages/jaydebeapi/__init__.py", line 165, in _handle_sql_exception_jpype
reraise(exc_type, exc_info[1], exc_info[2])
File "/Users/trancongminh/Pelago/pelago-ds-env/lib/python3.9/site-packages/jaydebeapi/__init__.py", line 57, in reraise
raise value.with_traceback(tb)
File "/Users/trancongminh/Pelago/pelago-ds-env/lib/python3.9/site-packages/jaydebeapi/__init__.py", line 534, in execute
is_rs = self._prep.execute()
jaydebeapi.DatabaseError: org.postgresql.util.PSQLException: ERROR: could not open file "s3://events-streaming/traveller/v2/ym_202210/d_04/hm_131901.parquet" for reading: No such file or directory
Hint: COPY FROM instructs the PostgreSQL server process to read a file. You may want a client-side facility such as psql's \copy
Anyy body has exp with this pattern just help, thanks
Solutions or keywords that helpful for further investigation

Scope Tektronix TDS 1012 gives an error with pyvisa

I'm trying to connect a Tektronix TDS 1012B oscilloscope to my PC using Pyvisa.
I'm using Kubuntu 20.04, Python version 3.9.7 and Pyvisa version 1.11.3.
I do the following :
>>> import pyvisa
>>> rm = pyvisa.ResourceManager()
>>> rm.list_resources()
('USB0::1689::870::C021027::0::INSTR')
>>> scope = rm.open_resource('USB0::1689::870::C021027::0::INSTR')
>>> scope.query("*IDN?")
But I get this error message:
Traceback (most recent call last):
File "/home/pierre/anaconda3/lib/python3.9/site-packages/pyvisa_py/protocols/usbtmc.py", line 256, in write
return self.usb_send_ep.write(data)
File "/home/pierre/anaconda3/lib/python3.9/site-packages/usb/core.py", line 408, in write
return self.device.write(self, data, timeout)
File "/home/pierre/anaconda3/lib/python3.9/site-packages/usb/core.py", line 989, in write
return fn(
File "/home/pierre/anaconda3/lib/python3.9/site-packages/usb/backend/libusb1.py", line 837, in bulk_write
return self.__write(self.lib.libusb_bulk_transfer,
File "/home/pierre/anaconda3/lib/python3.9/site-packages/usb/backend/libusb1.py", line 938, in __write
_check(retval)
File "/home/pierre/anaconda3/lib/python3.9/site-packages/usb/backend/libusb1.py", line 602, in _check
raise USBTimeoutError(_strerror(ret), ret, _libusb_errno[ret])
usb.core.USBTimeoutError: [Errno 110] Operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/pierre/anaconda3/lib/python3.9/site-packages/pyvisa/resources/messagebased.py", line 638, in query
self.write(message)
File "/home/pierre/anaconda3/lib/python3.9/site-packages/pyvisa/resources/messagebased.py", line 197, in write
count = self.write_raw(message.encode(enco))
File "/home/pierre/anaconda3/lib/python3.9/site-packages/pyvisa/resources/messagebased.py", line 157, in write_raw
return self.visalib.write(self.session, message)[0]
File "/home/pierre/anaconda3/lib/python3.9/site-packages/pyvisa_py/highlevel.py", line 543, in write
written, status_code = self.sessions[session].write(data)
File "/home/pierre/anaconda3/lib/python3.9/site-packages/pyvisa_py/usb.py", line 179, in write
count = self.interface.write(data)
File "/home/pierre/anaconda3/lib/python3.9/site-packages/pyvisa_py/protocols/usbtmc.py", line 436, in write
bytes_sent += raw_write(data)
File "/home/pierre/anaconda3/lib/python3.9/site-packages/pyvisa_py/protocols/usbtmc.py", line 258, in write
raise ValueError(str(e))
ValueError: [Errno 110] Operation timed out
When I use another type of oscilloscope this very simple code works fine. And when I use Pyvisa under Windows instead of Kubuntu, it also works.
Can someone help me fix this annoying issue?

Check out this link to a post from Tek in Tek's forums. They suggest using the pyvisa-py native Python VISA backend. This command instructs to use the pyvisa-py backend
rm = visa.ResourceManager('#py')
and you install by
pip install pyvisa-py
I don't have an Ubuntu machine at my fingertips so I don't know if there are packages available for installing from apt.

Connecting my TBS 1064 in SUSE has problems due to write permissions. This results in an time-out on requests like ask(). It is likely due to udev/rules, although I did not figure out what to put exactly to make it always work. For now one has to put the write permission (admin) for every USB reconnect, but then it is working.
Other devices do not have this problem and it is only a linux issue.

Is there a way to use Torsocks directly on a Python script that runs Selenium?

I have a simple Python script that runs Selenium, and I have tried using Torsocks (as usual) simply like this: torsocks python script.py. However, it failed with this error:
Traceback (most recent call last):
File "script.py", line 21, in <module>
browser = webdriver.Firefox(options=options)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/firefox/webdriver.py", line 163, in __init__
log_path=service_log_path)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/firefox/service.py", line 47, in __init__
self, executable_path, port=port, log_file=log_file, env=env)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/common/service.py", line 42, in __init__
self.port = utils.free_port()
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/common/utils.py", line 37, in free_port
free_socket.listen(5)
PermissionError: [Errno 1] Operation not permitted
Is it actually possible to use Torsocks like this?
I realize that I could send the request with SOCKS5 proxy, but I wonder if it could run using Torsocks, and if not, it would be great to get an explanation.

SUDS Exception Imported Schema Failed

I'm getting the error:
Exception: imported schema (http://www.w3.org/2001/XMLSchema) at
(http://www.w3.org/2001/XMLSchema.x sd), failed
when passing a Doctor (constructed with ImportDoctor) to the suds Client constructor.
I'm working on two Windows machines, both of them got the same version of suds installed, but only one of them rises the error above.
Could someone guide me here to know why this error rises?, so I can figure out what's missing on the machine where it happens?.
Thanks in advance!!!.
UPDATE: I don't really know if this is important, but it's worth noting that my Windows machine that is rising the error is an Amazon Web Services instance. At my local machine everything's working well!.
UPDATE: Here's some code I ran at the python interpreter of the machine I mentioned. Here you can detail how the error is rising...
>>> from suds.client import Client
>>> from suds.xsd.doctor import ImportDoctor, Import
>>> missing_import = Import("http://www.w3.org/2001/XMLSchema")
>>> missing_import.filter.add("http://tempuri.org/")
>>> doctor = ImportDoctor(missing_import)
>>> client = Client("http://etcfulfill.ebooks.com/Fulfillment.asmx?wsdl")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "suds\client.py", line 112, in __init__
self.wsdl = reader.open(url)
File "suds\reader.py", line 152, in open
d = self.fn(url, self.options)
File "suds\wsdl.py", line 159, in __init__
self.build_schema()
File "suds\wsdl.py", line 220, in build_schema
self.schema = container.load(self.options)
File "suds\xsd\schema.py", line 95, in load
child.dereference()
File "suds\xsd\schema.py", line 323, in dereference
midx, deps = x.dependencies()
File "suds\xsd\sxbasic.py", line 422, in dependencies
raise TypeNotFound(self.ref)
suds.TypeNotFound: Type not found: '(schema, http://www.w3.org/2001/XMLSchema, )'
>>> client = Client("http://etcfulfill.ebooks.com/Fulfillment.asmx?wsdl", doctor=doctor)
No handlers could be found for logger "suds.xsd.sxbasic"
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "suds\client.py", line 112, in __init__
self.wsdl = reader.open(url)
File "suds\reader.py", line 152, in open
d = self.fn(url, self.options)
File "suds\wsdl.py", line 159, in __init__
self.build_schema()
File "suds\wsdl.py", line 220, in build_schema
self.schema = container.load(self.options)
File "suds\xsd\schema.py", line 93, in load
child.open_imports(options)
File "suds\xsd\schema.py", line 305, in open_imports
imported = imp.open(options)
File "suds\xsd\sxbasic.py", line 542, in open
result = self.download(options)
File "suds\xsd\sxbasic.py", line 567, in download
raise Exception(msg)
Exception: imported schema (http://www.w3.org/2001/XMLSchema) at (http://www.w3.org/2001/XMLSchema.xsd), failed
UPDATE:
I realized that suds connections always open in TCP increasing ports, and if it reaches the maximum TCP port (65535) then it starts opening again from the minimum TCP port available, so there's no problem with this.
The problem shows up when using suds ImportDoctor, because it has to open a previous connection to the location where the import should be retrieved, and for some reason, if the system reaches the maximum TCP port count, then suds somehow assumes that there's no TCP port available to open the connection for obtaining the import, and in consecuence it throws the exception:
Exception: imported schema (http://www.w3.org/2001/XMLSchema) at (http://www.w3.org/2001/XMLSchema.xsd), failed
I repeat, this only happens if suds has to open this previous connection for obtaining the import. If ImportDoctor is not used, then suds has no problem if the TCP port count reaches its maximum, it just restarts at the minimum port available.
Does anyone has any clue on how to resolve this issue???. I'd really appreciate the help!!!.

I've figured out what the problem was. The schema that was missing from the WSDL I was trying to use with suds was:
http://www.w3.org/2001/XMLSchema
And the XSD file for this schema is at:
http://www.w3.org/2001/XMLSchema.xsd
So when I used suds ImportDoctor to add this schema import, sometimes the w3.org domain was denying my access (don't know why really) and that's why this error was rising:
Exception: imported schema (http://www.w3.org/2001/XMLSchema) at (http://www.w3.org/2001/XMLSchema.xsd), failed
What did I do to solve this problem?. I just downloaded this schema to my machine and used suds ImportDoctor to retrieve this import locally.
And that was it!!!. Confusing bug!!!. But SOLVED.

Trouble with Sqlite on Python/Django

I'm trying to learn Django and following along the Django Book tutorial and I'm getting an error when I type these lines into the Python shell:
>>> from django.db import connection
>>> cursor = connection.cursor()
Here's the traceback I get:
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "C:\Python27\lib\site-packages\django\db\backends\__init__.py", line 306,
in cursor
cursor = self.make_debug_cursor(self._cursor())
File "C:\Python27\lib\site-packages\django\db\backends\sqlite3\base.py", line
288, in _cursor
self._sqlite_create_connection()
File "C:\Python27\lib\site-packages\django\db\backends\sqlite3\base.py", line
278, in _sqlite_create_connection
self.connection = Database.connect(**kwargs)
OperationalError: unable to open database file
Any ideas on how to solve this? On my settings.py I have DATABASES ENGINE set to: django.db.backends.sqlite3 and NAME set to: C:\Python27/PythonProjects/mysite. Thanks!

I find (in modern Django) that just giving a filename like database.sqlite for the filename as the name is the best option. Django should interpret that by sticking the new database in the root folder of the project, which is perfect for my needs.