Python 3.5, psycopg2 2.7.4, connects to postgresql database and gets error psycopg2.DatabaseError: SSL error: bad length
What does this error mean - some SSL certificate is not valid?
Here's traceback:
Traceback (most recent call last):
File "src/db_importer.py", line 278, in <module>
main()
File "src/db_importer.py", line 255, in main
run_process(args, db_params)
File "src/db_importer.py", line 198, in run_process
import_csv(cur, schema, table_name, args.raport_data)
File "src/db_importer.py", line 130, in import_csv
cur.copy_from(f, '{}.{}'.format(schema, table_name), sep=';', columns=cols)
psycopg2.DatabaseError: SSL error: bad length
After freeing some space the script worked. So tt was lack of space that caused this error.
try 'vacuum full analyze your table' on postgres level, it worked for me. Deadtulips was the issue and 'After freeing some space' might have been autovaccum running
Related
I would like to build redshift and s3 locally, and then use them for tasks that may run from airflow, tools ... to reduce CI/CD code when have to deploy them to dev, also want to avoid conflict about resources, files, ...
Currently can use LocalStack's S3, but for Redshift, jusr looking for solutions but only get combination using redshift-fake-driver along with package JayDeBeApi in python, but it seems not working properly
import jpype # JPype1==1.4.1
import jaydebeapi # JayDeBeApi==1.2.3
jars = "/Users/trancongminh/Downloads/jars/*"
jpype.startJVM(classpath=jars)
driverName = "jp.ne.opt.redshiftfake.postgres.FakePostgresqlDriver"
print(jpype.JClass(driverName))
# as I spin up a docker container for postgresQL
connectionString = "jdbc:postgresqlredshift://localhost:5432/docker"
uid = "docker"
pwd = "docker"
driverFileName = "/Users/trancongminh/Downloads/jars/redshift-fake-driver_2.12-1.0.15.jar"
conn = jaydebeapi.connect(
jclassname=driverName,
url=connectionString,
driver_args={'user': uid, 'password': pwd},
jars=driverFileName
)
curs = conn.cursor()
curs.execute("SELECT * FROM pg_catalog.pg_tables limit 10;")
curs.fetchall()
curs.execute("copy db_table_name_v2 from 'http://localhost:4566/events-streaming/traveller/v2/ym_202210/d_04/hm_131901.parquet' CREDENTIALS 'aws_access_key_id=test;aws_secret_access_key=test' ")
But get errors like No such file or directory, or smth like this
Traceback (most recent call last):
File "FakeConnection.scala", line 31, in jp.ne.opt.redshiftfake.FakeConnection.prepareStatement
Exception: Java Exception
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/trancongminh/Pelago/pelago-ds-env/lib/python3.9/site-packages/jaydebeapi/__init__.py", line 531, in execute
self._prep = self._connection.jconn.prepareStatement(operation)
java.lang.NoSuchMethodError: java.lang.NoSuchMethodError: 'void scala.util.parsing.combinator.Parsers.$init$(scala.util.parsing.combinator.Parsers)'
or may be like this:
Traceback (most recent call last):
File "FakePreparedStatement.scala", line 138, in jp.ne.opt.redshiftfake.FakePreparedStatement$FakeAsIsPreparedStatement.execute
Exception: Java Exception
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/trancongminh/Pelago/pelago-ds-env/lib/python3.9/site-packages/jaydebeapi/__init__.py", line 534, in execute
is_rs = self._prep.execute()
org.postgresql.util.PSQLException: org.postgresql.util.PSQLException: ERROR: could not open file "s3://events-streaming/traveller/v2/ym_202210/d_04/hm_131901.parquet" for reading: No such file or directory
Hint: COPY FROM instructs the PostgreSQL server process to read a file. You may want a client-side facility such as psql's \copy.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/trancongminh/Pelago/pelago-ds-env/lib/python3.9/site-packages/jaydebeapi/__init__.py", line 536, in execute
_handle_sql_exception()
File "/Users/trancongminh/Pelago/pelago-ds-env/lib/python3.9/site-packages/jaydebeapi/__init__.py", line 165, in _handle_sql_exception_jpype
reraise(exc_type, exc_info[1], exc_info[2])
File "/Users/trancongminh/Pelago/pelago-ds-env/lib/python3.9/site-packages/jaydebeapi/__init__.py", line 57, in reraise
raise value.with_traceback(tb)
File "/Users/trancongminh/Pelago/pelago-ds-env/lib/python3.9/site-packages/jaydebeapi/__init__.py", line 534, in execute
is_rs = self._prep.execute()
jaydebeapi.DatabaseError: org.postgresql.util.PSQLException: ERROR: could not open file "s3://events-streaming/traveller/v2/ym_202210/d_04/hm_131901.parquet" for reading: No such file or directory
Hint: COPY FROM instructs the PostgreSQL server process to read a file. You may want a client-side facility such as psql's \copy
Anyy body has exp with this pattern just help, thanks
Solutions or keywords that helpful for further investigation
I'm trying to connect a Tektronix TDS 1012B oscilloscope to my PC using Pyvisa.
I'm using Kubuntu 20.04, Python version 3.9.7 and Pyvisa version 1.11.3.
I do the following :
>>> import pyvisa
>>> rm = pyvisa.ResourceManager()
>>> rm.list_resources()
('USB0::1689::870::C021027::0::INSTR')
>>> scope = rm.open_resource('USB0::1689::870::C021027::0::INSTR')
>>> scope.query("*IDN?")
But I get this error message:
Traceback (most recent call last):
File "/home/pierre/anaconda3/lib/python3.9/site-packages/pyvisa_py/protocols/usbtmc.py", line 256, in write
return self.usb_send_ep.write(data)
File "/home/pierre/anaconda3/lib/python3.9/site-packages/usb/core.py", line 408, in write
return self.device.write(self, data, timeout)
File "/home/pierre/anaconda3/lib/python3.9/site-packages/usb/core.py", line 989, in write
return fn(
File "/home/pierre/anaconda3/lib/python3.9/site-packages/usb/backend/libusb1.py", line 837, in bulk_write
return self.__write(self.lib.libusb_bulk_transfer,
File "/home/pierre/anaconda3/lib/python3.9/site-packages/usb/backend/libusb1.py", line 938, in __write
_check(retval)
File "/home/pierre/anaconda3/lib/python3.9/site-packages/usb/backend/libusb1.py", line 602, in _check
raise USBTimeoutError(_strerror(ret), ret, _libusb_errno[ret])
usb.core.USBTimeoutError: [Errno 110] Operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/pierre/anaconda3/lib/python3.9/site-packages/pyvisa/resources/messagebased.py", line 638, in query
self.write(message)
File "/home/pierre/anaconda3/lib/python3.9/site-packages/pyvisa/resources/messagebased.py", line 197, in write
count = self.write_raw(message.encode(enco))
File "/home/pierre/anaconda3/lib/python3.9/site-packages/pyvisa/resources/messagebased.py", line 157, in write_raw
return self.visalib.write(self.session, message)[0]
File "/home/pierre/anaconda3/lib/python3.9/site-packages/pyvisa_py/highlevel.py", line 543, in write
written, status_code = self.sessions[session].write(data)
File "/home/pierre/anaconda3/lib/python3.9/site-packages/pyvisa_py/usb.py", line 179, in write
count = self.interface.write(data)
File "/home/pierre/anaconda3/lib/python3.9/site-packages/pyvisa_py/protocols/usbtmc.py", line 436, in write
bytes_sent += raw_write(data)
File "/home/pierre/anaconda3/lib/python3.9/site-packages/pyvisa_py/protocols/usbtmc.py", line 258, in write
raise ValueError(str(e))
ValueError: [Errno 110] Operation timed out
When I use another type of oscilloscope this very simple code works fine. And when I use Pyvisa under Windows instead of Kubuntu, it also works.
Can someone help me fix this annoying issue?
Check out this link to a post from Tek in Tek's forums. They suggest using the pyvisa-py native Python VISA backend. This command instructs to use the pyvisa-py backend
rm = visa.ResourceManager('#py')
and you install by
pip install pyvisa-py
I don't have an Ubuntu machine at my fingertips so I don't know if there are packages available for installing from apt.
Connecting my TBS 1064 in SUSE has problems due to write permissions. This results in an time-out on requests like ask(). It is likely due to udev/rules, although I did not figure out what to put exactly to make it always work. For now one has to put the write permission (admin) for every USB reconnect, but then it is working.
Other devices do not have this problem and it is only a linux issue.
I am trying to use pandas, sqlalchemy and pypyodbc in order to write my dataframe (df2) to a table in my sql database. I have successfully connected to the database with pypyodbc. However, when I try to use the engine with to_sql it throws the error in the title. I think I'm missing something in the urllib section when the engine creation string is parsed.
Here is my code snippet:
connStr = "Driver=SQL Server Native Client 10.0; Server=Sname-L\SQLEXPRESS;database=DBname;trusted_connection=yes;"
connEncodedStr = urllib.parse.quote_plus(connStr)
engine = sqlalchemy.create_engine("mssql+pyodbc:///?odbc_connect={}".format(connEncodedStr),module=pypyodbc,echo=True)
df2.to_sql('db_table2', engine, if_exists='replace')
print(engine)
result:
Engine(mssql+pyodbc:///?odbc_connect=Driver=SQL Server Native Client 10.0; Server=Sname-L\SQLEXPRESS;database=DBname;trusted_connection=yes;)
Here is the traceback:
Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Continuum\Anaconda3\lib\site-packages\s
qlalchemy\pool.py", line 1122, in _do_get
return self._pool.get(wait, self._timeout)
File "C:\Users\user\AppData\Local\Continuum\Anaconda3\lib\site-packages\s
qlalchemy\util\queue.py", line 145, in get
raise Empty
sqlalchemy.util.queue.Empty
However, this also happened:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "noaadatamanip_datafram.py", line 50, in <module>
df2.to_sql('db_table2', engine, if_exists='replace')
File "C:\Users\user\AppData\Local\Continuum\Anaconda3\lib\site-packages\p
andas\core\generic.py", line 1201, in to_sql
chunksize=chunksize, dtype=dtype)
The above code went on for a lot of lines and returned the error in the title. Let me know if posting the whole thing will be helpful.
Sorry in advance, my error handling skills are very low.
At first I tried:
import pymongo
MONGOHQ_URL = "mongodb://username:password#kahana.mongohq.com:10025/dbname"
conn = pymongo.MongoClient(MONGOHQ_URL)
but apparently it failed and it throws following error:
Traceback (most recent call last):
File "bot.py", line 95, in <module>
conn = pymongo.Connection(MONGOHQ_URL)
File "/var/lib/openshift/53abb500028e/python/virtenv/lib/python2.7/site-packages/pymongo/connection.py", line 236, in __init__
max_pool_size, document_class, tz_aware, _connect, **kwargs)
File "/var/lib/openshift/53abb500028e/python/virtenv/lib/python2.7/site-packages/pymongo/mongo_client.py", line 369, in __init__
raise ConnectionFailure(str(e))
pymongo.errors.ConnectionFailure: [Errno 13] Permission denied
They have an example, but it's in Ruby and I have no idea. If I am not wrong, I guess the connection is happening in this script.
Can anyone makeout from this ruby code and help me so that I can use in my python script with pymongo? I have already set environment variable MONGO_URL
I'm getting the error:
Exception: imported schema (http://www.w3.org/2001/XMLSchema) at
(http://www.w3.org/2001/XMLSchema.x sd), failed
when passing a Doctor (constructed with ImportDoctor) to the suds Client constructor.
I'm working on two Windows machines, both of them got the same version of suds installed, but only one of them rises the error above.
Could someone guide me here to know why this error rises?, so I can figure out what's missing on the machine where it happens?.
Thanks in advance!!!.
UPDATE: I don't really know if this is important, but it's worth noting that my Windows machine that is rising the error is an Amazon Web Services instance. At my local machine everything's working well!.
UPDATE: Here's some code I ran at the python interpreter of the machine I mentioned. Here you can detail how the error is rising...
>>> from suds.client import Client
>>> from suds.xsd.doctor import ImportDoctor, Import
>>> missing_import = Import("http://www.w3.org/2001/XMLSchema")
>>> missing_import.filter.add("http://tempuri.org/")
>>> doctor = ImportDoctor(missing_import)
>>> client = Client("http://etcfulfill.ebooks.com/Fulfillment.asmx?wsdl")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "suds\client.py", line 112, in __init__
self.wsdl = reader.open(url)
File "suds\reader.py", line 152, in open
d = self.fn(url, self.options)
File "suds\wsdl.py", line 159, in __init__
self.build_schema()
File "suds\wsdl.py", line 220, in build_schema
self.schema = container.load(self.options)
File "suds\xsd\schema.py", line 95, in load
child.dereference()
File "suds\xsd\schema.py", line 323, in dereference
midx, deps = x.dependencies()
File "suds\xsd\sxbasic.py", line 422, in dependencies
raise TypeNotFound(self.ref)
suds.TypeNotFound: Type not found: '(schema, http://www.w3.org/2001/XMLSchema, )'
>>> client = Client("http://etcfulfill.ebooks.com/Fulfillment.asmx?wsdl", doctor=doctor)
No handlers could be found for logger "suds.xsd.sxbasic"
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "suds\client.py", line 112, in __init__
self.wsdl = reader.open(url)
File "suds\reader.py", line 152, in open
d = self.fn(url, self.options)
File "suds\wsdl.py", line 159, in __init__
self.build_schema()
File "suds\wsdl.py", line 220, in build_schema
self.schema = container.load(self.options)
File "suds\xsd\schema.py", line 93, in load
child.open_imports(options)
File "suds\xsd\schema.py", line 305, in open_imports
imported = imp.open(options)
File "suds\xsd\sxbasic.py", line 542, in open
result = self.download(options)
File "suds\xsd\sxbasic.py", line 567, in download
raise Exception(msg)
Exception: imported schema (http://www.w3.org/2001/XMLSchema) at (http://www.w3.org/2001/XMLSchema.xsd), failed
UPDATE:
I realized that suds connections always open in TCP increasing ports, and if it reaches the maximum TCP port (65535) then it starts opening again from the minimum TCP port available, so there's no problem with this.
The problem shows up when using suds ImportDoctor, because it has to open a previous connection to the location where the import should be retrieved, and for some reason, if the system reaches the maximum TCP port count, then suds somehow assumes that there's no TCP port available to open the connection for obtaining the import, and in consecuence it throws the exception:
Exception: imported schema (http://www.w3.org/2001/XMLSchema) at (http://www.w3.org/2001/XMLSchema.xsd), failed
I repeat, this only happens if suds has to open this previous connection for obtaining the import. If ImportDoctor is not used, then suds has no problem if the TCP port count reaches its maximum, it just restarts at the minimum port available.
Does anyone has any clue on how to resolve this issue???. I'd really appreciate the help!!!.
I've figured out what the problem was. The schema that was missing from the WSDL I was trying to use with suds was:
http://www.w3.org/2001/XMLSchema
And the XSD file for this schema is at:
http://www.w3.org/2001/XMLSchema.xsd
So when I used suds ImportDoctor to add this schema import, sometimes the w3.org domain was denying my access (don't know why really) and that's why this error was rising:
Exception: imported schema (http://www.w3.org/2001/XMLSchema) at (http://www.w3.org/2001/XMLSchema.xsd), failed
What did I do to solve this problem?. I just downloaded this schema to my machine and used suds ImportDoctor to retrieve this import locally.
And that was it!!!. Confusing bug!!!. But SOLVED.