We Keep Coding

Simple HTTPS python server to detect weather its connected through ca-cert or not?

first i create a ca-cert key pair with
openssl req -new -x509 -keyout private_key.pem -out public_cert.pem -days 365 -nodes
Generating a RSA private key
..+++++
.................................+++++
writing new private key to 'private_key.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:.
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:.
Organization Name (eg, company) [Internet Widgits Pty Ltd]:.
Organizational Unit Name (eg, section) []:.
Common Name (e.g. server FQDN or YOUR name) []:35.222.65.55 <----------------------- this ip should be server ip very important
Email Address []:
now i run a server with python code
# libraries needed:
from http.server import HTTPServer, SimpleHTTPRequestHandler
import ssl , socket
# address set
server_ip = '0.0.0.0'
server_port = 3389
# configuring HTTP -> HTTPS
httpd = HTTPServer((server_ip, server_port), SimpleHTTPRequestHandler)
httpd.socket = ssl.wrap_socket(httpd.socket, certfile='./public_cert.pem',keyfile='./private_key.pem', server_side=True)
httpd.serve_forever()
now this server can be connected for both secure ca-cert case and ingore-ca-cert server connections when using SSL case
that is
curl --cacert public_cert.pem --cert-type PEM https://35.222.65.55:3389
and
curl -k https://35.222.65.55:3389
will work
how to detect if the request is ingnore-ca-cert or not from server side ?

how to not allow insecure connection from server side ?
The server side has no control over the certificate validation done at the client side. The server has no knowledge if the client has verified the certificate or not. Nothing in the exchanged data indicates if the client is doing a curl -k or a curl without this option. Thus it is not possible to stop clients with broken or disabled validation from connecting to the server.

Try to reach docker container from python script (Interactive brokers)

On my synology I have this docker container running: https://registry.hub.docker.com/r/mgvazquez/ibgateway/
In the "manual" is says: "In this example you will launch the Interactive Brokers Gateway in paper mode listening on port 4001, and the VNC Server listening on port 5900"
So in the docker container I did the following port mapping:
Local port 32778 to container 5900 and local port 32776 to container 4001. My Synology Nas is 192.168.2.6.
When I connect from my local pc using vnc to 192.168.2.6:32778 it works perfectly.
Now, In my Python script I do:
from ib_insync import *
ib = IB()
# use this instead for IB Gateway
ib.connect('192.168.2.6:32776', 4002, clientId=1)
The 4002 is a socket port setting inside the gateway.
When I run the script I get "Getaddrinfo failed". Does not make sense to me.
What can be the issue here?

according to API document at https://ib-insync.readthedocs.io/api.html#module-ib_insync.ib
connect use following syntax:
connect(host='127.0.0.1', port=7497, clientId=1, timeout=4, readonly=False, account='')
host (str) – Host name or IP address.
port (int) – Port number.
clientId (int) – ID number to use for this client; must be unique per connection. Setting clientId=0 will automatically merge manual TWS trading with this client.
timeout (float) – If establishing the connection takes longer than timeout seconds then the asyncio.TimeoutError exception is raised. Set to 0 to disable timeout.
readonly (bool) – Set to True when API is in read-only mode.
account (str) – Main account to receive updates for.
so your code:
# use this instead for IB Gateway
ib.connect('192.168.2.6:32776', 4002, clientId=1)
should be changed to:
# use this instead for IB Gateway
ib.connect('192.168.2.6', 32776, clientId=1)

First, just for testing, try and use port 4001 directly:
ib.connect('192.168.2.6:32776', 4002, clientId=1)
Second, check your IB socat service is running, since it is that service which establishes two bidirectional byte streams and transfers data between 4001 and 4002:
echo "Starting Interactive Brokers Controller" | info
exec socat TCP-LISTEN:4001,fork TCP:127.0.0.1:4002 2>&1 | info
The Dockerfile registers it.
Try and add a mapping for port 4002.

Remote Postgresql is very slow

I run .py files using Django ORM, that connected to Postgresql server on another server.
Both servers working on Ubuntu 20.04
when i run the same file it takes the following time:
2-3 seconds on server with postgresql
8-12 seconds on another server.
When run .py file with more processes, it can take 20 seconds. If I run the same script at the same time on the postgresql server, it takes 2-3 seconds anyway
I tried:
Turn off firewall on both servers (sudo ufw disable)
Change postgres configs and then restart postgres server
Use pgBouncer
I checked internet speed on the servers and its normal
This is postgresql.conf
# Generated by PGConfig 2.0 beta
## http://pgconfig.org
# Memory Configuration
shared_buffers = 2GB
effective_cache_size = 6GB
work_mem = 41MB
maintenance_work_mem = 512MB
# Checkpoint Related Configuration
min_wal_size = 512MB
max_wal_size = 2GB
checkpoint_completion_target = 0.9
wal_buffers = 16MB
# Network Related Configuration
listen_addresses = '*'
max_connections = 1000
# Storage Configuration
random_page_cost = 1.1
effective_io_concurrency = 200
# Worker Processes
max_worker_processes = 8
max_parallel_workers_per_gather = 4
max_parallel_workers = 8
# Logging configuration for pgbadger
logging_collector = on
log_checkpoints = on
log_connections = on
log_disconnections = on
log_lock_waits = on
log_temp_files = 0
lc_messages = 'C'
# Adjust the minimum time to collect data
log_min_duration_statement = '10s'
log_autovacuum_min_duration = 0
# 'csvlog' format configuration
log_destination = 'csvlog'
In pg_hba i just insert 1 string
#
# Database administrative login by Unix domain socket
local all postgres peer
# TYPE DATABASE USER ADDRESS METHOD
# "local" is for Unix domain socket connections only
local all all peer
# IPv4 local connections:
host all all 127.0.0.1/32 md5
# IPv6 local connections:
host all all ::1/128 md5
# Allow replication connections from localhost, by a user with the
# replication privilege.
local replication all peer
host replication all 127.0.0.1/32 md5
host replication all ::1/128 md5
host all all all md5
Is it normal to have this speed or i can configure that?

Service and version displayed via nmap scan for simple python socket server

I've got a simple python socket server. Here's the code:
import socket
host = "0.0.0.0" # address to bind on.
port = 8081
def listen_serv():
try:
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((host,port))
s.listen(4)
...
messages back and forth between the server and client
...
if __name__ == "__main__":
while True:
listen_serv()
When I run the python server locally and then scan with nmap localhost i see the open port 8081 with the service blackice-icecap running on it. A quick google search revealed that this is a firewall service that uses the port 8081 for a service called ice-cap remote. If I change the port to 12000 for example, I get another service called cce4x.
A further scan with nmap localhost -sV returns the contents of the python script
1 service unrecognized despite returning data. If you know the service/version,
please submit the following fingerprint at https://nmap.org/cgi-bin/submit.cgi?new-service :
SF-Port8081-TCP:V=7.25BETA1%I=7%D=8/18%Time=57B58EE7%P=x86_64-pc-linux-gn
SF:u%r(NULL,1A4,"\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\
SF:*\*\*\*\*\*\*\*\*\*\*\*\*\n\*\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x
SF:20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\
SF:x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\*\n\*\x20\x20\x20\x20\x20\x
SF:20Welcome\x20to\x20ScapeX\x20Mail\x20Server\x20\x20\x20\x20\*\n\*\x20\x
SF:20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\
SF:x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20
SF:\x20\x20\*\n\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\
SF:*\*\*\*\*\*\*\*\*\*\*\*\nHere\x20is\x20a\x20quiz\x20to\x20test\x20your\
SF:x20knowledge\x20of\x20hacking\.\.\.\n\n\nAnswer\x20correctly\x20and\x20
SF:we\x20will\x20reward\x20you\x20with\x20a\x20shell\x20:-\)\x20\nQuestion
etc...
etc...
Is there a way I can customize the service and version descriptions that are displayed by nmap for my simple python server?

Found a solution by sending the following line as the first message from the server
c.send("HTTP/1.1 200 OK\r\nServer: Netscape-Enterprise/6.1\r\nDate: Fri, 19 Aug 2016 10:28:43 GMT\r\nContent-Type: text/html; charset=UTF-8\r\nConnection: close\r\nVary: Accept-Encoding\n\nContent-Length: 32092\r\n\n\n""")

Receiving multicast data on specific interface

tcmpdump can view all the multicast traffic to specific group and port on eth2, but my Python program cannot. The Python program, running on Ubuntu 12.04:
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
# Multicast port is 52122
sock.bind(('', 52122))
# Interface eth2 IP is 1.2.3.4, multicast group is 6.7.8.9
mreq = socket.inet_aton('6.7.8.9')+socket.inet_aton('1.2.3.4')
sock.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, mreq)
while True:
print '\nwaiting to receive message'
data, address = sock.recvfrom(1024)
print data
When I use another program to send a multicast packet to eth2, it works and prints the packet. But it fails to see all the current multicast traffic. If I run tcpdump on eth2 on the same port and group as the above program:
sudo tcpdump -i eth2 host 6.7.8.9 and port 52122
it sees both the packets I send from another program AND all the current multicast traffic. It's output looks likes this...
# Packet sent from my other program
09:52:51.952714 IP 1.2.3.4.57940 > 6.7.8.9.52122: UDP, length 19
# Packet send from the outside world
09:52:52.143339 IP 9.9.9.9.39295 > 6.7.8.9.52122: UDP, length 62
Why can't my program see the packets from the outside world? How can I modify it (or something else) to fix this?
Edit:
I should have mentioned, the interface this going over is not eth2 but eth2.200 a VLAN. (The local IP and the tcpdump commands are all run with eth2.200, I just changed that in this question to make it simpler.) Based on this answer that could be the problem?
Edit #2:
netstat -ng when the program is running shows eth2.200 subscribed to 224.0.0.1 and 6.7.8.9`.
tshark -i eth2.200 igmp shows three repeated 1.2.3.4 -> 6.7.8.9 IGMP 46 V2 Membership Report / Join group 6.7.8.9 when the program first starts. When the program process is killed, it shows 1.2.3.4 -> 224.0.0.2 IGMP 46 V2 Leave group 6.7.8.9. There is also an infrequent 1.2.3.1 -> 224.0.0.1 IGMP 60 V2 Membership Query, general, where 1.2.3.1 is 1.2.3.4's gateway.
Not sure if it will help, but the routing table looks like:
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 1.2.5.6 0.0.0.0 UG 0 0 0 eth1
1.2.3.0 0.0.0.0 255.255.255.240 U 0 0 0 eth2.200
Thank you!

Finally! Found this question on ServerFault that addresses the same thing. Basically the kernel was not forwarding on / was filtering out the packets because it thought the sourced address was spoofed.
Changed the settings in /etc/sysctl.conf to match:
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.all.rp_filter = 0
net.ipv4.ip_forward = 1
Rebooted and everything works.