We Keep Coding

Connect to a remote jupyter lab instance via proxy

I'm trying to connect to a remote jupyter notebook.
Server-side actions:
I simply use jupyter notebook command that produces the following output:
http://localhost:8888/?token=c7a760ee5387de7d6a1cb797a0685a116621f8b3b5a1a5ba
I know that there's password authentication procedure as well. I set up the jupyter_notebook_config.py file. I changed the c.NotebookApp.password and c.NotebookApp.open_browser variables.
Client-side actions:
Since I'm using Putty I can copy/paste the HTTP address (with token) to a local browser. I get the following page:
Even though I'm using the token from the link (step 1) and password (step 2) I get the Invalid Credentials error.
There's a proxy server between the local machine and the server but I don't know how to account for that fact.
Update 11.10.2019
I decided to tweak some parameters in .jupyter/jupyter_notebook_config.py file. By sacrificing security I did away with token authentication by setting c.NotebookApp.token=''. I still get the Invalid credentials message.

jupyter_notebook_config.py which resides in .jupyter directory (in the home directory of the user) is created with jupyter notebook --generate-config. All of the options are commented out. You need to find c.NotebookApp.ip variable that must not be commented out and set its value to the IP of the server the Jupyter is running on. If you want to connect to that instance of Jupyter from another machine you need to find localhost:port webpage with a browser of your choosing. My answer assumes that you've forwarded the port you are using.

python request remote web server via CLI through a ssh tunnel?

I installed a web server on a remote machine that can be only access through a ssh tunnel. Therefore, I have run with putty a ssh tunnel by specifying a port forwarding (in my case 8159). I have also configured the socks proxy on my browser to access to my remote webserver. Futhermore, with a curl command I can get the webpages if I add the following option --sock5-hostname localhost:8159.
Now, I would like to use python to request those webpages by passing through the ssh tunnel that I have configured with putty. I tried pysocks and proxy environment variables in my python code but it did not work. I would like to know if you have an idea to solve this problem.
Thank you in advance.

A Python script, a proxy and Microsoft Forefront - Auto-Authentication

Today I'm dealing with a Python3 script that has to do a http post request and send a mail.
The Python script is launched on a Windows PC that is in a corporate network protected by Forefront.
The user is logged with his secret credentials and can access to the internet through a proxy.
Like the other non-Microsoft applications (i.e. Chrome), I want my script to connect to the internet without prompt the user for his username and password.
How can I do this?

On Microsoft OSes, the authentication used is Kerberos, so you won't be able to use directly your ID + password.
I'm on Linux, so I can't test it directly but I think that you can create a proxy with fiddler which can negociate the authentication for you, and you can use this proxy with python.
Fiddler's Composer will automatically respond to authentication challenges (including the Negotiate protocol which wraps Kerberos) if you tick the Authentication box on the Options subtab, in the menus.

Scrapy with TOR (Windows)

I created a Scrapy project with several spiders to crawl some websites. Now I want to use TOR to:
Hide my ip from the crawled servers;
Associate my requests to different ips, simulating accesses from different users.
I have read some info about this, for example:
using tor with scrapy framework, How to connect to https site with Scrapy via Polipo over TOR?
The answers from these links weren't helpful to me. What are the steps that I should take to make Scrapy work properly with TOR?
EDIT 1:
Considering answer 1, I started by installing TOR. As I am using Windows I downloaded the TOR Expert Bundle (https://www.torproject.org/dist/torbrowser/5.0.1/tor-win32-0.2.6.10.zip) and read the chapter about how to configure TOR as a relay (https://www.torproject.org/docs/tor-doc-windows.html.en). Unfortunately there is little or any information about how to do it on Windows. If I unzip the downloaded archive and run the file Tor\Tor.exe nothing happens. However, I can see in the Task Manager that a new process is instantiated. I don't know what is the best way to proceed from here.

After a lot of research, I found a way to setup my Scrapy project to work with TOR on Windows OS:
Download TOR Expert Bundle for Windows (1) and unzip the files to a folder (ex. \tor-win32-0.2.6.10).
The recent TOR's versions for Windows don't come with a graphical user interface (2). It is probably possible to setup TOR only through config files and cmd commands but for me, the best option was to use Vidalia. Download it (3) and unzip the files to a folder (ex. vidalia-standalone-0.2.21-win32). Run "Start Vidalia.exe" and go to Settings. On the "General" tab, point Vidalia to TOR (\tor-win32-0.2.6.10\Tor\tor.exe).
Check on "Advanced" tab and "Tor Configuration File" section the torrc file. I have the next ports configured:
ControlPort 9151
SocksPort 9050
Click Start Tor on the Vidalia Control Panel UI. After some processing you should se on the status the message "Connected to the Tor network!".
Download Polipo proxy (4) and unzip the files to a folder (ex. polipo-1.1.0-win32). Read about this proxy on the link 5.
Edit the file config.sample and add the next lines to it (in the beginning of the file, for example):
socksParentProxy = "localhost:9050"
socksProxyType = socks5
diskCacheRoot = ""
Start Polipo through cmd. Go to the folder where you unzipped the files and enter the next command "polipo.exe -c config.sample".
Now you have Polipo and TOR up and running. Polipo will redirect any request to TOR through port 9050 with SOCKS protocol. Polipo will receive any HTTP request to redirect trough port 8123.
Now you can follow the rest of the tutorial "Torifying Scrapy Project On Ubuntu" (6). Continue in the step where the tutorial explains how to test the TOR/Polipo communications.
Links:
https://www.torproject.org/download/download.html.en
https://tor.stackexchange.com/questions/6496/tor-expert-bundle-on-windows-no-installation-instructions
https://people.torproject.org/~erinn/vidalia-standalone-bundles/
http://www.pps.univ-paris-diderot.fr/~jch/software/files/polipo/
http://www.pps.univ-paris-diderot.fr/~jch/software/polipo/tor.html
http://blog.privatenode.in/torifying-scrapy-project-on-ubuntu

A detailed step-by-step Explanation is here
http://blog.privatenode.in/torifying-scrapy-project-on-ubuntu/
The Basic steps there are:
Install Tor and Polipo (for linux this might require to add a repository).
Configure Polipo to talk with TOR using SOCK Connection (see above link).
Create a custom Middleware to use tor as a http proxy and to randomly change the scrapy user agent
to suppress depreciation warning from above example, write
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
instead of 'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,
What is your szenario? Have you thought about renting Proxy Servers?

How to clone a mercurial repository over an ssh connection initiated by fabric when http authorization is required?

I'm attempting to use fabric for the first time and I really like it so far, but at a certain point in my deployment script I want to clone a mercurial repository. When I get to that point I get an error:
err: abort: http authorization required
My repository requires http authorization and fabric doesn't prompt me for the user and password. I can get around this by changing my repository address from:
https://hostname/repository
to:
https://user:password#hostname/repository
But for various reasons I would prefer not to go this route.
Are there any other ways in which I could bypass this problem?

Here are four options with various security trade-offs and requiring various amounts of sys admin mojo:
With newer mercurial's you could put the password in the [auth] section of the local user's .hgrc file. The password will still be on disk in plaintext, but at least not in the URL
Or
You could locally set up a HTTP proxy that presents as no-auth locally and does the auth for you when communicating with remote.
Or
Of you're able to alter configuration on the hosting server you could set it (Apache?) to not require a user/pass when accessed from localhost, and then use a SSH tunnel to make the local machine look like it's coming from localhost when it access the server:
ssh -L 8080:localhost:80 user#hostname # run in background and leave running
and then have fabric connect to http://localhost:8080/repository
Or
Newer mercurial's support client side certificates for authentication, so you could configure your Apache to honor those as authorization/authentcation and then tweak your local hg to provide the certificate.

Depending on your fabfile, you might be able to reframe the problem. Instead of doing a hg clone on the remote system you could do your mercurial commands on your local system, and then ship the artifact you've constructed across with fabric.
Specifically, you could clone the mercurial repository by using fabric's local() commands, and run a 'hg archive' command to prepare a tarball. Then you can use fabrics put() to upload that tarball, and fabrics run() to unpack it in the correct location.
A code snippet for the clone, pack, put might look a bit like the following:
from fabric.api import local
def task():
local("hg clone ssh://hg#host/repo tmpdir")
with lcd("tmpdir"):
local("hg archive ../repo.tgz")
local("rm tmpdir")
put("repo.tgz")