FYI: I am trying to replicate the web archive.
Right now, all the urls I am crawling are being send to the path "D:\website\dateoftoday". My code will remove every "/" from the urls, because you can't save a file with a slash in it. I've created a nginx web server to browse through these files and it works until I try clicking on a Root-Relative path (e.g. /blog/something). This is to be expected, because that link does not exist in my path (because its called blogsomething and not /blog/something).
My question is: how do I remove the "/" in the middle of the url for EVERY url on the web server?
Related
I have a simple task to do: script reads a free-text string which will contain SharePoint URLs. Now those URLs are provided by the users, basically a copy-paste from their browser. The thing that my app has to do is go to those links and check if there are any files under it.
So from what I can gather, there are many possible SharePoint URLs, for example:
<host>/sites/<site_name>/SitePages/something.aspx - for example a simple post
<host>/:w/r/sites/<site_name>/_layouts/15/something.aspx (like a shortcut URL) - for example a MS Office Word document
<host>/sites/<site_name>/<drive_name>/Forms/something.aspx?[...]&id=%2Fsites%2F<site-name>%2F<drive_name>%2F<path> - a URL to a file tree view of some files on a drive
<host>/:f:/r/sites/<site_name>/<drive_name>/<path_to_a_file>
The last one is perfect, because it contains the path to the directory in the url path. The 3rd one does have it as well, but in the urlencoded query params part.
What I do in this scenario is I parse the URL, extracting:
site name
drive name (not ID)
path (from the path in url or from the encoded &id= part)
Then, I can connect to SharePoint, get a site, list all the site drives (/drives), check if their "web_url" is a substring of my Sharepoint URL (I could search the appropriate drive by name, but the thing returned from the API is the "display name" and in my URL resides an "actual drive name"). Okay, so I've got my drive and now I can get my item by path. This all can be done via the regular MS Graph API (each step is needed for getting the object - site/drive ID) or via a python wrapper (I use python-o365).
As you can see, this is a real pain. Is there a standard way to deal with this? I mean, if I had the site and drive IDs, I could do it in a single API call, but given the fact that I only have a SharePoint link, I can't get those two, right? And how about the URL parsing?
I am trying to create a link that allows a user to download a zip file that's been generated earlier in the python script. The script then writes an HTML link to a web page. The user should be able to click the link and download their zip file.
import os,sys
downloadZip = ("http://<server>/folder/structure/here/" + zipFileName + ".zip")
print """<h3><a href="{}" download>Download zip file</a></h3>""".format(downloadZip)
The result is a link that when clicked opens a 404 page. I've noticed that on that page, it displays
Physical Path C:\inetpub\wwwroot\inputted\path\here\file.zip
I am testing this on the same server the processing is occurring on. I wouldn't think that should make a difference, but here I am. The end result should be a zip file downloaded to the user's pc.
Not sure if this would be helpful or not but I have noticed that some 'server package apps' disallow the execution/download of certain filetypes. I had a similar thing happen years ago.
To test if this is the case, create a new folder in your web directory and add an index.html page with some random writing (to identify that you have the correct page). Quickly try to access this page.
Next create a .zip file, put it in the same folder as the index.html file you just created, and add a Download link on the index.html page.
Now revisit the page, and try to download the file you created. If it works then there is a problem elsewhere, if it doesn't then whatever server package application you are using probably set Apache to block .zip files by default. Hope this helps buddy :)
I do a simple web application written in Python using cherrypy and Mako. So, my question is also simple.
I have one page with URL http://1.2.3.4/a/page_first. Also there is an image that available on URL http://1.2.3.4/a/page_first/my_image.png. And I want to locate my_image.png on the page_first.
I added a tag <img src="my_image.png"/>, but it is not shown. I looked at web developer tools->Network and saw that request URL for image was http://1.2.3.4/a/my_image.png, instead of http://1.2.3.4/a/page_first/my_image.png.
Why does it happen?
Thanks.
The page address needs to be http://1.2.3.4/a/page_first/ (with trailing slash).
ADDED:
You don't seem to understand relative URLs, so let me explain. When you reference an image like this <img src="my_image.png"/>, the image URL in the tag doesn't have any host/path info, so path is taken from the address of the HTML page that refers to the image. Since path is everything up to the last slash, in your case it is http://1.2.3.4/a/. So the full image URL that the browser will request becomes http://1.2.3.4/a/my_image.png.
You want it to be http://1.2.3.4/a/page_first/my_image.png, so the path part of the HTML page must be /a/page_first/.
Note that the browser will not assume page_first is "a directory" just because it doesn't have an "extension", and will not add the trailing slash automatically. When you access a server publishing static dirs and files and specify a directory name for the path and omit the trailing slash (e. g. http://www.example.com/some/path/here), the server is able to determine that you actually request a directory, and it adds the slash (and usually also a default/index file name) for you. It's not generally the case with dynamic web sites where URLs are programmed.
So basically you need to explicitly include the trailing slash in your page path: dispatcher.connect('page','/a/:number_of_page/', controller=self, action='page_method') and always refer to it with the trailing slash (http://1.2.3.4/a/page_first/), otherwise the route will not be matched.
As a side note, usually you put the images and other static files into a dedicated dir and serve them either with CherryPy's static dir tool, or, if it's a high load site, with a dedicated server.
Try <img src="/page_first/my_image.png"/>
I have a simple python module that generates a web page containing a list of links. This page is automatically opened in a browser through webbrowser.open('file://' + file.name): no problems here.
From this web page, when clicking on on a link (which has a href like http://localhost/cgi-bin/script.py?param1=Something) it lunches a cgi script that performs some operations with the passed value and at the end generates a new web page, stored locally in my machine (e.g. in a path like /home/user/web/out/) in a folder where I correctly set all rwx permissions.
Well, I've been trying to automatically open this new page in the browser for two days now, attempting with all the solutions I found searching through documentations and forums. I tried with webbrowser.open() again, but then I realized that I can't use it because from the web server I can't open a new browser window itself. Then I tried with redirection, printing first a header (print "Content-type: text/html\n\n") and then
print "Status: 302 Moved"
print "Location: file:///home/user/web/out/abc.html\n\n"`
but this just shows a plain blank page. I gave a try to other redirecting solutions, like
print "<meta http-equiv='refresh' content='0' url='file:///home/user/web/out/abc.html'>"
print "<script type="text/javascript">location.href='file:///home/user/web/out/abc.html';</script>"
print "<script type="text/javascript">windows.open('file:///home/user/web/out/abc.html');</script>"
I even tried inserting a Button with a POST method that triggers the opening of the new page, without success: I keep getting a plain blank page.
It is worth to say that, if I manually open this abc.html page in the browser, it is properly shown.
I think this has to do with the fact that the html page I'm opening from the web server is in locally stored, but I don't know how to solve this. Can somebody point me to the right direction?
A CGI-Script can not redirect I remember.
Note that status code 200 is sent prior to execution of a CGI script, so
scripts cannot send other status codes such as 302 (redirect).
This is some code:
self.send_response(200, "Script output follows")
module: http.server (Python 3)
But you can just open the file and use sys.stdout.write or shutil to copy its content to stdout.
Hi there.
I have a URL list. I do not know how to check is this address to a file or directory.
examples:
url = "http://example.com/path/to/file.html"
if '.' in url.split('/')[-1]:
return True
but if url is
url = "http://example.com/path/domains/domain.com"
domain.com is a directory not a file. How to detect it?
Checking the file extension is not good, maybe some headers? But I want to do as little as possible internet transfer usage.
Edit:
I need to download a large number of links and map their path to the location in my operating system. eg
example.com/path/to/file.html
~/Downloads/example.com/path/to/
and here download file.html.
eg:
example.com/directory/
create ~/Downlods/example.com/directory/
next url: example.com/directory/dir2
create ~/Downloads/example.com/directory/dir2
next url: example.com/directory/file.html
Download file.html in too ~/Downloads/example.com/directory/
not too create file.html directory
In short, you can't. Accessing the URL http://example.com/path/domains/domain.com would send a 302 redirect (if I remember correctly) to http://example.com/path/domains/domain.com/ by default. There are no headers in the response that indicates if a URL points to a directory. May I ask why you need to know this? I suppose you can add a slash to a URL and see what happens from there. That might get you the results you are looking for.
On HTTP servers, there is no such "file" or "directory" things. You just send an URI to the server which identify a specific resource that depends of the server's configuration.
By default, most of the HTTP servers use the files and directories of your system, but it can be configured (URL Rewriting, ...).