We Keep Coding

Trying to check if the webdirectory is showing the same thing as index.html

I'm on a blackbox penetration training, last time i asked a question about sql injection which so far im making a progress on it i was able to retrieve the database and the column.
This time i need to find the admin login, so i used dirsearch for that, i checked each webdirectories from dirsearch and sometimes it would show the same page as index.html.
So i'm trying to fix this by automating the process with a script:
import requests
url = "http://depedqc.ph";
webdirectory_path = "C:/PentestingLabs/Dirsearch/reports/depedqc.ph/scanned_webdirectory9-3-2022.txt";
index = requests.get(url);
same = index.content
for webdirectory in open(webdirectory_path, "r").readlines():
webdirectory_split = webdirectory.split();
result = result = [i for i in webdirectory_split if i.startswith(url)];
result = ''.join(result);
print(result);
response = requests.get(result);
if response.content == same:
print("same content");
Only problem is, i get this error:
Invalid URL '': No scheme supplied. Perhaps you meant http://?
Even though the printed result is: http://depedqc.ph/html
What am i doing wrong here? i appreciate a feedback

Scraping values from a request response

Unfortunately i can provide only the output of the request and not the full code since it contains quite private infos, basically when printing the request as text file i get a json one, something like that:
{"paymentResource":{"paymentToken":"PAYID-MEJ------","intent":"authorize","redirectUrl":"https://www.paypal.com/checkoutnow?nolegacy=1\u0026token=EC-5JS-----2S","authenticateUrl":null}}
How can i scrape that Paypal url? tried by doing this but it didn't worked (ppstep2 is the name of the request):
content = ppstep2.json()
pp = content["redirectUrl"]
I only get this error while doing it:
pp = content["redirectUrl"]
KeyError: 'redirectUrl'

Your variable content is a dictionary.
To get the value for "redirectUrl" you can do this:
pp = content['paymentResource']['redirectUrl']
The key error was caused by not including ['paymentResource']
I would recommend reviewing python dictionaries and the .get() method as well.
https://docs.python.org/3/library/stdtypes.html?highlight=dictionary%20get#dict.get

Try adding a print(json.dumps(content, indent=4)) before you try to access it and look at the output. You might spot why then.
redirectUrl isn't part of content. It's in the content['paymentResource'] dictionary in that response content.
using content['paymentResource']['redirectUrl'] should work.
Edit: If you want to try and get a value without ending up with an exception, try using .get():
# This will result in a KeyError as you experienced:
pp = content["redirectUrl"]
# This will instead set pp to None if 'redirectUrl' doesn't exist as a Key
pp = content.get("redirectUrl", None)

How to download list data from SharePoint Online to a csv (preferably) or json file?

I have accessed a list in SharePoint Online with Python and want to save the list data to a file (csv or json) to transform it and sort some metadata for a migration
I have full access to the Sharepoint site I am connecting(client ID, secret..).
from office365.runtime.auth.authentication_context import AuthenticationContext
from office365.runtime.client_request import ClientRequest
from office365.sharepoint.client_context import ClientContext
I have set my settings:
app_settings = {
'url': 'https://company.sharepoint.com/sites/abc',
'client_id': 'id',
'client_secret': 'secret'
}
Connecting to the site:
context_auth = AuthenticationContext(url=app_settings['url'])
context_auth.acquire_token_for_app(client_id=app_settings['client_id'],
client_secret=app_settings['client_secret'])
ctx = ClientContext(app_settings['url'], context_auth)
Getting the lists and checking the titles:
lists = ctx.web.lists
ctx.load(lists)
ctx.execute_query()
for lista in lists:
print(lista.properties["Title"]) # this gives me the titles of each list and it works.
lists is a ListCollection Object
From the previous code, I see that I want to get the list titled: "Analysis A":
a1 = lists.get_by_title("Analysis A")
ctx.load(a1)
ctx.execute_query() # a1 is a List item - non-iterable
Then I get the data in that list:
a1w = a1.get_items()
ctx.load(a1w)
ctx.execute_query() # a1w is a ListItemCollection - iterable
idea 1: df to json/csv
df1 = pd.DataFrame(a1w) #doens't work)
idea 2:
follow this link: How to save a Sharepoint list as a file?
I get an error while executing the json.loads command:
JSONDecodeError: Extra data: line 1 column 5 (char 4)
Alternatives:
I tried Shareplum, but can't connect with it, like I did with office365-python-rest. My guess is that it doesn't have an authorisation option with client id and client secret (as far as I can see)
How would you do it? Or am I missing something?

Sample test demo for your reference.
context_auth = AuthenticationContext(url=app_settings['url'])
context_auth.acquire_token_for_app(client_id=app_settings['client_id'],
client_secret=app_settings['client_secret'])
ctx = ClientContext(app_settings['url'], context_auth)
list = ctx.web.lists.get_by_title("ListA")
items = list.get_items()
ctx.load(items)
ctx.execute_query()
dataList = []
for item in items:
dataList.append({"Title":item.properties["Title"],"Created":item.properties["Created"]})
print("Item title: {0}".format(item.properties["Title"]))
pandas.read_json(json.dumps(dataList)).to_csv("output.csv", index = None,header=True)

Idea 1
It's hard to tell what can go wrong without the error trace. But I suspect it's likely to do with malformed data that you are passing as the argument. See here from the documentation to know exactly what's expected.
Do also consider updating your question with relevant stack error traces.
Idea 2
JSONDecodeError: Extra data: line 1 column 5 (char 4)
This error simply means that the Json string is not a valid format. You can validate JSON strings by using this service. This often tells you the point of error which you can then use it to manually fix the problem.
This error could also be caused if the object that is being parsed is a python object. You can avoid this by jsonifying each line as you go
data_list= []
for line in open('file_name.json', 'r'):
data_list.append(json.loads(line))
This avoids storing intermediate python objects. Also see this related issue if nothing works.

How to convert suds object to xml string

This is a duplicate to this question:
How to convert suds object to xml
But the question has not been answered: "totxt" is not an attribute on the Client class.
Unfortunately I lack of reputation to add comments. So I ask again:
Is there a way to convert a suds object to its xml?
I ask this because I already have a system that consumes wsdl files and sends data to a webservice. But now the customers want to alternatively store the XML as files (to import them later manually). So all I need are 2 methods for writing data: One writes to a webservice (implemented and tested), the other (not implemented yet) writes to files.
If only I could make something like this:
xml_as_string = My_suds_object.to_xml()
The following code is just an example and does not run. And it's not elegant. Doesn't matter. I hope you get the idea what I want to achieve:
I have the function "write_customer_obj_webservice" that works. Now I want to write the function "write_customer_obj_xml_file".
import suds
def get_customer_obj():
wsdl_url = r'file:C:/somepathhere/Customer.wsdl'
service_url = r'http://someiphere/Customer'
c = suds.client.Client(wsdl_url, location=service_url)
customer = c.factory.create("ns0:CustomerType")
return customer
def write_customer_obj_webservice(customer):
wsdl_url = r'file:C:/somepathhere/Customer.wsdl'
service_url = r'http://someiphere/Customer'
c = suds.client.Client(wsdl_url, location=service_url)
response = c.service.save(someparameters, None, None, customer)
return response
def write_customer_obj_xml_file(customer):
output_filename = r'C\temp\testxml'
# The following line is the problem. "to_xml" does not exist and I can't find a way to do it.
xml = customer.to_xml()
fo = open(output_filename, 'a')
try:
fo.write(xml)
except:
raise
else:
response = 'All ok'
finally:
fo.close()
return response
# Get the customer object always from the wsdl.
customer = get_customer_obj()
# Since customer is an object, setting it's attributes is very easy. There are very complex objects in this system.
customer.name = "Doe J."
customer.age = 42
# Write the new customer to a webservice or store it in a file for later proccessing
if later_processing:
response = write_customer_obj_xml_file(customer)
else:
response = write_customer_obj_webservice(customer)

I found a way that works for me. The trick is to create the Client with the option "nosend=True".
In the documentation it says:
nosend - Create the soap envelope but don't send. When specified, method invocation returns a RequestContext instead of sending it.
The RequestContext object has the attribute envelope. This is the XML as string.
Some pseudo code to illustrate:
c = suds.client.Client(url, nosend=True)
customer = c.factory.create("ns0:CustomerType")
customer.name = "Doe J."
customer.age = 42
response = c.service.save(someparameters, None, None, customer)
print response.envelope # This prints the XML string that would have been sent.

You have some issues in write_customer_obj_xml_file function:
Fix bad path:
output_filename = r'C:\temp\test.xml'
The following line is the problem. "to_xml" does not exist and I can't find a way to do it.
What's the type of customer? type(customer)?
xml = customer.to_xml() # to be continued...
Why mode='a'? ('a' => append, 'w' => create + write)
Use a with statement (file context manager).
with open(output_filename, 'w') as fo:
fo.write(xml)
Don't need to return a response string: use an exception manager. The exception to catch can be EnvironmentError.
Analyse
The following call:
customer = c.factory.create("ns0:CustomerType")
Construct a CustomerType on the fly, and return a CustomerType instance customer.
I think you can introspect your customer object, try the following:
vars(customer) # display the object attributes
help(customer) # display an extensive help about your instance
Another way is to try the WSDL URLs by hands, and see the XML results.
You may obtain the full description of your CustomerType object.
And then?
Then, with the attributes list, you can create your own XML. Use an XML template and fill it with the object attributes.
You may also found the magic function (to_xml) which do the job for you. But, not sure the XML format matches your need.

client = Client(url)
client.factory.create('somename')
# The last XML request by client
client.last_sent()
# The last XML response from Web Service
client.last_received()

jQuery getJSON Output using Python/Django

So, I'm trying to make a simple call using jQuery .getJSON to my local web server using python/django to serve up its requests. The address being used is:
http://localhost:8000/api/0.1/tonight-mobile.json?callback=jsonp1290277462296
I'm trying to write a simple web view that can access this url and return a JSON packet as the result (worried about actual element values/layout later).
Here's my simple attempt at just alerting/returning the data:
$.getJSON("http://localhost:8000/api/0.1/tonight-mobile.json&callback=?",
function(json){
alert(json);
<!--$.each(json.items, function(i,item){
});-->
});
I am able to access this URL directly, either at http://localhost:8000/api/0.1/tonight-mobile.json or http://localhost:8000/api/0.1/tonight-mobile.json&callback=jsonp1290277462296 and get back a valid JSON packet... So I'm assuming it's in my noob javascript:)
My views.py function that is generating this response looks as follows:
def tonight_mobile(request):
callback = request.GET.get('callback=?', '')
def with_rank(rank, place):
return (rank > 0)
place_data = dict(
Places = [make_mobile_place_dict(request, p) for p in Place.objects.all()]
)
xml_bytes = json.dumps(place_data)
xml_bytes = callback + '(' + xml_bytes + ');'
return HttpResponse(xml_bytes, mimetype="application/json")
With corresponding urls.py configuration:
(r'^tonight-mobile.json','iphone_api.views.tonight_mobile'),
I am still somewhat confused on how to use callbacks, so maybe that is where my issue lies. Note I am able to call directly a 'blah.json' file that is giving me a response, but not through a wired URL. Could someone assist me with some direction?

First, callback = request.GET.get('callback=?', '') won't get you the value of callback.
callback = request.GET.get( 'callback', None )
Works much better.
To debug this kind of thing. You might want to include print statements in your Django view function so you can see what's going on. For example: print repr(request.GET) is a helpful thing to put in a view function so that you can see the GET dictionary.