We Keep Coding

Why downloading Facebook images with requests.get() gives corrupted files?

I am a very new to Python and Facebook Graph API and hope you can help me out with this:
I have writted (in Python) a peace of code that uploads images tu a page on facebook (in a post, so it contains some text too) and this works as expected. Now I am trying to write a peace of code capable of downloading thhe image inside a post (given post_id). Unfortunately I always get "file corrupted " error.
Here is the code I use to download the image:
# this function uploads an image from a web url
ret = upload_img_to_fb(url)
# decode JSON from the request
post_id = json.loads(ret)
ret = get_json_from_fb_postID(post_id['post_id'])
perm_url = json.loads(ret)
print('Perm url = ' + perm_url['permalink_url'] + '\n')
img_response = requests.get(perm_url['permalink_url'])
image = open("foto4.jpg","wb")
image.write(img_response.content)
image.close()
Now, the print will print the following:
Perm url = https://www.facebook.com/102956292308044/photos/a.103716555565351/107173241886349/?type=3
which, acording to what I understood makes everything wrong because it is not a picture, even if a picture is displayed on the screen. So I right clicked the pic and opened it's link and I got:
https://scontent.fbzo1-2.fna.fbcdn.net/v/t39.30808-6/273008252_107171558553184_3697853178128736286_n.jpg?_nc_cat=103&ccb=1-5&_nc_sid=9267fe&_nc_ohc=d0ZvWSTzIogAX-PsuzN&_nc_ht=scontent.fbzo1-2.fna&oh=00_AT8GWh0wDHgB6tGCzHuPE2VZFus9EgWhllaJfVkZ-Nqtow&oe=620465E4
and if I pass this last link as parameter to img_response = requests.get() it works.
How do I get around this?

How to make pytube print all available resolutions and the user inputs one

Intro: So I am trying to make a pytube project but I am stuck on this one step,
Problem: I can't figure out how to make pytube list all the available resolutions
from pytube import YouTube
# import the package
print("Please Paste The URL of the youtube video")
url = input()
# URL (user input)
my_video = YouTube(url)
print(my_video.title)
# Title of The Video
#Now for the Thumbnail Image
print("Thumbnail URL")
print(my_video.thumbnail_url)
#To Download the video with the users Choice of resolution
print("Choose A Resolution Please")
for stream in my_video.stream:
print(stream)
#command for downloading the video
my_video.download()

There's an attribute for the stream object that goes by resolution. For example
You have:
for stream in my_video.stream:
print(stream)
But since you want to display the resolution of each stream objects, you can try:
for stream in my_video.stream:
print(stream.resolution)
I took the time to write a script to test my thought.
from pytube import YouTube
def download(video_resolutions, videos):
while True:
# Looping through the video_resolutions list to be displayed on the screen for user selection...
i = 1
for resolution in video_resolutions:
print(f'{i}. {resolution}')
i += 1
# To Download the video with the users Choice of resolution
choice = int(input('\nChoose A Resolution Please: '))
# To validate if the user enters a number displayed on the screen...
if 1 <= choice < i:
resolution_to_download = video_resolutions[choice - 1]
print(f"You're now downloading the video with resolution {resolution_to_download}...")
# command for downloading the video
videos[choice - 1].download()
print("\nVideo was successfully downloaded!")
break
else:
print("Invalid choice!!\n\n")
def sort_resolutions(url):
# URL (user input)
my_video = YouTube(url)
print(my_video.title)
# Title of The Video
# Now for the Thumbnail Image
print("Thumbnail URL")
print(my_video.thumbnail_url)
video_resolutions = []
videos = []
for stream in my_video.streams.order_by('resolution'):
# print(stream)
video_resolutions.append(stream.resolution)
videos.append(stream)
# print(video_resolutions)
return video_resolutions, videos
print("Please Paste The URL of the youtube video")
url = "https://youtu.be/o9aaoiyJlcM"
video_resolutions, videos = sort_resolutions(url)
download(video_resolutions, videos)
The url = "https://youtu.be/o9aaoiyJlcM" is just a static line that I don't have to be re-entering the url, you can change it back to the input if you wish. After assigning the link to the url variable I then pass that url to a function called sort_resolutions(url) where the link will be used and extract all that we will need. I used the function because it just makes the code more organized.
Within the sort_resolution function notice where I have created two lists objects... video_resolutions and videos, video_resolutions = [], videos = [], I have these populated by the stream objects.
for stream in my_video.streams.order_by('resolution'):
video_resolutions.append(stream.resolution) # Populating the resolution list
videos.append(stream) # Populating the video list
my_video.streams.order_by('resolution') this is just sorting the stream objects by resolution in order.
return video_resolutions, videos is just returning the lists that has been populated, video_resolutions, videos = sort_resolutions(url).
The values returned is now going to be passed to the download function download(video_resolutions, videos). Please note that within this function the while loop is to keep a display of the menu on the screen of all the available resolutions that can be downloaded. If the user selects a valid number that choice variable collects the value and we will then use choice - 1 to find that index of the desired resolution by resolution_to_download = video_resolutions[choice - 1], but this will only find the resolution. In order to download the video that matches the same index number in the videos list, you'll have to videos[choice - 1].download(). In order words, videos[choice - 1] is a stream object so by calling the download method videos[choice - 1].download() still works.
One more notice, the resolution list may contain duplication for the resolution. So maybe you can fine tune that.

DocuSign API: When populating template's PDF form fields by tag, why do Text tabs get duplicated in the upper-left corner of every document?

Our team has been using the Docusign Python API for about a year now and we've noticed this issue pretty consistently, then disappear for a few weeks, and now it's back again. Every time we generate a new template with Text tabs anchored somewhere on the document, the Tabs get duplicated into the upper-left corner of the document. The resulting document has a bunch of overlapping text fields in the corner of the first page, which looks very unprofessional and messy.
Here's what the template looks like. The two fields stacked on top of each other are hard to see since they've been shrunk down to the minimum size, but if I click on them I can see that their tab labels match the two fields on the page (broker_name and sign_date).
Then during the signing ceremony for an envelope built from this template, the empty fields get populated with data, and show up in the corner.
We generate the Tabs object with the following code:
# Returns a Tabs object with name and date tabs. For envelopes that only contain those two fields.
def generate_name_date_tabs(name, date):
name = Text(document_id = '1', page_number = '1', recipient_id = '1', value=name, tab_label = 'eh_broker_name')
date = Text(document_id = '1', page_number = '1', recipient_id = '1', value=date, tab_label = 'eh_sign_date')
return Tabs(text_tabs=[name, date])
And we generate the template and envelope here:
def create_application_envelope(destination_email, base64_file_content, signer_name="PlanProvide", signer_tabs=None, is_embedded_signing=False):
# Create an application envelope
# docusign_auth.DSClient.login("jwt")
account_id = session["ds_account_id"]
envelope_api = docusign_api_handler.get_envelope_api()
templates_api = docusign_api_handler.get_templates_api()
access_code = generate_access_code()
signer = Signer(
email=destination_email,
name=signer_name,
recipient_id="1",
routing_order="1",
role_name="signer",
# Use a six-character access code so URL doesn't expire
access_code=access_code,
tabs=signer_tabs
)
doc = Document(
document_base64 = base64_file_content,
name = "Your application generated by PlanProvide",
file_extension = "pdf",
document_id = 1,
# Detect Adobe form field names
transform_pdf_fields = True
)
documents = [doc]
template_req_object = EnvelopeTemplate(
documents=documents, email_subject="Please sign this document",
recipients=Recipients(signers=[signer]),
description="Template created via the API",
name="Application Template",
shared="false",
status="created"
)
try:
res = templates_api.create_template(account_id=account_id, envelope_template=template_req_object)
except Exception as e:
raise DocusignError(e)
try:
template_id = res.templates[0].template_id
except:
template_id = res.template_id
envelope_definition = EnvelopeDefinition(
email_subject="Please sign this document sent from the Python SDK",
template_id=template_id,
template_roles=[signer],
status="sent" # requests that the envelope be created and sent.
)
# Send it to an email
try:
results = envelope_api.create_envelope(account_id=account_id, envelope_definition=envelope_definition)
except Exception as e:
raise DocusignError(e)
return {
"envelope_id": results.envelope_id,
"access_code": access_code
}
As far as I can tell, the Tabs object still has only the two Text fields all the way up to the template creation, so I'm not sure why the duplicates appear. Any help or ideas with this issue would be greatly appreciated!
EDIT:
Here's where we call generate_name_date_tabs():
tabs = docusign_api_handler.generate_name_date_tabs(name, datetime.today().strftime('%m-%d-%y'))
try:
resp = application_services.create_application_envelope(
destination_email,
base64_file_content,
signer_name=name,
signer_tabs=tabs, is_embedded_signing=False)

I think I see the problem.
You're defining the tabs inside signer by providing the tabs field. Later down the code, when you're creating the envelope, you again provide the same signer in templateRoles:
except:
template_id = res.template_id
envelope_definition = EnvelopeDefinition(
email_subject="Please sign this document sent from the Python SDK",
template_id=template_id,
template_roles=[signer],
status="sent"
I'd change template_roles=[signer] to templateRoles=[{"email":"email#domain.com", "name": "name", "roleName" : "signer"}] and update
signer = Signer(
recipient_id="1",
routing_order="1",
role_name="signer",
# Use a six-character access code so URL doesn't expire
access_code=access_code,
tabs=signer_tabs
)
to not include name and email.
If that doesn't work, we'd need to dig in some more.Can you please show where you call generate_name_date_tabs?
Also can you show what's assigned to signer_tabs in the code below?
signer = Signer(
email=destination_email,
name=signer_name,
recipient_id="1",
routing_order="1",
role_name="signer",
# Use a six-character access code so URL doesn't expire
access_code=access_code,
**tabs=signer_tabs**
)
If you're seeing the issue disappearing and appearing with different releases, I might very well be a bug. Let us know if any of above is useful in narrowing down the problem.

First off, if you want a date/time when it's signed, don't use a textTab, but a dateSignedTab. That one automatically fill in the date/time when the document is signed.
Second, you are using a template. If your template has tabs define - they would be there in addition to the tabs in your code.
Third, if you don't provide location (X/Y coordinates) and no anchorText, the tab would be placed in position 0/0 or upper left corner like you see.
So, the one in the right place is probably coming from the template, the one in your code is showing in the upper left corner cause no position was specified.

python: get all youtube video urls of a channel

I want to get all video url's of a specific channel. I think json with python or java would be a good choice. I can get the newest video with the following code, but how can I get ALL video links (>500)?
import urllib, json
author = 'Youtube_Username'
inp = urllib.urlopen(r'http://gdata.youtube.com/feeds/api/videos?max-results=1&alt=json&orderby=published&author=' + author)
resp = json.load(inp)
inp.close()
first = resp['feed']['entry'][0]
print first['title'] # video title
print first['link'][0]['href'] #url

After the youtube API change, max k.'s answer does not work. As a replacement, the function below provides a list of the youtube videos in a given channel. Please note that you need an API Key for it to work.
import urllib
import json
def get_all_video_in_channel(channel_id):
api_key = YOUR API KEY
base_video_url = 'https://www.youtube.com/watch?v='
base_search_url = 'https://www.googleapis.com/youtube/v3/search?'
first_url = base_search_url+'key={}&channelId={}&part=snippet,id&order=date&maxResults=25'.format(api_key, channel_id)
video_links = []
url = first_url
while True:
inp = urllib.urlopen(url)
resp = json.load(inp)
for i in resp['items']:
if i['id']['kind'] == "youtube#video":
video_links.append(base_video_url + i['id']['videoId'])
try:
next_page_token = resp['nextPageToken']
url = first_url + '&pageToken={}'.format(next_page_token)
except:
break
return video_links

Short answer:
Here's a library That can help with that.
pip install scrapetube
import scrapetube
videos = scrapetube.get_channel("UC9-y-6csu5WGm29I7JiwpnA")
for video in videos:
print(video['videoId'])
Long answer:
The module mentioned above was created by me due to a lack of any other solutions. Here's what i tried:
Selenium. It worked but had three big drawbacks: 1. It requires a web browser and driver to be installed. 2. has big CPU and memory requirements. 3. can't handle big channels.
Using youtube-dl. Like this:
import youtube_dl
youtube_dl_options = {
'skip_download': True,
'ignoreerrors': True
}
with youtube_dl.YoutubeDL(youtube_dl_options) as ydl:
videos = ydl.extract_info(f'https://www.youtube.com/channel/{channel_id}/videos')
This also works for small channels, but for bigger ones i would get blocked by youtube for making so many requests in such a short time (because youtube-dl downloads more info for every video in the channel).
So i made the library scrapetube which uses the web API to get all the videos.

Increase max-results from 1 to however many you want, but beware they don't advise grabbing too many in one call and will limit you at 50 (https://developers.google.com/youtube/2.0/developers_guide_protocol_api_query_parameters).
Instead you could consider grabbing the data down in batches of 25, say, by changing the start-index until none came back.
EDIT: Here's the code for how I would do it
import urllib, json
author = 'Youtube_Username'
foundAll = False
ind = 1
videos = []
while not foundAll:
inp = urllib.urlopen(r'http://gdata.youtube.com/feeds/api/videos?start-index={0}&max-results=50&alt=json&orderby=published&author={1}'.format( ind, author ) )
try:
resp = json.load(inp)
inp.close()
returnedVideos = resp['feed']['entry']
for video in returnedVideos:
videos.append( video )
ind += 50
print len( videos )
if ( len( returnedVideos ) < 50 ):
foundAll = True
except:
#catch the case where the number of videos in the channel is a multiple of 50
print "error"
foundAll = True
for video in videos:
print video['title'] # video title
print video['link'][0]['href'] #url

Based on the code found here and at some other places, I've written a small script that does this. My script uses v3 of Youtube's API and does not hit against the 500 results limit that Google has set for searches.
The code is available over at GitHub: https://github.com/dsebastien/youtubeChannelVideosFinder

Independent way of doing things. No api, no rate limit.
import requests
username = "marquesbrownlee"
url = "https://www.youtube.com/user/username/videos"
page = requests.get(url).content
data = str(page).split(' ')
item = 'href="/watch?'
vids = [line.replace('href="', 'youtube.com') for line in data if item in line] # list of all videos listed twice
print(vids[0]) # index the latest video
This above code will scrap only limited number of video url's max upto 60. How to grab all the videos url which is present in the channel. Can you please suggest.
This above code snippet will display only the list of all the videos which is listed twice. Not all the video url's in the channel.

Using Selenium Chrome Driver:
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
import time
driverPath = ChromeDriverManager().install()
driver = webdriver.Chrome(driverPath)
url = 'https://www.youtube.com/howitshouldhaveended/videos'
driver.get(url)
height = driver.execute_script("return document.documentElement.scrollHeight")
previousHeight = -1
while previousHeight < height:
previousHeight = height
driver.execute_script(f'window.scrollTo(0,{height + 10000})')
time.sleep(1)
height = driver.execute_script("return document.documentElement.scrollHeight")
vidElements = driver.find_elements_by_id('thumbnail')
vid_urls = []
for v in vidElements:
vid_urls.append(v.get_attribute('href'))
This code has worked the few times I've tried it; however, you might need to tweak the sleep time, or add a way to recognize when the browser is still loading the extra information. It easily worked for me for getting a channel with 300+ videos, but it was having an issue with one that had 7000+ videos due to the time required to load the new videos on the browser becoming inconsistent.

I modified the script originally posted by dermasmid to fit my needs. This is the result:
import scrapetube
import sys
path = '_list.txt'
sys.stdout = open(path, 'w')
videos = scrapetube.get_channel("UC9-y-6csu5WGm29I7JiwpnA")
for video in videos:
print("https://www.youtube.com/watch?v="+str(video['videoId']))
# print(video['videoId'])
Basically it is saves all the URLs from the playlist into a "_list.txt" file. I am using this "_list.txt" file to download all the videos using the yt-dlp.exe. All the downloaded files have the .mp4 extension.
Now I do need to create another "_playlist.txt" file that contains all the FILENAMES coresponding to each URL from the "_List.txt".
For example, for: "https://www.youtube.com/watch?v=yG1m7oGZC48" to have "Apple M1 Ultra & NUMA - Computerphile.mp4" as output into the "_playlist.txt"

I do made some further improvements, to be able to add the channel URL into the console, print the result on screen and also into an external file called "_list.txt".
import scrapetube
import sys
path = '_list.txt'
print('**********************\n')
print("The result will be saved in '_list.txt' file.")
print("Enter Channel ID:")
# Prints the output in the console and into the '_list.txt' file.
class Logger:
def __init__(self, filename):
self.console = sys.stdout
self.file = open(filename, 'w')
def write(self, message):
self.console.write(message)
self.file.write(message)
def flush(self):
self.console.flush()
self.file.flush()
sys.stdout = Logger(path)
# Strip the: "https://www.youtube.com/channel/"
channel_id_input = input()
channel_id = channel_id_input.strip("https://www.youtube.com/channel/")
videos = scrapetube.get_channel(channel_id)
for video in videos:
print("https://www.youtube.com/watch?v="+str(video['videoId']))
# print(video['videoId'])

Invalid request URI while adding a video to playlist via youtube api

I have been unable to overcome this error while trying to add a video to my playlist using the youtube gdata python api.
gdata.service.RequestError: {'status':
400, 'body': 'Invalid request URI',
'reason': 'Bad Request'}
This seems to be the same error, but there are no solutions as yet. Any help guys?
import getpass
import gdata.youtube
import gdata.youtube.service
yt_service = gdata.youtube.service.YouTubeService()
# The YouTube API does not currently support HTTPS/SSL access.
yt_service.ssl = False
yt_service = gdata.youtube.service.YouTubeService()
yt_service.email = #myemail
yt_service.password = getpass.getpass()
yt_service.developer_key = #mykey
yt_service.source = #text
yt_service.client_id= #text
yt_service.ProgrammaticLogin()
feed = yt_service.GetYouTubePlaylistFeed(username='default')
# iterate through the feed as you would with any other
for entry in feed.entry:
if (entry.title.text == "test"):
lst = entry;
print entry.title.text, entry.id.text
custom_video_title = 'my test video on my test playlist'
custom_video_description = 'this is a test video on my test playlist'
video_id = 'Ncakifd_16k'
playlist_uri = lst.id.text
playlist_video_entry = yt_service.AddPlaylistVideoEntryToPlaylist(playlist_uri, video_id, custom_video_title, custom_video_description)
if isinstance(playlist_video_entry, gdata.youtube.YouTubePlaylistVideoEntry):
print 'Video added'
The confounding thing is that updating the playlist works, but adding a video does not.
playlist_entry_id = lst.id.text.split('/')[-1]
original_playlist_description = lst.description.text
updated_playlist = yt_service.UpdatePlaylist(playlist_entry_id,'test',original_playlist_description,playlist_private=False)
The video_id is not wrong because its the video from the sample code. What am I missing here? Somebody help!
Thanks.

Gdata seems to use v1 API. So, the relevant documentation is here: http://code.google.com/apis/youtube/1.0/developers_guide_protocol.html#Retrieving_a_playlist
This means, your "playlist_uri" should not take the value of "lst.id.text", but should take the "feedLink" element's "href" attribute in order to be used with "AddPlaylistVideoEntryToPlaylist"
Even if you happen to use v2 API, you should take the URI from the "content" element's "src" attribute as explained in the documentation, you get by substituting 2.0, in the above URL! (SO doesn't allow me to put two hyperlinks because i don't have enough reputations! :))