We Keep Coding

What is wrong with my implementation of importing csv into google sheets?

I am using gspread package in python, and I try to import a csv into a google spreadsheet, but I got an error.
Here is my code:
import requests
from bs4 import BeautifulSoup
from csv import writer
import gspread
from oauth2client.service_account import ServiceAccountCredentials
scope = ["https://spreadsheets.google.com/feeds", 'https://www.googleapis.com/auth/spreadsheets',
"https://www.googleapis.com/auth/drive.file", "https://www.googleapis.com/auth/drive"]
creds = ServiceAccountCredentials.from_json_keyfile_name("creds.json", scope)
client = gspread.authorize(creds)
# Open a sheet from a spreadsheet
spreadsheet = client.open("BlogScraping")
print(spreadsheet.id)
res = requests.get("https://www.rithmschool.com/blog")
soup = BeautifulSoup(res.text, "html.parser")
articles = soup.find_all("article")
with open("blog_data.csv", "w") as csv_file:
csv_writer = writer(csv_file)
csv_writer.writerow(["title", "link", "date"])
for article in articles:
title = article.find("a").get_text()
url = article.find("a")["href"]
date = article.find("time")["datetime"]
csv_writer.writerow([title, url, date])
content = open('blog_data.csv', 'r').read()
client.import_csv(spreadsheet.id, content)
Here's the errors I got:
Traceback (most recent call last):
File "/Users/chrisrosana/PycharmProjects/webScrapeProject/scraping.py", line 32, in <module>
client.import_csv(spreadsheet.id, content)
File "/Users/chrisrosana/PycharmProjects/webScrapeProject/venv/lib/python3.9/site-packages/gspread/client.py", line 293, in import_csv
self.request(
File "/Users/chrisrosana/PycharmProjects/webScrapeProject/venv/lib/python3.9/site-packages/gspread/client.py", line 61, in request
response = getattr(self.session, method)(
File "/Users/chrisrosana/PycharmProjects/webScrapeProject/venv/lib/python3.9/site-packages/requests/sessions.py", line 602, in put
return self.request('PUT', url, data=data, **kwargs)
File "/Users/chrisrosana/PycharmProjects/webScrapeProject/venv/lib/python3.9/site-packages/google/auth/transport/requests.py", line 464, in request
response = super(AuthorizedSession, self).request(
File "/Users/chrisrosana/PycharmProjects/webScrapeProject/venv/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/Users/chrisrosana/PycharmProjects/webScrapeProject/venv/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/Users/chrisrosana/PycharmProjects/webScrapeProject/venv/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/Users/chrisrosana/PycharmProjects/webScrapeProject/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/Users/chrisrosana/PycharmProjects/webScrapeProject/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/Users/chrisrosana/PycharmProjects/webScrapeProject/venv/lib/python3.9/site-packages/urllib3/connection.py", line 234, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1300, in _send_request
body = _encode(body, 'body')
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 164, in _encode
raise UnicodeEncodeError(
UnicodeEncodeError: 'latin-1' codec can't encode character '\u2013' in position 1980: Body ('–') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.
I think I did the implementation correctly and my spreadsheet.id are correct also. I don't know what am I doing wrong?

I found the solution.
It's basically the content that I am sending contains a character that is not valid. It has something to do on utf-8 encoding. Adding content.encode('utf-8') will solve it.
So instead of
client.import_csv(spreadsheet.id, content)
do this:
client.import_csv(spreadsheet.id, data=content.encode('utf-8')
Thanks #Ludo21South

Exchangelib: Can't connect to server

so recently I tried myself with Exchangelib, but I currently can't solve the issue. Heres my code:
from exchangelib import DELEGATE, Account, Credentials, IMPERSONATION
from exchangelib.configuration import Configuration
config = Configuration(
server='https://XXX',
credentials=Credentials(username='XXX\\XXX', password='XXX')
)
account = Account(
primary_smtp_address='mail#mail.com',
config=config,
autodiscover=False,
access_type=DELEGATE,
)
for item in account.inbox.all().order_by('-datetime_received')[:20]:
print(item.subject, item.sender, item.datetime_received)
Currently I receive the following error:
File "Project/Exchange/standardoutlook2.py", line 23, in <module>
config = Configuration(server='https://XXX', credentials=credentials)
File "Projectvenv36\lib\site-packages\exchangelib\configuration.py", line 46, in __init__
version=version
File "Projectvenv36\lib\site-packages\exchangelib\protocol.py", line 176, in __call__
protocol = super(CachingProtocol, cls).__call__(*args, **kwargs)
File "Projectvenv36\lib\site-packages\exchangelib\protocol.py", line 209, in __init__
name=self.credentials.username)
File "Projectvenv36\lib\site-packages\exchangelib\transport.py", line 149, in get_service_authtype
timeout=BaseProtocol.TIMEOUT)
File "Projectvenv36\lib\site-packages\requests\sessions.py", line 559, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "Projectvenv36\lib\site-packages\requests\sessions.py", line 512, in request
resp = self.send(prep, **send_kwargs)
File "Projectvenv36\lib\site-packages\requests\sessions.py", line 622, in send
r = adapter.send(request, **kwargs)
File "Projectvenv36\lib\site-packages\requests\adapters.py", line 412, in send
self.cert_verify(conn, request.url, verify, cert)
File "Project/Exchange/standardoutlook2.py", line 13, in cert_verify
}[urlparse(url).hostname]
KeyError: 'https'
I've added the certificate to the certificate of certifi, so when I start any request to the server outside of the code above it works well.

Crash, Python and Google Calendar API

Ahoy !
I followed the Documentation (https://developers.google.com/google-apps/calendar/quickstart/python ) for google calendar API and this tutorial (http://www.esologic.com/?p=634 ) to create an alarm clock. The principle is quite simple. I put an event in my calendar, and if it is on the calendar "Raspberry" and that the description is "wake" when the time = the time specified, it plays me my music on the Pi.
Except that when I run my script, it will work for a variable time, and then will crash with a random error.
Traceback (most recent call last):
File "alarmclock.py", line 65, in <module>
calendar_event_query()
File "alarmclock.py", line 37, in calendar_event_query
events = service.events().list(singleEvents=True, calendarId=CALENDAR_ID).execute()
File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 140, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/googleapiclient/http.py", line 722, in execute
body=self.body, headers=self.headers)
File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 596, in new_request
redirections, connection_type)
File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1609, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1351, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1272, in _conn_request
conn.connect()
File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1036, in connect
self.disable_ssl_certificate_validation, self.ca_certs)
File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 80, in _ssl_wrap_socket
cert_reqs=cert_reqs, ca_certs=ca_certs)
File "/usr/lib/python2.7/ssl.py", line 891, in wrap_socket
ciphers=ciphers)
File "/usr/lib/python2.7/ssl.py", line 566, in __init__
self.do_handshake()
File "/usr/lib/python2.7/ssl.py", line 788, in do_handshake
self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:581)
or
Traceback (most recent call last):
File "alarmclock.py", line 80, in <module>
calendar_event_query()
File "alarmclock.py", line 52, in calendar_event_query
events = service.events().list(singleEvents=True, calendarId=CALENDAR_ID).execute()
File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 140, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/googleapiclient/http.py", line 722, in execute
body=self.body, headers=self.headers)
File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 596, in new_request
redirections, connection_type)
File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1609, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1351, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1278, in _conn_request
raise ServerNotFoundError("Unable to find the server at %s" % conn.host)
httplib2.ServerNotFoundError: Unable to find the server at www.googleapis.com
So I do not understand if it is a problem with my script, or if it is directly GoogleAPI that misfire and then crash my script?
Here is my script.
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
from __future__ import print_function
import httplib2
from apiclient import discovery
import oauth2client
from oauth2client import client
from oauth2client import tools
from datetime import datetime
import logging
import time
import random
import os
import argparse
try:
flags = argparse.ArgumentParser(parents=[tools.argparser]).parse_args()
except ImportError:
flags = None
CALENDAR_ID = 'MY_CALENDAR_ID#group.calendar.google.com'
CLIENT_SECRET_FILE = 'raspi-wake.json'
SCOPE = 'https://www.googleapis.com/auth/calendar.readonly'
MP3_FOLDER = 'mp3'
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
def get_credentials():
home_dir = os.path.expanduser('~')
credential_dir = os.path.join(home_dir, '.credentials')
if not os.path.exists(credential_dir):
os.makedirs(credential_dir)
credential_path = os.path.join(credential_dir, 'raspberry-wake.json')
store = oauth2client.file.Storage(credential_path)
credentials = store.get()
if not credentials or credentials.invalid:
flow = client.flow_from_clientsecrets(CLIENT_SECRET_FILE, scope=SCOPE)
flow.user_agent = 'Raspberry Wake'
if flags:
credentials = tools.run_flow(flow, store, flags)
else:
credentials = tools.run(flow, store)
print('Storing credentials to ' + credential_path)
return credentials
def calendar_event_query():
credentials = get_credentials()
http = credentials.authorize(httplib2.Http())
service = discovery.build('calendar', 'v3', http=http)
today = datetime.today()
events = service.events().list(singleEvents=True, calendarId=CALENDAR_ID).execute()
for i, event in enumerate(events['items']):
try:
start = event['start']['dateTime'][:-9]
except KeyError:
start = ''
description = event.get('description', '')
now = today.strftime('%Y-%m-%dT%H:%M')
if start >= now and description.lower() == 'wake':
if start == now:
mp3_files = random.choice(os.listdir(MP3_FOLDER))
command = 'mpg123 \'{}/{}\''.format(MP3_FOLDER, mp3_files)
os.system(command)
time.sleep(60)
while True:
calendar_event_query()
time.sleep(5)

Socket error when using gdata Youtube API in Python

I'm using gdata to map YouTube URLs to video titles, using the following code:
import gdata.youtube.service as youtube
import re
import queue
import urlparse
ytservice = youtube.YouTubeService()
ytservice.ssl = True
ytservice.developer_key = '' # snip
class youtube(mediaplugin):
def __init__(self, parsed_url):
self.url = parsed_url
self.video_id = urlparse.parse_qs(parsed_url.query)['v'][0]
self.ytdata = ytservice.GetYouTubeVideoEntry(self.video_id)
print self.ytdata
I get the following socket exception when calling service.GetYouTubeVideoEntry():
File "/Users/haldean/Documents/qpi/qpi/media.py", line 21, in __init__
self.ytdata = ytservice.GetYouTubeVideoEntry(self.video_id)
File "/Users/haldean/Documents/qpi/lib/python2.7/site-packages/gdata/youtube/service.py", line 210, in GetYouTubeVideoEntry
return self.Get(uri, converter=gdata.youtube.YouTubeVideoEntryFromString)
File "/Users/haldean/Documents/qpi/lib/python2.7/site-packages/gdata/service.py", line 1069, in Get
headers=extra_headers)
File "/Users/haldean/Documents/qpi/lib/python2.7/site-packages/atom/__init__.py", line 93, in optional_warn_function
return f(*args, **kwargs)
File "/Users/haldean/Documents/qpi/lib/python2.7/site-packages/atom/service.py", line 186, in request
data=data, headers=all_headers)
File "/Users/haldean/Documents/qpi/lib/python2.7/site-packages/atom/http_interface.py", line 148, in perform_request
return http_client.request(operation, url, data=data, headers=headers)
File "/Users/haldean/Documents/qpi/lib/python2.7/site-packages/atom/http.py", line 163, in request
connection.endheaders()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 937, in endheaders
self._send_output(message_body)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 797, in _send_output
self.send(msg)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 759, in send
self.connect()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1140, in connect
self.timeout, self.source_address)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 553, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
gaierror: [Errno 8] nodename nor servname provided, or not known
I'm at a loss as to how to even begin debugging this. Any ideas appreciated. Thanks!
Edit:
In response to a question asked in comments, video_id is qh-mwjF-OMo and parsed_url is:
ParseResult(scheme=u'http', netloc=u'www.youtube.com', path=u'/watch', params='', query=u'v=qh-mwjF-OMo&feature=g-user-u', fragment='')

My mistake was that the video_id should be passed as a keyword parameter, like so:
self.ytdata = ytservice.GetYouTubeVideoEntry(video_id=self.video_id)
It seems that the socket exception is the only layer of gdata that will throw an exception; it tries to get a URL blindly based on the arguments and it only fails when the URL fetch fails.

Problem passing XML to HTTP Post api using urllib in Python

I am trying to access a REST api and need to call it with a line of XML for a filter condition. My apologies for providing code that others cannot access. When I execute this code, I get the error message listed below.
import urllib2
import urllib
import hashlib
import hmac
import time
import random
import base64
def MakeRequest():
url = 'https://api01.marketingstudio.com/API/Gateway/9'
publickey = ''
privatekey = ''
method = 'Query'
nonce = random.randrange(123400, 9999999)
age = int(time.time())
final = str(age) + '&' + str(nonce) + '&' + method.lower() + '&' + url.lower()
converted = hmac.new(privatekey, final, hashlib.sha1).digest()
authorization = 'AMS ' + publickey + ':' + base64.b64encode(converted)
xml_string = "<list><FilterItems><FilterItem attribute='pageNumber' value='1'/></FilterItems></list>"
form = {'XML':xml_string}
data = urllib.urlencode(form)
headers = {'Content-Type': 'application/xml'}
req = urllib2.Request(url,data,headers)
req.add_header('ams-method', method)
req.add_header('ams-nonce', nonce)
req.add_header('ams-age', age)
req.add_header('Authorization', authorization)
r = urllib2.urlopen(req)
print r.read()
MakeRequest();
Here is the error message.
Data at the root level is invalid. Line 1, position 1.
at Aprimo.REST.Core.RESTService.GetRequest(String URI, HttpRequest req)
at Aprimo.REST.RESTHandler.GetRequest(String apiUrl, HttpContext context)
at Aprimo.REST.RESTHandler.ProcessRequest(HttpContext context)
I think this has the correct logic and filter conditions, what should I look at to get this to work. Thanks.
Per #Mark's suggestion I removed the urlencode for the XML string and got the following TraceBack:
Traceback (most recent call last):
File "file.py", line 36, in <module>
MakeRequest();
File "file.py", line 32, in MakeRequest
r = urllib2.urlopen(req)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 126, in urlopen
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 392, in open
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 410, in _open
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 370, in _call_chain
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1194, in https_open
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1155, in do_open
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 941, in request
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 975, in _send_request
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 937, in endheaders
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 801, in _send_output
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 773, in send
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 207, in sendall
TypeError: unhashable type

So the problem was with the formatting of the form variable and the encoding I was trying to do. Revising the following lines gets the call to work. I did not need to specify the headers.
xml_string = "<list><FilterItems><FilterItem attribute='pageNumber' value='1'/></FilterItems></list>"
data = (xml_string)
req = urllib2.Request(url,data)