I have a gz format file. The file is very big and the first line is as follow:
{"originaltitle":"Leasing Specialist - WPM Real Estate Management","workexperiences":[{"company":"Home Properties","country":"US","customizeddaterange":"","daterange":{"displaydaterange":"","startdate":null,"enddate":null},"description":"Responsibilities: Inspect tour routes, models and show apartments daily to ensure cleanliness. Greeting prospective residents; determining the needs and preferences of the prospect and professionally present specific apartments while providing information regarding features and benefits. Answering incoming calls in a cheerful and professional manner. Handle each call accordingly whether it is a prospect call or an irate resident that just moved in. Develop and maintain Resident relations through the courtesy of on-site personnel, promptness of maintenance calls, and knowledge of community policies. Learn to develop professional sales and closing techniques. Accompany prospects to model apartments and discusses size and layout of rooms, available facilities, such as swimming pool and saunas, location of shopping centers, services available, and terms of lease. Demonstrate thorough knowledge and use of lead tracking system. Make follow-up calls to prospective Residents who did not fill out an application. Compile and update listings of available rental units.","location":"Baltimore, MD","normalizedtitle":"leasing specialist","title":"Leasing Specialist"},{"company":"WPM Real Estate Management","country":"US","customizeddaterange":"1 year, 3 months","daterange":{"displaydaterange":"July 2017 to October 2018","startdate":{"displaydate":"July 2017","granularity":"MONTH","isodate":{"date":null}},"enddate":{"displaydate":"October 2018","granularity":"MONTH","isodate":{"date":null}}},"description":"Responsibilities: Inspect tour routes, models and show apartments daily to ensure cleanliness. Greeting prospective residents; determining the needs and preferences of the prospect and professionally present specific apartments while providing information regarding features and benefits. Answering incoming calls in a cheerful and professional manner. Handle each call accordingly whether it is a prospect call or an irate resident that just moved in. Develop and maintain Resident relations through the courtesy of on-site personnel, promptness of maintenance calls, and knowledge of community policies. Learn to develop professional sales and closing techniques. Accompany prospects to model apartments and discusses size and layout of rooms, available facilities, such as swimming pool and saunas, location of shopping centers, services available, and terms of lease. Demonstrate thorough knowledge and use of lead tracking system. Make follow-up calls to prospective Residents who did not fill out an application. Compile and update listings of available rental units.","location":"Baltimore, MD","normalizedtitle":"leasing specialist","title":"Leasing Specialist"},{"company":"Westminster Management","country":"US","customizeddaterange":"1 year","daterange":{"displaydaterange":"June 2016 to June 2017","startdate":{"displaydate":"June 2016","granularity":"MONTH","isodate":{"date":null}},"enddate":{"displaydate":"June 2017","granularity":"MONTH","isodate":{"date":null}}},"description":"Responsibilities: Tour vacant units and model with future prospects.Process applications. Answer emails and incoming phone calls. Prepare lease agreement for signing. Collect all monies that is due on dateof move-in. Enter resident repair orders for resident. Walk vacant units to ensure that the unit is ready for show. Complete residency and employment verifications. Income qualify all applicants.","location":"Baltimore, MD","normalizedtitle":"leasing consultant","title":"Leasing Consultant"},{"company":"MARYLAND MANAGEMENT COMPANY","country":"US","customizeddaterange":"1 year, 1 month","daterange":{"displaydaterange":"April 2015 to May 2016","startdate":{"displaydate":"April 2015","granularity":"MONTH","isodate":{"date":null}},"enddate":{"displaydate":"May 2016","granularity":"MONTH","isodate":{"date":null}}},"description":"Responsibilities: Lease apartments, sign lease agreements, complete residence maintenance repairrequest, answer phones, customer service, processed prospects applications, opened and closedinventory, responded to Level One emails Accomplishments: I was able to successfully finish FairHousing requirements. The first month I was able to properly and accurately process a application and move-in documents. Skills Used: The skills I used while at Americana were strong team work, strongcommunication, interpersonal, and leadership.","location":"Glen Burnie, MD","normalizedtitle":"leasing agent","title":"Leasing Agent"},{"company":"Amazon.com","country":"US","customizeddaterange":"1 year, 5 months","daterange":{"displaydaterange":"September 2014 to February 2016","startdate":{"displaydate":"September 2014","granularity":"MONTH","isodate":{"date":null}},"enddate":{"displaydate":"February 2016","granularity":"MONTH","isodate":{"date":null}}},"description":"Responsibilities: I assure customers are receiving the correct merchandise in a timely fashion.And evaluate inventoryAccomplishments:I exceeded Amazon expectations of receiving 2800 items per hour, which allowed me to train otherassociates, building confidence and skills.Skills Used:The skills i used while performing my task were strong leadership, strong communications, and beingdetailed orientated.","location":"Baltimore, MD","normalizedtitle":"customer service representative","title":"Customer Service Representative"},{"company":"Carmax Superstore","country":"US","customizeddaterange":"1 year, 2 months","daterange":{"displaydaterange":"February 2014 to April 2015","startdate":{"displaydate":"February 2014","granularity":"MONTH","isodate":{"date":null}},"enddate":{"displaydate":"April 2015","granularity":"MONTH","isodate":{"date":null}}},"description":"Responsibilities:Greet customersSearch for the right vehicle that best suits the customers needs and wantsSubmit financial applicationsAssist customer with the purchasing process and document signingEnter customers information for appraisal offerAssist customer with purchasing Car Max extended warrantiesConducted follow- up on a daily, weekly, and monthy basisAccomplishments:I was acknowledged by the district for having 100% in Car Max extended warranties. Also I wasacknowledged by the district for having one of the highest Voice Of Customer survey scores. I passedthe 6 week training, obtaining my sales licenseSkills Used:I demonstrate strong communication, interpersonal and listening skills. I also have strongorganizational skills.","location":"Nottingham, MD","normalizedtitle":"sales consultant","title":"Certified Sales Consultant"},{"company":"rue21","country":"US","customizeddaterange":"1 year, 8 months","daterange":{"displaydaterange":"June 2011 to February 2013","startdate":{"displaydate":"June 2011","granularity":"MONTH","isodate":{"date":null}},"enddate":{"displaydate":"February 2013","granularity":"MONTH","isodate":{"date":null}}},"description":"Responsibilities: Managed profit goals on a daily basisCustomer ServiceReceived Incoming shipmentDelivered daily bank depsoitsMaintained store appearanceOverlooked sales associates performanceCreated daily goals for each sales associateAccomplishments:The impact that I was able to have during my time at Rue21, I was able to build a strong team of individuals who were scored top in the region for Customer Service.Skills Used:I demonstrated strong leadership and verbal communication.","location":"Dundalk, MD","normalizedtitle":"assistant store manager","title":"Assistant Store Manager"},{"company":"Shaws Jewelers","country":"US","customizeddaterange":"1 year, 5 months","daterange":{"displaydaterange":"November 2009 to April 2011","startdate":{"displaydate":"November 2009","granularity":"MONTH","isodate":{"date":null}},"enddate":{"displaydate":"April 2011","granularity":"MONTH","isodate":{"date":null}}},"description":"Responsibilities: Customer serviceGeneral office( typing, faxing, )Made outgoing calls to valued customersCleaned and maintained show cases and lunch roomPrepared jewlery repair tickets for outgoing shipmentAccomplishments:During my time at Shaws Jewelers I was able to demonstrate excellent customer service.Also I wasable to achieve personal profit goals and credit application goals on a daily basis. I was acknowledged and rewarded by my DM for excellent team participation and over achieving the 6 standards on a dailybasis.Skills Used:I demonstrated strong verbal and listening skills. Also I have excellent interpersonal skills.","location":"Dundalk, MD","normalizedtitle":"sales associate","title":"Sales Associate"}],"skillslist":[{"monthsofexperience":0,"text":"yardi"},{"monthsofexperience":0,"text":"marketing"},{"monthsofexperience":0,"text":"outlook"},{"monthsofexperience":0,"text":"receptionist"},{"monthsofexperience":0,"text":"management"}],"url":"/r/Lashannon-Felton/1062d3b8cbb13886","additionalinfo":""}\n'
I am not familiar with gzip.GzipFile format.
Is there a way to make it a dictionary?
You will want to make use of the json module and the gzip module in Python, both of which are part of the Python Standard Library.
The gzip module provides the GzipFile class, as well as the open(),
compress() and decompress() convenience functions. The GzipFile class
reads and writes gzip-format files, automatically compressing or
decompressing the data so that it looks like an ordinary file object.
To read the compressed file, you can call gzip.open().
Opening the file with the default rb mode, will return a gzip.GzipFile object, from which you can obtain a bytes-like object by calling read().
Then, using json.loads(), you can convert the raw data into a usable Python object -- a dictionary.
The snippet below is a simple demonstration of this in action:
import gzip
import json
with gzip.open('gzipped_file.json.gz', 'rb') as f:
raw_json = f.read()
data = json.loads(raw_json)
print(type(data))
# Prints <class 'dict'>
print(data)
# Prints {'originaltitle': 'Leasing Specialist - WPM Real Estate Management', 'workexperience ...
print(data['workexperiences'][0]['company'])
# Prints Home Properties
I'm creating a stock analysis web application. I want to store financial data for multiple stocks. Then I want to use a stock screener on them. This screener involves retrieving multiple stocks from the backend and performing a technical indicator test on them. Stocks that pass the indicator test will be returned to the user. Let's say i want to store a pandas.dataframe for exampleStock:
open high low close volume
date
2017-08-01 247.46 247.50 246.716 247.32 55050401
2017-08-02 247.47 247.60 246.370 247.44 47211216
2017-08-03 247.31 247.34 246.640 246.96 40855997
2017-08-04 247.52 247.79 246.970 247.41 60191838
2017-08-07 247.49 247.87 247.370 247.87 31995021
....
I have been using DataStore. I create entities for each stock setting the key as the stocks symbol. I use a model like this:
from google.appengine.ext import ndb
class Stocks(ndb.Model):
dates = ndb.StringProperty(repeated=True)
open = ndb.FloatProperty(repeated=True)
high = ndb.FloatProperty(repeated=True)
low = ndb.FloatProperty(repeated=True)
close = ndb.FloatProperty(repeated=True)
volume = ndb.FloatProperty(repeated=True)
Then I retrieve multiple entities to loop over with the techncial indicator check:
import numpy
listOfStocks = ndb.get_multi(list_of_keys)
for stock in listOfStocks:
doIndicatorCheck(numpy.array(stock.close))
I want to make a query for stocks, do the indicator check and then return results to the user as fast as possible. Should I be using Bigtable for this or Datastore is fine? If Datastore is fine is this the ideal way to do it?
Thanks in advance.
Disclosure: I am a product manager for Cloud Bigtable.
If you plan to have a large amount of financial data, covering the entire stock market, Cloud Bigtable is a good choice: it scales to terabytes and petabytes, and you can get low-latency responses to your requests, it is already in use in financial, risk and anti-fraud applications, and natively supports time series via its third dimension. See this blog post and video on how FIS used Cloud Bigtable for their bid on the SEC CAT project.
That said, Cloud Bigtable is strongly consistent in a single cluster, but eventually-consistent if you use replication, so you have to keep that in mind. If your users expect strong consistency, your options are:
use a single cluster instance (replication only within a single zone)
if you use cross-zone replication, route requests to a single cluster via application profiles
consider using a different system which provides strong consistency
Firestore will provide a serverless document database with strong consistency for your applications, so you should consider Firestore if that is important for your use case.
If you want to be able to run SQL queries on your data, consider:
Google Cloud SQL — managed MySQL or PostgreSQL
Google Cloud Spanner — petabyte-scale, transactional database
Google BigQuery — serverless data warehouse
Hope this helps!
As you know, Datastore is implemented using Bigtable. So you can expect them to perform similarly. In term of use case suitability, Datastore, which is going to be replaced by the Firestore soon, is more suitable for storing user or user-session related data. On top of that. Also, Bigtable is explicitly recommended for finance related workloads.
There is a page that is specifically dedicated to selecting the most you can use this page as a guide.
Let's say I have the following text:
Steps toward this goal include: Increasing efficiency of mobile networks, data centers, data transmission, and spectrum allocation Reducing the amount of data apps have to pull from networks through caching, compression, and futuristic technologies like peer-to-peer data transfer Making investments in accessibility profitable by educating people about the uses of data, creating business models that thrive when free data access is offered initially, and building out credit card infrastructure so carriers can move from pre-paid to post-paid models that facilitate investment If the plan works, mobile operators will gain more customers and invest more in accessibility; phone makers will see people wanting better devices; Internet providers will get to connect more people; and people will receive affordable Internet so they can join the knowledge economy and connect with the people they care about.
As you can tell by reading the text, these are multiple sentences (a list of points). How can I split this text into sentences? I've tried using python NLTK but no luck. Checking for uppercase letters won't work either, as it isn't very reliable.
Any ideas on how to solve this problem?
Thanks.
if i understood you correctly this little code could help: (Note tested on python 2.7.5)
paragraph = 'Steps toward this goal include: Increasing efficiency of mobile networks, data centers, data transmission, and spectrum allocation Reducing the amount of data apps have to pull from networks through caching, compression, and futuristic technologies like peer-to-peer data transfer Making investments in accessibility profitable by educating people about the uses of data, creating business models that thrive when free data access is offered initially, and building out credit card infrastructure so carriers can move from pre-paid to post-paid models that facilitate investment If the plan works, mobile operators will gain more customers and invest more in accessibility; phone makers will see people wanting better devices; Internet providers will get to connect more people; and people will receive affordable Internet so they can join the knowledge economy and connect with the people they care about.'
words = []
separators = ['.',',',':',';']
oldValue = 0
for value in range(len(paragraph)):
if paragraph[value] in separators:
words.append(paragraph[oldValue:value+1])
oldValue = value+2
for word in words:
print word
[EDIT]
also you could add uppercase letter check easily with
if paragraph[value] == paragraph[value].upper():
words.append(paragraph[oldValue:value+1])
...
I'm trying to build a python solution where a user can enter a credit card which will be submitted and saved to the Payflow pro servers and can be billed on an on-demand basis. I know python-payflowpro supports recurring billing, but that occurs on a regular schedule, such as weekly or monthly. I'm looking to find a solution that will bill a user's card at their request without them having to enter in their card information.
I've looked through the payflow pro api docs and it looks like there is some feature where you can bill a user's account multiple times if you have the transaction id that payflow pro gives you. However, I'm not sure if this is only so merchants can make adjustments to an existing order (such as the customer wishes to later add an additional item). And I don't think that python-payflowpro supports this.
Has anyone used payflow in this way to store credit cards online and make on-demand payments to them? Is there an existing python api for this, whether it be python-payflowpro or something else? Or do I have to roll my own API for this?
I'm pretty new to payflow, so maybe I'm missing something obvious. Was wondering how other people approached this situation.
Thank you for reading and for your consideration.
Joe
This is the python-payflowpro package that I am currently using:
https://github.com/bkeating/python-payflowpro/blob/master/payflowpro/tests/client.py
Looking closer at the api source code, I found an undocumented function called reference_transaction, which allows reference transactions. It seems to allow you to make use of PayPal's Reference Transactions and allow you to store credit cards online and charge them on an ad-hoc basis.
https://github.com/bkeating/python-payflowpro/blob/master/payflowpro/client.py#L259
With a little digging, I found a way to utilize this api method, but had to do a few tricks to pass in the proper arguments. I've documented them here:
https://github.com/bkeating/python-payflowpro/issues/5
Does anyone know if there is an API for Oneworld Alliance ? The idea is to program in Python a Traveling Salesman, who visits all airports in the system based on actual available flight connections.
I don't think Oneworld Alliance themselves, or other airlines or alliances, have their own APIs. Not sure whether asking this is ontopic to SO.
Try the search engines and booking sites: Travelocity, Expedia, Hotwire, cheaptickets...
For example by Google here's Expedia Affiliate Network.
Kayak apparently used to have a beta API but it was pulled due to misuse.
Not sure how easy it is to scrape Oneworld's site or timetables, I wouldn't start there.
Remember the airlines have a negative incentive to allow their data to be scraped, whereas the search engines have a positive incentive (within reasonable limits). So start with the latter.
When you say "based on actual available flight connections", I presume you just check whether airline X has a route connecting city A to city B, not at actual seat inventory on specific dates and times, which seems needless. Do you need durations and frequencies?
Btw, there are 900 hits on SO on "Traveling Salesman", you might be able to reuse someone else's data.