We Keep Coding

scrapy-elasticsearch pipeline only for specific item

I want to use the scrapy-elasticsearch pipeline in my scrapy project. In this project I have different items / models. These items are stored in a mysql server. In addition I want to index ONE of these items in an ElasticSearchServer.
In the documentation, however, I only find the way to index all defined items like in the code example from the settings.py below.
ELASTICSEARCH_INDEX = 'scrapy'
ELASTICSEARCH_TYPE = 'items'
ELASTICSEARCH_UNIQ_KEY = 'url'
Like you can see the ELASTICSEARCH_TYPE shows that alle items have to be indexed. Is there a possibility to limit this to only one item?

The current implementation does not support sending only some items.
You could create a subclass of the original pipeline and override the process_item method to do what you want.
If you have the time, you could also send a pull request upstream with a proposal to allow filtering items before sending them to Elasticsearch.

Use only available SOAP bindings?

Currently using Suds with Python 2.7. Open to trying a different client.
I'm connecting to a SOAP service that returns different documents where only certain fields are available depending on the document. Introspecting the service doesn't help because I just get a list of all available fields rather than those returned on a doc by doc basis. This has led me to some particularly annoying error handling, which currently looks like:
try:
client.service.Something
except AttributeError:
pass
else:
doSomethingWith(client.service.Something)
For every possible field in the WSDL schema.
Alternatively, parsing the response to generate a list of available fields doesn't help much, because then I end up with a long list of
if client.service.Something is not None:
doSomethingWith(client.service.Something)
I suppose I could parse the doc and build a list of available fields, and then do something like:
for field in listOfFields:
doSomethingWith(field)
But I do different things with the field's contents depending on what the field is, so that doesn't help much.
Is there any simple/elegant solution? I can't be the only one to have encountered this.

Tastypie - get user groups resource_uri

I'm trying to update the groups assigned to users via an API (via rest) with Tastypie.
I tried passing the group id's directly in, however it says that the URL provided is not a valid resource. I then tried passing in a URL such as '/api/v1/groups/1/' but that is saying that's not a link to a valid resource.
Any hints? I'm creating user records just fine from a standard django view/form, but I would like to do this as a REST action.

Do you need to specify the format, such as: /api/v1/groups/1/?format=json ?

Google Reader API Listing feed from a single tag or category

I'm writing a python script (in fact a Calibre Recipe) to retrieve all the items under a specific tag or category in my readlist.
I'm able to retrieve the whole items from the category but I'ld like to retrieve items feed-by-feed, so I need a way to list the feeds filed under a specific category (just as Google Reader UI does when you click on a folder).
I'm unable to find a API for doing that.
Any suggestions?
Thanks.

You can use https://www.google.com/reader/api/0/subscription/list to list the user's subscriptions. The folders each subscription is in are given by the categories list. You can append ?output=json to get the list of subscriptions as JSON.

Getting custom tags from AWS instance browser function

I'm trying to get the "name" tag from the AWS instance via their internal browser method. In my batch/script I have the following:
.\wget.exe -O - http://169.254.169.254/latest/meta-data/instance-id > c:\instance.id
Is there a way to get custom tags you create when you spin-up the instance? I wanted to save off some information that I push when I create the instance via boto. I can essentially match up the instance.id with the attribute that I create, in this case, a "name" tag. But there doesn't seem to be a way to get that name tag from within the instance itself.
The above code represents an operation that I run within a batch to grab the instance.id, I want to do the same for a custom "tag".

No, you cannot access the tags using the meta-data provided to the instance. What i recommend is using the user_data property of run_instances. Any data you save in the user_data property you can retrieve by downloading the file http://169.254.169.254/latest/user-data.