We Keep Coding

Semeval twitter data download is not working

I want to download the training set of the following dataset: http://alt.qcri.org/semeval2017/task4/index.php?id=data-and-tools
I am required to use a script to download the tweets from their respective ids and the script is on this github: https://github.com/seirasto/twitter_download
When I run the following command in powershell:
python download_tweets_user_api.py --dist input.txt --output output.txt --user
I get
https://alt.qcri.org/semeval2017/task4/
^
SyntaxError: invalid syntax
When trying to run this command:
python download_tweets_api.py --dist=tweeti-a.dist.tsv
I get the following error:
usage: download_tweets_api.py [-h] [--partial PARTIAL] --dist DIST --output OUTPUT
download_tweets_api.py: error: argument --dist: can't open 'tweeti-a.dist.tsv': [Errno 2] No such file or directory: 'tweeti-a.dist.tsv'
What am I doing wrong?

The syntax error can be avoided by removing or commenting the following line 7 of the download_tweets_user_api.py file:
https://alt.qcri.org/semeval2017/task4/
For the second error, use:
python download_tweets_api.py --dist={name-of-file} --output ouput.txt
instead of:
python download_tweets_api.py --dist=tweeti-a.dist.tsv
where {name-of-file} is the filename of the file that you downloaded from https://alt.qcri.org/semeval2017/task4/index.php?id=data-and-tools
Make sure the file is in the same directory where you run the command prompt and where download_tweets_api.py lies.
Assuming you have authorized twitter, change line 19:
MY_TWITTER_CREDS = os.path.expanduser('~/.my_app_credentials')
to:
MY_TWITTER_CREDS = os.path.expanduser('~/.twitter_oauth')
If you have not authorized twitter yet, open command prompt and import twitter package using pip install twitter and then run twitter authorize. Once you are redirected to a page, allow authorization and copy the pin back to command prompt.
Once this is done, your .twitter_oauth file will be created and you will be able to run the command that caused the error.

Save an output file on disk using shell commands in a django app

I am creating a script to run shell commands for simulation purposes using a web app. I want to run a shell command in a django app and then save the output to a file.
The problem I am facing is that when running the shell command, the output tries to get saved in the url that is invoked (for example: localhost:8000/projects) which is understandable.
I want to save the output to for example:
/home/myoutput/output.txt rather than /projects or /tasks
I have to run a whole script and save it's output to the txt file later but that is easy once this is done.
Tried os.chdir() function to change directory to /desiredpath already
from subprocess import run
#the function invoked from views.py
def invoke_mpiexec():
run('echo "this is a test file" > fahadTest.txt')
FileNotFoundError at /projects
Exception Type: FileNotFoundError

First I want to say that directly calling external programs from a web request in Django is a bit of an anti-pattern. The preferred approach is to use a work queue like Celery or rq, but that comes with a bit of added complexity.
That being said, you can solve your problem with the argument shell=True:
from subprocess import run
#the function invoked from views.py
def invoke_mpiexec():
run('echo "this is a test file" > fahadTest.txt', shell=True)
Here is the documentation:
If shell is True, the specified command will be executed through the
shell. This can be useful if you are using Python primarily for the
enhanced control flow it offers over most system shells and still want
convenient access to other shell features such as shell pipes,
filename wildcards, environment variable expansion, and expansion of ~
to a user’s home directory. However, note that Python itself offers
implementations of many shell-like features (in particular, glob,
fnmatch, os.walk(), os.path.expandvars(), os.path.expanduser(), and
shutil).
Note: Using shell=True can lead to security issues:
If the shell is invoked explicitly, via shell=True, it is the
application’s responsibility to ensure that all whitespace and
metacharacters are quoted appropriately to avoid shell injection
vulnerabilities.

You should use subprocess.call with stdout argument
def invoke_mpiexec():
f = open("fahadTest.txt", "w")
subprocess.call(['echo', '"this is a test file"'], stdout=f)
or use write function
def invoke_mpiexec():
f = open('fahadTest.txt', 'w')
f.write("Now the file has more content!")
f.close()

So I figured it out.
Below is the fix:
run('mkdir -p $HOME/phdata/test/ && echo "this is a test file" > $HOME/phdata/test/fahadTest.txt', shell=True)
mkdir -p creates a directory if it doesn't exist
$HOME is used to go to the home directory and from there you can
navigate to folders.
shell=True argument is required to run it as shell command
You can also create a ssh connection and run the commands/ scripts on the remote server. For this, my approach will be to create a script on the remote server, call it through my app and provide arguments to it. Another workaround which is not that good but works is to create a script on the server using the above line and then call it.

Error in check_call() subprocess, executing 'mv' unix command: "Syntax error: '(' unexpected"

I'm making a python script for Travis CI.
.travis.yml
...
script:
- support/travis-build.py
...
The python file travis-build.py is something like this:
#!/usr/bin/env python
from subprocess import check_call
...
check_call(r"mv !(my_project|cmake-3.0.2-Darwin64-universal) ./my_project/final_folder", shell=True)
...
When Travis building achieves that line, I'm getting an error:
/bin/sh: 1: Syntax error: "(" unexpected
I just tried a lot of different forms to write it, but I get the same result. Any idea?
Thanks in advance!
Edit
My current directory layout:
- my_project/final_folder/
- cmake-3.0.2-Darwin64-universal/
- fileA
- fileB
- fileC
I'm trying with this command to move all the current files fileA, fileB and fileC, excluding my_project and cmake-3.0.2-Darwin64-universal folders into ./my_project/final_folder. If I execute this command on Linux shell, I get my aim but not through check_call() command.
Note: I can't move the files one by one, because there are many others
I don't know which shell Travis are using by default because I don't specify it, I only know that if I write the command in my .travis.yml:
.travis.yml
...
script:
# Here is the previous Travis code
- mv !(my_project|cmake-3.0.2-Darwin64-universal) ./my_project/final_folder
...
It works. But If I use the script, it fails.
I found this command from the following issue:
How to use 'mv' command to move files except those in a specific directory?

You're using the bash feature extglob, to try to exclude the files that you're specifying. You'll need to enable it in order to have it exclude the two entries you're specifying.
The python subprocess module explicitly uses /bin/sh when you use shell=True, which doesn't enable the use of bash features like this by default (it's a compliance thing to make it more like original sh).
If you want to get bash to interpret the command; you have to pass it to bash explicitly, for example using:
subprocess.check_call(["bash", "-O", "extglob", "-c", "mv !(my_project|cmake-3.0.2-Darwin64-universal) ./my_project/final_folder"])
I would not choose to do the job in this manner, though.

Let me try again: in which shell do you expect your syntax !(...) to work? Is it bash? Is it ksh? I have never used it, and a quick search for a corresponding bash feature led nowhere. I suspect your syntax is just wrong, which is what the error message is telling you. In that case, your problem is entirely independent form python and the subprocess module.
If a special shell you have on your system supports this syntax, you need to make sure that Python is using the same shell when invoking your command. It tells you which shell it has been using: /bin/sh. This is usually just a link to the real shell executable. Does it point to the same shell you have tested your command in?
Edit: the SO solution you referenced contains the solution in the comments:
Tip: Note however that using this pattern relies on extglob. You can
enable it using shopt -s extglob (If you want extended globs to be
turned on by default you can add shopt -s extglob to .bashrc)
Just to demonstrate that different shells might deal with your syntax in different ways, first using bash:
$ !(uname)
-bash: !: event not found
And then, using /bin/dash:
$ !(uname)
Linux

The argument to a subprocess.something method must be a list of command line arguments. Use e.g. shlex.split() to make the string be split into correct command line arguments:
import shlex, subprocess
subprocess.check_call( shlex.split("mv !(...)") )
EDIT:
So, the goal is to move files/directories, with the exemption of some file(s)/directory(ies). By playing around with bash, I could get it to work like this:
mv `ls | grep -v -e '\(exclusion1\|exclusion2\)'` my_project
So in your situation that would be:
mv `ls | grep -v -e '\(myproject\|cmake-3.0.2-Darwin64-universal\)'` my_project
This could go into the subprocess.check_call(..., shell=True) and it should do what you expect it to do.

windows wget & being cut off

System Info:
Windows 7,
GNU Wget 1.11.4,
Python 2.6
The problem:
Im running a python script that fires a wget shortcut, the problem is that wget (even when run purely in command line from the exe) cuts off '&''s. For example when i run the code below, this is what i get:
C:\Program Files\GnuWin32\bin>wget.exe
http://www.imdb.com/search/title?genres=action&sort=alpha,asc&start=51&title_type=feature
SYSTEM_WGETRC = c:/progra~1/wget/etc/wgetrc syswgetrc = C:\Program
Files\GnuWin32/etc/wgetrc
--2013-01-18 12:48:43-- http://www.imdb.com/search/title?genres=action Resolving
www.imdb.com... 72.21.215.52 Connecting to
www.imdb.com|72.21.215.52|:80... failed: Connection refused.
=alpha,ascThe system cannot find the file specified.
The system cannot find the file 51. 'title_type' is not recognized as an internal or
external command, operable program or batch file.
As you can see, wget counts all text before the '&' as the URL in question, and windows take the last half as a new command(s).
There has got to be some way of allowing wget to capture that whole string as the URL.
Thanks in advance.
EDIT:
When i call the command in command line with brackets around it, it works great, however, when i run the script through python:
subprocess.Popen(['start /B wget.lnk --directory-prefix=' + output_folder + ' --output-document=' + output_folder + 'this.html "http://www.imdb.com/search/title?genres=action&sort=alpha,asc&start=51&title_type=feature"'], shell=True)
I get the following error:
SYSTEM_WGETRC = c:/progra~1/wget/etc/wgetrc
syswgetrc = C:\Program Files (x86)\GnuWin32/etc/wgetrc
"http://www.imdb.com/search/title?genres=action&sort=alpha,asc&start=51&title_ty
pe=feature": Unsupported scheme.

It's not Wget that cuts off the URL, but the command interpreter, which uses & to separate two commands, akin to ;. This is indicated by the =alpha,ascThe system cannot find the file specified. error on the following line.
To prevent this from happening, quote the entire URL:
wget.exe "http://www.imdb.com/search/title?genres=action&sort=alpha,asc&start=51&title_type=feature"

Execute bash script from URL using python

Assume I have a file at http://mysite.com/myscript.sh that contains:
#!/bin/bash
echo "Hello $1"
From the command line, I can execute my script (without downloading it) using the following command:
bash <(curl -s http://mysite.com/myscript.sh) World
Now, instead of executing the above command from the command line, I want to execute it from a python script. I tried doing the following:
import os
os.system('bash <(curl -s http://mysite.com/myscript.sh) World')
...but I get the following error:
sh: -c: line 0: syntax error near unexpected token `('
How do I make this execute correctly in python?

Evidently, os.system runs its command through /bin/sh, which usually causes whichever shell it's linked to to drop to a compatibility mode that doesn't include the <(...) construction. You can get around it by either storing the result in a temporary file or using another level of shell. Ugly, but it works.
os.system('bash -c "bash <(curl -s http://mysite.com/myscript.sh) World"')

There is a libcurl for python so you don't have to go the way around to command line behaviour. Here's the function list that should really do it - have never run remote scripts myself though. If you need installing the python binding, the instructions are here.
import curl