I've created a package to do the following:
Truncate Staging Tables -> Run Python Script (this pulls data from an API that populates the staging tables) -> run 3 execute SQL tasks to move last 5 days of data from staging to prod
What I'm trying to do is:
Create Parameters/Variables to allow someone to change the URL, Username, and Password to be editable in SSMS. I'm fairly new to Python and SSIS, so I don't really know what I'm doing; but, this is the line of code that I would be looking to have the variables.
rTotal = requests.get('https://jira.tpp.company.com/rest/zapi/latest/zql/executeSearch?zqlQuery=creationDate%20>%3D%20startOfDay(-5d)&offset=0&maxRecords=1', auth=HTTPBasicAuth(uid, pwd) , verify=False).json()
How would I go about doing this? For the URL, I don't know if it would be better to do something like: 'https://jira.tpp.company.com/rest/zapi/latest/zql/executeSearch?zqlQuery=' + #[User::url]
if thats even how you would do it. For the uid and pwd, can I create variables/parameters like #[User::uid] & #[User::pwd] to replace uid and pwd? Do I need to wrap it in anything or define it somewhere?
Thank in advance for any help you may provide.
I'm not 100% sure that I understand you, but I will try to answer anyway.
My suggestion would be that you create multiple variables and with these variables make a new one, the final variable with all data.
Check screenshot and see what is my suggestion:
As you see, I marked with red squares where I put variables. Also, at bottom of picture check how I created variables.
With this usage of variables, you can change value when you run package!
Here is expression for rTotal:
"requests.get('https://jira.tpp.tsysecom.com/rest/zapi/latest/zql/executeSearch?zqlQuery=" + #[User::URL] +
"', auth=HTTPBasicAuth(" + #[User::uid] + ", " + #[User::pwd] + ") , verify=False).json()"
I'm assuming you're executing this script as an Execute Process Task, with the path to your python.exe in the Executable field and the path to the script itself in the Arguments field.
There's no built-in way to make your SSIS variables available to the Python script the way you could with a C# or VB script task, but you can pass them in at the end of the Arguments field and then grab them using sys.argv as if they were command line arguments.
The way you add the variables to Arguments is in the Expressions tab of the Execute Process Task Editor window:
On that tab, there's a "Misc" header and a blank Expressions field. Click into the field and a "..." button will show up. Click that
In the new window that appears, select Arguments as the Property, then click the "..." in this window's Expression field
Here, you need to set up the argument, including both your filepath and the variables you want to pass in, using the SSIS Expression syntax. Note that you'll need to escape "s and \s with a \ before each. You're shooting for something like this:
"\"C:\\your_file_path_here\\your_script.py\" \"" + #[User::url] + "\" " + #[User::uid] + " " + #[User::pwd]
You might need to quote your UID/PW fields as well depending on what characters are allowed in them.
Once you have that set up, those variables will pass into your script the same way any other command line argument would. You can tack an "import sys" at the top of your script and use sys.argv[0] for the URL, sys.argv[1] for the UID, and sys.argv[2] for the PW.
Related
I am having a problem using a path variable (that contains a space) in a Python subprocess command.
This should be so simple, yet I have wasted almost 3 hours trying to get the full value to work?
HELP!
Can you change repo to:
repo='/Users/derekm/"BGGoPlan Home"/"99.0 Repo"/Response/response-dashboard'
That should fix your issue.
If you'd like to programmatically solve this issue then I recommend doing the following:
repo_loc = repo_loc.replace(" ", "\ ")
Just before your sp2 = ... line.
I'm sending I'm receiving a JSON message through MQTT in Python, and I would like to start a command line program with what the JSON gives as variables.
The problem with this is that I don't know what values are going to come through and thus this is where I have trouble.
The easiest would be if I knew all the variables that would come through and do something like this:
data = json.loads(msg.payload)
os.system("'command +f ' + data[arg1] + ' +g ' + data[arg2]")
But as mentioned previously, I don't know if they are being passed through, and as such, how can I break it down so that the command line command is build up?
Maybe:
if 'arg1' in data:
command = "+f " + data[arg1]
else:
pass
if 'arg2' in data:
command + "+g " + data[arg2]
else:
pass
Would this work? Is there a better idea?
You can use a for loop to iterate over the json, and construct the command string.
commandArgs = ["+f ","+g "]
commandCount=0
for element in data:
command= command + commandArgs[commandCount] + element
commandCount = commandCount +1
Although you could do this as described it's not something you should do. Running user-inputted commands is one of the most unsecure things a program can do. Scrubbing the commands thoroughly is possible but quite difficult to do comprehensively. The usual approach is to have a table of acceptable commands, match against the table, and then use the entries from that table to populate the command line. Nothing typed by the user ever makes it into the command line with that method.
If you do wish to take user input directly, be extremely careful about scrubbing all special characters, characters outside your preferred locale, double-byte characters, path delimiter characters, etc. Perhaps you could start with the snippet Jeff provided and add a lot of data scrubbing code.
Also, be aware that the probability that whatever you do not code for will eventually be submitted for processing corresponds roughly to the risk of that command. For example, if you fail to catch and remove cat ~/.ssh/* there's a moderately good chance one of your users will execute it or someone will break in and do so. But if you do not catch and remove rm -r /* the chance someone will submit that command approaches certainty.
I am new to python scripting and am required to build a script for WebLogic Server. The requirement is to use a file (type .rpd) from assigned location for deployment. The file type would remain same throughout, however, the file name would change.
Given this condition, I tried the below methods on assigning file to the variable. Both the below methods are not working.
rpdfile = "*.rpd"
rpdlocation = "/u02/RPD_Upload/"+rpdfile+
or
rpdlocation = "/u02/RPD_Upload/*.rpd
I get invalid syntax error while running the python. Experts please help me understand on assigning such variables.
For your first method, you are not appending correctly. You have an extra '+' at the end of the line. It should look like this:
rpdfile = "*.rpd"
rpdlocation = "/u02/RPD_Upload/" + rpdfile
For your second method, you are not closing the double-quote. See below:
rpdlocation = "/u02/RPD_Upload/*.rpd"
Complete newbie question.
I am developing a class for searching a movie database using an API.
Here is how you make a query using the movie title and year from within the Python interpreter:
import moviesearch
movie_db = moviesearch.MovieSearch()
result = movie_db.search('Clockwork Orange', year='1971')
print result
Doing this particular query yields one result from the service I am using.
To save typing for testing, I have created the following script m.py:
from sys import argv
import moviesearch
script, movie_name, params = argv
movie_db = moviesearch.MovieSearch()
result = movie_db.search(movie_name, params)
print result
When I execute this script like so:
python m.py 'Clockwork Orange', year='1971'
the service yields two results.
So this means that there is something wrong with the way I am formatting my parameters when I am testing the class with the script. Can someone please tell me what I am doing wrong?
I am not including the class for querying the movie database because I don't think that's necessary for figuring out what's wrong. What I need to know is how to properly format my parameters when using the script so that it executes exactly as it does in the example above where I am using the class directly from the Python interpreter.
Thanks in advance for any help.
When you do this in the shell (I'm assuming a *nix platform with an sh-compatible shell; Windows is probably slightly different):
python m.py 'Clockwork Orange', year='1971'
The sys.argv list will be:
['m.py', 'Clockwork Orange,', 'year=1971']
So, when you do this:
script, movie_name, params = argv
You end up with movie_name set to the string 'Clockwork Orange,' (with an extra comma), and params set to the string 'year=1971'. So, your call is ultimately:
result = movie_db.search('Clockwork Orange,', 'year=1971')
This is not the same as:
result = movie_db.search('Clockwork Orange', year='1971')
So, how do you get what you want?
Well, you can parse the params, with code something like this:
params = dict(pair.split('=') for pair in argv[2:])
result = movie_db.search(movie_name, **params)
The first line will give you a dict with {'year': '1971'}. Then you can use the ** syntax to forward that dict as the keyword arguments to the function.
Finally, you should take out the extra comma in the arguments you use to run the script. (You could write code to strip it, but it seems like it's just a typo, so the better fix is to not make that typo.)
I'm not sure if function is the word I am looking for. In fact I don't really know what I'm saying but I have some code and it's not quite doing it what I want to. Basically I want to copy and paste this code I've got and email it to someone. I want them to be able to simply copy and paste it into their Terminal and perform calculations.:
## SCSAC.py
def round(x, base=5):
return int(base * round(float(x)/base))
option = 'yes'
while (option == 'yes'):
x=float(raw_input('How many accumulated orders do you have from retailers: '));
y=float(raw_input('How many units are in the inventory: '));
z=float(raw_input('How many accumulated orders have you placed: '));
print 'Place an order of %s units' % round(((x / 25 + y / 10 + z / 25) + 115));
print ;
option=raw_input("Do you wish to calculate another order? (Enter 'yes' to continue or any other key to quit):: ");
print
Whenever I type this code in line for line, it works perfectly. That's because there are basically 3 seperate things happening here.
I define "round" which rounds a value to the nearest 5.
I define an option to loop the program upon completion
I define the actual program, and in that I round the answer and conclude with the option to loop. You may notice 2 print's that don't print anything, but they are only there to have blank lines.
When I copy and paste it, I get a syntax error.
I am not a programmer and I have just been playing with this all day. I just want to know how I can edit this code so it is copy/paste-able and will run the way it is supposed to.
Try using IPython instead of the regular Python interpret at the shell. With IPython, you can type %cpaste, and then paste a whole chunk of code, which it will execute for you step by step, saving the intermediate variables into working memory.
If you insist on pasting it in the regular interpreter, then do it line by line, and take special care for the indentations. The indentations are usually where paste syntax errors come from.
Even better, use Emacs. Then you can just save the pasted code into a file, like test.py, type M-x shell, and then python test.py to quickly run it. Or, if you saved it to a file like test.py then in IPython you can also type %run "test.py" and it will run the code and again store intermediate variables into working memory.
If you're copy/pasting this after making changes more than once or twice, just save it to a file and run it like a script.
You can save the code in a file, and run it using Python.
You can add #!/usr/bin/env python to the beginning of the file, so it can run on *nix systems (if you have execute permission).
Or, you can do python SCSAC.py and run your code. This works on all systems (AFAIK). You can email the file to your person, and she can run it using Python. This seems to be the easiest way to do it.