I have a python file that calls a function in another directory.
I would like to use the config variable DATA_DIR in the function directly without importing configuration at each time.
The main file looks like this :
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import ConfigParser
config = ConfigParser.ConfigParser()
config.read('static.cfg')
global __DATA_DIR__
__DATA_DIR__ = config.get('Directories', '__DATA_DIR__')
from src.directory import file
file.function()
The function looks like this :
#!/usr/bin/env python
# -*- coding: utf-8 -*-
def function():
global __DATA_DIR__
print (__DATA_DIR__)
The configuration file looks like this :
[Directories]
__DATA_DIR__=/directorie/to/config.cfg
When executing the first main program, I had this error :
NameError: global name 'DATA_DIR' is not defined
Why not pass the config argument to the function. This would need confirmation but I would imagine that only the read method actually reads and parses the actual file, and the config.get method only gives you data from an internal datastructure, so passing the config object and doing a config.get inside the function would be pretty efficient.
so :
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import ConfigParser
from src.directory import file
config = ConfigParser.ConfigParser()
config.read('static.cfg')
file.function( config )
and in your file :
#!/usr/bin/env python
# -*- coding: utf-8 -*-
def function( cfg ):
print ( cfg.get("__DATA_DIR__") )
Since you are importing your function, you should pass in your __DATA_DIR__ variable to it:
from src.directory import file
file.function(__DATA_DIR__)
#your other file
def function(data):
print (data)
Related
I have a python script:
# -*- coding: utf-8 -*-
import pandas as pd
print('Hello World')
I'm trying to run it in my Scala project using a PythonRunner object:
import org.apache.spark.deploy.PythonRunner
import java.io.File
import java.nio.file.Paths
object PythonRunnerApp extends App{
val pyFilePath = this.getClass.getResource("").getPath + "/hello.py"
PythonRunner.main(Array(pyFilePath, "hello.py"))
}
As a result, I get an import error: ImportError: No module named pandas
Traceback (most recent call last):
File "/Users/a19562665/IdeaProjects/PythonRunner/target/scala-2.12/classes//hello.py", line 3, in <module>
import pandas as pd
ImportError: No module named pandas
Exception in thread "main" org.apache.spark.SparkUserAppException: User application exited with 1
at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:103)
at PythonRunnerApp$.runUsingSpark(PythonRunnerApp.scala:15)
at PythonRunnerApp$.delayedEndpoint$PythonRunnerApp$1(PythonRunnerApp.scala:27)
at PythonRunnerApp$delayedInit$body.apply(PythonRunnerApp.scala:8)
at scala.Function0.apply$mcV$sp(Function0.scala:39)
at scala.Function0.apply$mcV$sp$(Function0.scala:39)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
at scala.App.$anonfun$main$1$adapted(App.scala:80)
at scala.collection.immutable.List.foreach(List.scala:431)
at scala.App.main(App.scala:80)
at scala.App.main$(App.scala:78)
at PythonRunnerApp$.main(PythonRunnerApp.scala:8)
at PythonRunnerApp.main(PythonRunnerApp.scala)
Is there any way I can ask PythonRunner to install pandas?
UPD:
Here is another example of running python scripts:
#!/usr/bin/python
# -*- coding: utf-8 -*-
#import pandas as pd
import sys
for line in sys.stdin:
print('Hello, ' + line)
# this is hello.py
And Scala application:
spark.sparkContext.addFile(getClass.getResource("hello.py").getPath, true)
val test = spark.sparkContext.parallelize(List("Body!")).repartition(1)
val piped = test.pipe(SparkFiles.get("./hello.py"))
val c = piped.collect()
c.foreach(println)
Output: Hello, Body!
But the question remains open to me. Can I, as a cluster user, install pandas on workers?
You need to install all necessary Python dependencies across every executor before submitting Spark applications into your cluster
Keep in mind that you're not using Pandas here, and SparkSQL should probably be used instead
I want to write a script that takes a POST (HTTP) request and sends a response. If the POST data has english characters, they are returned/displayed normally. But cyrillic chars look like this:
������ ������ �������������� ������
My Python script is as follows:
# -*- coding: utf-8 -*-
print('Content-type: text/html\n')
import cgi
from imp import reload
form = cgi.FieldStorage()
text2 = form.getfirst("postvar1", "не задано")
print(text2)
I use a custom server script
# -*- coding: utf-8 -*-
from http.server import HTTPServer, CGIHTTPRequestHandler
server_address = ("", 8080)
httpd = HTTPServer(server_address, CGIHTTPRequestHandler)
httpd.serve_forever()
The python script is saved as UTF-8.
How can this problem be solved?
Hi i have a python script which is reading the values defined in ConfigParser sections.I am able to get the values from the sections and putting into a diffrent variable to use it further.
For example,
[path]
DB_CONFIGURATION_PATH:/opt/aor/regression/test_scenario/Config.cfg
[section1]
localhost:127.0.0.1
username:root
password:root
DB_name:test
Script1:
#!/usr/bin/python
import ConfigParser
configParser = ConfigParser.RawConfigParser()
configParser.read('/opt/aor/regression/test_scenarios/Config.cfg')
dbconfig_path = configParser.get("Regression_script_path","DB_CONFIGURATION_PATH")
Script2:
#!/usr/bin/pythoni
import os
import sys
#import ReadConfigurationFile
from ReadConfigurationFile import *
db_path=dbconfig_path
config = ConfigParser.ConfigParser()
config.read('db_path')
host=config.get("section1","localhost")
print host
This is throwing error as :
ConfigParser.NoSectionError: No section: 'section1'
Can you tell me what is the problem here?
I have a encoding problem, When I try to crawl youtube (arabic channel) :
#!/usr/bin/python
# -*- coding: utf8 -*-
from django.core.management.base import BaseCommand, CommandError
import requests, lxml, re
from lxml import html
class Command(BaseCommand):
def handle(self, *args, **options):
r = requests.get("https://www.youtube.com/user/aljazeerachannel/videos?view=0")
root = lxml.html.fromstring(r.content)
for data in root.xpath('.//*[#id="branded-page-body"]/div/div/div[1]/div/div[2]/ul/li[1]/span/span/a'):
print data.text
The result is :
[root#vmi9105 buzzbal]# python manage.py youtube
اÙتخابات اÙÙجاÙس اÙبÙدÙØ© Ù٠سÙØ·ÙØ© عÙÙاÙ
try this it sloved my problem in python:
f"{yourString}".encode('latin-1').decode("utf-8")
I want the process i initiate through the script to run on webserver even if user closes the page.
It doesn't seem to be working with this:
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
import cgi,cgitb,subprocess
print "Content-Type: text/plain;charset=utf-8"
print
form = cgi.FieldStorage()
ticker = form['ticker'].value
print subprocess.Popen(['/usr/bin/env/python','options.py',ticker])
Please help! Thanks!
I guess this is wrong:
'/usr/bin/env/python'
it should be usually:
'/usr/bin/env python'
but better use this:
>>> import sys
>>> sys.executable # contains the executable running this python process
'C:\\Python27\\pythonw.exe'
I use to do it like this:
p = subprocess.Popen([sys.executable,'options.py',ticker])