The following Python code (from Read in parallel and write sequentially?) processes (reads) the data for ID 0 to N. There readers run in parallel and it writes the result in the order of ID. How to implement it in C# efficiently?
async def reader(q, id_to_data, data_ready): # Multiple readers are run in parallel
while True:
id = await q.get()
data = await read_async(id)
id_to_data[id] = data
data_ready.set()
async def main():
q = asyncio.Queue()
for id in range(1000):
await q.put(id)
id_to_data = {}
data_ready = asyncio.Event()
readers = [asyncio.create_task(reader(q, id_to_data, data_ready))
for _ in range(3)]
for id in range(1000):
while True:
# wait for the current ID to appear before writing
if id in id_to_data:
data = id_to_data.pop(id)
await data.write_async(f'{id}.csv')
break
# move on to the next ID
else:
# wait for new data and try again
await data_ready.wait()
data_ready.clear()
for r in readers:
r.cancel()
C#:
void main() {
List<int> ids = get_IDs();
ProcessAllIDs(ids);
}
async Task ProcessAllIDs(List<int> ids) {
// A low efficient implementation
for (var id in ids) {
var result = await Process1(id); // Can be run concurrently
await Process2(result); // Need to be run sequentially
}
}
A simple way would be to create a List of Tasks and then await in them in sequence:
var list = new List<Task<int>> { Calculate(2), Calculate(7), Calculate(19) };
foreach(var task in list)
{
Console.WriteLine(await task);
}
static async Task<int> Calculate(int x)
{
await Task.Delay(200);
return x*x;
}
This will start 3 Tasks that calculate stuff simultaneously and then print the results in order.
So for your example, it would look like this:
async Task main() {
List<int> ids = get_IDs();
await ProcessAllIDs(ids);
}
async Task ProcessAllIDs(List<int> ids) {
var tasks = ids.Select(id => Process1(id)).ToList();
for (var t in tasks) {
await Process2(await t); // Need to be run sequentially
}
}
Related
My Discord bot allows users to play a song starting from a timestamp.
The problem is that playback is delayed and audio plays faster and is jumbled if start times >= 30s are set.
Results from testing different start times. Same URL, 30 second duration:
Entered Start Time (s)
Playback Delay (s)
Song Playback Time (s)
0
3
30
30
10
22
60
17
17
120
31
2
150
120
<1
I am setting the start time using ffmpeg_options as suggested in this question.
Does anyone understand why the audio playback is being delayed/jumbled? How can I improve playback delay and allow users to start in the middle of a multi-chapter YouTube video?
Code:
import discord
import youtube_dl
import asyncio
# Suppress noise about console usage from errors
youtube_dl.utils.bug_reports_message = lambda: ""
ytdl_format_options = {
"format": "bestaudio/best",
"outtmpl": "%(extractor)s-%(id)s-%(title)s.%(ext)s",
"restrictfilenames": True,
"noplaylist": False,
"yesplaylist": True,
"nocheckcertificate": True,
"ignoreerrors": False,
"logtostderr": False,
"quiet": True,
"no_warnings": True,
"default_search": "auto",
"source_address": "0.0.0.0", # Bind to ipv4 since ipv6 addresses cause issues at certain times
}
ytdl = youtube_dl.YoutubeDL(ytdl_format_options)
class YTDLSource(discord.PCMVolumeTransformer):
def __init__(self, source: discord.AudioSource, *, data: dict, volume: float = 0.5):
super().__init__(source, volume)
self.data = data
self.title = data.get("title")
self.url = data.get("url")
#classmethod
async def from_url(cls, url, *, loop=None, stream=False, timestamp = 0):
ffmpeg_options = {
"options": f"-vn -ss {timestamp}"}
loop = loop or asyncio.get_event_loop()
data = await loop.run_in_executor(None, lambda: ytdl.extract_info(url, download=not stream))
if "entries" in data:
# Takes the first item from a playlist
data = data["entries"][0]
filename = data["url"] if stream else ytdl.prepare_filename(data)
return cls(discord.FFmpegPCMAudio(filename, **ffmpeg_options), data=data)
intents = discord.Intents.default()
bot = discord.Bot(intents=intents)
#bot.slash_command()
async def play(ctx, audio: discord.Option(), seconds: discord.Option(), timestamp: discord.Option()):
channel = ctx.author.voice.channel
voice = await channel.connect()
player = await YTDLSource.from_url(audio, loop=bot.loop, stream=True, timestamp=int(timestamp))
voice.play(player)
await asyncio.sleep(int(seconds))
await voice.disconnect()
token = token value
bot.run(token)
I want to run .py file from my C# project, and get the result. The python script is making an API request, and returns an auth_key token, which I want to use in my C# code. The only problem is that, for some reason the C# code doesn't wait for the process to finish, and thus that not every account has auth_key. Here is my C# code.
private static void GenerateTokens()
{
var url = ConfigurationManager.AppSetting[GeSettingsNode() + ":ip"];
for (int i = 0; i < accounts.Count; i++)
{
ProcessStartInfo start = new ProcessStartInfo();
start.FileName = ConfigurationManager.AppSetting["PythonPath"];
start.Arguments = string.Format($"python_operation_processor.py {accounts[i].client_key_id} {accounts[i].key_sercret_part} {url}");
start.UseShellExecute = false;
start.RedirectStandardOutput = true;
Process process = Process.Start(start);
using (StreamReader reader = process.StandardOutput)
{
accounts[i].auth_key = reader.ReadToEnd().Trim();
}
}
}
And here is my Python script ( python_operation_processor.py )that's making the API requests.
if __name__ == '__main__':
client_key_id = sys.argv[1]
client_secret = sys.argv[2]
API_URL = sys.argv[3]
nonce = str(uuid.uuid4())
d = datetime.datetime.now() - datetime.timedelta(hours=3)
timestamp = d.strftime('%Y-%m-%dT%H:%M:%S.%f')[:-3] + 'Z'
signature = b64encode(hmac.new(b64decode(client_secret), msg=bytes(client_key_id + nonce + timestamp, 'utf-8'),
digestmod=hashlib.sha256).digest()).decode('utf-8')
r = requests.post(API_URL + '/v1/authenticate',
json={'client_key_id': client_key_id, 'timestamp': timestamp, 'nonce': nonce,
'signature': signature})
if r.status_code != 200:
raise Exception('Failed to authenticate: ' + r.text)
auth_token = r.json()['token']
print(auth_token)
Do you have any idea, how I can wait for the execution of every process, and get the token for every account ?
I recently created something similar and ended up with this because, whilst waiting for the process is easy, it is tricky to get the output stream filled correctly.
The method presented also allow you to display the output into a textblock or similar in your application.
If you use it like this, the token will be written to the StringBuilder, and used as return value.
private async Task<string> RunCommand(string fileName, string args)
{
var timeoutSignal = new CancellationTokenSource(TimeSpan.FromMinutes(3));
ProcessStartInfo start = new ProcessStartInfo();
start.FileName = fileName;
start.Arguments = string.Format("{0}", args);
start.RedirectStandardOutput = true;
start.RedirectStandardError = true;
start.UseShellExecute = false;
start.CreateNoWindow = true;
var sb = new StringBuilder();
using (Process process = new Process())
{
process.StartInfo = start;
process.OutputDataReceived += (sender, eventArgs) =>
{
sb.AppendLine(eventArgs.Data); //allow other stuff as well
};
process.ErrorDataReceived += (sender, eventArgs) => {};
if (process.Start())
{
process.EnableRaisingEvents = true;
process.BeginOutputReadLine();
process.BeginErrorReadLine();
await process.WaitForExitAsync(timeoutSignal.Token);
//allow std out to be flushed
await Task.Delay(100);
}
}
return sb.ToString();
}
To render this to a textblock in a UI application, you'll need to:
implement an event which signals a new line has been read, which means forwarding the process.OutputDataReceived event.
if your thinking about a live feed, make sure you flush the stdio buffer in python setting flush to true: print(""hello world"", flush=True)
If you're using an older .net version; you can implement the WaitForExitAsync as described here: https://stackoverflow.com/a/17936541/2416958 as an extention method:
public static class ProcessHelpers
{
public static Task<bool> WaitForExitAsync(this Process process, TimeSpan timeout)
{
ManualResetEvent processWaitObject = new ManualResetEvent(false);
processWaitObject.SafeWaitHandle = new SafeWaitHandle(process.Handle, false);
TaskCompletionSource<bool> tcs = new TaskCompletionSource<bool>();
RegisteredWaitHandle registeredProcessWaitHandle = null;
registeredProcessWaitHandle = ThreadPool.RegisterWaitForSingleObject(
processWaitObject,
delegate(object state, bool timedOut)
{
if (!timedOut)
{
registeredProcessWaitHandle.Unregister(null);
}
processWaitObject.Dispose();
tcs.SetResult(!timedOut);
},
null /* state */,
timeout,
true /* executeOnlyOnce */);
return tcs.Task;
}
}
I am new at python so sry if question is stupid but i hope u guys will help me.
Bot not updating status every 5 sec(i put more time like 5 min and it didn't work too). It shows number of servers and not changing to second status.
from discord.ext import commands, tasks
from itertools import cycle
#tasks.loop( seconds = 12 )
async def changeStatus():
status = cycle( [f' on { len(client.guilds) } servers', '~help'] )
await client.change_presence( activity = discord.Activity( type = discord.ActivityType.playing, name = next(status) ) )
#client.event
async def on_ready():
print( 'bot connected' )
changeStatus.start()
Since
status = cycle( [f' on { len(client.guilds) } servers', '~help'] )
is called everytime you call your function, it will be reinterpreted, which means that the next() function always return the first element. To fix this, you will need a different approach. For example, create a global iteration-variable and declare you cycle-list as only a list.
iterationPosition = 0
#tasks.loop( seconds = 12 )
async def changeStatus():
status = [f' on { len(client.guilds) } servers', '~help']
await client.change_presence(activity=discord.Activity(type=discord.ActivityType.playing, name=status[iterationPosition]))
iterationPosition = 0 if (iterationPosition == len(status) - 1) else (iterationPosition + 1)
You will need to keep track if you've reached the end of your list. This is done by the last line of code.
It is not changing the status as you got a Presence Update rate-limit of 5/60secs per-session. These rate-limits are dynamic so don't get too close to them
status = cycle( [f' on { len(client.guilds) } servers', '~help'] ) This line should be outside of the changeStatus function as whenever it is called, it makes [f' on { len(client.guilds) } servers the first item.
I am trying to develop a Flutter app for audio fingerprints processing. I am using Starflut for python integration. Here is simple example:
//python file for using from dart
def tt(a,b) :
print (a, b)
return 666,777
g1 = 123
def yy(a,b,z) :
print(a,b,z)
return {'jack': 4098, 'sape': 4139}
class Multiply :
def __init__(self):
pass
def multiply(self, a,b):
print("multiply....",a,b)
return a * b
//dart code which uses python
void _initStarCore() async {
StarCoreFactory starcore = await Starflut.getFactory();
StarServiceClass service = await starcore.initSimple("test", "123", 0, 0, []);
srvGroup = await service["_ServiceGroup"];
bool isAndroid = await Starflut.isAndroid();
if (isAndroid == true) {
String libraryDir = await Starflut.getNativeLibraryDir();
String docPath = await Starflut.getDocumentPath();
if (libraryDir.indexOf("arm64") > 0) {
Starflut.unzipFromAssets("lib-dynload-arm64.zip", docPath, true);
} else if (libraryDir.indexOf("x86_64") > 0) {
Starflut.unzipFromAssets("lib-dynload-x86_64.zip", docPath, true);
} else if (libraryDir.indexOf("arm") > 0) {
Starflut.unzipFromAssets("lib-dynload-armeabi.zip", docPath, true);
} else {
Starflut.unzipFromAssets("lib-dynload-x86.zip", docPath, true);
}
await Starflut.copyFileFromAssets("python3.6.zip",
"flutter_assets/starfiles", null);
}
await srvGroup.initRaw("python36", service);
String resPath = await Starflut.getResourcePath();
srvGroup.loadRawModule("python", "",
resPath + "/flutter_assets/starfiles/" + "testpy.py", false);
dynamic python = await service.importRawContext("python", "", false, "");
StarObjectClass retobj = await python.call("tt", ["hello ", "world"]);
print(await retobj[0]);
print(await retobj[1]);
print(await python["g1"]);
StarObjectClass yy = await python.call("yy", ["hello ", "world", 123]);
print(await yy.call("__len__",[]));
StarObjectClass multiply = await service.importRawContext("python", "Multiply", true, "");
StarObjectClass multiply_inst = await multiply.newObject(["", "", 33, 44]);
print(await multiply_inst.getString());
print(await multiply_inst.call("multiply", [11, 22]));
await srvGroup.clearService();
await starcore.moduleExit();
}
Now I need to import python library Dejavu for audio fingerprinting, but I do not know how to do that. There is nothing about it in starflut documentation or issues in their repository.
Has somebody faced same problem? Or maybe somebody has any suggestion how i can try to solve it?
Sorry for mistakes, hope the text is understandable:) English is not my native language.
Did you read the repo readme and installation readme file?
If not, try this:
In your command prompt:
pip install PyDejavu
In the module where you need to import Dejavu:
from dejavu import Dejavu
Have you tried chaquopy plugin for flutter, as it supports 90% of the python packages that you can use and integrate inside your flutter app.
I have a task wherein I have to make 700 REST API calls. I have used a loop, but the run time of the loop is more than the timeout of AWS Lambda service. Is there any way I can make the calls concurrently and merge the results. I am using node JS, but a solution in python is also welcome
Here is a sample of the code I am running now:
HitBTCData = {}
for item in exchanges:
itemlist = item.split('-')
response = requests.get('https://min-
api.cryptocompare.com/data/histominute?
fsym='+itemlist[0]+'&tsym='+itemlist[1]+'&limit=2000&aggregate=1
&e=HitBTC').json()
if itemlist[0]+itemlist[1] not in HitBTCData:
HitBTCData[itemlist[0]+itemlist[1]] = []
HitBTCData[itemlist[0]+itemlist[1]].append(response['Data'])
while response['Type'] != 1:
time = response['TimeFrom']
response = requests.get('https://min-
api.cryptocompare.com/data/histominute?
fsym='+itemlist[0]+'&tsym='+itemlist[1]+
'&limit=2000&aggregate=1&toTs='+str(time)+'&e=HitBTC').json()
if response['Type'] != 1:
HitBTCData[itemlist[0]+itemlist[1]]
.append(response['Data'])
HitBTCData[itemlist[0]+itemlist[1]] =
HitBTCData[itemlist[0]+itemlist[1]][::-1]
Use a loop in Node.js and the calls will run concurrently, but to make sure all of your calls are processed before exiting the lambda, you need to have some form of counter to make sure all of the queries have ended.
example:
var request = require('request');
var results = [];
var runningQueries = 0;
exchanges.forEach(function (item) {
++runningQueries;
callAPI (item, function (err, data) {
if (err) {
results.push(err)
--runningQueries;
} else {
results.push(data)
--runningQueries;
}
if (runningQueries == 0) {
// all call results have been collated, so do something
}
})
})
You'll need to build your own callAPI function. Also, I'm being a bit lazy with what I'm pushing to the array here, but it's just to give you a framework.
You could very easily do this with Promise.all():
const urls = [...];
const requestFunction = async (url) => {
// all your ifs go here
let data = await request(url);
// transform/sanitize your data
return data;
}
const allResults = await Promise.all(urls.map(requestFunction));
// you now have an array of results
The only downside to Promise.all() is that if any of the promises rejects, everything fails. So I would suggest catching errors and resolving with null or something similar.
You could do a for loop to generate your array of URLs, new Array(700).map(...), or since it looks like you already have some kind of array, just .map() that directly and do more transforms inside the request function.