We Keep Coding

Linux: SyntaxError for error-free Python script

I am trying to run a python file that works fine on my Windows machine on a remote server:
import collections
import pandas as pd
from pathlib import Path
import shelve
import cloudpickle
import numpy as np
import typing
from typing import List
import ray
from ray.rllib.agents import ppo
from ray.rllib.utils.spaces.space_utils import flatten_to_single_ndarray
from ray.tune import register_env
from ray.rllib.env.base_env import _DUMMY_AGENT_ID
from ray.rllib.policy.sample_batch import DEFAULT_POLICY_ID
.
.
.
if __name__ == '__main__':
WIN = False
ray.init(dashboard_port=8263)
daterange = pd.date_range('2017-01-01', periods=35040, freq='15T')
norming_factor = 10
actions_module = ActionsModuleContinuous()
batch_n_days = 1
kappa = 1000
seed = 1234
steps_per_episode = batch_n_days * 24 * 4
num_episodes = 5
device_config: List = list()
device_config.append(Device1(WIN=WIN, norming_factor=norming_factor, n_data_points=steps_per_episode))
device_config.append(Device2(WIN=WIN))
device_config.append(Device3(WIN=WIN))
It returns the following SyntaxError, for no apparent reason:
(venv) [<username>#<server> examples]$ python test.py
File "test.py", line 183
device_config: List = list()
^
SyntaxError: invalid syntax
I have tried deleting : List, which just had the effect of moving the same error to a seemingly arbitrary place further down the script. Any help is greatly appreciated.

device_config: List = list()
That syntax is adding type annotations for variable device_config.
If you get SyntaxError: invalid syntax exception, it means that your python interpreter used to run your code is not new enough. Nothing to do with your code, it just means that when you enter python in your shell, interpreter that gets executed is not new enough.

Handling wildcard expansion with python subprocess call

I'm calling this function and using %s*silent to read files that have names with the following format: name.number.silent.
I get the name from start_model.split('/')[-1].split('.')[0] so don't worry about it.
This is obviously not working because these commands are actually never passed to the shell. If I were to use glob, how can I modify my code to do what I'm doing below?
from subprocess import call
def fragment_score(rosetta_path, silent_input_and_score_output, start_model):
call([rosetta_path,
'-mode score',
'-in::file::silent', '%s/%s*silent' % (silent_input_and_score_output, start_model.split('/')[-1].split('.')[0]),
'-scorefile', '%s/scores1' % silent_input_and_score_output,
'-n_matches', '50'])

Use the Python glob module to generate a list of glob results, and splice it into your argument list at the same position where you would otherwise have a shell replacing a glob expression with the list of associated matches:
from subprocess import call
from glob import glob
def fragment_score(rosetta_path, silent_input_and_score_output, start_model):
glob_exp = '%s/%s*silent' % (silent_input_and_score_output, start_model.split('/')[-1].split('.')[0])
glob_results = glob(glob_exp)
call([rosetta_path,
'-mode score',
'-in::file::silent'
] + glob_results + [
'-scorefile', '%s/scores1' % silent_input_and_score_output,
'-n_matches', '50'])
In current Python 3.x, there's syntax that makes this a bit more natural:
call([rosetta_path,
'-mode score',
'-in::file::silent',
*glob_results,
'-scorefile', '%s/scores1' % silent_input_and_score_output,
'-n_matches', '50'])

Python subprocess script failing

Have written the below script to delete files in a folder not matching the dates in the "keep" period. Eg. Delete all except files partly matching this name.
The command works from the shell but fails with the subprocess call.
/bin/rm /home/backups/!(*"20170920"*|*"20170919"*|*"20170918"*|*"20170917"*|*"20170916"*|*"20170915"*|*"20170914"*)
#!/usr/bin/env python
from datetime import datetime
from datetime import timedelta
import subprocess
### Editable Variables
keepdays=7
location="/home/backups"
count=0
date_string=''
for count in range(0,keepdays):
if(date_string!=""):
date_string+="|"
keepdate = (datetime.now() - timedelta(days=count)).strftime("%Y%m%d")
date_string+="*\""+keepdate+"\"*"
full_cmd="/bin/rm "+location+"/!("+date_string+")"
subprocess.call([full_cmd], shell=True)
This is what the script returns:
#./test.py
/bin/rm /home/backups/!(*"20170920"*|*"20170919"*|*"20170918"*|*"20170917"*|*"20170916"*|*"20170915"*|*"20170914"*)
/bin/sh: 1: Syntax error: "(" unexpected
Python version is Python 2.7.12

Just as #hjpotter said, subprocess will use /bin/sh as default shell, which doesn't support the kind of globbing you want to do. See official documentation. You can change that using the executable parameter to subprocess.call() with a more appropriate shell (/bin/bash or /bin/zsh for example): subprocess.call([full_cmd], executable="/bin/bash", shell=True)
BUT you can be a lot better served by Python itself, you don't need to call a subprocess to delete a file:
#!/usr/bin/env python
from datetime import datetime
from datetime import timedelta
import re
import os
import os.path
### Editable Variables
keepdays=7
location="/home/backups"
now = datetime.now()
keeppatterns = set((now - timedelta(days=count)).strftime("%Y%m%d") for count in range(0, keepdays))
for filename in os.listdir(location):
dates = set(re.findall(r"\d{8}", filename))
if not dates or dates.isdisjoint(keeppatterns):
abs_path = os.path.join(location, filename)
print("I am about to remove", abs_path)
# uncomment the line below when you are sure it won't delete any valuable file
#os.path.delete(abs_path)

Reading TDMS files in python_ how to use tdmsinfo command?

I would like to know what is the content of a tdms file, which is produced by Labview.
Following this site, I write in Python:
import numpy as np
from nptdms import TdmsFile
from nptdms import tdms
#read a tdms file
filenameS = "RESULTS.tdms"
tdms_file = TdmsFile(filenameS)
tdmsinfo [--properties] tdms_file
I receive the following error:
tdmsinfo [--properties] tdms_file
^
SyntaxError: invalid syntax
I do not how to fix it.
Thank you for your help :)

What you are looking for is:
First create a TMDS objet from file:
tdms_file = TdmsFile("C:\\Users\\XXXX\\Desktop\\xx Python\\XXXX.tdms")
then get the group names with:
tdms_groups = tdms_file.groups()
after you can figure out which group names you have into the file, just write
tdms_groups
It will print the following:
['Variables_1', 'Variables_2', 'Variables_3', 'Variables_4', etc..]
With group names now u will be able to get channels with the following:
tdms_Variables_1 = tdms_file.group_channels("Variables_1")
Next print your channels contain into that group:
tdms_Variables_1
It will show:
[ TdmsObject with path /'Variables_1'/'Channel_1', TdmsObject with path /'Variables_1'/'Channel_2', etc..]
At the end get the vectors and its data:
MessageData_channel_1 = tdms_file.object('Variables_1', 'Channel_1')
MessageData_data_1 = MessageData_channel_1.data
Check your data
MessageData_data_1
do stuff with your data!
cheers!

To loop over all properties from the root object try this:
#read a tdms file
filenameS = "RESULTS.tdms"
tdms_file = TdmsFile(filenameS)
root_object = tdms_file.object()
# Iterate over all items in the properties dictionary and print them
for name, value in root_object.properties.items():
print("{0}: {1}".format(name, value))
That should give you all properties names.

Your problem seems to be that tdmsinfo will not work inside a Python script as it is not a python command: it's "a command line program".
The solution is to either use 'tdmsinfo' from a windows shell, or make a wrapper in python so that it runs the command in a subprocess for you. For instance in Python3 with the subprocess package
import subprocess
tdmsfile='my_file.tdms'
# startup info to hide the windows shell
si = subprocess.STARTUPINFO()
si.dwFlags |= subprocess.STARTF_USESHOWWINDOW
#si.wShowWindow = subprocess.SW_HIDE # default
# run tdmsinfo in a subprocess and capture the output in a
a = subprocess.run(['tdmsinfo',tdmsfile],
stdout=subprocess.PIPE,
startupinfo=si).stdout
a = a.decode('utf-8')
print(a)
the code above should give you only the channels and groups, but you can also run with the -p flag to include all the TDMS object properties
a = subprocess.run(['tdmsinfo','-p',tdmsfile],
stdout=subprocess.PIPE,
startupinfo=si).stdout

python error handling filepath is incorrect when run as a module

So I have a 2 files.
test.py
import sys
import error_handler
sys.excepthook = error_handler;
value = 23/0; #this line will throw Zero division error
error_handler.py
def custom_error_handler(ex_class, ex, tb):
fileName = os.path.split(tb.tb_frame.f_code.co_filename)[1];
lineNo = tb.tb_lineno;
print fileName;
print lineNo;
These are the outputs when I run the file and when I run it as a module.
python test.py
test.py
4
But if I run
python -m test
runpy.py
162
Any ideas on why running it the second way produces that result? is there another way I should be doing this?
Thankyou.

ok I solved this by following viraptors comment and the python documentation.
I had to import traceback, split the traceback into an array
import traceback;
tb_array = traceback.extract_tb(tb);
each element in the array is an array that contains:
file path
line number
method name
line string