We Keep Coding

Gracefully handle variable number of input and output files

You have a pipeline where some samples have one input file and produce one output file while other samples have two input files and produce two output files. The typical case for those in bioinformatics is NGS sequencing libraries where some samples are paired-end and other samples are single-end sequenced. If you need to trim reads and align them you have to account for the variable number of input/output files.
What is the most appropriate way to handle this? I feel using checkpoints is overkill since in my opinion checkpoints are a bit cryptic, but I may be wrong...
Here's how I would do it for the case where the number of input/output of files can be only 1 or 2: Have an if-else in the run directive based first on the number of input files. If the number of inputs is 1 touch the second files. For the following rules have again an if-else this time checking whether the second file has 0 bytes.
Here's an example (not tested but it should be about right):
import os
samples = {'S1': ['s1.R1.fastq'],
'S2': ['s2.R1.fastq', 's2.R2.fastq']}
rule all:
input:
expand('bam/{sample_id}.bam', sample_id= samples),
rule cutadapt:
input:
fastq= lambda wc: samples[wc.sample_id],
output:
r1= 'cutadapt/{sample_id}.R1.fastq',
r2= touch('cutadapt/{sample_id}.R2.fastq'),
run:
if len(input.fastq) == 1:
# {output.r1} created, {output.r2} touched
shell('cutadapt ... -o {output.r1} {input.fastq}')
else:
shell('cutadapt ... -o {output.r1} -p {output.r2} {input.fastq}')
rule align:
input:
r1= 'cutadapt/{sample_id}.R1.fastq',
r2= 'cutadapt/{sample_id}.R2.fastq',
output:
'bam/{sample_id}.bam',
run:
if os.path.getsize(input.r2) == 0:
# or
# if len(samples[wildcards.sample_id]) == 1:
shell('hisat2 ... {input.r1} > {output.bam}')
else:
shell('hisat2 ... -1 {input.r1} -2 {input.r2} > {output.bam}')
This works, but I find it awkward to artificially create the second file just to keep the workflow happy. Are there better solutions?

There shall be two separate rules for single ended and pair ended case:
rule single_end:
input:
fastq = 's{n}.R1.fastq'
output:
r1 = 'cutadapt/S{n}.R1.fastq'
shell:
'cutadapt ... -o {output.r1} {input.fastq}'
rule pair_ends:
input:
r1 = 's{n}.R1.fastq',
r2 = 's{n}.R2.fastq'
output:
r1 = 'cutadapt/S{n}.R1.fastq',
r2 = 'cutadapt/S{n}.R2.fastq'
shell:
'cutadapt ... -o {output.r1} -p {output.r2} {input}'
Now there is an ambiguity: whenever the pair_ends can be applied, the single_end may be applied too. This problem can be solved with can be ruleorder:
ruleorder pair_ends > single_end

I agree that checkpoints would be overkill here.
You can compose arbitrary inputs and command lines using input functions and parameter functions. The trick is optional r2 output from your cutadapt rule, which may or may not be present, which complicates the workflow's DAG. In this case it's probably appropriate to use Snakemake's directories as output functionality:
CUTADAPT_R1 = 'cutadapt/{sample_id}/R1.fastq'
CUTADAPT_R2 = 'cutadapt/{sample_id}/R2.fastq'
def get_cutadapt_outargs(wldc, input):
r1 = CUTADAPT_R1.format(**wldc)
r2 = CUTADAPT_R2.format(**wldc)
ret = f"-o '{r1}'"
if len(input.fastq) > 1:
ret += f" -p '{r2}'"
return ret
def get_align_inargs(wldc):
r1 = CUTADAPT_R1.format(**wldc)
r2 = CUTADAPT_R2.format(**wldc)
if len(samples[wldc.sample_id]) > 1:
return "-1 '{}' -2 '{}'".format(r1, r2)
else:
return "'{}'".format(r1)
rule all:
input:
expand('bam/{sample_id}.bam', sample_id= samples),
rule cutadapt:
input:
fastq= lambda wc: samples[wc.sample_id],
output:
fq = directory('cutadapt/{sample_id}'),
params:
out_args = get_cutadapt_outargs,
shell:
"""
cutadapt ... {params.out_args} {input.fastq}
"""
rule align:
input:
fq = rules.cutadapt.output.fq,
output:
bam = 'bam/{sample_id}.bam',
params:
input_args = get_align_inargs,
shell:
"""
hisat2 ... {params.input_args} > {output.bam}
"""

How to transform Serial Data from digital inclinometer to readable format in Python?

I bought a device (inclinometer) that produces data which I want to attach to my post in a readable format. How can I do this using python?
I have the following script:
import binascii
import serial
ser = serial.Serial('COM1', 9600)
s = ser.read()
hex_string = binascii.hexlify(s).decode('utf-8')
print(hex_string)
I have added an example of how the inclinometer is sending data over COM port.

As not all information is given to tailor the best answer some assumptions are made to narrow down the posibile answers.
In the example the assumption is taken that the datastream giving by the inclination-meter includes more than just x-axis and y-axis coordination. Therefor the while-loop is introduced. There is also a data-request pause from inclination-meter build in to prevent data-flooding in your destination device where you run this python script (raspberry-pi?).
Perhaps this script is used in multiprocessing mode? Then you might want to include a step where you compare current x,y coordinates against defined reference values (not included but should at the end of the while-loop).
import binascii
import serial
import time
set_sloop = 0 # control flag for result
loop_checks = 0 # control flag to prevent data-flooding.
ser = serial.Serial('COM1', 9600)
while set_sloop == 0: # get hex-data from device every 2 seconds (see pause interval)
s = ser.read()
hex_string = binascii.hexlify(s).decode('utf-8')
print(hex_string)
if str(hex_string[0:9]) == '680D008400': # used (x,y) string identifier based on example 0x84.
xh = hex_string[10:13]
yh = hex_string[16:19]
x = '%s.%s' % xh[0:1], xh[2:3]
y = '%s.%s' % yh[0:1], yh[2:3]
xt = 'x = %s deg' % x
yt = 'y = %s deg' % y
print (xt, yt)
set_sloop = 1
loop_checks += 1
if loop_checks == 10:
time.sleep(2) # pause interval: wait 2 seconds before requesting new data.
loop_checks = 0
print ('got inclination: (%s , %s) deg' % x, y)
# ... remaining code to "auto-post inclination" goes here ...
Note: take into account that the searchstring '680D008400' is not completely according the description given in the example. There is missing a set of digits. Typo in company text?

Error with quantifier in Z3Py

I would like Z3 to check whether it exists an integer t that satisfies my formula. I'm getting the following error:
Traceback (most recent call last):
File "D:/z3-4.6.0-x64-win/bin/python/Expl20180725.py", line 18, in <module>
g = ForAll(t, f1(t) == And(t>=0, t<10, user[t].rights == ["read"] ))
TypeError: list indices must be integers or slices, not ArithRef
Code:
from z3 import *
import random
from random import randrange
class Struct:
def __init__(self, **entries): self.__dict__.update(entries)
user = [Struct() for i in range(10)]
for i in range(10):
user[i].uid = i
user[i].rights = random.choice(["create","execute","read"])
s=Solver()
f1 = Function('f1', IntSort(), BoolSort())
t = Int('t')
f2 = Exists(t, f1(t))
g = ForAll(t, f1(t) == And(t>=0, t<10, user[t].rights == ["read"] ))
s.add(g)
s.add(f2)
print(s.check())
print(s.model())

You are mixing and matching Python and Z3 expressions, and while that is the whole point of Z3py, it definitely does not mean that you can mix/match them arbitrarily. In general, you should keep all the "concrete" parts in Python, and relegate the symbolic parts to "z3"; carefully coordinating the interaction in between. In your particular case, you are accessing a Python list (your user) with a symbolic z3 integer (t), and that is certainly not something that is allowed. You have to use a Z3 symbolic Array to access with a symbolic index.
The other issue is the use of strings ("create"/"read" etc.) and expecting them to have meanings in the symbolic world. That is also not how z3py is intended to be used. If you want them to mean something in the symbolic world, you'll have to model them explicitly.
I'd strongly recommend reading through http://ericpony.github.io/z3py-tutorial/guide-examples.htm which is a great introduction to z3py including many of the advanced features.
Having said all that, I'd be inclined to code your example as follows:
from z3 import *
import random
Right, (create, execute, read) = EnumSort('Right', ('create', 'execute', 'read'))
users = Array('Users', IntSort(), Right)
for i in range(10):
users = Store(users, i, random.choice([create, execute, read]))
s = Solver()
t = Int('t')
s.add(t >= 0)
s.add(t < 10)
s.add(users[t] == read)
r = s.check()
if r == sat:
print s.model()[t]
else:
print r
Note how the enumerated type Right in the symbolic land is used to model your "permissions."
When I run this program multiple times, I get:
$ python a.py
5
$ python a.py
9
$ python a.py
unsat
$ python a.py
6
Note how unsat is produced, if it happens that the "random" initialization didn't put any users with a read permission.

extract number from exponent in Python

Hi everyone / Python Gurus
I would like to know how to accomplish the following task, which so far I've been unable to do so.
Here's what I have:
Q1 = 20e-6
Now this is an exponential number that if you print(Q1) as is it will show: 2e-5 which is fine. Mathematically speaking.
However, here's what I want to do with it:
I want Q1 to print only the number 20. And based on the whether this is e-6 then print uC or if this e-9 the print nC.
Here's an example for better understanding:
Q1=20e-6
When I run print(Q1) show: 20uC.
Q2=20e-9
When I run print(Q2) show: 20nC.
Can you please help me figure this out?

just replace the exponent using str.replace:
q1 = 'XXXXXX'
q1 = q1.replace('e-9', 'nC').replace('e-6', 'uC')
print(q1)

I recommend you using si-prefix.
You can install it using pip:
sudo pip install si-prefix
Then you can use something like this:
from si_prefix import si_format
# precision after the point
# char is the unity's char to be used
def get_format(a, char='C', precision=2):
temp = si_format(a, precision)
try:
num, prefix = temp.split()
except ValueError:
num, prefix = temp , ''
if '.' in num:
aa, bb = num.split('.')
if int(bb) == 0:
num = aa
if prefix:
return num + ' ' + prefix + char
else:
return num
tests = [20e-6, 21.46e05, 33.32e-10, 0.5e03, 0.33e-2, 112.044e-6]
for k in tests:
print get_format(k)
Output:
20 uC
2.15 MC
3.33 nC
500
3.30 mC
112.04 uC

You can try by splitting the string:
'20e-9'.split('e')
gives
['20', '-9']
From there on, you can insert whatever you want in between those values:
('u' if int(a[1]) > 0 else 'n').join(a)
(with a = '20e-9'.split('e'))

You can not. The behaviour you are looking for is called "monkey patching". And this is not allowed for int and float.
You can refer to this stackoverflow question
The only way I can think of is to create a class that extends float and then implement a __str__ method that shows as per your requirement.
------- More explanation -----
if you type
Q1 = 20e-6
in python shell and then
type(Q1)
your will get a
float
So basically your Q1 is considered as float by python type system
when you type print(Q1)
the _str__ method of float is called
The process of extending core class is one example of "monkey patch" and that is what I was refereing to.
Now the problem is that you can not "monkey patch" (or extend if you prefer that) core classes in python (which you can in some languages like in Ruby).
[int, float etc are core classes and written in C for your most common python distribution.]
So how do you solve it?
you need to create a new class like this
class Exponent(float):
def init(self, value):
self.value = value
def __str__(self):
return "ok"
x = Exponent(10.0)
print(x) ==> "ok"
hope this helps

python - pretty print errorbars

I'm using python with numpy, scipy and matplotlib for data evaluation. As results I obtain averages and fitting parameters with errorbars.
I would like python to automatically pretty-print this data according to a given precision. For example:
Suppose I got the result x = 0.012345 +/- 0.000123.
Is there a way to automatically format this as 1.235(12) x 10^-2 when a precision of 2 was specified. That is, counting the precision in the errorbar, rather than in the value.
Does anyone know a package that provides such functionality, or would I have to implement this myself?
Is there a way to inject this into the python string formatting mechanism? I.e. being able to write something like "%.2N" % (0.012345, 0.0000123).
I already looked through the docs of numpy and scipy and googled around, but I couldn't find anything. I think this would be a useful feature for everyone who deals with statistics.
Thanks for your help!
EDIT:
As requested by Nathan Whitehead I'll give a few examples.
123 +- 1 ----precision 1-----> 123(1)
123 +- 1.1 ----precision 2-----> 123.0(11)
0.0123 +- 0.001 ----precision 1-----> 0.012(1)
123.111 +- 0.123 ----precision 2-----> 123.11(12)
The powers of ten are omitted for clarity.
The number inside the parenthesis is a shorthand notation for the standard error. The last digit of the number before the parens and the last digit of the number inside the parens have to be at the same decimal power. For some reason I cannot find a good explanation of this concept online. Only thing I got is this German Wikpedia article here. However, it is a quite common and very handy notation.
EDIT2:
I implemented the shorthand notation thing myself:
#!/usr/bin/env python
# *-* coding: utf-8 *-*
from math import floor, log10
# uncertainty to string
def un2str(x, xe, precision=2):
"""pretty print nominal value and uncertainty
x - nominal value
xe - uncertainty
precision - number of significant digits in uncertainty
returns shortest string representation of `x +- xe` either as
x.xx(ee)e+xx
or as
xxx.xx(ee)"""
# base 10 exponents
x_exp = int(floor(log10(x)))
xe_exp = int(floor(log10(xe)))
# uncertainty
un_exp = xe_exp-precision+1
un_int = round(xe*10**(-un_exp))
# nominal value
no_exp = un_exp
no_int = round(x*10**(-no_exp))
# format - nom(unc)exp
fieldw = x_exp - no_exp
fmt = '%%.%df' % fieldw
result1 = (fmt + '(%.0f)e%d') % (no_int*10**(-fieldw), un_int, x_exp)
# format - nom(unc)
fieldw = max(0, -no_exp)
fmt = '%%.%df' % fieldw
result2 = (fmt + '(%.0f)') % (no_int*10**no_exp, un_int*10**max(0, un_exp))
# return shortest representation
if len(result2) <= len(result1):
return result2
else:
return result1
if __name__ == "__main__":
xs = [123456, 12.34567, 0.123456, 0.001234560000, 0.0000123456]
xes = [ 123, 0.00123, 0.000123, 0.000000012345, 0.0000001234]
precs = [ 1, 2, 3, 4, 1]
for (x, xe, prec) in zip(xs, xes, precs):
print '%.6e +- %.6e #%d --> %s' % (x, xe, prec, un2str(x, xe, prec))
Output:
1.234560e+05 +- 1.230000e+02 #1 --> 1.235(1)e5
1.234567e+01 +- 1.230000e-03 #2 --> 12.3457(12)
1.234560e-01 +- 1.230000e-04 #3 --> 0.123456(123)
1.234560e-03 +- 1.234500e-08 #4 --> 0.00123456000(1235)
1.234560e-05 +- 1.234000e-07 #1 --> 1.23(1)e-5

For people that are still interested in this question, see the gvar library and here for an example of (at last part of) the desired behavior by the OP.

since x +- y is not a standard type (it could be seen as a complex with real and imaginary as x and y i guess, but that does not simplify anything...) but you can get full control over the presentation by creating a type and overriding the string function, i.e. something like this
class Res(object):
def __init__(self, res, delta):
self.res = res
self.delta = delta
def __str__(self):
return "%f +- %f"%(self.res,self.delta)
if __name__ == '__main__':
x = Res(0.2710,0.001)
print(x)
print(" a result: %s" % x)
you could naturally do something a bit more fancy inside the __str__ function...