We Keep Coding

A Problem on getting docker stats by Python

I tried to use Python to get the docker stats, by using Python's docker module.
the code is:
import docker
cli = docker.from_env()
for container in cli.containers.list():
stream = container.stats()
print(next(stream))
I run 6 docker containers, but when I run the code, It needs a few second to get all containers' stats, so is there have some good methods to get the stats immediately?

Docker stats inherently takes a little while, a large part of this is waiting for the next value to come through the stream
$ time docker stats 1339f13154aa --no-stream
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
...
real 0m1.556s
user 0m0.020s
sys 0m0.015s
You could reduce the time it takes to execute by running the commands in parralell, as opposed to one at a time.
To achieve this, you could use the wonderful threading or multiprocessing library.
Digital Ocean provides a good tutorial on how to accomplish this with a ThreadPoolExecutor:
import requests
import concurrent.futures
def get_wiki_page_existence(wiki_page_url, timeout=10):
response = requests.get(url=wiki_page_url, timeout=timeout)
page_status = "unknown"
if response.status_code == 200:
page_status = "exists"
elif response.status_code == 404:
page_status = "does not exist"
return wiki_page_url + " - " + page_status
wiki_page_urls = [
"https://en.wikipedia.org/wiki/Ocean",
"https://en.wikipedia.org/wiki/Island",
"https://en.wikipedia.org/wiki/this_page_does_not_exist",
"https://en.wikipedia.org/wiki/Shark",
]
with concurrent.futures.ThreadPoolExecutor() as executor:
futures = []
for url in wiki_page_urls:
futures.append(executor.submit(get_wiki_page_existence, wiki_page_url=url))
for future in concurrent.futures.as_completed(futures):
print(future.result())

This is what I use to get the stats directly for each running docker (still takes like 1 second per container, so I don't think it can be helped). Besides this if it helps https://docker-py.readthedocs.io/en/stable/containers.html documentation for the arguments. Hope it helps.
import docker
client = docker.from_env()
containers = client.containers.list()
for x in containers:
print(x.stats(decode=None, stream=False))
As the TheQueenIsDead suggested, it might require threading if you want to get it faster.

segfault when using shout-python inside a thread

Recently a machine was upgraded/reinstalled from an older Ubuntu version to the most recent Debian version. Since then, I get a segfault in an application I wrote which sends data to an icecast daemon.
After some searching, I focused it down to threading. As soon as I call shout.send from inside a thread, I get a segfault. Below is a minimal piece of code which reproduces the error.
The error occurs on the second-last line (icy_handle.send(chunk))
I am running:
Python 2.7
shout-python==0.2.1
Debian 8.0
Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt2-1 (2014-12-08) x86_64 GNU/Linux
I am honestly a bit stumped. After roughly 4 hours of hit-and-miss I am out of ideas. I have not yet tried to set up an old machine representing the old Ubuntu environment. I also don't remember the exact Ubuntu release. I think it was 12.04 but I am not sure. But I do know that it was running on that system. By the way: there were no hardware changes!
Any ideas?
The example code:
from threading import Thread
import shout
class Player(Thread):
def __init__(self, filename, *args, **kwargs):
super(Player, self).__init__(*args, **kwargs)
self.filename = filename
def run(self):
icy_handle = shout.Shout()
icy_handle.format = 'mp3'
icy_handle.audio_info = {
"bitrate": str(128),
"samplerate": str(44100),
"channels": str(1)}
icy_handle.user = "source"
icy_handle.name = "test stream"
icy_handle.url = "http://stream.example.com"
icy_handle.password = "password123"
icy_handle.mount = "/test.mp3"
icy_handle.port = 8000
icy_handle.open()
chunk_size = 1024
with open(self.filename, 'rb') as afile:
chunk = afile.read(chunk_size)
while chunk:
icy_handle.send(chunk)
icy_handle.sync()
chunk = afile.read(chunk_size)
if __name__ == "__main__":
import sys
if len(sys.argv) < 2:
print """
USAGE: %s <filename>
""" % sys.argv[0]
sys.exit(1)
p = Player(sys.argv[1])
p.start()
p.join()

I can reproduce your problem with 0.2.1, but not with trunk.
Please try:
svn co https://svn.xiph.org/icecast/trunk/shout-python
I'm pretty sure that this is the relevant fix:
https://trac.xiph.org/changeset/19174
We should release a new version. Let me get right on to that. (JFTR, this will involve me migrating the repository from SVN to git, so the above URL becomes invalid outside of this particular scope)

Finding Bluetooth low energy with python

Is it possible for this code to be modified to include Bluetooth Low Energy devices as well? https://code.google.com/p/pybluez/source/browse/trunk/examples/advanced/inquiry-with-rssi.py?r=1
I can find devices like my phone and other bluetooth 4.0 devices, but not any BLE. If this cannot be modified, is it possible to run the hcitool lescan and pull the data from hci dump within python? I can use the tools to see the devices I am looking for and it gives an RSSI in hcidump, which is what my end goal is. To get a MAC address and RSSI from the BLE device.
Thanks!

As I said in the comment, that library won't work with BLE.
Here's some example code to do a simple BLE scan:
import sys
import os
import struct
from ctypes import (CDLL, get_errno)
from ctypes.util import find_library
from socket import (
socket,
AF_BLUETOOTH,
SOCK_RAW,
BTPROTO_HCI,
SOL_HCI,
HCI_FILTER,
)
if not os.geteuid() == 0:
sys.exit("script only works as root")
btlib = find_library("bluetooth")
if not btlib:
raise Exception(
"Can't find required bluetooth libraries"
" (need to install bluez)"
)
bluez = CDLL(btlib, use_errno=True)
dev_id = bluez.hci_get_route(None)
sock = socket(AF_BLUETOOTH, SOCK_RAW, BTPROTO_HCI)
sock.bind((dev_id,))
err = bluez.hci_le_set_scan_parameters(sock.fileno(), 0, 0x10, 0x10, 0, 0, 1000);
if err < 0:
raise Exception("Set scan parameters failed")
# occurs when scanning is still enabled from previous call
# allows LE advertising events
hci_filter = struct.pack(
"<IQH",
0x00000010,
0x4000000000000000,
0
)
sock.setsockopt(SOL_HCI, HCI_FILTER, hci_filter)
err = bluez.hci_le_set_scan_enable(
sock.fileno(),
1, # 1 - turn on; 0 - turn off
0, # 0-filtering disabled, 1-filter out duplicates
1000 # timeout
)
if err < 0:
errnum = get_errno()
raise Exception("{} {}".format(
errno.errorcode[errnum],
os.strerror(errnum)
))
while True:
data = sock.recv(1024)
# print bluetooth address from LE Advert. packet
print(':'.join("{0:02x}".format(x) for x in data[12:6:-1]))
I had to piece all of that together by looking at the hcitool and gatttool source code that comes with Bluez. The code is completely dependent on libbluetooth-dev so you'll have to make sure you have that installed first.
A better way would be to use dbus to make calls to bluetoothd, but I haven't had a chance to research that yet. Also, the dbus interface is limited in what you can do with a BLE connection after you make one.
EDIT:
Martin Tramšak pointed out that in Python 2 you need to change the last line to print(':'.join("{0:02x}".format(ord(x)) for x in data[12:6:-1]))

You could also try pygattlib. It can be used to discover devices, and (currently) there is a basic support for reading/writing characteristics. No RSSI for now.
You could discover using the following snippet:
from gattlib import DiscoveryService
service = DiscoveryService("hci0")
devices = service.discover(2)
DiscoveryService accepts the name of the device, and the method discover accepts a timeout (in seconds) for waiting responses. devices is a dictionary, with BL address as keys, and names as values.
pygattlib is packaged for Debian (or Ubuntu), and also available as a pip package.

how can the directory of a usb drive connected to a system be obtained?

I need to obtain the path to the directory created for a usb drive(I think it's something like /media/user/xxxxx) for a simple usb mass storage device browser that I am making. Can anyone suggest the best/simplest way to do this? I am using an Ubuntu 13.10 machine and will be using it on a linux device.
Need this in python.

This should get you started:
#!/usr/bin/env python
import os
from glob import glob
from subprocess import check_output, CalledProcessError
def get_usb_devices():
sdb_devices = map(os.path.realpath, glob('/sys/block/sd*'))
usb_devices = (dev for dev in sdb_devices
if 'usb' in dev.split('/')[5])
return dict((os.path.basename(dev), dev) for dev in usb_devices)
def get_mount_points(devices=None):
devices = devices or get_usb_devices() # if devices are None: get_usb_devices
output = check_output(['mount']).splitlines()
is_usb = lambda path: any(dev in path for dev in devices)
usb_info = (line for line in output if is_usb(line.split()[0]))
return [(info.split()[0], info.split()[2]) for info in usb_info]
if __name__ == '__main__':
print get_mount_points()
How does it work?
First, we parse /sys/block for sd* files (courtesy of https://stackoverflow.com/a/3881817/1388392) to filter out usb devices.
Later you call mount and parse output for lines only for those devices.
Of course they might be some edge cases, when this won't work, portability issues etc. Or better ways to do it. But for more information you should rather seek help on SuperUser or ServerFault, with more experienced linux hackers.

I had to modify #m.wasowski 's code to make it work on Python3.5.4 as follows.
def get_mount_points(devices=None):
devices = devices or get_usb_devices() # if devices are None: get_usb_devices
output = check_output(['mount']).splitlines()
output = [tmp.decode('UTF-8') for tmp in output]
def is_usb(path):
return any(dev in path for dev in devices)
usb_info = (line for line in output if is_usb(line.split()[0]))
return [(info.split()[0], info.split()[2]) for info in usb_info]

Using m.wasowski code, unexpected behavior can occur:
return [(info.split()[0], info.split()[2]) for info in usb_info]
This part of code can produce bug, if your USB device name has white space character in it. I got that behavior with device named "USB DEVICE".
info.split()[2]
Returned media/home/USB for me, when it is media/home/USB DEVICE.
I modified that part, so it was founding word "type", and replaced that line with this:
#return [(info.split()[0], info.split()[2]) for info in usb_info]
fullInfo = []
for info in usb_info:
print(info)
mountURI = info.split()[0]
usbURI = info.split()[2]
print(info.split().__sizeof__())
for x in range(3, info.split().__sizeof__()):
if info.split()[x].__eq__("type"):
for m in range(3, x):
usbURI += " "+info.split()[m]
break
fullInfo.append([mountURI, usbURI])
return fullInfo

I had to further modify #nick-sikrier and #m-wasowski response to handle LUKs encrypted devices.
def get_usb_devices():
sdb_devices = map(os.path.realpath, glob('/sys/block/sd*'))
usb_devices = (dev for dev in sdb_devices
if any(['usb' in dev.split('/')[5],
'usb' in dev.split('/')[6]]))
return dict((os.path.basename(dev), dev) for dev in usb_devices)
def get_mount_points(
devices = get_usb_devices()
fullInfo = []
for dev in devices:
output = subprocess.check_output(['lsblk', '-lnpo', 'NAME,MOUNTPOINT', '/dev/' + dev]).splitlines()
for mnt_point in output:
mnt_point_split = mnt_point.split(' ', 1)
if len(mnt_point_split) > 1 and mnt_point_split[1].strip():
fullInfo.append([mnt_point_split[0], mnt_point_split[1]])
return fullInfo

With a simple shell pipe executed in python:
import subprocess
driver_name = "my_usb_stick"
path = subprocess.check_output("cat /proc/mounts | grep '"+driver_name+"' | awk '{print $2}'", shell=True)
path = path.decode('utf-8') # convert bytes in string
>>> "/media/user/my_usb_stick"
Explanations
/proc/mounts/ : Is a file listing all mounted devices
The 1st column specifies the device that is mounted.
The 2nd column reveals the mount point.
The 3rd column tells the file-system type.
The 4th column tells you if it is mounted read-only (ro) or read-write (rw).
The 5th and 6th columns are dummy values designed to match the format used in /etc/mtab
More details see this answer : How to interpret /proc/mounts?
grep returns the line containing your driver's name
awk returns the 2nd columns, aka the mount point, aka your path.

Determine Device of Filesystem in Python

How do you use Python to determine which Linux device/partition contains a given filesystem?
e.g.
>>> get_filesystem_device('/')
/dev/sda
>>> get_filesystem_partition('/')
/dev/sda1

Your question was about Linux, so this is (more or less) linux specific.
Below is code example for three variants for mapping major/minor to a device name.
Parse /proc/partitions.
Ask hal. Hal also keeps track of "parent" device, meaning you can easily get the disk aswell as the partition.
Check sysfs yourself. This is where hal gets its information from.
I'd say that /proc/partitions is simplest - it is just one file to open and check. hal gives you most information, and abstracts away lots of details. sysfs may be viewed as more correct that /proc/partitions and doesn't require hal to be running.
For a desktop program I would go for hal. On an embedded system I'd go with sysfs.
import os
def main():
dev = os.stat("/home/").st_dev
major, minor = os.major(dev), os.minor(dev)
print "/proc/partitions says:", ask_proc_partitions(major, minor)
print "HAL says:", ask_hal(major, minor)
print "/sys says:", ask_sysfs(major, minor)
def _parse_proc_partitions():
res = {}
for line in file("/proc/partitions"):
fields = line.split()
try:
tmaj = int(fields[0])
tmin = int(fields[1])
name = fields[3]
res[(tmaj, tmin)] = name
except:
# just ignore parse errors in header/separator lines
pass
return res
def ask_proc_partitions(major, minor):
d = _parse_proc_partitions()
return d[(major, minor)]
def ask_hal(major, minor):
import dbus
bus = dbus.SystemBus()
halobj = bus.get_object('org.freedesktop.Hal', '/org/freedesktop/Hal/Manager')
hal = dbus.Interface(halobj, 'org.freedesktop.Hal.Manager')
def getdevprops(p):
bdevi = dbus.Interface(bus.get_object('org.freedesktop.Hal', p),
"org.freedesktop.Hal.Device")
return bdevi.GetAllProperties()
bdevs = hal.FindDeviceByCapability("block")
for bdev in bdevs:
props = getdevprops(bdev)
if (props['block.major'], props['block.minor']) == (major, minor):
parentprops = getdevprops(props['info.parent'])
return (str(props['block.device']),
str(parentprops['block.device']))
def ask_sysfs(major, minor):
from glob import glob
needle = "%d:%d" % (major, minor)
files = glob("/sys/class/block/*/dev")
for f in files:
if file(f).read().strip() == needle:
return os.path.dirname(f)
return None
if __name__ == '__main__':
main()

It looks like this post has some of your answer (still not sure just how to grab the major/minor out of the /dev/sda2 entry to match it up with what os.stat() returns for /:
Device number in stat command output
>>> import os
>>> print hex(os.stat('/')[2])
0x802
\ \minor device number
\major device number
[me#server /]$ ls -l /dev/sda2
brw-rw---- 1 root disk 8, 2 Jun 24 2004 /dev/sda2
[me#server jgaines2]$ \ \minor device number
\major device number

I recently had a need for this solution also. After seeing all the convoluted methods of getting the result I wanted through pure python, I decided to turn to the shell for help.
import subprocess
device = subprocess.check_output("grep '/filesystem' /proc/mounts | awk '{printf $1}'", shell=True)
print device
This gives me exactly what I want, the device string for where my filesystem is mounted.
Short, sweet, and runs in python. :)

There are problems with quite a few of the above solutions. There's actually a problem with the question as well.
The last answer (searching /proc/mounts) just doesn't work: searching for "/" will match every line in /proc/mounts. Even correcting this like this won't work:
import subprocess
device = subprocess.check_output("awk '$2 == \"/filesystem\" { print $1}' /proc/mounts", shell=True)
print device
When "/filesystem" is "/" you'll typically get two entries, one for "rootfs" and one for the actual device. It also won't work when the mounted file system name has spaces in it (the space appears as \040 in /proc/mounts).
The problem is made worse with btrfs subvolumes. Each subvolume is mounted separately but they all share the same device. If you're trying to use a btrfs snapshot for backups (as I was) then you need the subvolume name and an indication of the filesystem type.
This function returns a tuple of (device, mountpoint, filesystem) and seems to work:
import os
def get_filesystem_partition(fs):
res = None
dev = os.lstat(fs).st_dev
for line in file('/proc/mounts'):
# lines are device, mountpoint, filesystem, <rest>
# later entries override earlier ones
line = [s.decode('string_escape') for s in line.split()[:3]]
if dev == os.lstat(line[1]).st_dev:
res = tuple(line)
return res
That seems to work for all the cases I can think of, although I expect that there are still pathological cases where it falls to bits.

It is not the purdiest, but this will get you started:
#!/usr/bin/python
import os, stat, subprocess, shlex, re, sys
dev=os.stat('/')[stat.ST_DEV]
major=os.major(dev)
minor=os.minor(dev)
out = subprocess.Popen(shlex.split("df /"), stdout=subprocess.PIPE).communicate()
m=re.search(r'(/[^\s]+)\s',str(out))
if m:
mp= m.group(1)
else:
print "cannot parse df"
sys.exit(2)
print "'/' mounted at '%s' with dev number %i, %i" % (mp,major,minor)
On OS X:
'/' mounted at '/dev/disk0s2' with dev number 14, 2
On Ubuntu:
'/' mounted at '/dev/sda1' with dev number 8, 1
To get the device name, chop off the minor number from the partition name. On OS X, also chop the 's' + minor number.

How about using the (linux) blkid command (/sbin/blkid)
$ uname --kernel-name --kernel-release
Linux 3.2.0-4-amd64
$ python --version
Python 2.7.3
-
#!/usr/bin/env python
import subprocess
sys_command = "/sbin/blkid"
proc = subprocess.Popen(sys_command,
stdout=subprocess.PIPE,
shell=True)
# proc.communicate() returns a tuple (stdout,stderr)
blkid_output = proc.communicate()[0]
print blkid_output
Here's the output on a dual-boot laptop with an (unmounted) USB drive (sdb1)
$ ./blkid.py
/dev/sda1: LABEL="RECOVERY" UUID="xxxx-xxxx" TYPE="vfat"
/dev/sda2: LABEL="OS" UUID="xxxxxxxxxxxxxxx" TYPE="ntfs"
/dev/sda5: UUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" TYPE="ext4"
/dev/sda6: UUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" TYPE="swap"
/dev/sda7: UUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" TYPE="ext4"
/dev/sdb1: LABEL="CrunchBang" TYPE="iso9660"

Here is how you can simply get the devices major and minor numbers:
import os
major, minor = divmod(os.stat('/').st_dev, 256)