We Keep Coding

Get the size of a folder in Linux server

While the following code works well in windows, in Linux server (of pythonanywhere) the function only returns 0, without errors. What am I missing?
import os
def folder_size(path):
total = 0
for entry in os.scandir(path):
if entry.is_file():
total += entry.stat().st_size
elif entry.is_dir():
total += folder_size(entry.path)
return total
print(folder_size("/media"))
Ref: Code from https://stackoverflow.com/a/37367965/6546440

The solution was given by #gilen-tomas in the comments:
import os
def folder_size(path):
total = 0
for entry in os.scandir(path):
if entry.is_file():
total += entry.stat().st_size
elif entry.is_dir():
total += folder_size(entry.path)
return total
print(folder_size("/home/your-user/your-proyect/media/"))
A complete path is needed!

Depending on the filesystem, the underlying struct dirent may not know if any given entry is a file or directory (or something else). Perhaps, on the filesystem used by pythonanywhere, you need to stat first (stat_result.st_type ought to be valid).
Edit: A look in discussion on os.scandir suggests the DT_UNKNOWN case is handled by doing another stat. I'd still try confirming those checks work as expected.

you can try this..
For linux :
import os
path = '/home/user/Downloads'
folder = sum([sum(map(lambda fname: os.path.getsize(os.path.join(directory, fname)), files)) for directory, folders, files in os.walk(path)])
MB=1024*1024.0
print "%.2f MB"%(folder/MB)
For windows :
import win32com.client as com
folderPath = r"/home/user/Downloads"
fso = com.Dispatch("Scripting.FileSystemObject")
folder = fso.GetFolder(folderPath)
MB=1024*1024.0
print "%.2f MB"%(folder.Size/MB)

It's worked for me in linux (Ubuntu server 16.04, python 3.5), but there could be some permission errors if the process doesn't have permission for reading a file.

Not a solution for this, but other way to get the size is using the cmd from python:
import subprocess
import re
cmd = ["du", "-sh", "-b", "media"]
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE)
tmp = str(proc.stdout.read())
tmp = re.findall('\d+', tmp)[0]
print(tmp)
If you are executing this from your proyect (instead of manually in the terminal) a complete path is needed in "media" ("/home/your-user/your-proyect/media/")

How to create a shortcut in startmenu using setuptools windows installer

I want to create a start menu or Desktop shortcut for my Python windows installer package. I am trying to follow https://docs.python.org/3.4/distutils/builtdist.html#the-postinstallation-script
Here is my script;
import sys
from os.path import dirname, join, expanduser
pyw_executable = sys.executable.replace('python.exe','pythonw.exe')
script_file = join(dirname(pyw_executable), 'Scripts', 'tklsystem-script.py')
w_dir = expanduser(join('~','lsf_files'))
print(sys.argv)
if sys.argv[1] == '-install':
print('Creating Shortcut')
create_shortcut(
target=pyw_executable,
description='A program to work with L-System Equations',
filename='L-System Tool',
arguments=script_file,
workdir=wdir
)
I also specified this script in scripts setup option, as indicated by aforementioned docs.
Here is the command I use to create my installer;
python setup.py bdist_wininst --install-script tklsystem-post-install.py
After I install my package using created windows installer, I can't find where my shorcut is created, nor I can confirm whether my script run or not?
How can I make setuptools generated windows installer to create desktop or start menu shortcuts?

Like others have commented here and elsewhere the support functions don't seem to work at all (at least not with setuptools). After a good day's worth of searching through various resources I found a way to create at least the Desktop shortcut. I'm sharing my solution (basically an amalgam of code I found here and here). I should add that my case is slightly different from yasar's, because it creates a shortcut to an installed package (i.e. an .exe file in Python's Scripts directory) instead of a script.
In short, I added a post_install function to my setup.py, and then used the Python extensions for Windows to create the shortcut. The location of the Desktop folder is read from the Windows registry (there are other methods for this, but they can be unreliable if the Desktop is at a non-standard location).
#!/usr/bin/env python
import os
import sys
import sysconfig
if sys.platform == 'win32':
from win32com.client import Dispatch
import winreg
def get_reg(name,path):
# Read variable from Windows Registry
# From https://stackoverflow.com/a/35286642
try:
registry_key = winreg.OpenKey(winreg.HKEY_CURRENT_USER, path, 0,
winreg.KEY_READ)
value, regtype = winreg.QueryValueEx(registry_key, name)
winreg.CloseKey(registry_key)
return value
except WindowsError:
return None
def post_install():
# Creates a Desktop shortcut to the installed software
# Package name
packageName = 'mypackage'
# Scripts directory (location of launcher script)
scriptsDir = sysconfig.get_path('scripts')
# Target of shortcut
target = os.path.join(scriptsDir, packageName + '.exe')
# Name of link file
linkName = packageName + '.lnk'
# Read location of Windows desktop folder from registry
regName = 'Desktop'
regPath = r'Software\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders'
desktopFolder = os.path.normpath(get_reg(regName,regPath))
# Path to location of link file
pathLink = os.path.join(desktopFolder, linkName)
shell = Dispatch('WScript.Shell')
shortcut = shell.CreateShortCut(pathLink)
shortcut.Targetpath = target
shortcut.WorkingDirectory = scriptsDir
shortcut.IconLocation = target
shortcut.save()
setup(name='mypackage',
...,
...)
if sys.argv[1] == 'install' and sys.platform == 'win32':
post_install()
Here's a link to a full setup script in which I used this:
https://github.com/KBNLresearch/iromlab/blob/master/setup.py

If you want to confirm whether the script is running or not, you can print to a file instead of the console. Looks like text you print to console in the post-install script won't show up.
Try this:
import sys
from os.path import expanduser, join
pyw_executable = join(sys.prefix, "pythonw.exe")
shortcut_filename = "L-System Toolsss.lnk"
working_dir = expanduser(join('~','lsf_files'))
script_path = join(sys.prefix, "Scripts", "tklsystem-script.py")
if sys.argv[1] == '-install':
# Log output to a file (for test)
f = open(r"C:\test.txt",'w')
print('Creating Shortcut', file=f)
# Get paths to the desktop and start menu
desktop_path = get_special_folder_path("CSIDL_COMMON_DESKTOPDIRECTORY")
startmenu_path = get_special_folder_path("CSIDL_COMMON_STARTMENU")
# Create shortcuts.
for path in [desktop_path, startmenu_path]:
create_shortcut(pyw_executable,
"A program to work with L-System Equations",
join(path, shortcut_filename),
script_path,
working_dir)

At least with Python 3.6.5, 32bit on Windows, setuptools does work for this. But based on the accepted answer, by trial and error I found some issues that may have caused your script to fail to do what you wanted.
create_shortcut does not accept keyword arguments, only positional, so its usage in your code is invalid
You must add a .lnk extension for Windows to recognise the shortcut
I found sys.executable will be the name of the installer executable, not the python executable
As mentioned, you can't see stdout or stderr so you might want to log to a text file. I would suggest also redirecting sys.stdout and sys.stderr to the log file.
(Maybe not relevant) as mentioned in this question there appears to be a bug with the version string generated by bdist_wininst. I used the hexediting hack from an answer there to work around this. The location in the answer is not the same, you have to find the -32 yourself.
Full example script:
import sys
import os
import datetime
global datadir
datadir = os.path.join(get_special_folder_path("CSIDL_APPDATA"), "mymodule")
def main(argv):
if "-install" in argv:
desktop = get_special_folder_path("CSIDL_DESKTOPDIRECTORY")
print("Desktop path: %s" % repr(desktop))
if not os.path.exists(datadir):
os.makedirs(datadir)
dir_created(datadir)
print("Created data directory: %s" % repr(datadir))
else:
print("Data directory already existed at %s" % repr(datadir))
shortcut = os.path.join(desktop, "MyModule.lnk")
if os.path.exists(shortcut):
print("Remove existing shortcut at %s" % repr(shortcut))
os.unlink(shortcut)
print("Creating shortcut at %s...\n" % shortcut)
create_shortcut(
r'C:\Python36\python.exe',
"MyModuleScript",
shortcut,
"",
datadir)
file_created(shortcut)
print("Successfull!")
elif "-remove" in sys.argv:
print("Removing...")
pass
if __name__ == "__main__":
logfile = r'C:\mymodule_install.log' # Fallback location
if os.path.exists(datadir):
logfile = os.path.join(datadir, "install.log")
elif os.environ.get("TEMP") and os.path.exists(os.environ.get("TEMP"),""):
logfile = os.path.join(os.environ.get("TEMP"), "mymodule_install.log")
with open(logfile, 'a+') as f:
f.write("Opened\r\n")
f.write("Ran %s %s at %s" % (sys.executable, " ".join(sys.argv), datetime.datetime.now().isoformat()))
sys.stdout = f
sys.stderr = f
try:
main(sys.argv)
except Exception as e:
raise
f.close()
sys.exit(0)

UPD: on an off chance that the client machine has pywin32 installed, we try in-process creation first. Somewhat cleaner that way.
Here is another take. This assumes the package is called myapp, and that also becomes the executable that you want a shortcut to. Substitute your own package name and your own shortcut text.
Uses a Windows Scripting Host COM class - in process if possible, inside a Powershell command line as a subprocess if not. Tested on Python 3.6+.
from setuptools import setup
from setuptools.command.install import install
import platform, sys, os, site
from os import path, environ
def create_shortcut_under(root, exepath):
# Root is an env variable name -
# either ALLUSERSPROFILE for the all users' Start menu,
# or APPDATA for the current user specific one
profile = environ[root]
linkpath = path.join(profile, "Microsoft", "Windows", "Start Menu", "Programs", "My Python app.lnk")
try:
from win32com.client import Dispatch
from pywintypes import com_error
try:
sh = Dispatch('WScript.Shell')
link = sh.CreateShortcut(linkpath)
link.TargetPath = exepath
link.Save()
return True
except com_error:
return False
except ImportError:
import subprocess
s = "$s=(New-Object -COM WScript.Shell).CreateShortcut('" + linkpath + "');$s.TargetPath='" + exepath + "';$s.Save()"
return subprocess.call(['powershell', s], stdout = subprocess.DEVNULL, stderr = subprocess.DEVNULL) == 0
def create_shortcut(inst):
try:
exepath = path.join(path.dirname(sys.executable), "Scripts", "myapp.exe")
if not path.exists(exepath):
# Support for "pip install --user"
exepath = path.join(path.dirname(site.getusersitepackages()), "Scripts", "myapp.exe")
# If can't modify the global menu, fall back to the
# current user's one
if not create_shortcut_under('ALLUSERSPROFILE', exepath):
create_shortcut_under('APPDATA', exepath)
except:
pass
class my_install(install):
def run(self):
install.run(self)
if platform.system() == 'Windows':
create_shortcut(self)
#...
setup(
#...
cmdclass={'install': my_install},
entry_points={"gui_scripts": ["myapp = myapp.__main__:main"]},

Using Python to ssh with no modules

I'm in a pickle with writing a script that can SSH into device, run a command and parse that data out to a file. I've written this using Pyparsing and Exscript then I found out that the device I'm going to be using this on is using Python 2.4.4 and Debian 4.1.1 so the modules will not work on this. Now I am back to the drawing board trying to find out how to do this with NO modules. Anyone have any reference or point me in the right direction for this? Thank you in advance.
Here is my code:
from Exscript.util.interact import read_login
from Exscript.protocols import SSH2
import uuid
from pyparsing import *
import re
import yaml
account = read_login()
conn = SSH2()
conn.connect('172.0.0.1')
conn.login(account)
conn.execute('foobar')
data = conn.response
conn.send('exit\r')
conn.close()
###### PARSER ######
date_regex = re.compile(r'\d\d-\d\d-\d\d')
time_regex = re.compile(r'\d\d:\d\d:\d\d')
pairs = [{'category': 'General Information',
'kv': Group(Word(alphanums) + Word(alphanums))},
{'category': 'Last Reset:',
'kv': Group(Word(alphas, max=1) + Word(alphas)) + Literal(':').suppress()
+ Group(Regex(date_regex) + Regex(time_regex)
+ Optional(SkipTo(LineEnd())))
}
]
# build list of categories with associated parsing rules
categories = [Word("# ").suppress() + x['category']
+ OneOrMore(Group(x['kv']))
for x in pairs]
# account for thing you don't have specific rules for
categories.append(Word("#").suppress() + Optional(SkipTo(LineEnd())) +
Group(OneOrMore(Combine(Word(alphanums) + SkipTo(LineEnd()))))
)
# OR all the categories together
categories_ored = categories[0]
for c in categories[1:]:
categories_ored |= c
configDef = OneOrMore(categories_ored)
suppress_tokens = ["show all", "SSH>", "Active System Configuration"]
suppresses = [Literal(x).suppress() for x in suppress_tokens]
for s in suppresses:
configDef.ignore(s)
result = configDef.parseString(data)
for e in result:
print(e)
with open('/Users/MyMac/development/data.yml', 'w') as outfile:
outfile.write( yaml.dump(e))
UPDATE
I have followed the advice below and now have Pexpect installed and found a older version of Python-Pyparsing that I have also installed. So I'm on my way again to getting my scripts to work with modules. Thanks!

Looks like this is already solved, but...
As long as your SSH is configured for this host (or the host doesn't require you to log-in), you should be able to do the following.
import os
""" This will execute foobar on the remote host
and store the command output to a text file
on your machine."""
os.system("ssh 172.0.0.1 foobar > ~/data.txt")
""" Commence processing """
data = open("data.txt", mode='r')
# and so on and so on
You can also use the subprocess library, but os.system for these types of tasks is the simplest IMO.

Get total length of videos in a particular directory in python

I have downloaded a bunch of videos from coursera.org and have them stored in one particular folder. There are many individual videos in a particular folder (Coursera breaks a lecture into multiple short videos). I would like to have a python script which gives the combined length of all the videos in a particular directory. The video files are .mp4 format.

First, install the ffprobe command (it's part of FFmpeg) with
sudo apt install ffmpeg
then use subprocess.run() to run this bash command:
ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 -- <filename>
(which I got from http://trac.ffmpeg.org/wiki/FFprobeTips#Formatcontainerduration), like this:
from pathlib import Path
import subprocess
def video_length_seconds(filename):
result = subprocess.run(
[
"ffprobe",
"-v",
"error",
"-show_entries",
"format=duration",
"-of",
"default=noprint_wrappers=1:nokey=1",
"--",
filename,
],
capture_output=True,
text=True,
)
try:
return float(result.stdout)
except ValueError:
raise ValueError(result.stderr.rstrip("\n"))
# a single video
video_length_seconds('your_video.webm')
# all mp4 files in the current directory (seconds)
print(sum(video_length_seconds(f) for f in Path(".").glob("*.mp4")))
# all mp4 files in the current directory and all its subdirectories
# `rglob` instead of `glob`
print(sum(video_length_seconds(f) for f in Path(".").rglob("*.mp4")))
# all files in the current directory
print(sum(video_length_seconds(f) for f in Path(".").iterdir() if f.is_file()))
This code requires Python 3.7+ because that's when text= and capture_output= were added to subprocess.run. If you're using an older Python version, check the edit history of this answer.

Download MediaInfo and install it (don't install the bundled adware)
Go to the MediaInfo source downloads and in the "Source code, All included" row, choose the link next to "libmediainfo"
Find MediaInfoDLL3.py in the downloaded archive and extract it anywhere.
Example location: libmediainfo_0.7.62_AllInclusive.7z\MediaInfoLib\Source\MediaInfoDLL\MediaInfoDLL3.py
Now make a script for testing (sources below) in the same directory.
Execute the script.
MediaInfo works on POSIX too. The only difference is that an so is loaded instead of a DLL.
Test script (Python 3!)
import os
os.chdir(os.environ["PROGRAMFILES"] + "\\mediainfo")
from MediaInfoDLL3 import MediaInfo, Stream
MI = MediaInfo()
def get_lengths_in_milliseconds_of_directory(prefix):
for f in os.listdir(prefix):
MI.Open(prefix + f)
duration_string = MI.Get(Stream.Video, 0, "Duration")
try:
duration = int(duration_string)
yield duration
print("{} is {} milliseconds long".format(f, duration))
except ValueError:
print("{} ain't no media file!".format(f))
MI.Close()
print(sum(get_lengths_in_milliseconds_of_directory(os.environ["windir"] + "\\Performance\\WinSAT\\"
)), "milliseconds of content in total")

In addition to Janus Troelsen's answer above, I would like to point out a small problem I
encountered when implementing his answer. I followed his instructions one by one but had different results on windows (7) and linux (ubuntu). His instructions worked perfectly under linux but I had to do a small hack to get it to work on windows. I am using a 32-bit python 2.7.2 interpreter on windows so I utilized MediaInfoDLL.py. But that was not enough to get it to work for me I was receiving this error at this point in the process:
"WindowsError: [Error 193] %1 is not a valid Win32 application".
This meant that I was somehow using a resource that was not 32-bit, it had to be the DLL MediaInfoDLL.py was loading. If you look at the MediaInfo intallation directory you will see 3 dlls MediaInfo.dll is 64-bit while MediaInfo_i386.dll is 32-bit. MediaInfo_i386.dll is the one which I had to use because of my python setup. I went to
MediaInfoDLL.py (which I already had included in my project) and changed this line:
MediaInfoDLL_Handler = windll.MediaInfo
to
MediaInfoDLL_Handler = WinDLL("C:\Program Files (x86)\MediaInfo\MediaInfo_i386.dll")
I didn't have to change anything for it to work in linux

Nowadays pymediainfo is available, so Janus Troelsen's answer could be simplified.
You need to install MediaInfo and pip install pymediainfo. Then the following code would print you the total length of all video files:
import os
from pymediainfo import MediaInfo
def get_track_len(file_path):
media_info = MediaInfo.parse(file_path)
for track in media_info.tracks:
if track.track_type == "Video":
return int(track.duration)
return 0
print(sum(get_track_len(f) for f in os.listdir('directory with video files')))

This link shows how to get the length of a video file https://stackoverflow.com/a/3844467/735204
import subprocess
def getLength(filename):
result = subprocess.Popen(["ffprobe", filename],
stdout = subprocess.PIPE, stderr = subprocess.STDOUT)
return [x for x in result.stdout.readlines() if "Duration" in x]
If you're using that function, you can then wrap it up with something like
import os
for f in os.listdir('.'):
print "%s: %s" % (f, getLength(f))

Here's my take. I did this on Windows. I took the answer from Federico above, and changed the python program a little bit to traverse a tree of folders with video files. So you need to go above to see Federico's answer, to install MediaInfo and to pip install pymediainfo, and then write this program, summarize.py:
import os
import sys
from pymediainfo import MediaInfo
number_of_video_files = 0
def get_alternate_len(media_info):
myJson = media_info.to_data()
myArray = myJson['tracks']
for track in myArray:
if track['track_type'] == 'General' or track['track_type'] == 'Video':
if 'duration' in track:
return int(track['duration'] / 1000)
return 0
def get_track_len(file_path):
global number_of_video_files
media_info = MediaInfo.parse(file_path)
for track in media_info.tracks:
if track.track_type == "Video":
number_of_video_files += 1
if type(track.duration) == int:
len_in_sec = int(track.duration / 1000)
elif type(track.duration) == str:
len_in_sec = int(float(track.duration) / 1000)
else:
len_in_sec = get_alternate_len(media_info)
if len_in_sec == 0:
print("File path = " + file_path + ", problem in type of track.duration")
return len_in_sec
return 0
sum_in_secs = 0.0
os.chdir(sys.argv[1])
for root, dirs, files in os.walk("."):
for name in files:
sum_in_secs += get_track_len(os.path.join(root, name))
hours = int(sum_in_secs / 3600)
remain = sum_in_secs - hours * 3600
minutes = int(remain / 60)
seconds = remain - minutes * 60
print("Directory: " + sys.argv[1])
print("Total number of video files is " + str(number_of_video_files))
print("Length: %d:%02d:%02d" % (hours, minutes, seconds))
Run it: python summarize.py <DirPath>
Have fun. I found I have about 1800 hours of videos waiting for me to have some free time. Yeah sure

How can I configure IPython 0.12 to log all input to log-files split out by timestamp? I have this working in IPython 0.10.1

Here's the relevant part of my ~/.ipython/ipy_user_conf.py file on a system running IPython .10.1:
import IPython.ipapi
ip = IPython.ipapi.get()
import os
from time import strftime
def main():
try:
basedir = '/home/stu/.ipython/log/'
logdir = strftime('%Y-%m-%d')
directory = os.path.join(basedir, logdir)
if not os.path.exists(directory):
os.mkdir(directory)
logfilename = strftime("%s") + ".log"
filename = os.path.join(directory, logfilename)
ip.IP.logger.logstart(logfname=filename, logmode='append')
except RuntimeError:
print "something went wrong!"
main()
This logs all input during interactive sessions to files of the form: /home/stu/.ipython/log/2012-04-26/1335454387.log
How can I configure IPython 0.12 to create logs in this manner?
Is it ideal to create a ~/.ipython/profile_default/startup/00-logstuff.py file for this?
Does anyone have a set of ~.ipython/<profile>/startup/ scripts which could serve as good practical examples?
Thanks

Have a look at this recipe. You can modify it to do folder-per-day rather than file-per-day easily enough.
http://wiki.ipython.org/Cookbook/DatedLog