We Keep Coding

Python ValueError: not enough values to unpack (expected 3, got 1)

i tried writing this code but had an error. used windows 10 and python 3.8.5
#usage
#python3 coex.py combo.txt extracted.txt
from sys import argv
import re
script , combo_file , ex_file = argv
cfile = open(combo_file)
xfile = open(ex_file, 'w')
def rexmail(cfile):
rexmail = re.compile(r'[a-zA-Z0-9_.+-]+#[a-zA-Z0-9.-]+:[a-zA-Z0-9._-]+')
cfile = rexmail.findall(cfile.read())
lenofclist = len(cfile)
for i in range(lenofclist):
xfile.write("\n")
xfile.write(str(cfile[i]))
print("[+]*********EXTRACTING DONE***********[+]\n")
print("[+]*********CHECK extracted.txt FILE FOR EMAIL:PASS COMBOS*************[+]\n")
def header():
print('''
made with <3
_______ ___ ___ _________ ________ ________ ________ _________ ________ ________
|\ ___ \ |\ \ / /| |\___ ___\ |\ __ \ |\ __ \ |\ ____\ |\___ ___\ |\ __ \ |\ __ \
\ \ __/| \ \ \/ / / \|___ \ \_| \ \ \|\ \ \ \ \|\ \ \ \ \___| \|___ \ \_| \ \ \|\ \ \ \ \|\ \
\ \ \_|/__ \ \ / / \ \ \ \ \ _ _\ \ \ __ \ \ \ \ \ \ \ \ \ \\\ \ \ \ _ _\
\ \ \_|\ \ / \/ \ \ \ \ \ \\ \| \ \ \ \ \ \ \ \____ \ \ \ \ \ \\\ \ \ \ \\ \|
\ \_______\ / /\ \ \ \__\ \ \__\\ _\ \ \__\ \__\ \ \_______\ \ \__\ \ \_______\ \ \__\\ _\
\|_______| /__/ /\ __\ \|__| \|__|\|__| \|__|\|__| \|_______| \|__| \|_______| \|__|\|__|
|__|/ \|__|
EMAIL:PASS extractor from any txt file .
''')
header()
rexmail(cfile)
Error:
Traceback (most recent call last):
File "C:\Users\BRS\Desktop\minecraft\Combo-Extractor-master\coex.py", line 8, in <module>
script , combo_file , ex_file = argv
ValueError: not enough values to unpack (expected 3, got 1)
I dont really get what's off. please help me by correcting this code. and if possible tell me why this happens
is it a problem with tuples or what pls help with this

argv actually return a list whose first element (i.e. index 0 element) is the location of the python file. To correct this error, use
script, combo_file, ex_file = argv[1:]
Alternatively, you can also use _, script, combo_file, ex_file = argv
Relevant Documentation - https://docs.python.org/3/library/sys.html
(Tested on Windows 10 (64 Bit) Python 3.7.4)

subprocess call of ffmpeg returns negative value

Here is the original command line call that works at shell:
ffmpeg -i /Users/abc/Desktop/Test/Full_Mov.mov -vf "drawtext=fontfile=/System/Library/Fonts/Keyboard.ttf: text='SCENE BLAH BLAH - %{frame_num}': start_number=1: x=(w-tw)/2: y=h-(2*lh): fontcolor=white: fontsize=20: box=1: boxcolor=black: boxborderw=5","format=yuv420p" -f segment -segment_frames 123 -reset_timestamps 1 -c:a copy -map 0 "/Users/abc/Desktop/Test/%03d_test40.mov"
Getting a negative value when trying to run ffmpeg via subprocess, causing it to fail and not process. I've even tried dumbing down the call to just run ffmpeg by itself with no arguments, and it still returns a negative value.
import subprocess
ffmpeg = "/usr/local/bin/ffmpeg"
source = "/Users/abc/Desktop/Test/Full_Mov.mov"
destination = "/Users/abc/Desktop/Test/%03d_test40.mov"
cmd = "%s -i %s -vf \"drawtext=fontfile=/System/Library/Fonts/Keyboard.ttf: text='SCENE BLAH BLAH - %%{frame_num}': start_number=1: x=(w-tw)/2: y=h-(2*lh): fontcolor=black: fontsize=20: box=1: boxcolor=white: boxborderw=5\",\"format=yuv420p\" -f segment -segment_frames 123 -reset_timestamps 1 -c:a copy -map 0 \"%s\"" % (ffmpeg, source, destination)
log("ffmpeg cmd: %s" % cmd)
log(subprocess.check_output(cmd, shell=True))
When running full command:
CalledProcessError: Command '/usr/local/bin/ffmpeg -i /Users/abc/Desktop/Full_Mov.mov -vf "drawtext=fontfile=/System/Library/Fonts/Keyboard.ttf: text='SCENE BLAH BLAH - %{frame_num}': start_number=1: x=(w-tw)/2: y=h-(2*lh): fontcolor=black: fontsize=20: box=1: boxcolor=white: boxborderw=5","format=yuv420p" -f segment -segment_frames 318 -reset_timestamps 1 -c:a copy -map 0 "/Users/abc/Desktop/Test/%03d.tmp.mov"' returned non-zero exit status -8
raise CalledProcessError(retcode, cmd, output=output)
When running just ffmpeg no arguments:
CalledProcessError: Command '/usr/local/bin/ffmpeg' returned non-zero exit status 1
raise CalledProcessError(retcode, cmd, output=output)
Latest output with suggested changes below:
subprocess.CalledProcessError: Command '['/usr/local/bin/ffmpeg', '-i', '/Users/szaharak/Desktop/Flix_Test/ep888_sq66_main_mov_2019_05_20_14_15.mov', '-vf', 'drawtext=fontfile=/System/Library/Fonts/Keyboard.ttf:', 'text=SCENE BLAH BLAH - %%{frame_num}:', 'start_number=1:', 'x=(w-tw)/2:', 'y=h-(2*lh):', 'fontcolor=black:', 'fontsize=20:', 'box=1:', 'boxcolor=white:', 'boxborderw=5', 'format=yuv420p', '-f', 'segment', '-segment_frames', '123', '-reset_timestamps', '1', '-c:a', 'copy', '-map', '0', '/Users/szaharak/Desktop/Flix_Test/%03d_test40.mov']' returned non-zero exit status 1
[NULL # 0x7f88c3012200] Unable to find a suitable output format for 'text=SCENE BLAH BLAH - %%{frame_num}:'
text=SCENE BLAH BLAH - %%{frame_num}:: Invalid argument
And here is latest...
>>> rc = subprocess.check_call(cmd)
ffmpeg version N-93891-ge1839283bc-tessus https://evermeet.cx/ffmpeg/ Copyright (c) 2000-2019 the FFmpeg developers
built with Apple LLVM version 10.0.1 (clang-1001.0.46.4)
configuration: --cc=/usr/bin/clang --prefix=/opt/ffmpeg --extra-version=tessus --enable-avisynth --enable-fontconfig --enable-gpl --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libfreetype --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libmysofa --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-version3 --pkg-config-flags=--static --disable-ffplay
libavutil 56. 28.100 / 56. 28.100
libavcodec 58. 52.102 / 58. 52.102
libavformat 58. 27.103 / 58. 27.103
libavdevice 58. 7.100 / 58. 7.100
libavfilter 7. 53.101 / 7. 53.101
libswscale 5. 4.101 / 5. 4.101
libswresample 3. 4.100 / 3. 4.100
libpostproc 55. 4.100 / 55. 4.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/szaharak/Desktop/Flix_Test/ep888_sq66_main_mov_2019_05_20_14_15.mov':
Metadata:
major_brand : qt
minor_version : 537199360
compatible_brands: qt
creation_time : 2019-05-14T17:58:24.000000Z
com.apple.quicktime.player.movie.audio.gain: 1.000000
com.apple.quicktime.player.movie.audio.treble: 0.000000
com.apple.quicktime.player.movie.audio.bass: 0.000000
com.apple.quicktime.player.movie.audio.balance: 0.000000
com.apple.quicktime.player.movie.audio.pitchshift: 0.000000
com.apple.quicktime.player.movie.audio.mute:
com.apple.quicktime.player.movie.visual.brightness: 0.000000
com.apple.quicktime.player.movie.visual.color: 1.000000
com.apple.quicktime.player.movie.visual.tint: 0.000000
com.apple.quicktime.player.movie.visual.contrast: 1.000000
com.apple.quicktime.player.version: 7.6.6 (7.6.6)
com.apple.quicktime.version: 7.7.3 (2943.14) 0x7738000 (Mac OS X, 10.11.6, 15G22010)
Duration: 00:01:12.67, start: 0.000000, bitrate: 23379 kb/s
Stream #0:0(eng): Video: mjpeg (Baseline) (jpeg / 0x6765706A), yuvj422p(pc, bt470bg/unknown/unknown), 1280x720 [SAR 72:72 DAR 16:9], 21838 kb/s, 24 fps, 24 tbr, 24k tbn, 24k tbc (default)
Metadata:
creation_time : 2019-05-14T17:58:24.000000Z
handler_name : Apple Video Media Handler
encoder : Photo - JPEG
Stream #0:1(eng): Audio: pcm_s16be (twos / 0x736F7774), 48000 Hz, stereo, s16, 1536 kb/s (default)
Metadata:
creation_time : 2019-05-14T17:58:24.000000Z
handler_name : Apple Sound Media Handler
[NULL # 0x7f8ddc8ce200] Unable to find a suitable output format for 'text=SCENE BLAH BLAH - %{frame_num}: start_number=1: x=(w-tw)/2: y=h-(2*lh): fontcolor=black: fontsize=20: box=1: boxcolor=white: boxborderw=5: format=yuv420p'
text=SCENE BLAH BLAH - %{frame_num}: start_number=1: x=(w-tw)/2: y=h-(2*lh): fontcolor=black: fontsize=20: box=1: boxcolor=white: boxborderw=5: format=yuv420p: Invalid argument
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/Cellar/python#2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 190, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/local/bin/ffmpeg', '-i', '/Users/szaharak/Desktop/Flix_Test/ep888_sq66_main_mov_2019_05_20_14_15.mov', '-vf', 'drawtext=fontfile=/System/Library/Fonts/Keyboard.ttf:', 'text=SCENE BLAH BLAH - %{frame_num}: start_number=1: x=(w-tw)/2: y=h-(2*lh): fontcolor=black: fontsize=20: box=1: boxcolor=white: boxborderw=5: format=yuv420p', '-f', 'segment', '-segment_frames', '123', '-reset_timestamps', '1', '-c:a', 'copy', '-map', '0', '/Users/szaharak/Desktop/Flix_Test/%03d_test40.mov']' returned non-zero exit status 1

ffmpeg command line is tricky as there are a lot of special arguments and mistakes are sometimes difficult to understand, the program sometimes interprets/discards wrong arguments, resulting in cryptic error messages.
Another difficulty is using check_output. You don't really need to do this, and if the program fails you won't have any output at all.
I would try the following (that I could not test), which is to:
drop shell=True
pass a list of arguments instead of composing the string yourself. It allows to forget about the quoting/escaping. You don't need to format the command, source & destination since they're standalone arguments:
Don't log, just print with check_call.
As a bonus, you'll get the output in real-time, instead of in the end...
note that I have removed the single quotes in "text=SCENE BLAH BLAH - %{frame_num}:, too and unescaped the % char.
like this:
cmd = [ffmpeg,"-i",source,"-vf",
"drawtext=fontfile=/System/Library/Fonts/Keyboard.ttf:",
"text=SCENE BLAH BLAH - %{frame_num}: start_number=1: x=(w-tw)/2: y=h-(2*lh): fontcolor=black: fontsize=20: box=1: boxcolor=white: boxborderw=5: format=yuv420p",
"-f","segment","-segment_frames","123",
"-reset_timestamps","1","-c:a","copy","-map","0",destination]
log("ffmpeg cmd: %s" % cmd)
rc = subprocess.check_call(cmd)
if rc:
raise Exception("ffmpeg failed")
for the case of ffmpeg, you could migrate to ffmpeg python module. Could save some argument parsing trouble too.

How can I convert this Python2.x code to Python3.x?

File "gridworld.py", line 19, in <module>
import util
File "/home/user/Desktop/reinforcement/util.py", line 23
fixedState = (3, 2147483648L, 507801126L, 683453281L, 310439348L, 2597246090L, \
^
SyntaxError: invalid syntax
class FixedRandom:
def __init__(self):
fixedState = (3, (2147483648L, 507801126L, 683453281L, 310439348L, 2597246090L, \
2209084787L, 2267831527L, 979920060L, 3098657677L, 37650879L, 807947081L, 3974896263L, \
881243242L, 3100634921L, 1334775171L, 3965168385L, 746264660L, 4074750168L, 500078808L, \
776561771L, 702988163L, 1636311725L, 2559226045L, 157578202L, 2498342920L, 2794591496L, \
4130598723L, 496985844L, 2944563015L, 3731321600L, 3514814613L, 3362575829L, 3038768745L, \
2206497038L, 1108748846L, 1317460727L, 3134077628L, 988312410L, 1674063516L, 746456451L, \
3958482413L, 1857117812L, 708750586L, 1583423339L, 3466495450L, 1536929345L, 1137240525L, \
3875025632L, 2466137587L, 1235845595L, 4214575620L, 3792516855L, 657994358L, 1241843248L, \
1695651859L, 3678946666L, 1929922113L, 2351044952L, 2317810202L, 2039319015L, 460787996L, \
3654096216L, 4068721415L, 1814163703L, 2904112444L, 1386111013L, 574629867L, 2654529343L, \
3833135042L, 2725328455L, 552431551L, 4006991378L, 1331562057L, 3710134542L, 303171486L, \
1203231078L, 2670768975L, 54570816L, 2679609001L, 578983064L, 1271454725L, 3230871056L, \
2496832891L, 2944938195L, 1608828728L, 367886575L, 2544708204L, 103775539L, 1912402393L, \
1098482180L, 2738577070L, 3091646463L, 1505274463L, 2079416566L, 659100352L, 839995305L, \
1696257633L, 274389836L, 3973303017L, 671127655L, 1061109122L, 517486945L, 1379749962L, \
3421383928L, 3116950429L, 2165882425L, 2346928266L, 2892678711L, 2936066049L, 1316407868L, \
2873411858L, 4279682888L, 2744351923L, 3290373816L, 1014377279L, 955200944L, 4220990860L, \
2386098930L, 1772997650L, 3757346974L, 1621616438L, 2877097197L, 442116595L, 2010480266L, \
2867861469L, 2955352695L, 605335967L, 2222936009L, 2067554933L, 4129906358L, 1519608541L, \
1195006590L, 1942991038L, 2736562236L, 279162408L, 1415982909L, 4099901426L, 1732201505L, \
2934657937L, 860563237L, 2479235483L, 3081651097L, 2244720867L, 3112631622L, 1636991639L, \
3860393305L, 2312061927L, 48780114L, 1149090394L, 2643246550L, 1764050647L, 3836789087L, \
3474859076L, 4237194338L, 1735191073L, 2150369208L, 92164394L, 756974036L, 2314453957L, \
323969533L, 4267621035L, 283649842L, 810004843L, 727855536L, 1757827251L, 3334960421L, \
3261035106L, 38417393L, 2660980472L, 1256633965L, 2184045390L, 811213141L, 2857482069L, \
2237770878L, 3891003138L, 2787806886L, 2435192790L, 2249324662L, 3507764896L, 995388363L, \
856944153L, 619213904L, 3233967826L, 3703465555L, 3286531781L, 3863193356L, 2992340714L, \
413696855L, 3865185632L, 1704163171L, 3043634452L, 2225424707L, 2199018022L, 3506117517L, \
3311559776L, 3374443561L, 1207829628L, 668793165L, 1822020716L, 2082656160L, 1160606415L, \
3034757648L, 741703672L, 3094328738L, 459332691L, 2702383376L, 1610239915L, 4162939394L, \
557861574L, 3805706338L, 3832520705L, 1248934879L, 3250424034L, 892335058L, 74323433L, \
3209751608L, 3213220797L, 3444035873L, 3743886725L, 1783837251L, 610968664L, 580745246L, \
4041979504L, 201684874L, 2673219253L, 1377283008L, 3497299167L, 2344209394L, 2304982920L, \
3081403782L, 2599256854L, 3184475235L, 3373055826L, 695186388L, 2423332338L, 222864327L, \
1258227992L, 3627871647L, 3487724980L, 4027953808L, 3053320360L, 533627073L, 3026232514L, \
2340271949L, 867277230L, 868513116L, 2158535651L, 2487822909L, 3428235761L, 3067196046L, \
3435119657L, 1908441839L, 788668797L, 3367703138L, 3317763187L, 908264443L, 2252100381L, \
764223334L, 4127108988L, 384641349L, 3377374722L, 1263833251L, 1958694944L, 3847832657L, \
1253909612L, 1096494446L, 555725445L, 2277045895L, 3340096504L, 1383318686L, 4234428127L, \
1072582179L, 94169494L, 1064509968L, 2681151917L, 2681864920L, 734708852L, 1338914021L, \
1270409500L, 1789469116L, 4191988204L, 1716329784L, 2213764829L, 3712538840L, 919910444L, \
1318414447L, 3383806712L, 3054941722L, 3378649942L, 1205735655L, 1268136494L, 2214009444L, \
2532395133L, 3232230447L, 230294038L, 342599089L, 772808141L, 4096882234L, 3146662953L, \
2784264306L, 1860954704L, 2675279609L, 2984212876L, 2466966981L, 2627986059L, 2985545332L, \
2578042598L, 1458940786L, 2944243755L, 3959506256L, 1509151382L, 325761900L, 942251521L, \
4184289782L, 2756231555L, 3297811774L, 1169708099L, 3280524138L, 3805245319L, 3227360276L, \
3199632491L, 2235795585L, 2865407118L, 36763651L, 2441503575L, 3314890374L, 1755526087L, \
17915536L, 1196948233L, 949343045L, 3815841867L, 489007833L, 2654997597L, 2834744136L, \
417688687L, 2843220846L, 85621843L, 747339336L, 2043645709L, 3520444394L, 1825470818L, \
647778910L, 275904777L, 1249389189L, 3640887431L, 4200779599L, 323384601L, 3446088641L, \
4049835786L, 1718989062L, 3563787136L, 44099190L, 3281263107L, 22910812L, 1826109246L, \
745118154L, 3392171319L, 1571490704L, 354891067L, 815955642L, 1453450421L, 940015623L, \
796817754L, 1260148619L, 3898237757L, 176670141L, 1870249326L, 3317738680L, 448918002L, \
4059166594L, 2003827551L, 987091377L, 224855998L, 3520570137L, 789522610L, 2604445123L, \
454472869L, 475688926L, 2990723466L, 523362238L, 3897608102L, 806637149L, 2642229586L, \
2928614432L, 1564415411L, 1691381054L, 3816907227L, 4082581003L, 1895544448L, 3728217394L, \
3214813157L, 4054301607L, 1882632454L, 2873728645L, 3694943071L, 1297991732L, 2101682438L, \
3952579552L, 678650400L, 1391722293L, 478833748L, 2976468591L, 158586606L, 2576499787L, \
662690848L, 3799889765L, 3328894692L, 2474578497L, 2383901391L, 1718193504L, 3003184595L, \
3630561213L, 1929441113L, 3848238627L, 1594310094L, 3040359840L, 3051803867L, 2462788790L, \
954409915L, 802581771L, 681703307L, 545982392L, 2738993819L, 8025358L, 2827719383L, \
770471093L, 3484895980L, 3111306320L, 3900000891L, 2116916652L, 397746721L, 2087689510L, \
721433935L, 1396088885L, 2751612384L, 1998988613L, 2135074843L, 2521131298L, 707009172L, \
2398321482L, 688041159L, 2264560137L, 482388305L, 207864885L, 3735036991L, 3490348331L, \
1963642811L, 3260224305L, 3493564223L, 1939428454L, 1128799656L, 1366012432L, 2858822447L, \
1428147157L, 2261125391L, 1611208390L, 1134826333L, 2374102525L, 3833625209L, 2266397263L, \
3189115077L, 770080230L, 2674657172L, 4280146640L, 3604531615L, 4235071805L, 3436987249L, \
509704467L, 2582695198L, 4256268040L, 3391197562L, 1460642842L, 1617931012L, 457825497L, \
1031452907L, 1330422862L, 4125947620L, 2280712485L, 431892090L, 2387410588L, 2061126784L, \
896457479L, 3480499461L, 2488196663L, 4021103792L, 1877063114L, 2744470201L, 1046140599L, \
2129952955L, 3583049218L, 4217723693L, 2720341743L, 820661843L, 1079873609L, 3360954200L, \
3652304997L, 3335838575L, 2178810636L, 1908053374L, 4026721976L, 1793145418L, 476541615L, \
973420250L, 515553040L, 919292001L, 2601786155L, 1685119450L, 3030170809L, 1590676150L, \
1665099167L, 651151584L, 2077190587L, 957892642L, 646336572L, 2743719258L, 866169074L, \
851118829L, 4225766285L, 963748226L, 799549420L, 1955032629L, 799460000L, 2425744063L, \
2441291571L, 1928963772L, 528930629L, 2591962884L, 3495142819L, 1896021824L, 901320159L, \
3181820243L, 843061941L, 3338628510L, 3782438992L, 9515330L, 1705797226L, 953535929L, \
764833876L, 3202464965L, 2970244591L, 519154982L, 3390617541L, 566616744L, 3438031503L, \
1853838297L, 170608755L, 1393728434L, 676900116L, 3184965776L, 1843100290L, 78995357L, \
2227939888L, 3460264600L, 1745705055L, 1474086965L, 572796246L, 4081303004L, 882828851L, \
1295445825L, 137639900L, 3304579600L, 2722437017L, 4093422709L, 273203373L, 2666507854L, \
3998836510L, 493829981L, 1623949669L, 3482036755L, 3390023939L, 833233937L, 1639668730L, \
1499455075L, 249728260L, 1210694006L, 3836497489L, 1551488720L, 3253074267L, 3388238003L, \
2372035079L, 3945715164L, 2029501215L, 3362012634L, 2007375355L, 4074709820L, 631485888L, \
3135015769L, 4273087084L, 3648076204L, 2739943601L, 1374020358L, 1760722448L, 3773939706L, \
1313027823L, 1895251226L, 4224465911L, 421382535L, 1141067370L, 3660034846L, 3393185650L, \
1850995280L, 1451917312L, 3841455409L, 3926840308L, 1397397252L, 2572864479L, 2500171350L, \
3119920613L, 531400869L, 1626487579L, 1099320497L, 407414753L, 2438623324L, 99073255L, \
3175491512L, 656431560L, 1153671785L, 236307875L, 2824738046L, 2320621382L, 892174056L, \
230984053L, 719791226L, 2718891946L, 624L), None)
self.random = random.Random()
self.random.setstate(fixedState)
fixedState = (3, (2147483648L, 507801126L, 683453281L, 310439348L, 2597246090L, \
^
SyntaxError: invalid syntax

There is no trailing L needed to define an integer in Python3, which you are using in your code. You need to hence remove the trailing L from your integers and define that without that.
From the docs: https://docs.python.org/3/whatsnew/3.0.html
Integer literals no longer support a trailing l or L.
In [11]: a = 1
In [12]: a = 1L
File "<ipython-input-12-61aa3f5b6495>", line 1
a = 1L
^
SyntaxError: invalid syntax

custom sorting for find command output

I'm trying to get sorted directory/file list with unix "find" command.
# find . -type f
.
./bin
./data
./data/disks
./inc
./inc/calls
./inc/calls/show
./inc/calls/show/system
./inc/calls/show/cli
./inc/calls/show/network
./inc/calls/show/stats
./inc/calls/services
./inc/calls/services/ntp
./inc/calls/services/tsa
./inc/calls/services/webgui
./inc/calls/services/engine
./inc/calls/system
./inc/calls/change
./inc/calls/change/password
./inc/calls/change/network
./inc/calls/disk
./inc/calls/disk/encr
./inc/etc
I want to sort it like:
./inc/calls/show/system \
./inc/calls/show/cli \
./inc/calls/show/network \
./inc/calls/show/stats \
./inc/calls/services/ntp \
./inc/calls/services/tsa \
./inc/calls/services/webgui \
./inc/calls/services/engine \
./inc/calls/change/password \
./inc/calls/change/network \
./inc/calls/disk/encr \
./inc/calls/system \
./inc/calls/change \
./inc/calls/services \
./inc/calls/disk \
./inc/calls/show \
./inc/calls \
./data/disks \
./inc/etc \
./bin \
./data \
./inc
Which node (directory/file) has more child (directory/files) should be first... i want to do it with bash or python... What is the best way to do that?

Match lines containing / and prepend the number of fields to the line using / as the separator, sort on the numbers of fields and remove the count.
$ awk -F/ '/\//{print NF,$0}' file | sort -nrk1 | cut -d' ' -f2-
./inc/calls/show/system
./inc/calls/show/stats
./inc/calls/show/network
./inc/calls/show/cli
./inc/calls/services/webgui
./inc/calls/services/tsa
./inc/calls/services/ntp
./inc/calls/services/engine
./inc/calls/disk/encr
./inc/calls/change/password
./inc/calls/change/network
./inc/calls/system
./inc/calls/show
./inc/calls/services
./inc/calls/disk
./inc/calls/change
./inc/etc
./inc/calls
./data/disks
./inc
./data
./bin

I would use python and try to convert:
a/b
a/c
b/e/f
b/e/g
in something like:
{'a': {'b': {}, 'c': {}},
'b': {'e': {'f': {}, 'g': {}}},
}
To achieve this:
def add_list_to_dict(lst,d):
key, lst = lst[0], lst[1:]
if not key in d:
d[key] = {}
if lst:
add_list_to_dict(lst,d[key])
d = {}
for path in paths:
add_list_to_dict(path.split('/'),d)

How do I parse lines from a log file?

I need to extract the values of following output:
Oct 6 17:29:52 FW kernel: [ 5470.058450] ipTables: IN= OUT=eth0 SRC=192.168.1.116 DST=192.168.1.110 LEN=516 TOS=0x10 PREC=0x00 TTL=64 ID=4949 DF PROTO=TCP SPT=22 DPT=46216 WINDOW=446 RES=0x00 ACK PSH URGP=0
I'd need for example the value of PROTO stored in a value. tried shellscripting, my problem is that it works only if the log entry is in the same order everytime.
so this doens't work:
while read line
do
in_if=`echo $line | cut -d ' ' -f 10 | cut -d '=' -f 2`;
out_if=`echo $line | cut -d ' ' -f 11 | cut -d '=' -f 2`;
src_ip=`echo $line | cut -d ' ' -f 12 | cut -d '=' -f 2`;
dst_ip=`echo $line | cut -d ' ' -f 13 | cut -d '=' -f 2`;
pro=`echo $line | cut -d ' ' -f 20 | cut -d '=' -f 2`;
echo "$in_if,$out_if,$src_ip,$dst_ip,$pro" >> output.csv;
done < $tmp_file

Python does this conveniently. A general solution that gets all the KEY=value pairs is:
import re
import fileinput
pair_re = re.compile('([^ ]+)=([^ ]+)') # Matches KEY=value pair
for line in fileinput.input(): # The script accepts both data from stdin or a filename
line = line.rstrip() # Removes final spaces and newlines
data = dict(pair_re.findall(line)) # Fetches all the KEY=value pairs and puts them in a dictionary
# Example of usage:
print "PROTO =", data['PROTO'], "SRC =", data['SRC'] # Easy access to any value
This is arguably more legible, flexible and convenient than a shell script.

You can do this without touching Perl. You were on the right track, but with a regex you can search by name, not position.
Also, you should put quotes around $line so you don't get burned by any pipes or semicolons hanging around.
pro=`echo "$line" | grep -o 'PROTO=\w+\+' | cut -d '=' -f 2`;
Of course, if you did want to use Perl, you could make a much slicker solution:
#!/usr/bin/perl
while(<>) {
/IN=(\S*) .*OUT=(\S*) .*SRC=(\S*) .*DST=(\S*) .*PROTO=(\S*)/
and print "$1,$2,$3,$4,$5\n";
}
Then call:
./thatScript.pl logFile.txt >>output.csv

you don't even need to cut:
grep -Po "(?<=PROTO=)\w+" yourFile
OR
sed -r 's/.*PROTO=(\w+).*/\1/' yourFile
OR
awk -F'PROTO=' '{split($2,a," ");print a[1]}' yourfile
test:
kent$ echo "Oct 6 17:29:52 FW kernel: [ 5470.058450] ipTables: IN= OUT=eth0 SRC=192.168.1.116 DST=192.168.1.110 LEN=516 TOS=0x10 PREC=0x00 TTL=64 ID=4949 DF PROTO=TCP SPT=22 DPT=46216 WINDOW=446 RES=0x00 ACK PSH URGP=0"|grep -Po "(?<=PROTO=)\w+"
TCP
kent$ echo "Oct 6 17:29:52 FW kernel: [ 5470.058450] ipTables: IN= OUT=eth0 SRC=192.168.1.116 DST=192.168.1.110 LEN=516 TOS=0x10 PREC=0x00 TTL=64 ID=4949 DF PROTO=TCP SPT=22 DPT=46216 WINDOW=446 RES=0x00 ACK PSH URGP=0"|sed -r 's/.*PROTO=(\w+).*/\1/'
TCP
kent$ echo "Oct 6 17:29:52 FW kernel: [ 5470.058450] ipTables: IN= OUT=eth0 SRC=192.168.1.116 DST=192.168.1.110 LEN=516 TOS=0x10 PREC=0x00 TTL=64 ID=4949 DF PROTO=TCP SPT=22 DPT=46216 WINDOW=446 RES=0x00 ACK PSH URGP=0"|awk -F'PROTO=' '{split($2,a," ");print a[1]}'
TCP

A straightforward Perl solution might be the most readable one:
#!/usr/bin/env perl
use strict; use warnings;
my $s = q{Oct 6 17:29:52 FW kernel: [ 5470.058450] ipTables: IN= OUT=eth0
SRC=192.168.1.116 DST=192.168.1.110 LEN=516 TOS=0x10 PREC=0x00 TTL=64
ID=4949 DF PROTO=TCP SPT=22 DPT=46216 WINDOW=446 RES=0x00 ACK PSH URGP=0};
while ($s =~ /(?<k> [A-Z]+) = (?<v> \S*)/xg) {
print "'$+{k}' = '$+{v}'\n";
}
C:\Temp> z
'IN' = ''
'OUT' = 'eth0'
'SRC' = '192.168.1.116'
'DST' = '192.168.1.110'
'LEN' = '516'
'TOS' = '0x10'
'PREC' = '0x00'
'TTL' = '64'
'ID' = '4949'
'PROTO' = 'TCP'
'SPT' = '22'
'DPT' = '46216'
'WINDOW' = '446'
'RES' = '0x00'
'URGP' = '0'
You can also assign the information in the log line to a hash:
my %entry = ($s =~ /(?<k> [A-Z]+) = (?<v> \S*)/xg);

In perl this should do it
#consider the $a variable has the log file my
$a = <<log file>>;
my $desired_answer;
#regex
if ($a =~ m/PROTO=(.*?) /ig)
{ $desired_answer=$1; }

thanks for all the responses!
i chose the way of shellscripting using egrep and regex...
in_if=`echo "$line" | egrep -Eo 'IN=eth[0-9]*\b' | cut -d '=' -f 2`;
out_if=`echo "$line" | egrep -Eo 'OUT=eth[0-9]*\b' | cut -d '=' -f 2`;
src_ip=`echo "$line" | egrep -Eo 'SRC=[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | cut -d '=' -f 2`;
dst_ip=`echo "$line" | egrep -Eo 'DST=[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | cut -d '=' -f 2`;
pro=`echo "$line" | grep -o 'PROTO=[A-Z]*\b' | cut -d '=' -f 2`;