Using subprocess on Popen option - python
I use python 2.7 and ffmpeg on Mac OS 10.10.
When use some command from python, coded like below
p = subprocess.Popen(cmd.strip().split(" "))
This goes well all the time, but not in ffmpeg filter_complex case.
Below code can run on the terminal with directly input.
ffmpeg -i input1.mp4 -i input2.mp4 -filter_complex " nullsrc=size=1920x1440 [base]; [0:v] setpts=PTS-STARTPTS, scale=1920x1440 [frame0];[1:v] setpts=PTS-STARTPTS, scale=640x480 [frame1]; [base] [frame0] overlay=shortest=1:x=1920:y=1440 [tmp1]; [tmp1] [frame1] overlay=shortest=1:x=640:y=480 output.mp4
But cannot run with python Popen script like below.
cmd = "ffmpeg -i input1.mp4 -i input2.mp4 -filter_complex " nullsrc=size=1920x1440 [base]; [0:v] setpts=PTS-STARTPTS, scale=1920x1440 [frame0];[1:v] setpts=PTS-STARTPTS, scale=640x480 [frame1]; [base] [frame0] overlay=shortest=1:x=1920:y=1440 [tmp1]; [tmp1] [frame1] overlay=shortest=1:x=640:y=480 output.mp4"
p = subprocess.Popen(cmd.strip().split(" "))
This has problem at split(" ") works unexpected subdivision.
So I prepare the split command directly like
split_cmd = [ffmpeg_exe,'-i','input1.mp4','-i','input2.mp4','-filter_complex','\" nullsrc=size=1920x1440 [base]; [0:v] setpts=PTS-STARTPTS, scale=1920x1440 [frame0];[1:v] setpts=PTS-STARTPTS, scale=640x480 [frame1]; [base] [frame0] overlay=shortest=1:x=1920:y=1440 [tmp1]; [tmp1] [frame1] overlay=shortest=1:x=640:y=480\"','output.mp4'];
p = subprocess.Popen(split_cmd)
Even in this case, system return
[AVFilterGraph # 0x7fb1c9f000c0] No such filter: '" nullsrc'
Error configuring filters.
this could be system confused the -filter_complex option as ffmpeg option.
Anyone can help this, please.
You are actually passing in the opening and closing double-quote ", since there is no shell in-between that uses them. Use just
' nullsrc=size=1920x1440 [base]; [0:v] setpts=PTS-STARTPTS, scale=1920x1440 [frame0];[1:v] setpts=PTS-STARTPTS, scale=640x480 [frame1]; [base] [frame0] overlay=shortest=1:x=1920:y=1440 [tmp1]; [tmp1] [frame1] overlay=shortest=1:x=640:y=480'
instead, i.e. without the opening and closing \".
split_cmd = [ffmpeg_exe,'-i','input1.mp4','-i','input2.mp4','-filter_complex',' nullsrc=size=1920x1440 [base]; [0:v] setpts=PTS-STARTPTS, scale=1920x1440 [frame0];[1:v] setpts=PTS-STARTPTS, scale=640x480 [frame1]; [base] [frame0] overlay=shortest=1:x=1920:y=1440 [tmp1]; [tmp1] [frame1] overlay=shortest=1:x=640:y=480','output.mp4']
You can also try using shlex.split, as described in the subprocess documentation.
Related
GeoJson object (layer) not being added to Folium map
Have looked at previous similar questions and implemented suggestions but I am still having trouble getting a layer to appear on a Folium map. Basically I have used Here's isoline API to create an array of GeoJson data objects (these are bascially areas reacjable within certain drivetimes). I can render these drivetimes on a Here map but would like to use Folium as I think it's easier to set layer colors etc. So the GeoJson objects (drivetimes) are stored in an array called "values". My code is as follows: # Create the map m = folium.Map(location=[latitude, longitude], zoom_start=13) # Add a marker to the map folium.Marker( [latitude, longitude] ).add_to(m) # Add the layer folium.GeoJson(data=value[0], name="geojson").add_to(m) folium.LayerControl().add_to(m) m An example of one of the drivetimes is as follows, any help would be appreciated: {"data": {"features": [{"geometry": {"coordinates": [[[-6.643639, 53.382568], [-6.635742, 53.382568], [-6.633682, 53.381882], [-6.632309, 53.380508], [-6.630249, 53.379822], [-6.622009, 53.379822], [-6.619949, 53.379135], [-6.618576, 53.377762], [-6.616516, 53.377075], [-6.611023, 53.377075], [-6.608963, 53.376389], [-6.60759, 53.375015], [-6.60553, 53.374329], [-6.600037, 53.374329], [-6.597977, 53.375015], [-6.596603, 53.376389], [-6.594543, 53.377075], [-6.586304, 53.377075], [-6.584244, 53.376389], [-6.58287, 53.372269], [-6.580811, 53.371582], [-6.564331, 53.371582], [-6.562271, 53.372269], [-6.560898, 53.373642], [-6.558838, 53.374329], [-6.556091, 53.374329], [-6.554031, 53.373642], [-6.552658, 53.372269], [-6.550598, 53.371582], [-6.548538, 53.372269], [-6.547852, 53.374329], [-6.547852, 53.377075], [-6.547165, 53.379135], [-6.544418, 53.381882], [-6.542358, 53.382568], [-6.540298, 53.381882], [-6.538925, 53.380508], [-6.537552, 53.380508], [-6.536179, 53.381882], [-6.534119, 53.382568], [-6.525879, 53.382568], [-6.523819, 53.381882], [-6.522446, 53.377762], [-6.520386, 53.377075], [-6.514893, 53.377075], [-6.512833, 53.376389], [-6.511459, 53.375015], [-6.509399, 53.374329], [-6.50116, 53.374329], [-6.4991, 53.373642], [-6.497726, 53.372269], [-6.495667, 53.371582], [-6.493607, 53.372269], [-6.493607, 53.373642], [-6.497726, 53.377762], [-6.498413, 53.379822], [-6.497726, 53.381882], [-6.49498, 53.384628], [-6.49292, 53.385315], [-6.490173, 53.385315], [-6.488113, 53.384628], [-6.48674, 53.383255], [-6.48468, 53.382568], [-6.479187, 53.382568], [-6.477127, 53.381882], [-6.475754, 53.380508], [-6.471634, 53.379135], [-6.470261, 53.377762], [-6.468201, 53.377075], [-6.465454, 53.377075], [-6.463394, 53.377762], [-6.463394, 53.379135], [-6.466141, 53.381882], [-6.470261, 53.383255], [-6.473007, 53.386002], [-6.473694, 53.388062], [-6.473694, 53.396301], [-6.47438, 53.398361], [-6.4785, 53.399734], [-6.479187, 53.401794], [-6.479187, 53.404541], [-6.479874, 53.406601], [-6.481934, 53.407288], [-6.483994, 53.406601], [-6.485367, 53.402481], [-6.487427, 53.401794], [-6.495667, 53.401794], [-6.497726, 53.402481], [-6.4991, 53.406601], [-6.500473, 53.406601], [-6.501846, 53.405228], [-6.503906, 53.404541], [-6.514893, 53.404541], [-6.516953, 53.405228], [-6.517639, 53.407288], [-6.517639, 53.410034], [-6.516953, 53.412094], [-6.514893, 53.412781], [-6.512146, 53.412781], [-6.510086, 53.413467], [-6.508713, 53.414841], [-6.504593, 53.416214], [-6.50322, 53.417587], [-6.4991, 53.418961], [-6.498413, 53.421021], [-6.4991, 53.42308], [-6.50116, 53.423767], [-6.517639, 53.423767], [-6.519699, 53.424454], [-6.521072, 53.428574], [-6.525192, 53.429947], [-6.526566, 53.43132], [-6.530685, 53.432693], [-6.533432, 53.43544], [-6.534119, 53.4375], [-6.533432, 53.43956], [-6.530685, 53.442307], [-6.528625, 53.442993], [-6.525879, 53.442993], [-6.523819, 53.442307], [-6.522446, 53.440933], [-6.518326, 53.43956], [-6.516953, 53.438187], [-6.514893, 53.4375], [-6.512146, 53.4375], [-6.510086, 53.436813], [-6.508713, 53.43544], [-6.506653, 53.434753], [-6.50116, 53.434753], [-6.4991, 53.434067], [-6.497726, 53.432693], [-6.493607, 53.43132], [-6.492233, 53.429947], [-6.49086, 53.429947], [-6.489487, 53.43132], [-6.485367, 53.432693], [-6.48468, 53.434753], [-6.485367, 53.436813], [-6.489487, 53.438187], [-6.490173, 53.440247], [-6.489487, 53.442307], [-6.488113, 53.44368], [-6.488113, 53.445053], [-6.49086, 53.4478], [-6.49292, 53.448486], [-6.498413, 53.448486], [-6.500473, 53.4478], [-6.501846, 53.446426], [-6.503906, 53.44574], [-6.509399, 53.44574], [-6.511459, 53.446426], [-6.512146, 53.448486], [-6.512146, 53.451233], [-6.512833, 53.453293], [-6.514893, 53.453979], [-6.517639, 53.453979], [-6.519699, 53.454666], [-6.521072, 53.456039], [-6.525192, 53.457413], [-6.527939, 53.460159], [-6.528625, 53.462219], [-6.527939, 53.464279], [-6.525192, 53.467026], [-6.523132, 53.467712], [-6.521072, 53.467026], [-6.519699, 53.465652], [-6.517639, 53.464966], [-6.515579, 53.465652], [-6.515579, 53.467026], [-6.536179, 53.487625], [-6.536865, 53.489685], [-6.536865, 53.492432], [-6.536179, 53.494492], [-6.534119, 53.495178], [-6.528625, 53.495178], [-6.526566, 53.494492], [-6.511459, 53.479385], [-6.509399, 53.478699], [-6.506653, 53.478699], [-6.504593, 53.479385], [-6.50322, 53.480759], [-6.50116, 53.481445], [-6.4991, 53.480759], [-6.497726, 53.479385], [-6.493607, 53.478012], [-6.481247, 53.465652], [-6.479874, 53.465652], [-6.4785, 53.469772], [-6.477127, 53.471146], [-6.477127, 53.472519], [-6.4785, 53.473892], [-6.479187, 53.475952], [-6.4785, 53.478012], [-6.475754, 53.480759], [-6.473694, 53.481445], [-6.470947, 53.481445], [-6.468887, 53.480759], [-6.466141, 53.478012], [-6.465454, 53.475952], [-6.465454, 53.470459], [-6.464767, 53.468399], [-6.462021, 53.465652], [-6.457901, 53.464279], [-6.457214, 53.462219], [-6.457214, 53.459473], [-6.457901, 53.457413], [-6.459961, 53.456726], [-6.465454, 53.456726], [-6.467514, 53.456039], [-6.470261, 53.453293], [-6.470261, 53.45192], [-6.467514, 53.449173], [-6.463394, 53.4478], [-6.462021, 53.446426], [-6.459961, 53.44574], [-6.448975, 53.44574], [-6.446915, 53.446426], [-6.445541, 53.450546], [-6.443481, 53.451233], [-6.437988, 53.451233], [-6.435928, 53.450546], [-6.434555, 53.449173], [-6.430435, 53.4478], [-6.429749, 53.44574], [-6.430435, 53.44368], [-6.431808, 53.442307], [-6.431808, 53.440933], [-6.430435, 53.43956], [-6.429749, 53.4375], [-6.430435, 53.43544], [-6.434555, 53.434067], [-6.435928, 53.432693], [-6.437988, 53.432007], [-6.440735, 53.432007], [-6.442795, 53.432693], [-6.444168, 53.434067], [-6.445541, 53.434067], [-6.446915, 53.429947], [-6.451035, 53.428574], [-6.451721, 53.426514], [-6.451035, 53.424454], [-6.446915, 53.42308], [-6.445541, 53.421707], [-6.444168, 53.421707], [-6.437302, 53.428574], [-6.433182, 53.429947], [-6.431808, 53.43132], [-6.429749, 53.432007], [-6.424255, 53.432007], [-6.422195, 53.43132], [-6.418076, 53.4272], [-6.416016, 53.426514], [-6.405029, 53.426514], [-6.402969, 53.425827], [-6.401596, 53.424454], [-6.397476, 53.42308], [-6.396103, 53.421707], [-6.394043, 53.421021], [-6.391983, 53.421707], [-6.387863, 53.425827], [-6.383743, 53.4272], [-6.383057, 53.42926], [-6.383743, 53.43132], [-6.385803, 53.432007], [-6.391296, 53.432007], [-6.393356, 53.432693], [-6.39473, 53.434067], [-6.39679, 53.434753], [-6.399536, 53.434753], [-6.401596, 53.43544], [-6.402283, 53.4375], [-6.402283, 53.440247], [-6.401596, 53.442307], [-6.399536, 53.442993], [-6.383057, 53.442993], [-6.380997, 53.44368], [-6.376877, 53.4478], [-6.374817, 53.448486], [-6.369324, 53.448486], [-6.367264, 53.4478], [-6.365891, 53.446426], [-6.363831, 53.44574], [-6.361771, 53.446426], [-6.361084, 53.448486], [-6.361084, 53.451233], [-6.361771, 53.453293], [-6.363831, 53.453979], [-6.369324, 53.453979], [-6.371384, 53.454666], [-6.37207, 53.456726], [-6.37207, 53.459473], [-6.371384, 53.461533], [-6.37001, 53.462906], [-6.37001, 53.464279], [-6.371384, 53.465652], [-6.372757, 53.469772], [-6.383743, 53.480759], [-6.385803, 53.481445], [-6.38855, 53.481445], [-6.39061, 53.482132], [-6.391983, 53.483505], [-6.396103, 53.484879], [-6.397476, 53.486252], [-6.399536, 53.486938], [-6.410522, 53.486938], [-6.412582, 53.487625], [-6.413956, 53.488998], [-6.418076, 53.490372], [-6.423569, 53.495865], [-6.424255, 53.497925], [-6.424255, 53.500671], [-6.424942, 53.502731], [-6.426315, 53.504105], [-6.427002, 53.506165], [-6.427002, 53.514404], [-6.426315, 53.516464], [-6.424255, 53.517151], [-6.418762, 53.517151], [-6.416702, 53.516464], [-6.413956, 53.513718], [-6.413269, 53.511658], [-6.413269, 53.506165], [-6.412582, 53.504105], [-6.411209, 53.502731], [-6.409836, 53.498611], [-6.408463, 53.498611], [-6.407089, 53.499985], [-6.405029, 53.500671], [-6.399536, 53.500671], [-6.397476, 53.501358], [-6.39679, 53.503418], [-6.39679, 53.506165], [-6.396103, 53.508224], [-6.394043, 53.508911], [-6.38855, 53.508911], [-6.38649, 53.508224], [-6.379623, 53.501358], [-6.377563, 53.500671], [-6.37207, 53.500671], [-6.37001, 53.499985], [-6.368637, 53.498611], [-6.364517, 53.497238], [-6.363831, 53.495178], [-6.363831, 53.492432], [-6.364517, 53.490372], [-6.368637, 53.488998], [-6.368637, 53.487625], [-6.367264, 53.486252], [-6.365891, 53.482132], [-6.363144, 53.479385], [-6.361084, 53.478699], [-6.352844, 53.478699], [-6.350784, 53.478012], [-6.348038, 53.475266], [-6.348038, 53.473892], [-6.349411, 53.472519], [-6.350098, 53.470459], [-6.349411, 53.468399], [-6.348038, 53.468399], [-6.346664, 53.469772], [-6.344604, 53.470459], [-6.339111, 53.470459], [-6.337051, 53.469772], [-6.331558, 53.464279], [-6.330872, 53.462219], [-6.331558, 53.460159], [-6.335678, 53.458786], [-6.335678, 53.457413], [-6.332932, 53.454666], [-6.330872, 53.453979], [-6.319885, 53.453979], [-6.317825, 53.454666], [-6.317139, 53.456726], [-6.317139, 53.459473], [-6.316452, 53.461533], [-6.314392, 53.462219], [-6.303406, 53.462219], [-6.301346, 53.462906], [-6.299973, 53.464279], [-6.297913, 53.464966], [-6.295166, 53.464966], [-6.293106, 53.464279], [-6.290359, 53.461533], [-6.288986, 53.457413], [-6.286926, 53.456726], [-6.27594, 53.456726], [-6.27388, 53.456039], [-6.271133, 53.453293], [-6.26976, 53.449173], [-6.268387, 53.449173], [-6.267014, 53.450546], [-6.264954, 53.451233], [-6.256714, 53.451233], [-6.254654, 53.450546], [-6.253281, 53.448486], [-6.251907, 53.447113], [-6.250534, 53.4478], [-6.249161, 53.449173], [-6.247787, 53.453293], [-6.245728, 53.453979], [-6.234741, 53.453979], [-6.232681, 53.454666], [-6.228561, 53.458786], [-6.226501, 53.459473], [-6.221008, 53.459473], [-6.218948, 53.460159], [-6.218948, 53.461533], [-6.220322, 53.462906], [-6.221008, 53.464966], [-6.221008, 53.475952], [-6.220322, 53.478012], [-6.218262, 53.478699], [-6.215515, 53.478699], [-6.213455, 53.479385], [-6.212769, 53.481445], [-6.212769, 53.489685], [-6.212082, 53.491745], [-6.207962, 53.495865], [-6.207275, 53.497925], [-6.207275, 53.508911], [-6.206589, 53.510971], [-6.204529, 53.511658], [-6.199036, 53.511658], [-6.196976, 53.510971], [-6.194229, 53.508224], [-6.193542, 53.506165], [-6.193542, 53.495178], [-6.194229, 53.493118], [-6.195602, 53.491745], [-6.195602, 53.490372], [-6.194229, 53.488998], [-6.193542, 53.486938], [-6.193542, 53.484192], [-6.192856, 53.482132], [-6.191483, 53.480759], [-6.190796, 53.478699], [-6.191483, 53.476639], [-6.195602, 53.472519], [-6.196289, 53.470459], [-6.196289, 53.451233], [-6.195602, 53.449173], [-6.191483, 53.4478], [-6.188736, 53.445053], [-6.188049, 53.442993], [-6.188736, 53.440933], [-6.191483, 53.438187], [-6.193542, 53.4375], [-6.199036, 53.4375], [-6.201096, 53.436813], [-6.206589, 53.43132], [-6.206589, 53.429947], [-6.204529, 53.42926], [-6.193542, 53.42926], [-6.191483, 53.428574], [-6.190109, 53.4272], [-6.188049, 53.426514], [-6.185303, 53.426514], [-6.183243, 53.425827], [-6.182556, 53.423767], [-6.183243, 53.421707], [-6.183929, 53.420334], [-6.182556, 53.418961], [-6.17981, 53.418274], [-6.174316, 53.418274], [-6.172256, 53.417587], [-6.170883, 53.413467], [-6.166763, 53.412094], [-6.164017, 53.409348], [-6.162643, 53.405228], [-6.158524, 53.403854], [-6.155777, 53.401108], [-6.15509, 53.399048], [-6.155777, 53.396988], [-6.158524, 53.394241], [-6.162643, 53.392868], [-6.16333, 53.390808], [-6.16333, 53.388062], [-6.162643, 53.386002], [-6.160583, 53.385315], [-6.157837, 53.385315], [-6.155777, 53.386002], [-6.154404, 53.387375], [-6.152344, 53.388062], [-6.144104, 53.388062], [-6.142044, 53.387375], [-6.141357, 53.385315], [-6.141357, 53.382568], [-6.142044, 53.380508], [-6.144791, 53.377762], [-6.146851, 53.377075], [-6.149597, 53.377075], [-6.151657, 53.376389], [-6.152344, 53.374329], [-6.152344, 53.371582], [-6.15303, 53.369522], [-6.15509, 53.368835], [-6.157837, 53.368835], [-6.159897, 53.368149], [-6.16127, 53.366776], [-6.16539, 53.365402], [-6.168137, 53.362656], [-6.168137, 53.361282], [-6.166763, 53.359909], [-6.166077, 53.357849], [-6.166077, 53.355103], [-6.166763, 53.353043], [-6.168823, 53.352356], [-6.177063, 53.352356], [-6.179123, 53.353043], [-6.180496, 53.354416], [-6.182556, 53.355103], [-6.185303, 53.355103], [-6.187363, 53.354416], [-6.188049, 53.352356], [-6.188049, 53.346863], [-6.188736, 53.344803], [-6.190109, 53.34343], [-6.190796, 53.34137], [-6.190796, 53.338623], [-6.191483, 53.336563], [-6.193542, 53.335876], [-6.201782, 53.335876], [-6.203842, 53.33519], [-6.203842, 53.333817], [-6.202469, 53.332443], [-6.201096, 53.328323], [-6.199722, 53.32695], [-6.199036, 53.32489], [-6.199036, 53.319397], [-6.198349, 53.317337], [-6.195602, 53.31459], [-6.191483, 53.313217], [-6.190109, 53.311844], [-6.185989, 53.310471], [-6.185303, 53.308411], [-6.185303, 53.305664], [-6.185989, 53.303604], [-6.188049, 53.302917], [-6.190796, 53.302917], [-6.192856, 53.302231], [-6.194229, 53.298111], [-6.195602, 53.296738], [-6.196976, 53.292618], [-6.199036, 53.291931], [-6.207275, 53.291931], [-6.209335, 53.292618], [-6.210709, 53.293991], [-6.212769, 53.294678], [-6.221008, 53.294678], [-6.223068, 53.293991], [-6.224442, 53.289871], [-6.232681, 53.281631], [-6.236801, 53.280258], [-6.237488, 53.278198], [-6.236801, 53.276138], [-6.234741, 53.275452], [-6.210022, 53.275452], [-6.207962, 53.274765], [-6.206589, 53.273392], [-6.202469, 53.272018], [-6.201782, 53.269958], [-6.201782, 53.267212], [-6.202469, 53.265152], [-6.204529, 53.264465], [-6.210022, 53.264465], [-6.212082, 53.265152], [-6.213455, 53.266525], [-6.215515, 53.267212], [-6.237488, 53.267212], [-6.239548, 53.266525], [-6.240921, 53.265152], [-6.242981, 53.264465], [-6.245728, 53.264465], [-6.247787, 53.263779], [-6.249161, 53.262405], [-6.251221, 53.261719], [-6.25946, 53.261719], [-6.26152, 53.262405], [-6.262894, 53.263779], [-6.264954, 53.264465], [-6.270447, 53.264465], [-6.272507, 53.263779], [-6.27388, 53.262405], [-6.27594, 53.261719], [-6.289673, 53.261719], [-6.291733, 53.262405], [-6.293106, 53.263779], [-6.295166, 53.264465], [-6.300659, 53.264465], [-6.302719, 53.265152], [-6.304092, 53.266525], [-6.306152, 53.267212], [-6.314392, 53.267212], [-6.316452, 53.266525], [-6.316452, 53.265152], [-6.312332, 53.263779], [-6.311646, 53.261719], [-6.312332, 53.259659], [-6.315079, 53.256912], [-6.317139, 53.256226], [-6.322632, 53.256226], [-6.324692, 53.256912], [-6.328812, 53.261032], [-6.332932, 53.262405], [-6.334305, 53.263779], [-6.336365, 53.264465], [-6.347351, 53.264465], [-6.349411, 53.265152], [-6.350784, 53.266525], [-6.352158, 53.266525], [-6.353531, 53.265152], [-6.355591, 53.264465], [-6.363831, 53.264465], [-6.365891, 53.265152], [-6.366577, 53.267212], [-6.366577, 53.269958], [-6.367264, 53.272018], [-6.369324, 53.272705], [-6.377563, 53.272705], [-6.379623, 53.273392], [-6.380997, 53.277512], [-6.383057, 53.278198], [-6.385117, 53.277512], [-6.38649, 53.276138], [-6.38855, 53.275452], [-6.405029, 53.275452], [-6.407089, 53.276138], [-6.408463, 53.276825], [-6.409836, 53.275452], [-6.411209, 53.273392], [-6.413269, 53.272705], [-6.421509, 53.272705], [-6.423569, 53.273392], [-6.424942, 53.274765], [-6.426315, 53.274765], [-6.427689, 53.270645], [-6.429749, 53.269958], [-6.440735, 53.269958], [-6.442795, 53.269272], [-6.444168, 53.267899], [-6.446228, 53.267212], [-6.451721, 53.267212], [-6.453781, 53.267899], [-6.457901, 53.272018], [-6.459961, 53.273392], [-6.462708, 53.274765], [-6.465454, 53.275452], [-6.468201, 53.275452], [-6.470261, 53.274765], [-6.471634, 53.273392], [-6.475754, 53.272018], [-6.477127, 53.270645], [-6.4785, 53.270645], [-6.479874, 53.272018], [-6.481934, 53.272705], [-6.498413, 53.272705], [-6.500473, 53.272018], [-6.501846, 53.270645], [-6.503906, 53.269958], [-6.506653, 53.269958], [-6.508713, 53.269272], [-6.510086, 53.267899], [-6.514206, 53.266525], [-6.515579, 53.265152], [-6.517639, 53.264465], [-6.523132, 53.264465], [-6.525192, 53.265152], [-6.525879, 53.267212], [-6.525879, 53.269958], [-6.525192, 53.272018], [-6.521072, 53.273392], [-6.519699, 53.274765], [-6.515579, 53.276138], [-6.514206, 53.277512], [-6.510086, 53.278885], [-6.508713, 53.280258], [-6.506653, 53.280945], [-6.498413, 53.280945], [-6.496353, 53.281631], [-6.49498, 53.283005], [-6.49292, 53.283691], [-6.487427, 53.283691], [-6.485367, 53.284378], [-6.483994, 53.285751], [-6.481934, 53.286438], [-6.47644, 53.286438], [-6.47438, 53.287125], [-6.473694, 53.289185], [-6.473694, 53.291931], [-6.473007, 53.293991], [-6.470947, 53.294678], [-6.465454, 53.294678], [-6.463394, 53.293991], [-6.462021, 53.292618], [-6.459961, 53.291931], [-6.454468, 53.291931], [-6.452408, 53.292618], [-6.451035, 53.293991], [-6.446915, 53.295364], [-6.445541, 53.299484], [-6.444168, 53.300858], [-6.444168, 53.302231], [-6.446228, 53.302917], [-6.448975, 53.302917], [-6.451035, 53.303604], [-6.452408, 53.307724], [-6.454468, 53.308411], [-6.459961, 53.308411], [-6.462021, 53.307724], [-6.463394, 53.306351], [-6.465454, 53.305664], [-6.473694, 53.305664], [-6.475754, 53.306351], [-6.477127, 53.310471], [-6.479187, 53.311157], [-6.481247, 53.310471], [-6.48262, 53.309097], [-6.48468, 53.308411], [-6.490173, 53.308411], [-6.492233, 53.309097], [-6.49498, 53.311844], [-6.495667, 53.313904], [-6.49498, 53.315964], [-6.49086, 53.317337], [-6.48674, 53.321457], [-6.48468, 53.322144], [-6.473694, 53.322144], [-6.471634, 53.32283], [-6.467514, 53.32695], [-6.463394, 53.328323], [-6.463394, 53.329697], [-6.467514, 53.33107], [-6.468887, 53.332443], [-6.470947, 53.33313], [-6.47644, 53.33313], [-6.4785, 53.333817], [-6.479874, 53.33519], [-6.483994, 53.336563], [-6.485367, 53.337936], [-6.48674, 53.337936], [-6.49086, 53.333817], [-6.49498, 53.332443], [-6.495667, 53.330383], [-6.495667, 53.327637], [-6.496353, 53.325577], [-6.4991, 53.32283], [-6.50116, 53.322144], [-6.514893, 53.322144], [-6.516953, 53.321457], [-6.518326, 53.320084], [-6.520386, 53.319397], [-6.523132, 53.319397], [-6.525192, 53.320084], [-6.527939, 53.32283], [-6.528625, 53.32489], [-6.527939, 53.32695], [-6.523819, 53.33107], [-6.523132, 53.33313], [-6.523819, 53.33519], [-6.525879, 53.335876], [-6.527939, 53.33519], [-6.529312, 53.333817], [-6.531372, 53.33313], [-6.542358, 53.33313], [-6.544418, 53.333817], [-6.545105, 53.335876], [-6.545105, 53.338623], [-6.545792, 53.340683], [-6.548538, 53.34343], [-6.550598, 53.344116], [-6.553345, 53.344116], [-6.555405, 53.344803], [-6.556778, 53.348923], [-6.560898, 53.353043], [-6.562271, 53.357162], [-6.564331, 53.357849], [-6.569824, 53.357849], [-6.571884, 53.358536], [-6.573257, 53.359909], [-6.575317, 53.360596], [-6.583557, 53.360596], [-6.585617, 53.359909], [-6.58699, 53.355789], [-6.59111, 53.351669], [-6.592484, 53.347549], [-6.59523, 53.344803], [-6.59729, 53.344116], [-6.600037, 53.344116], [-6.602097, 53.344803], [-6.604843, 53.347549], [-6.60553, 53.349609], [-6.60553, 53.352356], [-6.604843, 53.354416], [-6.600723, 53.358536], [-6.600037, 53.360596], [-6.600723, 53.362656], [-6.60347, 53.365402], [-6.60553, 53.366089], [-6.61377, 53.366089], [-6.615829, 53.366776], [-6.617203, 53.368149], [-6.619263, 53.368835], [-6.624756, 53.368835], [-6.626816, 53.369522], [-6.628189, 53.370895], [-6.630249, 53.371582], [-6.635742, 53.371582], [-6.637802, 53.372269], [-6.639175, 53.373642], [-6.641235, 53.374329], [-6.643982, 53.374329], [-6.646042, 53.375015], [-6.646729, 53.377075], [-6.646729, 53.379822], [-6.646385, 53.381882], [-6.645355, 53.382568], [-6.643639, 53.382568]]], "type": "Polygon"}, "properties": {"range": {"type": "time", "value": 1500}}, "type": "Feature"}], "type": "FeatureCollection"}, "type": "GeoJSON"}
To specify polygons without changing the data in your question, you need to specify geometry values in dictionary form. import folium latitude = 53.372 longitude = -6.49 # Create the map m = folium.Map(location=[latitude, longitude], zoom_start=10) # Add a marker to the map folium.Marker( [latitude, longitude] ).add_to(m) # Add the layer folium.GeoJson(data=value['data']['features'][0]['geometry'], name="geojson").add_to(m) folium.LayerControl().add_to(m) m
Creating video from list of images using ffmpeg
Summary From given code, second function taking UI values and passing to first one. from log, first two lines shows command list and command string. What's irritating me is, if I copied second line from log (dos cmd) and paste it in a cmd prompt, it works successfully and produces video correctly. but running it via gui subprocess, errors out. Task Description Directory contains thousand of images each after few minutes of a construction project. I need to create a video from data (images) per day basis. I have successfully extracted list of images, etc, and have data. utils.py: def framesToVideoViaCount(srcFile,outputFile,start,count,rate): # ffmpeg -start_number 1 -i test_%04d.png -vframes 100 -vcodec mpeg4 test.mp4 # "C:\\Users\\lalat\\Desktop\\03008427633\\output_%04d.png" # ffmpeg -start_number 50 -i "C:\\Users\\lalat\\Desktop\\03008427633\\output_%04d.png" -vframes 200 -vcodec mpeg4 "C:\\Users\\lalat\\Desktop\\day1.mp4" command = [ 'ffmpeg', # '-loglevel', 'fatal', '-r %d' %(rate), '-start_number %d' %(start) , '-i "%s"' %(srcFile), '-vframes %d' %(count), '-vcodec mpeg4' ] # command.append ('fps=%d' % (fps)) command.append (outputFile) print ("COMMAND: " , command) print ("DOS CMD: ", ' '.join(command)) ffmpeg = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE ) out, err = ffmpeg.communicate() if(err) : print('error',err); return None; print (out) Actual GUI function call in tk gui app: def approachByCount(self): tdate = datetime(year=int(self.fyear.get()), month=int(self.fmonth.get()), day=int(self.fdate.get())) # now = datetime.datetime.now() outputName = str(tdate.strftime("%Y%m%d")) + ".txt" outputVid = str(tdate.strftime("%Y%m%d")) + ".mp4" files = os.listdir(self.userDir) selected = [] for afile in files: afile = os.path.join(self.userDir,afile) sdate = datetime.fromtimestamp(os.path.getmtime(afile)) if sdate.date() == tdate.date(): selected.append(afile) # get first file name srcfilename = os.path.basename(selected[0]) srcfilename,ext = srcfilename.split(".") filename,pad = srcfilename.split("_") srcfilename = os.path.join(self.userDir,filename+"_%0"+str(len(pad))+"d."+ext) start = int(pad) count = len(selected) utils.framesToVideoViaCount(srcfilename,outputVid,start,count,25) ERROR log COMMAND: ['ffmpeg', '-r 25', '-start_number 1', '-i "C:\\Users\\lalat\\Desktop\\03008427633\\output_%04d.png"', '-vframes 382', '-vcodec mpeg4', '20190810.mp4'] DOC CMD: ffmpeg -r 25 -start_number 1 -i "C:\Users\lalat\Desktop\03008427633\output_%04d.png" -vframes 382 -vcodec mpeg4 20190810.mp4 error b"ffmpeg version 4.1 Copyright (c) 2000-2018 the FFmpeg developers\r\n built with gcc 8.2.1 (GCC) 20181017\r\n configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth\r\n libavutil 56. 22.100 / 56. 22.100\r\n libavcodec 58. 35.100 / 58. 35.100\r\n libavformat 58. 20.100 / 58. 20.100\r\n libavdevice 58. 5.100 / 58. 5.100\r\n libavfilter 7. 40.101 / 7. 40.101\r\n libswscale 5. 3.100 / 5. 3.100\r\n libswresample 3. 3.100 / 3. 3.100\r\n libpostproc 55. 3.100 / 55. 3.100\r\n Unrecognized option 'r 25'.\r\n Error splitting the argument list: Option not found\r\n"
subprocess call of ffmpeg returns negative value
Here is the original command line call that works at shell: ffmpeg -i /Users/abc/Desktop/Test/Full_Mov.mov -vf "drawtext=fontfile=/System/Library/Fonts/Keyboard.ttf: text='SCENE BLAH BLAH - %{frame_num}': start_number=1: x=(w-tw)/2: y=h-(2*lh): fontcolor=white: fontsize=20: box=1: boxcolor=black: boxborderw=5","format=yuv420p" -f segment -segment_frames 123 -reset_timestamps 1 -c:a copy -map 0 "/Users/abc/Desktop/Test/%03d_test40.mov" Getting a negative value when trying to run ffmpeg via subprocess, causing it to fail and not process. I've even tried dumbing down the call to just run ffmpeg by itself with no arguments, and it still returns a negative value. import subprocess ffmpeg = "/usr/local/bin/ffmpeg" source = "/Users/abc/Desktop/Test/Full_Mov.mov" destination = "/Users/abc/Desktop/Test/%03d_test40.mov" cmd = "%s -i %s -vf \"drawtext=fontfile=/System/Library/Fonts/Keyboard.ttf: text='SCENE BLAH BLAH - %%{frame_num}': start_number=1: x=(w-tw)/2: y=h-(2*lh): fontcolor=black: fontsize=20: box=1: boxcolor=white: boxborderw=5\",\"format=yuv420p\" -f segment -segment_frames 123 -reset_timestamps 1 -c:a copy -map 0 \"%s\"" % (ffmpeg, source, destination) log("ffmpeg cmd: %s" % cmd) log(subprocess.check_output(cmd, shell=True)) When running full command: CalledProcessError: Command '/usr/local/bin/ffmpeg -i /Users/abc/Desktop/Full_Mov.mov -vf "drawtext=fontfile=/System/Library/Fonts/Keyboard.ttf: text='SCENE BLAH BLAH - %{frame_num}': start_number=1: x=(w-tw)/2: y=h-(2*lh): fontcolor=black: fontsize=20: box=1: boxcolor=white: boxborderw=5","format=yuv420p" -f segment -segment_frames 318 -reset_timestamps 1 -c:a copy -map 0 "/Users/abc/Desktop/Test/%03d.tmp.mov"' returned non-zero exit status -8 raise CalledProcessError(retcode, cmd, output=output) When running just ffmpeg no arguments: CalledProcessError: Command '/usr/local/bin/ffmpeg' returned non-zero exit status 1 raise CalledProcessError(retcode, cmd, output=output) Latest output with suggested changes below: subprocess.CalledProcessError: Command '['/usr/local/bin/ffmpeg', '-i', '/Users/szaharak/Desktop/Flix_Test/ep888_sq66_main_mov_2019_05_20_14_15.mov', '-vf', 'drawtext=fontfile=/System/Library/Fonts/Keyboard.ttf:', 'text=SCENE BLAH BLAH - %%{frame_num}:', 'start_number=1:', 'x=(w-tw)/2:', 'y=h-(2*lh):', 'fontcolor=black:', 'fontsize=20:', 'box=1:', 'boxcolor=white:', 'boxborderw=5', 'format=yuv420p', '-f', 'segment', '-segment_frames', '123', '-reset_timestamps', '1', '-c:a', 'copy', '-map', '0', '/Users/szaharak/Desktop/Flix_Test/%03d_test40.mov']' returned non-zero exit status 1 [NULL # 0x7f88c3012200] Unable to find a suitable output format for 'text=SCENE BLAH BLAH - %%{frame_num}:' text=SCENE BLAH BLAH - %%{frame_num}:: Invalid argument And here is latest... >>> rc = subprocess.check_call(cmd) ffmpeg version N-93891-ge1839283bc-tessus https://evermeet.cx/ffmpeg/ Copyright (c) 2000-2019 the FFmpeg developers built with Apple LLVM version 10.0.1 (clang-1001.0.46.4) configuration: --cc=/usr/bin/clang --prefix=/opt/ffmpeg --extra-version=tessus --enable-avisynth --enable-fontconfig --enable-gpl --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libfreetype --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libmysofa --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-version3 --pkg-config-flags=--static --disable-ffplay libavutil 56. 28.100 / 56. 28.100 libavcodec 58. 52.102 / 58. 52.102 libavformat 58. 27.103 / 58. 27.103 libavdevice 58. 7.100 / 58. 7.100 libavfilter 7. 53.101 / 7. 53.101 libswscale 5. 4.101 / 5. 4.101 libswresample 3. 4.100 / 3. 4.100 libpostproc 55. 4.100 / 55. 4.100 Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/szaharak/Desktop/Flix_Test/ep888_sq66_main_mov_2019_05_20_14_15.mov': Metadata: major_brand : qt minor_version : 537199360 compatible_brands: qt creation_time : 2019-05-14T17:58:24.000000Z com.apple.quicktime.player.movie.audio.gain: 1.000000 com.apple.quicktime.player.movie.audio.treble: 0.000000 com.apple.quicktime.player.movie.audio.bass: 0.000000 com.apple.quicktime.player.movie.audio.balance: 0.000000 com.apple.quicktime.player.movie.audio.pitchshift: 0.000000 com.apple.quicktime.player.movie.audio.mute: com.apple.quicktime.player.movie.visual.brightness: 0.000000 com.apple.quicktime.player.movie.visual.color: 1.000000 com.apple.quicktime.player.movie.visual.tint: 0.000000 com.apple.quicktime.player.movie.visual.contrast: 1.000000 com.apple.quicktime.player.version: 7.6.6 (7.6.6) com.apple.quicktime.version: 7.7.3 (2943.14) 0x7738000 (Mac OS X, 10.11.6, 15G22010) Duration: 00:01:12.67, start: 0.000000, bitrate: 23379 kb/s Stream #0:0(eng): Video: mjpeg (Baseline) (jpeg / 0x6765706A), yuvj422p(pc, bt470bg/unknown/unknown), 1280x720 [SAR 72:72 DAR 16:9], 21838 kb/s, 24 fps, 24 tbr, 24k tbn, 24k tbc (default) Metadata: creation_time : 2019-05-14T17:58:24.000000Z handler_name : Apple Video Media Handler encoder : Photo - JPEG Stream #0:1(eng): Audio: pcm_s16be (twos / 0x736F7774), 48000 Hz, stereo, s16, 1536 kb/s (default) Metadata: creation_time : 2019-05-14T17:58:24.000000Z handler_name : Apple Sound Media Handler [NULL # 0x7f8ddc8ce200] Unable to find a suitable output format for 'text=SCENE BLAH BLAH - %{frame_num}: start_number=1: x=(w-tw)/2: y=h-(2*lh): fontcolor=black: fontsize=20: box=1: boxcolor=white: boxborderw=5: format=yuv420p' text=SCENE BLAH BLAH - %{frame_num}: start_number=1: x=(w-tw)/2: y=h-(2*lh): fontcolor=black: fontsize=20: box=1: boxcolor=white: boxborderw=5: format=yuv420p: Invalid argument Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/Cellar/python#2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 190, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['/usr/local/bin/ffmpeg', '-i', '/Users/szaharak/Desktop/Flix_Test/ep888_sq66_main_mov_2019_05_20_14_15.mov', '-vf', 'drawtext=fontfile=/System/Library/Fonts/Keyboard.ttf:', 'text=SCENE BLAH BLAH - %{frame_num}: start_number=1: x=(w-tw)/2: y=h-(2*lh): fontcolor=black: fontsize=20: box=1: boxcolor=white: boxborderw=5: format=yuv420p', '-f', 'segment', '-segment_frames', '123', '-reset_timestamps', '1', '-c:a', 'copy', '-map', '0', '/Users/szaharak/Desktop/Flix_Test/%03d_test40.mov']' returned non-zero exit status 1
ffmpeg command line is tricky as there are a lot of special arguments and mistakes are sometimes difficult to understand, the program sometimes interprets/discards wrong arguments, resulting in cryptic error messages. Another difficulty is using check_output. You don't really need to do this, and if the program fails you won't have any output at all. I would try the following (that I could not test), which is to: drop shell=True pass a list of arguments instead of composing the string yourself. It allows to forget about the quoting/escaping. You don't need to format the command, source & destination since they're standalone arguments: Don't log, just print with check_call. As a bonus, you'll get the output in real-time, instead of in the end... note that I have removed the single quotes in "text=SCENE BLAH BLAH - %{frame_num}:, too and unescaped the % char. like this: cmd = [ffmpeg,"-i",source,"-vf", "drawtext=fontfile=/System/Library/Fonts/Keyboard.ttf:", "text=SCENE BLAH BLAH - %{frame_num}: start_number=1: x=(w-tw)/2: y=h-(2*lh): fontcolor=black: fontsize=20: box=1: boxcolor=white: boxborderw=5: format=yuv420p", "-f","segment","-segment_frames","123", "-reset_timestamps","1","-c:a","copy","-map","0",destination] log("ffmpeg cmd: %s" % cmd) rc = subprocess.check_call(cmd) if rc: raise Exception("ffmpeg failed") for the case of ffmpeg, you could migrate to ffmpeg python module. Could save some argument parsing trouble too.
using ffmepg in python, shell and stdout
I want to run the following line in python: ffmpeg -i test.avi -ss 0 -r 25 -vframes 100 ./out/image-%3d.jpg 2>&1 | grep output which should, if I directly run it in shell, output: >>Output #0, image2, to './out/image-%3d.jpg': However, when I do this in python: command = 'ffmpeg -i '+video_name+' -ss '+str(T) + ' -r '+str(25) + ' -vframes '+str(N)+' '+out_dir+'/image-%3d.jpg 2>&1 | grep output' argx = shlex.split(command) print argx proc = subprocess.Popen(argx,stdout=subprocess.PIPE,shell = True) (out,err) = proc.communicate() it outputs this: ['ffmpeg', '-i', 'test.avi', '-ss', '0', '-r', '25', '-vframes', '100', './out/image-%3d.jpg', '2>&1', '|', 'grep', 'output'] ffmpeg version 1.2.6-7:1.2.6-1~trusty1 Copyright (c) 2000-2014 the FFmpeg developers built on Apr 26 2014 18:52:58 with gcc 4.8 (Ubuntu 4.8.2-19ubuntu1) configuration: --arch=amd64 --disable-stripping --enable-avresample --enable-pthreads --enable-runtime-cpudetect --extra-version='7:1.2.6-1~trusty1' --libdir=/usr/lib/x86_64-linux-gnu --prefix=/usr --enable-bzlib --enable-libdc1394 --enable-libfreetype --enable-frei0r --enable-gnutls --enable-libgsm --enable-libmp3lame --enable-librtmp --enable-libopencv --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-vaapi --enable-vdpau --enable-libvorbis --enable-libvpx --enable-zlib --enable-gpl --enable-postproc --enable-libcdio --enable-x11grab --enable-libx264 --shlibdir=/usr/lib/x86_64-linux-gnu --enable-shared --disable-static libavutil 52. 18.100 / 52. 18.100 libavcodec 54. 92.100 / 54. 92.100 libavformat 54. 63.104 / 54. 63.104 libavdevice 53. 5.103 / 53. 5.103 libavfilter 3. 42.103 / 3. 42.103 libswscale 2. 2.100 / 2. 2.100 libswresample 0. 17.102 / 0. 17.102 libpostproc 52. 2.100 / 52. 2.100 Hyper fast Audio and Video encoder usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}... Use -h to get full help or, even better, run 'man ffmpeg' where apparently ffmpeg didn't get the proper arguments where is wrong? Thx
When shell=True, you should pass the command as a string, not as a list of arguments: command = 'ffmpeg -i '+video_name+' -ss '+str(T) + ' -r '+str(25) + ' -vframes '+str(N)+' '+out_dir+'/image-%3d.jpg 2>&1 | grep output' proc = subprocess.Popen(command, stdout=subprocess.PIPE, shell=True) out, err = proc.communicate() Note that using shell=True is a security risk if command depends on user input. If you wish to use shell=False, then you'll need to replace the shell pipeline with two subprocess.Popen calls, with proc1.stdout connected to proc2.stdin: import subprocess PIPE = subprocess.PIPE filename = out_dir+'/image-%3d.jpg' args = ['ffmpeg', '-i', video_name, '-ss', T, '-r', 25, '-vframes', N, filename] proc1 = subprocess.Popen(args, stdout=PIPE, stderr=PIPE, shell=False) proc2 = subprocess.Popen(['grep', 'output'), stdin=proc1.stdout, stdout=PIPE, stderr=PIPE) proc1.stdout.close() # Allow proc1 to receive a SIGPIPE if proc2 exits. out, err = proc2.communicate()
How do I parse lines from a log file?
I need to extract the values of following output: Oct 6 17:29:52 FW kernel: [ 5470.058450] ipTables: IN= OUT=eth0 SRC=192.168.1.116 DST=192.168.1.110 LEN=516 TOS=0x10 PREC=0x00 TTL=64 ID=4949 DF PROTO=TCP SPT=22 DPT=46216 WINDOW=446 RES=0x00 ACK PSH URGP=0 I'd need for example the value of PROTO stored in a value. tried shellscripting, my problem is that it works only if the log entry is in the same order everytime. so this doens't work: while read line do in_if=`echo $line | cut -d ' ' -f 10 | cut -d '=' -f 2`; out_if=`echo $line | cut -d ' ' -f 11 | cut -d '=' -f 2`; src_ip=`echo $line | cut -d ' ' -f 12 | cut -d '=' -f 2`; dst_ip=`echo $line | cut -d ' ' -f 13 | cut -d '=' -f 2`; pro=`echo $line | cut -d ' ' -f 20 | cut -d '=' -f 2`; echo "$in_if,$out_if,$src_ip,$dst_ip,$pro" >> output.csv; done < $tmp_file
Python does this conveniently. A general solution that gets all the KEY=value pairs is: import re import fileinput pair_re = re.compile('([^ ]+)=([^ ]+)') # Matches KEY=value pair for line in fileinput.input(): # The script accepts both data from stdin or a filename line = line.rstrip() # Removes final spaces and newlines data = dict(pair_re.findall(line)) # Fetches all the KEY=value pairs and puts them in a dictionary # Example of usage: print "PROTO =", data['PROTO'], "SRC =", data['SRC'] # Easy access to any value This is arguably more legible, flexible and convenient than a shell script.
You can do this without touching Perl. You were on the right track, but with a regex you can search by name, not position. Also, you should put quotes around $line so you don't get burned by any pipes or semicolons hanging around. pro=`echo "$line" | grep -o 'PROTO=\w+\+' | cut -d '=' -f 2`; Of course, if you did want to use Perl, you could make a much slicker solution: #!/usr/bin/perl while(<>) { /IN=(\S*) .*OUT=(\S*) .*SRC=(\S*) .*DST=(\S*) .*PROTO=(\S*)/ and print "$1,$2,$3,$4,$5\n"; } Then call: ./thatScript.pl logFile.txt >>output.csv
you don't even need to cut: grep -Po "(?<=PROTO=)\w+" yourFile OR sed -r 's/.*PROTO=(\w+).*/\1/' yourFile OR awk -F'PROTO=' '{split($2,a," ");print a[1]}' yourfile test: kent$ echo "Oct 6 17:29:52 FW kernel: [ 5470.058450] ipTables: IN= OUT=eth0 SRC=192.168.1.116 DST=192.168.1.110 LEN=516 TOS=0x10 PREC=0x00 TTL=64 ID=4949 DF PROTO=TCP SPT=22 DPT=46216 WINDOW=446 RES=0x00 ACK PSH URGP=0"|grep -Po "(?<=PROTO=)\w+" TCP kent$ echo "Oct 6 17:29:52 FW kernel: [ 5470.058450] ipTables: IN= OUT=eth0 SRC=192.168.1.116 DST=192.168.1.110 LEN=516 TOS=0x10 PREC=0x00 TTL=64 ID=4949 DF PROTO=TCP SPT=22 DPT=46216 WINDOW=446 RES=0x00 ACK PSH URGP=0"|sed -r 's/.*PROTO=(\w+).*/\1/' TCP kent$ echo "Oct 6 17:29:52 FW kernel: [ 5470.058450] ipTables: IN= OUT=eth0 SRC=192.168.1.116 DST=192.168.1.110 LEN=516 TOS=0x10 PREC=0x00 TTL=64 ID=4949 DF PROTO=TCP SPT=22 DPT=46216 WINDOW=446 RES=0x00 ACK PSH URGP=0"|awk -F'PROTO=' '{split($2,a," ");print a[1]}' TCP
A straightforward Perl solution might be the most readable one: #!/usr/bin/env perl use strict; use warnings; my $s = q{Oct 6 17:29:52 FW kernel: [ 5470.058450] ipTables: IN= OUT=eth0 SRC=192.168.1.116 DST=192.168.1.110 LEN=516 TOS=0x10 PREC=0x00 TTL=64 ID=4949 DF PROTO=TCP SPT=22 DPT=46216 WINDOW=446 RES=0x00 ACK PSH URGP=0}; while ($s =~ /(?<k> [A-Z]+) = (?<v> \S*)/xg) { print "'$+{k}' = '$+{v}'\n"; } C:\Temp> z 'IN' = '' 'OUT' = 'eth0' 'SRC' = '192.168.1.116' 'DST' = '192.168.1.110' 'LEN' = '516' 'TOS' = '0x10' 'PREC' = '0x00' 'TTL' = '64' 'ID' = '4949' 'PROTO' = 'TCP' 'SPT' = '22' 'DPT' = '46216' 'WINDOW' = '446' 'RES' = '0x00' 'URGP' = '0' You can also assign the information in the log line to a hash: my %entry = ($s =~ /(?<k> [A-Z]+) = (?<v> \S*)/xg);
In perl this should do it #consider the $a variable has the log file my $a = <<log file>>; my $desired_answer; #regex if ($a =~ m/PROTO=(.*?) /ig) { $desired_answer=$1; }
thanks for all the responses! i chose the way of shellscripting using egrep and regex... in_if=`echo "$line" | egrep -Eo 'IN=eth[0-9]*\b' | cut -d '=' -f 2`; out_if=`echo "$line" | egrep -Eo 'OUT=eth[0-9]*\b' | cut -d '=' -f 2`; src_ip=`echo "$line" | egrep -Eo 'SRC=[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | cut -d '=' -f 2`; dst_ip=`echo "$line" | egrep -Eo 'DST=[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | cut -d '=' -f 2`; pro=`echo "$line" | grep -o 'PROTO=[A-Z]*\b' | cut -d '=' -f 2`;