I worked on code published on GitHub https://github.com/jrterven/audio-visual-dataset/blob/master/extract_detailed_text_watson.py the code was design to use 5 words but I want to change it to 1 word so I try do that in the code but there is an error in the run as shown below
found 2 files
Processing video: health_news_2.mp4
video resolution: 608 x 1080
video framerate: 29.97002997002997
entry: <class 'dict'> {'link': 'build_Dataset', 'text': 'صورة', 'conf': 0.61, 'start': 2.07, 'end': 2.55, 'bounding_box': []}
s_sec, s_millisec: 2.0 69.99999999999984
entry: <class 'dict'> {'link': 'build_Dataset', 'text': 'مجهرية', 'conf': 0.97, 'start': 2.55, 'end': 3.24, 'bounding_box': []}
s_sec, s_millisec: 2.0 549.9999999999998
/Users/shaimaa/opt/anaconda3/lib/python3.8/site-packages/torch/nn/functional.py:780: UserWarning: Note that order of the arguments: ceil_mode and return_indices will changeto match the args list in nn.MaxPool2d in a future release.
warnings.warn("Note that order of the arguments: ceil_mode and return_indices will change"
entry: <class 'dict'> {'link': 'build_Dataset', 'text': 'مجهرية', 'conf': 0.97, 'start': 2.55, 'end': 3.24, 'bounding_box': [230, 126, 131, 171]}
ffmpeg version 4.4.1 Copyright (c) 2000-2021 the FFmpeg developers
built with Apple clang version 13.0.0 (clang-1300.0.29.3)
configuration: --prefix=/usr/local/Cellar/ffmpeg/4.4.1_3 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libdav1d --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-avresample --enable-videotoolbox
libavutil 56. 70.100 / 56. 70.100
libavcodec 58.134.100 / 58.134.100
libavformat 58. 76.100 / 58. 76.100
libavdevice 58. 13.100 / 58. 13.100
libavfilter 7.110.100 / 7.110.100
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 9.100 / 5. 9.100
libswresample 3. 9.100 / 3. 9.100
libpostproc 55. 9.100 / 55. 9.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'build_Dataset/news/health_news_2.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf58.45.100
Duration: 00:04:50.88, start: 0.000000, bitrate: 603 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 608x1080 [SAR 1:1 DAR 76:135], 468 kb/s, 29.97 fps, 29.97 tbr, 30k tbn, 59.94 tbc (default)
Metadata:
handler_name : ISO Media file produced by Google Inc.
vendor_id : [0][0][0][0]
Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 127 kb/s (default)
Metadata:
handler_name : ISO Media file produced by Google Inc.
vendor_id : [0][0][0][0]
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264))
Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[libx264 # 0x7f7fe8810800] using SAR=1/1
[libx264 # 0x7f7fe8810800] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 # 0x7f7fe8810800] profile High, level 1.1, 4:2:0, 8-bit
[libx264 # 0x7f7fe8810800] 264 - core 163 r3060 5db6aa6 - H.264/MPEG-4 AVC codec - Copyleft 2003-2021 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=5 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to '/Users/shaimaa/Downloads/LIP_Reading/Code/audio-visual-dataset-master/results_news/news/health_news_2/1.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf58.76.100
Stream #0:0(und): Video: h264 (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 130x170 [SAR 1:1 DAR 13:17], q=2-31, 29.97 fps, 30k tbn (default)
Metadata:
handler_name : ISO Media file produced by Google Inc.
vendor_id : [0][0][0][0]
encoder : Lavc58.134.100 libx264
Side data:
cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 127 kb/s (default)
Metadata:
handler_name : ISO Media file produced by Google Inc.
vendor_id : [0][0][0][0]
frame= 21 fps=0.0 q=-1.0 Lsize= 18kB time=00:00:00.67 bitrate= 222.3kbits/s dup=1 drop=0 speed=4.86x
video:5kB audio:11kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 13.600434%
[libx264 # 0x7f7fe8810800] frame I:1 Avg QP:23.12 size: 1388
[libx264 # 0x7f7fe8810800] frame P:5 Avg QP:25.05 size: 401
[libx264 # 0x7f7fe8810800] frame B:15 Avg QP:30.82 size: 85
[libx264 # 0x7f7fe8810800] consecutive B-frames: 4.8% 0.0% 0.0% 95.2%
[libx264 # 0x7f7fe8810800] mb I I16..4: 3.0% 80.8% 16.2%
[libx264 # 0x7f7fe8810800] mb P I16..4: 1.4% 2.6% 0.4% P16..4: 51.3% 19.2% 6.3% 0.0% 0.0% skip:18.8%
[libx264 # 0x7f7fe8810800] mb B I16..4: 0.0% 0.1% 0.1% B16..8: 31.9% 2.7% 0.5% direct: 0.7% skip:64.0% L0:31.9% L1:64.7% BI: 3.4%
[libx264 # 0x7f7fe8810800] 8x8 transform intra:76.4% inter:84.5%
[libx264 # 0x7f7fe8810800] coded y,uvDC,uvAC intra: 75.0% 89.4% 42.3% inter: 7.0% 6.6% 0.3%
[libx264 # 0x7f7fe8810800] i16 v,h,dc,p: 0% 40% 10% 50%
[libx264 # 0x7f7fe8810800] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 15% 32% 8% 5% 7% 8% 7% 10% 9%
[libx264 # 0x7f7fe8810800] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 21% 26% 9% 7% 7% 10% 4% 10% 6%
[libx264 # 0x7f7fe8810800] i8c dc,h,v,p: 38% 34% 20% 8%
[libx264 # 0x7f7fe8810800] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 # 0x7f7fe8810800] ref P L0: 68.6% 9.7% 15.4% 6.2%
[libx264 # 0x7f7fe8810800] ref B L0: 91.5% 5.4% 3.1%
[libx264 # 0x7f7fe8810800] ref B L1: 89.9% 10.1%
[libx264 # 0x7f7fe8810800] kb/s:53.25
entry: <class 'dict'> {'link': 'build_Dataset', 'text': 'تظهر', 'conf': 1.0, 'start': 3.24, 'end': 3.66, 'bounding_box': []}
s_sec, s_millisec: 3.0 240.00000000000023
entry: <class 'dict'> {}
Traceback (most recent call last):
File "extract_subvideos.py", line 424, in <module>
main(args)
File "extract_subvideos.py", line 140, in main
s_sec, s_millisec = divmod(float(entry['start']), 1)
KeyError: 'start'
In this part of the code I change in it an only the number of word
fa = FaceAlignment()
videos_directory = args.videos_dir
results_dir = args.results_dir
vids_name = args.category
vid_proc_name = args.log_file
dataset_annotation_file = args.ann_file
if args.save_videos == 'True':
save_videos = True
else:
save_videos = False
# Create video window
cv2.namedWindow('Vid')
# load or create list with processed files
processed_files = []
videos_processed_exists = os.path.isfile(
os.path.join(results_dir, vid_proc_name))
if not videos_processed_exists:
with open(os.path.join(results_dir, vid_proc_name), "w") as fp:
for pfiles in processed_files:
print(pfiles, file=fp)
else:
with open(os.path.join(results_dir, vid_proc_name)) as fp:
processed_files = fp.read().splitlines()
# Create annotation file the first time
annotation_exists = os.path.isfile(os.path.join(
results_dir, dataset_annotation_file))
if not annotation_exists:
try:
with open(os.path.join(
results_dir, dataset_annotation_file), 'w') as csvfile:
writer = csv.DictWriter(csvfile, fieldnames=csv_columns)
writer.writeheader()
except IOError:
print("Error creating annotaton file. I/O error")
# Get json files list names in videos directory
files_list = []
for ann_file in os.listdir(os.path.join(videos_directory, vids_name)):
if ann_file.endswith(".json"):
files_list.append(ann_file[0:-5])
files_list = natsorted(files_list)
num_files = len(files_list)
print('found', num_files, 'files')
# traverse all the files
stop_videos = False
for file in files_list:
if stop_videos:
break
# check if current video is not in alredy processed
if file in processed_files:
print(file, 'has already been processed. Skipping it.')
continue
num_output_video = 0
# Search for the video files in videos_directory
video_name = file + '.mp4'
print('Processing video:', video_name)
if save_videos:
# create output directory
output_dir = os.path.join(results_dir, vids_name, file)
if not os.path.isdir(output_dir):
os.mkdir(output_dir)
# Load watson results
with open(os.path.join(
videos_directory, vids_name, file + '.json')) as f:
stt_results = json.load(f)
# Extract all the words with confidence >90
words_data = extract_words_from_watson_results(stt_results, max_words=5)
# Start the video capture
cap = cv2.VideoCapture(os.path.join(
videos_directory, vids_name, video_name))
# Extract video metadata
height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
fps = cap.get(cv2.CAP_PROP_FPS)
print('video resolution:', width, ' x ', height)
print('video framerate:', fps)
frame_count = 0
fps_processing = 30.0 # fps holder
t = cv2.getTickCount() # initiate the tickCounter
count = 0
for entry in words_data:
# Extract speech to text data
print('entry:', type(entry), entry)
s_sec, s_millisec = divmod(float(entry['start']), 1)
e_sec, e_millisec = divmod(float(entry['end']), 1)
s_min = 0
e_min = 0
s_millisec = s_millisec * 1000
e_millisec = e_millisec * 1000
print('s_sec, s_millisec:', s_sec, s_millisec)
if s_sec >= 60:
s_min = math.floor(s_sec / 60.0)
s_sec = s_sec % 60
if e_sec >= 60:
e_min = math.floor(e_sec / 60.0)
e_sec = e_sec % 60
# Determine video frames involved in stt entry
min_frame = s_min*fps*60 + (s_sec*fps)
max_frame = e_min*fps*60 + (e_sec*fps)
# go to min_frame
cap.set(cv2.CAP_PROP_POS_FRAMES, min_frame)
frame_count = min_frame
# read frames from min_frame to max_frame
num_people = 0
valid_video = True
landmarks = []
angles = []
consecutive_frames_no_people = 0
while frame_count < max_frame:
if count == 0:
t = cv2.getTickCount()
# capture next frame
ret, frame = cap.read()
if not ret:
continue
frame_count += 1
# resize frame for faster processing
if frame.shape[0] <= 0 or frame.shape[1] <= 0:
continue
frame_small = cv2.resize(frame, (0, 0), fx=scale, fy=scale,
interpolation=cv2.INTER_LINEAR)
# detect faces and landmarjs
fa.update_features(frame_small)
landmarks.append(fa.get_mouth_features(scale=scale))
num_people = fa.get_num_people()
angles.append(fa.get_yaw())
# if it detects less than or more than 1 person
# go to next subtitle
if num_people != 1:
consecutive_frames_no_people += 1
if consecutive_frames_no_people >= max_bad_frames:
print(consecutive_frames_no_people,
' frames without 1 person. Skiping to next subtitle')
valid_video = False
break
# if only one person in the scene
if num_people == 1:
consecutive_frames_no_people = 0
fa.renderMouth(frame_small)
# Put fps at which we are processing camera feed on frame
cv2.putText(frame_small, "{0:.2f}-fps".format(fps_processing),
(50, height-50), cv2.FONT_HERSHEY_COMPLEX,
1, (0, 0, 255), 2)
# Display the image
cv2.imshow('Vid',frame_small)
# Read keyboard and exit if ESC was pressed
k = cv2.waitKey(1) & 0xFF
if k ==27:
exit()
elif k == ord('q'):
stop_videos = True
# increment frame counter
count = count + 1
# calculate fps at an interval of 100 frames
if (count == 30):
t = (cv2.getTickCount() - t)/cv2.getTickFrequency()
fps_processing = 30.0/t
count = 0
# if this was a valid video
if valid_video and len(landmarks) > 0:
num_output_video += 1
entry['mouth3d'] = landmarks
entry['angle'] = angles
if save_videos:
s_hr = 0
e_hr = 0
if s_min >= 60:
s_hr = math.floor(s_min / 60)
s_min = s_min % 60
if e_min >= 60:
e_hr = math.floor(e_min / 60)
e_min = e_min % 60
# cut and crop video
# ffmpeg -i input.mp4 -ss hh:mm:ss -filter:v crop=w:h:x:y -c:a copy -to hh:mm:ss output.mp4
ss = "{0:02d}:{1:02d}:{2:02d}.{3:03d}".format(
s_hr, s_min, int(s_sec), math.ceil(s_millisec))
es = "{0:02d}:{1:02d}:{2:02d}.{3:03d}".format(
e_hr, e_min, int(e_sec), math.ceil(e_millisec))
crop = "crop={0:1d}:{1:1d}:{2:1d}:{3:1d}".format(
bbw, bbh, bbx1, bby1)
out_name = os.path.join(output_dir, str(num_output_video))
subprocess.call(['ffmpeg', #'-hide_banner', '-loglevel', 'panic',
'-i', os.path.join(
videos_directory, vids_name, video_name),
'-ss', ss,
'-filter:v', crop, '-c:a', 'copy',
'-to', es, out_name +'.mp4'])
# save recognized speech
text_file = open(out_name +'.txt', "w")
text_file.write(entry['text'] + '\n')
text_file.write(str(entry['conf']))
text_file.close()
# append results to annotation file
append_annotation_file(os.path.join(
results_dir, dataset_annotation_file), words_data)
# save name of processed file
processed_files.append(file)
with open(os.path.join(results_dir, vid_proc_name), "w") as fp:
for p_file in processed_files:
print(p_file, file=fp)
# Release resources
cap.release()
cv2.destroyAllWindows()
def extract_text_conf_ts(s_idx, max_words, num_words, timestamps, conf, link):
text = ''
avg_conf = 0
start = timestamps[int(s_idx * max_words)][1]
end = timestamps[int(s_idx * max_words + num_words-1)][2]
for w_idx in range(num_words):
text = text + ' ' + timestamps[int(s_idx*max_words + w_idx)][0]
avg_conf += conf[int(s_idx*max_words + w_idx)][1]
avg_conf = round(avg_conf/num_words, 2)
if len(text.strip()) >= 4:
out_entry = {'link': link, 'text': text.strip(), 'conf': avg_conf,
'start':start, 'end': end, 'mouth3d': [],
'angle': [] }
else:
out_entry = {}
return out_entry
def extract_words_from_watson_results(stt_results, max_words=5):
data = stt_results['results']
link = stt_results['link']
link = link.rsplit('/', 1)[-1]
out_data = []
for sentence_idx, ann in enumerate(data):
data_ann = ann['alternatives'][0]
text = data_ann['transcript']
conf = data_ann['word_confidence']
timestamps = data_ann['timestamps']
num_words = len(timestamps)
num_splits = num_words//max_words
rest = num_words%max_words
if num_words < max_words:
maxx_words = num_words
else:
maxx_words = max_words
for s_idx in range(num_splits):
out_entry = extract_text_conf_ts(s_idx, maxx_words, maxx_words,
timestamps, conf, link)
out_data.append(out_entry)
if rest > 0:
out_entry = extract_text_conf_ts(num_splits, maxx_words, rest,
timestamps, conf, link)
if out_entry:
out_data.append(out_entry)
return out_data
def append_annotation_file(csv_file, data):
try:
with open(csv_file, 'a') as csvfile:
writer = csv.DictWriter(csvfile, fieldnames=csv_columns)
for entry in data:
writer.writerow(entry)
except IOError:
print("I/O error")
def remove_accents(input_str):
nfkd_form = unicodedata.normalize('NFKD', input_str)
only_ascii = nfkd_form.encode('ASCII', 'ignore')
return only_ascii
if __name__== "__main__":
# Parse input arguments
parser = argparse.ArgumentParser(description='Extract subvideos')
parser.add_argument('--dir', dest='videos_dir',
help='Directory with videos', type=str)
parser.add_argument('--cat', dest='category',
help='Video category', type=str)
parser.add_argument('--vids_log', dest='log_file',
help='Name of log file', type=str)
parser.add_argument('--results_dir', dest='results_dir',
help='Directory with results', type=str)
parser.add_argument('--ann_file', dest='ann_file',
help='Annotations file (csv)', type=str)
parser.add_argument('--save_videos', dest='save_videos',
help='Save videos', type=str, default='False')
args = parser.parse_args()
main(args)
I have this spider in scrapy
import scrapy
class ProductDetailSpider(scrapy.Spider):
name = 'productdetail'
allowed_domains = ['bisabeli.id']
start_urls = [
"https://bisabeli.id/index.php?route=product/product&product_id=7874"
]
def parse(self, response):
description = response.css('div.producttab div#tab-description div#collapse-description').get().strip()
yield {
'description': description,
}
And when I run this spider in console with
scrapy crawl productdetail
The result is:
{
"description": "<div id=\"co
llapse-description\" class=\"desc-collapse showup\">\n\t\t\t\t\t\t\t\t\t\t<p>Deskripsi Realme X2 Pro 12GB/256GB - Pre Order - Biru</p><p>barang repack</p><p>Apa itu Repack :</p><p>Kartu garansi Konsumen dpt lembar pembeli lalu penjual ambil bagian KG l
embar penjual, utk kami bantu daftarkan klaim warranty nya nanti 1 tahun</p><p><br></p><p><br></p><p>Garansi resmi 1 tahun</p><p><br></p><p>OS Android</p><p>OS ver Android 9.0 (Pie); ColorOS 6.1</p><p>SIM Nano SIM , Dual SIM , Dual Standby</p><p>CPU
Qualcomm SDM855 Snapdragon 855+ (7nm)</p><p>Octa-core</p><p>Kecepatan CPU 2.96 GHz (1x2.96 GHz Kryo 485 & 3x2.42 GHz Kryo 485 & 4x1.8 GHz Kryo 485)</p><p>Storage 64GB , 128GB , 256GB</p><p>RAM 8GB , 12GB , 6GB</p><p>External Storage No</p><
p>Battery 4000mAh</p><p>50W SuperVOOC Flash Charge</p><p>Ukuran Layar 6.5 inches</p><p>Resolusi FHD+ 2400 x 1080 pixels at 402 ppi</p><p>Super AMOLED, 90Hz display</p><p>Network Tipe 2G , 3G , 4G (LTE)</p><p>2G GSM: 850/900/1800/1900</p><p>3G WCDMA:
B1/B2/B4/B5/B6/B8/B19</p><p>4G (LTE) LTE FDD: B1/B2/B3/B4/B5/B7/B8/B12/B17/B18/B19/B20/B26/B28</p><p>TD-LTE: B34/B38/B39/B40/B41</p><p>Speed HSPA 42.2/11.5 Mbps, LTE-A</p><p>Kamera Utama 64MP + 13MP + 8MP + 2MP</p><p>Kamera Depan 16MP</p><p>Fitur W
i-Fi , Hotspot/Tethering , GPS , Bluetooth , Flash , Fingerprint Scanner , NFC , 3.5mm Headphone Jack , Quad Cameras</p><p>Ukuran Dimensi 161 x 75.7 x 8.7 mm</p>\n\t\t\t\t\t\t\t\t\t</div>"}
What I want to get is all tags inside div tag ('<div id="collapse-description" class="desc-collapse showup">'....'</div>'),
How I write the code for it?
Update
The result I want is like this:
{
"description": "\n\t\t\t\t\t\t\t\t\t\t<p>Deskripsi Realme X2 Pro 12GB/256GB - Pre Order - Biru</p><p>barang repack</p><p>Apa itu Repack :</p><p>Kartu garansi Konsumen dpt lembar pembeli lalu penjual ambil bagian KG l
embar penjual, utk kami bantu daftarkan klaim warranty nya nanti 1 tahun</p><p><br></p><p><br></p><p>Garansi resmi 1 tahun</p><p><br></p><p>OS Android</p><p>OS ver Android 9.0 (Pie); ColorOS 6.1</p><p>SIM Nano SIM , Dual SIM , Dual Standby</p><p>CPU
Qualcomm SDM855 Snapdragon 855+ (7nm)</p><p>Octa-core</p><p>Kecepatan CPU 2.96 GHz (1x2.96 GHz Kryo 485 & 3x2.42 GHz Kryo 485 & 4x1.8 GHz Kryo 485)</p><p>Storage 64GB , 128GB , 256GB</p><p>RAM 8GB , 12GB , 6GB</p><p>External Storage No</p><
p>Battery 4000mAh</p><p>50W SuperVOOC Flash Charge</p><p>Ukuran Layar 6.5 inches</p><p>Resolusi FHD+ 2400 x 1080 pixels at 402 ppi</p><p>Super AMOLED, 90Hz display</p><p>Network Tipe 2G , 3G , 4G (LTE)</p><p>2G GSM: 850/900/1800/1900</p><p>3G WCDMA:
B1/B2/B4/B5/B6/B8/B19</p><p>4G (LTE) LTE FDD: B1/B2/B3/B4/B5/B7/B8/B12/B17/B18/B19/B20/B26/B28</p><p>TD-LTE: B34/B38/B39/B40/B41</p><p>Speed HSPA 42.2/11.5 Mbps, LTE-A</p><p>Kamera Utama 64MP + 13MP + 8MP + 2MP</p><p>Kamera Depan 16MP</p><p>Fitur W
i-Fi , Hotspot/Tethering , GPS , Bluetooth , Flash , Fingerprint Scanner , NFC , 3.5mm Headphone Jack , Quad Cameras</p><p>Ukuran Dimensi 161 x 75.7 x 8.7 mm</p>\n\t\t\t\t\t\t\t\t\t"}
Update v2
Trying some code :
description = response.xpath("//div[#class='producttab']//div[#id='tab-description']//div[#id='collapse-description']").get()
Result:
{'description': '<div id="co
llapse-description" class="desc-collapse showup">\n\t\t\t\t\t\t\t\t\t\t<p>Deskripsi Realme X2 Pro 12GB/256GB - Pre Order - Biru</p><p>barang repack</p><p>Apa itu Repack :</p><p>Kartu garansi Konsumen dpt lembar pembeli lalu penjual ambil bagian KG l
embar penjual, utk kami bantu daftarkan klaim warranty nya nanti 1 tahun</p><p><br></p><p><br></p><p>Garansi resmi 1 tahun</p><p><br></p><p>OS Android</p><p>OS ver Android 9.0 (Pie); ColorOS 6.1</p><p>SIM Nano SIM , Dual SIM , Dual Standby</p><p>CPU
Qualcomm SDM855 Snapdragon 855+ (7nm)</p><p>Octa-core</p><p>Kecepatan CPU 2.96 GHz (1x2.96 GHz Kryo 485 & 3x2.42 GHz Kryo 485 & 4x1.8 GHz Kryo 485)</p><p>Storage 64GB , 128GB , 256GB</p><p>RAM 8GB , 12GB , 6GB</p><p>External Storage No</p><
p>Battery 4000mAh</p><p>50W SuperVOOC Flash Charge</p><p>Ukuran Layar 6.5 inches</p><p>Resolusi FHD+ 2400 x 1080 pixels at 402 ppi</p><p>Super AMOLED, 90Hz display</p><p>Network Tipe 2G , 3G , 4G (LTE)</p><p>2G GSM: 850/900/1800/1900</p><p>3G WCDMA:
B1/B2/B4/B5/B6/B8/B19</p><p>4G (LTE) LTE FDD: B1/B2/B3/B4/B5/B7/B8/B12/B17/B18/B19/B20/B26/B28</p><p>TD-LTE: B34/B38/B39/B40/B41</p><p>Speed HSPA 42.2/11.5 Mbps, LTE-A</p><p>Kamera Utama 64MP + 13MP + 8MP + 2MP</p><p>Kamera Depan 16MP</p><p>Fitur W
i-Fi , Hotspot/Tethering , GPS , Bluetooth , Flash , Fingerprint Scanner , NFC , 3.5mm Headphone Jack , Quad Cameras</p><p>Ukuran Dimensi 161 x 75.7 x 8.7 mm</p>\n\t\t\t\t\t\t\t\t\t</div>'}
description = response.xpath("//div[#class='producttab']//div[#id='tab-description']//div[#id='collapse-description']/text()").get()
result:
{'description': '\n\t\t\t\t\
t\t\t\t\t\t'}
description = response.xpath("//div[#class='producttab']//div[#id='tab-description']//div[#id='collapse-description']//text()").get()
result:
{'description': '\n\t\t\t\t\
t\t\t\t\t\t'}
Still... Not the result I desired...
OLD VERSION:
At this moment I found only .re(".+") to get all (more or less) as list
[opening_tag, item, item, ..., closing_tag]
If I skip first and last element and use "".join() then I can get inner HTML.
# without `get()`
description = response.xpath("//div[#class='producttab']//div[#id='tab-description']//div[#id='collapse-description']")
description = description.re('.+')
description = "".join(description[1:-1]).strip()
EDIT:
Old version works only because there are \n in HTML - and it splits HTML into list on \n
["<opening tag>text", "children", "<closing_tag>"]
When there is no \n then old version create one string
[ "<opening tag> text children <closing_tag>" ]
similar to get().
Besides every item has structure
<tag> text children_tags </tag> tail
and when \n is between text and children_tagsthen old version skipstext` from final result.
Code which works for me for different HTML is
import lxml.html
html = '<body>hello<span>good</span>world</body>'
tree = lxml.html.fromstring(html)
text = tree.text or ''
children = [lxml.html.tostring(x).decode() for x in item.getchildren()]
inner_html = text + "".join(children).strip()
print(inner_html)
Result:
hello<span>good</span>world
It seems scrapy already uses lxml in Selectors so lxml should be already installed.
Minimal working code:
import scrapy
import lxml.html
def get_inner_html(html):
tree = lxml.html.fromstring(html)
text = tree.text or '' # to skip `None`
children = [lxml.html.tostring(x).decode() for x in tree.getchildren()]
inner_html = text + "".join(children).strip()
return inner_html
class ProductDetailSpider(scrapy.Spider):
name = 'productdetail'
allowed_domains = ['bisabeli.id']
start_urls = [
"https://bisabeli.id/index.php?route=product/product&product_id=7874"
]
def parse(self, response):
print('--- example 1 ---')
html = response.xpath("//div[#class='producttab']//div[#id='tab-description']//div[#id='collapse-description']").get()
results = get_inner_html(html)
print(results.strip())
print('--- example 2 ---')
html = '<body>hello<span>good</span>world</body>'
results = get_inner_html(html)
print(results.strip())
This question already has answers here:
How do I split a list into equally-sized chunks?
(66 answers)
Closed 2 years ago.
I have string and I will do split for this string and then I will get 320 elements. I need to create 4 groups. Every group will be with 100 elements and the last group must be with 20 last elements. And the last step is that all groups must be string and not list. how can I do that?
I can do that if I know how many elements I have:
s_l = 'AA AAL AAOI ACLS ADT ADTX ADVM AEL AIG ALEC ALLY ALT AMCX ANGI APA APDN APLS APPS APRN AQUA ARMK ARNC ARVN ATNM ATOM ATRA ATSG AVCT AVT AX AXDX BCLI BE BEAM BJRI BKE BKU BLDR BLNK BMRA BOOT BXS BYD CAKE CALX CAPR CARG CARR CARV CATM CC CCL CELH CEQP CFG CHEF CHRS CIT CLDX CLR CLSK CNK CNST CODX COLB COOP CPE CRS CTVA CUK CVET CVI CVM CYTK DAL DBX DCP DDS DEI DISCA DISCK DK DNB DRNA DVAX DXC ECOM EIGR ELAN ELF ELY ENVA EQ EQT EXEL FE FHI FIXX FL FLWS FMCI FORM FOX FOXA FRTA FUN GBX GIII GM GNMK GOCO GPRE GRAF GRPN GRWG GTHX GWB HALO HCC HCSG HEAR HFC HGV HIBB HMSY HOG HOME HP HSC HTH HWC IMUX IMVT INO INOV INSG INSM INT IOVA IRDM ITCI JELD JWN KMT KODK KPTI KSS KTB KTOS KURA LAKE LB LCA LL LPI LPRO LSCC LYFT MAXR MBOT MCRB MCS MD MDP MGM MGNX MIC MLHR MOS MRSN MTOR MXL MYGN NCLH NCR NK NKTR NLS NMIH NOVA NTLA NTNX NUAN NVST NXTC ODP OFC OKE OMER OMF OMI ONEM OSPN OSUR OXY OZK PACW PD PDCE PDCO PEAK PGNY PLAY PLCE PLT PLUG PPBI PRPL PRTS PRVB PS PSNL PSTX PSXP PTGX PVAC RCUS REAL REZI RKT RMBL RPAY RRGB RRR RVLV RVP RXN SANM SAVE SBGI SC SCPL SEAS SEM SFIX SFM SGMS SGRY SHLL SHOO SHYF SIX SKX SLQT SMCI SNAP SNDX SNV SONO SPAQ SPCE SPR SPWH SPWR SRG SRNE SSNT SSSS STOR SUM SUN SUPN SVMK SWBI SYF SYRS TBIO TCDA TCF TCRR TDC TEX TFFP TGTX THC TMHC TRGP TRIP TSE TUP TVTY UBX UCBI UCTT UFS UNFI UONE UPWK URBN USFD VCRA VERI VIAC VIRT VIVO VREX VSLR VSTO VXRT WAFD WBS WFC WHD WIFI WKHS WORK WORX WRK WRTC WW WWW WYND XEC XENT XPER XRX YELP ZGNX ZUMZ ZYXI'
split_s_l = s_l.split(" ")
part_1 = ' '.join(split_s_l[:100])
part_2 = ' '.join(split_s_l[100:200])
part_3 = ' '.join(split_s_l[200:300])
part_4 = ' '.join(split_s_l[300:])
for part in (part_1, part_2, part_3, part_4):
print(part)
but I don't know how to do that If I have many elements in list.
For a variable number of items, you can loop using:
sep = ' '
num = 100
split_s_l = s_l.split(sep)
for i in range(0, len(split_s_l), num):
part = sep.join(split_s_l[i : i+num])
print(part)
Bear in mind that for the last slice in the example case ([300:400]) it does not matter that there are only 320 elements -- just the last 20 items will be included (no error).
Something like this?
def break_up(s, nwords, separator):
words = s.split()
return [separator.join(words[n:n+nwords]) for n in range(0, len(words), nwords)]
print(break_up('a b c d e f g h', 3, ' '))
Result:
['a b c', 'd e f', 'g h']
Of course, you might call as print(break_up(s_l, 100, ' '))
Below
s_l = 'AA AAL AAOI ACLS ADT ADTX ADVM AEL AIG ALEC ALLY ALT AMCX ANGI APA APDN APLS APPS APRN AQUA ARMK ARNC ARVN ATNM ATOM ATRA ATSG AVCT AVT AX AXDX BCLI BE BEAM BJRI BKE BKU BLDR BLNK BMRA BOOT BXS BYD CAKE CALX CAPR CARG CARR CARV CATM CC CCL CELH CEQP CFG CHEF CHRS CIT CLDX CLR CLSK CNK CNST CODX COLB COOP CPE CRS CTVA CUK CVET CVI CVM CYTK DAL DBX DCP DDS DEI DISCA DISCK DK DNB DRNA DVAX DXC ECOM EIGR ELAN ELF ELY ENVA EQ EQT EXEL FE FHI FIXX FL FLWS FMCI FORM FOX FOXA FRTA FUN GBX GIII GM GNMK GOCO GPRE GRAF GRPN GRWG GTHX GWB HALO HCC HCSG HEAR HFC HGV HIBB HMSY HOG HOME HP HSC HTH HWC IMUX IMVT INO INOV INSG INSM INT IOVA IRDM ITCI JELD JWN KMT KODK KPTI KSS KTB KTOS KURA LAKE LB LCA LL LPI LPRO LSCC LYFT MAXR MBOT MCRB MCS MD MDP MGM MGNX MIC MLHR MOS MRSN MTOR MXL MYGN NCLH NCR NK NKTR NLS NMIH NOVA NTLA NTNX NUAN NVST NXTC ODP OFC OKE OMER OMF OMI ONEM OSPN OSUR OXY OZK PACW PD PDCE PDCO PEAK PGNY PLAY PLCE PLT PLUG PPBI PRPL PRTS PRVB PS PSNL PSTX PSXP PTGX PVAC RCUS REAL REZI RKT RMBL RPAY RRGB RRR RVLV RVP RXN SANM SAVE SBGI SC SCPL SEAS SEM SFIX SFM SGMS SGRY SHLL SHOO SHYF SIX SKX SLQT SMCI SNAP SNDX SNV SONO SPAQ SPCE SPR SPWH SPWR SRG SRNE SSNT SSSS STOR SUM SUN SUPN SVMK SWBI SYF SYRS TBIO TCDA TCF TCRR TDC TEX TFFP TGTX THC TMHC TRGP TRIP TSE TUP TVTY UBX UCBI UCTT UFS UNFI UONE UPWK URBN USFD VCRA VERI VIAC VIRT VIVO VREX VSLR VSTO VXRT WAFD WBS WFC WHD WIFI WKHS WORK WORX WRK WRTC WW WWW WYND XEC XENT XPER XRX YELP ZGNX ZUMZ ZYXI'
words= s_l.split(" ")
num_of_100_groups = int(len(words) / 100)
groups = []
for i in range(0,num_of_100_groups):
groups.append(words[i * 100 : (i+1) * 100])
groups.append(words[num_of_100_groups * 100:])
for sub_group in groups:
print(' '.join(sub_group))
I have 2 pythons on my mac (10.8.3): The default, and 2.7 version from homebrew.
So far, I could install modules and use them with my brew python. I installed mapnik with brew install mapnik (mapnik-2.1.0) and it compiled correctly. But, if I open python and enter import mapnik, the following error comes and python exits:
>>> import mapnik
Fatal Python error: PyThreadState_Get: no current thread
Abort trap: 6
Mac os x also shows an error window, with the following content:
Process: Python [60666]
Path: /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
Identifier: Python
Version: 2.7.3 (2.7.3)
Code Type: X86-64 (Native)
Parent Process: bash [60454]
User ID: 501
Date/Time: 2013-03-28 10:21:08.535 +0100
OS Version: Mac OS X 10.8.3 (12D78)
Report Version: 10
Interval Since Last Report: 128837 sec
Crashes Since Last Report: 5
Per-App Crashes Since Last Report: 4
Anonymous UUID: 567121E6-7BAC-335F-E3B0-DD24D1F9E6BA
Crashed Thread: 0 Dispatch queue: com.apple.main-thread
Exception Type: EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Application Specific Information:
abort() called
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 0x00007fff94500d46 __kill + 10
1 libsystem_c.dylib 0x00007fff92afddf0 abort + 177
2 org.python.python 0x0000000105a77eaa Py_FatalError + 49
3 org.python.python 0x0000000105a76370 PyThreadState_Get + 28
4 org.python.python 0x0000000105a71f16 Py_InitModule4_64 + 58
5 libboost_python-mt.dylib 0x0000000104153288 boost::python::detail::init_module(char const*, void (*)()) + 40
6 org.python.python 0x00000001029fa716 _PyImport_LoadDynamicModule + 150
7 org.python.python 0x00000001029fa428 import_submodule + 296
8 org.python.python 0x00000001029f9ff1 load_next + 268
9 org.python.python 0x00000001029f80fb PyImport_ImportModuleLevel + 794
10 org.python.python 0x00000001029de9d3 builtin___import__ + 132
11 org.python.python 0x000000010296e335 PyObject_Call + 101
12 org.python.python 0x00000001029e8e10 PyEval_CallObjectWithKeywords + 93
13 org.python.python 0x00000001029e61b6 PyEval_EvalFrameEx + 11727
14 org.python.python 0x00000001029e32bd PyEval_EvalCodeEx + 1638
15 org.python.python 0x00000001029e2c51 PyEval_EvalCode + 54
16 org.python.python 0x00000001029f6e58 PyImport_ExecCodeModuleEx + 247
17 org.python.python 0x00000001029f99bf load_source_module + 1053
18 org.python.python 0x00000001029f9c7e load_package + 334
19 org.python.python 0x00000001029fa428 import_submodule + 296
20 org.python.python 0x00000001029f9ff1 load_next + 268
21 org.python.python 0x00000001029f80fb PyImport_ImportModuleLevel + 794
22 org.python.python 0x00000001029de9d3 builtin___import__ + 132
23 org.python.python 0x000000010296e335 PyObject_Call + 101
24 org.python.python 0x00000001029e8e10 PyEval_CallObjectWithKeywords + 93
25 org.python.python 0x00000001029e61b6 PyEval_EvalFrameEx + 11727
26 org.python.python 0x00000001029e32bd PyEval_EvalCodeEx + 1638
27 org.python.python 0x00000001029e2c51 PyEval_EvalCode + 54
28 org.python.python 0x0000000102a01a2b run_mod + 53
29 org.python.python 0x0000000102a0184d PyRun_InteractiveOneFlags + 365
30 org.python.python 0x0000000102a01367 PyRun_InteractiveLoopFlags + 188
31 org.python.python 0x0000000102a01218 PyRun_AnyFileExFlags + 60
32 org.python.python 0x0000000102a120b2 Py_Main + 3210
33 libdyld.dylib 0x00007fff966bf7e1 start + 1
Thread 0 crashed with X86 Thread State (64-bit):
rax: 0x0000000000000000 rbx: 0x00007fff5d29c2b0 rcx: 0x00007fff5d29c298 rdx: 0x0000000000000000
rdi: 0x000000000000ecfa rsi: 0x0000000000000006 rbp: 0x00007fff5d29c2c0 rsp: 0x00007fff5d29c298
r8: 0x00000000000003f5 r9: 0x00007fff5d29c270 r10: 0x00007fff94502342 r11: 0x0000000000000206
r12: 0x00007fff5d29cb00 r13: 0x0000000102e9b656 r14: 0x00000000000003f5 r15: 0x0000000000000000
rip: 0x00007fff94500d46 rfl: 0x0000000000000206 cr2: 0x00007fff7bff7ff0
Logical CPU: 0
Binary Images:
0x102961000 - 0x102961fff +org.python.python (2.7.3 - 2.7.3) <62DA7BCA-2A0C-3753-A043-7459827F56D1> /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
0x102964000 - 0x102a6cfff +org.python.python (2.7.3, [c] 2004-2012 Python Software Foundation. - 2.7.3) <81E6FB4A-287C-37C3-A26D-D045B604933F> /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/Python
0x102c4c000 - 0x102c4efff +readline.so (0) <C4F1219C-CDFD-37D2-A762-22974D3F2918> /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/readline.so
0x102c54000 - 0x102c77fff +libreadline.6.2.dylib (6.2) <2131C2A4-E75D-3680-9C8D-E42D78A5E1B9> /usr/local/opt/readline/lib/libreadline.6.2.dylib
0x102cfc000 - 0x102d01fff +itertools.so (0) <F857A819-40A9-3F72-A0B1-3E97BA5F3DAA> /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/itertools.so
0x102d0a000 - 0x102f2cfff +_mapnik.so (0) <433DAD47-AB4D-37C5-A0EA-FE1134B63D3D> /usr/local/lib/python2.7/site-packages/mapnik/_mapnik.so
0x103603000 - 0x103978ff7 +libmapnik.dylib (0) <0E898678-9D86-35B4-A772-0248924F5BF3> /usr/local/Cellar/mapnik/2.1.0/lib/libmapnik.dylib
0x104138000 - 0x104170ff7 +libboost_python-mt.dylib (0) <9B5DD680-BEC9-3988-9F55-F59DA38CC175> /usr/local/lib/libboost_python-mt.dylib
0x1041b3000 - 0x1041c2ff7 +libboost_thread-mt.dylib (0) <24432300-9373-30B6-89F6-857CBEF9C105> /usr/local/lib/libboost_thread-mt.dylib
0x1041de000 - 0x10424dff7 +libfreetype.6.dylib (0) <490EDE42-5B78-34BA-85E7-D0547BE778E1> /usr/local/lib/libfreetype.6.dylib
0x104262000 - 0x104268fff +libltdl.7.dylib (0) <E437E59F-FC2E-3837-BDF8-301BEF13C7D6> /usr/local/lib/libltdl.7.dylib
0x10426c000 - 0x10428bff7 +libpng15.15.dylib (0) <52907DBF-A04C-325D-9E77-DF56A771026F> /usr/local/lib/libpng15.15.dylib
0x104293000 - 0x1042e6fff +libtiff.5.dylib (0) <AE85D7B1-F4E5-3B27-832D-5756EFCDE912> /usr/local/lib/libtiff.5.dylib
0x1042f2000 - 0x104326fff +libproj.0.dylib (0) <C799D9A7-0DDA-334D-9AEA-13A4AE7A67A2> /usr/local/lib/libproj.0.dylib
0x104333000 - 0x104441ff7 +libicuuc.50.1.dylib (0) <5FDBBF25-EEA6-3649-9ED6-B18CE89CE510> /usr/local/opt/icu4c/lib/libicuuc.50.1.dylib
0x1044a3000 - 0x1044b2fff +libboost_filesystem-mt.dylib (0) <B2C03485-5FA6-3744-BC99-DB4E87DA4D87> /usr/local/lib/libboost_filesystem-mt.dylib
0x1044c3000 - 0x1044c5ff7 +libboost_system-mt.dylib (0) <FE5E6E11-C7FB-3895-9976-526D22997EDC> /usr/local/lib/libboost_system-mt.dylib
0x1044ca000 - 0x104578fff +libboost_regex-mt.dylib (0) <BC1D7F46-F6AB-3964-B344-E692EFC5D3B7> /usr/local/lib/libboost_regex-mt.dylib
0x10461c000 - 0x10464bff7 +libjpeg.8.dylib (0) <9EC07360-CF18-3529-AE54-E60DBF1313DC> /usr/local/lib/libjpeg.8.dylib
0x104652000 - 0x105a24fff +libicudata.50.dylib (0) <B74D5789-7082-3084-9BF0-9A3AE6F2B046> /usr/local/opt/icu4c/lib/libicudata.50.dylib
0x105a26000 - 0x105b33fff org.python.python (2.7.2 - 2.7.2) <E7F3EED1-E55D-32AF-9649-77C814693F6A> /System/Library/Frameworks/Python.framework/Versions/2.7/Python
0x7fff62561000 - 0x7fff6259593f dyld (210.2.3) <36CAA36E-72BC-3E48-96D9-B96A2DF77730> /usr/lib/dyld
0x7fff8ae57000 - 0x7fff8ae7cff7 libc++abi.dylib (26) <D86169F3-9F31-377A-9AF3-DB17142052E4> /usr/lib/libc++abi.dylib
0x7fff8c50f000 - 0x7fff8c510ff7 libsystem_sandbox.dylib (220.2) <6838A6FD-8626-3356-BB4F-BB4787216207> /usr/lib/system/libsystem_sandbox.dylib
0x7fff8c511000 - 0x7fff8c523ff7 libz.1.dylib (43) <2A1551E8-A272-3DE5-B692-955974FE1416> /usr/lib/libz.1.dylib
0x7fff8d337000 - 0x7fff8d342fff libsystem_notify.dylib (98.5) <C49275CC-835A-3207-AFBA-8C01374927B6> /usr/lib/system/libsystem_notify.dylib
0x7fff8d9e6000 - 0x7fff8d9ebfff libcompiler_rt.dylib (30) <08F8731D-5961-39F1-AD00-4590321D24A9> /usr/lib/system/libcompiler_rt.dylib
0x7fff8dd72000 - 0x7fff8dd73fff libDiagnosticMessagesClient.dylib (8) <8548E0DC-0D2F-30B6-B045-FE8A038E76D8> /usr/lib/libDiagnosticMessagesClient.dylib
0x7fff902d7000 - 0x7fff90326ff7 libcorecrypto.dylib (106.2) <CE0C29A3-C420-339B-ADAA-52F4683233CC> /usr/lib/system/libcorecrypto.dylib
0x7fff9064f000 - 0x7fff90650fff libsystem_blocks.dylib (59) <D92DCBC3-541C-37BD-AADE-ACC75A0C59C8> /usr/lib/system/libsystem_blocks.dylib
0x7fff90a75000 - 0x7fff90a7dff7 libsystem_dnssd.dylib (379.37) <616FC901-151E-38BF-B2C4-24A351C5FAAD> /usr/lib/system/libsystem_dnssd.dylib
0x7fff914fd000 - 0x7fff9150afff libbz2.1.0.dylib (29) <CE9785E8-B535-3504-B392-82F0064D9AF2> /usr/lib/libbz2.1.0.dylib
0x7fff92899000 - 0x7fff928a7fff libcommonCrypto.dylib (60027) <BAAFE0C9-BB86-3CA7-88C0-E3CBA98DA06F> /usr/lib/system/libcommonCrypto.dylib
0x7fff92aa4000 - 0x7fff92b70ff7 libsystem_c.dylib (825.26) <4C9EB006-FE1F-3F8F-8074-DFD94CF2CE7B> /usr/lib/system/libsystem_c.dylib
0x7fff92bba000 - 0x7fff92bc0ff7 libunwind.dylib (35.1) <21703D36-2DAB-3D8B-8442-EAAB23C060D3> /usr/lib/system/libunwind.dylib
0x7fff92ed3000 - 0x7fff92f3cfff libstdc++.6.dylib (56) <EAA2B53E-EADE-39CF-A0EF-FB9D4940672A> /usr/lib/libstdc++.6.dylib
0x7fff93019000 - 0x7fff93065ff7 libauto.dylib (185.1) <73CDC482-16E3-3FC7-9BB4-FBA2DA44DBC2> /usr/lib/libauto.dylib
0x7fff930cf000 - 0x7fff930d5fff libmacho.dylib (829) <BF332AD9-E89F-387E-92A4-6E1AB74BD4D9> /usr/lib/system/libmacho.dylib
0x7fff93177000 - 0x7fff93199ff7 libxpc.dylib (140.42) <BBE558BD-5E55-35E4-89ED-1AA6B056D05A> /usr/lib/system/libxpc.dylib
0x7fff931f6000 - 0x7fff931f7ff7 libremovefile.dylib (23.2) <6763BC8E-18B8-3AD9-8FFA-B43713A7264F> /usr/lib/system/libremovefile.dylib
0x7fff931f8000 - 0x7fff93260ff7 libc++.1.dylib (65.1) <20E31B90-19B9-3C2A-A9EB-474E08F9FE05> /usr/lib/libc++.1.dylib
0x7fff9345d000 - 0x7fff9365dfff libicucore.A.dylib (491.11.2) <FD6282D8-DF3F-3842-8C2E-CF478D2B9669> /usr/lib/libicucore.A.dylib
0x7fff9365e000 - 0x7fff9365efff libkeymgr.dylib (25) <CC9E3394-BE16-397F-926B-E579B60EE429> /usr/lib/system/libkeymgr.dylib
0x7fff936d4000 - 0x7fff936dcfff liblaunch.dylib (442.26.2) <2F71CAF8-6524-329E-AC56-C506658B4C0C> /usr/lib/system/liblaunch.dylib
0x7fff9424d000 - 0x7fff9427bff7 libsystem_m.dylib (3022.6) <B434BE5C-25AB-3EBD-BAA7-5304B34E3441> /usr/lib/system/libsystem_m.dylib
0x7fff94284000 - 0x7fff94286fff libquarantine.dylib (52) <4BE2E642-A14F-340A-B482-5BD2AEFD9C24> /usr/lib/system/libquarantine.dylib
0x7fff94381000 - 0x7fff94382ff7 libdnsinfo.dylib (453.19) <14202FFB-C3CA-3FCC-94B0-14611BF8692D> /usr/lib/system/libdnsinfo.dylib
0x7fff944ef000 - 0x7fff9450aff7 libsystem_kernel.dylib (2050.22.13) <5A961E2A-CFB8-362B-BC43-122704AEB047> /usr/lib/system/libsystem_kernel.dylib
0x7fff94579000 - 0x7fff94587ff7 libsystem_network.dylib (77.10) <0D99F24E-56FE-380F-B81B-4A4C630EE587> /usr/lib/system/libsystem_network.dylib
0x7fff94821000 - 0x7fff9493992f libobjc.A.dylib (532.2) <90D31928-F48D-3E37-874F-220A51FD9E37> /usr/lib/libobjc.A.dylib
0x7fff9493a000 - 0x7fff9494fff7 libdispatch.dylib (228.23) <D26996BF-FC57-39EB-8829-F63585561E09> /usr/lib/system/libdispatch.dylib
0x7fff95c25000 - 0x7fff95d22ff7 libxml2.2.dylib (22.3) <47B09CB2-C636-3024-8B55-6040F7829B4C> /usr/lib/libxml2.2.dylib
0x7fff96350000 - 0x7fff96351ff7 libSystem.B.dylib (169.3) <9089D72D-E714-31E1-80C8-698A8E8B05AD> /usr/lib/libSystem.B.dylib
0x7fff96397000 - 0x7fff9639cfff libcache.dylib (57) <65187C6E-3FBF-3EB8-A1AA-389445E2984D> /usr/lib/system/libcache.dylib
0x7fff96646000 - 0x7fff9664dfff libcopyfile.dylib (89) <876573D0-E907-3566-A108-577EAD1B6182> /usr/lib/system/libcopyfile.dylib
0x7fff966bd000 - 0x7fff966c0ff7 libdyld.dylib (210.2.3) <F59367C9-C110-382B-A695-9035A6DD387E> /usr/lib/system/libdyld.dylib
0x7fff96961000 - 0x7fff96963ff7 libunc.dylib (25) <92805328-CD36-34FF-9436-571AB0485072> /usr/lib/system/libunc.dylib
0x7fff974ba000 - 0x7fff974f0fff libsystem_info.dylib (406.17) <4FFCA242-7F04-365F-87A6-D4EFB89503C1> /usr/lib/system/libsystem_info.dylib
0x7fff97850000 - 0x7fff97888fff libncurses.5.4.dylib (37.3) <68D5B5F5-8252-3F1E-AFF1-C6AFE145DBC1> /usr/lib/libncurses.5.4.dylib
0x7fff978d3000 - 0x7fff97abdff7 com.apple.CoreFoundation (6.8 - 744.18) <A60C3C9B-3764-3291-844C-C487ACF77C2C> /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
External Modification Summary:
Calls made by other processes targeting this process:
task_for_pid: 0
thread_create: 0
thread_set_state: 0
Calls made by this process:
task_for_pid: 0
thread_create: 0
thread_set_state: 0
Calls made by all processes on this machine:
task_for_pid: 18730
thread_create: 2
thread_set_state: 0
VM Region Summary:
ReadOnly portion of Libraries: Total=109.5M resident=53.3M(49%) swapped_out_or_unallocated=56.2M(51%)
Writable regions: Total=20.6M written=2644K(13%) resident=3256K(15%) swapped_out=0K(0%) unallocated=17.5M(85%)
REGION TYPE VIRTUAL
=========== =======
MALLOC 11.8M
MALLOC guard page 32K
STACK GUARD 56.0M
Stack 8192K
__DATA 2356K
__LINKEDIT 68.3M
__TEXT 41.2M
__UNICODE 544K
shared memory 12K
=========== =======
TOTAL 188.2M
Model: MacBookPro8,2, BootROM MBP81.0047.B27, 4 processors, Intel Core i7, 2 GHz, 8 GB, SMC 1.69f4
Graphics: Intel HD Graphics 3000, Intel HD Graphics 3000, Built-In, 512 MB
Graphics: AMD Radeon HD 6490M, AMD Radeon HD 6490M, PCIe, 256 MB
Memory Module: BANK 0/DIMM0, 4 GB, DDR3, 1333 MHz, 0x80CE, 0x4D34373142353237334348302D4348392020
Memory Module: BANK 1/DIMM0, 4 GB, DDR3, 1333 MHz, 0x80CE, 0x4D34373142353237334348302D4348392020
AirPort: spairport_wireless_card_type_airport_extreme (0x14E4, 0xD6), Broadcom BCM43xx 1.0 (5.106.98.100.16)
Bluetooth: Version 4.1.3f3 11349, 2 service, 11 devices, 1 incoming serial ports
Network Service: Wi-Fi, AirPort, en1
Serial ATA Device: ST9500420AS, 500.11 GB
Serial ATA Device: MATSHITADVD-R UJ-898
USB Device: FaceTime HD Camera (Built-in), apple_vendor_id, 0x8509, 0xfa200000 / 3
USB Device: hub_device, 0x0424 (SMSC), 0x2513, 0xfa100000 / 2
USB Device: Apple Internal Keyboard / Trackpad, apple_vendor_id, 0x0246, 0xfa120000 / 5
USB Device: BRCM2070 Hub, 0x0a5c (Broadcom Corp.), 0x4500, 0xfa110000 / 4
USB Device: Bluetooth USB Host Controller, apple_vendor_id, 0x821a, 0xfa113000 / 8
USB Device: hub_device, 0x0424 (SMSC), 0x2513, 0xfd100000 / 2
USB Device: IR Receiver, apple_vendor_id, 0x8242, 0xfd110000 / 3
Any problems with linking my pythons? how can I fix it? thanks!
It looks like you are running the homebrew python but either boost python or mapnik's python bindings ended up linking against the system python provided by apple. If a full clean and reinstall of boost and Mapnik does not fix this then I recommend stopping by #mapnik irc on freenode for help debugging. Generally to fix you need to call otool -L on boost_python.dylib and or on site_packages/mapnik/_mapnik.so to see which ended up with a linkage to the system python. Then you can fix by using install_name_tool.
I encountered the same problem between python27-apple and python27 (installed by Macport). I tried these steps, and it worked for me.
According to #Evert's comments of different libraries used by different python, I tried to revert the python27-apple to be "active".
sudo port select --list python
For my computer, it showed:
Available versions for python:
none
python25-apple
python26
python26-apple
python27 (active)
python27-apple
in which python27(by Macport) was currently active. Then,
sudo port select python python27-apple
to make python27-apple active as the default lib were called previously. To double check,
sudo port select --list python
It now showed:
Available versions for python:
none
python25-apple
python26
python26-apple
python27
python27-apple (active)
And the python crash had gone when I ran the scripts.
You can use the homebrew python like this:
/usr/bin/python2.7 yourscript.py
I face the similar problem in using androguard. Though it's not quite the same, it can be a reference. My crash also is Fatal Python error: PyThreadState_Get: no current thread
Abort trap: 6. Run otool -L <lib>, suggest the dylib link to the system python.
Try solution:
python-config --ldflags
Copy the output to corresponding MakeFile. For example change LDFLAGS += -lm -lbz2 -lz -llzma -lsnappy -lmuparser -lpython to LDFLAGS += -lm -lbz2 -lz -llzma -lsnappy -lmuparser -L/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config -lpython2.7 -ldl -framework CoreFoundation.
make clean, then make again.
Now the dylib is OK, and no error anymore.
Alternatively ...
... when you use otool -L <lib> to find the wrongly linked python dylib you can use install_name_tool to change the symbol like:
install_name_tool -change /System/Library/Frameworks/Python.framework/Versions/2.7/Python /usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib <lib>