Related
10.223.157.186 - - [15/Jul/2009:14:58:59 -0700] "GET / HTTP/1.1" 403 202
10.223.157.186 - - [15/Jul/2009:14:58:59 -0700] "GET /favicon.ico HTTP/1.1" 404 209
10.223.157.186 - - [15/Jul/2009:15:50:35 -0700] "GET / HTTP/1.1" 200 9157
10.211.47.159 - - [10/Aug/2009:20:52:19 -0700] "GET /assets/js/lowpro.js HTTP/1.1" 304 -
10.216.113.172 - - [12/Aug/2009:06:04:50 -0700] "GET /release-schedule/ HTTP/1.1" 200 9306
10.216.113.172 - - [12/Aug/2009:06:04:50 -0700] "GET /release-schedule/ HTTP/1.1" 200 9306
10.216.113.172 - - [12/Aug/2009:06:04:52 -0700] "GET /displaytitle.php?id=10 HTTP/1.1" 200 10234
10.216.113.172 - - [12/Aug/2009:06:04:52 -0700] "GET /displaytitle.php?id=10 HTTP/1.1" 200 10234
Lets say i have a column have all the host which means (10.223.157.186) and I want to find the most frequent used host.
result = Mainpanda['host'].mode()
print("Mode:\n",result)
I know by using host can find them but somehow it only display the top 1, I need to put them in to a list from 1 to the N
may someone please help me?
Assuming you have a single column as string, first extract the IP, then use mode:
result = Mainpanda['host'].str.extract(r'(\d+\.\d+\.\d+\.\d+)', expand=False).mode()
Output:
0 10.216.113.172
Name: host, dtype: object
If you want the top N, with the counts, use value_counts:
N = 10
result = (Mainpanda['host']
.str.extract(r'(\d+\.\d+\.\d+\.\d+)', expand=False)
.value_counts().head(10)
)
Output:
10.216.113.172 4
10.223.157.186 3
10.211.47.159 1
...
Name: host, dtype: int64
If you want only a list of IPs:
N = 10
result = (Mainpanda['host']
.str.extract(r'(\d+\.\d+\.\d+\.\d+)', expand=False)
.value_counts().index[:N].tolist()
)
Output:
['10.216.113.172', '10.223.157.186', '10.211.47.159', ...]
I created python script to scrape facebook help page. I wanted to scrape cms_object_id, cmsID, name. so these values are in a script tag then firstly tried to find all <script> tags then tried to iterate over this and then there is __bbox inside the tags which contains the values wanted to scrape.
so this is my script:
import json
import requests
import bs4
from Essentials import Static
class CmsIDs:
def GetIDs():
# cont = requests.get(""https://www.facebook.com:443/help"", headers=Static.headers) # syntax error
cont = requests.get("https://www.facebook.com:443/help", headers=Static.headers)
soup = bs4.BeautifulSoup(cont.content, "html5lib")
text = soup.find_all("script")
start = ""
txtstr = ""
for i in range(len(text)):
mystr = text[i]
# mystr = text[i]
print("this is: ", mystr.find('__bbox":'))
if text[i].get_text().find('__bbox":') != -1:
# print(i, text[i].get_text())
txtstr += text[i].get_text()
start = text[i].get_text().find('__bbox":') + len('__bbox":')
print('start:', start)
count = 0
for end, char in enumerate(txtstr[start:], start):
if char == '{':
count += 1
if char == '}':
count -= 1
if count == 0:
break
print('end:', end)
# --- convert JSON string to Python structure (dict/list) ---
data = json.loads(txtstr[start:end+1])
# pp.pprint(data)
print('--- search ---')
CmsIDs.search(data)
# --- use recursion to find all 'cms_object_id', 'cmsID', 'name' ---
def search(data):
if isinstance(data, dict):
found = False
if 'cms_object_id' in data:
print('cms_object_id', data['cms_object_id'])
found = True
if 'cmsID' in data:
print('cmsID', data['cmsID'])
found = True
if 'name' in data:
print('name', data['name'])
found = True
if found:
print('---')
for val in data.values():
CmsIDs.search(val)
if isinstance(data, list):
for val in data:
CmsIDs.search(val)
if __name__ == '__main__':
CmsIDs.GetIDs()
the page contains cms_object_id, cmsID, name. so wanted to scrape all these 3 values but I am getting an error:
for end, char in enumerate(txtstr[start:], start):
TypeError: slice indices must be integers or None or have an __index__ method
so how can I solve this error and reach ultimate goal?
Note: Since I'm unfamiliar with and failed to install Essentials, and also because ""https://www.facebook.com:443/help"" raises a syntax error (there should only be one quote on each side of the string), I changed the requests line in my code.
cont = requests.get('https://www.facebook.com:443/help', headers={'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'})
TypeError: slice indices must be integers or None or have an __index__ method
You've initiated start as a string [start = ""] and it needs to be an integer. Unless the if text[i].get_text().find('__bbox":') != -1.... block is entered, start remains a string.
if you just want to not get this error, you could just exit the program if start hasn't been updated [indicating that __bbox": wasn't found in any of the script tags].
print('start:', start)
if start == "":
print('{__bbox":} not found')
return
count = 0
But that still leaves the problem of __bbox": not being found; I'm not sure why, but the issue is resolved for me if I don't use the html5lib parser - just changing to BeautifulSoup(cont.content) resolved this issue.
# soup = bs4.BeautifulSoup(cont.content, "html5lib")
soup = bs4.BeautifulSoup(cont.content) # don't define the parser
# soup = bs4.BeautifulSoup(cont.content, "html.parser") # you could also try other parsers
Other suggestions
Your code will probably work without these, but you might want to consider these suggested improvements for error-handling:
Filter the script Tags
If the text ResultSet only has script tags that contain __bbox":, you avoid looping unnecessarily through the 100+ other scripts, and you won't have to check with if....find('__bbox":') anymore.
text = soup.find_all(lambda t: t.name == 'script' and '"__bbox"' in t.get_text())
for mystr in [s.get_text() for s in text]:
print("this is: ", mystr.find('__bbox":'))
txtstr += mystr
start = mystr.find('__bbox":') + len('__bbox":')
Initiate end
You should initiate the end variable [like end = 0] before the for end, char ... loop because you're using it after the loop as well.
print('end:', end)
data = json.loads(txtstr[start:end+1])
If txtstr[start:] is empty somehow, these lines will raise error/s since end would not be defined yet.
Use a JavaScript Parser
This will make the previous suggestions unnecessary, but as it is json.loads will raise an error if txtstr[start:end+1] is empty somehow, or if it contains any unpaired [and likely escaped] }or{. So, it might be more reliable to use a parser rather than just trying to walk through the string.
I have this function that uses slimit to find values from strings containing JavaScript code. (It's far from perfect, but it seems to for this script, at least.) GetIDs() could be re-written as below.
# import json, requests, bs4, slimit
# from slimit.visitors import nodevisitor
# def findObj_inJS... ## PASTE FROM https://pastebin.com/UVcLniSG
# class CmsIDs:
def GetIDs():
# cont=requests.get('https://www.facebook.com:443/help',headers=Static.headers)
cont = requests.get('https://www.facebook.com:443/help', headers={
'accept': ';'.join(
[ 'text/html,application/xhtml+xml,application/xml',
'q=0.9,image/avif,image/webp,image/apng,*/*',
'q=0.8,application/signed-exchange',
'v=b3', 'q=0.9' ])})
## in case of request errors ##
try: cont.raise_for_status()
except Exception as e:
print('failed to fetch page HTML -', type(e), e)
return
print('fetched', cont.url, 'with', cont.status_code, cont.reason)
soup = bs4.BeautifulSoup(cont.content)
scrCond = lambda t: t.name == 'script' and '"__bbox"' in t.get_text()
jScripts = [s.get_text() for s in soup.find_all(scrCond)]
print(f'Found {len(jScripts)} script tags containing {{"__bbox"}}')
data = [findObj_inJS(s,'"__bbox"') for s in jScripts]
print('--- search ---')
CmsIDs.search(data)
# def search(data)....
Return the Data
This isn't for error-handling, but if you return the data printed by CmsIDs.search you could save it for further use.
def search(data):
rList, dKeys = [], ['cms_object_id', 'cmsID', 'name']
if isinstance(data, dict):
dObj = {k: data[k] for k in dKeys if k in data}
rList += [dObj] if dObj else []
for k, v in dObj.items(): print(k, v)
if dObj: print('---')
for val in data.values(): rList += CmsIDs.search(val)
if isinstance(data, list):
for val in data: rList += CmsIDs.search(val)
return rList
The printed result will be the same as before, but if you change the last line of GetIDs to
return CmsIDs.search(data)
and then define a variable cmsList = CmsIDs.GetIDs() then cmsList will be a list of dctionaries, which you could then [for example] save to csv with pandas and view as a table on a spreadsheet.
# import pandas
pandas.DataFrame(cmsList).to_csv('CmsIDs_GetIDs.csv', index=False)
or print the markdown for the table [of the results I got] below
print(pandas.DataFrame(cmsList, dtype=str).fillna('').to_markdown())
[index]
cms_object_id
name
cmsID
0
Використання Facebook
1
570785306433644
Створення облікового запису
2
396528481579093
Your Profile
3
Додати й редагувати інформацію у своєму профілі
1017657581651994
4
Ваші основна світлина й обкладинка
1217373834962306
5
Поширення дописів у своєму профілі та керування ними
1640261589632787
6
Усунення проблем
191128814621591
7
1540345696275090
Додавання в друзі
8
Додавання друзів
246750422356731
9
Люди, яких ви можете знати
336320879782850
10
Control Who Can Friend and Follow You
273948399619967
11
Upload Your Contacts to Facebook
1041444532591371
12
Видалення з друзів чи блокування користувача
1000976436606344
13
312959615934334
Facebook Dating
14
753701661398957
Ваша головна сторінка
15
How Feed Works
1155510281178725
16
Control What You See in Feed
964154640320617
17
Like and React to Posts
1624177224568554
18
Пошук
821153694683665
19
Translate Feed
1195058957201487
20
Memories
1056848067697293
21
1071984682876123
Повідомлення
22
Надсилання повідомлень
487151698161671
23
Переглянути повідомлення й керувати ними
1117039378334299
24
Поскаржитися на повідомлення
968185709965912
25
Відеовиклики
287631408243374
26
Fix a Problem
1024559617598844
27
753046815962474
Reels
28
Watching Reels
475378724739085
29
Creating Reels
867690387846185
30
Managing Your Reels
590925116168623
31
862926927385914
Розповіді
32
Як створити розповідь і поширити її
126560554619115
33
View and Reply to Stories
349797465699432
34
Page Stories
425367811379971
35
1069521513115444
Світлини й відео
36
Світлини
1703757313215897
37
Відео
1738143323068602
38
Going Live
931327837299966
39
Albums
490693151131920
40
Додавання позначок
267689476916031
41
Усунення проблеми
507253956146325
42
1041553655923544
Відео у Watch
43
Перегляд шоу та відео
401287967326510
44
Fix a Problem
270093216665260
45
2402655169966967
Gaming
46
Gaming on Facebook
385894640264219
47
Платежі в іграх
248471068848455
48
282489752085908
Сторінки
49
Interact with Pages
1771297453117418
50
Створити сторінку й керувати нею
135275340210354
51
Імена й імена користувачів
1644118259243888
52
Керування налаштуваннями сторінки
1206330326045914
53
Customize a Page
1602483780062090
54
Publishing
1533298140275888
55
Messaging
994476827272050
56
Insights
794890670645072
57
Banning and Moderation
248844142141117
58
Усунути проблему
1020132651404616
59
1629740080681586
Групи
60
Join and Choose Your Settings
1210322209008185
61
Post, Participate and Privacy
530628541788770
62
Create, Engage and Manage Settings
408334464841405
63
Керування групою для адміністраторів
1686671141596230
64
Community Chats
3397387057158160
65
Pages in Groups
1769476376397128
66
Fix a Problem
1075368719167893
67
1076296042409786
Події
68
Create and Manage an Event
572885262883136
69
View and Respond to Events
1571121606521970
70
Facebook Classes
804063877226739
71
833144153745643
Fundraisers and Donations
72
Creating a Fundraiser
356680401435429
73
Пожертва в рамках збору коштів
1409509059114623
74
Особисті збори коштів
332739730519432
75
For Nonprofits
1640008462980459
76
Fix a Problem
2725517974129416
77
1434403039959381
Meta Pay
78
Платежі в іграх
248471068848455
79
Payments in Messages
863171203733904
80
Пожертва в рамках збору коштів
1409509059114623
81
Квитки на заходи
1769557403280350
82
Monetization and Payouts
1737820969853848
83
1713241952104830
Marketplace
84
Як працює Marketplace
1889067784738765
85
Buying on Marketplace
272975853291364
86
Продаж на Marketplace
153832041692242
87
Sell with Shipping on Marketplace
773379109714742
88
Using Checkout on Facebook
1411280809160810
89
Групи з купівлі й продажу
319768015124786
90
Get Help with Marketplace
1127970530677256
91
1642635852727373
Додатки
92
Manage Your Apps
942196655898243
93
Видимість і конфіденційність додатка
1727608884153160
94
866249956813928
Додатки Facebook для мобільних пристроїв
95
Додаток для Android
1639918076332350
96
iPhone and iPad Apps
1158027224227668
97
Facebook Lite App
795302980569545
98
273947702950567
Спеціальні можливості
99
Керування обліковим записом
100
1573156092981768
Вхід і пароль
101
Вхід в обліковий запис
1058033620955509
102
Змінення пароля
248976822124608
103
Виправлення проблеми із входом
283100488694834
104
Завантаження посвідчення особи
582999911881572
105
239070709801747
Налаштування облікового запису
106
Як змінити налаштування облікового запису
1221288724572426
107
Ваше ім’я користувача
1740158369563165
108
Спадкоємці
991335594313139
109
1090831264320592
Імена у Facebook
110
1036755649750898
Сповіщення
111
Push, Email and Text Notifications
530847210446227
112
Виберіть, про що отримувати сповіщення
269880466696699
113
Усунення проблем
1719980288275077
114
109378269482053
Налаштування реклами
115
Як працює реклама у Facebook
516147308587266
116
Контроль реклами, яку ви бачите
1075880512458213
117
Ваша інформація та реклама у Facebook
610457675797481
118
1701730696756992
Доступ до вашої інформації та її завантаження
119
250563911970368
Деактивація або видалення облікового запису
120
Конфіденційність і безпека
121
238318146535333
Ваша конфіденційність
122
Керуйте тим, хто може переглядати контент, який ви поширюєте у Facebook
1297502253597210
123
Керування своїми дописами
504765303045427
124
Control Who Can Find You
1718866941707011
125
592679377575472
Безпека
126
Джерела щодо боротьби з жорстоким поводженням
726709730764837
127
Ресурси з допомоги для протидії самогубству та самоушкодженню
1553737468262661
128
Crisis Response
141874516227713
129
Ресурси з правил безпеки для допомоги батькам
1079477105456277
130
Інформація для правоохоронних органів
764592980307837
131
235353253505947
Захист облікового запису
132
Функції безпеки та поради з її забезпечення
285695718429403
133
Сповіщення про вхід і двоетапна перевірка
909243165853369
134
Уникайте спаму та шахрайства
1584206335211143
135
236079651241697
Безпека під час здійснення покупок
136
Розпізнавання шахрайства
1086141928978559
137
Уникнення шахрайства
2374002556073992
138
Купівля на Marketplace
721562085854101
139
Поради щодо безпечної купівлі
123884166448529
140
Купуйте впевнено
1599248863596914
141
Політики та скарги
142
1753719584844061
Скарга на порушення
143
Як поскаржитися на щось?
1380418588640631
144
Don't Have an Account?
1723400564614772
145
1126628984024935
Як повідомити про проблему у Facebook
146
186614050293763
Being Your Authentic Self on Facebook
147
1561472897490627
Повідомлення про порушення конфіденційності
148
1216349518398524
Зламані та фальшиві облікові записи
149
275013292838654
Керування обліковим записом померлої людини
150
About Memorialized Accounts
1017717331640041
151
Request to Memorialize or Remove an Account
1111566045566400
152
399224883474207
Інтелектуальна власність
153
Авторське право
1020633957973118
154
Торговельна марка
507663689427413
155
1735443093393986
Про наші політики
I am training a classifier for an imbalanced dataset. In the logs below one can see that the trainingset-auc increases over 1 after the 78th booster is added. What could AUC-ROC value greater than 1 signify? Since the area under the curve is not expected to be greater than 1 could there be something wrong with my oversampled data which is causing this issue? For this training I am using auc-pr as a metric for evaluation to output the best iteration.
I am using the python XGBoost library for this. The hyperparams being used here were tuned for an older version of the same model. Is hyperparam retuning for this version recommended to resolve this?
param = {
'eta':0.05,
'subsample':0.5,
'colsample_bytree': 0.9387046046168422,
'gamma': 4.533133064606544,
'max_depth': 6.167312302144592,
'min_child_weight': 5.369834798013086
}
param['eval_metric']=['auc','aucpr']
param['objective']='binary:logistic'
param['nthread']=6
param['max_depth']=int(param['max_depth'])
watchlist = [(dtrain,'trainingset'), (dtest,'evalualtionset')]
bst = xgb.train(param, dtrain, num_round_opt, watchlist,early_stopping_rounds=10, maximize=True)
[0] trainingset-auc:0.97164 trainingset-aucpr:0.98648 evalualtionset-auc:0.96008 evalualtionset-aucpr:0.65317
[1] trainingset-auc:0.97540 trainingset-aucpr:0.98780 evalualtionset-auc:0.96368 evalualtionset-aucpr:0.67202
[2] trainingset-auc:0.97850 trainingset-aucpr:0.99092 evalualtionset-auc:0.96786 evalualtionset-aucpr:0.71809
[3] trainingset-auc:0.97884 trainingset-aucpr:0.99107 evalualtionset-auc:0.96856 evalualtionset-aucpr:0.72405
[4] trainingset-auc:0.98322 trainingset-aucpr:0.99278 evalualtionset-auc:0.97693 evalualtionset-aucpr:0.79977
[5] trainingset-auc:0.98384 trainingset-aucpr:0.99299 evalualtionset-auc:0.97684 evalualtionset-aucpr:0.79582
[6] trainingset-auc:0.98400 trainingset-aucpr:0.99327 evalualtionset-auc:0.97822 evalualtionset-aucpr:0.80698
[7] trainingset-auc:0.98405 trainingset-aucpr:0.99329 evalualtionset-auc:0.97830 evalualtionset-aucpr:0.80772
[8] trainingset-auc:0.98406 trainingset-aucpr:0.99329 evalualtionset-auc:0.97806 evalualtionset-aucpr:0.80557
[9] trainingset-auc:0.98392 trainingset-aucpr:0.99328 evalualtionset-auc:0.97838 evalualtionset-aucpr:0.80744
[10] trainingset-auc:0.98417 trainingset-aucpr:0.99335 evalualtionset-auc:0.97845 evalualtionset-aucpr:0.80712
[11] trainingset-auc:0.98423 trainingset-aucpr:0.99337 evalualtionset-auc:0.97835 evalualtionset-aucpr:0.80620
[12] trainingset-auc:0.98497 trainingset-aucpr:0.99364 evalualtionset-auc:0.97885 evalualtionset-aucpr:0.80868
[13] trainingset-auc:0.98496 trainingset-aucpr:0.99364 evalualtionset-auc:0.97885 evalualtionset-aucpr:0.80832
[14] trainingset-auc:0.98585 trainingset-aucpr:0.99380 evalualtionset-auc:0.97940 evalualtionset-aucpr:0.80942
[15] trainingset-auc:0.98620 trainingset-aucpr:0.99380 evalualtionset-auc:0.98022 evalualtionset-aucpr:0.81149
[16] trainingset-auc:0.98633 trainingset-aucpr:0.99385 evalualtionset-auc:0.98035 evalualtionset-aucpr:0.81172
[17] trainingset-auc:0.98660 trainingset-aucpr:0.99393 evalualtionset-auc:0.98026 evalualtionset-aucpr:0.81135
[18] trainingset-auc:0.98666 trainingset-aucpr:0.99395 evalualtionset-auc:0.98014 evalualtionset-aucpr:0.81041
[19] trainingset-auc:0.98720 trainingset-aucpr:0.99414 evalualtionset-auc:0.98070 evalualtionset-aucpr:0.81233
[20] trainingset-auc:0.98726 trainingset-aucpr:0.99418 evalualtionset-auc:0.98115 evalualtionset-aucpr:0.81617
[21] trainingset-auc:0.98748 trainingset-aucpr:0.99434 evalualtionset-auc:0.98128 evalualtionset-aucpr:0.81832
[22] trainingset-auc:0.98778 trainingset-aucpr:0.99448 evalualtionset-auc:0.98151 evalualtionset-aucpr:0.82042
[23] trainingset-auc:0.98787 trainingset-aucpr:0.99451 evalualtionset-auc:0.98158 evalualtionset-aucpr:0.82183
[24] trainingset-auc:0.98803 trainingset-aucpr:0.99456 evalualtionset-auc:0.98171 evalualtionset-aucpr:0.82221
[25] trainingset-auc:0.98821 trainingset-aucpr:0.99464 evalualtionset-auc:0.98194 evalualtionset-aucpr:0.82412
[26] trainingset-auc:0.98836 trainingset-aucpr:0.99459 evalualtionset-auc:0.98231 evalualtionset-aucpr:0.82474
[27] trainingset-auc:0.98872 trainingset-aucpr:0.99467 evalualtionset-auc:0.98255 evalualtionset-aucpr:0.82629
[28] trainingset-auc:0.98948 trainingset-aucpr:0.99503 evalualtionset-auc:0.98272 evalualtionset-aucpr:0.82888
[29] trainingset-auc:0.98964 trainingset-aucpr:0.99510 evalualtionset-auc:0.98270 evalualtionset-aucpr:0.82944
[30] trainingset-auc:0.98982 trainingset-aucpr:0.99513 evalualtionset-auc:0.98285 evalualtionset-aucpr:0.83007
[31] trainingset-auc:0.98996 trainingset-aucpr:0.99516 evalualtionset-auc:0.98299 evalualtionset-aucpr:0.83081
[32] trainingset-auc:0.99007 trainingset-aucpr:0.99518 evalualtionset-auc:0.98307 evalualtionset-aucpr:0.83084
[33] trainingset-auc:0.99016 trainingset-aucpr:0.99520 evalualtionset-auc:0.98322 evalualtionset-aucpr:0.83247
[34] trainingset-auc:0.99044 trainingset-aucpr:0.99526 evalualtionset-auc:0.98339 evalualtionset-aucpr:0.83269
[35] trainingset-auc:0.99055 trainingset-aucpr:0.99530 evalualtionset-auc:0.98356 evalualtionset-aucpr:0.83398
[36] trainingset-auc:0.99069 trainingset-aucpr:0.99534 evalualtionset-auc:0.98371 evalualtionset-aucpr:0.83408
[37] trainingset-auc:0.99082 trainingset-aucpr:0.99542 evalualtionset-auc:0.98383 evalualtionset-aucpr:0.83512
[38] trainingset-auc:0.99143 trainingset-aucpr:0.99562 evalualtionset-auc:0.98401 evalualtionset-aucpr:0.83669
[39] trainingset-auc:0.99149 trainingset-aucpr:0.99565 evalualtionset-auc:0.98410 evalualtionset-aucpr:0.83738
[40] trainingset-auc:0.99178 trainingset-aucpr:0.99572 evalualtionset-auc:0.98423 evalualtionset-aucpr:0.83810
[41] trainingset-auc:0.99228 trainingset-aucpr:0.99579 evalualtionset-auc:0.98449 evalualtionset-aucpr:0.83873
[42] trainingset-auc:0.99235 trainingset-aucpr:0.99582 evalualtionset-auc:0.98461 evalualtionset-aucpr:0.83915
[43] trainingset-auc:0.99250 trainingset-aucpr:0.99585 evalualtionset-auc:0.98470 evalualtionset-aucpr:0.83996
[44] trainingset-auc:0.99277 trainingset-aucpr:0.99593 evalualtionset-auc:0.98486 evalualtionset-aucpr:0.84172
[45] trainingset-auc:0.99284 trainingset-aucpr:0.99595 evalualtionset-auc:0.98501 evalualtionset-aucpr:0.84308
[46] trainingset-auc:0.99306 trainingset-aucpr:0.99602 evalualtionset-auc:0.98517 evalualtionset-aucpr:0.84485
[47] trainingset-auc:0.99330 trainingset-aucpr:0.99608 evalualtionset-auc:0.98522 evalualtionset-aucpr:0.84564
[48] trainingset-auc:0.99348 trainingset-aucpr:0.99611 evalualtionset-auc:0.98520 evalualtionset-aucpr:0.84604
[49] trainingset-auc:0.99359 trainingset-aucpr:0.99612 evalualtionset-auc:0.98533 evalualtionset-aucpr:0.84591
[50] trainingset-auc:0.99370 trainingset-aucpr:0.99616 evalualtionset-auc:0.98534 evalualtionset-aucpr:0.84641
[51] trainingset-auc:0.99389 trainingset-aucpr:0.99621 evalualtionset-auc:0.98532 evalualtionset-aucpr:0.84722
[52] trainingset-auc:0.99406 trainingset-aucpr:0.99625 evalualtionset-auc:0.98531 evalualtionset-aucpr:0.84768
[53] trainingset-auc:0.99418 trainingset-aucpr:0.99629 evalualtionset-auc:0.98536 evalualtionset-aucpr:0.84730
[54] trainingset-auc:0.99431 trainingset-aucpr:0.99631 evalualtionset-auc:0.98540 evalualtionset-aucpr:0.84793
[55] trainingset-auc:0.99447 trainingset-aucpr:0.99635 evalualtionset-auc:0.98537 evalualtionset-aucpr:0.84761
[56] trainingset-auc:0.99468 trainingset-aucpr:0.99639 evalualtionset-auc:0.98535 evalualtionset-aucpr:0.84717
[57] trainingset-auc:0.99501 trainingset-aucpr:0.99644 evalualtionset-auc:0.98553 evalualtionset-aucpr:0.84739
[58] trainingset-auc:0.99513 trainingset-aucpr:0.99646 evalualtionset-auc:0.98559 evalualtionset-aucpr:0.84769
[59] trainingset-auc:0.99541 trainingset-aucpr:0.99648 evalualtionset-auc:0.98569 evalualtionset-aucpr:0.84786
[60] trainingset-auc:0.99556 trainingset-aucpr:0.99650 evalualtionset-auc:0.98570 evalualtionset-aucpr:0.84787
[61] trainingset-auc:0.99587 trainingset-aucpr:0.99655 evalualtionset-auc:0.98580 evalualtionset-aucpr:0.84833
[62] trainingset-auc:0.99604 trainingset-aucpr:0.99657 evalualtionset-auc:0.98585 evalualtionset-aucpr:0.84825
[63] trainingset-auc:0.99636 trainingset-aucpr:0.99659 evalualtionset-auc:0.98586 evalualtionset-aucpr:0.84789
[64] trainingset-auc:0.99656 trainingset-aucpr:0.99662 evalualtionset-auc:0.98599 evalualtionset-aucpr:0.84885
[65] trainingset-auc:0.99668 trainingset-aucpr:0.99664 evalualtionset-auc:0.98601 evalualtionset-aucpr:0.84901
[66] trainingset-auc:0.99678 trainingset-aucpr:0.99665 evalualtionset-auc:0.98604 evalualtionset-aucpr:0.84928
[67] trainingset-auc:0.99696 trainingset-aucpr:0.99666 evalualtionset-auc:0.98606 evalualtionset-aucpr:0.84882
[68] trainingset-auc:0.99711 trainingset-aucpr:0.99668 evalualtionset-auc:0.98611 evalualtionset-aucpr:0.84903
[69] trainingset-auc:0.99740 trainingset-aucpr:0.99672 evalualtionset-auc:0.98619 evalualtionset-aucpr:0.84895
[70] trainingset-auc:0.99751 trainingset-aucpr:0.99674 evalualtionset-auc:0.98615 evalualtionset-aucpr:0.84869
[71] trainingset-auc:0.99783 trainingset-aucpr:0.99679 evalualtionset-auc:0.98622 evalualtionset-aucpr:0.84856
[72] trainingset-auc:0.99815 trainingset-aucpr:0.99682 evalualtionset-auc:0.98628 evalualtionset-aucpr:0.84832
[73] trainingset-auc:0.99838 trainingset-aucpr:0.99687 evalualtionset-auc:0.98628 evalualtionset-aucpr:0.84822
[74] trainingset-auc:0.99875 trainingset-aucpr:0.99690 evalualtionset-auc:0.98640 evalualtionset-aucpr:0.84883
[75] trainingset-auc:0.99884 trainingset-aucpr:0.99691 evalualtionset-auc:0.98646 evalualtionset-aucpr:0.84923
[76] trainingset-auc:0.99900 trainingset-aucpr:0.99692 evalualtionset-auc:0.98646 evalualtionset-aucpr:0.84933
[77] trainingset-auc:0.99951 trainingset-aucpr:0.99695 evalualtionset-auc:0.98674 evalualtionset-aucpr:0.84985
[78] trainingset-auc:1.00024 trainingset-aucpr:0.99699 evalualtionset-auc:0.98697 evalualtionset-aucpr:0.84964
[79] trainingset-auc:1.00058 trainingset-aucpr:0.99700 evalualtionset-auc:0.98719 evalualtionset-aucpr:0.84987
[80] trainingset-auc:1.00090 trainingset-aucpr:0.99702 evalualtionset-auc:0.98727 evalualtionset-aucpr:0.84965
[81] trainingset-auc:1.00111 trainingset-aucpr:0.99706 evalualtionset-auc:0.98733 evalualtionset-aucpr:0.85003
[82] trainingset-auc:1.00129 trainingset-aucpr:0.99710 evalualtionset-auc:0.98741 evalualtionset-aucpr:0.85001
[83] trainingset-auc:1.00143 trainingset-aucpr:0.99711 evalualtionset-auc:0.98744 evalualtionset-aucpr:0.85045
[84] trainingset-auc:1.00161 trainingset-aucpr:0.99716 evalualtionset-auc:0.98743 evalualtionset-aucpr:0.85045
[85] trainingset-auc:1.00235 trainingset-aucpr:0.99721 evalualtionset-auc:0.98778 evalualtionset-aucpr:0.85076
[86] trainingset-auc:1.00266 trainingset-aucpr:0.99724 evalualtionset-auc:0.98788 evalualtionset-aucpr:0.85038
[87] trainingset-auc:1.00302 trainingset-aucpr:0.99725 evalualtionset-auc:0.98803 evalualtionset-aucpr:0.85063
[88] trainingset-auc:1.00341 trainingset-aucpr:0.99729 evalualtionset-auc:0.98837 evalualtionset-aucpr:0.85198
[89] trainingset-auc:1.00393 trainingset-aucpr:0.99731 evalualtionset-auc:0.98868 evalualtionset-aucpr:0.85253
[90] trainingset-auc:1.00445 trainingset-aucpr:0.99732 evalualtionset-auc:0.98889 evalualtionset-aucpr:0.85306
[91] trainingset-auc:1.00450 trainingset-aucpr:0.99735 evalualtionset-auc:0.98897 evalualtionset-aucpr:0.85393
[92] trainingset-auc:1.00561 trainingset-aucpr:0.99737 evalualtionset-auc:0.98945 evalualtionset-aucpr:0.85371
[93] trainingset-auc:1.00588 trainingset-aucpr:0.99738 evalualtionset-auc:0.98951 evalualtionset-aucpr:0.85383
[94] trainingset-auc:1.00602 trainingset-aucpr:0.99739 evalualtionset-auc:0.98959 evalualtionset-aucpr:0.85399
[95] trainingset-auc:1.00644 trainingset-aucpr:0.99741 evalualtionset-auc:0.98996 evalualtionset-aucpr:0.85467
[96] trainingset-auc:1.00653 trainingset-aucpr:0.99742 evalualtionset-auc:0.98993 evalualtionset-aucpr:0.85481
[97] trainingset-auc:1.00665 trainingset-aucpr:0.99744 evalualtionset-auc:0.99003 evalualtionset-aucpr:0.85595
[98] trainingset-auc:1.00708 trainingset-aucpr:0.99747 evalualtionset-auc:0.99050 evalualtionset-aucpr:0.85680
[99] trainingset-auc:1.00771 trainingset-aucpr:0.99747 evalualtionset-auc:0.99068 evalualtionset-aucpr:0.85696
[100] trainingset-auc:1.00809 trainingset-aucpr:0.99749 evalualtionset-auc:0.99089 evalualtionset-aucpr:0.85706
[101] trainingset-auc:1.00842 trainingset-aucpr:0.99751 evalualtionset-auc:0.99103 evalualtionset-aucpr:0.85717
[102] trainingset-auc:1.00890 trainingset-aucpr:0.99753 evalualtionset-auc:0.99124 evalualtionset-aucpr:0.85699
[103] trainingset-auc:1.00901 trainingset-aucpr:0.99753 evalualtionset-auc:0.99127 evalualtionset-aucpr:0.85706
[104] trainingset-auc:1.00912 trainingset-aucpr:0.99755 evalualtionset-auc:0.99134 evalualtionset-aucpr:0.85719
[105] trainingset-auc:1.00919 trainingset-aucpr:0.99757 evalualtionset-auc:0.99130 evalualtionset-aucpr:0.85749
[106] trainingset-auc:1.00960 trainingset-aucpr:0.99759 evalualtionset-auc:0.99147 evalualtionset-aucpr:0.85720
[107] trainingset-auc:1.01100 trainingset-aucpr:0.99760 evalualtionset-auc:0.99188 evalualtionset-aucpr:0.85763
[108] trainingset-auc:1.01279 trainingset-aucpr:0.99762 evalualtionset-auc:0.99284 evalualtionset-aucpr:0.85757
[109] trainingset-auc:1.01298 trainingset-aucpr:0.99763 evalualtionset-auc:0.99288 evalualtionset-aucpr:0.85746
[110] trainingset-auc:1.01380 trainingset-aucpr:0.99765 evalualtionset-auc:0.99343 evalualtionset-aucpr:0.85782
[111] trainingset-auc:1.01421 trainingset-aucpr:0.99765 evalualtionset-auc:0.99352 evalualtionset-aucpr:0.85784
[112] trainingset-auc:1.01439 trainingset-aucpr:0.99767 evalualtionset-auc:0.99359 evalualtionset-aucpr:0.85771
[113] trainingset-auc:1.01446 trainingset-aucpr:0.99768 evalualtionset-auc:0.99367 evalualtionset-aucpr:0.85838
[114] trainingset-auc:1.01457 trainingset-aucpr:0.99769 evalualtionset-auc:0.99372 evalualtionset-aucpr:0.85821
[115] trainingset-auc:1.01500 trainingset-aucpr:0.99770 evalualtionset-auc:0.99406 evalualtionset-aucpr:0.85837
[116] trainingset-auc:1.01590 trainingset-aucpr:0.99771 evalualtionset-auc:0.99432 evalualtionset-aucpr:0.85814
[117] trainingset-auc:1.01639 trainingset-aucpr:0.99772 evalualtionset-auc:0.99467 evalualtionset-aucpr:0.85830
[118] trainingset-auc:1.01688 trainingset-aucpr:0.99773 evalualtionset-auc:0.99487 evalualtionset-aucpr:0.85819
[119] trainingset-auc:1.01817 trainingset-aucpr:0.99775 evalualtionset-auc:0.99528 evalualtionset-aucpr:0.85877
[120] trainingset-auc:1.01843 trainingset-aucpr:0.99776 evalualtionset-auc:0.99534 evalualtionset-aucpr:0.85879
[121] trainingset-auc:1.01848 trainingset-aucpr:0.99777 evalualtionset-auc:0.99536 evalualtionset-aucpr:0.85867
[122] trainingset-auc:1.01858 trainingset-aucpr:0.99777 evalualtionset-auc:0.99538 evalualtionset-aucpr:0.85854
[123] trainingset-auc:1.01896 trainingset-aucpr:0.99778 evalualtionset-auc:0.99546 evalualtionset-aucpr:0.85856
[124] trainingset-auc:1.01913 trainingset-aucpr:0.99779 evalualtionset-auc:0.99560 evalualtionset-aucpr:0.85925
[125] trainingset-auc:1.01920 trainingset-aucpr:0.99779 evalualtionset-auc:0.99561 evalualtionset-aucpr:0.85938
[126] trainingset-auc:1.01927 trainingset-aucpr:0.99781 evalualtionset-auc:0.99572 evalualtionset-aucpr:0.86009
[127] trainingset-auc:1.01976 trainingset-aucpr:0.99782 evalualtionset-auc:0.99592 evalualtionset-aucpr:0.86018
[128] trainingset-auc:1.01990 trainingset-aucpr:0.99782 evalualtionset-auc:0.99596 evalualtionset-aucpr:0.86008
[129] trainingset-auc:1.02059 trainingset-aucpr:0.99784 evalualtionset-auc:0.99628 evalualtionset-aucpr:0.86020
[130] trainingset-auc:1.02077 trainingset-aucpr:0.99784 evalualtionset-auc:0.99634 evalualtionset-aucpr:0.86014
[131] trainingset-auc:1.02081 trainingset-aucpr:0.99785 evalualtionset-auc:0.99641 evalualtionset-aucpr:0.86084
[132] trainingset-auc:1.02129 trainingset-aucpr:0.99787 evalualtionset-auc:0.99677 evalualtionset-aucpr:0.86086
I'm trying to apply a custom python function on every frame of a video, and create the video with modified frames as output. My input video is a mkv file, with variable framerate, and I'd like to get the same thing as output, so one frame in the input matches one in the output at the exact same time.
I tried to use this example of ffmpeg-python. However, it seems that the timestamp info are lost in the pipes. The output video has 689 frames when the input only has 300 (the durations also aren't a match, with 27s vs 11s for the input).
I also tried to first process each frame in my video and save the transformed version as PNGs. Then I "masked" the input video with the processed frames. This seems to be better because the output video has the same 11s duration than the input, but the frame count doesn't match (313 vs 300).
Code for the python-ffmpeg solution:
width = 1920
height = 1080
process1 = (
ffmpeg
.input(in_filename)
.output('pipe:', format='rawvideo', pix_fmt='rgb24')
.run_async(pipe_stdout=True)
)
process2 = (
ffmpeg
.input('pipe:', format='rawvideo', pix_fmt='rgb24', s='{}x{}'.format(width, height))
.output(out_filename, pix_fmt='yuv420p')
.overwrite_output()
.run_async(pipe_stdin=True)
)
while True:
in_bytes = process1.stdout.read(width * height * 3)
if not in_bytes:
break
in_frame = (
np
.frombuffer(in_bytes, np.uint8)
.reshape([height, width, 3])
)
# Just add 1 to the pixels for the example
out_frame = in_frame + 1
process2.stdin.write(
out_frame
.astype(np.uint8)
.tobytes()
)
process2.stdin.close()
process1.wait()
process2.wait()
Code for the overlay solution:
ffmpeg -i in.mkv -i test/%d.png -filter_complex "[0][1]overlay=0:0" -copyts out.mkv
Is there any other solution I didn't think about to perform what I'm trying to do? It doesn't seem to be that complicated but I can't find a way to do it.
Thanks for any help!
UPDATE:
Here are the logs for the input and output pipes of the python-ffmpeg solution.
Input
Input #0, matroska,webm, from 'in.mkv':
Metadata:
ENCODER : Lavf59.17.100
Duration: 00:00:11.48, start: 0.000000, bitrate: 45702 kb/s
Stream #0:0: Video: h264 (High 4:4:4 Predictive), yuvj420p(pc, gbr/unknown/unknown, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 60 fps, 60 tbr, 1k tbn (default)
Metadata:
ENCODER : Lavc58.134.100 h264_nvenc
DURATION : 00:00:11.483000000
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> rawvideo (native))
Output #0, rawvideo, to 'pipe:':
Metadata:
encoder : Lavf59.17.100
Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24(pc, gbr/unknown/unknown, progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 2985984 kb/s, 60 fps, 60 tbn (default)
Metadata:
DURATION : 00:00:11.483000000
encoder : Lavc59.20.100 rawvideo
frame= 689 fps=154 q=-0.0 Lsize= 4185675kB time=00:00:11.48 bitrate=2985984.1kbits/s dup=389 drop=0 speed=2.57x
Output
Input #0, rawvideo, from 'pipe:':
Duration: N/A, start: 0.000000, bitrate: 1244160 kb/s
Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 1920x1080, 1244160 kb/s, 25 tbr, 25 tbn
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
[libx264 # 0000025afaf11140] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
[libx264 # 0000025afaf11140] profile High, level 4.0, 4:2:0, 8-bit
[libx264 # 0000025afaf11140] 264 - core 164 r3081 19856cc - H.264/MPEG-4 AVC codec - Copyleft 2003-2021 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=18 lookahead_threads=3 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, matroska, to 'images/videos/out.mkv':
Metadata:
encoder : Lavf59.17.100
Stream #0:0: Video: h264 (H264 / 0x34363248), yuv420p(tv, progressive), 1920x1080, q=2-31, 25 fps, 1k tbn
Metadata:
encoder : Lavc59.20.100 libx264
Side data:
cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
frame= 689 fps= 11 q=-0.0 Lsize= 4185675kB time=00:00:11.48 bitrate=2985984.1kbits/s dup=389 drop=0 speed=0.181x
video:4185675kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%
I'll answer my own question, as I've been able to solve the issue with the help of kesh in the comments.
There are basically two things:
vsync passthrough is required for the input video, to keep the number of frames
another external tool (MKVToolNix) has to be used twice to extract timestamps from the initial video and apply them to the output
Below is the relevant code to perform the whole operation using python and subprocess. You can use the following line on both input and output video to check that the timestamps are indeed the same for each frame: ffprobe -show_entries packet=pts_time,duration_time,stream_index video.mkv
width = 1920
height = 1080
process1 = (
ffmpeg
.input('in.mkv', vsync='passthrough')
.output('pipe:', format='rawvideo', pix_fmt='rgb24')
.run_async(pipe_stdout=True)
)
process2 = (
ffmpeg
.input('pipe:', format='rawvideo', pix_fmt='rgb24', s='{}x{}'.format(width, height))
.output('temp.mkv', pix_fmt='yuv420p')
.overwrite_output()
.run_async(pipe_stdin=True)
)
while True:
in_bytes = process1.stdout.read(width * height * 3)
if not in_bytes:
break
in_frame = (
np
.frombuffer(in_bytes, np.uint8)
.reshape([height, width, 3])
)
# Keep thing simple, just add 1 to each pixel
out_frame = in_frame + 1
process2.stdin.write(
out_np
.astype(np.uint8)
.tobytes()
)
process2.stdin.close()
process1.wait()
process2.wait()
# Extract timestamps from input video
subprocess.run(['mkvextract', 'in.mkv', 'timestamps_v2', '0:timestamps.txt'])
# Apply extracted timestamps to create synchronized output video
subprocess.run(['mkvmerge', '-o', 'out.mkv', '--timestamps', '0:timestamps.txt', 'temp.mkv'])
# Clean up
os.remove('temp.mkv')
os.remove('timestamps.txt')
I have 2 pythons on my mac (10.8.3): The default, and 2.7 version from homebrew.
So far, I could install modules and use them with my brew python. I installed mapnik with brew install mapnik (mapnik-2.1.0) and it compiled correctly. But, if I open python and enter import mapnik, the following error comes and python exits:
>>> import mapnik
Fatal Python error: PyThreadState_Get: no current thread
Abort trap: 6
Mac os x also shows an error window, with the following content:
Process: Python [60666]
Path: /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
Identifier: Python
Version: 2.7.3 (2.7.3)
Code Type: X86-64 (Native)
Parent Process: bash [60454]
User ID: 501
Date/Time: 2013-03-28 10:21:08.535 +0100
OS Version: Mac OS X 10.8.3 (12D78)
Report Version: 10
Interval Since Last Report: 128837 sec
Crashes Since Last Report: 5
Per-App Crashes Since Last Report: 4
Anonymous UUID: 567121E6-7BAC-335F-E3B0-DD24D1F9E6BA
Crashed Thread: 0 Dispatch queue: com.apple.main-thread
Exception Type: EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Application Specific Information:
abort() called
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 0x00007fff94500d46 __kill + 10
1 libsystem_c.dylib 0x00007fff92afddf0 abort + 177
2 org.python.python 0x0000000105a77eaa Py_FatalError + 49
3 org.python.python 0x0000000105a76370 PyThreadState_Get + 28
4 org.python.python 0x0000000105a71f16 Py_InitModule4_64 + 58
5 libboost_python-mt.dylib 0x0000000104153288 boost::python::detail::init_module(char const*, void (*)()) + 40
6 org.python.python 0x00000001029fa716 _PyImport_LoadDynamicModule + 150
7 org.python.python 0x00000001029fa428 import_submodule + 296
8 org.python.python 0x00000001029f9ff1 load_next + 268
9 org.python.python 0x00000001029f80fb PyImport_ImportModuleLevel + 794
10 org.python.python 0x00000001029de9d3 builtin___import__ + 132
11 org.python.python 0x000000010296e335 PyObject_Call + 101
12 org.python.python 0x00000001029e8e10 PyEval_CallObjectWithKeywords + 93
13 org.python.python 0x00000001029e61b6 PyEval_EvalFrameEx + 11727
14 org.python.python 0x00000001029e32bd PyEval_EvalCodeEx + 1638
15 org.python.python 0x00000001029e2c51 PyEval_EvalCode + 54
16 org.python.python 0x00000001029f6e58 PyImport_ExecCodeModuleEx + 247
17 org.python.python 0x00000001029f99bf load_source_module + 1053
18 org.python.python 0x00000001029f9c7e load_package + 334
19 org.python.python 0x00000001029fa428 import_submodule + 296
20 org.python.python 0x00000001029f9ff1 load_next + 268
21 org.python.python 0x00000001029f80fb PyImport_ImportModuleLevel + 794
22 org.python.python 0x00000001029de9d3 builtin___import__ + 132
23 org.python.python 0x000000010296e335 PyObject_Call + 101
24 org.python.python 0x00000001029e8e10 PyEval_CallObjectWithKeywords + 93
25 org.python.python 0x00000001029e61b6 PyEval_EvalFrameEx + 11727
26 org.python.python 0x00000001029e32bd PyEval_EvalCodeEx + 1638
27 org.python.python 0x00000001029e2c51 PyEval_EvalCode + 54
28 org.python.python 0x0000000102a01a2b run_mod + 53
29 org.python.python 0x0000000102a0184d PyRun_InteractiveOneFlags + 365
30 org.python.python 0x0000000102a01367 PyRun_InteractiveLoopFlags + 188
31 org.python.python 0x0000000102a01218 PyRun_AnyFileExFlags + 60
32 org.python.python 0x0000000102a120b2 Py_Main + 3210
33 libdyld.dylib 0x00007fff966bf7e1 start + 1
Thread 0 crashed with X86 Thread State (64-bit):
rax: 0x0000000000000000 rbx: 0x00007fff5d29c2b0 rcx: 0x00007fff5d29c298 rdx: 0x0000000000000000
rdi: 0x000000000000ecfa rsi: 0x0000000000000006 rbp: 0x00007fff5d29c2c0 rsp: 0x00007fff5d29c298
r8: 0x00000000000003f5 r9: 0x00007fff5d29c270 r10: 0x00007fff94502342 r11: 0x0000000000000206
r12: 0x00007fff5d29cb00 r13: 0x0000000102e9b656 r14: 0x00000000000003f5 r15: 0x0000000000000000
rip: 0x00007fff94500d46 rfl: 0x0000000000000206 cr2: 0x00007fff7bff7ff0
Logical CPU: 0
Binary Images:
0x102961000 - 0x102961fff +org.python.python (2.7.3 - 2.7.3) <62DA7BCA-2A0C-3753-A043-7459827F56D1> /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
0x102964000 - 0x102a6cfff +org.python.python (2.7.3, [c] 2004-2012 Python Software Foundation. - 2.7.3) <81E6FB4A-287C-37C3-A26D-D045B604933F> /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/Python
0x102c4c000 - 0x102c4efff +readline.so (0) <C4F1219C-CDFD-37D2-A762-22974D3F2918> /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/readline.so
0x102c54000 - 0x102c77fff +libreadline.6.2.dylib (6.2) <2131C2A4-E75D-3680-9C8D-E42D78A5E1B9> /usr/local/opt/readline/lib/libreadline.6.2.dylib
0x102cfc000 - 0x102d01fff +itertools.so (0) <F857A819-40A9-3F72-A0B1-3E97BA5F3DAA> /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/itertools.so
0x102d0a000 - 0x102f2cfff +_mapnik.so (0) <433DAD47-AB4D-37C5-A0EA-FE1134B63D3D> /usr/local/lib/python2.7/site-packages/mapnik/_mapnik.so
0x103603000 - 0x103978ff7 +libmapnik.dylib (0) <0E898678-9D86-35B4-A772-0248924F5BF3> /usr/local/Cellar/mapnik/2.1.0/lib/libmapnik.dylib
0x104138000 - 0x104170ff7 +libboost_python-mt.dylib (0) <9B5DD680-BEC9-3988-9F55-F59DA38CC175> /usr/local/lib/libboost_python-mt.dylib
0x1041b3000 - 0x1041c2ff7 +libboost_thread-mt.dylib (0) <24432300-9373-30B6-89F6-857CBEF9C105> /usr/local/lib/libboost_thread-mt.dylib
0x1041de000 - 0x10424dff7 +libfreetype.6.dylib (0) <490EDE42-5B78-34BA-85E7-D0547BE778E1> /usr/local/lib/libfreetype.6.dylib
0x104262000 - 0x104268fff +libltdl.7.dylib (0) <E437E59F-FC2E-3837-BDF8-301BEF13C7D6> /usr/local/lib/libltdl.7.dylib
0x10426c000 - 0x10428bff7 +libpng15.15.dylib (0) <52907DBF-A04C-325D-9E77-DF56A771026F> /usr/local/lib/libpng15.15.dylib
0x104293000 - 0x1042e6fff +libtiff.5.dylib (0) <AE85D7B1-F4E5-3B27-832D-5756EFCDE912> /usr/local/lib/libtiff.5.dylib
0x1042f2000 - 0x104326fff +libproj.0.dylib (0) <C799D9A7-0DDA-334D-9AEA-13A4AE7A67A2> /usr/local/lib/libproj.0.dylib
0x104333000 - 0x104441ff7 +libicuuc.50.1.dylib (0) <5FDBBF25-EEA6-3649-9ED6-B18CE89CE510> /usr/local/opt/icu4c/lib/libicuuc.50.1.dylib
0x1044a3000 - 0x1044b2fff +libboost_filesystem-mt.dylib (0) <B2C03485-5FA6-3744-BC99-DB4E87DA4D87> /usr/local/lib/libboost_filesystem-mt.dylib
0x1044c3000 - 0x1044c5ff7 +libboost_system-mt.dylib (0) <FE5E6E11-C7FB-3895-9976-526D22997EDC> /usr/local/lib/libboost_system-mt.dylib
0x1044ca000 - 0x104578fff +libboost_regex-mt.dylib (0) <BC1D7F46-F6AB-3964-B344-E692EFC5D3B7> /usr/local/lib/libboost_regex-mt.dylib
0x10461c000 - 0x10464bff7 +libjpeg.8.dylib (0) <9EC07360-CF18-3529-AE54-E60DBF1313DC> /usr/local/lib/libjpeg.8.dylib
0x104652000 - 0x105a24fff +libicudata.50.dylib (0) <B74D5789-7082-3084-9BF0-9A3AE6F2B046> /usr/local/opt/icu4c/lib/libicudata.50.dylib
0x105a26000 - 0x105b33fff org.python.python (2.7.2 - 2.7.2) <E7F3EED1-E55D-32AF-9649-77C814693F6A> /System/Library/Frameworks/Python.framework/Versions/2.7/Python
0x7fff62561000 - 0x7fff6259593f dyld (210.2.3) <36CAA36E-72BC-3E48-96D9-B96A2DF77730> /usr/lib/dyld
0x7fff8ae57000 - 0x7fff8ae7cff7 libc++abi.dylib (26) <D86169F3-9F31-377A-9AF3-DB17142052E4> /usr/lib/libc++abi.dylib
0x7fff8c50f000 - 0x7fff8c510ff7 libsystem_sandbox.dylib (220.2) <6838A6FD-8626-3356-BB4F-BB4787216207> /usr/lib/system/libsystem_sandbox.dylib
0x7fff8c511000 - 0x7fff8c523ff7 libz.1.dylib (43) <2A1551E8-A272-3DE5-B692-955974FE1416> /usr/lib/libz.1.dylib
0x7fff8d337000 - 0x7fff8d342fff libsystem_notify.dylib (98.5) <C49275CC-835A-3207-AFBA-8C01374927B6> /usr/lib/system/libsystem_notify.dylib
0x7fff8d9e6000 - 0x7fff8d9ebfff libcompiler_rt.dylib (30) <08F8731D-5961-39F1-AD00-4590321D24A9> /usr/lib/system/libcompiler_rt.dylib
0x7fff8dd72000 - 0x7fff8dd73fff libDiagnosticMessagesClient.dylib (8) <8548E0DC-0D2F-30B6-B045-FE8A038E76D8> /usr/lib/libDiagnosticMessagesClient.dylib
0x7fff902d7000 - 0x7fff90326ff7 libcorecrypto.dylib (106.2) <CE0C29A3-C420-339B-ADAA-52F4683233CC> /usr/lib/system/libcorecrypto.dylib
0x7fff9064f000 - 0x7fff90650fff libsystem_blocks.dylib (59) <D92DCBC3-541C-37BD-AADE-ACC75A0C59C8> /usr/lib/system/libsystem_blocks.dylib
0x7fff90a75000 - 0x7fff90a7dff7 libsystem_dnssd.dylib (379.37) <616FC901-151E-38BF-B2C4-24A351C5FAAD> /usr/lib/system/libsystem_dnssd.dylib
0x7fff914fd000 - 0x7fff9150afff libbz2.1.0.dylib (29) <CE9785E8-B535-3504-B392-82F0064D9AF2> /usr/lib/libbz2.1.0.dylib
0x7fff92899000 - 0x7fff928a7fff libcommonCrypto.dylib (60027) <BAAFE0C9-BB86-3CA7-88C0-E3CBA98DA06F> /usr/lib/system/libcommonCrypto.dylib
0x7fff92aa4000 - 0x7fff92b70ff7 libsystem_c.dylib (825.26) <4C9EB006-FE1F-3F8F-8074-DFD94CF2CE7B> /usr/lib/system/libsystem_c.dylib
0x7fff92bba000 - 0x7fff92bc0ff7 libunwind.dylib (35.1) <21703D36-2DAB-3D8B-8442-EAAB23C060D3> /usr/lib/system/libunwind.dylib
0x7fff92ed3000 - 0x7fff92f3cfff libstdc++.6.dylib (56) <EAA2B53E-EADE-39CF-A0EF-FB9D4940672A> /usr/lib/libstdc++.6.dylib
0x7fff93019000 - 0x7fff93065ff7 libauto.dylib (185.1) <73CDC482-16E3-3FC7-9BB4-FBA2DA44DBC2> /usr/lib/libauto.dylib
0x7fff930cf000 - 0x7fff930d5fff libmacho.dylib (829) <BF332AD9-E89F-387E-92A4-6E1AB74BD4D9> /usr/lib/system/libmacho.dylib
0x7fff93177000 - 0x7fff93199ff7 libxpc.dylib (140.42) <BBE558BD-5E55-35E4-89ED-1AA6B056D05A> /usr/lib/system/libxpc.dylib
0x7fff931f6000 - 0x7fff931f7ff7 libremovefile.dylib (23.2) <6763BC8E-18B8-3AD9-8FFA-B43713A7264F> /usr/lib/system/libremovefile.dylib
0x7fff931f8000 - 0x7fff93260ff7 libc++.1.dylib (65.1) <20E31B90-19B9-3C2A-A9EB-474E08F9FE05> /usr/lib/libc++.1.dylib
0x7fff9345d000 - 0x7fff9365dfff libicucore.A.dylib (491.11.2) <FD6282D8-DF3F-3842-8C2E-CF478D2B9669> /usr/lib/libicucore.A.dylib
0x7fff9365e000 - 0x7fff9365efff libkeymgr.dylib (25) <CC9E3394-BE16-397F-926B-E579B60EE429> /usr/lib/system/libkeymgr.dylib
0x7fff936d4000 - 0x7fff936dcfff liblaunch.dylib (442.26.2) <2F71CAF8-6524-329E-AC56-C506658B4C0C> /usr/lib/system/liblaunch.dylib
0x7fff9424d000 - 0x7fff9427bff7 libsystem_m.dylib (3022.6) <B434BE5C-25AB-3EBD-BAA7-5304B34E3441> /usr/lib/system/libsystem_m.dylib
0x7fff94284000 - 0x7fff94286fff libquarantine.dylib (52) <4BE2E642-A14F-340A-B482-5BD2AEFD9C24> /usr/lib/system/libquarantine.dylib
0x7fff94381000 - 0x7fff94382ff7 libdnsinfo.dylib (453.19) <14202FFB-C3CA-3FCC-94B0-14611BF8692D> /usr/lib/system/libdnsinfo.dylib
0x7fff944ef000 - 0x7fff9450aff7 libsystem_kernel.dylib (2050.22.13) <5A961E2A-CFB8-362B-BC43-122704AEB047> /usr/lib/system/libsystem_kernel.dylib
0x7fff94579000 - 0x7fff94587ff7 libsystem_network.dylib (77.10) <0D99F24E-56FE-380F-B81B-4A4C630EE587> /usr/lib/system/libsystem_network.dylib
0x7fff94821000 - 0x7fff9493992f libobjc.A.dylib (532.2) <90D31928-F48D-3E37-874F-220A51FD9E37> /usr/lib/libobjc.A.dylib
0x7fff9493a000 - 0x7fff9494fff7 libdispatch.dylib (228.23) <D26996BF-FC57-39EB-8829-F63585561E09> /usr/lib/system/libdispatch.dylib
0x7fff95c25000 - 0x7fff95d22ff7 libxml2.2.dylib (22.3) <47B09CB2-C636-3024-8B55-6040F7829B4C> /usr/lib/libxml2.2.dylib
0x7fff96350000 - 0x7fff96351ff7 libSystem.B.dylib (169.3) <9089D72D-E714-31E1-80C8-698A8E8B05AD> /usr/lib/libSystem.B.dylib
0x7fff96397000 - 0x7fff9639cfff libcache.dylib (57) <65187C6E-3FBF-3EB8-A1AA-389445E2984D> /usr/lib/system/libcache.dylib
0x7fff96646000 - 0x7fff9664dfff libcopyfile.dylib (89) <876573D0-E907-3566-A108-577EAD1B6182> /usr/lib/system/libcopyfile.dylib
0x7fff966bd000 - 0x7fff966c0ff7 libdyld.dylib (210.2.3) <F59367C9-C110-382B-A695-9035A6DD387E> /usr/lib/system/libdyld.dylib
0x7fff96961000 - 0x7fff96963ff7 libunc.dylib (25) <92805328-CD36-34FF-9436-571AB0485072> /usr/lib/system/libunc.dylib
0x7fff974ba000 - 0x7fff974f0fff libsystem_info.dylib (406.17) <4FFCA242-7F04-365F-87A6-D4EFB89503C1> /usr/lib/system/libsystem_info.dylib
0x7fff97850000 - 0x7fff97888fff libncurses.5.4.dylib (37.3) <68D5B5F5-8252-3F1E-AFF1-C6AFE145DBC1> /usr/lib/libncurses.5.4.dylib
0x7fff978d3000 - 0x7fff97abdff7 com.apple.CoreFoundation (6.8 - 744.18) <A60C3C9B-3764-3291-844C-C487ACF77C2C> /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
External Modification Summary:
Calls made by other processes targeting this process:
task_for_pid: 0
thread_create: 0
thread_set_state: 0
Calls made by this process:
task_for_pid: 0
thread_create: 0
thread_set_state: 0
Calls made by all processes on this machine:
task_for_pid: 18730
thread_create: 2
thread_set_state: 0
VM Region Summary:
ReadOnly portion of Libraries: Total=109.5M resident=53.3M(49%) swapped_out_or_unallocated=56.2M(51%)
Writable regions: Total=20.6M written=2644K(13%) resident=3256K(15%) swapped_out=0K(0%) unallocated=17.5M(85%)
REGION TYPE VIRTUAL
=========== =======
MALLOC 11.8M
MALLOC guard page 32K
STACK GUARD 56.0M
Stack 8192K
__DATA 2356K
__LINKEDIT 68.3M
__TEXT 41.2M
__UNICODE 544K
shared memory 12K
=========== =======
TOTAL 188.2M
Model: MacBookPro8,2, BootROM MBP81.0047.B27, 4 processors, Intel Core i7, 2 GHz, 8 GB, SMC 1.69f4
Graphics: Intel HD Graphics 3000, Intel HD Graphics 3000, Built-In, 512 MB
Graphics: AMD Radeon HD 6490M, AMD Radeon HD 6490M, PCIe, 256 MB
Memory Module: BANK 0/DIMM0, 4 GB, DDR3, 1333 MHz, 0x80CE, 0x4D34373142353237334348302D4348392020
Memory Module: BANK 1/DIMM0, 4 GB, DDR3, 1333 MHz, 0x80CE, 0x4D34373142353237334348302D4348392020
AirPort: spairport_wireless_card_type_airport_extreme (0x14E4, 0xD6), Broadcom BCM43xx 1.0 (5.106.98.100.16)
Bluetooth: Version 4.1.3f3 11349, 2 service, 11 devices, 1 incoming serial ports
Network Service: Wi-Fi, AirPort, en1
Serial ATA Device: ST9500420AS, 500.11 GB
Serial ATA Device: MATSHITADVD-R UJ-898
USB Device: FaceTime HD Camera (Built-in), apple_vendor_id, 0x8509, 0xfa200000 / 3
USB Device: hub_device, 0x0424 (SMSC), 0x2513, 0xfa100000 / 2
USB Device: Apple Internal Keyboard / Trackpad, apple_vendor_id, 0x0246, 0xfa120000 / 5
USB Device: BRCM2070 Hub, 0x0a5c (Broadcom Corp.), 0x4500, 0xfa110000 / 4
USB Device: Bluetooth USB Host Controller, apple_vendor_id, 0x821a, 0xfa113000 / 8
USB Device: hub_device, 0x0424 (SMSC), 0x2513, 0xfd100000 / 2
USB Device: IR Receiver, apple_vendor_id, 0x8242, 0xfd110000 / 3
Any problems with linking my pythons? how can I fix it? thanks!
It looks like you are running the homebrew python but either boost python or mapnik's python bindings ended up linking against the system python provided by apple. If a full clean and reinstall of boost and Mapnik does not fix this then I recommend stopping by #mapnik irc on freenode for help debugging. Generally to fix you need to call otool -L on boost_python.dylib and or on site_packages/mapnik/_mapnik.so to see which ended up with a linkage to the system python. Then you can fix by using install_name_tool.
I encountered the same problem between python27-apple and python27 (installed by Macport). I tried these steps, and it worked for me.
According to #Evert's comments of different libraries used by different python, I tried to revert the python27-apple to be "active".
sudo port select --list python
For my computer, it showed:
Available versions for python:
none
python25-apple
python26
python26-apple
python27 (active)
python27-apple
in which python27(by Macport) was currently active. Then,
sudo port select python python27-apple
to make python27-apple active as the default lib were called previously. To double check,
sudo port select --list python
It now showed:
Available versions for python:
none
python25-apple
python26
python26-apple
python27
python27-apple (active)
And the python crash had gone when I ran the scripts.
You can use the homebrew python like this:
/usr/bin/python2.7 yourscript.py
I face the similar problem in using androguard. Though it's not quite the same, it can be a reference. My crash also is Fatal Python error: PyThreadState_Get: no current thread
Abort trap: 6. Run otool -L <lib>, suggest the dylib link to the system python.
Try solution:
python-config --ldflags
Copy the output to corresponding MakeFile. For example change LDFLAGS += -lm -lbz2 -lz -llzma -lsnappy -lmuparser -lpython to LDFLAGS += -lm -lbz2 -lz -llzma -lsnappy -lmuparser -L/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config -lpython2.7 -ldl -framework CoreFoundation.
make clean, then make again.
Now the dylib is OK, and no error anymore.
Alternatively ...
... when you use otool -L <lib> to find the wrongly linked python dylib you can use install_name_tool to change the symbol like:
install_name_tool -change /System/Library/Frameworks/Python.framework/Versions/2.7/Python /usr/local/Cellar/python/2.7.6_1/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib <lib>