We Keep Coding

Troubles when trying to build and sign a transaction with web3

I'm watching the tutorial of freeCodeCamp.org,
Solidity, Blockchain, and Smart Contract Course – Beginner to Expert Python Tutorial (link of the video course) and I'm stucked at 04:06:47 because when I try to build a transaction and sign it, my compiler gives me back a lot of errors:
INFORMAZIONI: impossibile trovare file corrispondenti ai criteri di
ricerca indicati. Traceback (most recent call last): File
"C:\Users\giuse\OneDrive\Desktop\Sol\web3_py_simple_storage\deploy.py",
line 44, in
transaction = SimpleStorage.constructor().buildTransaction( File "C:\Users\giuse\AppData\Local\Programs\Python\Python39\lib\site-packages\eth_utils\decorators.py",
line 18, in _wrapper
return self.method(obj, *args, **kwargs) File "C:\Users\giuse\AppData\Local\Programs\Python\Python39\lib\site-packages\web3\contract.py",
line 684, in buildTransaction
return fill_transaction_defaults(self.web3, built_transaction) File "cytoolz/functoolz.pyx", line 250, in
cytoolz.functoolz.curry.call
return self.func(*args, **kwargs) File "C:\Users\giuse\AppData\Local\Programs\Python\Python39\lib\site-packages\web3_utils\transactions.py",
line 121, in fill_transaction_defaults
default_val = default_getter(web3, transaction) File "C:\Users\giuse\AppData\Local\Programs\Python\Python39\lib\site-packages\web3_utils\transactions.py",
line 67, in
'gas': lambda web3, tx: web3.eth.estimate_gas(tx), File "C:\Users\giuse\AppData\Local\Programs\Python\Python39\lib\site-packages\web3\eth.py",
line 759, in estimate_gas
return self._estimate_gas(transaction, block_identifier) File "C:\Users\giuse\AppData\Local\Programs\Python\Python39\lib\site-packages\web3\module.py",
line 57, in caller
result = w3.manager.request_blocking(method_str, File "C:\Users\giuse\AppData\Local\Programs\Python\Python39\lib\site-packages\web3\manager.py",
line 197, in request_blocking
response = self._make_request(method, params) File "C:\Users\giuse\AppData\Local\Programs\Python\Python39\lib\site-packages\web3\manager.py",
line 150, in _make_request
return request_func(method, params) File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.call
return self.func(*args, **kwargs) File "C:\Users\giuse\AppData\Local\Programs\Python\Python39\lib\site-packages\web3\middleware\formatting.py",
line 76, in apply_formatters
response = make_request(method, params) File "C:\Users\giuse\AppData\Local\Programs\Python\Python39\lib\site-packages\web3\middleware\gas_price_strategy.py",
line 90, in middleware
return make_request(method, params) File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.call
return self.func(*args, **kwargs) File "C:\Users\giuse\AppData\Local\Programs\Python\Python39\lib\site-packages\web3\middleware\formatting.py",
line 74, in apply_formatters
response = make_request(method, formatted_params) File "C:\Users\giuse\AppData\Local\Programs\Python\Python39\lib\site-packages\web3\middleware\attrdict.py",
line 33, in middleware
response = make_request(method, params) File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.call
return self.func(*args, **kwargs) File "C:\Users\giuse\AppData\Local\Programs\Python\Python39\lib\site-packages\web3\middleware\formatting.py",
line 74, in apply_formatters
response = make_request(method, formatted_params) File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.call
return self.func(*args, **kwargs) File "C:\Users\giuse\AppData\Local\Programs\Python\Python39\lib\site-packages\web3\middleware\formatting.py",
line 73, in apply_formatters
formatted_params = formatter(params) File "cytoolz/functoolz.pyx", line 503, in
cytoolz.functoolz.Compose.call
ret = PyObject_Call(self.first, args, kwargs) File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.call
return self.func(*args, **kwargs) File "C:\Users\giuse\AppData\Local\Programs\Python\Python39\lib\site-packages\eth_utils\decorators.py",
line 91, in wrapper
return ReturnType(result) # type: ignore File "C:\Users\giuse\AppData\Local\Programs\Python\Python39\lib\site-packages\eth_utils\applicators.py",
line 22, in apply_formatter_at_index
yield formatter(item) File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.call
return self.func(*args, **kwargs) File "C:\Users\giuse\AppData\Local\Programs\Python\Python39\lib\site-packages\eth_utils\functional.py",
line 45, in inner
return callback(fn(*args, **kwargs)) File "C:\Users\giuse\AppData\Local\Programs\Python\Python39\lib\site-packages\eth_utils\applicators.py",
line 84, in apply_formatters_to_dict
yield key, formatterskey File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.call
return self.func(*args, **kwargs) File "C:\Users\giuse\AppData\Local\Programs\Python\Python39\lib\site-packages\eth_utils\applicators.py",
line 72, in apply_formatter_if
return formatter(value) File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.call
return self.func(*args, **kwargs) File "C:\Users\giuse\AppData\Local\Programs\Python\Python39\lib\site-packages\web3\middleware\validation.py",
line 57, in validate_chain_id
raise ValidationError( web3.exceptions.ValidationError: The transaction declared chain ID 5777, but the connected node is on 1337
I first searched on internet for solutions, but the only one was to add in the transation build one more parameter (the one of the gasPrice), but it didn't solve my problems.
I hope someone can help me, here is the full code I wrote:
from solcx import compile_standard, install_solc
import json
from web3 import Web3
from dotenv import load_dotenv
import os
load_dotenv()
install_solc("0.6.0")
with open("./simpleStorage.sol", "r") as file:
simple_storage_file = file.read()
compiled_sol = compile_standard(
{
"language": "Solidity",
"sources": {"simpleStorage.sol": {"content": simple_storage_file}},
"settings": {
"outputSelection": {
"*": {"*": ["abi", "metadata", "evm.bytecode", "evm.sourceMap"]}
}
},
},
solc_version="0.6.0",
)
with open("compiledCode.json", "w") as file:
json.dump(compiled_sol, file)
bytecode = compiled_sol["contracts"]["simpleStorage.sol"]["SimpleStorage"]["evm"][
"bytecode"
]["object"]
abi = compiled_sol["contracts"]["simpleStorage.sol"]["SimpleStorage"]["abi"]
w3 = Web3(Web3.HTTPProvider("HTTP://127.0.0.1:7545"))
chain_id = 5777
my_address = "0xd8BADAe3766759e7e298931dF01F452616dc6dde"
pvt_key = os.getenv("PRIVATE_KEY")
SimpleStorage = w3.eth.contract(abi=abi, bytecode=bytecode)
nonce = w3.eth.getTransactionCount(my_address)
transaction = SimpleStorage.constructor().buildTransaction(
{
"chainId": chain_id,
"gasPrice": w3.eth.gas_price,
"from": my_address,
"nonce": nonce,
},
)
signed_txn = w3.eth.sign_transaction(transaction, private_key=pvt_key)

Try changing your chain_id = 5777 to chain_id = 1337.

If you are using ganache quickstart, instead use New Workspace option so you can save it.
Go to settings on top right corner (If you are using GUI from tutorial video)
Go to server tab and select following values and save.
Change values accordingly in your code.
Also change create trasaction code to this
transaction = SimpleStorage.constructor().buildTransaction(
{
"chainId": chain_id,
"gasPrice": w3.eth.gas_price,
"from": my_address,
"nonce": nonce,
}
)
Doing these 2 steps should fix your problem. I just did it for myself.

jsonb join not working properly in sqlalchemy

I have a query that joins on a jsonb type column in postgres that I want to convert to sqlalchemy in django using the aldjemy package
SELECT anon_1.key AS tag, count(anon_1.value ->> 'polarity') AS count_1, anon_1.value ->> 'polarity' AS anon_2
FROM feedback f
JOIN tagging t ON t.feedback_id = f.id
JOIN jsonb_each(t.json_content -> 'entityMap') AS anon_3 ON true
JOIN jsonb_each(((anon_3.value -> 'data') - 'selectionState') - 'segment') AS anon_1 ON true
where f.id = 2
GROUP BY anon_1.value ->> 'polarity', anon_1.key;
The json_content field stores data in the following format:
{
"entityMap":
{
"0":
{
"data":
{
"people":
{
"labelId": 5,
"polarity": "positive"
},
"segment": "a small segment",
"selectionState":
{
"focusKey": "9xrre",
"hasFocus": true,
"anchorKey": "9xrre",
"isBackward": false,
"focusOffset": 75,
"anchorOffset": 3
}
},
"type": "TAG",
"mutability": "IMMUTABLE"
},
"1":
{
"data":
{
"product":
{
"labelId": 6,
"polarity": "positive"
},
"segment": "another segment",
"selectionState":
{
"focusKey": "9xrre",
"hasFocus": true,
"anchorKey": "9xrre",
"isBackward": false,
"focusOffset": 138,
"anchorOffset": 79
}
},
"type": "TAG",
"mutability": "IMMUTABLE"
}
}
}
I wrote the following sqlalchemy code to achieve the query
first_alias = aliased(func.jsonb_each(Tagging.sa.json_content["entityMap"]))
print(first_alias)
second_alias = aliased(
func.jsonb_each(
first_alias.c.value.op("->")("data")
.op("-")("selectionState")
.op("-")("segment")
)
)
polarity = second_alias.c.value.op("->>")("polarity")
p_tag = second_alias.c.key
_count = (
Feedback.sa.query()
.join(
CampaignQuestion,
CampaignQuestion.sa.question_id == Feedback.sa.question_id,
isouter=True,
)
.join(Tagging)
.join(first_alias, true())
.join(second_alias, true())
.filter(CampaignQuestion.sa.campaign_id == campaign_id)
.with_entities(p_tag.label("p_tag"), func.count(polarity), polarity)
.group_by(polarity, p_tag)
.all()
)
print(_count)
but it is giving me a NotImplementedError: Operator 'getitem' is not supported on this expression error on accessing first_alias.c
the stack trace:
Traceback (most recent call last):
File "/home/.cache/pypoetry/virtualenvs/api-FPSaTdE5-py3.8/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
response = handler(request, *args, **kwargs)
File "/home/work/api/app/campaign/views.py", line 119, in results_p_tags
d = campaign_service.get_p_tag_count_for_campaign_results(id)
File "/home/work/api/app/campaign/services/campaign.py", line 177, in get_p_tag_count_for_campaign_results
return campaign_selectors.get_p_tag_counts_for_campaign(campaign_id)
File "/home/work/api/app/campaign/selectors.py", line 196, in get_p_tag_counts_for_campaign
polarity = second_alias.c.value.op("->>")("polarity")
File "/home/.cache/pypoetry/virtualenvs/api-FPSaTdE5-py3.8/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 1093, in __get__
obj.__dict__[self.__name__] = result = self.fget(obj)
File "/home/.cache/pypoetry/virtualenvs/api-FPSaTdE5-py3.8/lib/python3.8/site-packages/sqlalchemy/sql/selectable.py", line 746, in columns
self._populate_column_collection()
File "/home/.cache/pypoetry/virtualenvs/api-FPSaTdE5-py3.8/lib/python3.8/site-packages/sqlalchemy/sql/selectable.py", line 1617, in _populate_column_collection
self.element._generate_fromclause_column_proxies(self)
File "/home/.cache/pypoetry/virtualenvs/api-FPSaTdE5-py3.8/lib/python3.8/site-packages/sqlalchemy/sql/selectable.py", line 703, in _generate_fromclause_column_proxies
fromclause._columns._populate_separate_keys(
File "/home/.cache/pypoetry/virtualenvs/api-FPSaTdE5-py3.8/lib/python3.8/site-packages/sqlalchemy/sql/base.py", line 1216, in _populate_separate_keys
self._colset.update(c for k, c in self._collection)
File "/home/.cache/pypoetry/virtualenvs/api-FPSaTdE5-py3.8/lib/python3.8/site-packages/sqlalchemy/sql/base.py", line 1216, in <genexpr>
self._colset.update(c for k, c in self._collection)
File "/home/.cache/pypoetry/virtualenvs/api-FPSaTdE5-py3.8/lib/python3.8/site-packages/sqlalchemy/sql/operators.py", line 434, in __getitem__
return self.operate(getitem, index)
File "/home/.cache/pypoetry/virtualenvs/api-FPSaTdE5-py3.8/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 831, in operate
return op(self.comparator, *other, **kwargs)
File "/home/.cache/pypoetry/virtualenvs/api-FPSaTdE5-py3.8/lib/python3.8/site-packages/sqlalchemy/sql/operators.py", line 434, in __getitem__
return self.operate(getitem, index)
File "/home/.cache/pypoetry/virtualenvs/api-FPSaTdE5-py3.8/lib/python3.8/site-packages/sqlalchemy/sql/type_api.py", line 75, in operate
return o[0](self.expr, op, *(other + o[1:]), **kwargs)
File "/home/.cache/pypoetry/virtualenvs/api-FPSaTdE5-py3.8/lib/python3.8/site-packages/sqlalchemy/sql/default_comparator.py", line 173, in _getitem_impl
_unsupported_impl(expr, op, other, **kw)
File "/home/.cache/pypoetry/virtualenvs/api-FPSaTdE5-py3.8/lib/python3.8/site-packages/sqlalchemy/sql/default_comparator.py", line 177, in _unsupported_impl
raise NotImplementedError(
NotImplementedError: Operator 'getitem' is not supported on this expression
Any help would be greatly appreciated
PS: The sqlalchemy version I'm using for this is 1.4.6
I used the same sqlalchmy query expression before in a flask project using sqlalchemy version 1.3.22 and it was working correctly

Fixed the issue by using table_valued functions as mentioned in the docs,
and accessing the ColumnCollection of the function using indices instead of keys. Code is as follows:
first_alias = func.jsonb_each(Tagging.sa.json_content["entityMap"]).table_valued(
"key", "value"
)
second_alias = func.jsonb_each(
first_alias.c[1].op("->")("data").op("-")("selectionState").op("-")("segment")
).table_valued("key", "value")
polarity = second_alias.c[1].op("->>")("polarity")
p_tag = second_alias.c[0]

From a single JSON create and insert multiple rows to BigQuery with Pub/Sub and Dataflow

I have created a Beam Dataflow pipeline that parses a single JSON from a PubSub topic:
{
"data": "test data",
"options": {
"test options": "test",
"test_units": {
"test": {
"test1": "test1",
"test2": "test2"
},
"test2": {
"test1": "test1",
"test2": "test2"
},
"test3": {
"test1": "test1",
"test2": "test2"
}
}
}
}
My output is something like this:
{
"data": "test data",
"test_test_unit": "test1",
"test_test_unit": "test2",
"test1_test_unit": "test1",
...
},
{
"data": "test data",
"test_test_unit": "test1",
"test_test_unit": "test2",
"test1_test_unit": "test1",
...
}
Basically what I'm doing is flattening the data based on how many test_units are in the JSON from the PubSub and returning that many rows in a single dict.
I have created a Class to flatten the data which returns a dict of rows.
Here is my Beam pipeline:
lines = ( p | 'Read from PubSub' >> beam.io.ReadStringsFromPubSub(known_args.input_topic)
| 'Parse data' >> beam.DoFn(parse_pubsub())
| 'Write to BigQuery' >> beam.io.WriteToBigQuery(
known_args.output_table,
schema=table_schema,
create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED
)
)
Here is some of the class to handle the flattening:
class parse_pubsub(beam.DoFn):
def process(self, element):
# ...
# flattens the data
# ...
return rows
Here is the error from the Stackdriver logs:
Error processing instruction -138. Original traceback is Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker.py", line 151, in _execute
response = task() File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker.py",
line 186, in <lambda> self._execute(lambda: worker.do_instruction(work), work) File "/usr/local/lib/python2.7/
dist-packages/apache_beam/runners/worker/sdk_worker.py", line 265, in do_instruction request.instruction_id)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/worker/sdk_worker.py", line 281, in
process_bundle delayed_applications = bundle_processor.process_bundle(instruction_id) File "/usr/local/lib/
python2.7/dist-packages/apache_beam/runners/worker/bundle_processor.py", line 552, in process_bundle op.finish()
File "apache_beam/runners/worker/operations.py", line 549, in
apache_beam.runners.worker.operations.DoOperation.finish def finish(self): File "apache_beam/runners/worker/
operations.py", line 550, in apache_beam.runners.worker.operations.DoOperation.finish with
self.scoped_finish_state: File "apache_beam/runners/worker/operations.py", line 551, in
apache_beam.runners.worker.operations.DoOperation.finish self.dofn_runner.finish() File "apache_beam/runners/
common.py", line 758, in apache_beam.runners.common.DoFnRunner.finish self._invoke_bundle_method
(self.do_fn_invoker.invoke_finish_bundle) File "apache_beam/runners/common.py", line 752, in
apache_beam.runners.common.DoFnRunner._invoke_bundle_method self._reraise_augmented(exn) File "apache_beam/
runners/common.py", line 777, in apache_beam.runners.common.DoFnRunner._reraise_augmented raise_with_traceback
(new_exn) File "apache_beam/runners/common.py", line 750, in
apache_beam.runners.common.DoFnRunner._invoke_bundle_method bundle_method() File "apache_beam/runners/common.py",
line 361, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle def invoke_finish_bundle(self): File
"apache_beam/runners/common.py", line 365, in apache_beam.runners.common.DoFnInvoker.invoke_finish_bundle
self.signature.finish_bundle_method.method_value()) File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/
gcp/bigquery.py", line 630, in finish_bundle self._flush_batch() File "/usr/local/lib/python2.7/dist-packages/
apache_beam/io/gcp/bigquery.py", line 637, in _flush_batch table_id=self.table_id, rows=self._rows_buffer) File
# HERE:
"/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/bigquery_tools.py",
line 611, in insert_rows for k, v in iteritems(row): File "/usr/local/lib/python2.7/dist-packages/future/utils/
__init__.py", line 308, in iteritems func = obj.items AttributeError: 'int' object has no attribute 'items'
[while running 'generatedPtransform-135']
I've also tried returning a list and had the same error that 'list' object has no 'items' therefore I'm converting the list rows to a dict like this:
0 {
"data": "test data",
"test_test_unit": "test1",
"test_test_unit": "test2",
"test1_test_unit": "test1",
...
},
1 {
"data": "test data",
"test_test_unit": "test1",
"test_test_unit": "test2",
"test1_test_unit": "test1",
...
}
I'm fairly new to this so any help will be appreciated!

You'll need to use the yield keyword to emit multiple outputs in your DoFn. For example:
class parse_pubsub(beam.DoFn):
def process(self, element):
# ...
# flattens the data
# ...
for row in rows:
yield row

How to fix ValueError: Expecting property name: line 4 column 1 (char 43)

when I tr to run python manage.py runserver code is giving error . And its traceback is strange ,
I tried
JSON ValueError: Expecting property name: line 1 column 2 (char 1)
and all similar questions but didn't get what exactly I am facing.
Traceback (most recent call last):
File "manage.py", line 22, in <module>
execute_from_command_line(sys.argv)
File "/home/tousif/.local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
utility.execute()
File "/home/tousif/.local/lib/python2.7/site-packages/django/core/management/__init__.py", line 308, in execute
settings.INSTALLED_APPS
File "/home/tousif/.local/lib/python2.7/site-packages/django/conf/__init__.py", line 56, in __getattr__
self._setup(name)
File "/home/tousif/.local/lib/python2.7/site-packages/django/conf/__init__.py", line 41, in _setup
self._wrapped = Settings(settings_module)
File "/home/tousif/.local/lib/python2.7/site-packages/django/conf/__init__.py", line 110, in __init__
mod = importlib.import_module(self.SETTINGS_MODULE)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/home/tousif/Desktop/ITP/ITP/itpcrm/itpcrm/settings.py", line 55, in <module>
cfg = json.loads(open('/home/tousif/Desktop/ITP/ITP/itpcrm/config.json', 'r').read())
File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 380, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Expecting property name: line 4 column 1 (char 43)
my config.json file which containt credentials etc (I have changed credentials to post here ) .And got this file from live server where its working fine but on local its giving this error.
{
"dev": {
"db": {
​
"ENGINE": "django.db.backends.mysql",
"NAME": "itpcrm",
"USER": "root",
"PASSWORD": "password",
"HOST": "localhost",
"PORT": "3306"
},
"jwt_key": "GRESDFwef3452fwefer",
"voice_api_url": "http://192.112.255.32:9040",
"voice_api_key": "3123",
"auth_api_key": "379h4f73f",
"provisioner_api_key": "abc",
"quote_approval_url": "http://192.112.255.145:9998/quotes/customer-approval?token=",
"docusign_base_url": "https://demo.docusign.net/restapi",
"docusign_integrator_key": "8a256bde-405b",
"docusign_oauth_base_url": "account-d.docusign.com",
"docusign_redirect_uri": "http://192.112.255.145:9998/api/callbacks/docusign",
"docusign_private_key_filename": "/home/itp/docusign-examples/keys/docusign_private_key.txt",
"docusign_user_id": "7f2444f-ae99-54922fec68f6",
"docusign_user_name": "dor.com"
},
"prod": {
"db": {
​
"ENGINE": "django.db.backends.mysql",
"NAME": "itp",
"USER": "it",
"PASSWORD": "password",
"HOST": "192.168.3.111",
"PORT": "3306"
},
"jwt_key": "rRregrgERg54g564heRGRfdsger",
"voice_api_url": "https://api.crm.itpscorp.com/itpvoice",
"voice_api_key": "abc1",
"auth_api_key": "379h4f73f3279fy927yf928oowqofabdbf",
"provisioner_api_key": "abc123123",
"quote_approval_url": "http://192.112.255.145:9998/quotes/customer-approval?token=",
"docusign_base_url": "https://demo.docusign.net/restapi",
"docusign_integrator_key": "8a256bde-405b-4032-bf24-be0245631f03",
"docusign_oauth_base_url": "account-d.docusign.com",
"docusign_redirect_uri": "http://192.112.255.145:9998/api/callbacks/docusign",
"docusign_private_key_filename": "/home/itp/docusign-examples/keys/docusign_private_key.txt",
"docusign_user_id": "7f26f6bb-8a39-444f-ae99-54922fec68f6",
"docusign_user_name": "docusign#itpfiber.com"
},
"mode": "dev"
}

The empty lines after "db" starts with the unicode codepoint 0x200B ('ZERO WIDTH SPACE'). That is what trips up the JSON decoder.
I copied the text into gvim and made a screenshot. See below.
Remove those characters (or the whole line) and it works...
(Looking at the JSON file with a hex editor would also show the problem clearly.)
If you look closely at the error message, you can that this correctly identifies the problem:
ValueError: Expecting property name: line 4 column 1 (char 43)
The moral of this story: look out for whitespace codepoints.

OverflowError: MongoDB can only handle up to 8-byte ints?

I have spent the last 12 hours scouring the web. I am completely lost, please help.
I am trying to pull data from an API endpoint and put it into MongoDB. The data looks like this:
{"_links": {
"self": {
"href": "https://us.api.battle.net/data/sc2/ladder/271302?namespace=prod"
}
},
"league": {
"league_key": {
"league_id": 5,
"season_id": 37,
"queue_id": 201,
"team_type": 0
},
"key": {
"href": "https://us.api.battle.net/data/sc2/league/37/201/0/5?namespace=prod"
}
},
"team": [
{
"id": 6956151645604413000,
"rating": 5321,
"wins": 131,
"losses": 64,
"ties": 0,
"points": 1601,
"longest_win_streak": 15,
"current_win_streak": 4,
"current_rank": 1,
"highest_rank": 10,
"previous_rank": 1,
"join_time_stamp": 1534903699,
"last_played_time_stamp": 1537822019,
"member": [
{
"legacy_link": {
"id": 9964871,
"realm": 1,
"name": "mTOR#378",
"path": "/profile/9964871/1/mTOR"
},
"played_race_count": [
{
"race": "Zerg",
"count": 195
}
],
"character_link": {
"id": 9964871,
"battle_tag": "Hellghost#11903",
"key": {
"href": "https://us.api.battle.net/data/sc2/character/Hellghost-11903/9964871?namespace=prod"
}
}
}
]
},
{
"id": 11611747760398664000, .....
....
Here's the code:
for ladder_number in ladder_array:
ladder_call_url = ladder_call+slash+str(ladder_number)+eng_locale+access_token
url = str(ladder_call_url)
response = requests.get(url)
print('trying ladder number '+str(ladder_number))
print('calling :'+url)
if response.status_code == 200:
print('status: '+str(response))
mmr_db.ladders.insert_one(response.json())
I get an error:
OverflowError: MongoDB can only handle up to 8-byte ints?
Is this because the data I am trying to load is too large? Are the "ID" integers too large?
Oh man, any help would be sincerely appreciated.
_______ EDIT ____________
Edited to include the Traceback:
Traceback (most recent call last):
File "C:\scripts\mmr_from_ladders.py", line 96, in <module>
mmr_db.ladders.insert_one(response.json(), bypass_document_validation=True)
File "C:\Users\me\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pymongo\collection.py", line 693, in insert_one
session=session),
File "C:\Users\me\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pymongo\collection.py", line 607, in _insert
bypass_doc_val, session)
File "C:\Users\me\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pymongo\collection.py", line 595, in _insert_one
acknowledged, _insert_command, session)
File "C:\Users\me\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pymongo\mongo_client.py", line 1243, in _retryable_write
return self._retry_with_session(retryable, func, s, None)
File "C:\Users\me\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pymongo\mongo_client.py", line 1196, in _retry_with_session
return func(session, sock_info, retryable)
File "C:\Users\me\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pymongo\collection.py", line 590, in _insert_command
retryable_write=retryable_write)
File "C:\Users\me\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pymongo\pool.py", line 584, in command
self._raise_connection_failure(error)
File "C:\Users\me\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pymongo\pool.py", line 745, in _raise_connection_failure
raise error
File "C:\Users\me\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pymongo\pool.py", line 579, in command
unacknowledged=unacknowledged)
File "C:\Users\me\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pymongo\network.py", line 114, in command
codec_options, ctx=compression_ctx)
File "C:\Users\me\AppData\Local\Programs\Python\Python37-32\lib\site-packages\pymongo\message.py", line 679, in _op_msg
flags, command, identifier, docs, check_keys, opts)
OverflowError: MongoDB can only handle up to 8-byte ints

The BSON spec — MongoDB’s native binary extended JSON format / data type — only supports 32 bit (signed) and 64 bit (signed) integers — 8 bytes being 64 bits.
The maximum integer value that can be stored in a 64 bit int is:
9,223,372,036,854,775,807
In your example you appear to have larger ids, for example:
11,611,747,760,398,664,000
I’m guessing that the app generating this data is using uint64 types (unsigned can hold x2-1 values).
I would start by looking at either of these potential solutions, if possible:
Changing the other side to use int64 (signed) types for the IDs.
Replacing the incoming IDs using ObjectId() as you then get a 12 byte ~ GUID for your unique IDs.