I am trying to use a JSON file in Mapbox studio for network analysis but it gives me an error:
Input failed. "type" member required on line 1.
The representative sample of JSON file is:
"version": 0.6,
"generator": "Overpass API 0.7.56.2 b688b00f",
"osm3s": {
"timestamp_osm_base": "2020-03-27T11:58:01Z",
"copyright": "The data included in this document is from www.openstreetmap.org. The data is made available under ODbL."
},
"elements": [
{
"type": "node",
"id": 123458059,
"lat": -38.3344495,
"lon": 143.5394486
},
{
"type": "node",
"id": 123458066,
"lat": -38.3394461,
"lon": 143.5923655,
"tags": {
"crossing": "traffic_signals",
"highway": "traffic_signals"
}
},
{
"type": "way",
"id": 769574290,
"nodes": [
7183581936,
681081177,
1561328098,
1539139562,
448021781
],
"tags": {
"highway": "trunk",
"lanes": "2",
"maxspeed": "80",
"name": "Princes Highway",
"ref": "A1",
"source:maxspeed:sign": "mapillary"
}
},
{
"type": "way",
"id": 776227225,
"nodes": [
1017428185,
317738200
],
"tags": {
"alt_name": "Princes Highway",
"highway": "trunk",
"lanes": "2",
"maxspeed": "50",
"name": "Murray Street",
"ref": "A1",
"source:maxspeed:sign": "OpenStreetCam",
"source:name": "services.land.vic.gov.au"
}
}
]
}
Does the error occur because of the specification of the format? Do we need to reformat the features or types?
To upload data to Mapbox you will need to convert your JSON file to GeoJSON, a subset of the JSON format. For example:
{"type": "FeatureCollection",
"features": [
{
"geometry": {
"type": "Point",
"coordinates": [
-76.9750541388,
38.8410857803
]
},
"type": "Feature",
"properties": {
"description": "Southern Ave",
"marker-symbol": "rail-metro",
"title": "Southern Ave",
"url": "http://www.wmata.com/rider_tools/pids/showpid.cfm?station_id=107",
"lines": [
"Green"
],
"address": "1411 Southern Avenue, Temple Hills, MD 20748"
}
},
{
"geometry": {
"type": "Point",
"coordinates": [
-76.935256783,
38.9081784965
]
},
"type": "Feature",
"properties": {
"description": "Deanwood",
"marker-symbol": "rail-metro",
"title": "Deanwood",
"url": "http://www.wmata.com/rider_tools/pids/showpid.cfm?station_id=65",
"lines": [
"Orange"
],
"address": "4720 Minnesota Avenue NE, Washington, DC 20019"
}
}
]}
Mapbox accepts GeoJSON, you'll need to ask Overpass to return data as GeoJSON.
Related
I have a .geojson file with many lineStrings with the location of transects used to monitor shoreline change, so each transect runs across a land/water boundary. When visualized as is, the transects originate offshore and end onshore. For an analysis tool I am using, I need the locations to be swapped: the first coordinate needs to start on land and end offshore. I will have many thousands of these transects to change and want to make sure I'm doing it correctly but can't seem to figure out this very simple task (sorry, I am new here). I am working in python and earth engine.
# original
{
"type": "FeatureCollection",
"name": "EastChukci_small_testArea",
"crs": { "type": "name", "properties": { "name": "urn:ogc:def:crs:EPSG::3857" } },
"features": [
{ "type": "Feature", "properties": { "name": 2722 }, "geometry": { "type": "LineString", "coordinates": [ [ -17592698.71288351342082, 11344741.029055444523692 ], [ -17592054.347651835530996, 11343198.733621645718813 ] ] } },
{ "type": "Feature", "properties": { "name": 2723 }, "geometry": { "type": "LineString", "coordinates": [ [ -17592838.831736516207457, 11344682.393273767083883 ], [ -17592194.440066188573837, 11343140.124529516324401 ] ] } },
{ "type": "Feature", "properties": { "name": 2724 }, "geometry": { "type": "LineString", "coordinates": [ [ -17592978.948162343353033, 11344623.755085829645395 ], [ -17592334.530055023729801, 11343081.513031836599112 ] ] } },
]
}
# desired
{
"type": "FeatureCollection",
"name": "EastChukci_small_testArea",
"crs": { "type": "name", "properties": { "name": "urn:ogc:def:crs:EPSG::3857" } },
"features": [
{ "type": "Feature", "properties": { "name": 2722 }, "geometry": { "type": "LineString", "coordinates": [[ -17592054.347651835530996, 11343198.733621645718813 ], [ -17592698.71288351342082, 11344741.029055444523692 ] ] } },
{ "type": "Feature", "properties": { "name": 2723 }, "geometry": { "type": "LineString", "coordinates": [ [ -17592194.440066188573837, 11343140.124529516324401 ], [ -17592838.831736516207457, 11344682.393273767083883 ] ] } },
{ "type": "Feature", "properties": { "name": 2724 }, "geometry": { "type": "LineString", "coordinates": [ [ -17592334.530055023729801, 11343081.513031836599112 ] ], [ -17592978.948162343353033, 11344623.755085829645395 ] ] } },
]
}
Thanks in advance.
To read and write a json file you can use json module. The code below should solve your problem, but the downside of it is that it loads the whole file to the memory at once.
import json
with open('data.json', 'r') as json_file:
data = json.load(json_file)
for feature_data in data['features']:
feature_data['geometry']['coordinates'].reverse()
with open('data.json', 'w') as json_file:
json.dump(data, json_file)
I have a geojson feature collection dataset with a lot of features. I want to add/update the properties of each feature with properties of a json file. The unique identifier of both datasets is the "uuid" value.
This is the geojson format:
mtx = {
"type": "FeatureCollection",
"crs": {
"type": "name",
"properties": {
"name": "EPSG:4326"
}
},
"features": [
{
"type": "Feature",
"id": 1,
"geometry": {
"type": "Point",
"coordinates": [
5.36516933279853,
51.5510854507331
]
},
"properties": {
"OBJECTID": 1,
"PK_UID": 1,
"uuid": "1efa8916-c854-465b-80f5-1f02fd25fb31",
"road": "A2",
"lane": 1,
"km": 134.96,
"bearing": 148.02261,
"locid": "A2134.96"
}
},
{
"type": "Feature",
"id": 2,
"geometry": {
"type": "Point",
"coordinates": [
5.05380200345974,
52.3264095459638
]
},
"properties": {
"OBJECTID": 2,
"PK_UID": 2,
"uuid": "73bf3758-6754-433f-9896-d03c0673ae55",
"road": "A1",
"lane": 3,
"km": 11.593,
"bearing": 113.404253,
"locid": "A111.593"
}
}
]
}
And this is the json format:
msi= [
{
"uuid": "1efa8916-c854-465b-80f5-1f02fd25fb31",
"road": "A2",
"carriageway": "R",
"lane": "1",
"km": "134.960",
"display": "blank",
"display_attrs": "{'flashing': 'false'}",
"speedlimit": "null"
},
{
"uuid": "73bf3758-6754-433f-9896-d03c0673ae55",
"road": "A1",
"carriageway": "R",
"lane": "3",
"km": "11.593",
"display": "blank",
"display_attrs": "{'flashing': 'false'}",
"speedlimit": "null"
}
]
So how can I make a python script that loop through the geojson features and update each feature properties with the matching properties from the json based on the "uuid" value?
I tried something like this but this didn't give me the expected result:
#Loop over GeoJSON features and update the new properties from msi json
for feat in mtx['features']:
for i in range(len(msi)):
if mtx['features'][i]['properties']['uuid'] == msi[i]['uuid']:
feat ['properties'].update(msi[i])
Thanks for helping me out.
I'd do something like this (completely untested, so beware of mistakes...):
# Turn msi into a dictionary for easy access
msi_dict = {m['uuid']: m for m in msi}
for f in mtx['features']:
f['properties'].update(msi_dict.get(f['properties']['uuid'], {}))
Here is a schema of the json output which I am trying to parse and write specific fields from it into a csv file (Example: cve id, description,....)
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "JSON Schema for NVD Vulnerability Data Feed version 1.1",
"id": "https://scap.nist.gov/schema/nvd/feed/1.1/nvd_cve_feed_json_1.1.schema",
"definitions": {
"def_cpe_name": {
"description": "CPE name",
"type": "object",
"properties": {
"cpe22Uri": {
"type": "string"
},
"cpe23Uri": {
"type": "string"
},
"lastModifiedDate": {
"type": "string"
}
},
"required": [
"cpe23Uri"
]
},
"def_cpe_match": {
"description": "CPE match string or range",
"type": "object",
"properties": {
"vulnerable": {
"type": "boolean"
},
"cpe22Uri": {
"type": "string"
},
"cpe23Uri": {
"type": "string"
},
"versionStartExcluding": {
"type": "string"
},
"versionStartIncluding": {
"type": "string"
},
"versionEndExcluding": {
"type": "string"
},
"versionEndIncluding": {
"type": "string"
},
"cpe_name": {
"type": "array",
"items": {
"$ref": "#/definitions/def_cpe_name"
}
}
},
"required": [
"vulnerable",
"cpe23Uri"
]
},
"def_node": {
"description": "Defines a node or sub-node in an NVD applicability statement.",
"properties": {
"operator": {"type": "string"},
"negate": {"type": "boolean"},
"children": {
"type": "array",
"items": {"$ref": "#/definitions/def_node"}
},
"cpe_match": {
"type": "array",
"items": {"$ref": "#/definitions/def_cpe_match"}
}
}
},
"def_configurations": {
"description": "Defines the set of product configurations for a NVD applicability statement.",
"properties": {
"CVE_data_version": {"type": "string"},
"nodes": {
"type": "array",
"items": {"$ref": "#/definitions/def_node"}
}
},
"required": [
"CVE_data_version"
]
},
"def_subscore": {
"description": "CVSS subscore.",
"type": "number",
"minimum": 0,
"maximum": 10
},
"def_impact": {
"description": "Impact scores for a vulnerability as found on NVD.",
"type": "object",
"properties": {
"baseMetricV3": {
"description": "CVSS V3.x score.",
"type": "object",
"properties": {
"cvssV3": {"$ref": "cvss-v3.x.json"},
"exploitabilityScore": {"$ref": "#/definitions/def_subscore"},
"impactScore": {"$ref": "#/definitions/def_subscore"}
}
},
"baseMetricV2": {
"description": "CVSS V2.0 score.",
"type": "object",
"properties": {
"cvssV2": {"$ref": "cvss-v2.0.json"},
"severity": {"type": "string"},
"exploitabilityScore": {"$ref": "#/definitions/def_subscore"},
"impactScore": {"$ref": "#/definitions/def_subscore"},
"acInsufInfo": {"type": "boolean"},
"obtainAllPrivilege": {"type": "boolean"},
"obtainUserPrivilege": {"type": "boolean"},
"obtainOtherPrivilege": {"type": "boolean"},
"userInteractionRequired": {"type": "boolean"}
}
}
}
},
"def_cve_item": {
"description": "Defines a vulnerability in the NVD data feed.",
"properties": {
"cve": {"$ref": "CVE_JSON_4.0_min_1.1.schema"},
"configurations": {"$ref": "#/definitions/def_configurations"},
"impact": {"$ref": "#/definitions/def_impact"},
"publishedDate": {"type": "string"},
"lastModifiedDate": {"type": "string"}
},
"required": ["cve"]
}
},
"type": "object",
"properties": {
"CVE_data_type": {"type": "string"},
"CVE_data_format": {"type": "string"},
"CVE_data_version": {"type": "string"},
"CVE_data_numberOfCVEs": {
"description": "NVD adds number of CVE in this feed",
"type": "string"
},
"CVE_data_timestamp": {
"description": "NVD adds feed date timestamp",
"type": "string"
},
"CVE_Items": {
"description": "NVD feed array of CVE",
"type": "array",
"items": {"$ref": "#/definitions/def_cve_item"}
}
},
"required": [
"CVE_data_type",
"CVE_data_format",
"CVE_data_version",
"CVE_Items"
]
}
# -*- coding: utf-8 -*-
"""
Created on Thu Dec 3 17:08:51 2020
#author: Rajat Varshney
"""
import requests, json
api_url = 'https://services.nvd.nist.gov/rest/json/cve/1.0/'
cveid = input('Enter CVE ID: ')
api_call = requests.get(api_url+cveid)
print(api_call.content)
with open('cve details.txt', 'w') as outfile:
json.dump(api_call.content, outfile)
I am developing a json schema and I am trying to test if files validate against it properly. Still new to the whole json schema world (since today), apologies if my terminology is not correct.
I have different types of files, and they will differ with regards to their biomaterial_type. Each of them should be tested for "#/definitions/basic", some of them for "#/definitions/donor", and they all will have unique fields to test for.
Here is a (shortened) example containing one biomaterial_type:
{
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"basic": {
"type": "object",
"description": "Objects shared across all samples",
"properties": {
"sample_ontology_uri" : {
"type": "array", "minItems": 1,
"items": {
"type": "string",
"format": "uri",
"description": "(Ontology: EFO) links to sample ontology information."}},
"disease_ontology_uri" : {
"type": "array", "minItems": 1,
"items": {
"type": "string",
"format": "uri",
"description": "(Ontology: NCIM)"}},
"disease" : {
"type": "array", "minItems": 1, "maxItems": 1,
"items": {
"type": "string",
"description": "Free form field "}},
"biomaterial_provider" : {
"type": "array", "minItems": 1, "maxItems": 1,
"items": {
"type": "string",
"description": "The name of the company, laboratory or person that provided the biological material."}},
"biomaterial_type" : {
"type": "array", "minItems": 1, "maxItems": 1,
"items": {
"type": "string",
"description": "The type of the biosample used (Cell Line, Primary Cell, Primary Cell Culture, Primary Tissue)",
"enum":["Cell Line", "Primary Cell", "Primary Cell Culture", "Primary Tissue"]}},
"treatment" : {
"type": "array", "minItems": 1, "maxItems": 1,
"items": {
"type": "string",
"description": "Any artificial modification (differentiation, activation, genome editing, etc)."}},
"biological_replicates": {
"type": "array",
"items": {
"type": "string",
"description": "List of biological replicate sample accessions"}}
},
"required": ["sample_ontology_curie", "disease_ontology_curie", "disease", "biomaterial_provider", "biomaterial_type", "treatment", "biological_replicates"]
},
"donor": {
"type": "object",
"description": "Additional set of properties for samples coming from a donor.",
"properties": {
"donor_id" : {
"type": "array", "minItems": 1, "maxItems": 1,
"items": {
"type": "string",
"description": "An identifying designation for the donor that provided the cells/tissues."}},
"donor_age" : {
"type": "array", "minItems": 1, "maxItems": 1,
"items": {
"description": "The age of the donor that provided the cells/tissues. NA if not available. If over 90 years enter as 90+. If entering a range of ages use the format “{age}-{age}”.",
"oneOf": [
{ "type": "number" },
{ "type": "string", "enum": ["90+", "NA"] },
{ "type": "string", "format": "uri" }
]
}},
"donor_age_unit" : {
"type": "array", "minItems": 1, "maxItems": 1,
"items": {
"type": "string",
"description": "The unit of measurement used to describe the age of the sample (year, month, week, day)",
"enum": ["year", "month", "week", "day"]}},
"donor_life_stage": {
"type": "array", "minItems": 1, "maxItems": 1,
"items": {
"type": "string",
"description": "The stage or phase of the donor when the sample was taken (embryonic, fetal, postnatal, newborn, child, adult, unknown)",
"enum": ["embryonic", "fetal", "postnatal", "newborn", "child", "adult", "unknown"]}},
"donor_health_status" : {
"type": "array", "minItems": 1, "maxItems": 1, "items": {
"type": "string",
"description": "The health status of the donor that provided the primary cell. NA if not available."}},
"donor_health_status_ontology_uri" : {
"type": "array", "minItems": 1,
"items": {
"type": "string",
"format": "uri",
"description": "(Ontology: NCIM) "}},
"donor_sex" : {"type": "array", "minItems": 1, "maxItems": 1, "items": {"type": "string", "enum": ["Male", "Female", "Unknown", "Mixed"], "description": "'Male', 'Female', 'Unknown', or 'Mixed' for pooled samples."}},
"donor_ethnicity" : {
"type": "array", "minItems": 1, "maxItems": 1,
"items": {
"type": "string",
"description": "The ethnicity of the donor that provided the primary cell. NA if not available. If dealing with small/vulnerable populations consider identifiability issues."}}
},
"required": ["donor_id", "donor_age", "donor_age_unit", "donor_life_stage", "donor_health_status_uri", "donor_health_status", "donor_sex", "donor_ethnicity"]
}
},
"type" : "object",
"if":
{"properties":
{ "biomaterial_type": {"const": "Primary Tissue"}},
"required": ["biomaterial_type"] },
"then": {
"allOf": [
{ "$ref": "#/definitions/donor" },
{
"properties": {
"tissue_type" : {
"type": "array", "minItems": 1, "maxItems": 1,
"items": {
"type": "string",
"description": "The type of tissue."}},
"tissue_depot" : {
"type": "array", "minItems": 1, "maxItems": 1,
"items": {
"type": "string",
"description": "Details about the anatomical location from which the primary tissue was collected."}},
"collection_method" : {
"type": "array", "minItems": 1, "maxItems": 1,
"items": {
"type": "string",
"description": "The protocol for collecting the primary tissue."}}
},
"required": ["tissue_type", "tissue_depot", "collection_method"]
}
]
}
}
Additional biomaterial_type will be added via additional if-conditions.
Here is a example json:
{
"SAMPLE_SET": {
"SAMPLE": [
{
"TITLE": "Homo sapiens male embryo (108 days) small intestine tissue",
"SAMPLE_NAME": {
"TAXON_ID": "9606",
"SCIENTIFIC_NAME": "Homo sapiens",
"COMMON_NAME": "human"
},
"SAMPLE_ATTRIBUTES": {
"SAMPLE_ATTRIBUTE": [
{
"TAG": "SAMPLE_ONTOLOGY_URI",
"VALUE": "http://purl.obolibrary.org/obo/UBERON:0002108"
},
{
"TAG": "DISEASE_ONTOLOGY_URI",
"VALUE": "https://ncit.nci.nih.gov/ncitbrowser/ConceptReport.jsp?dictionary=NCI_Thesaurus&code=C115935"
},
{
"TAG": "DISEASE",
"VALUE": "Healthy"
},
{
"TAG": "BIOMATERIAL_PROVIDER",
"VALUE": "Ian Glass at Congenital Defects Lab, University of Washington"
},
{
"TAG": "BIOMATERIAL_TYPE",
"VALUE": "Primary Tissue"
},
{
"TAG": "TISSUE_TYPE",
"VALUE": "small intestine"
},
{
"TAG": "TISSUE_DEPOT",
"VALUE": "Ian Glass at Congenital Defects Lab, University of Washington"
},
{
"TAG": "COLLECTION_METHOD",
"VALUE": "unknown"
},
{
"TAG": "DONOR_ID",
"VALUE": "ENCDO119ASK"
},
{
"TAG": "DONOR_AGE",
"VALUE": "NA"
},
{
"TAG": "DONOR_AGE_UNIT",
"VALUE": "day"
},
{
"TAG": "DONOR_LIFE_STAGE",
"VALUE": "embryonic"
},
{
"TAG": "DONOR_HEALTH_STATUS_ONTOLOGY_URI",
"VALUE": "https://ncit.nci.nih.gov/ncitbrowser/ConceptReport.jsp?dictionary=NCI_Thesaurus&code=C115935"
},
{
"TAG": "DONOR_HEALTH_STATUS",
"VALUE": "Healthy"
},
{
"TAG": "DONOR_SEX",
"VALUE": "Male"
},
{
"TAG": "DONOR_ETHNICITY",
"VALUE": "NA"
}
]
},
"_accession": "ENCBS054KUO",
"_center_name": "ENCODE"
},
]
}
}
I am trying to test if schema makes sense using jsonschema with python:
import json
import jsonschema
from jsonschema import validate
data = ''
schema = ''
with open('data.json', 'r') as file:
data = file.read()
with open(schema.json, 'r') as file:
schema = file.read()
try:
jsonschema.validate(json.loads(data), json.loads(schema))
print('ok')
except jsonschema.ValidationError as e:
print (e.message)
except jsonschema.SchemaError as e:
print (e)
I always get "ok", even if I provide json data with errors.
Is the problem with my Python script or with my schema?
Thanks for any pointers.
I am fairly new to the geojson spec...and formatting is causing havok.
All I am trying to do is build a new "features" list (for Points only) which I am adding new 'properties' for. Then I write it into (test.json).
Right now I am returning which has intermittent cases of "type": "FeatureCollection" (I was only expecting to see it once - at the top of the file) and some bad } syntax errors:
{"type": "FeatureCollection",
"features": [
{"geometry": {
"type": "Point",
"coordinates": [-122.3447075, 47.6821492]},
"type": "Feature",
"properties": {
"marker-color": "#808080",
"timestamp": "2013-08-17T22:41:18Z",
"version": 3,
"user": "seattlefyi",
"last_updated": "over a year ago",
"id": 427307160,
"marker-size": "small"
}}, ## what??
]} ## what??
{"type": "FeatureCollection",
"features": [
{"geometry": {
"type": "Point",
"coordinates": [-122.3447075, 47.6821492]},
"type": "Feature",
"properties": {
"marker-color": "#808080",
"timestamp": "2013-08-17T22:41:18Z",
"version": 3,
"user": "seattlefyi",
"last_updated": "over a year ago",
"id": 427307160,
"marker-size": "small"
}}, ## what...no "type": "FeatureCollection" on this one?
{"geometry": {
"type": "Point",
"coordinates": [-122.377932, 47.5641566]},
"type": "Feature",
"properties": {
"marker-color": "#808080",
"timestamp": "2009-07-11T04:04:51Z",
"version": 1,
"user": "Rob Lanphier",
"last_updated": "over a year ago",
"id": 439976119,
"marker-size": "small"
}
}
]
}
However, I'm trying to return
{"type": "FeatureCollection",
"features": [
{"type": "Feature",
"geometry": {
"type": "Point",
"coordinates": [-122.3447075, 47.6821492]},
"properties": {
"marker-color": "#808080",
"timestamp": "2013-08-17T22:41:18Z",
"version": 3,
"user": "ralph",
"last_updated": "over a year ago",
"id": 427307160,
"marker-size": "small"
}},
{"type": "Feature",
"geometry": {
"type": "Point",
"coordinates": [-122.377932, 47.5641566]},
"properties": {
"marker-color": "#808080",
"timestamp": "2009-07-11T04:04:51Z",
"version": 1,
"user": "Rob Lanphier",
"last_updated": "over a year ago",
"id": 439976119,
"marker-size": "small"
}
}
]
}
Code is:
def write_to_features(source, class_time, color):
""" write the json into geojson
takes all the items from one node ("lat", "lon", "id", "user", "tags", "timestamp")
writes all the items with new tags "last_updated","marker-color","marker-size"
returns a dict
"""
pt = {
"type": "Feature",
"geometry": {
"type": 'Point',
"coordinates": [float(source['lon']), float(source['lat'])]
},
"properties": {
"user": source['user'],
"id": source['id'],
"version": source['version'],
"timestamp": source['timestamp'],
"last_updated": classified_time,
"marker-color": marker_color,
"marker-size": "small"
}
}
return pt
def __main__():
geojson = { "type": "FeatureCollection", "features": [] }
outfile = r'.\test.json'
with open(outfile, 'w') as geojson_file:
for item in all_data_dict['elements']:
point_dict = write_to_features(item, data_w_update, data_item_color)
geojson['features'].append(point_dict)
json.dump(geojson, geojson_file)
Shouldn't your json.dump(geojson, geojson_file) be outside your loop?you append in the line above it..so I'm questioning why would you dump/write to the file multiple times?I would think you should only be calling json.dump once.