解析大型JSON文件(Python)

尼科

我是JSON的新手,但是在进行了一些有关读取JSON并进行解析的研究之后,我发现所有操作都是手动完成的:

 "image":{
        "full":"fullArt.png",
        "sprite":"sprite4.png",
        "group":"Art",
        "x":192,
        "y":0,
        "w":48,
        "h":48
   }

这意味着,如果我有这个小的JSON块要读取,我将使用类似以下内容:

jfile="image.json"
file=open(jfile, 'r')
jdata=json.load(file)
data=jdata["image"]
full=data["full"]
sprite=data["sprite"]
....

但这是假设我知道文件的内部位置,并且所有图像都是以这种方式写入的,这可能是错误的。

另一个问题是,我的目标是读取200,000行以上JSON的文件,以便将从文件中提取的数据放入数据库中。

所以我的问题是:如何读取大型JSON文件并将结果存储在数据库中?

感谢您阅读我。

编辑:添加了一些我想阅读的东西

    {
   "type":"champion",
   "format":"standAloneComplex",
   "version":"6.4.2",
   "data":{
      "Ekko":{
         "id":"Ekko",
         "key":"245",
         "name":"Ekko",
         "title":"the Boy Who Shattered Time",
         "image":{
            "full":"Ekko.png",
            "sprite":"champion4.png",
            "group":"champion",
            "x":192,
            "y":0,
            "w":48,
            "h":48
         },
         "skins":[
            {
               "id":"245000",
               "num":0,
               "name":"default",
               "chromas":false
            },
            {
               "id":"245001",
               "num":1,
               "name":"Sandstorm Ekko",
               "chromas":false
            },
            {
               "id":"245002",
               "num":2,
               "name":"Academy Ekko",
               "chromas":false
            }
         ],
         "lore":"A prodigy from the rough streets of Zaun, Ekko manipulates time to spin any situation to his advantage. Using his own invention, the Zero-Drive, he explores the branching possibilities of reality. As well as experimenting with multi-dimensional probability, Ekko spends his days running wild with the other lost children of Zaun. Though he revels in this freedom, when there's a threat to his friends, he'll endure anything to defend them. To the unknowing observer, Ekko accomplishes the impossible with ease, time and time again.",
         "blurb":"A prodigy from the rough streets of Zaun, Ekko manipulates time to spin any situation to his advantage. Using his own invention, the Zero-Drive, he explores the branching possibilities of reality. As well as experimenting with multi-dimensional ...",
         "allytips":[
            "Chronobreak is a potent escape tool, but it can also be quite powerful when used offensively. Don't underestimate its damage potential.",
            "If you can proc Z-Drive Resonance on an enemy champion, it's worth taking risks to do so. The bonus movement speed makes it easy to escape.",
            "Phase Dive's dash is a great tool for setting up Ekko's other abilities. Use it to get double hits with Timewinder or get into position to detonate Parallel Convergence."
         ],
         "enemytips":[
            "Ekko is significantly weaker when his ultimate is down. Watch for the trail he leaves behind to determine if Chronobreak is available.",
            "Ekko's stun zone takes 3 seconds to arm. Watch for the image he creates on cast and try to guess where the zone was placed.",
            "The second hit of Timewinder does more damage than the first; try to avoid it."
         ],
         "tags":[
            "Assassin",
            "Fighter"
         ],
         "partype":"MP",
         "info":{
            "attack":5,
            "defense":3,
            "magic":7,
            "difficulty":8
         },
         "stats":{
            "hp":580.0,
            "hpperlevel":80.0,
            "mp":280.0,
            "mpperlevel":50.0,
            "movespeed":340.0,
            "armor":27.0,
            "armorperlevel":3.0,
            "spellblock":32.0,
            "spellblockperlevel":1.25,
            "attackrange":125.0,
            "hpregen":9.0,
            "hpregenperlevel":0.9,
            "mpregen":6.0,
            "mpregenperlevel":0.8,
            "crit":0.0,
            "critperlevel":0.0,
            "attackdamage":59.0,
            "attackdamageperlevel":3.0,
            "attackspeedoffset":0.0,
            "attackspeedperlevel":3.3
         },
         "spells":[
            {
               "id":"EkkoQ",
               "name":"Timewinder",
               "description":"Ekko throws a temporal grenade that expands into a time-distortion field upon hitting an enemy champion, slowing and damaging anyone caught inside. After a delay, the grenade rewinds back to Ekko, dealing damage on its return.",
               "tooltip":"Ekko throws a device that deals {{ e1 }} <span class=\"color99FF99\">(+{{ a1 }})</span> magic damage to enemies it passes through. It expands into a slowing field on the first champion hit, slowing everything inside by {{ e2 }}%. It then returns to him after a delay, dealing {{ e3 }} <span class=\"color99FF99\">(+{{ a2 }})</span> magic damage to all targets hit upon return.",
               "leveltip":{
                  "label":[
                     "Outgoing Damage",
                     "Return Damage",
                     "Slow",
                     "Cooldown",
                     "Mana Cost"
                  ],
                  "effect":[
                     "{{ e1 }} -> {{ e1NL }}",
                     "{{ e3 }} -> {{ e3NL }}",
                     "{{ e2 }}% -> {{ e2NL }}%",
                     "{{ cooldown }} -> {{ cooldownnNL }}",
                     "{{ cost }} -> {{ costnNL }}"
                  ]
               },
               "maxrank":5,
               "cooldown":[
                  9,
                  8.5,
                  8,
                  7.5,
                  7
               ],
               "cooldownBurn":"9/8.5/8/7.5/7",
               "cost":[
                  50,
                  60,
                  70,
                  80,
                  90
               ],
               "costBurn":"50/60/70/80/90",
               "effect":[
                  null,
                  [
                     60,
                     75,
                     90,
                     105,
                     120
                  ],
                  [
                     32,
                     39,
                     46,
                     53,
                     60
                  ],
                  [
                     60,
                     85,
                     110,
                     135,
                     160
                  ],
                  [
                     100,
                     100,
                     100,
                     100,
                     100
                  ]
               ],
               "effectBurn":[
                  null,
                  "60/75/90/105/120",
                  "32/39/46/53/60",
                  "60/85/110/135/160",
                  "100"
               ],
               "vars":[
                  {
                     "link":"spelldamage",
                     "coeff":0.1,
                     "key":"a1"
                  },
                  {
                     "link":"spelldamage",
                     "coeff":0.6,
                     "key":"a2"
                  }
               ],
               "costType":"Mana",
               "maxammo":"-1",
               "range":[
                  1075,
                  1075,
                  1075,
                  1075,
                  1075
               ],
               "rangeBurn":"1075",
               "image":{
                  "full":"EkkoQ.png",
                  "sprite":"spell13.png",
                  "group":"spell",
                  "x":192,
                  "y":144,
                  "w":48,
                  "h":48
               },
               "resource":"{{ cost }} Mana"
            },
            {
               "id":"EkkoW",
               "name":"Parallel Convergence",
               "description":"Ekko splits the timeline, creating an anomaly after a few seconds that slows enemies caught inside. If Ekko enters the anomaly, he gains shielding and triggers a detonation, stunning enemies by suspending them in time.",
               "tooltip":"<span class=\"colorFF9900\">Passive:</span> Ekko's basic attacks deal bonus magic damage to enemies under 30% health equal to {{ e3 }}% <span class=\"color99FF99\">(+{{ a2 }}%)</span> of their missing health (max {{ e5 }} damage vs. minions and monsters).<br><br><span class=\"colorFF9900\">Active:</span> After a 3 second delay, Ekko creates a short-lived chronosphere at the target location that slows enemies who enter by 40%. If Ekko enters the sphere, he will detonate it, gaining a shield that absorbs up to {{ e4 }} <span class=\"color99FF99\">(+{{ a1 }})</span> damage for 2 seconds. Enemies caught inside are stunned for {{ e2 }} seconds.",
               "leveltip":{
                  "label":[
                     "Shield Amount",
                     "Cooldown",
                     "Mana Cost"
                  ],
                  "effect":[
                     "{{ e4 }} -> {{ e4NL }}",
                     "{{ cooldown }} -> {{ cooldownnNL }}",
                     "{{ cost }} -> {{ costnNL }}<"
                  ]
               },
               "maxrank":5,
               "cooldown":[
                  22,
                  20,
                  18,
                  16,
                  14
               ],
               "cooldownBurn":"22/20/18/16/14",
               "cost":[
                  50,
                  55,
                  60,
                  65,
                  70
               ],
               "costBurn":"50/55/60/65/70",
               "effect":[
                  null,
                  [
                     375,
                     375,
                     375,
                     375,
                     375
                  ],
                  [
                     2.25,
                     2.25,
                     2.25,
                     2.25,
                     2.25
                  ],
                  [
                     5,
                     5,
                     5,
                     5,
                     5
                  ],
                  [
                     80,
                     100,
                     120,
                     140,
                     160
                  ],
                  [
                     150,
                     150,
                     150,
                     150,
                     150
                  ]
               ],
               "effectBurn":[
                  null,
                  "375",
                  "2.25",
                  "5",
                  "80/100/120/140/160",
                  "150"
               ],
               "vars":[
                  {
                     "link":"spelldamage",
                     "coeff":0.018,
                     "key":"a2"
                  },
                  {
                     "link":"spelldamage",
                     "coeff":1.5,
                     "key":"a1"
                  }
               ],
               "costType":"Mana",
               "maxammo":"-1",
               "range":[
                  1600,
                  1600,
                  1600,
                  1600,
                  1600
               ],
               "rangeBurn":"1600",
               "image":{
                  "full":"EkkoW.png",
                  "sprite":"spell13.png",
                  "group":"spell",
                  "x":240,
                  "y":144,
                  "w":48,
                  "h":48
               },
               "resource":"{{ cost }} Mana"
            },
            {
               "id":"EkkoE",
               "name":"Phase Dive",
               "description":"Ekko rolls evasively while charging up his Z-Drive. His next attack deals bonus damage and warps reality, teleporting him to his target.",
               "tooltip":" Ekko dashes a short distance in the targeted direction. His next attack will deal {{ e1 }} <span class=\"color99FF99\">(+{{ a1 }})</span> bonus magic damage and teleport him to his target.",
               "leveltip":{
                  "label":[
                     "Damage",
                     "Cooldown",
                     "Mana Cost"
                  ],
                  "effect":[
                     "{{ e1 }} -> {{ e1NL }}",
                     "{{ cooldown }} -> {{ cooldownnNL }}",
                     "{{ cost }} -> {{ costnNL }}"
                  ]
               },
               "maxrank":5,
               "cooldown":[
                  13,
                  11.5,
                  10,
                  8.5,
                  7
               ],
               "cooldownBurn":"13/11.5/10/8.5/7",
               "cost":[
                  40,
                  50,
                  60,
                  70,
                  80
               ],
               "costBurn":"40/50/60/70/80",
               "effect":[
                  null,
                  [
                     50,
                     80,
                     110,
                     140,
                     170
                  ]
               ],
               "effectBurn":[
                  null,
                  "50/80/110/140/170"
               ],
               "vars":[
                  {
                     "link":"spelldamage",
                     "coeff":0.2,
                     "key":"a1"
                  }
               ],
               "costType":"Mana",
               "maxammo":"-1",
               "range":[
                  325,
                  325,
                  325,
                  325,
                  325
               ],
               "rangeBurn":"325",
               "image":{
                  "full":"EkkoE.png",
                  "sprite":"spell13.png",
                  "group":"spell",
                  "x":288,
                  "y":144,
                  "w":48,
                  "h":48
               },
               "resource":"{{ cost }} Mana"
            },
            {
               "id":"EkkoR",
               "name":"Chronobreak",
               "description":"Ekko shatters his timeline, becoming untargetable and rewinding to a more favorable point in time. He returns to whenever he was a few seconds ago, and heals for a percentage of the damage received in that duration. Enemies near his arrival zone take massive damage.",
               "tooltip":"Makes Ekko invulnerable and untargetable, and teleports him back to wherever he was 4 seconds ago. Ekko heals himself for {{ e3 }} + {{ e2 }}% <span class=\"color99FF99\">(+{{ a2 }}%)</span> of the damage taken over the last 4 seconds, and deals {{ e1 }} <span class=\"color99FF99\">(+{{ a1 }})</span> magic damage to all nearby enemies on arrival.",
               "leveltip":{
                  "label":[
                     "Damage",
                     "Flat Heal",
                     "Cooldown"
                  ],
                  "effect":[
                     "{{ e1 }} -> {{ e1NL }}",
                     "{{ e3 }} -> {{ e3NL }}",
                     "{{ cooldown }} -> {{ cooldownnNL }}"
                  ]
               },
               "maxrank":3,
               "cooldown":[
                  110,
                  90,
                  70
               ],
               "cooldownBurn":"110/90/70",
               "cost":[
                  100,
                  100,
                  100
               ],
               "costBurn":"100",
               "effect":[
                  null,
                  [
                     200,
                     350,
                     500
                  ],
                  [
                     20,
                     20,
                     20
                  ],
                  [
                     100,
                     150,
                     200
                  ]
               ],
               "effectBurn":[
                  null,
                  "200/350/500",
                  "20",
                  "100/150/200"
               ],
               "vars":[
                  {
                     "link":"spelldamage",
                     "coeff":0.06667,
                     "key":"a2"
                  },
                  {
                     "link":"spelldamage",
                     "coeff":1.3,
                     "key":"a1"
                  }
               ],
               "costType":"Mana",
               "maxammo":"-1",
               "range":[
                  850,
                  850,
                  850
               ],
               "rangeBurn":"850",
               "image":{
                  "full":"EkkoR.png",
                  "sprite":"spell13.png",
                  "group":"spell",
                  "x":336,
                  "y":144,
                  "w":48,
                  "h":48
               },
               "resource":"{{ cost }} Mana"
            }
         ],
         "passive":{
            "name":"Z-Drive Resonance",
            "description":"Ekko's Zero-Drive charges his spells and attacks with temporal energy. The third hit deals bonus damage and saps the target's movement speed. If the target is a champion, Ekko gains accelerated movement speed.",
            "image":{
               "full":"Ekko_P.png",
               "sprite":"passive4.png",
               "group":"passive",
               "x":384,
               "y":0,
               "w":48,
               "h":48
            }
         },
         "recommended":[
            " ANOTHER 1k lines here "
         ]
      }
   }
}
最大容量

正如@Hugo已经提到的那样,使用Pandas可以简化很多事情。除此之外,使用json_normalize()函数还可以帮助您平整嵌套的数据,以便将它们加载到数据库中变得更加容易。

我也强烈建议您仔细看看ujson(Ultra Fast JSON)模块,它比标准模块快

并且使用许多大的JSON文件,几乎不可避免地会在JSON文件/数据中出现错误-为此,我建议使用以下技巧:

python -m json.tool file.json

它应该告诉您JSON数据出了什么问题

附言:如果您需要进一步的帮助,请更新您的问题或打开一个新问题。

更新:

为了将解析的JSON数据存储在DB中,您想使用pandas.DataFrame.to_sql()函数

本文收集自互联网,转载请注明来源。

如有侵权,请联系 [email protected] 删除。

编辑于
0

我来说两句

0 条评论
登录 后参与评论

相关文章