Web scraping Python beautifulsoup

Question

I'm trying to create a crawler that scans the website https://www.superherodb.com/ and fetches the information on all the superheroes (seen on:https://www.superherodb.com/characters) from their individual pages. I want to fetch all the information on the hero: the stats, powers, equipment, origin, connections, etc. But I am having trouble accessing their stats from the hero's page.

For example, this page: https://www.superherodb.com/001/10-39302/

For the Power Stats section in the hero's page I tried:

  bs_test.find_all("div", {"class": "stat-value"})

and:

    bs_test.select(".stat-value")

But the output always outputs 0 as the value:

[<div class="stat-value">0</div>,
 <div class="stat-value">0</div>,
 <div class="stat-value">0</div>,
 <div class="stat-value">0</div>,
 <div class="stat-value">0</div>,
 <div class="stat-value">0</div>,
 <div class="stat-value">0</div>]

What am I missing here? Please help me.

Answer 1

They aren't visible there. Try scraping <class="note footnote"> rather than the stat_value. It provides the following data:

stats_10_39302_shdb = {"stats":{"int":140,"str":45,"spe":5,"dur":5,"pow":0,"com":20,"tie":0},"bars":{"int":70,"str":1,"spe":1,"dur":5,"pow":0,"com":20,"tie":0}

for the Han example.