Can't get text value for parsed html

Question

I need help fetching text values from the parsed HTML (uncomment print(meta_tags) to see parsed data). I've used .text and .get_text() but it doesn't work here. Once I get the text value I can create a panda dataframe just like df1. Can anyone help me with that, please?

import requests
from bs4 import BeautifulSoup as bs
import pandas as pd

article_list = []
for pages in range(1, 3):
  url = "someurl"
  page = requests.get(url+str(pages))
  soup = bs(page.content, "html.parser")

  all_articles = soup.find_all("li", class_="post-item tie-standard")

  for item in all_articles:
    titles = item.find('h2', class_='post-title').text
    links = item.find('a', class_='more-link button')['href']

    page = requests.get(links)
    soup = bs(page.content, "html.parser")

    head_h1 = soup.find_all("h1")
    head_h2 = soup.find_all("h2")
    head_h3 = soup.find_all("h3")

    meta_tags = [(h1_text, h2_text, h3_text)
    for h1_text in head_h1
    for h2_text in head_h2
    for h3_text in head_h3]
    # print(meta_tags)

    articles = {
        'title': titles,
        'links': links,
    }
    article_list.append(articles)
    
df1 = pd.DataFrame(article_list)
df2 = pd.DataFrame(meta_tags)

Answer 1

Part of your problem is that triple-nested list comprehension. That's not what you want. If there are 9 of each tag, that's going to get to get you 9x9x9 entries, when you are expecting 9+9+9.

See what you think of this. I've eliminated pandas for now; you can look at the final output and decide what do to with it.

import requests
from bs4 import BeautifulSoup as bs

article_list = []
for pages in range(1, 3):
    url = "https://lisbdnet.com/category/digital/advertising/page/"
    page = requests.get(url+str(pages))
    soup = bs(page.content, "html.parser")

    for item in soup.find_all("li", class_="post-item tie-standard"):
        titles = item.find('h2', class_='post-title').text
        links = item.find('a', class_='more-link button')['href']

        page = requests.get(links)
        soup = bs(page.content, "html.parser")

        head_h1 = soup.find_all("h1")
        head_h2 = soup.find_all("h2")
        head_h3 = soup.find_all("h3")

        meta_tags = [
            [h1.text for h1 in head_h1],
            [h2.text for h2 in head_h2 if h2.text],
            [h3.text for h3 in head_h3]
        ]

        article_list.append({
            'title': titles,
            'link': links,
            'meta': meta_tags
        })

from pprint import pprint
pprint(article_list)

Output:

[{'links': 'https://lisbdnet.com/how-does-advertising-affect-consumers-2/',
  'meta': [['A Study On The Influences of Advertisement On Consumer Buying '
            'Behavior:How does advertising affect consumers ?'],
           ['How does advertising affect consumers ?',
            'Brand Awareness',
            'Branded Advertising',
            'Consumers Get the Information They Need',
            'Reminders and Ad Repetition',
            'What are the benefits of advertising to consumers ?',
            'Negative of advertising',
            'How do advertisers use persuasive techniques to influence our '
            'decisions ?',
            'FAQ ?',
            'Conclusion'],
           ['Brand Awareness Effect',
            'The Behavioral Effect',
            'Attitude Toward The Ad Effect',
            'Advertising Recall Effect',
            'Why advertising affect children’s health ?',
            'How advertising affect demand of goods and services and '
            'purchasing decisions ?',
            'What is the difference between persuasion techniques and '
            'influence tactics ?',
            'Why is health advertising important for consumers ?',
            'What kind of person is influenced by advertising ?',
            'nhungoc',
            'Does Football Cause Brain Damage? Everything Footballer Should '
            'Know',
            'Top 13 Best Security Cameras in Australia 2021 Review, The Indoor '
            '& Outdoor Security Cameras',
            'Related Articles',
            'What is linear advertising ?',
            'What Is Cinema Advertising? How Cinema Advertising Can Help You '
            'Reach Millions of People?',
            'What is advertising sales ?',
            'What Is The Concept Of Duality? – Definitions and Examples',
            'What Is Concept Design? Definitions, The Purpose and Examples',
            'What Is Stored Program Concept? – Who, What, When and How',
            'Who does apple’s advertising ?',
            'what is linear advertising? Is Linear Marketing the small '
            'business’s best choice?',
            'Leave a Reply Cancel reply']],
  'title': 'A Study On The Influences of Advertisement On Consumer Buying '
           'Behavior:How does advertising affect consumers ?'},
 {'links': 'https://lisbdnet.com/what-is-linear-advertising-2/',
  'meta': [['What is linear advertising ?'],
           ['What is linear advertising\xa0?',
            'How does it work\xa0?',
            'Linear addressable advertising\xa0?',
            'Linear and digital media\xa0?',
...