Concatenate labels to an existing dataframe

Question

I want to use a list of names "headers" to create a new column in my dataframe. In the initial table, the name of each division is positioned above the results for each team in that division. I want to add that header to each row entry for each divsion to make the data more identifiable like this. I have the headers stored in the "header" object in my code. How can I multiply each division header by the number of rows that appear in the division and append to the dataset?

Edit: here is another snippet of what I want the get from the end product.

df3 = df.iloc[0:6]
df3.insert(0, 'Divisions', ['na','L5 Junior', 'L5 Junior', 'na', 
'L5 Senior - Medium', 'L5 Senior - Medium'])

df3

.

import pandas as pd
import numpy as np
from bs4 import BeautifulSoup
import requests

Import HTML

scr = 'https://tv.varsity.com/results/7361971-2022-spirit-unlimited-battle-at-the- 
boardwalk-atlantic-city-grand-ntls/31220'
scr1 = requests.get(scr)
soup = BeautifulSoup(scr1.text, "html.parser")

List of names to append

table_MN = pd.read_html(scr)
sp3 = soup.find(class_="full-content").find_all("h2")
headers = [elt.text for elt in sp3]

table_MN = pd.read_html(scr)

Extract text and header from division info

div = pd.DataFrame(headers)
div.columns = ["division"]

df = pd.concat(table_MN, ignore_index=True)

df.columns = df.iloc[0]

df

Answer 1

It is still not clear what is the output you are looking for. However, may I suggest the following, which accomplishes selecting common headers from tables in table_MN and the concatenating the results. If it is going in the right direction pls let me know, and indicate what else you want to extract from the resulting table:

tmn_1 = [tbl.T.set_index(0).T for tbl in table_MN]
pd.concat(tmn_1, axis=0, ignore_index = True)

output:

     Rank  Program Name                 Team Name            Raw Score    Deductions    Performance Score    Event Score
--  ------  ---------------------------  -----------------  -----------  ------------  -------------------  -------------
 0       1  Rockstar Cheer New Jersey    City Girls             47.8667          0                 95.7333        95.6833
 1       2  Cheer Factor                 Xtraordinary           46.6667          0.15              93.1833        92.8541
 2       1  Rockstar Cheer New Jersey    City Girls             47.7667          0                 95.5333        23.8833
 3       2  Cheer Factor                 Xtraordinary           46.0333          0.2               91.8667        22.9667
 4       1  Star Athletics               Roar                   47.5333          0.9               94.1667        93.9959
 5       1  Prime Time All Stars         Lady Onyx              43.9             1.35              86.45          86.6958
 6       1  Prime Time All Stars         Lady Onyx              44.1667          0.9               87.4333        21.8583
 7       1  Just Cheer All Stars         Jag 5                  46.4333          0.15              92.7167        92.2875
 8       1  Just Cheer All Stars         Jag 5                  45.8             0.6               91             22.75
 9       1  Quest Athletics              Black Ops              47.4333          0.45              94.4167        93.725
10       1  Quest Athletics              Black Ops              46.5             1.35              91.65          22.9125
11       1  The Stingray Allstars        X-Rays                 45.3             0.95              89.65          88.4375
12       1  Vortex Allstars              Lady Rays              45.7             0.5               90.9           91.1083
13       1  Vortex Allstars              Lady Rays              45.8667          0                 91.7333        22.9333
14       1  Upper Merion All Stars       Citrus                 46.4333          0                 92.8667        92.7
15       2  Cheer Factor                 JUNIOR X               45.9             1.1               90.7           90.6542
16       3  NJ Premier All Stars         Prodigy                44.6333          0.05              89.2167        89.8292
17       1  Upper Merion All Stars       Citrus                 46.1             0                 92.2           23.05
18       2  NJ Premier All Stars         Prodigy                45.8333          0                 91.6667        22.9167
19       3  Cheer Factor                 JUNIOR X               45.7333          0.95              90.5167        22.6292
20       1  Virginia Royalty Athletics   Dynasty                46.5             0                 93             92.9
21       1  Virginia Royalty Athletics   Dynasty                46.3             0                 92.6           23.15
22       1  South Jersey Storm           Lady Reign             47.2333          0                 94.4667        93.4875
...