'Webscrapping a site which contains JSON data

I am working on a site to get the job data from it. The site response does not have full information when I used beautifulsoup. So tried to achieve it using Pandas. Still no luck. Can someone help me here?

import pandas as pd
import requests
headers = {
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36'}
url = f'https://hirist.com'
# r = requests.get(url, headers, verify=False)

payload = {"pageNo": "1",
           "query": "software engineer",
           "loc": '17',
           "minexp": '0',
           "maxexp": '0',
           "range": '0',
           "boost": '0',
           "searchRange": '4',
           "searchOp": 'AND',
           "jobType": "1"
           }
jsonData = requests.post(url, headers=headers,
                         json=payload, verify=False).json()
df = pd.DataFrame(jsonData)

print(df)

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source

'Webscrapping a site which contains JSON data

Sources

Related Questions