'How to receive a json response using proxycrawl

from proxycrawl import CrawlingAPI
from bs4 import BeautifulSoup
api = CrawlingAPI({'token': 'wMtzOpKeL9CCASWQBGiRLA'})
# a = """GET 'https://api.proxycrawl.com/?token=wMtzOpKeL9CCASWQBGiRLA&url=https%3A%2F%2Fwww.amazon.com&format=json'
# Response:
# {
#   "original_status": "200",
#   "pc_status": 200,
#   "url": "https://www.jackson.org/provider-directory/?specialty=Allergy",
#   "body": "\u003C!doctype html\u003E\u003Chtml class=\"a-no-js\" data-19ax5a9jf\n... (all the html of the page)"
# }"""
targetURL = 'https://www.jackson.org/provider-directory/?specialty=Allergy'
response = api.get(targetURL)
if response['status_code'] == 200:
    b_soup = BeautifulSoup(response['body'], 'lxml')
    print(b_soup)

Is there anyway to receive the json response using proxycrawling and get the details from the webpage. Link: https://proxycrawl.com/docs/crawling-api/response/#html-response



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source