'Scrape Website that is running meteor, using python requests
You see, there is this website called edabit. All you need to know is that I want to scrape the website without using selenium.
I want to learn how.
What does selenium do under the hood that makes it step forward. I make a request and get html that references to a meteor runtime config. Once I get to that stage, how do I go from there?
import requests
url = "https://edabit.com/challenge/ARr5tA458o2tC9FTN"
r = requests.get(url)
print(r.text,r,sep="\n\n\t")
⇩⇩⇩⇩Outputs⇩⇩⇩⇩
<!DOCTYPE html>
<html lang="en">
<head>
<link rel="stylesheet" type="text/css" class="__meteor-css__" href="/c039cef9b47481baf0e9a343e536154438171f0f.css?meteor_css_resource=true">
<link rel="stylesheet" type="text/css" class="__meteor-css__" href="/52f02f273b0d7fd6013f016f05c3645aa114c8e6.css?meteor_css_resource=true">
<meta name="fragment" content="!">
<link href="https://fonts.googleapis.com/css?family=Lato" rel="stylesheet">
<link href="https://edabit-fonts.s3-us-west-1.amazonaws.com/avenir.css?family=Avenir" rel="stylesheet">
<link rel="icon" type="image/png" href="https://s3.amazonaws.com/edabit-images/logo_main_medium.png">
<title>Edabit</title>
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<link href="https://fonts.googleapis.com/css?family=Lato" rel="stylesheet">
<link href="https://edabit-fonts.s3-us-west-1.amazonaws.com/avenir.css?family=Avenir" rel="stylesheet">
<link rel="icon" type="image/png" href="https://s3.amazonaws.com/edabit-images/logo_main_medium.png">
<script src="https://script.tapfiliate.com/tapfiliate.js" type="text/javascript" async></script>
<script type="text/javascript">
(function(t,a,p){t.TapfiliateObject=a;t[a]=t[a]||function(){
(t[a].q=t[a].q||[]).push(arguments)}})(window,'tap');
</script>
</head>
<body>
<script type="text/javascript">__meteor_runtime_config__ = JSON.parse(decodeURIComponent("%7B%22meteorRelease%22%3A%22METEOR%401.8.2%22%2C%22gitCommitHash%22%3A%22b231bf6df48b60606f5acf2f54427d52feb3711f%22%2C%22meteorEnv%22%3A%7B%22NODE_ENV%22%3A%22production%22%2C%22TEST_METADATA%22%3A%22%7B%7D%22%7D%2C%22PUBLIC_SETTINGS%22%3A%7B%22analyticsSettings%22%3A%7B%22Google%20Analytics%22%3A%7B%22trackingId%22%3A%22UA-91229704-1%22%7D%7D%2C%22hotjar%22%3A%7B%22hjid%22%3A%22399651%22%2C%22hjsv%22%3A%221%22%7D%2C%22pricing%22%3A%7B%22lifetime%22%3A299%2C%22monthly%22%3A39%2C%22yearly%22%3A120%7D%2C%22stripeKey%22%3A%22pk_live_OW5zXRZem0eb8x31ZSaET8xO%22%2C%22tap%22%3A%7B%22accountId%22%3A%2218539-483fb9%22%2C%22integration%22%3A%22stripe%22%7D%2C%22trialLimit%22%3A15%7D%2C%22ROOT_URL%22%3A%22https%3A%2F%2Fedabit.com%22%2C%22ROOT_URL_PATH_PREFIX%22%3A%22%22%2C%22kadira%22%3A%7B%22appId%22%3A%22Mz7xd9SXTmvc2Cyw5%22%2C%22endpoint%22%3A%22https%3A%2F%2Fapm-engine.meteor.com%22%2C%22clientEngineSyncDelay%22%3A10000%2C%22enableErrorTracking%22%3Atrue%7D%2C%22autoupdate%22%3A%7B%22versions%22%3A%7B%22web.browser%22%3A%7B%22version%22%3A%224c5f400ef296a1f363c2ac037bcca994a67c05a8%22%2C%22versionRefreshable%22%3A%22eabbe41e49f1cfb764e2a02a33e852728e7a26f0%22%2C%22versionNonRefreshable%22%3A%22799cb33096d2bcc3b00eb190b902cd6840ce9e86%22%7D%2C%22web.browser.legacy%22%3A%7B%22version%22%3A%22f45f0508d12468a843658947c98fad067be0fea6%22%2C%22versionRefreshable%22%3A%22eabbe41e49f1cfb764e2a02a33e852728e7a26f0%22%2C%22versionNonRefreshable%22%3A%225fdcb5621ec7f5351fe9c6d265dc52791133253a%22%7D%7D%2C%22autoupdateVersion%22%3Anull%2C%22autoupdateVersionRefreshable%22%3Anull%2C%22autoupdateVersionCordova%22%3Anull%2C%22appId%22%3A%226oe24v3kjymx1952geg%22%7D%2C%22appId%22%3A%226oe24v3kjymx1952geg%22%2C%22isModern%22%3Afalse%7D"))</script>
<script type="text/javascript" src="/fa82c61660f6e946bc1c9dfcc6f33af930712e50.js?meteor_js_resource=true"></script>
</body>
</html>
<Response [200]>
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|