'Get Dynamiclly loaded Source Code from python using mechanize and bs4
I want to get source code of page that loads from javascript actually that page is linkedin profile page and i want to get job and education details.
I'm not using selenium i don't want browser window to open i know about headless but cookies problem
I have logedin through mechanize and i have get some data like phone number, address, headlines, emails, and Full Name. But as it is loaded from javascript so i can't get whole page data.
Data getting:
.....<code id="bpr-guid-892585" style="display: none">
{"data":{"entityUrn":"urn:li:collectionResponse:uPYuDSPXzooiHx+zPOguG1+f+JFMWTWFEfhiIQtEFMM=","elements":[],"paging":{"count":10,"start":0,"total":0,"links":[]},"$type":"com.linkedin.restli.common.CollectionResponse"},"included":[]}
</code>
<code id="datalet-bpr-guid-892585" style="display: none">
{"request":"/voyager/api/takeovers","status":200,"body":"bpr-guid-892585","method":"GET","headers":{"x-li-uuid":"AAXafRyXk/WxvhRuOZTrnA\u003D\u003D"}}
</code>
<img class="datalet-bpr-guid-892585" src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" style="display: none"/><code id="bpr-guid-892586" style="display: none">
{"data":{"entityUrn":"urn:li:collectionResponse:nZx6/1e1AAbOHh075gv083zrunZT186/K+rx5FP70A4=","elements":[{"lixTracking":{"urn":"urn:li:member:865882626","segmentIndex":2,"experimentId":4358724,"treatmentIndex":1,"$type":"com.linkedin.voyager.common.ChameleonConfigLixTrackingInfo"},"data":{"namespace":"premium/templates/components/chooser/plan-card","locale":"en_US","message":"Learn more","key":"i18n_card_select_plan","$type":"com.linkedin.voyager.common.ChameleonConfigDataI18n"},"displayName":"card_select_plan","description":"testing 'Learn more' on SKU cards, vs control of 'Select plan' ","lixTreatment":"VAR_t20152_PR_1","lixKey":"chameleon.PREMIUM:us.copy.17654","creatorDisplayName":"cyount","status":"PERMANENT_RAMP","$type":"com.linkedin.voyager.common.ChameleonConfigItem"},{"lixTracking":{"urn":"urn:li:member:865882626","segmentIndex":3,"experimentId":4395729,"treatmentIndex":0,"$type":"com.linkedin.voyager.common.ChameleonConfigLixTrackingInfo"},"data":{"namespace":"onboarding/templates/components/widgets/people-you-may-know","locale":"en_US","message":"Connecting with people lets you see updates and keep in touch","key":"i18n_onboarding_pymk_page_header_phase_3","$type":"com.linkedin.voyager.common.ChameleonConfigDataI18n"},"displayName":"onboarding_pymk_page_header_phase_3","description":"Onboarding PEOPLE_YOU_MAY_KNOW widget header copy test","lixTreatment":"control","lixKey":"chameleon.ONBOARDING:global.copy.19060","creatorDisplayName":"zihliu","status":"MAX_RAMP","$type":"com.linkedin.voyager.common.ChameleonConfigItem"},{"lixTracking":{"urn":"urn:li:member:865882626","segmentIndex":3,"experimentId":4395707,"treatmentIndex":1,"$type":"com.linkedin.voyager.common.ChameleonConfigLixTrackingInfo"},"data":{"namespace":"onboarding/templates/components/widgets/profile-edit-common","locale":"en_US","message":"What’s your most recent experience?","key":"i18n_onboarding_profile_edit_work_header_v2","$type":"com.linkedin.voyager.common.ChameleonConfigDataI18n"},"displayName":"onboarding_profile_edit_work_header_v2","description":"Onboarding PROFILE_EDIT widget header copy test","lixTreatment":"VAR_t21697_PR_1","lixKey":"chameleon.ONBOARDING:global.copy.19063","$recipeTypes":["com.linkedin.voyager.dash.deco.relationships.ProfileWithEmailRequired","com.linkedin.voyager.dash.deco.identity.profile.WebTopCardCore"],"$type":"com.linkedin.voyager.dash.identity.profile.Profile","firstName":"Adarsh ","profilePicture":{"displayImageWithFrameReferenceUnion":{"vectorImage":{"$recipeTypes":["com.linkedin.voyager.dash.deco.common.VectorImageOnlyRootUrlAndAttribution"],"rootUrl":"https://media-exp1.licdn.com/dms/image/C4E35AQEIVkoUWgLFvw/profile-framedphoto-shrink_","artifacts":[{"width":200,"$recipeTypes":["com.linkedin.voyager.dash.deco.common.VectorArtifact"],"fileIdentifyingUrlPathSegment":"200_200/0/1597096649541?e=1647694800&v=beta&t=XVOK0upwO6V3NaJtUWLwy-yLMDa8cZICzYH0do67vhU","expiresAt":1647694800000,"height":200,"$type":"com.linkedin.common.VectorArtifact"},{"width":400,"$recipeTypes":["com.linkedin.voyager.dash.deco.common.VectorArtifact"],"fileIdentif
</code>
<img class="terminatorlet" src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" style="display: none"/>
<div aria-live="polite" class="visually-hidden" id="a11y-notification" role="region"></div>
</body></html>
And also if there is way by using of selenium then please guide me but using headless attribute.
It contains all data that i mentiond above but when loaded from browser after login it's different.
Thanks for any help.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
