'Viewing Returned Result in Splash with Lua
Intro
Hello, world! Ignoring the heavy implications there that the Stackoverflow community is in and of itself the entirety of "the world", I have a question that hopefully should be easy for someone out there to answer. One, because I've been trying to find the answer for many hours, and two because it might be easy reputation points for you!
As I'm new to both Lua and Splash. There is a project on which I've been working in an effort to learn both Lua and Splash. The task was; make a webscraping script returns a JSON/dictionary-style data structure with the {description1: price1, description2: price2} (etc.) of every result on the Amazon's Daily Deals or whatever it's called. I first completed this task in Python and added a pprint statement at the end just so I could verify that the dictionary file it produces is correct, which it was. I'll show you that script here:
import requests
from bs4 import BeautifulSoup
from pprint import pprint
url = "https://www.amazon.com/b/ref=OUT_DQC_D_EN_US?node=17862038011&pf_rd_r=XW466BFCF4YAJBD5KMT7&pf_rd_p=487a9cd3-4b40-4b49-9a2b-ccced663286f&pd_rd_r=2d6aecc4-2464-4258-9e2a-2fc6af779869&pd_rd_w=MwTPT&pd_rd_wg=poTGp&ref_=pd_gw_unk&nocache=1629088796366"
# Returns the 20 results on the first page
def get_amazon_results(url):
r = requests.get('http://localhost:8050/render.html', params={'url': url, 'wait': 2})
soup = BeautifulSoup(r.text, 'html.parser')
results = soup.find_all(id="dealTitle")
prices = soup.find_all(class_='gb-font-size-medium inlineBlock unitLineHeight dealPriceText')
results_dict = dict()
for i in range(len(results)):
# Get the description
result = str(results[i])
separators = result.split(' ')
result = separators[-5].replace('\n', '')
# Get the price
price = str(prices[i])
price = price.replace('<span class="gb-font-size-medium inlineBlock unitLineHeight dealPriceText">', '')
price = price.replace('</span>', '')
price = price.replace('$', '')
# In the case of a price range, put min/max in an array
if '-' in price:
price = price.split(' - ')
for i in range(len(price)):
price[i] = float(price[i])
else:
price = float(price)
results_dict[result] = price
return results_dict
if __name__ == '__main__':
amazon_results = get_amazon_results(url)
pprint(amazon_results)
So I went to do this now with Lua and Splash, and I believe that I created a script that does this... but I can't really tell. I'm having difficulty confirming if the object I created is correct, because when I run the code on the Splash interface, it just shows: 
Which is technically great? I suppose I did want an object, but I want to actually see the object.
Here is the Lua script that produces that result (very similar to the Python script, you may notice):
function main(splash)
local url = "https://www.amazon.com/b/ref=OUT_DQC_D_EN_US?node=17862038011&pf_rd_r=XW466BFCF4YAJBD5KMT7&pf_rd_p=487a9cd3-4b40-4b49-9a2b-ccced663286f&pd_rd_r=2d6aecc4-2464-4258-9e2a-2fc6af779869&pd_rd_w=MwTPT&pd_rd_wg=poTGp&ref_=pd_gw_unk&nocache=1629088796366"
local ok = splash:go(url)
local results = {}
if ok then
-- Give the elements and page data 2secs to load...
splash:wait(2)
local descriptions = splash:select_all("#dealTitle")
local prices = splash:select_all(".gb-font-size-medium inlineBlock unitLineHeight dealPriceText")
for i in pairs(descriptions) do
results[descriptions[i]] = prices[i]
end
else return nil
end
return results
end
Note: The length of the arrays I get in Lua, 15, is the same length as the arrays and results I produced in Python
Is there a way within the Splash interface to just print(results)? Or at least display them on the screen in a way where I could see the plain text of the contents? I've been able to see, for example, the text of a single description within the descriptions array, but when I return anything other than text or a string, it'll just show me "array" or "object". I can also get it to give me numbers, like the length of an array, but I just want to see the dang object it created! Any help, advice, or pointing in the correct direction, would be highly appreciated! Thank you for taking the time to read this, I talk too much and I am aware that this might be some simple solution that I'm overlooking.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
