'Scrapy + splash: can't select element

I'm learning to use scrapy with splash. As an exercise, I'm trying to visit https://www.ubereats.com/stores/, click on the address text box, enter a location and then press the Enter button to move to next page containing the restaurants available for that location. I have the following lua code:

function main(splash)
  local url = splash.args.url
  assert(splash:go(url))
  assert(splash:wait(5))

  local element = splash:select('.base_29SQWm')
  local bounds = element:bounds()
  assert(element:mouseclick{x = bounds.width/2, y = bounds.height/2})
    assert(element:send_text("Wall Street"))
  assert(splash:send_keys("<Return>"))
  assert(splash:wait(5))

  return {
  html = splash:html(),
  }
end

When I click on "Render!" in the splash API, I get the following error message:

  {
      "info": {
          "message": "Lua error: [string \"function main(splash)\r...\"]:7: attempt to index local 'element' (a nil value)",
          "type": "LUA_ERROR",
          "error": "attempt to index local 'element' (a nil value)",
          "source": "[string \"function main(splash)\r...\"]",
          "line_number": 7
      },
      "error": 400,
      "type": "ScriptError",
      "description": "Error happened while executing Lua script"
  }

Somehow my css expression is false, resulting in splash trying to access an element that is undefined/nil! I've tried other expressions, but I can't seem to figure it out!

Q: Does anyone know how to solve this problem?

EDIT: Even though I still would like to know how to actually click on the element, I figured out how to get the same result by just using keys:

function main(splash)
    local url = splash.args.url
    assert(splash:go(url))
    assert(splash:wait(5))
    splash:send_keys("<Tab>")
    splash:send_keys("<Tab>")
    splash:send_text("Wall Street, New York")
    splash:send_keys("<Return>")
    assert(splash:wait(10))

    return {
    html = splash:html(),
    png = splash:png(),
    }
  end

However, returned html/images in the splash API are from the page where you enter the address, not the page that you see after you've entered your address and clicked enter.

Q2: How do I succesfully load the second page?



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source