Convert a a dictionary into an array in one line

Question

I'm trying to simplify this:

urls = []
soup = BeautifulSoup(r.text, "html.parser")
items = soup.select('.mydiv a')
for item in items:
    url = item['href']
    urls.append(url)

... into a single line. Why does the following not work?

urls = [].append(item['href'] for item in soup.select('.mydiv a'))

I know it's "less readable", but I want to know for educational purposes.

Answer 1

Why does the following not work?

urls = [].append(item['href'] for item in soup.select('.mydiv a'))

Your code appends a single object to your list, namely, a generator object.

The expression inside the parentheses in append() is a generator expression which yields a generator object. Usually generator expressions need to be enclosed in their own set of parentheses but in the case of one argument in the expression (as in your example), their own set of parentheses is not needed.

The syntax of a comprehension is virtually identical to the syntax for a generator expression, except that generator expressions tend to be enclosed within a pair of parentheses whereas comprehensions are not.

Even though syntactically comprehensions and generator expressions can be identical, whether Python interprets the identical syntax as a comprehension or a generator expression depends on the context.

For example

i for i in range(0, 10)

refers to a comprehension when enclosed by square brackets [ ] or { } (i.e. in list and set displays -- see previous reference):

# Example 1
>>> a = [i for i in range(0, 10)] 
>>> a
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

However, when it appears, say, as the argument to the append() method of lists, the above is interpreted as a generator expression:

# Example 2
>>> b = []
>>> b.append(i for i in range(0, 10)) 
>>> b
[<generator object <genexpr> at 0x7fda557477d0>]

EDIT: to address one of the comments.

By the way, every generator object is an iterator and so anywhere where an iterable is expected, a generator expression can be supplied.

Python list objects have an extend method that accepts one argument, namely, an iterable:

list.extend(iterable)

which "[e]xtend[s] the list by appending all the items from the iterable" (quote from the docs).

Contrast this with

list.append(x)

which "[a]dd[s] an item to the end of the list" (ibid.).

Your original one-liner code that uses the append method can therefore have your desired effect if you were to replace append with extend.

Of course, you should go with the list comprehension, unless you already have a list with elements and you wish to extend it!

Answer 2

List comprehension :

urls = [item['href'] for item in soup.select('.mydiv a')]