When do I need to use a GeoSeries when creating a GeoDataFrame, and when is a list enough?

Question

import geopandas as gpd
import matplotlib.pyplot as plt
from shapely.geometry import Polygon, Point
import numpy as np

---

I define a polygon:

polygon = Polygon([(0,0),(0,1),(1,1),(1,0)])

and create a list of random points:

np.random.seed(42)
points = [Point([np.random.uniform(low=-1,high=1),
                 np.random.uniform(low=-1,high=1)]) for _ in range(1000)]

I want to know which points are within the polygon. I create a GeoDataFrame with a column called points, by first converting the points list to GeoSeries:

gdf = gpd.GeoDataFrame(dict(points=gpd.GeoSeries(points)))

Then simply do:

gdf.points.within(polygon)

which returns a pandas.core.series.Series of booleans, indicating which points are within the polygon.

However, if I don't create the GeoDataFrame from a list, not a GeoSeries object:

gdf = gpd.GeoDataFrame(dict(points=points))

and then do:

gdf.points.within(polygon)

I get:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-171-831eddc859a1> in <module>()
----> 1 gdf.points.within(polygon)

/usr/local/lib/python3.7/dist-packages/pandas/core/generic.py in __getattr__(self, name)
   5485         ):
   5486             return self[name]
-> 5487         return object.__getattribute__(self, name)
   5488 
   5489     def __setattr__(self, name: str, value) -> None:

AttributeError: 'Series' object has no attribute 'within'

In the examples given on the geopandas.GeoDataFrame page, a GeoDataFrame is create from a list, not a GeoSeries of shapely.geometry.Point objects:

from shapely.geometry import Point
d = {'col1': ['name1', 'name2'], 'geometry': [Point(1, 2), Point(2, 1)]}
gdf = geopandas.GeoDataFrame(d, crs="EPSG:4326")

---

When do I need to convert my lists to GeoSeries first, and when can I keep them as lists when creating GeoDataFrames?