''NoneType' object is not subscriptable error when fetching the value

Editing the whole question and title to give more clarity... I collected a set of rows from a dataframe df using data_collect = df.collect() for sample logic building. There will be millions of rows in the df without filter. The rows in data_collect are

[Row(Origin='USA', YearWeek='202131', L4Weeks=['202132', '202133', '202134', '202135'], Value=2.0),
 Row(Origin='USA', YearWeek='202123', L4Weeks=['202123', '202124', '202125', '202126'], Value=10.0),
 Row(Origin='USA', YearWeek='202129', L4Weeks=['202127', '202128', '202129', '202130'], Value=22.0),
 Row(Origin='USA', YearWeek='202127', L4Weeks=['202127', '202128', '202129', '202130'], Value=9.0),
 Row(Origin='USA', YearWeek='202130', L4Weeks=['202132', '202133', '202134', '202135'], Value=0.0),
 Row(Origin='USA', YearWeek='202150', L4Weeks=['202149', '202150', '202151', '202152'], Value=1.0),
 Row(Origin='USA', YearWeek='202152', L4Weeks=['202149', '202150', '202151', '202152'], Value=1.0),
 Row(Origin='USA', YearWeek='202201', L4Weeks=['202202', '202203', '202204', '202205'], Value=0.0),
 Row(Origin='USA', YearWeek='202125', L4Weeks=['202123', '202124', '202125', '202126'], Value=13.0),
 Row(Origin='USA', YearWeek='202147', L4Weeks=['202145', '202146', '202147', '202148'], Value=0.0),
 Row(Origin='USA', YearWeek='202103', L4Weeks=['202101', '202102', '202103', '202104'], Value=0.0),
 Row(Origin='USA', YearWeek='202128', L4Weeks=['202127', '202128', '202129', '202130'], Value=3.0)]

Now I used the following code block to build my logic

for row in data_collect:
  value_sum = 0
  for week in row['L4Weeks']:
    value_sum += spark.sql("""select value from df where Origin = '{}' and YearWeek = {} """.format(row['Origin'], week)).first()[0]
  print(value_sum )

This is giving me that None subscription error. If I fetch value from df without passing that week filter, it is working fine.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source