How to order query results for parent by related field for one-to-many relationship in SQLAlchemy

Question

I have following DB models

class User(Base):
    __tablename__ = "user"

    user_id = Column("id", Integer(), primary_key=True)
    groups = relationship(
        "Group", back_populates="user", lazy="selectin", cascade="all, delete-orphan",
    )


class Group(Base):
    __tablename__ = "group"

    group_id = Column("id", Integer(), primary_key=True)
    user_id = Column(
        Integer,
        ForeignKey("user.id", ondelete="CASCADE"),
        index=True,
    )
    division_id = Column(
        String(96),
        ForeignKey("division.id", onupdate="CASCADE"),
        nullable=False,
    )
      name = Column(String(64), nullable=False)
    user = relationship("User", back_populates="groups", lazy="selectin")
    group = relationship("Division", back_populates="groups", lazy="selectin")


class Division(Base):
    __tablename__ = "division"

    division_id = Column("id", Integer, primary_key=True)
    name = Column(String(64), nullable=False)
    groups = relationship("Group", back_populates="group", lazy="selectin")

I want to fetch all the users ordered by their groups(can be something else as well, need to come from enduser), which I can easily achieve using the following query

session.query(User).join(Group).join(Division).order_by(Group.name).all()

And it might look that this works just fine but it doesn't, because since a user might have multiple groups, so in order to have correct result I first need to sort the groups for each user object i.e. something like sorted(User.group.order_by(Group.name) and then apply the order_by on the User model based on these sorted groups.

And the same thing can apply to division names as well. I know that we can provide default order_by fields while defining the relationship like below but that's not what I want since the order_by field need to come from enduser and can be any other field as well.

groups = relationship("Group", back_populates="user", lazy="selectin", cascade="all, delete-orphan",order_by=("Group.name"))

I can do this at data layer in python but that would not be ideal since there is already some ordering being done at DB layer.

So how can I achieve this at DB layer using SQLAlchemy or even with raw sql. Or is it even possible with sql?