What is the intuition behind using value objects in DDD when storing the references (ids) to other Aggregates

Question

Let me preface this by apologising if this was asked before (I could not find a similar question when doing a quick search).

The DDD pattern is centered around defining and isolating aggregates to split the business complexity into a more manageable chunks, and it is strictly forbidden for an aggregate to hold any kind of relation to other aggregates. The references to the other aggregates are instead to be stored as ids that can then be used to fetch the other aggregates on demand. Therefore a single aggregate would only contain its properties, value objects and references to other aggregates.

To use a common example (User, Order) it would look as such:

public class User {
  private Long id;
  private List<Long> orders;
}

and

public class Order {
  private Long id;
  private Long userId;
}

However I have seen multiple sources use another layer of encapsulation for the aggregate references, turning them from the property types (as defined in the example above) into value objects like shown below:

public class User {
  private Long id;
  private List<OrderId> orders;
}

and

public class Order {
  private Long id;
  private UserId userId;
}

I am rather new to DDD so I want to understand the benefit of doing so when working with non-composite ids. On the first glance I see a lot of pretty obvious drawbacks (or such they seem to me!), like explosion in quantity of common base types, serialization issues, extra complexity when working with the code and accessing the values stored within these holders, however I am sure that it would not be done so without a very good reason that I am overlooking somewhere.

Any comments, thoughts or feedback would be very welcome!