Unable to register node "node-name" with API server: Unauthorized

Question

I am using kubernetes 1.13.2 on bare metals (No Provider).

I already have a master and a worker node set up while ago, but now my new worker node cannot join to the cluster and receives "Unauthorized" message when it tries to register

I have renewed my token on my master, and created a new join command. But still getting "Unauthorized" response upon joining

After sending kubeadm join ... command, it times out

[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "node-name" as an annotation
[kubelet-check] Initial timeout of 40s passed.
error uploading crisocket: timed out waiting for the condition

and here is what I get in journalctl -u kubelet

Apr 22 20:31:13 node-name kubelet[18567]: I0422 20:31:13.399059   18567 kubelet_node_status.go:278] Setting node annotation to enable volume controller attach/detach
Apr 22 20:31:13 node-name kubelet[18567]: I0422 20:31:13.404930   18567 kubelet_node_status.go:72] Attempting to register node node-name
Apr 22 20:31:13 node-name kubelet[18567]: E0422 20:31:13.406863   18567 kubelet_node_status.go:94] Unable to register node "node-name" with API server: Unauthorized
Apr 22 20:31:13 node-name kubelet[18567]: E0422 20:31:13.407096   18567 event.go:203] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"node-name.1597fce5edba5ee6", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"node-name", UID:"node-name", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientMemory", Message:"Node node-name status is now: NodeHasSufficientMemory", Source:v1.EventSource{Component:"kubelet", Host:"node-name"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf27bf9d2c75d6e6, ext:897526251, loc:(*time.Location)(0x71d3440)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf27bfa05821f203, ext:13556483910, loc:(*time.Location)(0x71d3440)}}, Count:8, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'Unauthorized' (will not retry!)
Apr 22 20:31:13 node-name kubelet[18567]: E0422 20:31:13.409745   18567 event.go:203] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"node-name.1597fce5edba8b6c", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"node-name", UID:"node-name", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientPID", Message:"Node node-name status is now: NodeHasSufficientPID", Source:v1.EventSource{Component:"kubelet", Host:"node-name"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf27bf9d2c76036c, ext:897537648, loc:(*time.Location)(0x71d3440)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf27bfa0582242b8, ext:13556504573, loc:(*time.Location)(0x71d3440)}}, Count:8, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'Unauthorized' (will not retry!)
Apr 22 20:31:13 node-name kubelet[18567]: E0422 20:31:13.476603   18567 kubelet.go:2266] node "node-name" not found
Apr 22 20:31:13 node-name kubelet[18567]: E0422 20:31:13.576911   18567 kubelet.go:2266] node "node-name" not found
Apr 22 20:31:13 node-name kubelet[18567]: E0422 20:31:13.630766   18567 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Unauthorized
Apr 22 20:31:13 node-name kubelet[18567]: E0422 20:31:13.631616   18567 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Unauthorized
Apr 22 20:31:13 node-name kubelet[18567]: E0422 20:31:13.632799   18567 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Unauthorized

Answer 1

The issue was caused by Docker and/or Kubernetes version mismatch between Kubernetes nodes.
The problem was resolved after reinstalling Docker and Kubernetes to the correct versions.

Kubernetes version skew support policy describes the maximum version skew supported between various Kubernetes components. For more information, see the version-skew-policy documentation.

The Kubernetes release notes list which versions of Docker are compatible with that version of Kubernetes. For example in the CHANGELOG-1.13.md you can find validated docker versions for Kubernetes 1.13.