Garbage Collection(GC) Issue while running code on databricks

Question

Getting the below log on GC issue.

The Spark Configuration Settings are
spark.driver.extraJavaOptions -XX:+DisableExplicitGC -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:G1ReservePercent=20

spark.executor.extraJavaOptions -XX:+DisableExplicitGC -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:G1ReservePercent=20

Cluster Settings 14GB Memory, 4 Cores

Logs :

2022-05-12T04:14:08.479+0000: [GC pause (G1 Evacuation Pause) (young), 0.0058750 secs]
   [Parallel Time: 4.6 ms, GC Workers: 4]
      [GC Worker Start (ms): Min: 510868.9, Avg: 510868.9, Max: 510869.0, Diff: 0.1]
      [Ext Root Scanning (ms): Min: 2.2, Avg: 2.8, Max: 4.4, Diff: 2.3, Sum: 11.2]
      [Update RS (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.3]
         [Processed Buffers: Min: 0, Avg: 1.5, Max: 3, Diff: 3, Sum: 6]
      [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
      [Object Copy (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.3, Sum: 0.7]
      [Termination (ms): Min: 0.0, Avg: 1.4, Max: 1.8, Diff: 1.8, Sum: 5.4]
         [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
      [GC Worker Total (ms): Min: 4.4, Avg: 4.4, Max: 4.4, Diff: 0.0, Sum: 17.7]
      [GC Worker End (ms): Min: 510873.4, Avg: 510873.4, Max: 510873.4, Diff: 0.0]
   [Code Root Fixup: 0.0 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.1 ms]
   [Other: 1.2 ms]
      [Choose CSet: 0.0 ms]
      [Ref Proc: 0.3 ms]
      [Ref Enq: 0.0 ms]
      [Redirty Cards: 0.1 ms]
      [Humongous Register: 0.2 ms]
      [Humongous Reclaim: 0.1 ms]
      [Free CSet: 0.0 ms]
   [Eden: 0.0B(312.0M)->0.0B(312.0M) Survivors: 0.0B->0.0B Heap: 6232.2M(6260.0M)->6232.2M(6260.0M)]
[Times: user=0.02 sys=0.00, real=0.00 secs]
2022-05-12T04:14:08.486+0000: [Full GC (Allocation Failure)  6232M->6232M(6260M), 13.1145837 secs]
   [Eden: 0.0B(312.0M)->0.0B(312.0M) Survivors: 0.0B->0.0B Heap: 6232.2M(6260.0M)->6232.2M(6260.0M)], [Metaspace: 226660K->226660K(1265664K)]
[Times: user=20.55 sys=0.06, real=13.12 secs]
2022-05-12T04:14:21.601+0000: [Full GC (Allocation Failure)  6232M->6232M(6260M), 12.9015824 secs]
   [Eden: 0.0B(312.0M)->0.0B(312.0M) Survivors: 0.0B->0.0B Heap: 6232.2M(6260.0M)->6232.2M(6260.0M)], [Metaspace: 226660K->226660K(1265664K)]
[Times: user=20.46 sys=0.04, real=12.90 secs]
2022-05-12T04:14:34.503+0000: [GC pause (G1 Evacuation Pause) (young), 0.0059072 secs]
   [Parallel Time: 4.6 ms, GC Workers: 4]
      [GC Worker Start (ms): Min: 536892.9, Avg: 536893.0, Max: 536893.0, Diff: 0.1]
      [Ext Root Scanning (ms): Min: 2.2, Avg: 2.8, Max: 4.5, Diff: 2.3, Sum: 11.3]
      [Update RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.1]
         [Processed Buffers: Min: 0, Avg: 0.2, Max: 1, Diff: 1, Sum: 1]
      [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
      [Object Copy (ms): Min: 0.0, Avg: 0.2, Max: 0.3, Diff: 0.3, Sum: 0.6]
      [Termination (ms): Min: 0.0, Avg: 1.4, Max: 2.0, Diff: 2.0, Sum: 5.8]
         [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
      [GC Worker Total (ms): Min: 4.5, Avg: 4.5, Max: 4.5, Diff: 0.0, Sum: 17.9]
      [GC Worker End (ms): Min: 536897.4, Avg: 536897.4, Max: 536897.5, Diff: 0.0]
   [Code Root Fixup: 0.0 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.1 ms]
   [Other: 1.1 ms]
      [Choose CSet: 0.0 ms]
      [Ref Proc: 0.3 ms]
      [Ref Enq: 0.0 ms]
      [Redirty Cards: 0.1 ms]
      [Humongous Register: 0.2 ms]
      [Humongous Reclaim: 0.1 ms]
      [Free CSet: 0.0 ms]
  
 [Eden: 0.0B(312.0M)->0.0B(312.0M) Survivors: 0.0B->0.0B Heap: 6232.2M(6260.0M)->6232.2M(6260.0M)]
[Times: user=0.02 sys=0.00, real=0.00 secs]
2022-05-12T04:14:34.510+0000: [GC pause (G1 Evacuation Pause) (young) (initial-mark), 0.0069220 secs]
   [Parallel Time: 5.7 ms, GC Workers: 4]
     
 [GC Worker Start (ms): Min: 536900.4, Avg: 536900.4, Max: 536900.4, Diff: 0.1]
      [Ext Root Scanning (ms): Min: 4.2, Avg: 4.4, Max: 4.6, Diff: 0.4, Sum: 17.4]
      [Update RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.1]
         [Processed Buffers: Min: 0, Avg: 0.5, Max: 1, Diff: 1, Sum: 2]
      [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
      [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
      [Object Copy (ms): Min: 0.0, Avg: 0.3, Max: 0.5, Diff: 0.5, Sum: 1.1]
      [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.2]
         [Termination Attempts: Min: 1, Avg: 1.0, Max: 1, Diff: 0, Sum: 4]
      [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1]
      [GC Worker Total (ms): Min: 4.7, Avg: 4.7, Max: 4.7, Diff: 0.0, Sum: 18.9]
      [GC Worker End (ms): Min: 536905.1, Avg: 536905.1, Max: 536905.1, Diff: 0.0]
   [Code Root Fixup: 0.0 ms]
   [Code Root Purge: 0.0 ms]
   [Clear CT: 0.1 ms]
   [Other: 1.1 ms]
      [Choose CSet: 0.0 ms]
      [Ref Proc: 0.3 ms]
      [Ref Enq: 0.0 ms]
      [Redirty Cards: 0.1 ms]
      [Humongous Register: 0.1 ms]
      [Humongous Reclaim: 0.1 ms]
      [Free CSet: 0.0 ms]   
[Eden: 0.0B(312.0M)->0.0B(312.0M) Survivors: 0.0B->0.0B Heap: 6232.2M(6260.0M)->6232.2M(6260.0M)]
[Times: user=0.02 sys=0.00, real=0.01 secs]
2022-05-12T04:14:34.517+0000: [GC concurrent-root-region-scan-start]
2022-05-12T04:14:34.517+0000: [GC concurrent-root-region-scan-end, 0.0003873 secs]
2022-05-12T04:14:34.517+0000: [GC concurrent-mark-start]
2022-05-12T04:14:34.518+0000: [Full GC (Allocation Failure)  6232M->6232M(6260M), 13.0701579 secs]

   [Eden: 0.0B(312.0M)->0.0B(312.0M) Survivors: 0.0B->0.0B Heap: 6232.2M(6260.0M)->6232.2M(6260.0M)], [Metaspace: 226660K->226660K(1265664K)]
[Times: user=20.59 sys=0.01, real=13.07 secs]
2022-05-12T04:14:47.588+0000: [Full GC (Allocation Failure)