How to recover user data when Pod crashes

Question

My application deployed in Pod/Container receive 1000 user data records each time from the queue and process it to store in the respective databases. Once all the 1000 records processed then insert data into the respective database tables at once. Issue - If my application is in between processing these records and the Pod is crashed suddenly due to some reasons. How can I start with newly created Pod by just start processing from 801 record onwards (without starting from the beginning) ?. Please let me know the solution to this problem.

Thanks, Srinivas

Answer 1

Don't use volume type as emptyDir to store data. It will be gone when the pod is restarted or crashed.You need to create a PVC (persistent volume claim) . And bound it in pod. Store the data in mount path where you have mounted your pvc volume. Whenever you are recreating or restarting your pod just bound the pvc with your pod.

please see the documentation for more clearance.

link:

REF

Answer 2

Create a PersistentVolume and PersistentVolumeClaim and then mount the PVC with the Pod. Below is an example:

1. PV creation

apiVersion: v1
kind: PersistentVolume
metadata:
  name: datacachepv-1
  labels:
   data: cache
spec:
  persistentVolumeReclaimPolicy: Delete
  accessModes:
    - ReadWriteMany
  capacity:
    storage: 100Mi
  hostPath:
    path: /tmp/datacache/DBBackup

2. PVC creation

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: datacachepvc-1
  labels:
   data: cache
spec:
  accessModes:
- ReadWriteMany
  resources:
requests:
  storage: 100Mi

3. Mount PVC with the Pod:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    app: testrunner
  name: datacache
spec:
  containers:
  - image: armdocker.rnd.ericsson.se/sandbox/elimixu/datacache:0.3.6 
    name: datacache
    ports:
    - containerPort: 3645 
      hostPort: 3645 
    resources: {}
    volumeMounts:
            - name: datacache-conf
              mountPath: /bin/datacache.conf
              subPath: datacache.conf
              readOnly: true 
            - name: datacachepvc-1
              mountPath: /DBBackup
  volumes:
  - name: datacache-conf
    configMap:
     name: datacache-conf
     items:
     - key: datacache.conf
       path: datacache.conf
  - name: datacache-volume
    persistentVolumeClaim:
      claimName: datacachepvc-1
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}