'How to configure Prometheus outside of Kubernetes - Elastic Kubernetes Service

I need help configuring Prometheus outside of EKS Cluster?

What I have done:

  1. Created an EKS Cluster,
  2. Installed Prometheus in EC2 outside of EKS.
  3. Added IAM Role with EKS full access to EC2. (Able to access the cluster from aws-cli)
  4. Configured aws-auth configmap to include the above IAM role, can get the /healthz of EKS with IAM Role's token attached to the EC2 using
    curl -k https://kubeapi-url/healthz --header "Authorization: Bearer $TOKEN"

Issue:
Token has an expiration time of 15mins by default(I don't need to change this for security reasons). So, I can not use this Token directly inside the prometheus.yml config.

What do I need:

  1. How can I access EKS without Token and Kubectl?
  2. How to configure prometheus.yml to make it automatically obtain Token just like the Kubeconfig file of EKS does with
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - --region
      - ap-south-1
      - eks
      - get-token
      - --cluster-name
      - eks-cluster-name
      command: aws
  1. Or is there any other way to configure Prometheus without the kubernetes token?

(Prometheus config with my requirement for EKS is Greatly appreciated.)

kubernetesprometheusamazon-iamkubectlamazon-eks


Solution 1:[1]

Just for the 2nd question:

Since you have an automatically renewed Kubeconfig file, in prometheus.yml configuration file you could use it directly.

# Optional path to a kubeconfig file.
# Note that api_server and kube_config are mutually exclusive.
[ kubeconfig_file: <filename> ]

Not sure this is useful for your case.

Solution 2:[2]

Not the exact answer, but you can use Prometheus federation.

  1. Setup Prometheus inside the cluster, with default config, and it automatically scrapes everything. Not much configuration required.
  2. Setup the prometheus on EC2 Box and use federation config as mentioned here
scrape_configs:
  - job_name: 'federate'
    scrape_interval: 15s

    honor_labels: true
    metrics_path: '/federate'

    params:
      'match[]':
        - '{job="prometheus"}'
        - '{__name__=~"job:.*"}'

    static_configs:
      - targets:
        - '<k8s-prometheus-endpoint>'
        - '<someother-prometheus-endpoint>'

There won't be any token used, so comparatively much more secure. All using k8s serviceaccount and federation. Consumes lot more resources but still we get sort-of HA aswell.

For optimisation: You can reduced the data in k8s prometheus for 1-2 day(depends) and have data for longer duration in your EC2 prometheus.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 YwH
Solution 2 Sumit Murari

Related Questions