Getting Logs from GCP Alerting

Question

I am setting up alerting for GCP VMs. It works fine for email, but I'm trying to use the webhook option. It shows the incident was caught and the webhook triggered. But I don't see an alert on the receiving end. I don't know how to debug since GCP webhooks seems like a blackbox. Does anybody know where I can see the log for the actual webhook call? I'm not sure it is receiving an alert ID from the webhook call.

I'm using this document:

https://cloud.google.com/monitoring/alerts/using-channels-api#api-create-channels

Thanks!

Gary

Answer 1

You configure a webhook notification channel and expect to be notified when incidents occur, but you might not receive any notifications because of following reasons:

1.Private endpoint

You can't use webhooks for notifications unless the endpoint is public. To resolve this situation, use Pub/Sub notifications combined with a pull subscription to that notification topic.

When you configure a Pub/Sub notification channel, incident notifications are sent to a Pub/Sub queue that has Identity and Access Management controls. Any service that can query for, or listen to, a Pub/Sub topic can consume these notifications. For example, applications running on App Engine, Cloud Run, or Compute Engine virtual machines can consume these notifications.

2.Public endpoint

To identify why the delivery failed, examine your Cloud Logging log entries for failure information.

For example, you can search for log entries for the notification channel resource by using the Logs Explorer, with a filter like the following:

resource.type="stackdriver_notification_channel"

NOTE : Also check whether you have been mentioned in the recipient list, if not mentioned you will not be able to see the alert on the receiving end.

Refer Troubleshooting Alerting policies for information.