creating export task in aws cloudwatch log

Question

I am trying to create export task for moving the cloudwatch logs more than 30 day to move to s3 bucket. I am currently following this AWS article https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3ExportTasks.html. I am stuck with the creation of export task. when I ran the below command

 aws logs create-export-task --profile CWLExportUser --task-name "my-log-group-$(date +%Y-%m-%d)" --log-group-name "/aws/lambda/AMICreate" --from  60  --to  155520000 --destination "diego-cw-logs" --destination-prefix "export-task-output"

I am getting below error

An error occurred (InvalidParameterException) when calling the CreateExportTask operation: Specified time range of '60' and '155520000' is not valid. Please make sure the values are within the retention period of the log groups and from value is lesser than the to values

I am missing something. It would be great if some one render their hands to fix the issue.

Warmly, Muneesh

Answer 1

Use https://currentmillis.com/ to convert time in milisecond. Below is python code and use it with eventbridge to get transfer log as per your need.

import json import boto3 import botocore from datetime import datetime, timedelta

def hello(event, context):

# Date calculated in milisecond
time_seven_days_ago = (datetime.now() + timedelta(days=-103)).timestamp()
time_six_days_ago = (datetime.now() + timedelta(days=-102)).timestamp()
print(int(time_seven_days_ago)*1000) #printing to CloudWatch
print(int(time_six_days_ago)*1000) #printing to CloudWatch

#Boto3 objecets for CW and SNS
client = boto3.client('sns')
cli = boto3.client('logs')

try:
    response = cli.create_export_task(
    taskName='task-name',logGroupName='cloudwatch-log-group-name', fromTime=int(time_seven_days_ago)*1000, to=int(time_six_days_ago)*1000,  destination='s3-bucket-name',   destinationPrefix='its-your-wish')
    print(response)
    client.publish(
        TopicArn='topic--arn',
        #Sending Error in Email
        Message='Task completed',
        Subject='Create Task Job Status',
        MessageStructure='Create Task Job status'
        )

except BaseException as error:
    print('Error From Lambda Function', error) #Printing Error to CloudWatch
    try:
        client.publish(
        TopicArn='topic-arn',
        #Sending Error in Email
        Message=error,
        Subject='Create Task Job Failed',
        MessageStructure='Create Task Job Failed'+error
        )
    except BaseException as fail:
        print('Error while sending email', fail)

Answer 2

--to and --from should be:

The start time of the range for the request, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC

In your case you are using --from 60 --to 155520000 which means that you want to export values from 60 millseconds after 1970 00:00:00 to 155520000 milliseconds after (~two days) after. Obviously this does not make sense.

So basically you have to provide correct timestamps in milliseconds for the range you want to use.