Fleuntd Masking private data with in the log

Question

[2018-09-20 08:25:21,635: ERROR/MainProcess] test[b48600fc-70d1-4f44-9b7a-1988c3ca5cc4] raised unexpected: MaxRetriesExceededError("Can't retry test.api.tasks.send_webhook[b43-23-24442-3sfa-23] args:(u'http://test.com/private_token=0101032030301', {'content-type': 'application/json'}, {'date_created': '2018-09-19T22:33:57.602599', 'type': 'foo.created', 'data': {'status': 'settled', 'date_completed': '2018-09-20T02:25:17', 'target': {'merchant': None, 'redeemable_target': None, 'phone': None, 'user': {'username': u'tato', 'about': ' ', 'last_name': u'Teette', 'display_name': u'ttead', 'test_count': 90, 'is_active': True, 'is_group': False, 'trust_request': None, 'id': '0000000100203', 'phone': None, 'profile_picture_url': u'https://test.test.com/u/', 'first_name': u'Test', 'friend_status': None, 'is_blocked': False, 'email': None, 'identity': None, 'date_joined': '2015-08-19T11:16:36'}, 'type': 'user', 'email': None}, 'audience': u'friends', 'actor': {'username': u'test', 'about': ' ', 'last_name': u'test', 'display_name': u'Michael Bittinger', 'friends_count': 52, 'is_active': True, 'is_group': False, 'trust_request': None, 'id': '39219321939133333', 'phone': None, 'profile_picture_url': u'https://test.teee.com/u/', 'first_name': u'testt', 'ttt_status': None, 'is_blocked': False, 'email': None, 'identity': None, 'date_joined': '2017-09-01T22:20:12'}, 'note': u'\U0001f4bb\U0001f4b8', 'amount': 27.52, 'action': 'hit', 'date_created': '2018-09-20T02:25:17', 'date_reminded': None, 'id': '3213131333323', 'date_authorized': None}}) kwargs:{}",)

Hey guys the above log is what i'm passing via fluentd to Elasticsearch.

I need to be able to filter on "private_token:" and make sure to musk it or replace it with something else so it looks something like this. "http://test.com/private_token=######'" or anything but the actual value with FluentD.

Answer 1

When I wanted to mask password, I added a filter with the following conf in the fluentd conf file.

<filter **>
   @type record_transformer
    enable_ruby
    <record>
        message ${record["message"].gsub(/[Pp]assword*([^,])/,'****')}
     </record>
</filter>