'Saltstack - File Searialization - Keep Empty lines, File content sequence and Comments intact
I am trying to use salt file serialization, but it is removing all empty lines, sorting the complete file in alphabetical order and removing all comments. I dont want these to happen. Tried to check here and here, but could not find anything.
{% import_yaml "app-filebeat.yml" as config %}
manage_file:
file.managed:
- name: /etc/filebeat/filebeat.yml
- source: salt://filebeat.yml.tmpl
- template: jinja
conf_file:
file.serialize:
- name: /etc/filebeat/filebeat.yml
- dataset: {{ config }}
- formatter: yaml
- merge_if_exists: true
app_filebeat.yml:
output.logstash:
hosts: ['myhost.example.com:5158']
filebeat.yml.tmpl:
# Filebeat prospector configuration
filebeat:
config:
inputs:
enabled: true
path: /opt/filebeat/prospector-conf/*.yml
reload.enabled: true
reload.period: 300s
# Logstash configuration
output.logstash:
hosts:
worker: 1
compression_level: 3
loadbalance: true
ssl:
certificate: /usr/share/filebeat/file_beat.crt
key: /usr/share/filebeat/file_beat.key
verification_mode: none
# Filebeat Logging Configuration
logging:
level: debug
to_files: true
files:
path: /var/tellme/log/filebeat/
name: filebeat.log
rotateeverybytes: 10485760
keepfiles: 7
Ouput:
filebeat:
config:
inputs:
enabled: true
path: /opt/filebeat/prospector-conf/*.yml
reload.enabled: true
reload.period: 300s
logging:
files:
keepfiles: 7
name: filebeat.log
path: /var/tellme/log/filebeat/
rotateeverybytes: 10485760
level: debug
to_files: true
output.logstash:
compression_level: 3
hosts:
- myhost.example.com:5158
loadbalance: true
ssl:
certificate: /usr/share/filebeat/file_beat.crt
key: /usr/share/filebeat/file_beat.key
verification_mode: none
worker: 1
Salt State output:
----------
ID: manage_file
Function: file.managed
Name: /etc/filebeat/filebeat.yml
Result: True
Comment: File /etc/filebeat/filebeat.yml updated
Started: 04:53:53.678745
Duration: 118.489 ms
Changes:
----------
diff:
---
+++
@@ -0,0 +1,29 @@
+# Filebeat prospector configuration
+filebeat:
+ config:
+ inputs:
+ enabled: true
+ path: /opt/filebeat/prospector-conf/*.yml
+ reload.enabled: true
+ reload.period: 300s
+
+# Logstash configuration
+output.logstash:
+ hosts:
+ worker: 1
+ compression_level: 3
+ loadbalance: true
+ ssl:
+ certificate: /usr/share/filebeat/file_beat.crt
+ key: /usr/share/filebeat/file_beat.key
+ verification_mode: none
+
+# Filebeat Logging Configuration
+logging:
+ level: debug
+ to_files: true
+ files:
+ path: /var/tellme/log/filebeat/
+ name: filebeat.log
+ rotateeverybytes: 10485760
+ keepfiles: 7
----------
ID: conf_file
Function: file.serialize
Name: /etc/filebeat/filebeat.yml
Result: True
Comment: File /etc/filebeat/filebeat.yml updated
Started: 04:53:53.797737
Duration: 14.986 ms
Changes:
----------
diff:
---
+++
@@ -1,4 +1,3 @@
-# Filebeat prospector configuration
filebeat:
config:
inputs:
@@ -6,24 +5,21 @@
path: /opt/filebeat/prospector-conf/*.yml
reload.enabled: true
reload.period: 300s
-
-# Logstash configuration
+logging:
+ files:
+ keepfiles: 7
+ name: filebeat.log
+ path: /var/tellme/log/filebeat/
+ rotateeverybytes: 10485760
+ level: debug
+ to_files: true
output.logstash:
- hosts:
- worker: 1
compression_level: 3
+ hosts:
+ - myhost.example.com:5158
loadbalance: true
ssl:
certificate: /usr/share/filebeat/file_beat.crt
key: /usr/share/filebeat/file_beat.key
verification_mode: none
-
-# Filebeat Logging Configuration
-logging:
- level: debug
- to_files: true
- files:
- path: /var/tellme/log/filebeat/
- name: filebeat.log
- rotateeverybytes: 10485760
- keepfiles: 7
+ worker: 1
Please help.
Solution 1:[1]
The file.serialize module does not look like the right choice for this requirement.
You already have a template for filebeat.yml, and the requirement is to modify this file in-place and update only hosts under output.logstash: section.
For this the file.accumulated might be better with changes as below:
filebeat.yml.tmpl:
# Logstash configuration
output.logstash:
hosts:
{%- for host in accumulator['extra_config'] %}
- {{ host }}
{%- endfor %}
app-filebeat.yml:
hosts: ['myhost.example.com:5158']
Then the below states:
{% import_yaml "app-filebeat.yml" as config %}
extra_config:
file.accumulated:
- filename: /etc/filebeat/filebeat.yml
- text: {{ config.hosts }}
- require_in:
- file: manage_file
manage_file:
file.managed:
- name: /etc/filebeat/filebeat.yml
- source: salt://filebeat.yml.tmpl
- template: jinja
Solution 2:[2]
import_yaml and merge_if_exists load YAML as a python data structure. All formatting, ordering, and comments are lost. This is intentional, because they are irrelevant to the data structure.
If you want to manage the file as a plain text file, then use file.managed. If you want to manage it as structured data, then you don't get to choose the presentation details. Don't try to do both to the same file.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | |
| Solution 2 |