Linux File Monitoring

You can configure MetricsHub to monitor files on Linux systems.

In the example below, we configured MetricsHub to:

monitor all files matching the path pattern /var/log/myapp/*.log
search for the patterns "Error", "Exception", "Failure"
count how many times these patterns occur
expose the result through the system.file.match.count metric.

This use case allows you to track:

the number of errors in log files
the number of exceptions
the number of specific events or keywords.

Procedure

To achieve this use case:

Declare the resource to be monitored (prod-linux-web) and its attributes (host.name, host.type)

resources:
  prod-linux-web:
    attributes:
      host.name: prod-linux-web
      host.type: linux

Configure the SSH protocol with credentials and timeout

    protocols:
      ssh:
        username: USERNAME
        password: PASSWORD
        timeout: 240

Important: Monitoring large files may take time. Specify a sufficient timeout (for example, 240 seconds) to prevent connection timeouts.

Configure the monitor job to target the desired files
```
    monitors:
      file:
        simple:
```

Configure the File source

          sources:
            # FileSource: one row per file [path, content]; line is "path;content;" (path can contain spaces).
            fileSource:
              type: file
              paths: /var/log/myapp/*.log # Regular expression matching the file paths to monitor
              mode: log # File fetching mode. Use log for log file monitoring
              maxSizePerPoll: 1MB # Maximum size (in MB) to read per polling cycle.

Create an awk script to count how many times specific patterns appear in each file

              computes:
                - type: awk # The awk script processes each file and counts how many times the content pattern is found
                  script: |
                    BEGIN { pattern = "Error|Exception|Failure" } # pattern can be a single string ("Error") or a regular expression ("Error|Exception|Failure").
                    index($0, ";") > 0 {
                      path = substr($0, 1, index($0, ";") - 1)
                      content = substr($0, index($0, ";") + 1)
                      if (substr(content, length(content), 1) == ";") content = substr(content, 1, length(content) - 1)
                      if (pattern == "" || content == "") count = 0; else count = gsub(pattern, "&", content)
                      print path ";" count # Use an empty string "" to skip matching and return 0.
                    }

Define the identification attributes

          mapping:
            # Mapping is executed on the result produced by the source (after computes are applied).
            source: ${source::fileSource}
            attributes:
              id: $1
              system.file.path: $1
              system.file.keyword: Error|Exception|Failure

Expose the number of matches using the system.file.match.count metric

            metrics:
              # Emit a datapoint per file: number of pattern matches found in each file.
              system.file.match.count: $2

Here is the complete YAML configuration:

resources:
  prod-linux-web:
    attributes:
      host.name: prod-linux-web
      host.type: linux
    protocols:
      ssh:
        username: USERNAME
        password: PASSWORD
        timeout: 240
    monitors:
      file:
        keys:
          - id
          - system.file.keyword
        simple:
          sources:
            # FileSource: one row per file [path, content]; line is "path;content;" (path can contain spaces).
            fileSource:
              type: file
              paths: /var/log/myapp/*.log # Glob pattern matching the file paths to monitor
              mode: log # File fetching mode. Use log for log file monitoring
              maxSizePerPoll: 1MB # Maximum size (in MB) to read per polling cycle.
              computes:
                - type: awk # The awk script processes each file and counts how many times the content pattern is found
                  script: |
                    BEGIN { pattern = "Error|Exception|Failure" }  # pattern can be a single string ("Error") or a regular expression ("Error|Exception|Failure").
                    index($0, ";") > 0 {
                      path = substr($0, 1, index($0, ";") - 1)
                      content = substr($0, index($0, ";") + 1)
                      if (substr(content, length(content), 1) == ";") content = substr(content, 1, length(content) - 1)
                      if (pattern == "" || content == "") count = 0; else count = gsub(pattern, "&", content) # Use an empty string "" to skip matching and return 0.
                      print path ";" count
                    }
          mapping:
            # Mapping is executed on the result produced by the source (after computes are applied).
            source: ${source::fileSource}
            attributes:
              id: $1
              system.file.path: $1
              system.file.keyword: Error|Exception|Failure
            metrics:
              # Emit a datapoint per file: number of pattern matches found in each file.
              system.file.match.count: $2

Procedure​

Supporting Resources​

Procedure

Supporting Resources