On This Page

Home / Edge/ Integrations/ Common Data Flows/Collect CSV Files with File Monitor

Collect CSV Files with File Monitor ​

Configure the File Monitor Source to successfully collect CSV files.

CSV files, due to their structure which includes file headers that contain crucial information for data parsing, require specific configuration to be successfully collected with the File Monitor Source.

For a full description and reference of the Source’s settings, see File Monitor Source.

Create a New File Monitor Source ​

First, create a File Monitor Source dedicated to collecting CSV files.

  1. On the top bar, select Products, and then select Edge. Under Fleets, select a Fleet.
  2. Select More > Sources. Select File Manager.
  3. Next, select Add Source.
  4. Enter a unique identifier for the Source in Input ID.
  5. Make sure Discovery Mode is set to Manual.
  6. In Search path, provide the path to the directory you want to discover. The path can’t contain wildcards.
  7. In Filename allowlist, indicate CSV files so other file types don’t get collected by mistake. For example, if the search path is /tmp/csv/, set Filename allowlist to /tmp/csv/*.csv.

This basic configuration lets you collect files from the selected path. Now, fine-tune the Source settings to ensure CSV files are collected correctly.

Configure File Monitor to Collect CSV Files ​

  1. In File Monitor General Settings, disable Collect from end.

    This ensures that if collection of a specific file is interrupted, File Monitor will not lose track of the file header.

  2. In Processing Settings > Event Breakers, select Add Ruleset and choose the CSV Ruleset.

  3. Ensure that Disable Event Breaker buffer timeout is toggled on.

    This protects the File Monitor from losing track of the file header state due to timeout.

  4. In Advanced Settings, set Hash length to be higher than the predicted length of the CSV file header (however, do not exceed 16384).

    This ensures that File Monitor correctly tracks the state for each individual CSV file and doesn’t confuse files with similar headers.

  5. Additionally, or if setting Hash length is not enough, enable Salt file hash so that each file is given a unique hash based on the file name.

With this configuration, File Monitor will recognize individual CSV files, correctly recognize their headers, and track them even after collection is stopped and resumed.