Data Collections

Data Collections are the foundation of Octigen's data integration system. They organize your data files and define how those files are identified and accessed.

What is a Data Collection?

A Data Collection is:

• A container for related data files
• A schema defining how files are named and organized
• A connection point between your data and slide templates

Think of it as a smart folder that knows what kind of files belong inside and how to find the right one when asked.

Creating a Data Collection

Navigate to Data Collections

1. Go to your space's main page
2. Click the "Data" tab
3. Click "Add Data Collection"

Basic Information

Collection Name

• Human-readable name for the collection
• Example: "Monthly Portfolio Holdings"

Source Format

• CSV - Comma-separated values; We support various formats such as comma-separated, semicolon-separated and tab-separated; As for encodings, please ensure that your CSV files are UTF-8 encoded.
• Excel (.xlsx, .xls) - Microsoft Excel workbooks with single-sheet or multi-sheet raw data (no calculated columns supported at this point). Legacy .xls exports (including Crystal Reports) are supported.

Source Connection

Currently only one type of source connection is supported: Manual upload.

Data Type

Currently only one data type is supported: Simple 2D table with header row.

Parsing options

Many real-world workbooks include title rows, side margins, or footnotes. Configure how Octigen finds the table:

Setting	Purpose
Sheet	Sheet name or index (default 0) when data is not on the first sheet
Filename pattern	Wildcard for Data Landing Zone matching (e.g. Q*.xlsx)
Skipped header rows	Rows to drop before the header row
Skipped leading columns	Columns to drop from the left before reading the header
Skipped footer rows	Rows to drop from the bottom after the data (footnotes)
Skipped trailing columns	Columns to drop from the right after the data

Parsing order: select sheet → drop leading columns → drop header rows → first remaining row is the header → remaining rows are data → drop footer rows and trailing columns.

Example: A performance file with two title rows above the header uses Skipped header rows = 2. A growth chart with seven empty columns on the left uses Skipped leading columns = 7 and Skipped header rows = 6.

Defining Parameters

Parameters identify individual files within the collection. For each parameter, you define:

Name

• The parameter name (lowercase, use underscores)
• Examples: valuation_date, portfolio_id, version

Type

• date - Date values (represented in YYYY-MM-DD format within Octigen)
• string - Text values (portfolio IDs, names)
• integer - Numeric values (versions, sequence numbers)

Parameter mappings

For each parameter, choose how its value is determined:

• prefix - Parameter value appears at the start of the filename
• suffix - Parameter value appears at the end of the filename
• sheet_name - Parameter value chooses the workbook sheet
• row_value - Parameter value matches a full value in a configured column
• row_prefix - Parameter value matches the first N characters of a configured column

Quick guide:

Mapping	Use when
prefix / suffix	The parameter is encoded in the filename
sheet_name	One workbook file holds many series on different sheets
row_value	One file holds many funds/rows and you filter to one row per parameter set
row_prefix	The parameter is the first characters of a column (e.g. share class from a ticker)

For row_value and row_prefix, set the column name on the parameter. For row_prefix, also set prefix length.

Example: Monthly Sales Data

Settings:

• Collection Name: "Monthly Sales Data"
• Source Format: CSV
• Parameters:
• valuation_date (type: date, mapping: suffix, required: yes)

Expected filenames:

sales_2024-12-01.csv
sales_2024-11-01.csv
sales_2024-10-01.csv

Example: Multi-Portfolio Holdings

Settings:

• Display Name: "Portfolio Holdings"
• Object Type: portfolio-holdings
• File Type: Excel
• Parameters:
• portfolio_id (type: string, mapping: prefix, required: yes)
• valuation_date (type: date, mapping: suffix, required: yes)

Expected filenames:

GROWTH_FUND_2024-12-01.xlsx
INCOME_FUND_2024-12-01.xlsx
GROWTH_FUND_2024-11-01.xlsx

Uploading Data Files

You can upload in two ways:

• On the collection page - Drag files into one collection; filenames are validated against that collection’s parameters.
• Data Landing Zone - Drop many workbooks at once; files auto-match collections by filename pattern. See Data Landing Zone.

Collections can also be edited after creation (sheet, skip settings, parameters, and filename pattern).

Drag and Drop Upload

1. Open the data collection
2. Drag files into the upload area, or click to browse
3. Files are validated against the collection's parameter schema

Upload Validation

When you upload a file, Octigen validates:

• Filename matches the expected pattern
• Parameter values can be extracted
• Parameter types are correct (valid dates, integers)

Validation examples:

Collection Parameters	Filename	Result
valuation_date (suffix, date)	sales_2024-12-01.csv	✅ Valid
valuation_date (suffix, date)	sales_20241201.csv	✅ Valid (alternative format)
valuation_date (suffix, date)	sales_december.csv	❌ Invalid date
portfolio_id (prefix, string), valuation_date (suffix, date)	FUND_A_2024-12-01.csv	✅ Valid

Duplicate Detection

If you upload a file with the same parameter values as an existing file:

• System detects the duplicate
• You can choose to overwrite the existing file
• Or cancel and keep the original

This prevents accidental data loss while allowing intentional updates.

Viewing Collection Contents

File List

The collection detail page shows all uploaded files with:

• Filename
• Parameter values extracted from the filename
• File size
• Upload date / last modified
• Actions (download, delete)

Managing Data Files

Downloading

Click the download icon next to any file to download the original.

Deleting

Click the delete icon to remove a file. This action:

• Cannot be undone
• Affects future presentations only
• Does not affect already-committed presentations

Updating

To update data for the same parameters:

1. Delete previous file
2. Upload the new file with the same filename
3. Confirm you want to overwrite

Parameter Mapping Details

Prefix Mapping

The parameter value appears at the start of the filename, followed by an underscore.

{parameter_value}_{rest_of_filename}.{extension}

Example: portfolio_id with prefix mapping

• Parameter value: GROWTH_FUND
• Filename: GROWTH_FUND_2024-12-01.csv

Suffix Mapping

The parameter value appears at the end of the filename (before extension), preceded by an underscore.

{base_name}_{parameter_value}.{extension}

Example: valuation_date with suffix mapping

• Parameter value: 2024-12-01
• Filename: sales_data_2024-12-01.csv

Combining Mappings

Multiple parameters combine in the filename:

{prefix_param}_{base}_{suffix_param}.{extension}

Example: portfolio_id (prefix) + valuation_date (suffix)

• Parameters: portfolio_id=INCOME, valuation_date=2024-12-01
• Filename: INCOME_holdings_2024-12-01.xlsx

Date Format Flexibility

For date parameters, Octigen recognizes multiple formats in filenames:

Format	Example	Accepted
ISO	2024-12-01	✅
Compact	20241201	✅
Dot separator	2024.12.01	✅
Underscore separator	2024_12_01	✅

All formats are normalized internally, so:

• Upload: sales_20241201.csv
• Query: valuation_date=2024-12-01
• Result: ✅ Match found

Best Practices

Naming Conventions

Collection IDs:

• Be descriptive but concise
• Avoid special characters

Parameter Names:

• Use lowercase with underscores: valuation_date, portfolio_id
• Match industry conventions where applicable
• Be consistent across collections

Filenames:

• Follow the parameter mapping pattern exactly
• Use consistent date formats
• Avoid spaces and special characters

Organization

One data type per collection:

• ✅ "Sales Data" collection with sales files
• ❌ "All Data" collection mixing sales, inventory, and HR files

Logical groupings:

• By report type (monthly sales, quarterly reviews)
• By data domain (portfolio, market, client)
• By update frequency (daily, weekly, monthly)

Data Quality

Before uploading:

• Verify column headers are correct
• Check data completeness and quality

Consistent structure:

• Same columns across files in a collection
• Consistent data types per column
• No extra rows or formatting

Common Issues

"Filename doesn't match parameter pattern"

Cause: The filename doesn't match the expected pattern for the collection's parameters.

Solution:

• Check parameter mappings (prefix vs suffix)
• Verify date format if using date parameters
• Ensure all required parameters are in the filename

"Invalid parameter value"

Cause: A parameter value couldn't be parsed correctly.

Solution:

• For dates: use YYYY-MM-DD or YYYYMMDD format
• For integers: use only digits
• For strings: avoid special characters

"Duplicate file detected"

Cause: A file with the same parameters already exists.

Solution:

• First delete previous version of the data object with same parameters.

Next Steps

• Data Landing Zone - Bulk-upload workbooks into collections
• Transformations - Prepare data for specific slides
• Connect Data to Workflows - Create slide deck workflows that always update certain tables and charts in a deterministic way
• Configure Data for Slides - Connect collections to templates
• Use Data in Presentations - Generate data-driven slides