CLI Reference¶

Complete reference for all Ingenious Fabric Accelerator commands, options, and usage patterns.

Before you begin

Activate your environment and set defaults to simplify commands.

.\.venv\Scripts\Activate.ps1

# Set environment (development, UAT, production)
$env:FABRIC_ENVIRONMENT = "development"

# Set workspace directory 
$env:FABRIC_WORKSPACE_REPO_DIR = "dp"

Command Groups at a Glance¶

Group	Purpose	Common example
`init`	Create projects / configure workspace / generate storage artifacts	`ingen_fab init new --project-name MyProj` `ingen_fab init storage-config`
`ddl`	Compile notebooks from DDL scripts and assist in creating ddl scripts	`ingen_fab ddl compile -o fabric_workspace_repo -g Warehouse`
`deploy`	Deploy, upload libs, extract/compare metadata and dwonload artefacts	`ingen_fab deploy get-metadata --target both -f csv -o meta.csv`
`dbt`	Generate notebooks from dbt outputs and convert metadata to dbt-adapter format	`ingen_fab dbt create-notebooks -p my_dbt_proj`

Global Options¶

These options are available for all commands:

ingen_fab [GLOBAL_OPTIONS] COMMAND [COMMAND_OPTIONS]

Option	Short	Description	Environment Variable
`--fabric-workspace-repo-dir`	`-fwd`	Directory containing fabric workspace repository files	`FABRIC_WORKSPACE_REPO_DIR`
`--fabric-environment`	`-fe`	Target environment (e.g., development, production)	`FABRIC_ENVIRONMENT`
`--help`	-	Show help message	-

Command Groups¶

init¶

Initialize solutions and projects.

`init new`¶

Create a new Fabric workspace project with complete starter template including variable library, sample DDL scripts, and platform manifests.

ingen_fab init new --project-name "My Project"

Options: - --project-name (required): Name of the project - --path: Path where to create the project (default: current directory) - --with-samples: Use sample_project as template instead of project_templates (includes additional sample data and configurations)

What gets created: - Complete variable library with environment-specific configurations - Sample DDL scripts for both Lakehouse (Python) and Warehouse (SQL) - Pre-configured Lakehouse and Warehouse definitions - Platform manifests for deployment tracking - Comprehensive README with setup instructions

Template Options

By default, init new creates a minimal project structure from project_templates. Use --with-samples to include the full sample project with platform manifests and example configurations.

Examples:

# Create a new project with minimal template
ingen_fab init new --project-name "Data Analytics Platform"

# Create project with sample data and configurations
ingen_fab init new --project-name "My Project" --with-samples

# Create project in specific path
ingen_fab init new --project-name "ML Pipeline" --path ./projects

# Create project with samples in specific path
ingen_fab init new --project-name "ML Pipeline" --path ./projects --with-samples

Next steps after creation:

Update variable values in fabric_workspace_items/config/var_lib.VariableLibrary/valueSets/development.json
Generate DDL notebooks: ingen_fab ddl compile --output-mode fabric_workspace_repo --generation-mode Lakehouse
Deploy to Fabric: ingen_fab deploy deploy

`init workspace`¶

Initialize workspace configuration and automatically update artifact GUIDs (lakehouses, warehouses, notebooks).

ingen_fab init workspace

Features: - Sets workspace ID in variable library for the specified environment - Automatically discovers and updates lakehouse, warehouse, and notebook GUIDs - Matches variables ending in _lakehouse_id, _warehouse_id, _notebook_id with their corresponding _name variables

Variable Pattern Matching:

The command automatically updates GUIDs for variables following these naming patterns:

*_lakehouse_id ↔ *_lakehouse_name
*_warehouse_id ↔ *_warehouse_name
*_notebook_id ↔ *_notebook_name

For example, if your development.json contains:

{
  "variableOverrides": [
    {"name": "config_lakehouse_id", "value": "old-guid"},
    {"name": "config_lakehouse_name", "value": "config"},
    {"name": "analysis_notebook_id", "value": "old-guid"},
    {"name": "analysis_notebook_name", "value": "Data Analysis"}
  ]
}

Running ingen_fab init workspace will automatically find the "config" lakehouse and "Data Analysis" notebook in your workspace and update their GUIDs.

Options: - --workspace-id: Fabric workspace ID (auto-detected if not provided) - --workspace-name: Fabric workspace name (auto-detected if not provided) - --environment: Environment to configure (defaults to $FABRIC_ENVIRONMENT)

Examples:

# Configure workspace for development environment
$env:FABRIC_ENVIRONMENT = "development"
ingen_fab init workspace

# Specify workspace explicitly
ingen_fab init workspace --workspace-name "Analytics Workspace"

# Configure for specific environment
ingen_fab init workspace --environment production

`init storage-config`¶

Generate lakehouse, warehouse, and SQL database artifact folders from storage_config.yaml configuration file. This command automates the creation of Fabric-compatible artifact structures and updates variable definitions across all environments.

ingen_fab init storage-config

Prerequisites: - Valid storage_config.yaml file in {workspace}/fabric_config/ directory - FABRIC_WORKSPACE_REPO_DIR environment variable must be set

What it does:

Reads Configuration: Parses fabric_config/storage_config.yaml to identify lakehouses, warehouses, and SQL databases
Creates Artifact Folders: Generates properly structured folders for each resource:
Lakehouses: fabric_workspace_items/lakehouses/{name}.Lakehouse/
Warehouses: fabric_workspace_items/warehouses/{name}.Warehouse/
SQL Databases: fabric_workspace_items/sql_databases/{name}.SQLDatabase/
Generates Artifact Files: Creates required Fabric JSON files:
.platform - Contains metadata including type, displayName, and unique logicalId (GUID)
lakehouse.metadata.json - Lakehouse metadata (empty object {})
shortcuts.metadata.json - Shortcuts configuration (empty array [])
Updates Variable Definitions: Adds variable definitions to variables.json:
{name}_workspace_id - GUID of workspace containing the resource
{name}_lakehouse_name / {name}_warehouse_name / {name}_sqldatabase_name - Name of the resource
{name}_lakehouse_id / {name}_warehouse_id / {name}_sqldatabase_id - GUID of the resource
Updates All ValueSets: Updates all environment files (development.json, test.json, production.json) with placeholder GUIDs

Storage Configuration Format Sample:

storage:
  - lakehouse: local
    lh_bronze: lh_bronze
    lh_silver: lh_silver
    lh_gold: lh_gold

  - warehouses: local
    wh_gold: wh_gold
    wh_reporting: DO_NOT_CREATE  # Skip this warehouse

  - sqldatabases: local
    sqldb_analytics: sqldb_analytics
    sqldb_staging: DO_NOT_CREATE  # Skip this SQL database

Special Values: - DO_NOT_CREATE - Skip creating this resource - Empty string "" - Skip creating this resource

Generated Files Structure:

For each lakehouse (e.g., lh_bronze):

fabric_workspace_items/lakehouses/lh_bronze.Lakehouse/
├── .platform                    # Fabric metadata with unique logicalId
├── lakehouse.metadata.json      # Empty object {}
└── shortcuts.metadata.json      # Empty array []

For each warehouse (e.g., wh_gold):

fabric_workspace_items/warehouses/wh_gold.Warehouse/
└── .platform                    # Fabric metadata with unique logicalId

For each SQL database (e.g., db_analytics):

fabric_workspace_items/sql_databases/sqldb_analytics.SQLDatabase/
└── .platform                    # Fabric metadata with unique logicalId

Logical ID Generation:

Each artifact's .platform file contains a logicalId field with a randomly generated GUID (e.g., a1b2c3d4-e5f6-7890-abcd-ef1234567890). This GUID uniquely identifies the artifact in Fabric and is generated using uuid.uuid4(). The logicalId is purely for Fabric's internal tracking and does not need to match any workspace or resource ID.

LogicalId vs Resource IDs

The logicalId in the .platform file is a random GUID used by Fabric for artifact tracking. It is completely independent from: - Workspace IDs - Lakehouse/Warehouse resource IDs - Any other GUIDs in your configuration

These are separate values that serve different purposes in the Fabric platform.

Variable Updates Example for "lh_bronze":

After running the command, variables.json will include:

{
  "name": "lh_bronze_workspace_id",
  "note": "GUID of the workspace containing lh_bronze lakehouse.",
  "type": "String",
  "value": ""
},
{
  "name": "lh_bronze_lakehouse_name",
  "note": "Name of the lh_bronze lakehouse.",
  "type": "String",
  "value": ""
},
{
  "name": "lh_bronze_lakehouse_id",
  "note": "GUID of the lh_bronze lakehouse.",
  "type": "String",
  "value": ""
}

And each valueSet file (e.g., development.json) will include:

{
  "name": "lh_bronze_workspace_id",
  "value": "REPLACE_WITH_LH_BRONZE_WORKSPACE_GUID"
},
{
  "name": "lh_bronze_lakehouse_name",
  "value": "lh_bronze"
},
{
  "name": "lh_bronze_lakehouse_id",
  "value": "REPLACE_WITH_LH_BRONZE_LAKEHOUSE_GUID"
}

Examples:

# Generate all artifacts from storage_config.yaml
ingen_fab init storage-config

# After running, you'll see:
# ✓ Created 3 lakehouse folder(s)
# ✓ Created 1 warehouse folder(s)
# ✓ Created 2 SQL database folder(s)
# ✓ Updated variables.json with new variable definitions
# ✓ Updated 3 valueSet file(s) with new variables

Output Example:

Found 3 lakehouse(s): lh_bronze, lh_silver, lh_gold
Found 1 warehouse(s): wh_gold
Found 2 SQL database(s): sqldb_analytics, sqldb_staging

📦 Processing lakehouses...
  ✓ Created lh_bronze.Lakehouse (ID: a1b2c3d4...)
  ✓ Created lh_silver.Lakehouse (ID: e5f6g7h8...)
  ✓ Created lh_gold.Lakehouse (ID: i9j0k1l2...)

🏢 Processing warehouses...
  ✓ Created wh_gold.Warehouse (ID: m3n4o5p6...)

🗄️  Processing SQL databases...
  ✓ Created sqldb_analytics.SQLDatabase (ID: q7r8s9t0...)
  ✓ Created sqldb_staging.SQLDatabase (ID: u1v2w3x4...)

📝 Updating variable library definitions...
  ✓ Added 18 variable definition(s) to variables.json

📝 Updating variable library valueSets...
  ✓ Updated development.json (added 18 variables)
  ✓ Updated test.json (added 18 variables)
  ✓ Updated production.json (added 18 variables)

============================================================
✓ Successfully created 6 artifact folder(s)
  • 3 lakehouse(s)
  • 1 warehouse(s)
  • 2 SQL database(s)
✓ Updated variables.json with new variable definitions
✓ Updated 3 valueSet file(s) with new variables

💡 Next steps:
   1. Review the generated artifacts in fabric_workspace_items/
   2. Review the updated variables in valueSets/
   3. Deploy to Fabric: ingen_fab deploy deploy

Behavior: - Non-destructive: Skips folders that already exist (won't overwrite) - Idempotent: Safe to run multiple times - Smart Updates: Only adds new variables to variables.json and valueSet files - Duplicate Prevention: Checks for existing variables before adding

Next steps after running: 1. Review generated artifact folders and files 2. Update placeholder GUIDs in valueSet files with actual workspace and resource IDs 3. Deploy to Fabric using ingen_fab deploy deploy

ddl¶

Compile DDL notebooks from templates.

`ddl compile`¶

Generate DDL notebooks from SQL and Python scripts.

ingen_fab ddl compile --output-mode fabric_workspace_repo --generation-mode Warehouse

Options: - --output-mode / -o: Output destination (fabric_workspace_repo, local) - --generation-mode / -g: Target platform (Warehouse, Lakehouse) - --verbose / -v: Enable verbose output

Examples:

# Generate both lakehouse and warehouse notebooks (defaults to fabric_workspace_items folder)
ingen_fab ddl compile

# Generate lakehouse notebooks (defaults to fabric_workspace_items folder)
ingen_fab ddl compile -g Lakehouse

# Generate lakehouse notebooks
ingen_fab ddl compile --output-mode fabric_workspace_repo --generation-mode Lakehouse

# Generate warehouse notebooks for Fabric deployment
ingen_fab ddl compile --output-mode fabric_workspace_repo --generation-mode Warehouse

# Generate with verbose output
ingen_fab ddl compile -o fabric_workspace_repo -g Warehouse -v

`ddl ddls-from-metadata`¶

Generate DDL Python scripts from a metadata CSV file (defaults to metadata/lakehouse_metadata_all.csv).

If you used ingen_fab deploy get-metadata to generate a file with another name, please specify it using --metadata-file.

ingen_fab ddl ddls-from-metadata --lakehouse lh_bronze

Options: - --lakehouse / -l: Lakehouse name to generate DDLs for (required)

--table / -t: Specific table name to generate (optional, generates all tables if not specified)
--metadata-file / -m: Path to CSV metadata file (default: metadata/lakehouse_metadata_all.csv)
--subdirectory / -d: Subdirectory for output files (default: generated)
--sequence-numbers / --no-sequence-numbers / -s / -ns: Include sequence number prefixes in filenames (default: True)
--verbose / -v: Enable verbose output

Output: - With sequence numbers: 001_lh_silver2_customer.py, 002_lh_silver2_orders.py

Without sequence numbers: lh_silver2_customer.py, lh_silver2_orders.py
Output path: {workspace}/ddl_scripts/Lakehouses/{lakehouse}/{subdirectory}/

System Table Exclusion:

Tables with schemas sys or queryinsights are excluded.

Examples:

# Generate DDL for all tables in lakehouse
ingen_fab ddl ddls-from-metadata --lakehouse lh_bronze

# Generate DDL for specific table only
ingen_fab ddl ddls-from-metadata --lakehouse lh_bronze --table customer_data

# Generate without sequence numbers in filenames
ingen_fab ddl ddls-from-metadata --lakehouse lh_bronze --no-sequence-numbers

# Custom output subdirectory
ingen_fab ddl ddls-from-metadata --lakehouse lh_bronze --subdirectory staging_ddls

# Use custom metadata CSV file
ingen_fab ddl ddls-from-metadata --lakehouse lh_bronze --metadata-file custom_metadata.csv

# Use metadata file from different directory
ingen_fab ddl ddls-from-metadata --lakehouse lh_bronze --metadata-file /path/to/external_metadata.csv

# Combine options with short flags
ingen_fab ddl ddls-from-metadata -l lh_bronze -t orders -d production -ns -m custom_meta.csv

Typical Workflow:

# 1. Extract metadata from lakehouse (creates metadata/lakehouse_metadata_all.csv)
ingen_fab deploy get-metadata --lakehouse-name lh_bronze --format csv

# 2. Generate DDL scripts from the metadata (uses default file)
ingen_fab ddl ddls-from-metadata --lakehouse lh_bronze

# 3. OR use a custom metadata file
ingen_fab ddl ddls-from-metadata --lakehouse lh_bronze --metadata-file my_custom_metadata.csv

# 4. Review and customize the generated scripts in ddl_scripts/Lakehouses/lh_silver2/generated/

deploy¶

Deploy to environments and manage workspace items.

`deploy deploy`¶

Deploy all artifacts to a specific environment.

ingen_fab deploy deploy

Requires the global options --fabric-workspace-repo-dir and --fabric-environment to be set either via command line or environment variables.

Examples:

# Using environment variables (recommended)
$env:FABRIC_WORKSPACE_REPO_DIR = "dp"
$env:FABRIC_ENVIRONMENT = "development"
ingen_fab deploy deploy

`deploy delete-all`¶

Delete all workspace items in an environment.

ingen_fab deploy delete-all

Options: - --force / -f: Force deletion without confirmation

Uses the global options --fabric-workspace-repo-dir and --fabric-environment.

Examples:

# Delete all items in development environment (will prompt for confirmation)
ingen_fab deploy delete-all --fabric-environment development

# Force delete without confirmation
ingen_fab deploy delete-all --fabric-environment test --force

`deploy upload-python-libs`¶

Upload python_libs to the Fabric config lakehouse with variable injection performed during upload (via OneLake/ADLS APIs).

ingen_fab deploy upload-python-libs

Uses the global options --fabric-workspace-repo-dir and --fabric-environment.

`deploy get-metadata`¶

Extract schema/table/column metadata for Lakehouse, Warehouse, or both.

ingen_fab deploy get-metadata [OPTIONS]

Target selection: - --target / -tgt: lakehouse (default), warehouse, or both

Workspace/asset filters:

Workspace: --workspace-id | --workspace-name
Lakehouse: --lakehouse-id | --lakehouse-name
Warehouse: --warehouse-id | --warehouse-name

Metadata filters and connection:

--schema / -s: Schema name filter
--table / -t: Table name substring filter
--method / -m: sql-endpoint (default) or sql-endpoint-odbc
--sql-endpoint-id: Explicit SQL endpoint ID
--sql-endpoint-server: Endpoint server prefix (e.g., myws-abc123)

Output control:

--format / -f: csv (default), json, or table
--output / -o: Write output to a file; omit to print to console
Lakehouse only: --all: Extract all lakehouses in the workspace

Examples:

# Lakehouse metadata for one asset; write CSV to file
ingen_fab deploy get-metadata \
  --workspace-name "Analytics Workspace" \
  --lakehouse-name "Config Lakehouse" \
  --schema config --table meta \
  --format csv --output ./artifacts/lakehouse_metadata.csv

# Warehouse metadata printed as a table
ingen_fab deploy get-metadata \
  --workspace-name "Analytics Workspace" \
  --warehouse-name "EDW" \
  --target warehouse --format table

# Both lakehouse and warehouse (filter by schema) using default CSV outputs
ingen_fab deploy get-metadata \
  --workspace-name "Analytics Workspace" \
  --schema sales --target both

`deploy compare-metadata`¶

Compare two metadata CSV files and report differences between them.

ingen_fab deploy compare-metadata [OPTIONS]

Required options: - --file1 / -f1: First CSV metadata file for comparison - --file2 / -f2: Second CSV metadata file for comparison

Output control: - --format / -fmt: table (default), json, or csv - --output / -o: Write comparison report to file (optional)

Detection capabilities:

The command identifies and reports:

Missing tables: Tables present in one file but not the other
Missing columns: Columns present in one file but not the other
Data type differences: Same column with different data types
Nullable differences: Same column with different nullable settings

Output ordering:

Results are automatically sorted by: 1. Asset name (lakehouse/warehouse name) 2. Schema name
3. Table name 4. Column name 5. Difference type

Examples:

# Compare two lakehouse metadata files with table output
ingen_fab deploy compare-metadata \
  --file1 metadata_before.csv \
  --file2 metadata_after.csv

# Save detailed JSON comparison report
ingen_fab deploy compare-metadata \
  -f1 prod_metadata.csv \
  -f2 dev_metadata.csv \
  -o schema_diff_report.json --format json

# Generate CSV output for further analysis
ingen_fab deploy compare-metadata \
  --file1 old_schema.csv \
  --file2 new_schema.csv \
  --format csv --output differences.csv

Sample output:

Found 4 differences between metadata_before.csv and metadata_after.csv:
  - Missing Table: 1
  - Missing Column: 1
  - Data Type Diff: 1
  - Nullable Diff: 1

┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
┃ Type           ┃ Identifier                                       ┃ metadata_before  ┃ metadata_after   ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
│ Missing Table  │ Analytics.dbo.new_products                       │ missing          │ present          │
│ Missing Column │ Analytics.dbo.customers.phone                    │ missing          │ present          │
│ Data Type Diff │ Analytics.dbo.orders.total_amount                │ decimal(10,2)    │ decimal(12,2)    │
│ Nullable Diff  │ Analytics.dbo.customers.email                    │ true             │ false            │
└────────────────┴──────────────────────────────────────────────────┴──────────────────┴──────────────────┘

`deploy download-artefact`¶

Download artifacts from the Fabric workspace to local directory for version control and redeployment.

ingen_fab deploy download-artefact [OPTIONS]

Options:

--artefact-name / -n: Name of the Fabric artefact to download (required)
--artefact-type / -t: Type of Fabric artefact (required) - see supported types below
--output-path / -o: Directory to save the downloaded artefact (default: fabric_workspace_items)
--workspace-id / -w: Target workspace ID (overrides environment workspace)
--force / -f: Overwrite existing files without confirmation

Supported Artefact Types:

The following artefact types can be downloaded using the Fabric REST API:

Notebook - Jupyter notebooks with source code
Report - Power BI report definitions
SemanticModel - Tabular semantic models (datasets)
DataPipeline - Data pipeline configurations
GraphQLApi - GraphQL API definitions
DataflowGen2 - Dataflow Gen2 definitions
SparkJobDefinition - Spark job definitions
DataWarehouse - Data warehouse schemas
KQLDatabase - KQL database definitions

Note: Some Fabric item types (like Lakehouse, Environment, VariableLibrary) cannot be downloaded via the REST API and must be managed through other means.

Uses the global options --fabric-workspace-repo-dir and --fabric-environment.

Examples:

# Download a notebook from the workspace
ingen_fab deploy download-artefact --artefact-name "My Notebook" --artefact-type Notebook

# Download a Power BI report
ingen_fab deploy download-artefact --artefact-name "rp_test" --artefact-type Report

# Download a data pipeline
ingen_fab deploy download-artefact -n "ETL Pipeline" -t DataPipeline

# Download from a specific workspace
ingen_fab deploy download-artefact -n "Sales Report" -t Report -w abc123-def456

# Download with custom output path
ingen_fab deploy download-artefact -n "My Notebook" -t Notebook -o ./downloads

`deploy upload-dbt-project`¶

Upload a dbt project to the Fabric workspace for execution.

ingen_fab deploy upload-dbt-project [OPTIONS]

Options:

--project-path / -p: Path to dbt project directory (default: current directory)
--target-lakehouse / -l: Target lakehouse name for dbt execution
--target-workspace / -w: Target workspace name (uses environment config if not specified)
--models / -m: Specific dbt models to include (comma-separated)
--exclude / -e: Models to exclude from upload (comma-separated)
--include-profiles: Include dbt profiles.yml file (default: false)
--overwrite: Overwrite existing dbt project files (default: false)

Uses the global options --fabric-workspace-repo-dir and --fabric-environment.

Examples:

# Upload entire dbt project to default lakehouse
ingen_fab deploy upload-dbt-project --project-path ./my_dbt_project

# Upload specific models to named lakehouse
ingen_fab deploy upload-dbt-project \
  --project-path ./analytics_dbt \
  --target-lakehouse "Analytics Lakehouse" \
  --models "staging.customers,marts.customer_summary"

# Upload project excluding test models
ingen_fab deploy upload-dbt-project \
  --project-path ./dbt_project \
  --exclude "tests.*,staging.test_*" \
  --overwrite

# Upload with profiles and specific workspace
ingen_fab deploy upload-dbt-project \
  --project-path ./enterprise_dbt \
  --target-workspace "Production Analytics" \
  --target-lakehouse "Gold Layer" \
  --include-profiles

dbt¶

Generate notebooks from dbt outputs and proxy commands to dbt_wrapper. The dbt integration includes automatic profile management that helps you select and configure the appropriate lakehouse for your dbt models.

Automatic Profile Management¶

When running dbt commands through ingen_fab, the system automatically manages your dbt profile:

Discovers Available Lakehouses: Scans your environment configuration for all configured lakehouses
Interactive Selection: Prompts you to choose which lakehouse to use (if multiple are available)
Saves Preferences: Remembers your selection for future use in the same environment
Environment-Specific: Maintains different selections for different environments (development, test, production)

The dbt profile is automatically created or updated at ~/.dbt/profiles.yml with the name fabric-spark-testnb.

`dbt create-notebooks`¶

Generate Fabric notebooks from dbt models and tests. This command will prompt you to select a lakehouse if multiple options are available.

ingen_fab dbt create-notebooks --dbt-project my_dbt_project

Options:

--dbt-project / -p: Name of the dbt project directory under the workspace repo
--skip-profile-confirmation: Skip confirmation prompt when updating dbt profile

Examples:

# Create notebooks for a dbt project
ingen_fab dbt create-notebooks --dbt-project analytics_models

# Skip profile confirmation prompt
ingen_fab dbt create-notebooks -p data_mart --skip-profile-confirmation

`dbt convert-metadata`¶

Convert cached lakehouse metadata to dbt metaextracts format.

ingen_fab dbt convert-metadata --dbt-project _dbt_project

Options:

--dbt-project / -p: Name of the dbt project directory under the workspace repo
--metadata-file / -m: Path to CSV metadata file (default: metadata/lakehouse_metadata_all.csv)
Can be absolute or relative to workspace root
Use this if you generated metadata with a custom filename
--skip-profile-confirmation: Skip confirmation prompt when updating dbt profile

Prerequisites: The metadata must first be extracted using:

ingen_fab deploy get-metadata --target lakehouse

This creates the required metadata/lakehouse_metadata_all.csv file (or custom path if specified).

What it does: - Reads from {workspace}/metadata/lakehouse_metadata_all.csv (or specified custom path) - Creates JSON files in {workspace}/{dbt_project}/metaextracts/ for dbt_wrapper to use

Examples:

# Convert metadata for dbt project (using default metadata file)
ingen_fab dbt convert-metadata --dbt-project analytics_models

# Use custom metadata file (relative path)
ingen_fab dbt convert-metadata --dbt-project analytics_models --metadata-file metadata/custom_lakehouse_data.csv

# Use custom metadata file (absolute path)
ingen_fab dbt convert-metadata -p analytics_models -m C:\data\lakehouse_export.csv

# Skip profile confirmation
ingen_fab dbt convert-metadata -p my_dbt_project --skip-profile-confirmation

`dbt generate-schema-yml`¶

Convert cached lakehouse metadata to dbt schema.yml format for a specific lakehouse and layer.

ingen_fab dbt generate-schema-yml --dbt-project my_dbt_project --lakehouse lh_bronze --layer staging --dbt-type model

Options:

--dbt-project / -p: Name of the dbt project directory under the workspace repo
--lakehouse: Name of the lakehouse to use as a filter for the csv
--layer: Name of the dbt layer
--dbt-type: Is this for a model or a snapshot?
--skip-profile-confirmation: Skip confirmation prompt when updating dbt profile

Prerequisites: The metadata must first be extracted using:

ingen_fab deploy get-metadata --target lakehouse

This creates the required metadata/lakehouse_metadata_all.csv file.

What it does: - Reads from {workspace}/metadata/lakehouse_metadata_all.csv - Creates schema.yml file in the appropriate dbt project directory - Filters metadata for the specified lakehouse and layer - Generates dbt-compatible schema definitions

Examples:

# Generate schema.yml for staging models in bronze lakehouse
ingen_fab dbt generate-schema-yml --dbt-project analytics_models --lakehouse lh_bronze --layer staging --dbt-type model

# Generate schema.yml for snapshots with short flags
ingen_fab dbt generate-schema-yml -p data_mart --lakehouse lh_silver --layer marts --dbt-type snapshot

# Skip profile confirmation
ingen_fab dbt generate-schema-yml -p my_dbt_project --lakehouse lh_gold --layer reporting --dbt-type model --skip-profile-confirmation

`dbt exec` (proxy command)¶

Proxy any dbt command to dbt_wrapper inside the Fabric workspace repo. The exec command provides intelligent lakehouse selection:

With saved preference: Notifies user of chosen lakehouse and continues
Without saved preference: Always prompts for interactive selection
Never fails silently: Ensures user knows which lakehouse is being used

ingen_fab dbt [DBT_COMMAND] [DBT_OPTIONS]

Examples:

# build dbt models and snapshots
Ingen_fab dbt exec -- stage run build --project-dir dbt_project

# build dbt master notebooks
Ingen_fab dbt exec -- stage run post-scripts --project-dir dbt_project

Typical Output:

# With saved preference:
Using saved lakehouse preference: Bronze Layer (Environment: development)

# Without saved preference:
No valid lakehouse preference found for environment 'development'. Please select a lakehouse:
[Interactive selection table appears]

Configuration¶

Environment Variables¶

Set these to avoid specifying options repeatedly:

# Core configuration
$env:FABRIC_WORKSPACE_REPO_DIR = "dp"
$env:FABRIC_ENVIRONMENT = "development"

# For local testing
$env:FABRIC_ENVIRONMENT = "local"  # Required for test local commands

# Authentication (for deployment)
$env:AZURE_TENANT_ID = "your-tenant-id"
$env:AZURE_CLIENT_ID = "your-client-id"
$env:AZURE_CLIENT_SECRET = "your-client-secret"

Configuration Files¶

The tool uses the following configurations:

platform_manifest_*.yml files in your project root to determine what has been deployed to the current environment and uses file hashing to determine if any artefacts have been updated
Variable library files in fabric_workspace_items/config/var_lib.VariableLibrary/valueSets/
Environment variables

Common Usage Patterns¶

Development Workflow¶

# 1. Create project
ingen_fab init new --project-name "My Project"

# 2. Configure environment variables
$env:FABRIC_WORKSPACE_REPO_DIR = "dp"
$env:FABRIC_ENVIRONMENT = "development"

# 3. Generate lakehouse and warehouse artifacts from storage_config.yaml
ingen_fab init storage-config

# 4. Update configuration
# Edit fabric_workspace_items/config/var_lib.VariableLibrary/valueSets/development.json
# Replace placeholder GUIDs with your actual workspace and lakehouse IDs

# 5. Develop and build dbt notebooks, copy them to fabric_workspace_items

ingen_fab dbt exec -- stage run build --project-dir dbt_project
ingen_fab dbt exec -- stage run post-scripts --project-dir dbt_project
ingen_fab dbt create-notebooks -p my_dbt_project

# 6. Generate ddl notebooks
ingen_fab ddl compile --output-mode fabric_workspace_repo --generation-mode Warehouse
ingen_fab ddl compile --output-mode fabric_workspace_repo --generation-mode Lakehouse

# 7. Test locally
$env:FABRIC_ENVIRONMENT = 'local'
ingen_fab test local python
ingen_fab test local pyspark

# 8. Deploy to development
$env:FABRIC_ENVIRONMENT = 'development'
ingen_fab deploy deploy

# 9. Generate and run platform tests
ingen_fab test platform generate

Multi-Environment Deployment¶

# Deploy to different environments using environment variables
@('development', 'test', 'production') | ForEach-Object {
    $env:FABRIC_ENVIRONMENT = $_
    $env:FABRIC_WORKSPACE_REPO_DIR = "dp"
    ingen_fab deploy deploy
}

Working with Packages¶

# Flat File Ingestion
ingen_fab package ingest compile --include-samples
ingen_fab package ingest compile --target-datastore warehouse --include-samples
ingen_fab package ingest run --config-id "customers_import" --environment development

# Extract Generation
ingen_fab package extract compile --include-samples
ingen_fab package extract run --extract-name CUSTOMER_DAILY_EXPORT

# Synapse Sync
ingen_fab package synapse compile --include-samples
ingen_fab package synapse run --master-execution-id "12345"

# Synthetic Data Generation
ingen_fab package synthetic-data list-datasets
ingen_fab package synthetic-data generate retail_oltp_small --target-rows 10000 --seed 42
ingen_fab package synthetic-data compile --dataset-id retail_star_large --target-rows 1000000

# Note: The run commands display what parameters would be used for execution in a production environment

Error Handling¶

Common exit codes: - 0: Success - 1: General error or invalid parameters - Non-zero: Command-specific failure

Troubleshooting Tips¶

Use --help with any command for detailed usage
Check environment variables if commands fail unexpectedly
Set FABRIC_ENVIRONMENT=local for local testing
Enable verbose output with --verbose where available
Check log files in your workspace for detailed error messages

Getting Help¶

Command help: ingen_fab COMMAND --help
Subcommand help: ingen_fab COMMAND SUBCOMMAND --help
Global help: ingen_fab --help
Version: ingen_fab --version

Version Output¶

Use ingen_fab --version to verify what type of build is installed:

Release install from tag (for example @v1.4.2) prints a version: ingen_fab version v1.4.2
Non-release install from git without a tag prints: ingen_fab version installed from non-release (main)

Built-in --help (Synced)¶

The following sections embed live --help output generated by a script to keep docs in sync.

For the most up-to-date help information, use the built-in help commands:

# Get main help
ingen_fab --help

# Get help for specific command groups  
ingen_fab deploy --help
ingen_fab ddl --help
ingen_fab package --help

# Get help for specific commands
ingen_fab deploy get-metadata --help
ingen_fab ddl compile --help
ingen_fab package ingest compile --help

CLI Reference¶

Command Groups at a Glance¶

Global Options¶

Command Groups¶

init¶

init new¶

init workspace¶

init storage-config¶

ddl¶

ddl compile¶

ddl ddls-from-metadata¶

deploy¶

deploy deploy¶

deploy delete-all¶

deploy upload-python-libs¶

deploy get-metadata¶

deploy compare-metadata¶

deploy download-artefact¶

deploy upload-dbt-project¶

dbt¶

Automatic Profile Management¶

dbt create-notebooks¶

dbt convert-metadata¶

dbt generate-schema-yml¶

dbt exec (proxy command)¶

Configuration¶

Environment Variables¶

Configuration Files¶

Common Usage Patterns¶

Development Workflow¶

Multi-Environment Deployment¶

Working with Packages¶

Error Handling¶

Troubleshooting Tips¶

Getting Help¶

Version Output¶

Built-in --help (Synced)¶

`init new`¶

`init workspace`¶

`init storage-config`¶

`ddl compile`¶

`ddl ddls-from-metadata`¶

`deploy deploy`¶

`deploy delete-all`¶

`deploy upload-python-libs`¶

`deploy get-metadata`¶

`deploy compare-metadata`¶

`deploy download-artefact`¶

`deploy upload-dbt-project`¶

`dbt create-notebooks`¶

`dbt convert-metadata`¶

`dbt generate-schema-yml`¶

`dbt exec` (proxy command)¶