Deploy Guide¶

Home > User Guide > Deploy Guide

Practical guide to deploy changes, upload Python libraries with variable injection, and extract metadata from Fabric assets.

Summary¶

Prerequisites¶

Set up environment variables to avoid specifying them on each command:

# Project location
export FABRIC_WORKSPACE_REPO_DIR="./sample_project"

# Target environment
export FABRIC_ENVIRONMENT="development"

# Authentication (for deployment)
export AZURE_TENANT_ID="your-tenant-id"
export AZURE_CLIENT_ID="your-client-id"
export AZURE_CLIENT_SECRET="your-client-secret"

# Project location
$env:FABRIC_WORKSPACE_REPO_DIR = "dp"

# Target environment
$env:FABRIC_ENVIRONMENT = "development"

# Authentication (for deployment)
$env:AZURE_TENANT_ID = "your-tenant-id"
$env:AZURE_CLIENT_ID = "your-client-id"
$env:AZURE_CLIENT_SECRET = "your-client-secret"

See Environment Variables for a complete list.

Deploy artifacts¶

Deploy all items under fabric_workspace_items to the selected environment.

ingen_fab deploy deploy

The deployment process automatically performs variable replacement for environment-specific configuration:

Artifacts with Variable Replacement: - Notebooks (notebook-content.py) - Semantic Models (.tmdl files) - GraphQL APIs (graphql-definition.json) - Data Pipelines (pipeline-content.json) - Power BI Reports (definition.pbir)

Tips: - Use semantic, ordered DDL under ddl_scripts and generate notebooks with ingen_fab ddl compile ... before deploying. - Validate your variable library value set for the target environment.

Auto-update Item IDs¶

Enable automatic Item ID tracking after deployment by setting the AUTO_UPDATE_ITEM_IDS environment variable:

# Enable auto-update
export AUTO_UPDATE_ITEM_IDS=true
ingen_fab deploy deploy

# Or inline
AUTO_UPDATE_ITEM_IDS=true ingen_fab deploy deploy

How it works: - After successful deployment, queries the workspace for Item IDs of deployed artifacts - Updates variables following convention: {artifact_name}_{artifact_type}_id - Example: config lakehouse → config_lakehouse_id - Example: wh_gold warehouse → wh_gold_warehouse_id - Example: my_model semantic model → my_model_semanticmodel_id - Supports: Lakehouse, Warehouse, Notebook, SemanticModel, SQLDatabase, Eventhouse - Only updates variables that already exist in your valueSet - Skips artifacts without corresponding variables

When to use: - ✅ CI/CD pipelines (automation) - ✅ Initial deployments (bootstrap scenarios) - ✅ Single workspace deployments - ❌ Multi-workspace deployments (use ingen_fab init workspace instead)

Note: This feature complements but doesn't replace ingen_fab init workspace, which discovers all artifacts (not just newly deployed ones) and provides interactive configuration.

Upload Python libraries to OneLake¶

Upload python_libs to the config lakehouse with variable injection during upload.

ingen_fab deploy upload-python-libs

Notes: - Code between injection markers is resolved using the active value set. - Upload progress and failures are shown in the console.

Upload dbt project to OneLake/Config Lakehouse¶

Upload dbt project to the /Files section of the Config Lakehouse. Useful for Fabric Warehouse based projects to mount to Fabric Python notebooks

ingen_fab deploy upload-dbt-project --dbt-project <project_name>

Extract metadata (Lakehouse/Warehouse)¶

Discover schemas, tables, and columns via Fabric SQL endpoints.

# Lakehouse example (write CSV)
ingen_fab deploy get-metadata \
  --workspace-name "Analytics Workspace" \
  --lakehouse-name "Config Lakehouse" \
  --schema config --table meta \
  --format csv --output ./artifacts/lakehouse_metadata.csv

# Warehouse example (pretty table)
ingen_fab deploy get-metadata \
  --workspace-name "Analytics Workspace" \
  --warehouse-name "EDW" \
  --target warehouse --format table

# Both lakehouse and warehouse (filter by schema)
ingen_fab deploy get-metadata --workspace-name "Analytics Workspace" --schema sales --target both

Common flags: - --target / -tgt: lakehouse (default), warehouse, both - --schema / -s, --table / -t: filters - --format / -f: csv, json, table - --output / -o: write to file

Compare metadata files¶

Analyze differences between two metadata CSV files generated by get-metadata. Perfect for schema change tracking, environment comparisons, and migration validation.

# Basic comparison with table output
ingen_fab deploy compare-metadata \
  --file1 production_metadata.csv \
  --file2 staging_metadata.csv

# Generate JSON report for automated processing
ingen_fab deploy compare-metadata \
  -f1 before_migration.csv \
  -f2 after_migration.csv \
  -o migration_differences.json --format json

# CSV output for Excel analysis
ingen_fab deploy compare-metadata \
  --file1 lakehouse_v1.csv \
  --file2 lakehouse_v2.csv \
  --format csv --output schema_changes.csv

What it detects: - Missing tables: Tables added or removed between versions - Missing columns: Columns added or removed from existing tables - Data type changes: Column data type modifications (e.g., varchar(50) → varchar(100)) - Nullable changes: Column nullability modifications (NULL → NOT NULL)

Output features: - Logical ordering: Results sorted by asset → schema → table → column - Multiple formats: Human-readable tables, structured JSON, or CSV for analysis - Rich context: Each difference includes detailed descriptions and location information - Summary statistics: Count of differences by type for quick assessment

Common use cases:

# Track schema evolution over time
ingen_fab deploy get-metadata --target lakehouse -o schema_v1.csv
# ... make changes ...
ingen_fab deploy get-metadata --target lakehouse -o schema_v2.csv
ingen_fab deploy compare-metadata -f1 schema_v1.csv -f2 schema_v2.csv

# Compare environments before migration
ingen_fab deploy compare-metadata -f1 prod_schema.csv -f2 test_schema.csv

# Validate post-deployment changes
ingen_fab deploy compare-metadata \
  --file1 pre_deploy.csv --file2 post_deploy.csv \
  -o deployment_validation.json --format json

Download artifacts from workspace¶

Download artifacts from the Fabric workspace to local directory for backup, analysis, or version control.

# Download a notebook from the workspace
ingen_fab deploy download-artefact --artefact-name "My Notebook" --artefact-type Notebook

# Download a Power BI report
ingen_fab deploy download-artefact --artefact-name "rp_test" --artefact-type Report

# Download a data pipeline
ingen_fab deploy download-artefact -n "ETL Pipeline" -t DataPipeline

Supported Artefact Types: - Notebook - Jupyter notebooks with source code - Report - Power BI report definitions - SemanticModel - Tabular semantic models (datasets) - DataPipeline - Data pipeline configurations - GraphQLApi - GraphQL API definitions - DataflowGen2 - Dataflow Gen2 definitions - SparkJobDefinition - Spark job definitions - DataWarehouse - Data warehouse schemas - KQLDatabase - KQL database definitions

Common Options: - --artefact-name / -n: Name of the artefact (required) - --artefact-type / -t: Type of artefact (required) - --output-path / -o: Local directory destination (default: fabric_workspace_items) - --workspace-id / -w: Specific workspace ID (uses environment config if not specified) - --force / -f: Overwrite existing files without confirmation

Use Cases: - Backup: Create local backups of Fabric workspace artifacts - Version Control: Download artifacts for Git repository storage - Migration: Export artifacts from one workspace for import to another - Analysis: Download notebooks for local code review and analysis

Clean up¶

Delete all workspace items in an environment (use with care):

ingen_fab deploy delete-all --force

Remove any orphaned artefacts in Fabric Workspace

Notes:

• Preserves Lakehouses, Warehouses, VariableLibrary, Environment and SQL endpoints items
• --dry-run shows what would be deleted without actually deleting
• Prompts for confirmation unless --force is used
• --sync flag on deploy command for automatic cleanup

Troubleshooting¶

• Ensure FABRIC_WORKSPACE_REPO_DIR points to your workspace repo root.
• FABRIC_ENVIRONMENT must map to a value set in your variable library.
• Validate Azure authentication and permissions for the target workspace.