Power BI Model Extract

Overview

Two-stage workflow: (1) extract TMDL definitions of semantic models from the Power BI Service via REST API, then (2) compress the verbose TMDL into a compact Markdown summary (~95% smaller) that preserves the business logic — measures, columns, relationships, DAX, and Power Query/SQL source — while stripping metadata noise. batch_extract.py does both stages in one run; compress_tmdl.py can also run standalone on any local .SemanticModel folder (e.g. a PBIP project).

All scripts live in scripts/. On Windows, always set PYTHONIOENCODING=utf-8 PYTHONUTF8=1 — the scripts print emoji/Chinese and will crash under the default GBK codec otherwise.

Common tasks

Run from the skill's scripts/ directory (or reference scripts by absolute path).

Extract all models in a workspace (download + compress):

python batch_extract.py <workspace_id>

Output goes to extracted_models/ next to the script: one <name>.SemanticModel/ folder (raw TMDL) plus one <name>_summary.md (compressed) per model.

Useful flags (see the docstring at the top of batch_extract.py for the full list):
- --filter "<name>" — only models whose name contains the string
- --scheduled-only — only models with a Scheduled refresh configured (i.e. live production models)
- --no-compress — download raw TMDL only, skip the Markdown step
- --output-dir <path> — change the output location
- --token "<bearer>" — supply a Fabric bearer token explicitly

Compress a local model that's already on disk (e.g. a PBIP .SemanticModel folder, no API/auth needed):

python compress_tmdl.py "<path>/Something.SemanticModel" output_summary.md

List all accessible workspaces (to find a workspace_id):

python list_all.py

Check which datasets in a workspace have refresh history / scheduled refresh:

python check_refresh.py <workspace_id>

Authentication

batch_extract.py (and the helpers that import it) acquire a Fabric/Power BI token in this priority order — the first that works wins:

1. FABRIC_TOKEN env var or --token — a raw bearer token. Fastest ad-hoc method: in a browser on app.powerbi.com, open DevTools → Network → click any request → copy the Authorization: Bearer … value.
2. Azure CLI — if az is installed and az login has been run, the token is fetched automatically.
3. MSAL device-code flow — requires pip install msal. Prints a code + URL to log in via browser; caches a refresh token to ~/.pbip_token_cache.json so subsequent runs are silent for ~90 days.

If all fail, the script prints instructions and exits.

How compression works (when adjusting output)

compress_tmdl.py reduces TMDL ~95% by:
- Dropping noise lines — lineageTag, annotation PBI_*, formatString, displayFolder, queryGroup, etc. (see NOISE_PATTERNS).
- Skipping auto-generated boilerplate — parameter queries and "Combine Files from Folder" helper queries (is_trivial_helper), and empty/LocalDateTable cruft.
- Restructuring into Markdown — a relationships table, a Data Sources/ETL section (Power Query M), then per-table sections with Measures (DAX in fenced blocks), Columns (a type table), and the partition source. Tables are ordered 01Measures_ → dim_* → fact_* → other.
- Expanding Power Query escapes (#(lf), #(tab)) into real newlines for readability.

To tune what's kept vs. dropped, edit NOISE_PATTERNS, HELPER_FILE_OPS/COMPLEX_OPS, or the section builders in compress_tmdl.py. batch_extract.py imports these functions directly, so changes apply to both paths.

Notes

• If the user doesn't provide a workspace_id, run list_all.py to enumerate accessible workspaces and let them pick.
• Fabric API workspace/model listing can be blocked by tenant DLP policy (HTTP 403); the scripts automatically fall back to the classic Power BI REST API, so this is handled transparently.
• getDefinition is a long-running operation; batch_extract.py polls the operation until it succeeds — extraction of a large model can take tens of seconds.