Statistical Data Format Converter
> Safe by default — preview, not execute: the skill shows what it will read/convert and only writes a file when you explicitly ask. Every R-invoking path is opt-in and disabled by default.
Language
- English guide → README.md
- 中文指南 → README_zh-CN.md
This skill responds in the user's input language and auto-switches; runtime prompts switch by locale. SKILL.md body is English-only (agent-facing); bilingual walkthroughs live in the two READMEs.
Purpose
Read 50+ statistical-software and clinical-trial data formats into a pandas DataFrame, and inter-convert between most formats (SPSS ↔ Stata ↔ R ↔ SAS XPT ↔ Excel ↔ Parquet ↔ HDF5 ↔ JSON …). For statistical binary formats it preserves full variable/value labels and special-missing-value metadata; text/JSON formats preserve only a retainable subset.
Features
| Capability | Description | Typical Scenario |
|---|---|---|
| Read | Extract data + all metadata from 50+ formats into a pandas DataFrame; clearly report what is preserved vs lost | read data.sav and show metadata |
| Convert | Inter-convert most stats formats; export to universal formats (Parquet/Feather/HDF5/JSON/CSV/Excel) with labels embedded | convert data.sav to .dta keeping variable labels |
| Embed metadata | Labels embedded in Arrow schema.metadata / sidecar JSON for lossless round-trips |
save to parquet but keep value labels |
| Warn | Auto-detect and report metadata loss per conversion path | audit before exporting to CSV |
Supported Formats
50+ formats, sorted alphabetically.
| Format | Extension | Meta Preserve |
|---|---|---|
| CDISC ODM | .odm |
⚠️ Clinical data only |
| dBASE / FoxPro | .dbf |
⚠️ Read+Write, uppercase names |
| EpiData | .rec |
⚠️ Via R (opt-in) |
| EpiInfo | .prj .xml |
✅ XML structure |
| Excel | .xlsx .xls .xlsm |
⚠️ Extra sheet for labels; merged-cell fill |
| EViews | .wf1 .wf2 |
⚠️ JSON structure |
| Feather | .feather .arrow |
✅ Via schema |
| FST | .fst |
✗ Detect-only (proprietary) |
| GraphPad Prism | .pzfx .pz |
⚠️ Multi-table |
| Gretl | .gdt .gdtb |
✅ String-tables |
| HDF5 | .h5 .hdf5 |
⚠️ Hierarchy + attribute labels |
| HTML | .html |
⚠️ Tables only |
| jamovi | .omv |
✅ JSON analysis |
| JMP | .jmp |
⚠️ Multi-table |
| JSON | .json |
✅ stat-full-meta |
| MATLAB | .mat |
⚠️ v7.3+ via h5py fallback |
| Mathematica | .wdx |
⚠️ Best-effort XML |
| Minitab | .mtw .mpj |
⚠️ Via R (opt-in) |
| MS Access | .mdb .accdb |
⚠️ Multi-table; needs system driver |
| ODS | .ods |
⚠️ Data only |
| ORC | .orc |
✅ Via schema |
| Origin | .opju .oggu |
⚠️ Best-effort |
| Parquet | .parquet |
✅ Via schema; partitioned datasets |
| R | .rda .rds .rdata |
✅ pyreadr; R fallback opt-in (allow_r_exec) |
| SAS | .sas7bdat .xpt .sas7bcat |
✅ |
| SPSS | .sav .zsav .por |
✅ |
| Stata | .dta |
✅ |
| Weka ARFF | .arff |
✅ Nominal mapping |
| XML | .xml |
⚠️ Structure preserved |
> ✅=Full · ⚠️=Partial/conditional · ✗=Not preserved
>
> 12 detect-only formats (SAS CPORT .cpt, Statistica .sta, OxMetrics .in7, SYSTAT .sys/.syd, Paradox .db/.px, LIMDEP .lpw, NCSS .ncss, FST) give clear export guidance — see README.
Return Structure
{
"dataframe": pd.DataFrame,
"metadata": {
"file_format": "spss_sav",
"row_count": 100, "column_count": 10,
"variable_labels": {"q1": "Question 1"},
"value_labels": {"q1": {1: "Yes", 2: "No"}},
"special_missing": {...},
},
"warnings": [],
"column_report": {"q1": {"source_type": "int", "pandas_dtype": "int64"}},
}
Quick Start
# Check environment (optional install on request)
python scripts/check_env.py --install
In WorkBuddy (bilingual, auto-detects your language):
> convert data.sav to .dta
> read data.sav and show metadata
> 把 data.sav 转成 .dta 并保留变量标签
> For complete code examples, see references/usage_examples.py.
Dependencies
requires:
bins: [python3]
packages:
core: [pyreadstat>=1.3.5,<2, pyreadr>=0.4,<0.5, pandas>=2.0,<3]
extended: [openpyxl, xlrd, scipy, h5py, pyarrow, lxml, odfpy, tableauhyperapi, dbfread, dbf, pyodbc]
> Full list: requirements.txt
⚠️ Safety
- All R-invoking paths are opt-in and disabled by default; they only run when you pass
allow_r_exec=Trueon a trusted file. - Pure-Python parsers (
pyreadr,mtbpy) are tried first and never execute code. - No silent R fallback — if the pure-Python parser fails and
allow_r_execis not set, the skill raises a clear error. - Writing an existing
.hyperbacks up to.bakbefore overwrite; on failure the original is untouched. - Output for reference only; validate before regulatory submissions.
Security model (transparent disclosure)
| Behavior | Description |
|---|---|
| R invocation (opt-in) | Reading .rda/.rds/.RData (readRDS()/load()), Minitab .mtw/.mpj, EpiData .rec, and writing R formats run only with allow_r_exec=True on a trusted file. Pure-Python parsers tried first. |
| No silent fallback | On pure-Python parser failure without allow_r_exec, raises a clear error instead of launching R — avoids executing embedded code from untrusted files. |
| Static R templates | When the opt-in R path runs, all R scripts are static templates; user input passes only as CLI args (commandArgs(trailingOnly=TRUE) via jsonlite) — never concatenated into executable R code. |
| Temp CSV bridge | Opt-in R writes data to a temp CSV then reads back; deleted after use, but on crash could briefly persist — avoid highly sensitive data through R-backed formats. |
| No destructive writes | .hyper write → temp file → rotate existing to .bak (prior .bak → .bak.1, never silently deleted) → atomic swap. Original untouched on failure. |
| Sidecar metadata files | Writing CSV/TSV also emits <name>_metadata.json (full 17-field metadata) next to the data file; Parquet/Arrow embed metadata inside the file schema. No writes occur outside the output path you specify. |
| Pinned dependencies | Core deps carry upper-bound pins (pandas, pyreadstat, pyreadr) — see requirements.txt. |
| Optional install | python scripts/check_env.py --install only on explicit request. |
| Permissions | Read the input file; write the output file to a path you specify. No network unless you explicitly request package install. |
License
MIT. See LICENSE.
