mirror of
https://github.com/github/awesome-copilot.git
synced 2026-08-01 23:12:29 +00:00
Merge branch 'main' into add-skill-copilot-java-helidon
This commit is contained in:
Suren K
GitHub
@@ -0,0 +1,131 @@
|
||||
---
|
||||
name: convert-excel-to-md
|
||||
description: 'Converts Excel (.xlsx) workbooks into Markdown so their contents can be accurately analyzed, summarized, searched, or extracted from. Use this skill whenever the user shares, references, or asks about a .xlsx file — even if they don''t say "convert" or "markdown" explicitly. This includes requests to "read", "summarize", "review", "extract data from", "compare", "chart", or "analyze" a spreadsheet, workbook, budget, data export, or tracker. Always run the bundled conversion script to produce Markdown first; do not attempt to parse .xlsx content directly or write ad-hoc extraction code. Also use this skill for batch requests involving a whole folder of Excel workbooks. IMPORTANT: When the user references a folder or set of documents containing multiple file types (.pdf, .docx, .xlsx), invoke ALL three sibling skills — convert-pdf-to-md, convert-word-to-md, and convert-excel-to-md — so no file type is silently skipped.'
|
||||
---
|
||||
|
||||
# Convert Excel to Markdown
|
||||
|
||||
## When to use this skill
|
||||
|
||||
Trigger this skill any time there is a `.xlsx` file that needs to be
|
||||
understood or processed — for example, a user attaches a spreadsheet and
|
||||
asks questions about it, wants a summary of the data, wants specific rows or
|
||||
values pulled out, or wants multiple workbooks in a folder processed
|
||||
together. Excel's native `.xlsx` format is a zipped XML bundle that is not
|
||||
reliably readable as plain text, so always convert it to Markdown first
|
||||
using the script in this skill rather than trying to open or parse the file
|
||||
directly.
|
||||
|
||||
This skill only supports `.xlsx`. If asked to convert a legacy `.xls` file,
|
||||
tell the user it isn't supported and ask them to re-save it as `.xlsx`
|
||||
(Excel: File > Save As > Excel Workbook (.xlsx)) first.
|
||||
|
||||
**Mixed file types:** When the user references a folder or set of documents
|
||||
containing multiple supported file types (`.pdf`, `.docx`, `.xlsx`), this
|
||||
skill handles only `.xlsx` files. The agent MUST also invoke the sibling
|
||||
skills in parallel:
|
||||
- `convert-pdf-to-md` for any `.pdf` files
|
||||
- `convert-word-to-md` for any `.docx` files
|
||||
|
||||
Never process a folder and silently skip a supported file type. All three
|
||||
skills must be invoked together when mixed types are present.
|
||||
|
||||
## Setup (once per environment)
|
||||
|
||||
Before the first conversion in a given environment, follow
|
||||
[`references/setup.md`](references/setup.md) step by step to ensure Python,
|
||||
pip, and the `markitdown` package are installed. Do this proactively rather
|
||||
than guessing whether the environment is ready — the script itself will
|
||||
also fail with a clear pointer back to that file if `markitdown` turns out
|
||||
to be missing, so it's safe to just try the conversion first if you're
|
||||
reasonably confident setup was already done.
|
||||
|
||||
## Usage
|
||||
|
||||
The conversion script lives at `scripts/convert_excel_to_md.py`.
|
||||
|
||||
**Output structure:** MarkItDown's XLSX converter renders each sheet as its
|
||||
own `## <SheetName>` Markdown table — it has no support for embedded images
|
||||
at all. This script separately extracts real embedded images (raster
|
||||
pictures, not charts) and maps them to the sheet they belong to, writing a
|
||||
self-contained folder per document:
|
||||
|
||||
```
|
||||
<name>/
|
||||
img/
|
||||
sheet001_<sheetname>_img001.<ext>
|
||||
sheet002_<sheetname>_img001.<ext>
|
||||
...
|
||||
<name>.md (each sheet's images appear right after its table,
|
||||
under a "#### Images in this sheet" heading)
|
||||
```
|
||||
|
||||
This is per-sheet placement, not exact cell position — the finest
|
||||
granularity MarkItDown's stable output anchors (the `## <SheetName>`
|
||||
headings) allow. If a workbook has no embedded images, no `img/` folder or
|
||||
image sections are created. Native Excel **charts** are not extracted as
|
||||
images (only actual embedded pictures are — charts would need to be
|
||||
rendered by Excel/LibreOffice, which this lightweight skill does not do).
|
||||
|
||||
**Single file:**
|
||||
|
||||
```powershell
|
||||
python scripts\convert_excel_to_md.py "C:\path\to\workbook.xlsx"
|
||||
```
|
||||
|
||||
This creates a `workbook\` folder next to the source file (containing
|
||||
`workbook.md` and, if present, `workbook\img\`). To control the destination
|
||||
folder explicitly:
|
||||
|
||||
```powershell
|
||||
python scripts\convert_excel_to_md.py "C:\path\to\workbook.xlsx" -o "C:\path\to\output_folder"
|
||||
```
|
||||
|
||||
**A folder of workbooks (batch mode):**
|
||||
|
||||
```powershell
|
||||
python scripts\convert_excel_to_md.py "C:\path\to\folder"
|
||||
```
|
||||
|
||||
Add `--recursive` to also include subfolders:
|
||||
|
||||
```powershell
|
||||
python scripts\convert_excel_to_md.py "C:\path\to\folder" --recursive
|
||||
```
|
||||
|
||||
Each `.xlsx` found gets its own `<name>\` output folder next to it by
|
||||
default. Pass `-o "C:\path\to\output_parent"` to collect all the generated
|
||||
`<name>\` folders under a separate parent directory instead (subfolder
|
||||
structure is preserved when combined with `--recursive`).
|
||||
|
||||
After conversion, read the resulting `.md` file(s) to perform the actual
|
||||
analysis the user asked for — the script's job is only to produce accurate
|
||||
Markdown (and images), not to interpret the content.
|
||||
|
||||
## Deciding where output goes
|
||||
|
||||
**Default — always output next to the source file.** The `<name>/` folder
|
||||
is created in the same directory as the source `.xlsx`. This is the required
|
||||
default for every case. Do NOT override it unless the user explicitly asks
|
||||
for a different location.
|
||||
|
||||
**Only use `-o` when** the user explicitly provides an output path (e.g.,
|
||||
"save the output to `C:\output`", "put the results in `D:\work`"). Do NOT
|
||||
pass `-o` based on the agent's current working directory, the session state
|
||||
folder, or any implied location.
|
||||
|
||||
**If the source file path cannot be fully resolved** — for example, the
|
||||
user provides only a filename with no directory, or the path is ambiguous —
|
||||
use `ask_user` to confirm the full absolute path before running the
|
||||
conversion. Never guess or assume the directory.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Likely cause | Fix |
|
||||
|---|---|---|
|
||||
| `ModuleNotFoundError: No module named 'markitdown'` / exit code 2 | MarkItDown not installed | Follow `references/setup.md` |
|
||||
| `ERROR: Unsupported file type '.xls'` / exit code 3 | Legacy `.xls`, not `.xlsx` | Ask the user to re-save as `.xlsx` |
|
||||
| `ERROR: Input path not found` / exit code 3 | Wrong path, or file moved | Confirm the correct path with the user |
|
||||
| `FAILED <file> -> ...` in batch output | That specific file is corrupt, password-protected, or otherwise unreadable | Report which file(s) failed; other files in the batch still succeed |
|
||||
| `NOTE: skipped N non-.xlsx file(s)` | Folder contains non-Excel files | Expected — those files are intentionally ignored |
|
||||
| A sheet's charts don't appear as images | Charts are chart objects, not embedded pictures — this skill only extracts real embedded raster images | Expected; mention this limitation if the user specifically needs chart images |
|
||||
@@ -0,0 +1,73 @@
|
||||
# Environment Setup for convert-excel-to-md
|
||||
|
||||
Follow these steps exactly, in order, before running `scripts/convert_excel_to_md.py`
|
||||
for the first time in a given environment. Don't skip steps or improvise
|
||||
alternatives — they're written to be deterministic and safe to re-run.
|
||||
|
||||
## 1. Check Python is available (3.10+)
|
||||
|
||||
```powershell
|
||||
python --version
|
||||
```
|
||||
|
||||
- If this fails (command not found), install Python 3.10 or newer:
|
||||
- Windows: `winget install --id Python.Python.3.12 -e`
|
||||
- macOS: `brew install python@3.12`
|
||||
- Linux (Debian/Ubuntu): `sudo apt-get update && sudo apt-get install -y python3 python3-pip python-is-python3`
|
||||
- If the reported version is older than 3.10, install a newer Python using
|
||||
the same command above (MarkItDown requires 3.10+).
|
||||
|
||||
## 2. Check pip is available
|
||||
|
||||
```powershell
|
||||
python -m pip --version
|
||||
```
|
||||
|
||||
- If this fails, bootstrap pip:
|
||||
|
||||
```powershell
|
||||
python -m ensurepip --upgrade
|
||||
```
|
||||
|
||||
## 3. Install MarkItDown with Excel (.xlsx) support
|
||||
|
||||
Use the `scripts/requirements.txt` file bundled with this skill to install a pinned,
|
||||
known-good version of the dependency:
|
||||
|
||||
```powershell
|
||||
python -m pip install -r scripts/requirements.txt
|
||||
```
|
||||
|
||||
This pulls in `markitdown[xlsx]` (MarkItDown's XLSX table conversion
|
||||
dependencies, which include `pandas` and `openpyxl`). No extra package is needed for image extraction — this
|
||||
skill's script reads embedded images directly from the `.xlsx` zip
|
||||
structure using Python's built-in `zipfile` and `xml` modules.
|
||||
|
||||
## 4. Verify the install
|
||||
|
||||
```powershell
|
||||
python -c "from markitdown import MarkItDown; print('markitdown OK')"
|
||||
```
|
||||
|
||||
Expect to see `markitdown OK` printed with no errors. If you see
|
||||
`ModuleNotFoundError: No module named 'markitdown'`, repeat step 3 — pip may
|
||||
be installing into a different Python environment than the one being
|
||||
invoked (check `python -m pip --version` shows the same path as `python
|
||||
--version`'s interpreter).
|
||||
|
||||
## Notes
|
||||
|
||||
- This setup only needs to be done once per environment/virtual environment,
|
||||
not once per conversion.
|
||||
- `convert_excel_to_md.py` itself also checks for `markitdown` at startup
|
||||
and prints a pointer back to this file if it's missing, so re-running
|
||||
setup is safe and idempotent.
|
||||
- Only `.xlsx` is supported by this skill. Legacy binary `.xls` files are
|
||||
out of scope (a completely different, harder-to-parse file format) — ask
|
||||
the user to re-save the file as `.xlsx` (Excel: File > Save As > Excel
|
||||
Workbook (.xlsx)) if one is encountered.
|
||||
- Chart objects (as opposed to embedded pictures) are not extracted as
|
||||
images — only raster pictures actually embedded in the workbook's
|
||||
`xl/media` folder are. Native Excel charts would need to be rendered by
|
||||
Excel/LibreOffice to become images, which this lightweight skill does not
|
||||
attempt.
|
||||
@@ -0,0 +1,354 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Convert Excel (.xlsx) workbooks to Markdown using Microsoft's MarkItDown,
|
||||
with embedded images extracted to real files and placed under the correct
|
||||
sheet (MarkItDown's XLSX converter only extracts sheet data as tables -- it
|
||||
has no support for embedded images at all).
|
||||
|
||||
Usage:
|
||||
python convert_excel_to_md.py <input> [-o OUTPUT] [--recursive]
|
||||
|
||||
<input> may be either:
|
||||
- a path to a single .xlsx file, or
|
||||
- a path to a directory (batch mode: every .xlsx file directly inside it
|
||||
is converted; pass --recursive to also descend into subdirectories).
|
||||
|
||||
Output:
|
||||
For each source .xlsx (named "<name>.xlsx"), a folder is created
|
||||
containing the Markdown and its images, in this layout:
|
||||
|
||||
<name>/
|
||||
img/
|
||||
Sheet1_img001.<ext>
|
||||
Sheet2_img001.<ext>
|
||||
...
|
||||
<name>.md
|
||||
|
||||
MarkItDown renders each sheet as its own "## <SheetName>" section with a
|
||||
Markdown table. This script independently maps embedded images to the
|
||||
sheet they belong to (via the .xlsx zip's drawing relationships) and
|
||||
inserts a "#### Images in this sheet" block right after that sheet's
|
||||
table, before the next "## " heading. This is per-sheet placement (not
|
||||
exact cell position), which is the finest granularity MarkItDown's stable
|
||||
output anchors allow.
|
||||
|
||||
- Single file mode: the "<name>/" folder is created next to the source
|
||||
file, or at -o/--output (treated as the exact destination folder) if
|
||||
given.
|
||||
- Batch/directory mode: a "<name>/" folder is created next to each source
|
||||
file, or under -o/--output (treated as a parent directory, created if
|
||||
missing) if given, preserving relative subfolder structure when
|
||||
--recursive is used.
|
||||
- If a workbook has no embedded images, no "img/" folder or "Images in
|
||||
this sheet" sections are created.
|
||||
|
||||
Exit codes:
|
||||
0 - all requested conversions succeeded
|
||||
1 - one or more conversions failed (partial success in batch mode)
|
||||
2 - required dependency ("markitdown") is not installed
|
||||
3 - invalid input (path not found, or single-file input is not .xlsx)
|
||||
"""
|
||||
import argparse
|
||||
import posixpath
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
from xml.etree import ElementTree as ET
|
||||
|
||||
EXIT_OK = 0
|
||||
EXIT_CONVERSION_FAILED = 1
|
||||
EXIT_MISSING_DEPENDENCY = 2
|
||||
EXIT_INVALID_INPUT = 3
|
||||
|
||||
_REL_NS = "http://schemas.openxmlformats.org/package/2006/relationships"
|
||||
_MAIN_NS = "http://schemas.openxmlformats.org/spreadsheetml/2006/main"
|
||||
_R_NS = "http://schemas.openxmlformats.org/officeDocument/2006/relationships"
|
||||
_A_NS = "http://schemas.openxmlformats.org/drawingml/2006/main"
|
||||
|
||||
# Matches MarkItDown's per-sheet heading, e.g. "## Sheet1"
|
||||
_SHEET_HEADER_RE = re.compile(r"^## (.+)$", re.MULTILINE)
|
||||
|
||||
|
||||
def _import_markitdown():
|
||||
"""Import MarkItDown, failing with a clear, actionable message if absent."""
|
||||
try:
|
||||
from markitdown import MarkItDown
|
||||
return MarkItDown
|
||||
except ImportError:
|
||||
print(
|
||||
"ERROR: The 'markitdown' package is not installed.\n"
|
||||
"See references/setup.md for this skill, or run:\n"
|
||||
' pip install "markitdown[xlsx]"',
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(EXIT_MISSING_DEPENDENCY)
|
||||
|
||||
|
||||
def _normalize_rel_path(base_dir: str, target: str) -> str:
|
||||
"""Resolve a (possibly relative, e.g. '../media/image1.png') relationship
|
||||
target against the directory containing the part that referenced it."""
|
||||
if target.startswith("/"):
|
||||
return target.lstrip("/")
|
||||
return posixpath.normpath(posixpath.join(base_dir, target))
|
||||
|
||||
|
||||
def _sheet_name_to_media(xlsx_path: Path):
|
||||
"""Return {sheet_name: [media_zip_path, ...]} in per-sheet document
|
||||
order, by walking workbook.xml -> worksheet -> drawing -> media
|
||||
relationships. Returns {} if anything is missing/malformed (falls back
|
||||
gracefully -- images just won't be extracted for that sheet)."""
|
||||
try:
|
||||
with zipfile.ZipFile(xlsx_path) as z:
|
||||
names = set(z.namelist())
|
||||
if "xl/workbook.xml" not in names or "xl/_rels/workbook.xml.rels" not in names:
|
||||
return {}
|
||||
workbook_xml = z.read("xl/workbook.xml")
|
||||
workbook_rels_xml = z.read("xl/_rels/workbook.xml.rels")
|
||||
|
||||
sheet_rid = {}
|
||||
for sheet_el in ET.fromstring(workbook_xml).iter(f"{{{_MAIN_NS}}}sheet"):
|
||||
name = sheet_el.get("name")
|
||||
rid = sheet_el.get(f"{{{_R_NS}}}id")
|
||||
if name and rid:
|
||||
sheet_rid[name] = rid
|
||||
|
||||
rid_target = {}
|
||||
for rel in ET.fromstring(workbook_rels_xml).findall(f"{{{_REL_NS}}}Relationship"):
|
||||
rid_target[rel.get("Id")] = rel.get("Target")
|
||||
|
||||
result = {}
|
||||
for sheet_name, rid in sheet_rid.items():
|
||||
target = rid_target.get(rid)
|
||||
if not target:
|
||||
continue
|
||||
# workbook.xml.rels targets are typically relative to "xl/",
|
||||
# but OOXML allows package-absolute targets (leading "/") too.
|
||||
sheet_path = _normalize_rel_path("xl", target)
|
||||
if sheet_path not in names or "/" not in sheet_path:
|
||||
continue
|
||||
sheet_dir, sheet_file = sheet_path.rsplit("/", 1)
|
||||
sheet_rels_path = f"{sheet_dir}/_rels/{sheet_file}.rels"
|
||||
if sheet_rels_path not in names:
|
||||
continue
|
||||
|
||||
drawing_rid = None
|
||||
for d in ET.fromstring(z.read(sheet_path)).iter(f"{{{_MAIN_NS}}}drawing"):
|
||||
drawing_rid = d.get(f"{{{_R_NS}}}id")
|
||||
break
|
||||
if not drawing_rid:
|
||||
continue
|
||||
|
||||
drawing_target = None
|
||||
for rel in ET.fromstring(z.read(sheet_rels_path)).findall(f"{{{_REL_NS}}}Relationship"):
|
||||
if rel.get("Id") == drawing_rid:
|
||||
drawing_target = rel.get("Target")
|
||||
break
|
||||
if not drawing_target:
|
||||
continue
|
||||
drawing_path = _normalize_rel_path(sheet_dir, drawing_target)
|
||||
if drawing_path not in names or "/" not in drawing_path:
|
||||
continue
|
||||
drawing_dir, drawing_file = drawing_path.rsplit("/", 1)
|
||||
drawing_rels_path = f"{drawing_dir}/_rels/{drawing_file}.rels"
|
||||
if drawing_rels_path not in names:
|
||||
continue
|
||||
|
||||
drawing_rel_map = {}
|
||||
for rel in ET.fromstring(z.read(drawing_rels_path)).findall(f"{{{_REL_NS}}}Relationship"):
|
||||
drawing_rel_map[rel.get("Id")] = rel.get("Target")
|
||||
|
||||
media_paths = []
|
||||
for blip in ET.fromstring(z.read(drawing_path)).iter(f"{{{_A_NS}}}blip"):
|
||||
embed_rid = blip.get(f"{{{_R_NS}}}embed")
|
||||
if not embed_rid:
|
||||
continue
|
||||
rel_target = drawing_rel_map.get(embed_rid)
|
||||
if not rel_target:
|
||||
continue
|
||||
media_path = _normalize_rel_path(drawing_dir, rel_target)
|
||||
if media_path in names:
|
||||
media_paths.append(media_path)
|
||||
|
||||
if media_paths:
|
||||
result[sheet_name] = media_paths
|
||||
return result
|
||||
except (zipfile.BadZipFile, KeyError, OSError, ET.ParseError):
|
||||
return {}
|
||||
|
||||
|
||||
def _sanitize_filename_part(name: str) -> str:
|
||||
safe = re.sub(r"[^A-Za-z0-9_.-]+", "_", name).strip("_")
|
||||
return safe or "sheet"
|
||||
|
||||
|
||||
def extract_images(xlsx_path: Path, img_dir: Path):
|
||||
"""Extract embedded images from xlsx_path, grouped by sheet name.
|
||||
Returns {sheet_name: [filename, ...]} in per-sheet order. Files are
|
||||
named '<sanitized_sheet_name>_img{N:03d}.<ext>'."""
|
||||
sheet_media = _sheet_name_to_media(xlsx_path)
|
||||
if not sheet_media:
|
||||
return {}
|
||||
|
||||
written = {}
|
||||
with zipfile.ZipFile(xlsx_path) as z:
|
||||
names_in_zip = set(z.namelist())
|
||||
for sheet_idx, (sheet_name, media_paths) in enumerate(sheet_media.items(), start=1):
|
||||
safe_name = f"sheet{sheet_idx:03d}_{_sanitize_filename_part(sheet_name)}"
|
||||
files = []
|
||||
for idx, media_path in enumerate(media_paths, start=1):
|
||||
if media_path not in names_in_zip:
|
||||
print(f"WARNING: {media_path} not found in {xlsx_path}", file=sys.stderr)
|
||||
continue
|
||||
ext = Path(media_path).suffix.lstrip(".").lower() or "bin"
|
||||
if ext == "jpg":
|
||||
ext = "jpeg"
|
||||
out_name = f"{safe_name}_img{idx:03d}.{ext}"
|
||||
img_dir.mkdir(parents=True, exist_ok=True)
|
||||
(img_dir / out_name).write_bytes(z.read(media_path))
|
||||
files.append(out_name)
|
||||
if files:
|
||||
written[sheet_name] = files
|
||||
return written
|
||||
|
||||
|
||||
def insert_sheet_images(markdown_text: str, sheet_images) -> str:
|
||||
"""Insert a '#### Images in this sheet' block right after each sheet's
|
||||
section (before the next '## ' heading or end of text). If a sheet has
|
||||
no images, or no '## ' headings are found at all, the text is returned
|
||||
unchanged for that part."""
|
||||
if not sheet_images:
|
||||
return markdown_text
|
||||
matches = list(_SHEET_HEADER_RE.finditer(markdown_text))
|
||||
if not matches:
|
||||
return markdown_text
|
||||
|
||||
pieces = []
|
||||
last_end = 0
|
||||
for i, m in enumerate(matches):
|
||||
sheet_name = m.group(1).removesuffix("\r")
|
||||
start = m.start()
|
||||
end = matches[i + 1].start() if i + 1 < len(matches) else len(markdown_text)
|
||||
pieces.append(markdown_text[last_end:start])
|
||||
section = markdown_text[start:end].rstrip("\n")
|
||||
images = sheet_images.get(sheet_name)
|
||||
if images:
|
||||
section += "\n\n#### Images in this sheet\n\n"
|
||||
section += "\n".join(f"" for name in images)
|
||||
pieces.append(section + "\n\n")
|
||||
last_end = end
|
||||
pieces.append(markdown_text[last_end:])
|
||||
return "".join(pieces).rstrip() + "\n"
|
||||
|
||||
|
||||
def convert_one(md, source: Path, dest_dir: Path) -> bool:
|
||||
"""Convert a single .xlsx file to a '<name>/' folder containing the
|
||||
Markdown file and an 'img/' folder of extracted images. Returns True on
|
||||
success."""
|
||||
try:
|
||||
result = md.convert(str(source))
|
||||
except Exception as exc: # noqa: BLE001 - surface any conversion error
|
||||
print(f"FAILED {source} -> {exc}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
try:
|
||||
img_dir = dest_dir / "img"
|
||||
if dest_dir.exists():
|
||||
if img_dir.exists():
|
||||
shutil.rmtree(img_dir)
|
||||
dest_dir.mkdir(parents=True, exist_ok=True)
|
||||
sheet_images = extract_images(source, img_dir)
|
||||
text = insert_sheet_images(result.text_content, sheet_images)
|
||||
md_path = dest_dir / f"{source.stem}.md"
|
||||
md_path.write_text(text, encoding="utf-8")
|
||||
except OSError as exc:
|
||||
print(f"FAILED {source} -> could not write output in {dest_dir}: {exc}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
img_count = sum(len(v) for v in sheet_images.values())
|
||||
img_note = f", {img_count} image(s)" if img_count else ""
|
||||
print(f"OK {source} -> {md_path}{img_note}")
|
||||
return True
|
||||
|
||||
|
||||
def find_xlsx_files(root: Path, recursive: bool):
|
||||
"""Return (xlsx_files, skipped_count) for files directly/recursively under root."""
|
||||
pattern_iter = root.rglob("*") if recursive else root.iterdir()
|
||||
xlsx_files = []
|
||||
skipped = 0
|
||||
for entry in pattern_iter:
|
||||
if entry.is_dir():
|
||||
continue
|
||||
if entry.suffix.lower() == ".xlsx":
|
||||
xlsx_files.append(entry)
|
||||
else:
|
||||
skipped += 1
|
||||
return sorted(xlsx_files), skipped
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
parser.add_argument("input", help="Path to a .xlsx file or a directory of .xlsx files")
|
||||
parser.add_argument(
|
||||
"-o", "--output",
|
||||
help=(
|
||||
"Destination folder for the '<name>/' output (single-file mode), "
|
||||
"or parent directory under which each '<name>/' output folder is "
|
||||
"created (batch mode)"
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--recursive", action="store_true",
|
||||
help="When input is a directory, also search subdirectories",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
#MarkItDown = _import_markitdown()
|
||||
#md = MarkItDown()
|
||||
|
||||
source = Path(args.input)
|
||||
if not source.exists():
|
||||
print(f"ERROR: Input path not found: {source}", file=sys.stderr)
|
||||
return EXIT_INVALID_INPUT
|
||||
|
||||
if source.is_file() and source.suffix.lower() != ".xlsx":
|
||||
print(
|
||||
f"ERROR: Unsupported file type '{source.suffix}'. "
|
||||
"This skill only converts .xlsx files.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return EXIT_INVALID_INPUT
|
||||
|
||||
MarkItDown = _import_markitdown()
|
||||
md = MarkItDown()
|
||||
|
||||
if source.is_file():
|
||||
dest_dir = Path(args.output) if args.output else source.parent / source.stem
|
||||
return EXIT_OK if convert_one(md, source, dest_dir) else EXIT_CONVERSION_FAILED
|
||||
|
||||
# Directory / batch mode
|
||||
xlsx_files, skipped = find_xlsx_files(source, args.recursive)
|
||||
if skipped:
|
||||
print(f"NOTE: skipped {skipped} non-.xlsx file(s) in {source}")
|
||||
if not xlsx_files:
|
||||
print(f"ERROR: No .xlsx files found under {source}", file=sys.stderr)
|
||||
return EXIT_INVALID_INPUT
|
||||
|
||||
out_dir = Path(args.output) if args.output else None
|
||||
success_count = 0
|
||||
for xlsx_path in xlsx_files:
|
||||
if out_dir is not None:
|
||||
rel = xlsx_path.relative_to(source)
|
||||
dest_dir = out_dir / rel.parent / xlsx_path.stem
|
||||
else:
|
||||
dest_dir = xlsx_path.parent / xlsx_path.stem
|
||||
if convert_one(md, xlsx_path, dest_dir):
|
||||
success_count += 1
|
||||
|
||||
total = len(xlsx_files)
|
||||
print(f"\nConverted {success_count}/{total} file(s).")
|
||||
return EXIT_OK if success_count == total else EXIT_CONVERSION_FAILED
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1 @@
|
||||
markitdown[xlsx]>=0.1.0
|
||||
@@ -0,0 +1,130 @@
|
||||
---
|
||||
name: convert-pdf-to-md
|
||||
description: 'Converts PDF (.pdf) documents into Markdown so their contents can be accurately analyzed, summarized, searched, or extracted from. Use this skill whenever the user shares, references, or asks about a .pdf file — even if they don''t say "convert" or "markdown" explicitly. This includes requests to "read", "summarize", "review", "extract data from", "compare", or "analyze" a PDF report, paper, invoice, form, contract, or scanned document. Always run the bundled conversion script to produce Markdown first; do not attempt to parse PDF content directly or write ad-hoc extraction code. Also use this skill for batch requests involving a whole folder of PDF documents. IMPORTANT: When the user references a folder or set of documents containing multiple file types (.pdf, .docx, .xlsx), invoke ALL three sibling skills — convert-pdf-to-md, convert-word-to-md, and convert-excel-to-md — so no file type is silently skipped.'
|
||||
---
|
||||
|
||||
# Convert PDF to Markdown
|
||||
|
||||
## When to use this skill
|
||||
|
||||
Trigger this skill any time there is a `.pdf` file that needs to be
|
||||
understood or processed — for example, a user attaches a PDF and asks
|
||||
questions about it, wants a summary, wants specific data or tables pulled
|
||||
out, or wants multiple PDFs in a folder processed together. PDF is a
|
||||
layout/print format, not reliably readable as plain text, so always convert
|
||||
it to Markdown first using the script in this skill rather than trying to
|
||||
open or parse the file directly.
|
||||
|
||||
This skill only supports `.pdf` — that's MarkItDown's only PDF-family
|
||||
format, so there's no legacy format to worry about here (unlike Word's
|
||||
`.doc` or Excel's `.xls`).
|
||||
|
||||
**Mixed file types:** When the user references a folder or set of documents
|
||||
containing multiple supported file types (`.pdf`, `.docx`, `.xlsx`), this
|
||||
skill handles only `.pdf` files. The agent MUST also invoke the sibling
|
||||
skills in parallel:
|
||||
- `convert-word-to-md` for any `.docx` files
|
||||
- `convert-excel-to-md` for any `.xlsx` files
|
||||
|
||||
Never process a folder and silently skip a supported file type. All three
|
||||
skills must be invoked together when mixed types are present.
|
||||
|
||||
## Setup (once per environment)
|
||||
|
||||
Before the first conversion in a given environment, follow
|
||||
[`references/setup.md`](references/setup.md) step by step to ensure Python,
|
||||
pip, `markitdown`, and `pymupdf` (for image extraction) are installed. Do
|
||||
this proactively rather than guessing whether the environment is ready — the
|
||||
script itself will also fail with a clear pointer back to that file if a
|
||||
dependency turns out to be missing, so it's safe to just try the conversion
|
||||
first if you're reasonably confident setup was already done.
|
||||
|
||||
## Usage
|
||||
|
||||
The conversion script lives at `scripts/convert_pdf_to_md.py`.
|
||||
|
||||
**Output structure:** MarkItDown's PDF converter extracts text and tables
|
||||
only — it has no concept of embedded images at all. This script separately
|
||||
extracts real embedded images via PyMuPDF and writes a self-contained folder
|
||||
per document:
|
||||
|
||||
```
|
||||
<name>/
|
||||
img/
|
||||
page001_img001.<ext>
|
||||
page002_img001.<ext>
|
||||
...
|
||||
<name>.md
|
||||
```
|
||||
|
||||
Because MarkItDown's PDF text does not preserve reliable per-page markers,
|
||||
there's no safe way to know exactly where inline an image belongs. Rather
|
||||
than risk misplacing images next to the wrong paragraph, the script appends
|
||||
a `## Extracted Images` section at the end of the Markdown, with a
|
||||
`### Page N` subheading per page that has images — read this section
|
||||
separately from the main body text. If the document has no embedded images,
|
||||
no `img/` folder or `Extracted Images` section is created.
|
||||
|
||||
**Single file:**
|
||||
|
||||
```powershell
|
||||
python scripts\convert_pdf_to_md.py "C:\path\to\document.pdf"
|
||||
```
|
||||
|
||||
This creates a `document\` folder next to the source file (containing
|
||||
`document.md` and, if present, `document\img\`). To control the destination
|
||||
folder explicitly:
|
||||
|
||||
```powershell
|
||||
python scripts\convert_pdf_to_md.py "C:\path\to\document.pdf" -o "C:\path\to\output_folder"
|
||||
```
|
||||
|
||||
**A folder of PDFs (batch mode):**
|
||||
|
||||
```powershell
|
||||
python scripts\convert_pdf_to_md.py "C:\path\to\folder"
|
||||
```
|
||||
|
||||
Add `--recursive` to also include subfolders:
|
||||
|
||||
```powershell
|
||||
python scripts\convert_pdf_to_md.py "C:\path\to\folder" --recursive
|
||||
```
|
||||
|
||||
Each `.pdf` found gets its own `<name>\` output folder next to it by
|
||||
default. Pass `-o "C:\path\to\output_parent"` to collect all the generated
|
||||
`<name>\` folders under a separate parent directory instead (subfolder
|
||||
structure is preserved when combined with `--recursive`).
|
||||
|
||||
After conversion, read the resulting `.md` file(s) to perform the actual
|
||||
analysis the user asked for — the script's job is only to produce accurate
|
||||
Markdown (and images), not to interpret the content.
|
||||
|
||||
## Deciding where output goes
|
||||
|
||||
**Default — always output next to the source file.** The `<name>/` folder
|
||||
is created in the same directory as the source `.pdf`. This is the required
|
||||
default for every case. Do NOT override it unless the user explicitly asks
|
||||
for a different location.
|
||||
|
||||
**Only use `-o` when** the user explicitly provides an output path (e.g.,
|
||||
"save the output to `C:\output`", "put the results in `D:\work`"). Do NOT
|
||||
pass `-o` based on the agent's current working directory, the session state
|
||||
folder, or any implied location.
|
||||
|
||||
**If the source file path cannot be fully resolved** — for example, the
|
||||
user provides only a filename with no directory, or the path is ambiguous —
|
||||
use `ask_user` to confirm the full absolute path before running the
|
||||
conversion. Never guess or assume the directory.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Likely cause | Fix |
|
||||
|---|---|---|
|
||||
| `ModuleNotFoundError: No module named 'markitdown'` or `'fitz'` / exit code 2 | MarkItDown or PyMuPDF not installed | Follow `references/setup.md` |
|
||||
| `ERROR: Unsupported file type '...'` / exit code 3 | Not a `.pdf` file | Ask the user for the correct file, or if it's `.doc`/`.docx`/`.xlsx`, use the matching sibling skill instead |
|
||||
| `ERROR: Input path not found` / exit code 3 | Wrong path, or file moved | Confirm the correct path with the user |
|
||||
| `FAILED <file> -> ...` in batch output | That specific file is corrupt, password-protected, or otherwise unreadable | Report which file(s) failed; other files in the batch still succeed |
|
||||
| `NOTE: skipped N non-.pdf file(s)` | Folder contains non-PDF files | Expected — those files are intentionally ignored |
|
||||
| Markdown body is empty or near-empty despite images being extracted | The PDF is scanned/image-only with no embedded text layer; MarkItDown does not perform OCR | Tell the user OCR isn't supported — the extracted page images are still available for them to view |
|
||||
| Images appear in an appendix instead of inline with the text | Deliberate limitation — MarkItDown's PDF text has no reliable per-page markers to place images inline | Expected behavior; cross-reference the `### Page N` heading with the surrounding text context if needed |
|
||||
@@ -0,0 +1,71 @@
|
||||
# Environment Setup for convert-pdf-to-md
|
||||
|
||||
Follow these steps exactly, in order, before running `scripts/convert_pdf_to_md.py`
|
||||
for the first time in a given environment. Don't skip steps or improvise
|
||||
alternatives — they're written to be deterministic and safe to re-run.
|
||||
|
||||
## 1. Check Python is available (3.10+)
|
||||
|
||||
```powershell
|
||||
python --version
|
||||
```
|
||||
|
||||
- If this fails (command not found), install Python 3.10 or newer:
|
||||
- Windows: `winget install --id Python.Python.3.12 -e`
|
||||
- macOS: `brew install python@3.12`
|
||||
- Linux (Debian/Ubuntu): `sudo apt-get update && sudo apt-get install -y python3 python3-pip python-is-python3`
|
||||
- If the reported version is older than 3.10, install a newer Python using
|
||||
the same command above (MarkItDown requires 3.10+).
|
||||
|
||||
## 2. Check pip is available
|
||||
|
||||
```powershell
|
||||
python -m pip --version
|
||||
```
|
||||
|
||||
- If this fails, bootstrap pip:
|
||||
|
||||
```powershell
|
||||
python -m ensurepip --upgrade
|
||||
```
|
||||
|
||||
## 3. Install MarkItDown with PDF support, plus PyMuPDF for image extraction
|
||||
|
||||
Use the `scripts/requirements.txt` file bundled with this skill to install pinned,
|
||||
known-good versions of the dependencies:
|
||||
|
||||
```powershell
|
||||
python -m pip install -r scripts/requirements.txt
|
||||
```
|
||||
|
||||
This pulls in `markitdown[pdf]` and `pymupdf>=1.24.0`. PyMuPDF (imported as `fitz`)
|
||||
is required separately because MarkItDown's PDF
|
||||
converter only extracts text and tables — it has no support for embedded
|
||||
images at all, so this skill's script extracts them itself.
|
||||
|
||||
## 4. Verify the install
|
||||
|
||||
```powershell
|
||||
python -c "from markitdown import MarkItDown; import fitz; print('markitdown + pymupdf OK')"
|
||||
```
|
||||
|
||||
Expect to see `markitdown + pymupdf OK` printed with no errors. If you see a
|
||||
`ModuleNotFoundError`, repeat step 3 — pip may be installing into a
|
||||
different Python environment than the one being invoked (check
|
||||
`python -m pip --version` shows the same path as `python --version`'s
|
||||
interpreter).
|
||||
|
||||
## Notes
|
||||
|
||||
- This setup only needs to be done once per environment/virtual environment,
|
||||
not once per conversion.
|
||||
- `convert_pdf_to_md.py` itself also checks for `markitdown` and `fitz` at
|
||||
startup and prints a pointer back to this file if either is missing, so
|
||||
re-running setup is safe and idempotent.
|
||||
- Only `.pdf` is supported by this skill — it's MarkItDown's only PDF-family
|
||||
format, so there's no legacy-format equivalent to worry about (unlike
|
||||
Word's `.doc` or Excel's `.xls`).
|
||||
- Scanned/image-only PDFs (no embedded text layer) will produce little or
|
||||
no text from MarkItDown, since it does not perform OCR. The images
|
||||
themselves will still be extracted and appended, but the text body may be
|
||||
empty or near-empty in that case — mention this to the user if it happens.
|
||||
@@ -0,0 +1,321 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Convert PDF documents to Markdown using Microsoft's MarkItDown, with
|
||||
embedded images extracted to real files via PyMuPDF (MarkItDown's PDF
|
||||
converter only extracts text/tables -- it does not detect or emit anything
|
||||
for embedded images at all).
|
||||
|
||||
Usage:
|
||||
python convert_pdf_to_md.py <input> [-o OUTPUT] [--recursive]
|
||||
|
||||
<input> may be either:
|
||||
- a path to a single .pdf file, or
|
||||
- a path to a directory (batch mode: every .pdf file directly inside it
|
||||
is converted; pass --recursive to also descend into subdirectories).
|
||||
|
||||
Output:
|
||||
For each source .pdf (named "<name>.pdf"), a folder is created containing
|
||||
the Markdown and its images, in this layout:
|
||||
|
||||
<name>/
|
||||
img/
|
||||
page001_img001.<ext>
|
||||
page001_img002.<ext>
|
||||
page002_img001.<ext>
|
||||
...
|
||||
<name>.md
|
||||
|
||||
IMPORTANT: MarkItDown's PDF text extraction does not preserve reliable
|
||||
per-page markers in the returned Markdown (pages are simply joined
|
||||
together, or in some cases returned as a single unmarked block of text).
|
||||
That means there is no safe way to know exactly where, inline, an image
|
||||
should go. Rather than guess and risk misplacing an image next to the
|
||||
wrong paragraph, this script appends a clearly labeled "## Extracted
|
||||
Images" section at the end of the Markdown, with a "### Page N"
|
||||
subheading per page that contains images. This is a deliberate, honest
|
||||
tradeoff -- read the images section separately from the main body text.
|
||||
|
||||
- Single file mode: the "<name>/" folder is created next to the source
|
||||
file, or at -o/--output (treated as the exact destination folder) if
|
||||
given.
|
||||
- Batch/directory mode: a "<name>/" folder is created next to each source
|
||||
file, or under -o/--output (treated as a parent directory, created if
|
||||
missing) if given, preserving relative subfolder structure when
|
||||
--recursive is used.
|
||||
- If a document has no embedded images, no "img/" folder or "Extracted
|
||||
Images" section is created.
|
||||
|
||||
Exit codes:
|
||||
0 - all requested conversions succeeded
|
||||
1 - one or more conversions failed (partial success in batch mode)
|
||||
2 - a required dependency ("markitdown" or "pymupdf") is not installed
|
||||
3 - invalid input (path not found, or single-file input is not .pdf)
|
||||
"""
|
||||
import argparse
|
||||
import sys
|
||||
import hashlib
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
EXIT_OK = 0
|
||||
EXIT_CONVERSION_FAILED = 1
|
||||
EXIT_MISSING_DEPENDENCY = 2
|
||||
EXIT_INVALID_INPUT = 3
|
||||
|
||||
|
||||
def _import_markitdown():
|
||||
"""Import MarkItDown, failing with a clear, actionable message if absent."""
|
||||
try:
|
||||
from markitdown import MarkItDown
|
||||
return MarkItDown
|
||||
except ImportError:
|
||||
print(
|
||||
"ERROR: The 'markitdown' package is not installed.\n"
|
||||
"See references/setup.md for this skill, or run:\n"
|
||||
' pip install "markitdown[pdf]"',
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(EXIT_MISSING_DEPENDENCY)
|
||||
|
||||
|
||||
def _import_fitz():
|
||||
"""Import PyMuPDF (module name 'fitz'), failing with a clear message if absent."""
|
||||
try:
|
||||
import fitz
|
||||
import hashlib
|
||||
return fitz
|
||||
except ImportError:
|
||||
print(
|
||||
"ERROR: The 'pymupdf' package is not installed (needed for image "
|
||||
"extraction).\nSee references/setup.md for this skill, or run:\n"
|
||||
" pip install pymupdf",
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(EXIT_MISSING_DEPENDENCY)
|
||||
|
||||
|
||||
def extract_images(fitz, pdf_path: Path, img_dir: Path):
|
||||
"""Extract embedded images from pdf_path, grouped by 1-based page number.
|
||||
Returns {page_num: [filename, ...]} in per-page image order. Files are
|
||||
named 'page{P:03d}_img{N:03d}.<ext>'. Corrupt/unreadable images are
|
||||
skipped with a warning rather than aborting the whole conversion.
|
||||
|
||||
Two sources are combined and deduplicated:
|
||||
1. Image XObjects via page.get_images(full=True) -- covers most embedded
|
||||
images in modern PDFs.
|
||||
2. Inline image blocks via page.get_text("dict") -- covers images stored
|
||||
directly in the page content stream, which get_images() misses entirely.
|
||||
Deduplication is by image bytes hash so the same raster is never written twice
|
||||
on the same page regardless of which source reported it."""
|
||||
written_by_page = {}
|
||||
try:
|
||||
doc = fitz.open(str(pdf_path))
|
||||
except Exception as exc: # noqa: BLE001
|
||||
print(f"WARNING: could not open {pdf_path} for image extraction: {exc}", file=sys.stderr)
|
||||
return written_by_page
|
||||
|
||||
try:
|
||||
for page_index in range(len(doc)):
|
||||
page = doc[page_index]
|
||||
page_label = page_index + 1
|
||||
seen_hashes: set = set()
|
||||
raw_images: list[tuple[bytes, str]] = [] # (image_bytes, ext)
|
||||
|
||||
# --- Source 1: XObject images ---
|
||||
try:
|
||||
xobjects = page.get_images(full=True)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
print(
|
||||
f"WARNING: failed to enumerate XObject images on page {page_label} "
|
||||
f"of {pdf_path}: {exc}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
xobjects = []
|
||||
|
||||
for img in xobjects:
|
||||
xref = img[0]
|
||||
try:
|
||||
base_image = doc.extract_image(xref)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
print(
|
||||
f"WARNING: failed to extract XObject image xref={xref} on page "
|
||||
f"{page_label} of {pdf_path}: {exc}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
continue
|
||||
img_bytes = base_image.get("image") or b""
|
||||
if not img_bytes:
|
||||
continue
|
||||
ext = (base_image.get("ext") or "png").lower()
|
||||
raw_images.append((img_bytes, ext))
|
||||
|
||||
# --- Source 2: Inline images via get_text("dict") ---
|
||||
try:
|
||||
blocks = page.get_text("dict", flags=fitz.TEXT_PRESERVE_IMAGES).get("blocks", [])
|
||||
except Exception as exc: # noqa: BLE001
|
||||
print(
|
||||
f"WARNING: failed to extract text/image dict on page {page_label} "
|
||||
f"of {pdf_path}: {exc}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
blocks = []
|
||||
|
||||
for block in blocks:
|
||||
# Image blocks have type == 1
|
||||
if block.get("type") != 1:
|
||||
continue
|
||||
img_bytes = block.get("image") or b""
|
||||
if not img_bytes:
|
||||
continue
|
||||
# Derive extension from the block's "ext" key (fitz sets this)
|
||||
ext = (block.get("ext") or "png").lower()
|
||||
raw_images.append((img_bytes, ext))
|
||||
|
||||
# --- Write deduplicated images ---
|
||||
page_files = []
|
||||
img_idx = 1
|
||||
for img_bytes, ext in raw_images:
|
||||
h = hashlib.sha256(img_bytes).digest()
|
||||
if h in seen_hashes:
|
||||
continue
|
||||
seen_hashes.add(h)
|
||||
out_name = f"page{page_label:03d}_img{img_idx:03d}.{ext}"
|
||||
img_dir.mkdir(parents=True, exist_ok=True)
|
||||
(img_dir / out_name).write_bytes(img_bytes)
|
||||
page_files.append(out_name)
|
||||
img_idx += 1
|
||||
|
||||
if page_files:
|
||||
written_by_page[page_label] = page_files
|
||||
finally:
|
||||
doc.close()
|
||||
|
||||
return written_by_page
|
||||
|
||||
|
||||
def build_image_appendix(written_by_page) -> str:
|
||||
"""Build the '## Extracted Images' appendix text. Returns "" if empty."""
|
||||
if not written_by_page:
|
||||
return ""
|
||||
lines = ["", "## Extracted Images", ""]
|
||||
for page_num in sorted(written_by_page):
|
||||
lines.append(f"### Page {page_num}")
|
||||
lines.append("")
|
||||
for name in written_by_page[page_num]:
|
||||
lines.append(f"")
|
||||
lines.append("")
|
||||
return "\n".join(lines).rstrip() + "\n"
|
||||
|
||||
|
||||
def convert_one(md, fitz, source: Path, dest_dir: Path) -> bool:
|
||||
"""Convert a single .pdf file to a '<name>/' folder containing the
|
||||
Markdown file and an 'img/' folder of extracted images. Returns True on
|
||||
success."""
|
||||
try:
|
||||
result = md.convert(str(source))
|
||||
except Exception as exc: # noqa: BLE001 - surface any conversion error
|
||||
print(f"FAILED {source} -> {exc}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
try:
|
||||
if dest_dir.exists():
|
||||
shutil.rmtree(dest_dir)
|
||||
dest_dir.mkdir(parents=True, exist_ok=True)
|
||||
written_by_page = extract_images(fitz, source, dest_dir / "img")
|
||||
appendix = build_image_appendix(written_by_page)
|
||||
text = result.text_content.rstrip("\n")
|
||||
full_text = f"{text}\n{appendix}" if appendix else f"{text}\n"
|
||||
md_path = dest_dir / f"{source.stem}.md"
|
||||
md_path.write_text(full_text, encoding="utf-8")
|
||||
except OSError as exc:
|
||||
print(f"FAILED {source} -> could not write output in {dest_dir}: {exc}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
img_count = sum(len(v) for v in written_by_page.values())
|
||||
img_note = f", {img_count} image(s)" if img_count else ""
|
||||
print(f"OK {source} -> {md_path}{img_note}")
|
||||
return True
|
||||
|
||||
|
||||
def find_pdf_files(root: Path, recursive: bool):
|
||||
"""Return (pdf_files, skipped_count) for files directly/recursively under root."""
|
||||
pattern_iter = root.rglob("*") if recursive else root.iterdir()
|
||||
pdf_files = []
|
||||
skipped = 0
|
||||
for entry in pattern_iter:
|
||||
if entry.is_dir():
|
||||
continue
|
||||
if entry.suffix.lower() == ".pdf":
|
||||
pdf_files.append(entry)
|
||||
else:
|
||||
skipped += 1
|
||||
return sorted(pdf_files), skipped
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
parser.add_argument("input", help="Path to a .pdf file or a directory of .pdf files")
|
||||
parser.add_argument(
|
||||
"-o", "--output",
|
||||
help=(
|
||||
"Destination folder for the '<name>/' output (single-file mode), "
|
||||
"or parent directory under which each '<name>/' output folder is "
|
||||
"created (batch mode)"
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--recursive", action="store_true",
|
||||
help="When input is a directory, also search subdirectories",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
#MarkItDown = _import_markitdown()
|
||||
#fitz = _import_fitz()
|
||||
#md = MarkItDown()
|
||||
|
||||
source = Path(args.input)
|
||||
if not source.exists():
|
||||
print(f"ERROR: Input path not found: {source}", file=sys.stderr)
|
||||
return EXIT_INVALID_INPUT
|
||||
|
||||
if source.is_file() and source.suffix.lower() != ".pdf":
|
||||
print(
|
||||
f"ERROR: Unsupported file type '{source.suffix}'. "
|
||||
"This skill only converts .pdf files.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return EXIT_INVALID_INPUT
|
||||
|
||||
MarkItDown = _import_markitdown()
|
||||
fitz = _import_fitz()
|
||||
md = MarkItDown()
|
||||
|
||||
if source.is_file():
|
||||
dest_dir = Path(args.output) if args.output else source.parent / source.stem
|
||||
return EXIT_OK if convert_one(md, fitz, source, dest_dir) else EXIT_CONVERSION_FAILED
|
||||
|
||||
# Directory / batch mode
|
||||
pdf_files, skipped = find_pdf_files(source, args.recursive)
|
||||
if skipped:
|
||||
print(f"NOTE: skipped {skipped} non-.pdf file(s) in {source}")
|
||||
if not pdf_files:
|
||||
print(f"ERROR: No .pdf files found under {source}", file=sys.stderr)
|
||||
return EXIT_INVALID_INPUT
|
||||
|
||||
out_dir = Path(args.output) if args.output else None
|
||||
success_count = 0
|
||||
for pdf_path in pdf_files:
|
||||
if out_dir is not None:
|
||||
rel = pdf_path.relative_to(source)
|
||||
dest_dir = out_dir / rel.parent / pdf_path.stem
|
||||
else:
|
||||
dest_dir = pdf_path.parent / pdf_path.stem
|
||||
if convert_one(md, fitz, pdf_path, dest_dir):
|
||||
success_count += 1
|
||||
|
||||
total = len(pdf_files)
|
||||
print(f"\nConverted {success_count}/{total} file(s).")
|
||||
return EXIT_OK if success_count == total else EXIT_CONVERSION_FAILED
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,2 @@
|
||||
markitdown[pdf]>=0.1.0
|
||||
pymupdf>=1.24.0
|
||||
@@ -0,0 +1,129 @@
|
||||
---
|
||||
name: convert-word-to-md
|
||||
description: 'Converts Word (.docx) documents into Markdown so their contents can be accurately analyzed, summarized, searched, or extracted from. Use this skill whenever the user shares, references, or asks about a .docx file — even if they don''t say "convert" or "markdown" explicitly. This includes requests to "read", "summarize", "review", "extract data from", "compare", or "analyze" a Word document, resume, report, contract, or proposal. Always run the bundled conversion script to produce Markdown first; do not attempt to parse .docx content directly or write ad-hoc conversion code. Also use this skill for batch requests involving a whole folder of Word documents. IMPORTANT: When the user references a folder or set of documents containing multiple file types (.pdf, .docx, .xlsx), invoke ALL three sibling skills — convert-pdf-to-md, convert-word-to-md, and convert-excel-to-md — so no file type is silently skipped.'
|
||||
---
|
||||
|
||||
# Convert Word to Markdown
|
||||
|
||||
## When to use this skill
|
||||
|
||||
Trigger this skill any time there is a `.docx` file that needs to be
|
||||
understood or processed — for example, a user attaches a Word document and
|
||||
asks questions about it, wants a summary, wants specific data pulled out, or
|
||||
wants multiple Word documents in a folder processed together. Word's native
|
||||
`.docx` format is a zipped XML bundle that is not reliably readable as plain
|
||||
text, so always convert it to Markdown first using the script in this
|
||||
skill rather than trying to open or parse the file directly.
|
||||
|
||||
This skill only supports `.docx`. If asked to convert a legacy `.doc` file,
|
||||
tell the user it isn't supported and ask them to re-save it as `.docx`
|
||||
(Word: File > Save As > Word Document (.docx)) first.
|
||||
|
||||
**Mixed file types:** When the user references a folder or set of documents
|
||||
containing multiple supported file types (`.pdf`, `.docx`, `.xlsx`), this
|
||||
skill handles only `.docx` files. The agent MUST also invoke the sibling
|
||||
skills in parallel:
|
||||
- `convert-pdf-to-md` for any `.pdf` files
|
||||
- `convert-excel-to-md` for any `.xlsx` files
|
||||
|
||||
Never process a folder and silently skip a supported file type. All three
|
||||
skills must be invoked together when mixed types are present.
|
||||
|
||||
## Setup (once per environment)
|
||||
|
||||
Before the first conversion in a given environment, follow
|
||||
[`references/setup.md`](references/setup.md) step by step to ensure Python,
|
||||
pip, and the `markitdown` package are installed. Do this proactively rather
|
||||
than guessing whether the environment is ready — the script itself will
|
||||
also fail with a clear pointer back to that file if `markitdown` turns out
|
||||
to be missing, so it's safe to just try the conversion first if you're
|
||||
reasonably confident setup was already done.
|
||||
|
||||
## Usage
|
||||
|
||||
The conversion script lives at `scripts/convert_word_to_md.py`.
|
||||
|
||||
**Output structure:** MarkItDown embeds images as a truncated `data:image/png;base64...` URI
|
||||
placeholder (not real image data), so the script
|
||||
extracts real images directly from the `.docx` and writes a self-contained
|
||||
folder per document instead of a single loose `.md` file:
|
||||
|
||||
```
|
||||
<name>/
|
||||
img/
|
||||
img001.<ext>
|
||||
img002.<ext>
|
||||
...
|
||||
<name>.md (image references are relative: img/imgNNN.ext)
|
||||
```
|
||||
|
||||
If the document has no embedded images, no `img/` folder is created.
|
||||
|
||||
**Single file:**
|
||||
|
||||
```powershell
|
||||
# Windows
|
||||
python scripts\convert_word_to_md.py "C:\path\to\document.docx"
|
||||
```
|
||||
|
||||
```bash
|
||||
# macOS / Linux
|
||||
python scripts/convert_word_to_md.py "/path/to/document.docx"
|
||||
```
|
||||
|
||||
This creates a `document\` folder next to the source file (containing
|
||||
`document.md` and, if present, `document\img\`). To control the destination
|
||||
folder explicitly:
|
||||
|
||||
```powershell
|
||||
python scripts\convert_word_to_md.py "C:\path\to\document.docx" -o "C:\path\to\output_folder"
|
||||
```
|
||||
|
||||
**A folder of Word documents (batch mode):**
|
||||
|
||||
```powershell
|
||||
python scripts\convert_word_to_md.py "C:\path\to\folder"
|
||||
```
|
||||
|
||||
Add `--recursive` to also include subfolders:
|
||||
|
||||
```powershell
|
||||
python scripts\convert_word_to_md.py "C:\path\to\folder" --recursive
|
||||
```
|
||||
|
||||
Each `.docx` found gets its own `<name>\` output folder next to it by
|
||||
default. Pass `-o "C:\path\to\output_parent"` to collect all the generated
|
||||
`<name>\` folders under a separate parent directory instead (subfolder
|
||||
structure is preserved when combined with `--recursive`).
|
||||
|
||||
After conversion, read the resulting `.md` file(s) to perform the actual
|
||||
analysis the user asked for — the script's job is only to produce accurate
|
||||
Markdown (and images), not to interpret the content.
|
||||
|
||||
## Deciding where output goes
|
||||
|
||||
**Default — always output next to the source file.** The `<name>/` folder
|
||||
is created in the same directory as the source `.docx`. This is the required
|
||||
default for every case. Do NOT override it unless the user explicitly asks
|
||||
for a different location.
|
||||
|
||||
**Only use `-o` when** the user explicitly provides an output path (e.g.,
|
||||
"save the output to `C:\output`", "put the results in `D:\work`"). Do NOT
|
||||
pass `-o` based on the agent's current working directory, the session state
|
||||
folder, or any implied location.
|
||||
|
||||
**If the source file path cannot be fully resolved** — for example, the
|
||||
user provides only a filename with no directory, or the path is ambiguous —
|
||||
use `ask_user` to confirm the full absolute path before running the
|
||||
conversion. Never guess or assume the directory.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Likely cause | Fix |
|
||||
|---|---|---|
|
||||
| `ModuleNotFoundError: No module named 'markitdown'` / exit code 2 | MarkItDown not installed | Follow `references/setup.md` |
|
||||
| `ERROR: Unsupported file type '.doc'` / exit code 3 | Legacy `.doc`, not `.docx` | Ask the user to re-save as `.docx` |
|
||||
| `ERROR: Input path not found` / exit code 3 | Wrong path, or file moved | Confirm the correct path with the user |
|
||||
| `FAILED <file> -> ...` in batch output | That specific file is corrupt, password-protected, or otherwise unreadable | Report which file(s) failed; other files in the batch still succeed |
|
||||
| `NOTE: skipped N non-.docx file(s)` | Folder contains non-Word files | Expected — those files are intentionally ignored |
|
||||
| `WARNING: found N image placeholder(s) ... but extracted M image file(s)` | Mismatch between MarkItDown's placeholder count and images found in `word/media/` (unusual/malformed docx) | Placeholders are left unreplaced rather than risk wrong images; inspect the source file's media manually if images are needed |
|
||||
@@ -0,0 +1,66 @@
|
||||
# Environment Setup for convert-word-to-md
|
||||
|
||||
Follow these steps exactly, in order, before running `scripts/convert_word_to_md.py`
|
||||
for the first time in a given environment. Don't skip steps or improvise
|
||||
alternatives — they're written to be deterministic and safe to re-run.
|
||||
|
||||
## 1. Check Python is available (3.10+)
|
||||
|
||||
```powershell
|
||||
python --version
|
||||
```
|
||||
|
||||
- If this fails (command not found), install Python 3.10 or newer:
|
||||
- Windows: `winget install --id Python.Python.3.12 -e`
|
||||
- macOS: `brew install python@3.12`
|
||||
- Linux (Debian/Ubuntu): `sudo apt-get update && sudo apt-get install -y python3 python3-pip python-is-python3`
|
||||
- If the reported version is older than 3.10, install a newer Python using
|
||||
the same command above (MarkItDown requires 3.10+).
|
||||
|
||||
## 2. Check pip is available
|
||||
|
||||
```powershell
|
||||
python -m pip --version
|
||||
```
|
||||
|
||||
- If this fails, bootstrap pip:
|
||||
|
||||
```powershell
|
||||
python -m ensurepip --upgrade
|
||||
```
|
||||
|
||||
## 3. Install MarkItDown with Word (.docx) support
|
||||
|
||||
Use the `scripts/requirements.txt` file bundled with this skill to install a pinned,
|
||||
known-good version of the dependency:
|
||||
|
||||
```powershell
|
||||
python -m pip install -r scripts/requirements.txt
|
||||
```
|
||||
|
||||
This pulls in `markitdown[docx]` (MarkItDown's Word conversion dependency, which
|
||||
includes `mammoth` for `.docx` file parsing). No extra package is needed — this
|
||||
skill's script uses MarkItDown's built-in Word converter.
|
||||
|
||||
## 4. Verify the install
|
||||
|
||||
```powershell
|
||||
python -c "from markitdown import MarkItDown; print('markitdown OK')"
|
||||
```
|
||||
|
||||
Expect to see `markitdown OK` printed with no errors. If you see
|
||||
`ModuleNotFoundError: No module named 'markitdown'`, repeat step 3 — pip may
|
||||
be installing into a different Python environment than the one being
|
||||
invoked (check `python -m pip --version` shows the same path as `python
|
||||
--version`'s interpreter).
|
||||
|
||||
## Notes
|
||||
|
||||
- This setup only needs to be done once per environment/virtual environment,
|
||||
not once per conversion.
|
||||
- `convert_word_to_md.py` itself also checks for `markitdown` at startup and
|
||||
prints a pointer back to this file if it's missing, so re-running setup is
|
||||
safe and idempotent.
|
||||
- Only `.docx` is supported by this skill. Legacy binary `.doc` files are
|
||||
out of scope — ask the user to re-save the file as `.docx` (e.g., via
|
||||
Word's "Save As") if one is encountered.
|
||||
@@ -0,0 +1,301 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Convert Word (.docx) documents to Markdown using Microsoft's MarkItDown,
|
||||
with embedded images extracted to real files (MarkItDown only emits a
|
||||
truncated `data:image/...;base64...` placeholder, not real image data).
|
||||
|
||||
Usage:
|
||||
python convert_word_to_md.py <input> [-o OUTPUT] [--recursive]
|
||||
|
||||
<input> may be either:
|
||||
- a path to a single .docx file, or
|
||||
- a path to a directory (batch mode: every .docx file directly inside it
|
||||
is converted; pass --recursive to also descend into subdirectories).
|
||||
|
||||
Output:
|
||||
For each source .docx (named "<name>.docx"), a folder is created
|
||||
containing the Markdown and its images, in this layout:
|
||||
|
||||
<name>/
|
||||
img/
|
||||
img001.<ext>
|
||||
img002.<ext>
|
||||
...
|
||||
<name>.md (image references are relative: img/imgNNN.ext)
|
||||
|
||||
- Single file mode: the "<name>/" folder is created next to the source
|
||||
file, or at -o/--output (treated as the exact destination folder) if
|
||||
given.
|
||||
- Batch/directory mode: a "<name>/" folder is created next to each source
|
||||
file, or under -o/--output (treated as a parent directory, created if
|
||||
missing) if given, preserving relative subfolder structure when
|
||||
--recursive is used.
|
||||
- If a document has no embedded images, no "img/" folder is created.
|
||||
|
||||
Exit codes:
|
||||
0 - all requested conversions succeeded
|
||||
1 - one or more conversions failed (partial success in batch mode)
|
||||
2 - required dependency ("markitdown") is not installed
|
||||
3 - invalid input (path not found, or single-file input is not .docx)
|
||||
"""
|
||||
import argparse
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
from xml.etree import ElementTree as ET
|
||||
|
||||
EXIT_OK = 0
|
||||
EXIT_CONVERSION_FAILED = 1
|
||||
EXIT_MISSING_DEPENDENCY = 2
|
||||
EXIT_INVALID_INPUT = 3
|
||||
|
||||
_W_NS = "http://schemas.openxmlformats.org/wordprocessingml/2006/main"
|
||||
_R_NS = "http://schemas.openxmlformats.org/officeDocument/2006/relationships"
|
||||
_REL_NS = "http://schemas.openxmlformats.org/package/2006/relationships"
|
||||
|
||||
# MarkItDown embeds images as a literal truncated placeholder, e.g.
|
||||
#  -- NOT real base64 data. This pattern
|
||||
# matches that placeholder so it can be swapped for a real relative path.
|
||||
_PLACEHOLDER_IMAGE_RE = re.compile(
|
||||
r'!\[([^\]]*)\]\(data:image/[a-zA-Z0-9.+-]+;base64[^)]*\)'
|
||||
)
|
||||
|
||||
|
||||
def _import_markitdown():
|
||||
"""Import MarkItDown, failing with a clear, actionable message if absent."""
|
||||
try:
|
||||
from markitdown import MarkItDown
|
||||
return MarkItDown
|
||||
except ImportError:
|
||||
print(
|
||||
"ERROR: The 'markitdown' package is not installed.\n"
|
||||
"See references/setup.md for this skill, or run:\n"
|
||||
' pip install "markitdown[docx]"',
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(EXIT_MISSING_DEPENDENCY)
|
||||
|
||||
|
||||
def _document_order_media(docx_path: Path):
|
||||
"""Return [(rel_id, media_zip_path), ...] in the order images appear in
|
||||
word/document.xml (via r:embed / r:id), resolved through
|
||||
word/_rels/document.xml.rels. Returns [] if the document has no body
|
||||
part or no images (e.g. malformed docx falls back gracefully)."""
|
||||
try:
|
||||
with zipfile.ZipFile(docx_path) as z:
|
||||
if "word/document.xml" not in z.namelist() or \
|
||||
"word/_rels/document.xml.rels" not in z.namelist():
|
||||
return []
|
||||
rels_xml = z.read("word/_rels/document.xml.rels")
|
||||
doc_xml = z.read("word/document.xml")
|
||||
except (zipfile.BadZipFile, KeyError, OSError):
|
||||
return []
|
||||
|
||||
try:
|
||||
rels_root = ET.fromstring(rels_xml)
|
||||
doc_root = ET.fromstring(doc_xml)
|
||||
except ET.ParseError:
|
||||
return []
|
||||
|
||||
rel_map = {}
|
||||
for rel in rels_root.findall(f"{{{_REL_NS}}}Relationship"):
|
||||
rel_map[rel.get("Id")] = rel.get("Target")
|
||||
|
||||
ordered_rel_ids = []
|
||||
for elem in doc_root.iter():
|
||||
tag = elem.tag.rsplit("}", 1)[-1]
|
||||
if tag == "blip":
|
||||
rid = elem.get(f"{{{_R_NS}}}embed")
|
||||
elif tag == "imagedata":
|
||||
rid = elem.get(f"{{{_R_NS}}}id")
|
||||
else:
|
||||
rid = None
|
||||
if rid:
|
||||
ordered_rel_ids.append(rid)
|
||||
ordered_media = []
|
||||
for rid in ordered_rel_ids:
|
||||
target = rel_map.get(rid)
|
||||
if not target or "media/" not in target:
|
||||
continue
|
||||
import posixpath
|
||||
media_path = (
|
||||
target.lstrip("/")
|
||||
if target.startswith("/")
|
||||
else posixpath.normpath(
|
||||
target if target.startswith("word/") else posixpath.join("word", target)
|
||||
)
|
||||
)
|
||||
ordered_media.append((rid, media_path))
|
||||
return ordered_media
|
||||
|
||||
|
||||
def _extract_images(docx_path: Path, img_dir: Path):
|
||||
"""Extract embedded images from docx_path into img_dir as img001.ext,
|
||||
img002.ext, ... in document order. Returns the list of written filenames
|
||||
(relative to img_dir), in that same order."""
|
||||
ordered_media = _document_order_media(docx_path)
|
||||
if not ordered_media:
|
||||
return []
|
||||
|
||||
written = []
|
||||
with zipfile.ZipFile(docx_path) as z:
|
||||
names_in_zip = set(z.namelist())
|
||||
for idx, (rid, media_path) in enumerate(ordered_media, start=1):
|
||||
if media_path not in names_in_zip:
|
||||
print(f"WARNING: {media_path} (rel {rid}) not found in {docx_path}", file=sys.stderr)
|
||||
continue
|
||||
ext = Path(media_path).suffix.lstrip(".").lower() or "bin"
|
||||
if ext == "jpg":
|
||||
ext = "jpeg"
|
||||
out_name = f"img{idx:03d}.{ext}"
|
||||
img_dir.mkdir(parents=True, exist_ok=True)
|
||||
(img_dir / out_name).write_bytes(z.read(media_path))
|
||||
written.append(out_name)
|
||||
return written
|
||||
|
||||
|
||||
def _rewrite_image_refs(markdown_text: str, image_files) -> str:
|
||||
"""Replace MarkItDown's truncated base64 image placeholders with real
|
||||
relative img/imgNNN.ext references, in left-to-right order. If the
|
||||
counts don't match (unexpected), the placeholders are left as-is rather
|
||||
than risk mismatched references."""
|
||||
matches = list(_PLACEHOLDER_IMAGE_RE.finditer(markdown_text))
|
||||
if not matches:
|
||||
return markdown_text
|
||||
if len(matches) != len(image_files):
|
||||
print(
|
||||
f"WARNING: found {len(matches)} image placeholder(s) in markdown but "
|
||||
f"extracted {len(image_files)} image file(s); leaving placeholders "
|
||||
"unreplaced to avoid mismatched references.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return markdown_text
|
||||
|
||||
counter = {"i": 0}
|
||||
|
||||
def _replace(m):
|
||||
name = image_files[counter["i"]]
|
||||
counter["i"] += 1
|
||||
return f""
|
||||
|
||||
return _PLACEHOLDER_IMAGE_RE.sub(_replace, markdown_text)
|
||||
|
||||
|
||||
def convert_one(md, source: Path, dest_dir: Path) -> bool:
|
||||
"""Convert a single .docx file to a "<name>/" folder containing the
|
||||
Markdown file and an "img/" folder of extracted images. Returns True on
|
||||
success."""
|
||||
try:
|
||||
result = md.convert(str(source))
|
||||
except ImportError as exc:
|
||||
print(
|
||||
f"ERROR: A required dependency for converting '{source.name}' is not installed.\n"
|
||||
f" {exc}\n"
|
||||
"See references/setup.md for this skill, or run:\n"
|
||||
' pip install "markitdown[docx]"',
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(EXIT_MISSING_DEPENDENCY)
|
||||
except Exception as exc: # noqa: BLE001 - surface any conversion error
|
||||
print(f"FAILED {source} -> {exc}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
try:
|
||||
if dest_dir.exists():
|
||||
shutil.rmtree(dest_dir)
|
||||
dest_dir.mkdir(parents=True, exist_ok=True)
|
||||
image_files = _extract_images(source, dest_dir / "img")
|
||||
text = _rewrite_image_refs(result.text_content, image_files)
|
||||
md_path = dest_dir / f"{source.stem}.md"
|
||||
md_path.write_text(text, encoding="utf-8")
|
||||
except OSError as exc:
|
||||
print(f"FAILED {source} -> could not write output in {dest_dir}: {exc}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
img_note = f", {len(image_files)} image(s)" if image_files else ""
|
||||
print(f"OK {source} -> {md_path}{img_note}")
|
||||
return True
|
||||
|
||||
|
||||
def find_docx_files(root: Path, recursive: bool):
|
||||
"""Return (docx_files, skipped_count) for files directly/recursively under root."""
|
||||
pattern_iter = root.rglob("*") if recursive else root.iterdir()
|
||||
docx_files = []
|
||||
skipped = 0
|
||||
for entry in pattern_iter:
|
||||
if entry.is_dir():
|
||||
continue
|
||||
if entry.suffix.lower() == ".docx":
|
||||
docx_files.append(entry)
|
||||
else:
|
||||
skipped += 1
|
||||
return sorted(docx_files), skipped
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
parser.add_argument("input", help="Path to a .docx file or a directory of .docx files")
|
||||
parser.add_argument(
|
||||
"-o", "--output",
|
||||
help=(
|
||||
"Destination folder for the '<name>/' output (single-file mode), "
|
||||
"or parent directory under which each '<name>/' output folder is "
|
||||
"created (batch mode)"
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--recursive", action="store_true",
|
||||
help="When input is a directory, also search subdirectories",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
#MarkItDown = _import_markitdown()
|
||||
#md = MarkItDown()
|
||||
|
||||
source = Path(args.input)
|
||||
if not source.exists():
|
||||
print(f"ERROR: Input path not found: {source}", file=sys.stderr)
|
||||
return EXIT_INVALID_INPUT
|
||||
|
||||
if source.is_file() and source.suffix.lower() != ".docx":
|
||||
print(
|
||||
f"ERROR: Unsupported file type '{source.suffix}'. "
|
||||
"This skill only converts .docx files.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return EXIT_INVALID_INPUT
|
||||
|
||||
MarkItDown = _import_markitdown()
|
||||
md = MarkItDown()
|
||||
|
||||
if source.is_file():
|
||||
dest_dir = Path(args.output) if args.output else source.parent / source.stem
|
||||
return EXIT_OK if convert_one(md, source, dest_dir) else EXIT_CONVERSION_FAILED
|
||||
|
||||
# Directory / batch mode
|
||||
docx_files, skipped = find_docx_files(source, args.recursive)
|
||||
if skipped:
|
||||
print(f"NOTE: skipped {skipped} non-.docx file(s) in {source}")
|
||||
if not docx_files:
|
||||
print(f"ERROR: No .docx files found under {source}", file=sys.stderr)
|
||||
return EXIT_INVALID_INPUT
|
||||
|
||||
out_dir = Path(args.output) if args.output else None
|
||||
success_count = 0
|
||||
for docx_path in docx_files:
|
||||
if out_dir is not None:
|
||||
rel = docx_path.relative_to(source)
|
||||
dest_dir = out_dir / rel.parent / docx_path.stem
|
||||
else:
|
||||
dest_dir = docx_path.parent / docx_path.stem
|
||||
if convert_one(md, docx_path, dest_dir):
|
||||
success_count += 1
|
||||
|
||||
total = len(docx_files)
|
||||
print(f"\nConverted {success_count}/{total} file(s).")
|
||||
return EXIT_OK if success_count == total else EXIT_CONVERSION_FAILED
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1 @@
|
||||
markitdown[docx]>=0.1.0
|
||||
@@ -36,17 +36,22 @@ Scope to the target project only. List data access methods that interact with th
|
||||
- Follow seed file location and naming conventions from the existing project.
|
||||
- Reuse existing seed files when possible.
|
||||
- Avoid `TRUNCATE TABLE` — keep existing database data intact.
|
||||
- Assume existing business rows and lookup rows are already present; add only minimal, collision-safe seed records needed for the scenario.
|
||||
- Do not commit seed data; tests run in transactions that roll back.
|
||||
- Ensure seed data does not conflict with other tests.
|
||||
- Load and verify seed data before assertions depend on it.
|
||||
- Create or reuse a test `LookupConstants` class for stable lookup IDs/codes used across seed builders and assertions.
|
||||
|
||||
**Step 4: Write test cases**
|
||||
|
||||
- Inherit from the base test class to get automatic transaction create/rollback.
|
||||
- Ensure each database-touching method in scope has at least one integration test (or multiple tests for higher-risk behavior branches).
|
||||
- Assert logical outputs (rows, columns, counts, error types), not platform-specific messages.
|
||||
- Assert specific expected values — never assert that a value is merely non-null or non-empty when a concrete value is available from seed data.
|
||||
- Avoid testing code paths that do not exist or asserting behavior that cannot occur.
|
||||
- Avoid redundant assertions across tests targeting the same method.
|
||||
- For text parameters, include both empty-string and `NULL`/missing input coverage where applicable.
|
||||
- For datetime behavior, include explicit timezone-sensitive assertions when methods write/read `timestamp without time zone` or `timestamp(0)` targets.
|
||||
|
||||
**Step 5: Review determinism**
|
||||
|
||||
@@ -58,3 +63,4 @@ Re-examine every assertion against non-null values. Confirm each is deterministi
|
||||
- **DB-agnostic assertions** — no platform-specific error messages or syntax in assertions.
|
||||
- **Seed only against Oracle** — test project will be migrated to PostgreSQL later.
|
||||
- **Scoped to one project** — do not create tests for artifacts outside the target project.
|
||||
- **Preserve existing data** — never rewrite or wipe pre-existing business or lookup rows.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: migrating-oracle-to-postgres-stored-procedures
|
||||
description: 'Migrates Oracle PL/SQL stored procedures to PostgreSQL PL/pgSQL. Translates Oracle-specific syntax, preserves method signatures and type-anchored parameters, leverages orafce where appropriate, and applies COLLATE "C" for Oracle-compatible text sorting. Use when converting Oracle stored procedures or functions to PostgreSQL equivalents during a database migration.'
|
||||
description: 'Migrates Oracle PL/SQL stored procedures to PostgreSQL PL/pgSQL. Translates Oracle-specific syntax, preserves method signatures and type-anchored parameters, leverages orafce where appropriate, and applies explicit collation mapping (`COLLATE "C"` only when appropriate, locale collations when required). Use when converting Oracle stored procedures or functions to PostgreSQL equivalents during a database migration.'
|
||||
---
|
||||
|
||||
# Migrating Stored Procedures from Oracle to PostgreSQL
|
||||
@@ -32,7 +32,11 @@ Apply these translation rules:
|
||||
- Do not prefix object names with schema names unless already present in the Oracle source.
|
||||
- Leave exception handling and rollback logic unchanged.
|
||||
- Do not generate `COMMENT` or `GRANT` statements.
|
||||
- Use `COLLATE "C"` when ordering by text fields for Oracle-compatible sorting.
|
||||
- Apply collation intentionally when ordering text:
|
||||
- Use `COLLATE "C"` only when Oracle-compatible binary ordering is required and no other sort order is specified.
|
||||
- If Oracle used explicit linguistic sorting (for example `NLS_SORT = French`), map to an explicit PostgreSQL locale collation instead of `"C"`.
|
||||
- Use `SELECT collname, collprovider, collcollate, collctype FROM pg_collation ORDER BY collname;` to discover collations in the target environment.
|
||||
- Treat `UNION ALL` as a review checkpoint. Validate plan quality per branch and restructure if combined-branch planning causes regressions (for example, unexpected sequential scans on large tables).
|
||||
- Leverage the `orafce` extension when it improves clarity or fidelity.
|
||||
|
||||
Consult the PostgreSQL table/view definitions at `.github/oracle-to-postgres-migration/DDL/Postgres/Tables and Views/` for target schema details.
|
||||
|
||||
@@ -31,6 +31,12 @@ Write a markdown plan covering:
|
||||
- Recommended test cases per artifact
|
||||
- Seed data requirements
|
||||
- Known Oracle→PostgreSQL behavioral differences to validate
|
||||
- Coverage mapping that ensures every database touchpoint has at least one test case (or a justified set of cases for high-risk methods)
|
||||
|
||||
When defining recommended test cases, explicitly include:
|
||||
- Text parameter behavior for both empty string and `NULL`/missing values.
|
||||
- Datetime/timezone assertions, including round-trip and comparison behavior.
|
||||
- Cases where destination columns use `timestamp without time zone` or `timestamp(0)`, with explicit timezone-application expectations.
|
||||
|
||||
## Output
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: reviewing-oracle-to-postgres-migration
|
||||
description: 'Identifies Oracle-to-PostgreSQL migration risks by cross-referencing code against known behavioral differences (empty strings, refcursors, type coercion, sorting, timestamps, concurrent transactions, etc.). Use when planning a database migration, reviewing migration artifacts, or validating that integration tests cover Oracle/PostgreSQL differences.'
|
||||
description: 'Identifies Oracle-to-PostgreSQL migration risks by cross-referencing code against known behavioral differences (empty strings, refcursors, type coercion, sorting/collations, UNION ALL planner risks, materialized-view refresh requirements, timestamps, concurrent transactions, etc.). Use when planning a database migration, reviewing migration artifacts, or validating that integration tests cover Oracle/PostgreSQL differences.'
|
||||
---
|
||||
|
||||
# Oracle-to-PostgreSQL Database Migration
|
||||
@@ -60,7 +60,7 @@ For each reference in [references/REFERENCE.md](references/REFERENCE.md), confir
|
||||
|
||||
**Step 3: Verify integration test coverage**
|
||||
|
||||
Confirm tests exercise both the happy path and the failure scenarios highlighted in applicable insights (exceptions, sorting, refcursor consumption, concurrent transactions, timestamps, etc.).
|
||||
Confirm tests exercise both the happy path and the failure scenarios highlighted in applicable insights (exceptions, sorting, `UNION ALL` behavior/performance risks, refcursor consumption, concurrent transactions, timestamps, materialized-view freshness, etc.).
|
||||
|
||||
**Step 4: Gate the result**
|
||||
|
||||
|
||||
@@ -8,6 +8,8 @@
|
||||
| [oracle-to-postgres-sorting.md](oracle-to-postgres-sorting.md) | How to preserve Oracle-like ordering in PostgreSQL using COLLATE "C" and DISTINCT wrapper patterns. |
|
||||
| [oracle-to-postgres-to-char-numeric.md](oracle-to-postgres-to-char-numeric.md) | Oracle allows TO_CHAR(numeric) without format; PostgreSQL requires format string—use CAST(numeric AS TEXT) instead. |
|
||||
| [oracle-to-postgres-type-coercion.md](oracle-to-postgres-type-coercion.md) | PostgreSQL strict type checks vs. Oracle implicit coercion—fix comparison errors by quoting or casting literals. |
|
||||
| [postgres-union-all-planner.md](postgres-union-all-planner.md) | UNION ALL branches can produce poor plans when predicate pushdown is limited—review plans and split or reshape queries when needed. |
|
||||
| [postgres-materialized-view-refresh.md](postgres-materialized-view-refresh.md) | Materialized views are not auto-refreshed after base-table changes—application or jobs must explicitly refresh them. |
|
||||
| [postgres-concurrent-transactions.md](postgres-concurrent-transactions.md) | PostgreSQL allows only one active command per connection—materialize results or use separate connections to avoid concurrent operation errors. |
|
||||
| [postgres-refcursor-handling.md](postgres-refcursor-handling.md) | Differences in refcursor handling; PostgreSQL requires fetching by cursor name—C# patterns to unwrap and read results. |
|
||||
| [oracle-to-postgres-timestamp-timezone.md](oracle-to-postgres-timestamp-timezone.md) | CURRENT_TIMESTAMP / NOW() return UTC-normalised timestamptz in PostgreSQL; Npgsql surfaces DateTime.Kind=Unspecified—force UTC at connection open and in application code. |
|
||||
|
||||
+25
-6
@@ -3,13 +3,14 @@
|
||||
Purpose: Preserve Oracle-like sorting semantics when moving queries to PostgreSQL.
|
||||
|
||||
## Key points
|
||||
- Oracle often treats plain `ORDER BY` as binary/byte-wise, giving case-insensitive ordering for ASCII.
|
||||
- PostgreSQL defaults differ; to match Oracle behavior, use `COLLATE "C"` on sort expressions.
|
||||
- Oracle and PostgreSQL default collations can differ significantly.
|
||||
- Use `COLLATE "C"` only when you explicitly need Oracle-like binary ordering and no different sort rule is requested.
|
||||
- If Oracle uses explicit linguistic ordering (for example `NLS_SORT = French`), map to an explicit PostgreSQL locale collation instead of forcing `"C"`.
|
||||
|
||||
## 1) Standard `SELECT … ORDER BY`
|
||||
**Goal:** Keep Oracle-style ordering.
|
||||
|
||||
**Pattern:**
|
||||
**Pattern (only when Oracle-compatible binary ordering is required):**
|
||||
```sql
|
||||
SELECT col1
|
||||
FROM your_table
|
||||
@@ -17,9 +18,27 @@ ORDER BY col1 COLLATE "C";
|
||||
```
|
||||
|
||||
**Notes:**
|
||||
- Apply `COLLATE "C"` to each sort expression that must mimic Oracle.
|
||||
- Apply `COLLATE "C"` only to sort expressions that must mimic Oracle binary ordering.
|
||||
- Works with ascending/descending and multi-column sorts, e.g. `ORDER BY col1 COLLATE "C", col2 COLLATE "C" DESC`.
|
||||
|
||||
## 1b) Locale-aware ordering (when Oracle used NLS_SORT)
|
||||
|
||||
If Oracle used locale-specific sorting such as:
|
||||
```sql
|
||||
ORDER BY nlssort(Externalusers.UserID, 'NLS_SORT = French')
|
||||
```
|
||||
map to an explicit PostgreSQL collation, for example:
|
||||
```sql
|
||||
ORDER BY Externalusers.UserID COLLATE "ca_FR.utf-8"
|
||||
```
|
||||
|
||||
Use a collation that exists in the target environment. Discover available collations with:
|
||||
```sql
|
||||
SELECT collname, collprovider, collcollate, collctype
|
||||
FROM pg_collation
|
||||
ORDER BY collname;
|
||||
```
|
||||
|
||||
## 2) `SELECT DISTINCT … ORDER BY`
|
||||
**Issue:** PostgreSQL enforces that `ORDER BY` expressions appear in the `SELECT` list for `DISTINCT`, raising:
|
||||
`Npgsql.PostgresException: 42P10: for SELECT DISTINCT, ORDER BY expressions must appear in select list`
|
||||
@@ -38,14 +57,14 @@ ORDER BY col2 COLLATE "C";
|
||||
|
||||
**Why:**
|
||||
- The inner query performs the `DISTINCT` projection.
|
||||
- The outer query safely orders the result set and adds `COLLATE "C"` to align with Oracle sorting.
|
||||
- The outer query safely orders the result set and adds an explicit collation where needed to align with Oracle sorting.
|
||||
|
||||
**Tips:**
|
||||
- Ensure any columns used in the outer `ORDER BY` are included in the inner projection.
|
||||
- For multi-column sorts, collate each relevant expression: `ORDER BY col2 COLLATE "C", col3 COLLATE "C" DESC`.
|
||||
|
||||
## Validation checklist
|
||||
- [ ] Added `COLLATE "C"` to every `ORDER BY` that should follow Oracle sorting rules.
|
||||
- [ ] Applied explicit collation only where required (`"C"` for Oracle-style binary ordering, locale collation for linguistic ordering).
|
||||
- [ ] For `DISTINCT` queries, wrapped the projection and sorted in the outer query.
|
||||
- [ ] Confirmed ordered columns are present in the inner projection.
|
||||
- [ ] Re-ran tests or representative queries to verify ordering matches Oracle outputs.
|
||||
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
# PostgreSQL Materialized View Refresh Guide
|
||||
|
||||
Purpose: Ensure migrated applications keep materialized views current after base-table changes.
|
||||
|
||||
## Problem
|
||||
|
||||
PostgreSQL materialized views are static snapshots. Updates to source tables do **not** automatically refresh dependent materialized views.
|
||||
|
||||
## Migration risk
|
||||
|
||||
- Oracle-era assumptions that derived data updates immediately may no longer hold.
|
||||
- Read paths can return stale rows unless refresh timing is explicitly managed.
|
||||
- Integration tests may pass once and then fail intermittently if refresh sequencing is not deterministic.
|
||||
|
||||
## Required review item
|
||||
|
||||
For every migrated path that writes to tables feeding a materialized view, verify the application workflow includes an explicit refresh strategy.
|
||||
|
||||
## Refresh patterns
|
||||
|
||||
- Immediate refresh in the write workflow when freshness is required:
|
||||
```sql
|
||||
REFRESH MATERIALIZED VIEW my_view;
|
||||
```
|
||||
- Concurrent refresh (when supported and indexed) to reduce read blocking:
|
||||
```sql
|
||||
REFRESH MATERIALIZED VIEW CONCURRENTLY my_view;
|
||||
```
|
||||
- Scheduled/batch refresh when stale windows are acceptable.
|
||||
|
||||
## Integration-test expectations
|
||||
|
||||
- [ ] Tests that modify source tables assert materialized-view contents only after the intended refresh action.
|
||||
- [ ] Tests assert stale behavior before refresh when applicable.
|
||||
- [ ] Tests document whether freshness is immediate or eventual for each affected feature.
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
# PostgreSQL UNION ALL Planner Risk Guide
|
||||
|
||||
Purpose: Avoid regressions where migrated `UNION ALL` queries run much slower in PostgreSQL than expected.
|
||||
|
||||
## Problem
|
||||
|
||||
`UNION ALL` keeps duplicate rows and combines branch outputs directly, but PostgreSQL does not always optimize each branch as aggressively as isolated queries. In large datasets this can produce poor plans (for example full scans where index-based plans are expected).
|
||||
|
||||
## Why it happens
|
||||
|
||||
- Predicate pushdown through `UNION ALL` branches can be limited depending on query shape.
|
||||
- Cardinality estimates across branches can be skewed.
|
||||
- Branch-local indexes may not be chosen when the optimizer evaluates the combined query.
|
||||
|
||||
## Review checklist
|
||||
|
||||
- [ ] Compare `EXPLAIN (ANALYZE, BUFFERS)` plans for the combined `UNION ALL` query and branch-isolated variants.
|
||||
- [ ] Confirm branch predicates are explicit and not hidden inside non-sargable expressions.
|
||||
- [ ] Check for unexpected sequential scans on large tables in either branch.
|
||||
- [ ] Verify indexes exist for each branch's filter and join predicates.
|
||||
|
||||
## Mitigation patterns
|
||||
|
||||
1. Test each branch independently to verify expected index usage.
|
||||
2. Push filters down into each branch instead of only filtering in the outer query.
|
||||
3. If plan quality remains poor, split the query into two separately executed statements and combine results in application code.
|
||||
4. Consider materializing branch results in temporary/intermediate structures only when measurement confirms benefit.
|
||||
|
||||
## Validation note
|
||||
|
||||
Treat `UNION ALL` performance behavior as a migration review item even when functional test results match Oracle.
|
||||
@@ -0,0 +1,105 @@
|
||||
---
|
||||
name: steno-mode
|
||||
description: 'Shorthand-first response compression that cuts ~40% of response tokens while preserving technical precision and exact literals. Use when the user says "steno mode", "shorthand mode", "compressed responses", "token reduction", "brief structured output", or invokes /steno. Supports four compression levels: lite, brief, court, machine. Do not trigger for requests needing polished prose such as onboarding/tutorial content, stakeholder or customer-facing copy, or teaching-focused explanations.'
|
||||
license: MIT
|
||||
---
|
||||
|
||||
# Steno Mode
|
||||
|
||||
Respond like an expert using disciplined shorthand. Dense, exact, readable. Do not imitate literal court-reporting notation.
|
||||
|
||||
## Persistence
|
||||
|
||||
ACTIVE EVERY RESPONSE after enabled. Stay active across turns and across agent switches, including Ask, Edit, Agent, and custom agents. Turn off only when the user says "stop steno" or "normal mode".
|
||||
|
||||
Default level: **brief**. Switch with `/steno lite|brief|court|machine`.
|
||||
|
||||
## Contract
|
||||
|
||||
Goal: reduce tokens by compressing prose, not by sacrificing precision.
|
||||
|
||||
Priority order:
|
||||
|
||||
1. Exactness
|
||||
2. Readability
|
||||
3. Compression
|
||||
|
||||
If compression harms exactness, keep the full form.
|
||||
|
||||
## Core Rules
|
||||
|
||||
Cut:
|
||||
|
||||
- filler and pleasantries
|
||||
- low-value glue words when meaning stays clear
|
||||
- repeated framing before the answer
|
||||
|
||||
Keep exact (never compress):
|
||||
|
||||
- code blocks
|
||||
- commands
|
||||
- paths and filenames
|
||||
- API names and identifiers
|
||||
- env vars
|
||||
- quoted error text
|
||||
- versions, flags, and numbers
|
||||
|
||||
Compress with:
|
||||
|
||||
- stable abbreviations (examples): `cfg`, `auth`, `deps`, `env`, `req`, `resp`, `impl`, `perf`, `arch`, `ctx`, `conn`, `ctr`
|
||||
- symbolic joins: `->`, `=>`, `vs`, `w/`, `w/o`, `+`, `=`
|
||||
- list-first structure when content is naturally list-shaped
|
||||
- short causal chains: `X -> Y -> Z`
|
||||
|
||||
Avoid:
|
||||
|
||||
- random abbreviations
|
||||
- slang or text-message spelling
|
||||
- phonetic stenography glyphs
|
||||
- collapsing two distinct technical terms into one shorthand
|
||||
|
||||
Pattern: `[problem/point] -> [cause/decision] -> [action/result]`
|
||||
|
||||
## Levels
|
||||
|
||||
| Level | Behavior |
|
||||
|-------|----------|
|
||||
| **lite** | Tight professional prose. Full sentences mostly intact. Minimal filler. |
|
||||
| **brief** | Default. Shorthand + symbols + compact phrasing. High readability. |
|
||||
| **court** | Dense expert shorthand. Fragments allowed. Strong symbol use. |
|
||||
| **machine** | Max compression for expert users. Heavy abbreviation, minimal connectors. Use only when clarity still holds. |
|
||||
|
||||
## Examples
|
||||
|
||||
Example — "Why does this API retry loop never stop?"
|
||||
|
||||
- lite: "Retry state resets on each req, so the loop never reaches the terminal condition. Persist the ctr outside the req scope."
|
||||
- brief: "Retry state resets per req -> terminal condition never reached. Move ctr outside req scope."
|
||||
- court: "State resets per req -> no terminal hit -> loop. Persist ctr outside req scope."
|
||||
- machine: "Per-req reset -> no terminal -> loop. Persist ctr outside scope."
|
||||
|
||||
Example — "Review this bug fix."
|
||||
|
||||
- lite: "The fix handles null input, but it still mutates shared state. Clone before modifying."
|
||||
- brief: "Null case fixed. Shared state still mutated. Clone before write."
|
||||
- court: "Null fixed. Shared state mutates. Clone pre-write."
|
||||
- machine: "Null OK. Shared mutates. Clone pre-write."
|
||||
|
||||
Example — "Explain connection pooling."
|
||||
|
||||
- lite: "Connection pooling reuses open connections instead of creating a new one for every req. That cuts handshake overhead."
|
||||
- brief: "Pool reuses open conns vs new conn per req. Cuts handshake overhead."
|
||||
- court: "Pool = reuse open conns. No per-req open/close. Less handshake cost."
|
||||
- machine: "Pool reuse conns. Skip per-req handshake."
|
||||
|
||||
## Scope
|
||||
|
||||
Works well: code review comments, bug explanations, debugging Q&A, architecture summaries, API and config documentation, progress updates.
|
||||
|
||||
Does not work well: onboarding and tutorials, stakeholder communication, empathetic responses, teaching new concepts. For these, switch to lite or ask whether compression should stay on.
|
||||
|
||||
## Safety
|
||||
|
||||
- When exact wording matters, quote verbatim.
|
||||
- When ambiguity appears, expand once, then resume shorthand.
|
||||
- When the user asks for docs, legal text, customer copy, or polished prose, either switch to lite or ask whether compression should stay on.
|
||||
@@ -0,0 +1,312 @@
|
||||
---
|
||||
name: tm7-threat-model
|
||||
description: 'Creates valid Microsoft Threat Modeling Tool (.tm7) files compatible with the Microsoft Threat Modeling Tool v7.3+. Use this skill whenever asked to create, generate, or modify a .tm7 threat model file, or when performing STRIDE threat modeling that should output a .tm7 file that opens cleanly in the Microsoft Threat Modeling Tool.'
|
||||
---
|
||||
|
||||
# Microsoft Threat Modeling Tool (.tm7) Generator
|
||||
|
||||
You generate **valid `.tm7` files** for the Microsoft Threat Modeling Tool (v7.3+). A `.tm7`
|
||||
file is **not** generic XML — it is a **WCF `DataContractSerializer`** document with an exact
|
||||
namespace and element structure. If the structure is wrong, the tool refuses to open the file
|
||||
with:
|
||||
|
||||
> "File is not an actual threat model or the threat model may be corrupted."
|
||||
|
||||
Your job is to translate a described system (components, data stores, external actors, data
|
||||
flows, trust boundaries) into a diagram plus STRIDE threats, serialized in the exact `.tm7`
|
||||
format described below.
|
||||
|
||||
## Workflow
|
||||
|
||||
When asked to produce a `.tm7` file:
|
||||
|
||||
1. **Model the system.** Identify the elements:
|
||||
- **Processes** (web apps, services, functions) → `StencilEllipse`, `GE.P`
|
||||
- **Data stores** (databases, caches, queues, blobs) → `StencilParallelLines`, `GE.DS`
|
||||
- **External interactors** (users, browsers, third-party systems) → `StencilRectangle`, `GE.EI`
|
||||
- **Trust boundaries** → `BorderBoundary`, `GE.TB`
|
||||
- **Data flows** connecting the above → `Connector`, `GE.DF`
|
||||
2. **Assign a unique lowercase UUID** (e.g. `148ade68-5c80-40f3-8e1f-4e2cabdb5991`) to every
|
||||
stencil and every flow. Never use human-readable ids like `users-browser`.
|
||||
3. **Lay out coordinates** (`Left`/`Top`/`Width`/`Height`) so stencils don't overlap.
|
||||
4. **Generate STRIDE threats** per interaction and place them in `<ThreatInstances>`.
|
||||
5. **Serialize** using the structure in this guide, mirroring `assets/example-minimal.tm7`.
|
||||
6. **Validate** against the "Common Mistakes" checklist before returning the file.
|
||||
7. **Write the file with no XML declaration and no pretty-print indentation** (a single
|
||||
continuous XML stream is what the serializer emits).
|
||||
|
||||
Always open [`assets/example-minimal.tm7`](./assets/example-minimal.tm7) first and adapt it — reuse its exact
|
||||
serialization skeleton and only change stencil types, names, coordinates, flows, and threats.
|
||||
|
||||
## CRITICAL: Serialization format
|
||||
|
||||
TM7 files use **WCF `DataContractSerializer` XML**, not standard XML.
|
||||
|
||||
The file MUST start with this exact root element — **no `<?xml?>` declaration**:
|
||||
|
||||
```xml
|
||||
<ThreatModel xmlns="http://schemas.datacontract.org/2004/07/ThreatModeling.Model" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
|
||||
```
|
||||
|
||||
**NEVER use:**
|
||||
- `<?xml version="1.0" encoding="utf-8"?>` — causes deserialization failure.
|
||||
- `xmlns:xsi` / `xmlns:xsd` — these are standard XML namespaces, not DataContract namespaces.
|
||||
- Invented elements such as `<SecurityGaps>` or `<Mitigations>` — they do not exist in the
|
||||
TM7 schema.
|
||||
|
||||
> **Note:** `<MetaInformation>` (with children like `<Owner>`, `<Contributors>`,
|
||||
> `<Reviewer>`, `<Assumptions>`, `<ExternalDependencies>`, `<HighLevelSystemDescription>`,
|
||||
> `<ThreatModelName>`), `<Notes>`, and `<KnowledgeBase>` **are** part of the real schema and
|
||||
> are emitted by the tool — keep them (see the structure below and `assets/example-minimal.tm7`).
|
||||
> Just don't invent elements that the tool never produces.
|
||||
|
||||
## Required namespace prefixes
|
||||
|
||||
| Prefix | URI | Used for |
|
||||
|--------|-----|----------|
|
||||
| (default) | `http://schemas.datacontract.org/2004/07/ThreatModeling.Model` | Root `ThreatModel` |
|
||||
| `xmlns:i` | `http://www.w3.org/2001/XMLSchema-instance` | Type attributes |
|
||||
| `xmlns:z` | `http://schemas.microsoft.com/2003/10/Serialization/` | Reference ids (`z:Id`) |
|
||||
| `xmlns:a` | `http://schemas.microsoft.com/2003/10/Serialization/Arrays` | Arrays / collections |
|
||||
| `xmlns:b` | `http://schemas.datacontract.org/2004/07/ThreatModeling.KnowledgeBase` | Stencil properties |
|
||||
| `xmlns:c` | `http://www.w3.org/2001/XMLSchema` | Primitive type values |
|
||||
|
||||
## File structure (correct order)
|
||||
|
||||
A full tool export contains, in this order: `DrawingSurfaceList`, `MetaInformation`, `Notes`,
|
||||
`ThreatInstances`, `ThreatMetaData` (often empty/self-closing), then the large generic
|
||||
`KnowledgeBase` as a **top-level sibling** (not nested inside `ThreatMetaData`), and finally
|
||||
`Profile`.
|
||||
|
||||
```xml
|
||||
<ThreatModel xmlns="..." xmlns:i="...">
|
||||
<DrawingSurfaceList>
|
||||
<DrawingSurfaceModel z:Id="i1" xmlns:z="...">
|
||||
<GenericTypeId xmlns="...Abstracts">DRAWINGSURFACE</GenericTypeId>
|
||||
<Guid xmlns="...Abstracts">{guid}</Guid>
|
||||
<Properties xmlns="...Abstracts" xmlns:a="...Arrays">...</Properties>
|
||||
<TypeId xmlns="...Abstracts">DRAWINGSURFACE</TypeId>
|
||||
<Borders xmlns:a="...Arrays">
|
||||
<!-- Stencil elements: processes, data stores, external entities, boundaries -->
|
||||
</Borders>
|
||||
<Lines xmlns:a="...Arrays">
|
||||
<!-- Data flow lines connecting stencils -->
|
||||
</Lines>
|
||||
<Notes xmlns:a="...Arrays"/>
|
||||
</DrawingSurfaceModel>
|
||||
</DrawingSurfaceList>
|
||||
<MetaInformation>
|
||||
<!-- Owner, Contributors, Reviewer, Assumptions, ThreatModelName, etc. -->
|
||||
</MetaInformation>
|
||||
<Notes xmlns:a="...Arrays"/>
|
||||
<ThreatInstances>
|
||||
<!-- Threat entries -->
|
||||
</ThreatInstances>
|
||||
<ThreatMetaData/>
|
||||
<KnowledgeBase z:Id="i21" xmlns:a="...ThreatModeling.KnowledgeBase" xmlns:z="...">
|
||||
<!-- Generic SDL stencil/threat catalog — top-level sibling of ThreatMetaData -->
|
||||
</KnowledgeBase>
|
||||
<Profile>
|
||||
<PromptedKb xmlns=""/>
|
||||
</Profile>
|
||||
</ThreatModel>
|
||||
```
|
||||
|
||||
> The `<KnowledgeBase>` (the generic SDL stencil/threat catalog) is large but **required** —
|
||||
> the tool uses it to resolve every stencil `TypeId`. It is a **top-level sibling** placed after
|
||||
> `ThreatMetaData` and before `Profile`, **not** nested inside `ThreatMetaData`. Reuse it verbatim
|
||||
> from `assets/example-minimal.tm7`; only add stencils whose `TypeId` already appears in that
|
||||
> KnowledgeBase.
|
||||
|
||||
## Stencil elements
|
||||
|
||||
Each stencil in `<Borders>` is wrapped in `<a:KeyValueOfguidanyType>`:
|
||||
|
||||
```xml
|
||||
<a:KeyValueOfguidanyType>
|
||||
<a:Key>{guid}</a:Key>
|
||||
<a:Value z:Id="i2" i:type="StencilEllipse">
|
||||
<GenericTypeId xmlns="...Abstracts">GE.P</GenericTypeId>
|
||||
<Guid xmlns="...Abstracts">{guid}</Guid>
|
||||
<Properties xmlns="...Abstracts">
|
||||
<a:anyType i:type="b:HeaderDisplayAttribute" xmlns:b="...KnowledgeBase">
|
||||
<b:DisplayName>Web Application</b:DisplayName>
|
||||
<b:Name/>
|
||||
<b:Value i:nil="true"/>
|
||||
</a:anyType>
|
||||
<a:anyType i:type="b:StringDisplayAttribute" xmlns:b="...KnowledgeBase">
|
||||
<b:DisplayName>Name</b:DisplayName>
|
||||
<b:Name/>
|
||||
<b:Value i:type="c:string" xmlns:c="http://www.w3.org/2001/XMLSchema">My Component</b:Value>
|
||||
</a:anyType>
|
||||
<!-- Out Of Scope, Reason, configurable attributes -->
|
||||
</Properties>
|
||||
<TypeId xmlns="...Abstracts">SE.P.TMCore.WebApp</TypeId>
|
||||
<Height xmlns="...Abstracts">100</Height>
|
||||
<Left xmlns="...Abstracts">400</Left>
|
||||
<StrokeDashArray i:nil="true" xmlns="...Abstracts"/>
|
||||
<StrokeThickness xmlns="...Abstracts">1</StrokeThickness>
|
||||
<Top xmlns="...Abstracts">200</Top>
|
||||
<Width xmlns="...Abstracts">100</Width>
|
||||
</a:Value>
|
||||
</a:KeyValueOfguidanyType>
|
||||
```
|
||||
|
||||
### Stencil shape types
|
||||
|
||||
| Shape | `i:type` | `GenericTypeId` | Description |
|
||||
|-------|----------|-----------------|-------------|
|
||||
| Process (circle) | `StencilEllipse` | `GE.P` | Processes, web apps, services |
|
||||
| Data store (parallel lines) | `StencilParallelLines` | `GE.DS` | Databases, storage, caches |
|
||||
| External interactor (rectangle) | `StencilRectangle` | `GE.EI` | Users, external systems |
|
||||
| Trust boundary | `BorderBoundary` | `GE.TB` | Trust boundaries |
|
||||
|
||||
### Common `TypeId` values (SDL TM knowledge base)
|
||||
|
||||
| `TypeId` | Component |
|
||||
|----------|-----------|
|
||||
| `SE.P.TMCore.WebApp` | Web Application |
|
||||
| `SE.P.TMCore.AzureAppServiceWebApp` | Azure App Service Web App |
|
||||
| `SE.P.TMCore.AzureEventHub` | Azure Event Hub |
|
||||
| `SE.P.TMCore.DynamicsCRM` | Dynamics CRM |
|
||||
| `SE.DS.TMCore.SQL` | SQL Database |
|
||||
| `SE.DS.TMCore.AzureSQLDB` | Azure SQL Database |
|
||||
| `SE.EI.TMCore.Browser` | Browser |
|
||||
| `SE.EI.TMCore.Mobile` | Mobile Client |
|
||||
|
||||
## Data flow lines
|
||||
|
||||
Lines in `<Lines>` also use `<a:KeyValueOfguidanyType>`, with `i:type="Connector"`:
|
||||
|
||||
```xml
|
||||
<a:KeyValueOfguidanyType>
|
||||
<a:Key>{line-guid}</a:Key>
|
||||
<a:Value z:Id="i10" i:type="Connector">
|
||||
<GenericTypeId xmlns="...Abstracts">GE.DF</GenericTypeId>
|
||||
<Guid xmlns="...Abstracts">{line-guid}</Guid>
|
||||
<Properties xmlns="...Abstracts">...</Properties>
|
||||
<TypeId xmlns="...Abstracts">SE.DF.TMCore.Request</TypeId>
|
||||
<HandleX xmlns="...Abstracts">0</HandleX>
|
||||
<HandleY xmlns="...Abstracts">0</HandleY>
|
||||
<SourceGuid xmlns="...Abstracts">{source-stencil-guid}</SourceGuid>
|
||||
<SourceX xmlns="...Abstracts">0</SourceX>
|
||||
<SourceY xmlns="...Abstracts">0</SourceY>
|
||||
<TargetGuid xmlns="...Abstracts">{target-stencil-guid}</TargetGuid>
|
||||
<TargetX xmlns="...Abstracts">0</TargetX>
|
||||
<TargetY xmlns="...Abstracts">0</TargetY>
|
||||
</a:Value>
|
||||
</a:KeyValueOfguidanyType>
|
||||
```
|
||||
|
||||
## Property attribute types
|
||||
|
||||
Properties use typed `<a:anyType>` elements:
|
||||
|
||||
| `i:type` | Purpose | Value |
|
||||
|----------|---------|-------|
|
||||
| `b:HeaderDisplayAttribute` | Section header | `i:nil="true"` |
|
||||
| `b:StringDisplayAttribute` | Text value (Name, Reason) | `i:type="c:string"` |
|
||||
| `b:BooleanDisplayAttribute` | Boolean (Out Of Scope) | `i:type="c:boolean"` |
|
||||
| `b:ListDisplayAttribute` | Dropdown list | Has `<b:SelectedIndex>` |
|
||||
|
||||
## Threat instances
|
||||
|
||||
Threats go in `<ThreatInstances>` using `<a:KeyValueOfstringThreatpc_P0_PhOB>` (note the exact
|
||||
`PhOB` suffix). Unlike stencils, the threat `<a:Value>` fields are **`b:`-prefixed** (the
|
||||
`ThreatModeling.KnowledgeBase` namespace), and the `<a:Key>` is the literal concatenation
|
||||
`TH<id> + <SourceGuid> + <FlowGuid> + <TargetGuid>`:
|
||||
|
||||
```xml
|
||||
<ThreatInstances xmlns:a="...Arrays">
|
||||
<a:KeyValueOfstringThreatpc_P0_PhOB>
|
||||
<a:Key>TH117{source-guid}{flow-guid}{target-guid}</a:Key>
|
||||
<a:Value xmlns:b="...KnowledgeBase">
|
||||
<b:ChangedBy/>
|
||||
<b:DrawingSurfaceGuid>{drawing-surface-guid}</b:DrawingSurfaceGuid>
|
||||
<b:FlowGuid>{flow-guid}</b:FlowGuid>
|
||||
<b:Id>32</b:Id>
|
||||
<b:InteractionKey>{source-guid}:{flow-guid}:{target-guid}</b:InteractionKey>
|
||||
<b:InteractionString i:nil="true"/>
|
||||
<b:ModifiedAt>2025-01-01T00:00:00</b:ModifiedAt>
|
||||
<b:Priority>High</b:Priority>
|
||||
<b:Properties>
|
||||
<a:KeyValueOfstringstring>
|
||||
<a:Key>Title</a:Key>
|
||||
<a:Value>An adversary may spoof the user and gain access</a:Value>
|
||||
</a:KeyValueOfstringstring>
|
||||
<a:KeyValueOfstringstring>
|
||||
<a:Key>UserThreatCategory</a:Key>
|
||||
<a:Value>Spoofing</a:Value>
|
||||
</a:KeyValueOfstringstring>
|
||||
<a:KeyValueOfstringstring>
|
||||
<a:Key>UserThreatShortDescription</a:Key>
|
||||
<a:Value>Spoofing is when a process or entity is something other than its claimed identity.</a:Value>
|
||||
</a:KeyValueOfstringstring>
|
||||
<a:KeyValueOfstringstring>
|
||||
<a:Key>PossibleMitigations</a:Key>
|
||||
<a:Value>Enable multi-factor authentication and least-privilege access control.</a:Value>
|
||||
</a:KeyValueOfstringstring>
|
||||
<a:KeyValueOfstringstring>
|
||||
<a:Key>Priority</a:Key>
|
||||
<a:Value>High</a:Value>
|
||||
</a:KeyValueOfstringstring>
|
||||
<a:KeyValueOfstringstring>
|
||||
<a:Key>SDLPhase</a:Key>
|
||||
<a:Value>Design</a:Value>
|
||||
</a:KeyValueOfstringstring>
|
||||
</b:Properties>
|
||||
<b:SourceGuid>{source-stencil-guid}</b:SourceGuid>
|
||||
<b:State>Mitigated</b:State>
|
||||
<b:StateInformation i:nil="true"/>
|
||||
<b:TargetGuid>{target-stencil-guid}</b:TargetGuid>
|
||||
<b:Title i:nil="true"/>
|
||||
<b:TypeId>TH117</b:TypeId>
|
||||
<b:Upgraded>false</b:Upgraded>
|
||||
<b:Wide>false</b:Wide>
|
||||
</a:Value>
|
||||
</a:KeyValueOfstringThreatpc_P0_PhOB>
|
||||
</ThreatInstances>
|
||||
```
|
||||
|
||||
**Every GUID must resolve:** `SourceGuid` and `TargetGuid` must equal `<a:Key>` values of real
|
||||
stencils in `<Borders>`, and `FlowGuid` must equal the `<a:Key>` of a real connector in
|
||||
`<Lines>`. Dangling references produce a model that opens with missing diagram elements.
|
||||
|
||||
Use the standard STRIDE categories for `UserThreatCategory`: **S**poofing, **T**ampering,
|
||||
**R**epudiation, **I**nformation Disclosure, **D**enial of Service, **E**levation of Privilege.
|
||||
|
||||
## Common mistakes that break TM7 files
|
||||
|
||||
1. **Adding an `<?xml version="1.0"?>` declaration** — `DataContractSerializer` does not emit one.
|
||||
2. **Using `xmlns:xsi` / `xmlns:xsd`** instead of DataContract namespaces.
|
||||
3. **Using simple element names** like `<Border>`, `<Line>`, `<Stencil>` — you must use the
|
||||
DataContract wrapper types such as `<a:KeyValueOfguidanyType>`.
|
||||
4. **Inventing elements the tool never emits** like `<SecurityGaps>` or `<Mitigations>` — these
|
||||
are not in the schema. (`<MetaInformation>`, `<Notes>`, and `<KnowledgeBase>` **are** valid
|
||||
and must be preserved.)
|
||||
5. **Using human-readable GUIDs** like `users-browser` instead of real UUIDs
|
||||
(e.g. `148ade68-5c80-40f3-8e1f-4e2cabdb5991`).
|
||||
6. **Dangling references** — a `Line`, threat `SourceGuid`/`TargetGuid`, or threat `FlowGuid`
|
||||
that points to a stencil/flow GUID that isn't actually defined in `<Borders>`/`<Lines>`.
|
||||
Every reference must resolve to an included element.
|
||||
7. **Missing or duplicated `z:Id` reference attributes** — every serialized object needs a
|
||||
`z:Id`, and each `z:Id` (e.g. `i1`, `i2`, `i10`) must be **unique** across the whole file.
|
||||
When you duplicate a template block to add an element, always renumber its `z:Id` (and any
|
||||
nested ones) to values not used elsewhere; reusing an id creates duplicate DataContract
|
||||
object ids and makes deserialization fail.
|
||||
8. **Missing the `xmlns` on child elements** — each `GenericTypeId`, `Guid`, `Properties`,
|
||||
`TypeId`, etc. must carry its own
|
||||
`xmlns="http://schemas.datacontract.org/2004/07/ThreatModeling.Model.Abstracts"`.
|
||||
9. **Pretty-printing with indentation** — the correct output is a single continuous XML stream
|
||||
with no added newlines or indentation inside the content.
|
||||
|
||||
## Reference asset
|
||||
|
||||
Always use [`assets/example-minimal.tm7`](./assets/example-minimal.tm7) in this skill's
|
||||
directory as the structural reference. It is a fully synthetic, sanitized export (no personal or
|
||||
project data) that opens cleanly in the tool: two stencils connected by one data flow, with one
|
||||
STRIDE threat whose every reference resolves. Adapt the stencil types, names, properties,
|
||||
coordinates, data flows, and threats to the user's architecture, but **never** change the
|
||||
serialization format or namespace structure, and only use stencil `TypeId` values that already
|
||||
appear in its bundled `KnowledgeBase`. After generating, mentally diff your output's skeleton
|
||||
against the example to confirm every namespace, wrapper element, and GUID reference matches.
|
||||
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user