opthq/cpp-garmin-img-to-osm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

v0.6

Browse Source
This commit is contained in:
nq
2026-10-14 12:42:58 +00:00
parent dfaa54ed8e
commit d3ee396809
27 changed files with 13234 additions and 3839 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+46 -135
View File
@@ -1,155 +1,66 @@
# BGMountains Garmin IMG → OSM → OBF: C++ converter + POI audit toolkit (v0.5)
# BGMountains Garmin IMG → OSM v0.6 (legend-reviewed, spatially validated)
The source code, PNG icons, full-country inventory, editable 1,871-row POI configuration, source manifest, and comparison tools are included in this archive.
This is an **incremental source + QA release**, based on the v0.5 converter. Original map IMG/OBF binaries are **not bundled**. The source map's license is **not** the converter's GPL license: map-derived data, icons, CSV and examples are for private verification unless the rights holders authorize redistribution.
**Important:** The converter source is GPL-3.0-or-later. The original BGMountains map data and any extracted/converted geographic data have **separate** licenses. There is no blanket permission to merge this data into openstreetmap.org. No IMG or OBF binaries are bundled.
## Corrected
## Build
1. **67 exact extracted TYP icon ↔ supplied BGMountains legend matches**, 60 of them present as decoded POI types in the IMG (7 icons occur only in its TYP catalog), including 48 changes to the previous semantic defaults. Changes appear in `data/legend-corrections-log.csv`, and new mappings are embedded directly in `src/vector.cpp` as safe fallbacks. Every verified icon also gets a `garmin:legend` tag to aid inspection. Optional `config/poi-overrides-legend-reviewed.csv` permits further edits while retaining the original 1,871 type rows.
2. **Extended-point label pointer**: v0.5 read the label pointer at `p+len-3`; Garmin's layout places it at `p+6`, before the optional extra byte. Correcting this fixed or cleared **117 label outcomes**. See `data/extended-label-decoder-changes.csv`, with old/new names/coordinates. The string `есебър` was a decoder artifact at the *wrong geographic location*, not a misspelling to force to `Несебър`. The actual town of Несебър is in the normal classic point data at 42.6599550,27.7197504, mapset 02235056.
3. **Spatial sanity gate**: Extended POIs outside their own TRE subdivision bounds (beyond 0.00005° margin) are no longer emitted. **5,602 suspected/invalid point records** disappear; classic POI count remains **407,820**. This is a conservative defense against malformed extended streams, not independent Garmin specification conformance testing. Retained extended records may still warrant validation. Total kept POIs: **410,563**, distinct types: **263** (113 classic, 150 extended). The old 1,871-code inventory remains available, annotated with exclusion counts.
4. **Optional per-feature POI name corrections**, `--poi-name-fixes config/poi-name-fixes.csv` uses an exact source-map ID, hex code, latitude, longitude, previous name and replacement name. Never blanket-correct all points with a particular substring: it could mask a decoder error. No name corrections are active by default.
5. **Threads are off by default** (`--jobs 1`); use `--jobs 4` to enable four parallel workers. POI rules apply consistently in both cases.
```sh
sudo apt-get install -y cmake g++ make libpng-dev libsqlite3-dev python3
## Files
- `config/poi-overrides-all-types.csv` (same as `config/poi-overrides-legend-reviewed.csv`): **corrected editable 1,871-row override file** (schema `code,name_contains,mode,tags,note`). 60 active POI-code rows corrected (plus 7 extra verified icons embedded in C++ for future occurrences). Unverified rows are left inert and flagged in the notes.
- `config/poi-name-fixes.csv`: name correction template, no active fixes by default.
- `data/BGMountains-unique-poi-types.csv` (same as `data/BGMountains-unique-poi-types-STRICT-reviewed.csv`): **current useful catalog: 263 retained raw types**, current counts, reviewed classification status, up to five real source coordinates and names per type.
- `data/BGMountains-unique-poi-types-reviewed.csv`: **historical 1,871-type catalog**, with v0.5 counts, exclusions and review status; do not mistake rejected coordinates for correct map objects.
- `data/legend-corrections-log.csv`: 60 present-and-verified mappings; `data/all-67-icon-legend-matches.csv`: 67 TYP icon-to-legend matches (7 icon codes not observed as POIs), including original/new OSM semantic tags and one coordinate each.
- `data/extended-label-decoder-changes.csv`: 117 extended-label before/after results (some names vanish because they came from the wrong LBL offset).
- `data/legend-verified-POI-QA-waypoints.gpx`: 179 sample POIs of verified hex codes, for simultaneous visual comparison in GPXSee + original IMG. Exact count is in the GPX file.
- `data/BGMountains-POI-Explorer.html`: no-network/offline searchable icon reference, with copyable sample coordinates and basemap links.
- `data/poi-icons/*.png`: embedded Garmin icons extracted from the original TYP (copyright depends on source data).
- `tools/validate_pois.py`: previously tested source→OSM audit; limitation remains that it is not a third-party ground-truth or OBF assertion.
## Build and validation
```bash
sudo apt-get install -y cmake g++ libpng-dev libsqlite3-dev python3
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j "$(nproc)"
ctest --test-dir build --output-on-failure
```
## Bundled resources
./build/garmin2osm --input 'gmapsupp.img' \
--output BGMountains-legend-reviewed.osm \
--poi-overrides config/poi-overrides-legend-reviewed.csv \
--poi-name-fixes config/poi-name-fixes.csv \
--poi-manifest BGMountains-source-reviewed.csv \
--poi-types-out BGMountains-types-reviewed.csv
| File | Purpose |
|---|---|
| `data/BGMountains-unique-poi-types.csv` | **All 1,871 observed POI hex types**, counts, named counts, current default classification, icon availability and five source coordinates/names/mapset IDs per hex type |
| `config/poi-overrides-all-types.csv` | **Editable configurator**: one row per hex, with count, default and an example embedded in the notes. Rules initially have empty tags, therefore do not change output |
| `config/poi-overrides.csv` | Minimal template with explanation and an intentionally commented example |
| `data/BGMountains-source-pois.csv.gz` | Complete extracted inventory (416,165 source records) for audit/provenance comparisons; *this is map-derived data* |
| `data/poi-icons/*.png` | 96 successfully decoded Garmin TYP icons indexed by hex code; facilitates visual comparison with GPXSee |
| `tools/validate_pois.py` | Disk-backed, streaming comparison of source manifest against OSM POIs; optional semantic override assertions; OsmAnd inspector wrapper |
| `tests/demo.img` | Synthetic Garmin fixture suitable for reproducing baseline smoke tests |
## Produce a countrywide OSM export + inventory
Default execution is **serial** (`--jobs 1`, even if not supplied). Set `--jobs 4` or another integer 1–64 to opt into parallel mapset processing. Parallel processing uses independent SQLite stages; all OSM nodes are emitted before ways and node refs are rebased correctly.
```sh
./build/garmin2osm \
--input '/path/to/gmapsupp.img' \
--output Bulgaria.osm \
--poi-overrides config/poi-overrides-all-types.csv \
--poi-types-out unique-poi-types.csv \
--poi-manifest source-pois.csv \
--poi-examples 5 \
--jobs 4
```
For a regional check before rebuilding the whole country:
```sh
./build/garmin2osm --input '/path/to/gmapsupp.img' \
--bbox 23.28,42.11,23.47,42.27 --output Rila.osm \
--poi-overrides config/poi-overrides-all-types.csv \
--poi-types-out Rila-types.csv --poi-manifest Rila-source.csv
```
Generate **only** a catalog and manifest (no OSM output or staging DB):
```sh
./build/garmin2osm --input '/path/to/gmapsupp.img' \
--poi-types-out catalog.csv --poi-manifest source.csv --icons-dir my-icons
```
Other original flags remain: `--inspect`, `--contours`, `--all-levels`, `--keep-temp`, `--temp`, `--limit-maps`. Existing OSM, staging, and manifest output paths are never intentionally overwritten. Source inventory is generated even for ignored or `drop`-configured POIs, so intentional exclusions can be audited explicitly.
## Override known defaults and add new POI types
Edit `config/poi-overrides-all-types.csv` in LibreOffice/Excel or a text editor. Columns:
```
code,name_contains,mode,tags,note
0x6414,,replace,man_made=water_well,ONLY IF VERIFIED
0x6503,,append,information=guidepost;hiking=yes,Example: adds another attribute
```
Choose a valid mode: `append`, `replace`, `keep_raw`, or `drop`. `name_contains` is optional and case-sensitive; if populated, the rule applies only to names containing that text. Rule order matters within the same code. Actual `garmin:type`, `garmin:feature`, `garmin:mapset` and `name` cannot be overridden. Use `key=` to delete an existing default key. Unknown hex types will remain raw-only until mapped. **Never infer the meaning of a code solely from its hexadecimal number.**
An immediately usable example that modifies all type `0x6414` points follows:
```
code,name_contains,mode,tags,note
0x6414,,replace,man_made=water_well,TEST ONLY; not evidence of true semantics
```
NOTE: `0x6414` presently has a built-in `amenity=drinking_water` default. Its old duplicate fallback to `water_well` was unreachable and has been removed. The icon and field data need to establish the right category. **The sample replacement is not a verified mapping.**
Coordinates in `data/BGMountains-unique-poi-types.csv` refer to the decoded *original IMG* features. For each type, open the coordinates in GPXSee's Garmin map and inspect both the label and icon. `data/poi-icons/0x6414.png` may provide additional context. The fact that a rendered glyph looks like water is **not** enough to distinguish `natural=spring`, `man_made=water_well` and `amenity=drinking_water`.
## Automated validation: IMG → OSM
The `--poi-manifest` output is a separately emitted inventory of every parsed IMG point. To compare against the generated OSM:
```sh
python3 tools/validate_pois.py compare \
--source Rila-source.csv --osm Rila.osm \
--rules config/poi-overrides-all-types.csv \
--report Rila-report.json \
--mismatches Rila-missing-extra.csv
--source BGMountains-source-reviewed.csv \
--osm BGMountains-legend-reviewed.osm \
--rules config/poi-overrides-legend-reviewed.csv \
--report validated.json --mismatches mismatches.csv
```
Both `.osm` and `.osm.gz` are supported. The validator checks type, exact 7-digit quantized coordinates, name and multiplicity; it separately checks the assertions provided in the override CSV. Its failure exit code is nonzero; `--mismatches` includes direct OSM web links for investigation. Uses temporary SQLite to avoid holding the entire POI set in RAM. A `drop` override is treated as intentional absence, but a matching input record still appears in the source manifest.
Parallelism: append `--jobs 4` to the converter command if desired; otherwise it uses one worker.
**Proof limit:** This is an internal IMG-to-OSM conversion-consistency check, not independent verification of Garmin decoder fidelity or the semantic meaning of a hex code. It cannot by itself prove that your `.obf` contains all the classified features.
**Adding one verified name correction:** edit `config/poi-name-fixes.csv`, with this schema:
## Automated validation: OSM → OBF
Build with the official OsmAndMapCreator distribution (download separately):
```sh
./scripts/build_obf.sh /path/OsmAndMapCreator "$PWD/Rila.osm"
# Custom OsmAnd POI / rendering definitions may be supplied with their FULL respective config XMLs:
./scripts/build_obf.sh /path/OsmAndMapCreator "$PWD/Rila.osm" \
--poi-types /path/to/full-poi_types.xml \
--rendering-types /path/to/full-rendering_types.xml
```csv
mapset,code,lat,lon,old_name,new_name,note
# Example (illustrative; this row is NOT a real correction):
# 02234071,0x6609,42.1897745,23.2835913,Х.Иван Вазов,х. Иван Вазов,Check actual Garmin source before use
```
Inspect **your actual, externally compiled OBF**:
The original `name` and coordinates in the source manifest are quantized to seven decimals for exact matching. A manual name fix affects both OSM and the manifest, so it **cannot count as independent evidence of cartographic correctness**; record your verification separately.
```sh
python3 tools/validate_pois.py obf-inspect \
--inspector /path/to/OsmAndMapCreator/inspector.sh \
--obf /path/to/Rila.obf \
--bbox 23.28,42.11,23.47,42.27 \
--report obf-poi-inspection.txt
```
## What this does not prove
The wrapper invokes OsmAnd BinaryInspector `-vpoi -vpoiobjects` with an optional bounding box and saves the output. It does **not** yet reconcile OBF internal IDs with the Garmin origin; OsmAnd may exclude unsupported tags from its map and POI indexes. For OBF persistence checks, use a POI (known category, name and coordinate) and inspect it via the rendered map / a bounded BinaryInspector dump; unknown types often require extending the **full** OsmAnd `poi_types.xml`. For line colors or custom map tags, modify **full** `rendering_types.xml`, then rebuild the OBF. A CSV rule only affects the OSM export; it does not change OsmAnd's indexes without a rebuild.
The user's full `.obf` was not provided. Compile it again from the updated OSM source and inspect its POI categories/index; neither a successful OSM export nor this audit proves that OsmAnd indexing retains all custom tags. Geometry classification for the map's **area and line legend** remains a separate review (this patch principally covers POIs). Extended types other than those confirmed by icons are **unverified**, even when they pass the subdivision bounds check. Several extended nautical symbols (e.g., navigation lights) appear plausible but have no matching extracted TYP glyph.
## What is tested
## 0x10d1f / Несебър
- Existing bundled CTest suite: 3/3 tests passed.
- Supplied **441,942,016-byte** Bulgaria IMG: decoded **165 map regions**, **416,165 POI records**, **1,871 unique raw point types**, **96 icons**; full inventory generation completed.
- Rila test bbox `23.28,42.11,23.47,42.27`: source and exported OSM each contained **2,424 POIs**, with **zero missing/extra** after seven-decimal coordinate correction.
- Serial and `--jobs 4` Rila jobs: exactly the same multiset of **16,786 ways and 2,424 POIs** (including way geometry and tags).
- A deliberately overridden test type was exported and assertions passed.
- **Not tested:** user's full final `.obf` (not provided here); comprehensive semantic correspondence of 1,871 Garmin types; original Garmin route-network validity/turn-restrictions; all possible IMG/GMP variants.
## Concurrency caution
The Garmin container file uses a shared `std::ifstream`, protected by a mutex to ensure safe reads. Parser threads work on different `Map` objects with separate SQLite stores. For maximum throughput, `--jobs 4` may help on multi-core CPUs, but the disk read lock, SQLite writes and OSM XML output remain throughput bottlenecks. The default `--jobs 1` is deterministic and safest. Memory and temporary disk use increase with concurrency.
## Add independent POI records (optional)
Use `config/poi-additions.csv` with four CSV fields: `lat,lon,name,tags`.
The program includes them with `garmin:feature=manual_poi` and `source=user_manual`.
They are **not** presented as Garmin-extracted features; the source-POI validator
ignores these intentional extra nodes. Add the flag `--poi-additions config/poi-additions.csv`
to a command with `--output` (empty file is a no-op).
## Reproduce the bundled regional audit without the source IMG
```sh
python3 tools/validate_pois.py compare \
--source samples/Rila-source-pois.csv \
--osm samples/Rila-vector.osm.gz \
--report results.json --mismatches differences.csv
```
For the complete source inventory, use the gzip-compressed CSV directly:
`python3 tools/validate_pois.py compare --source data/BGMountains-source-pois.csv.gz --osm /your/full-map.osm`.
The v0.5 type list showed `есебър` at **43.0592179,27.7399421**. At that latitude it is ~44 km north of the original Nesebar town POI. The v0.6 label-pointer fix no longer returns `есебър` at that record, and the subdivision bounds check suppresses the spurious record. **No synthetic `Несебър` POI has been inserted.** The real original city/town record is preserved unchanged.