louis/gigafibre-fsm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2f1ebae587
gigafibre-fsm/scripts/migration
History
louispaulb 10afd696ae fix(migration): clean address_line + postal_code + connection_type at import 
Three legacy data-quality issues that were leaking into ERPNext on every
import run. Caught while auditing C-LPB4's mis-pinned dispatch job.

1. **Postal code embedded in address_line.** Legacy `gestionclient` had
   rows like `2200-3 chemin de la riviere de la guerre  J0S1B0` with
   the postal code concatenated at the end (and the same code repeated
   in the dedicated zip column). Caused 48-char address_line on what
   should have been a 39-char address. Now stripped at import: a regex
   matches `\\s+<FSA><LDU>\\s*$` (with or without space) and removes
   it; the dedicated postal_code field carries the canonical form.

2. **Abbreviations + Cobol-style capitalization.** Legacy stored
   `2066 Ch De La 1Re-Concession` instead of the canonical
   `2066 Chemin de la 1re-Concession`. ABBREV_MAP expands `Ch` →
   `Chemin`, `Av` → `Avenue`, `Bd`/`Boul` → `Boulevard`, `Rte` →
   `Route`, `St-` → `Saint-`, `Ste-` → `Sainte-`, `Mtl` → `Montréal`.
   Title-casing rule preserves French articles lowercase (`de`, `du`,
   `des`, `la`, `le`, `les`, `au`, `aux`, `à`, `et`, `sur`, `en`)
   and ordinal markers (`1re`, `2e`, `3e`). 96 SLs in production had
   the `1Re-Concession` style; they'll be re-normalized on next
   migration run.

3. **`connection_type` left empty even when ONT/CPE devices existed.**
   Pre-loads device→delivery mapping at import start; if the legacy
   delivery has any device whose category/name/model contains "ont",
   "onu", "cpe", "fibre", "gpon", or "ftth", we set
   connection_type='Fibre FTTH'. Without devices on file, the field
   stays empty (rep fills it later) — we don't guess.

4. **`postal_code` normalized too** — `j0s1b0` → `J0S 1B0` (uppercase
   + canonical space). Was being inserted in lowercase no-space form.

Self-tested on 8 representative cases including the actual broken
records found in production (LOC-15903, LOC-6227, LOC-4 / C-LPB4).

These changes affect only re-imports of locations. Existing data
needs a separate backfill script — a follow-up will cover that
either as a one-shot migration or by running the existing
`reimport_subscriptions.py` after this script.
2026-05-08 15:38:19 -04:00
..
genieacs-export feat: extract GenieACS WiFi/VoIP provisioning data from MariaDB 2026-04-03 08:49:26 -04:00
add_missing_custom_fields.py feat: ONT diagnostics — grouped mesh topology, signal RSSI, management link 2026-04-03 21:26:14 -04:00
add_office_extension.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
add_performance_indexes.sql feat: telephony UI, performance indexes, Twilio softphone, lazy-load invoices 2026-04-02 13:59:59 -04:00
analyze_pricing_cleanup.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
check_gl_dates.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
check_items.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
check_missing_cat26.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
check_naming.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
clean_reimport.py feat: flow editor, Gemini QR scanner with offline queue, dispatch planning v2 2026-04-22 10:44:17 -04:00
cleanup_customer_status.py feat: inline editing, search, notifications + full repo cleanup 2026-03-31 07:34:41 -04:00
create_custom_doctypes.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
create_portal_users.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
debug_idx.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
debug_match.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
debug_match2.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
explore_expro_payments.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
explore_expro_services.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
export_genieacs.sh feat: GenieACS config export + TR-069 to TR-369 migration plan 2026-04-02 21:03:41 -04:00
fix_and_invoices.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
fix_annual_billing_dates.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
fix_customer_links.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
fix_dates.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
fix_gl_entries.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
fix_income_accounts.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
fix_income_sql.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
fix_invoice_customer_names.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
fix_invoice_outstanding.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
fix_issue_cust2.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
fix_issue_owners.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
fix_no_rebate_discounts.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
fix_olt_port_ip.py refactor: major cleanup — remove dead dispatch app, commit all backend code, extract client composables 2026-04-08 17:38:38 -04:00
fix_reversals.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
fix_reversement.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
fix_sub_address.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
fix_subscription_details.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
geocode_locations.py feat: inline editing, search, notifications + full repo cleanup 2026-03-31 07:34:41 -04:00
import_customer_details.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
import_devices_and_enrich.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
import_employees.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
import_expro_payments.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
import_invoice_notes.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
import_invoices.py feat: flow editor, Gemini QR scanner with offline queue, dispatch planning v2 2026-04-22 10:44:17 -04:00
import_items.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
import_memos.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
import_missing_services.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
import_payment_arrangements.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
import_payment_methods.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
import_payments.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
import_services_and_enrich_customers.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
import_soumissions.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
import_suspensions.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
import_technicians.py feat: inline editing, search, notifications + full repo cleanup 2026-03-31 07:34:41 -04:00
import_terminated.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
import_ticket_msgs.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
import_voip_lines.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
invoice_preview.jinja feat: flow editor, Gemini QR scanner with offline queue, dispatch planning v2 2026-04-22 10:44:17 -04:00
logo-targo-green.svg feat: flow editor, Gemini QR scanner with offline queue, dispatch planning v2 2026-04-22 10:44:17 -04:00
migrate_all.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
migrate_direct.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
migrate_locations.py fix(migration): clean address_line + postal_code + connection_type at import 2026-05-08 15:38:19 -04:00
migrate_missing_data.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
migrate_phase3.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
migrate_phase5.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
migrate_provisioning_data.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
migrate_tickets.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
migrate_users.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
MIGRATION_MAP.md refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
nuke_data.py feat: inline editing, search, notifications + full repo cleanup 2026-03-31 07:34:41 -04:00
reconcile_subscriptions.py feat: subscription reimport, customer/doctype ID rename, zero-padded format 2026-04-01 17:17:23 -04:00
reimport_payments.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
reimport_subscriptions.py feat: telephony UI, performance indexes, Twilio softphone, lazy-load invoices 2026-04-02 13:59:59 -04:00
rename_all_doctypes.py feat: subscription reimport, customer/doctype ID rename, zero-padded format 2026-04-01 17:17:23 -04:00
rename_customers_c_prefix.py feat: telephony UI, performance indexes, Twilio softphone, lazy-load invoices 2026-04-02 13:59:59 -04:00
rename_customers.py feat: subscription reimport, customer/doctype ID rename, zero-padded format 2026-04-01 17:17:23 -04:00
rename_to_readable_ids.py feat: inline editing, search, notifications + full repo cleanup 2026-03-31 07:34:41 -04:00
setup_invoice_print_format.py feat: flow editor, Gemini QR scanner with offline queue, dispatch planning v2 2026-04-22 10:44:17 -04:00
setup_portal_auth_bridge.py feat: inline editing, search, notifications + full repo cleanup 2026-03-31 07:34:41 -04:00
setup_quote_print_format.py feat: flow editor, Gemini QR scanner with offline queue, dispatch planning v2 2026-04-22 10:44:17 -04:00
setup_scheduler_toggle.py feat: inline editing, search, notifications + full repo cleanup 2026-03-31 07:34:41 -04:00
setup_subscription_api.py feat: inline editing, search, notifications + full repo cleanup 2026-03-31 07:34:41 -04:00
setup_user_roles.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
simulate_payment_import.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
test_jinja_render.py feat: flow editor, Gemini QR scanner with offline queue, dispatch planning v2 2026-04-22 10:44:17 -04:00
update_assigned_staff.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
update_item_descriptions.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00
update_opened_by_staff.py refactor: reduce token count, DRY code, consolidate docs 2026-04-13 08:39:58 -04:00