gigafibre-fsm/scripts/migration/MIGRATION_MAP.md

Legacy → ERPNext Migration Map

Overview

Migration from legacy PHP/MariaDB billing system (gestionclient) to ERPNext v16 on PostgreSQL.

• Source: MariaDB in Docker container legacy-db on 96.125.196.67 (port 3307 on host), database gestionclient
• Target: ERPNext at erp.gigafibre.ca, company TARGO, currency CAD
• Scope: All historical data (no date cutoff)
• Method: Bulk SQL INSERT (bypasses Frappe ORM for speed — 4 min vs ~120 hours)

---

Key Accounting Problems Solved

The legacy system had several non-standard practices that broke standard accounting. Here is what was fixed during migration:

#	Legacy Problem	Impact	ERPNext Solution
1	Credit notes not linked to original invoices — negative invoices created with no reference back to the invoice they cancel	No audit trail; credits appear as free-floating	3 matching mechanisms reconstruct the links (16,830 credit notes linked via return_against)
2	Fake "reversement" payments — internal bookkeeping entries recorded as real payments when cancelling invoices	$955K phantom overpayment	Excluded from import; replaced by proper credit note allocation
3	Duplicate payments from portal — slow legacy backend causes same credit card charge to be submitted twice (same Stripe reference)	Invoices appear double-paid; bank balance overstated	Deduplicated by (account_id, reference) — 178,730 duplicates removed
4	Invoices both paid and reversed — customer pays, then invoice also gets a credit note reversal	Invoice shows negative outstanding (overpaid)	Extra reversal entries delinked; invoice marked as settled
5	Tax-inclusive totals — legacy stores total with tax included, no separate net amount	ERPNext needs both net and gross	Tax amounts back-calculated from invoice_tax table
6	No due dates — most invoices have no due date	Cannot determine overdue status	posting_date used as fallback
7	All revenue in one GL account — initial import put all invoice income into generic Autres produits d'exploitation - T	No revenue breakdown by service type; P&L unusable for analysis	Post-import fix maps SKU → product_cat.num_compte → ERPNext named account. GL entries updated to dominant account per invoice. Scripts: fix_income_accounts.py (row-by-row), fix_income_sql.py (bulk via temp table), fix_gl_entries.py (GL-only pass)

---

Glossary — Internal Prefixes

These prefixes are used in ERPNext document names to identify records created during migration:

Prefix	Stands for	Description
SINV-	Sales INVoice	Invoice document (e.g., SINV-638567)
PE-	Payment Entry	Payment received from customer
PER-	Payment Entry Reference	Allocation of a payment to a specific invoice
SII-	Sales Invoice Item	Line item on an invoice
stc-tps-	Sales Tax Charge — TPS	GST tax row (5%)
stc-tvq-	Sales Tax Charge — TVQ	QST tax row (9.975%)
ple-	Payment Ledger Entry	Tracks what's owed per invoice (ERPNext outstanding system)
plc-	PLE — Credit allocation	Credit note reducing a target invoice's balance
plr-	PLE — Reversal allocation	Reversal (from invoice notes) reducing a target invoice's balance
gir-	GL — Invoice Receivable	GL entry: debit to Accounts Receivable
gii-	GL — Invoice Income	GL entry: credit to Revenue
glt-	GL — TPS	GL entry: credit to TPS (GST) liability
glq-	GL — TVQ	GL entry: credit to TVQ (QST) liability
gpb-	GL — Payment Bank	GL entry: debit to Bank
gpr-	GL — Payment Receivable	GL entry: credit to Accounts Receivable

---

Data Flow Diagram

┌─────────────────────────────────────────────────────────────────────┐
│                        LEGACY (MariaDB)                             │
│                                                                     │
│  account ──────────────────────────────────┐                        │
│  invoice ──┬── invoice_item                │                        │
│            └── invoice_tax (TPS/TVQ rows)  │                        │
│  payment ──┬── payment_item (allocations)  │                        │
│            └── type='credit' (memo→#NNN)   │                        │
└────────────────────────┬───────────────────┘                        │
                         │                                            │
                         ▼                                            │
┌─────────────────────────────────────────────────────────────────────┐
│                        ERPNext (PostgreSQL)                          │
│                                                                     │
│  ┌──────────┐    ┌─────────────────┐    ┌──────────────────────┐   │
│  │ Customer │◄───│ Sales Invoice   │───►│ GL Entry (4 per inv) │   │
│  │          │    │  ├─ SI Item     │    │  gir- receivable     │   │
│  │          │    │  ├─ SI Tax(TPS) │    │  gii- income         │   │
│  │          │    │  └─ SI Tax(TVQ) │    │  glt- TPS            │   │
│  │          │    │                 │    │  glq- TVQ             │   │
│  │          │    │                 │───►│ PLE (1 per invoice)   │   │
│  │          │    │                 │    │  ple-SINV-            │   │
│  │          │    └─────────────────┘    └──────────────────────┘   │
│  │          │                                                       │
│  │          │    ┌─────────────────┐    ┌──────────────────────┐   │
│  │          │◄───│ Payment Entry   │───►│ GL Entry (2 per pmt) │   │
│  │          │    │  └─ PE Ref      │    │  gpb- bank           │   │
│  │          │    │    (allocations)│    │  gpr- receivable     │   │
│  │          │    │                 │───►│ PLE (1 per alloc)    │   │
│  │          │    │                 │    │  ple-PER-            │   │
│  └──────────┘    └─────────────────┘    └──────────────────────┘   │
│                                                                     │
│  Credit Allocations ──────────────────►  PLE (plc-)                │
│  (return inv → target inv)              against_voucher = target    │
└─────────────────────────────────────────────────────────────────────┘

---

Entity Relationships (FK Map)

Every entity references others by name (ERPNext primary key). The rename scripts update all these FKs in a single pass.

Customer (C-{legacy_account_id})
├── Sales Invoice.customer ─────────────► Customer.name
│   ├── SI Item.parent ─────────────────► Sales Invoice.name
│   ├── SI Tax.parent ──────────────────► Sales Invoice.name
│   ├── GL Entry.voucher_no ────────────► Sales Invoice.name  (4 per invoice)
│   ├── PLE.voucher_no ─────────────────► Sales Invoice.name  (1 per invoice)
│   ├── PLE.against_voucher_no ─────────► Sales Invoice.name  (from payments/credits)
│   ├── Payment Entry Reference.reference_name ► Sales Invoice.name
│   ├── Comment.reference_name ─────────► Sales Invoice.name  (invoice notes)
│   └── return_against ─────────────────► Sales Invoice.name  (credit note → original)
│
├── Payment Entry.party ────────────────► Customer.name
│   ├── PE Reference.parent ────────────► Payment Entry.name
│   ├── GL Entry.voucher_no ────────────► Payment Entry.name  (2 per payment)
│   ├── PLE.voucher_no ─────────────────► Payment Entry.name  (1 per allocation)
│   └── Comment.reference_name ─────────► Payment Entry.name
│
├── Subscription.party ─────────────────► Customer.name
│   ├── Subscription Plan Detail.parent ► Subscription.name
│   └── Subscription.service_location ──► Service Location.name
│
├── Service Subscription.customer ──────► Customer.name
│   └── Service Subscription.service_location ► Service Location.name
│
├── Issue.customer ─────────────────────► Customer.name
│   ├── Communication.reference_name ───► Issue.name
│   ├── Comment.reference_name ─────────► Issue.name  (ticket messages)
│   ├── Dispatch Job.issue ─────────────► Issue.name
│   └── Issue.service_location ─────────► Service Location.name
│
├── Comment.reference_name ─────────────► Customer.name  (memos)
│
├── Service Location ───────────────────► (linked via Subscription/Issue)
│   └── Service Equipment.service_location ► Service Location.name
│
└── Employee ───────────────────────────► (linked via Dispatch Technician)
    └── Dispatch Technician.employee ───► Employee.name

GL Entry ↔ PLE Linking

Sales Invoice SINV-{id}
  ├── GL: gir-SINV-{id}  → Receivable  (debit grand_total)
  ├── GL: gii-SINV-{id}  → Income*     (credit net_total)
  ├── GL: glt-SINV-{id}  → TPS         (credit tps)
  ├── GL: glq-SINV-{id}  → TVQ         (credit tvq)
  └── PLE: ple-SINV-{id} → voucher_no=SINV-{id}, against_voucher_no=SINV-{id}

Payment Entry PE-{id} allocated to SINV-{target}
  ├── GL: gpb-PE-{id}    → Bank        (debit paid_amount)
  ├── GL: gpr-PE-{id}    → Receivable  (credit paid_amount)
  └── PLE: ple-PER-{id}-{idx} → voucher_no=PE-{id}, against_voucher_no=SINV-{target}

Credit Note SINV-{credit} → SINV-{target}
  └── PLE: plc-{serial}  → voucher_no=SINV-{credit}, against_voucher_no=SINV-{target}

*Income account = SKU-mapped account from product_cat.num_compte (dominant per invoice)

---

Naming Series & New ID Allocation

Migrated records use legacy IDs. New records use autoincrement series set above the max legacy ID to avoid collisions.

DocType	Legacy Format	Zero-padded	New Format (post-migration)	Series Counter
Customer	C-{legacy_account_id}	C-{id:014d}	C-10000000034941+	C- → 10000000034940
Sales Invoice	SINV-{legacy_id}	SINV-{id:010d}	SINV-0000100001+	SINV- → 100000
Payment Entry	PE-{legacy_id}	PE-{id:010d}	PE-0000100001+	PE- → 100000
Issue	ISS-{legacy_ticket_id}	ISS-{id:010d}	ISS-0000250001+	ISS- → 250000
Service Location	LOC-{legacy_delivery_id}	LOC-{id:010d}	LOC-0000100001+	LOC- → 100000
Service Equipment	EQP-{legacy_device_id}	EQP-{id:010d}	EQP-0000100001+	EQP- → 100000
Service Subscription	SUB-{legacy_service_id}	SUB-{id:010d}	SUB-0000100001+	SUB- → 100000
Subscription (native)	ASUB-{legacy_service_id}	ASUB-{id:010d}	ASUB-0000100001+	ASUB- → 100000
Employee	HR-EMP-{seq}	—	HR-EMP-{N+1}+	HR-EMP- → max legacy
SI Item	SII-{inv_id}-{idx}	—	(child, no series)	—
PE Reference	PER-{pmt_id}-{idx}	—	(child, no series)	—
GL Entry	gir-/gii-/glt-/glq-/gpb-/gpr-	—	(child, no series)	—
PLE	ple-/plc-/plr-	—	(child, no series)	—

Rename Process (rename_all_doctypes.py)

Two-phase rename to avoid PK collisions:

1. Phase A: SINV-638567 → _TEMP_SINV-0000638567
2. Phase B: _TEMP_SINV-0000638567 → SINV-0000638567
3. FK cascade: All child tables updated via temp mapping table + UPDATE ... FROM JOIN before rename

FK tables updated per doctype:

• Sales Invoice → SI Item, SI Tax, GL Entry, PLE, PE Reference, Comment, Version, Dynamic Link
• Payment Entry → PE Reference, GL Entry, PLE, Comment, Version, Dynamic Link
• Issue → Dispatch Job, Comment, Version, Communication, Dynamic Link
• Service Location → Service Subscription, Subscription, Service Equipment, Issue, Dispatch Job
• Service Equipment → Comment, Dynamic Link
• Service/Native Subscription → Subscription Plan Detail, Comment, Dynamic Link

---

Entity Mapping

Customer

Legacy (account)              ERPNext (Customer)
─────────────────             ──────────────────
id                      →     legacy_account_id
first_name + last_name  →     customer_name
email                   →     (Contact)
                               name = CUST-{hash}

Sales Invoice

Legacy (invoice)              ERPNext (Sales Invoice)
─────────────────             ──────────────────────
id                      →     legacy_invoice_id
                               name = SINV-{legacy_id}
                               Example: 638567 → SINV-638567
account_id              →     customer (via account→Customer map)
total_amt               →     grand_total (TAX-INCLUSIVE)
total_amt - tps - tvq   →     net_total
date_orig (unix ts)     →     posting_date (YYYY-MM-DD)
total_amt < 0           →     is_return = 1
billed_amt              →     (not stored — derived from PLE)
                               outstanding_amount = SUM(PLE against this inv)

Key: Legacy total_amt is TAX-INCLUSIVE. ERPNext stores both grand_total (with tax) and net_total (without tax).

Invoice Items

Legacy (invoice_item)         ERPNext (Sales Invoice Item)
─────────────────────         ──────────────────────────
invoice_id              →     parent = SINV name
product_name            →     item_name, description
quantity                →     qty
unitary_price           →     rate
quantity × unitary_price →    amount
                               name = SII-{legacy_id}-{idx}
                               item_code = 'SVC'
                               income_account = mapped from SKU → product_cat.num_compte → ERPNext account
                                 (fallback: "Autres produits d'exploitation - T" if SKU unmapped)

Invoice Taxes

Legacy (invoice_tax)          ERPNext (Sales Taxes and Charges)
────────────────────          ─────────────────────────────────
invoice_id              →     parent = SINV name
tax_name = 'TPS'        →     description = 'TPS à payer - T'
  amount                →       tax_amount (5% GST, #834975559RT0001)
  tax_rate = 0.05       →       rate = 5.0
tax_name = 'TVQ'        →     description = 'TVQ à payer - T'
  amount                →       tax_amount (9.975% QST, #1213765929TQ0001)
  tax_rate = 0.09975    →       rate = 9.975
                               name = stc-tps-{legacy_id} / stc-tvq-{legacy_id}

Note: Legacy stores TPS and TVQ as separate rows per invoice. ERPNext also uses separate rows (idx=1 for TPS, idx=2 for TVQ).

Payment Entry

Legacy (payment)              ERPNext (Payment Entry)
────────────────              ──────────────────────
id                      →     legacy_payment_id
                               name = PE-{legacy_id}
account_id              →     party (via account→Customer map)
amount                  →     paid_amount = received_amount
date_orig (unix ts)     →     posting_date
memo                    →     remarks
type != 'credit'        →     (only non-credit payments imported as PE)
                               payment_type = 'Receive'
                               paid_from = 'Comptes clients - T'
                               paid_to = 'Banque - T'

Payment Allocations

Legacy (payment_item)         ERPNext (Payment Entry Reference)
─────────────────────         ───────────────────────────────
payment_id              →     parent = PE name
invoice_id              →     reference_name = SINV name
amount                  →     allocated_amount
                               name = PER-{legacy_pmt_id}-{idx}
                               reference_doctype = 'Sales Invoice'

Credit Notes (Return Invoices)

Legacy — Three linking mechanisms:

  1. Credit payment:
     payment (type='credit', memo='credit created by invoice #NNN')
       → payment_item.invoice_id = target invoice

  2. Invoice notes:
     invoice.notes = 'Renversement de la facture #NNN'
       → #NNN = target legacy invoice ID

  3. Reversement payment memo:
     payment (type='reversement', memo='create by invoice #CREDIT for invoice #TARGET')
       → payment_item.invoice_id = target invoice

ERPNext:
  Sales Invoice (is_return=1, return_against=target SINV)
    → PLE (plc-{id})   — from credit payments (mechanism 1)
    → PLE (plr-{id})   — from reversal notes + reversement memos (mechanisms 2+3)
        voucher_no = source return SINV
        against_voucher_no = target SINV
        amount = credit invoice's grand_total (negative)

---

Accounting Entries (Double-Entry)

Per Sales Invoice (4 GL entries)

                              Debit         Credit
                              ─────         ──────
Comptes clients - T (AR)      grand_total
Income account* (Revenue)                   net_total
TPS à payer - T (Tax)                       tps_amount
TVQ à payer - T (Tax)                       tvq_amount

* Income account = dominant SKU-mapped account for the invoice
  (from product_cat.num_compte). Falls back to "Autres produits d'exploitation - T".
  Note: GL uses ONE income account per invoice (the dominant/highest-amount one).
  Individual SI Items keep their per-SKU accounts for drill-down reporting.
                              ─────────     ─────────
                              grand_total = grand_total  ✓

For return invoices (negative amounts), debit/credit are swapped.

Per Payment Entry (2 GL entries)

                              Debit         Credit
                              ─────         ──────
Banque - T (Bank)             paid_amount
Comptes clients - T (AR)                    paid_amount

Payment Ledger Entry (PLE) — Outstanding Tracking

Type                    amount              against_voucher
────                    ──────              ───────────────
Invoice posting         +grand_total        self (SINV)
Payment allocation      -allocated_amount   target SINV
Credit allocation       -credit_amount      target SINV
Unallocated payment     -paid_amount        self (PE)

Outstanding = SUM(PLE.amount WHERE against_voucher = this invoice)

---

ERPNext Chart of Accounts

TARGO (Company, abbr: T)
├── Comptes clients - T          Receivable    (debit_to for all invoices)
├── Banque - T                   Bank          (paid_to for all payments)
├── Income accounts (mapped from legacy product_cat.num_compte):
│   ├── Autres produits d'exploitation - T  Income (fallback for unmapped SKUs)
│   ├── [4020] Mensualite fibre - T         Income (fibre monthly)
│   ├── [4xxx] Other mapped accounts...     Income (per product category)
│   └── (run fix_income_sql.py to see full distribution)
├── TPS à payer - T              Liability/Tax (5% GST)
└── TVQ à payer - T              Liability/Tax (9.975% QST)

---

Naming Conventions

Migrated Data (Legacy IDs)

Entity	Pattern	Example
Sales Invoice	SINV-{legacy_id}	SINV-638567
SI Item	SII-{legacy_id}-{idx}	SII-638567-0
SI Tax (TPS)	stc-tps-{legacy_id}	stc-tps-638567
SI Tax (TVQ)	stc-tvq-{legacy_id}	stc-tvq-638567
Payment Entry	PE-{legacy_id}	PE-76531
PE Reference	PER-{legacy_id}-{idx}	PER-76531-0
GL (inv receivable)	gir-SINV-{id}	gir-SINV-638567
GL (inv income)	gii-SINV-{id}	gii-SINV-638567
GL (inv TPS)	glt-SINV-{id}	glt-SINV-638567
GL (inv TVQ)	glq-SINV-{id}	glq-SINV-638567
GL (pmt bank)	gpb-PE-{id}	gpb-PE-76531
GL (pmt receivable)	gpr-PE-{id}	gpr-PE-76531
PLE (invoice)	ple-SINV-{id}	ple-SINV-638567
PLE (pmt alloc)	ple-PER-{id}-{idx}	ple-PER-76531-0
PLE (unallocated)	ple-PE-{id}	ple-PE-76531
PLE (credit alloc)	plc-{serial}	plc-1234
PLE (reversal alloc)	plr-{serial}	plr-567

Post-Migration (New Documents)

Entity	Pattern	Example
Sales Invoice	SINV-YYYY-NNNNN	SINV-2026-700001
Payment Entry	ERPNext autoname	PE-2026-00001

The different naming patterns between migrated and new documents ensure no collision if a reimport is needed after new documents have been created.

---

Legacy Database Schema

invoice

Column	Type	Notes
id	bigint PK	→ legacy_invoice_id in ERPNext
date_orig	bigint	UNIX timestamp
account_id	bigint FK	→ account.id
total_amt	double(20,2)	TAX-INCLUSIVE
billed_amt	double(20,2)	Amount paid
due_date	bigint	UNIX timestamp
correction	tinyint	1 = correction invoice
notes	mediumtext

invoice_item

Column	Type	Notes
id	bigint PK
invoice_id	bigint FK	→ invoice.id
product_name	varchar(512)	NOT description
quantity	double	NOT qty
unitary_price	double	NOT price
sku	varchar(128)

invoice_tax

Column	Type	Notes
id	bigint PK
invoice_id	bigint FK	→ invoice.id
tax_name	varchar(128)	'TPS' or 'TVQ' — separate rows, not columns
tax_rate	double	0.05 or 0.09975
amount	double(20,2)	Tax amount for this type

payment

Column	Type	Notes
id	bigint PK
account_id	bigint FK	→ account.id
date_orig	bigint	UNIX timestamp
amount	double
type	varchar(25)	'payment', 'credit', etc.
memo	varchar(512)	For credit: "credit created by invoice #NNN"
reference	varchar(128)	Stripe/processor transaction ID (e.g., pi_3Sad...)

payment_item

Column	Type	Notes
id	bigint PK
payment_id	bigint FK	→ payment.id
invoice_id	bigint FK	→ invoice.id
amount	double	Allocated to this invoice

account (customer)

Column	Type	Notes
id	bigint PK	→ legacy_account_id in Customer
first_name	varchar
last_name	varchar
email	varchar
company_name	varchar

---

Fiscal Year

Canadian fiscal year: July 1 – June 30

posting_date month >= 7  →  fiscal_year = "YYYY-(YYYY+1)"   e.g. "2025-2026"
posting_date month <  7  →  fiscal_year = "(YYYY-1)-YYYY"    e.g. "2024-2025"

---

Legacy Non-Compliance Changelog

The legacy PHP/MariaDB system uses several non-standard accounting practices that were corrected during migration to ERPNext.

1. Reversal Invoices — No Credit Link

Legacy behavior: To cancel an invoice, the system creates a negative invoice (same amount, opposite sign) on the same account. It sets billed_amt = total_amt on both invoices to mark them as "settled," but does not create a credit payment or any explicit reference between them. There is no return_against, no credit_of, and no payment linking the two.

Problem: With no link, there's no audit trail showing which invoice was cancelled by which credit note. The negative invoices appear as unallocated credits, creating phantom overpayment.

ERPNext fix: Three matching mechanisms were developed to reconstruct the links:

1. Credit payment allocations (15,694 matches) — payment.type='credit' with memo='credit created by invoice #NNN' → payment_item.invoice_id points to target. Zero overlap with mechanism 2.
2. Invoice notes field (5,095 matches) — invoice.notes contains 'Renversement de la facture #NNN' referencing the target legacy invoice ID.
3. Reversement payment memos (143 unique matches) — payment.type='reversement' with memo='create by invoice #CREDIT for invoice #TARGET' → payment_item.invoice_id = target. Only applied to invoices not already matched by mechanisms 1-2.

Total: 16,830 credit notes linked via return_against + PLE allocation entries.

2. Fake "Reversement" Payments

Legacy behavior: When a reversal invoice is created, the system also generates a payment.type = 'reversement' record allocated to the original invoice. This is not a real customer payment — it's an internal bookkeeping entry that marks the original invoice as "paid."

Problem: If imported as a real Payment Entry in ERPNext, the original invoice appears double-settled (once by the fake payment, once by the credit note PLE), creating ~$955K in phantom overpayment.

ERPNext fix: Excluded all ~5,000 "reversement" payments from import. These are not real financial transactions — the credit note relationship (mechanism 1/2/3 above) replaces them. The reversement memos are still parsed for matching purposes (mechanism 3).

3. Payment Types in Legacy

Legacy payment.type	Real Payment?	ERPNext Treatment
paiement direct	Yes	Payment Entry
carte credit	Yes	Payment Entry
ppa	Yes (pre-authorized)	Payment Entry
credit	No — credit note allocation	PLE (plc-) only, memo parsed for matching
cheque	Yes	Payment Entry
reversement	No — fake reversal payment	Excluded from PE, memo parsed for matching
comptant	Yes	Payment Entry
credit targo	No — internal credit	Excluded

4. Duplicate Payments (Double Form Submission)

Legacy behavior: The customer portal sometimes processes the same credit card charge twice when the legacy PHP backend is slow to respond. The submit handler fires again, creating a duplicate payment record. Both payments have the same reference field (Stripe transaction ID, e.g., pi_3SadKtAU3HUVhnM10KjBOebK) but different payment.id values.

Problem: Both payments are allocated to the same invoice via payment_item, causing the invoice to appear overpaid. The GL bank balance is also overstated.

ERPNext fix: Deduplicate payments by (account_id, reference) during import — keep only the first payment per unique reference. 178,730 duplicate payments removed (from 522K raw to 343K unique).

5. Tax-Inclusive Totals

Legacy behavior: invoice.total_amt is tax-inclusive (includes TPS 5% + TVQ 9.975%). Individual tax amounts are stored in separate invoice_tax rows, not as columns on the invoice.

ERPNext: Stores both grand_total (with tax) and net_total (without tax). Tax amounts are back-calculated from the invoice_tax table. When no tax record exists, taxes are estimated at 14.975% of total.

6. No Due Date Tracking

Legacy behavior: Many invoices have due_date = 0 (UNIX epoch) or NULL. The system doesn't enforce payment terms.

ERPNext fix: Uses posting_date as fallback when due_date is missing or invalid (before 2000).

7. Credit Notes with Both Payment and Reversal PLE

Legacy behavior: Some credit invoices have BOTH a credit payment allocation (mechanism 1) AND a separate reversal invoice reference (mechanism 2 or 3). If both are processed, the target invoice gets double-credited.

Problem: Creates phantom overpayment (negative outstanding) on target invoices. Originally caused $975K in overpaid balances.

ERPNext fix: Apply matching mechanisms in priority order. Track already_linked set — if a credit invoice was matched by credit payment (mechanism 1), skip it for reversal notes (mechanism 2) and reversement memos (mechanism 3). This prevents double-counting.

8. Orphaned Invoices (No Customer Account)

Legacy behavior: 9 invoices reference account_id values that don't exist in the account table (deleted customers or test data).

ERPNext fix: Skipped during import. Logged as unmapped: IDs 2712, 5627, 15119, 15234, 195096, 216370, 272051, 277963, 308988.

9. Invoice Naming for CRA Compliance

Legacy behavior: Invoices have integer IDs with no prefix. No formal naming convention.

Problem: CRA requires sequential invoice numbering for audit trail. Hex-encoded IDs (e.g., SINV-000009BE67) are not human-readable and can cause hash collisions on child tables.

ERPNext fix: Use SINV-{legacy_id} format (e.g., SINV-638567). Legacy IDs are already sequential integers. Post-migration invoices will use SINV-YYYY-NNNNN format (starting at SINV-2026-700001) to avoid collision with legacy IDs on reimport.

10. Customer Portal Credentials

Legacy behavior: 15,305 customer accounts with username/password (hashed, likely MD5/SHA1). Password reset tokens in client_pwd table (9,687 tokens). Stripe customer IDs stored in account.stripe_id.

ERPNext fix: Legacy password hashes are incompatible with Frappe's bcrypt auth. After migration, ERPNext Website Users will be created with customer emails, and bulk password reset emails will be sent. Stripe IDs will be linked to ERPNext's payment integration.

---

Migration Results (2026-03-29, full historical import)

Migration time: ~16 minutes (966 seconds) for full reimport including cleanup, data load from legacy MariaDB, bulk SQL inserts, GL entries, PLE entries, and outstanding recalculation. This compares to an estimated ~120 hours if using Frappe ORM.

Metric	Count
Sales Invoices	629,935
Payment Entries	343,684 (excl. reversement + credit + credit targo)
Payment References	426,018
GL Entries	3,135,184
PLE Entries	1,060,041
GL Balance	$130,120,226.76 = $130,120,226.76 (diff $0.00)

Credit Note Matching

Mechanism	Matches
Credit payment allocations (plc-)	15,508
Reversal notes + reversement memos (plr-)	5,238
Total linked	20,746

Invoice Status Breakdown

Status	Count
Paid	415,861
Overdue	197,190
Return	16,868
Credit Note Issued	15
Unpaid	1

Outstanding

• Owed: $20,500,562.62
• Overpaid: -$3,695.77 (15 invoices — unmatched credit notes with no reference to original in any of the 3 mechanisms)

Payment Deduplication

• Raw payments from legacy: 522,416
• After dedup by (account_id, reference): 343,686
• Duplicates removed: 178,730

---

How to Re-run

The migration is idempotent — the script deletes all existing data and reimports from scratch.

# From facturation.targo.ca:
ssh root@96.125.196.67 'docker cp /path/to/clean_reimport.py erpnext-backend-1:/home/frappe/frappe-bench/clean_reimport.py'
ssh root@96.125.196.67 'docker exec erpnext-backend-1 /home/frappe/frappe-bench/env/bin/python /home/frappe/frappe-bench/clean_reimport.py'

Important: Update the password in the script before running (redacted as ******* in git).

---

Complete Migration Phases & Script Inventory

Execution Order

The migration is split into phases. Each phase can be re-run independently (most scripts are idempotent or nuke-and-reimport).

Phase	Script	Description	Status
0	nuke_data.py	Delete all migrated data except Users, Items, Plans	Run before reimport
1a	clean_reimport.py	Master accounting import: 630K invoices, 344K payments, 3.1M GL, 1M PLE	DONE
1b	migrate_direct.py	Legacy account → Customer (15,303)	DONE (via migrate_all.py)
1c	import_items.py	Legacy product → Item (833) + Item Groups (41)	DONE
2a	migrate_phase3.py	Subscription Plans + Subscriptions from active services (cats 4,9,17,21,32,33)	DONE
2b	import_missing_services.py	Services from excluded categories (non-standard)	DONE
2c	fix_subscription_details.py	Populate actual_price, custom_description, item_code, item_group, billing_frequency from hijack data	DONE
2d	fix_annual_billing_dates.py	Fix annual billing dates from legacy date_next_invoice	DONE
2e	fix_sub_address.py	Link Subscription → Service Location via delivery_id	DONE
3a	migrate_locations.py	Legacy delivery → Service Location + device → Service Equipment	DONE
3b	import_devices_and_enrich.py	Import missing equipment + enrich locations with fibre OLT data	DONE
3c	import_fibre_sql.py (/tmp on server)	Direct SQL: Add OLT fields to Service Equipment (4,930 records from fibre table)	DONE
4a	migrate_tickets.py	Legacy ticket → Issue (98,524) + ticket_msg → Communication	DONE
4b	fix_issue_cust2.py	Link Issue → Customer via legacy account_id	DONE
4c	fix_issue_owners.py	Fix Issue owner/assignee from legacy staff_id	DONE
4d	import_memos.py	Legacy account_memo → Comments on Customer	DONE
5a	import_customer_details.py	Add 12+ custom fields to Customer (billing, contact, commercial flags)	DONE
5b	import_services_and_enrich_customers.py	Enrich Customer with phone, email, stripe_id, PPA, notes	DONE
5c	cleanup_customer_status.py	Disable Customers with no active subscriptions	DONE
5d	import_terminated.py	Import terminated customers (status 3,4,5)	DONE
6a	import_employees.py	Legacy staff → Employee (155), maps group_ad → Department	DONE
6b	import_technicians.py	Link Employee → Dispatch Technician	DONE
6c	add_office_extension.py	Add office_extension field to Employee from legacy staff.ext	DONE
7a	setup_user_roles.py	Create Role Profiles + assign Users (admin, tech, support, etc.)	DONE
7b	setup_scheduler_toggle.py	API endpoints for scheduler control (scheduler_status, toggle)	DONE
8a	fix_customer_links.py	Fix customer references in SINV, Subscription, Issue (name → CUST-xxx)	DONE
8b	fix_invoice_outstanding.py	Correct outstanding_amount from legacy billing_status	DONE
8c	fix_reversals.py	Link credit invoices → originals via return_against + PLE	DONE
8d	fix_reversement.py	Delete incorrectly imported reversement Payment Entries	DONE
8e	fix_dates.py	Fix creation/modified timestamps from legacy unix timestamps	DONE
8f	fix_and_invoices.py	Fix Subscription.party + import recent invoices	DONE
8g	fix_no_rebate_discounts.py	Restore catalog prices on deliveries (rebate handling)	DONE
9a	rename_to_readable_ids.py	Rename hex IDs → human-readable (CUST-xxx, LOC-addr, EQ-dev)	DONE
9b	geocode_locations.py	Geocode Service Locations via rqa_addresses (Quebec address DB)	DONE
9c	update_item_descriptions.py	Update Item descriptions from legacy French translations	DONE
10a	fix_invoice_customer_names.py	Fix customer_name on 630K invoices + 344K payments (was CUST-xxx, now real names)	DONE
10b	import_invoice_notes.py	Import 580,949 legacy invoice.notes as Comments on Sales Invoice	DONE
10c	import_ticket_msgs.py	Import legacy ticket_msg as Comments on Issue (784,290 messages)	DONE
10d	update_assigned_staff.py	Update assigned_staff on Issues from legacy ticket.assign_to	DONE
10e	update_opened_by_staff.py	Update opened_by_staff on Issues from legacy ticket.open_by	DONE
11a	fix_income_accounts.py	Fix income_account on SI Items: SKU → product_cat.num_compte → ERPNext account (row-by-row ORM)	DONE
11b	fix_income_sql.py	Same fix as 11a but via bulk SQL temp table JOIN (faster)	DONE
11c	fix_gl_entries.py	Update GL entries to match updated SI Item income accounts (dominant account per invoice)	DONE
12a	create_portal_users.py	Create ~11,800 Website Users with legacy MD5 password hashes	DONE
12b	setup_portal_auth_bridge.py	Server Script: MD5 → pbkdf2 conversion on first login	DONE
12c	setup_invoice_print_format.py	Custom Print Format for Sales Invoice (Gigafibre branding)	DONE
12d	setup_subscription_api.py	Fix Subscription DocField restrictions for REST API	DONE
12e	add_missing_custom_fields.py	Add ~73 custom fields to ERPNext doctypes	DONE
12f	migrate_provisioning_data.py	Migrate WiFi + VoIP provisioning data into Service Equipment	DONE
12g	fix_olt_port_ip.py	Replace OLT port data with actual OLT IP from legacy fibre table	DONE

Earlier Migration Scripts (superseded by clean_reimport.py)

These were used during development. clean_reimport.py now handles all accounting in one pass.

Script	Purpose
migrate_all.py	Original phase orchestrator (7 phases). Superseded by clean_reimport.py for accounting
migrate_direct.py	Direct PG insert for Customers — now embedded in migrate_all.py
migrate_phase3.py	Subscription Plans + Subscriptions — still used standalone
migrate_phase5.py	Opening Balance via Journal Entry — superseded by PLE approach
migrate_missing_data.py	Populate custom fields from legacy — superseded by targeted fix scripts
migrate_users.py	Create ERPNext Users from legacy staff + Authentik SSO
import_invoices.py	Import invoices via Frappe ORM — superseded by clean_reimport.py (bulk SQL)
import_payments.py	Import payments via Frappe ORM — superseded by clean_reimport.py (bulk SQL)
reimport_subscriptions.py	Clean reimport of native Subscriptions from Service Subscription
reconcile_subscriptions.py	Reconcile Service Subscription vs Subscription for discrepancies

Rename Scripts (run once, in order)

Script	Purpose
rename_all_doctypes.py	Rename all doctype IDs to zero-padded 10-digit numeric format
rename_customers.py	Rename Customer IDs to CUST-{legacy_id}
rename_customers_c_prefix.py	Prepend C- to customer names (visual distinction)

Analysis/Debug Scripts (read-only, not part of migration)

Script	Purpose
analyze_pricing_cleanup.py	Pricing analysis: catalog vs hijack, rebate absorption
check_items.py	Check item_code/item_name values in Sales Invoice Items
check_naming.py	Verify invoice naming and legacy_invoice_id mapping
check_gl_dates.py	Check GL entry date distribution by income account
check_missing_cat26.py	Identify missing services in non-imported categories
debug_match.py	Debug credit/reversal matching logic for income accounts
debug_match2.py	Debug specific unupdated invoices and test match logic
debug_idx.py	Debug idx matching between legacy and ERPNext rows
explore_expro_payments.py	Compare legacy vs ERPNext payments for Expro Transit
explore_expro_services.py	Show active services for Expro Transit with pricing
simulate_payment_import.py	DRY RUN for Expro payments: timeline of invoice vs balance
import_expro_payments.py	Import missing Expro Transit payments (account 3673)

Helper Scripts (in parent /scripts/)

Script	Purpose
bulk_submit.py	Bulk submit drafted Sales Invoices (docstatus 0 → 1)
fix_ple_groupby.py	Fix PostgreSQL GROUP BY errors in PLE queries
fix_ple_postgres.sh	Shell script to apply PostgreSQL patches
server_bulk_submit.py	Server-side bulk submit with progress tracking

---

Legacy Tables → ERPNext Mapping (Complete)

Core Data

Legacy Table	ERPNext DocType	Script	Notes
account	Customer	migrate_direct.py	15,303 records
account (contact data)	Contact	import_services_and_enrich_customers.py	Phone, email, cell
delivery	Service Location	migrate_locations.py	Delivery addresses
service	Subscription	migrate_phase3.py + import_missing_services.py	Active services
product	Item	import_items.py	833 items, 41 groups
product (plans)	Subscription Plan	migrate_phase3.py	Pricing plans
device	Service Equipment	migrate_locations.py + import_devices_and_enrich.py	~7,241 devices
fibre + fibre_olt	Service Equipment (OLT fields)	import_fibre_sql.py	4,930 with OLT data

Accounting

Legacy Table	ERPNext DocType	Script	Notes
invoice	Sales Invoice	clean_reimport.py	629,935 records
invoice_item	Sales Invoice Item	clean_reimport.py	Line items
invoice_tax	Sales Taxes and Charges	clean_reimport.py	TPS/TVQ rows
payment	Payment Entry	clean_reimport.py	343,684 (deduped)
payment_item	Payment Entry Reference	clean_reimport.py	426,018 allocations
—	GL Entry	clean_reimport.py	3,135,184 generated
—	Payment Ledger Entry	clean_reimport.py	1,060,041 generated

Support & HR

Legacy Table	ERPNext DocType	Script	Notes
ticket	Issue	migrate_tickets.py	98,524 tickets
ticket_msg	Communication	migrate_tickets.py	Ticket replies
ticket_dept	Issue Type	migrate_tickets.py	Department → type
account_memo	Comment	import_memos.py	Internal notes
staff	Employee	import_employees.py	155 employees
staff	User	migrate_users.py	Active staff → ERPNext users

Service Enrichment Fields

Legacy Source	ERPNext Field	Script
service.hijack_price	Subscription.actual_price	fix_subscription_details.py
service.hijack_desc	Subscription.custom_description	fix_subscription_details.py
service.product_id → product.sku	Subscription.item_code	fix_subscription_details.py
service.date_next_invoice	Subscription.current_invoice_start	fix_annual_billing_dates.py
service.billing_frequency	Subscription.billing_frequency (M/A)	fix_subscription_details.py
fibre.sn	Service Equipment.serial_number	import_fibre_sql.py
fibre.info_connect → fibre_olt	Service Equipment.olt_name/ip/slot/port	import_fibre_sql.py
account.stripe_id	Customer.stripe_id	import_services_and_enrich_customers.py
account.password	Customer.legacy_password_hash	Not yet migrated

---

Remaining Migration Tasks

Task	Priority	Notes
Migrate legacy password hashes	P1	DONE — create_portal_users.py + setup_portal_auth_bridge.py
Customer portal users	P1	DONE — ~11,800 Website Users created with MD5 bridge
Income account mapping	P1	DONE — fix_income_sql.py + fix_gl_entries.py (2026-04-09)
Investigate 15 overpaid invoices ($3,695.77)	P2	Unmatched credit notes
QR code on invoice PDFs	P2	Stripe payment link for customer portal
Scheduler reactivation	P0	PAUSED — need Louis-Paul approval before enabling
Password reset campaign	P3	Email/SMS to portal users who haven't logged in