gigafibre-fsm/scripts/migration/MIGRATION_MAP.md

# Legacy → ERPNext Migration Map

## Overview

Migration from legacy PHP/MariaDB billing system (`gestionclient`) to ERPNext v16 on PostgreSQL.
- **Source**: MariaDB in Docker container `legacy-db` on `96.125.196.67` (port 3307 on host), database `gestionclient`
- **Target**: ERPNext at `erp.gigafibre.ca`, company **TARGO**, currency **CAD**
- **Scope**: All historical data (no date cutoff)
- **Method**: Bulk SQL INSERT (bypasses Frappe ORM for speed — 4 min vs ~120 hours)

---

## Key Accounting Problems Solved

The legacy system had several non-standard practices that broke standard accounting. Here is what was fixed during migration:

| # | Legacy Problem | Impact | ERPNext Solution |
|---|---------------|--------|------------------|
| 1 | **Credit notes not linked to original invoices** — negative invoices created with no reference back to the invoice they cancel | No audit trail; credits appear as free-floating | 3 matching mechanisms reconstruct the links (16,830 credit notes linked via `return_against`) |
| 2 | **Fake "reversement" payments** — internal bookkeeping entries recorded as real payments when cancelling invoices | $955K phantom overpayment | Excluded from import; replaced by proper credit note allocation |
| 3 | **Duplicate payments from portal** — slow legacy backend causes same credit card charge to be submitted twice (same Stripe reference) | Invoices appear double-paid; bank balance overstated | Deduplicated by `(account_id, reference)` — 178,730 duplicates removed |
| 4 | **Invoices both paid and reversed** — customer pays, then invoice also gets a credit note reversal | Invoice shows negative outstanding (overpaid) | Extra reversal entries delinked; invoice marked as settled |
| 5 | **Tax-inclusive totals** — legacy stores total with tax included, no separate net amount | ERPNext needs both net and gross | Tax amounts back-calculated from `invoice_tax` table |
| 6 | **No due dates** — most invoices have no due date | Cannot determine overdue status | `posting_date` used as fallback |
| 7 | **All revenue in one GL account** — initial import put all invoice income into generic `Autres produits d'exploitation - T` | No revenue breakdown by service type; P&L unusable for analysis | Post-import fix maps SKU → `product_cat.num_compte` → ERPNext named account. GL entries updated to dominant account per invoice. Scripts: `fix_income_accounts.py` (row-by-row), `fix_income_sql.py` (bulk via temp table), `fix_gl_entries.py` (GL-only pass) |

---

## Glossary — Internal Prefixes

These prefixes are used in ERPNext document names to identify records created during migration:

| Prefix | Stands for | Description |
|--------|-----------|-------------|
| `SINV-` | **S**ales **INV**oice | Invoice document (e.g., `SINV-638567`) |
| `PE-` | **P**ayment **E**ntry | Payment received from customer |
| `PER-` | **P**ayment **E**ntry **R**eference | Allocation of a payment to a specific invoice |
| `SII-` | **S**ales **I**nvoice **I**tem | Line item on an invoice |
| `stc-tps-` | **S**ales **T**ax **C**harge — TPS | GST tax row (5%) |
| `stc-tvq-` | **S**ales **T**ax **C**harge — TVQ | QST tax row (9.975%) |
| `ple-` | **P**ayment **L**edger **E**ntry | Tracks what's owed per invoice (ERPNext outstanding system) |
| `plc-` | **PL**E — **C**redit allocation | Credit note reducing a target invoice's balance |
| `plr-` | **PL**E — **R**eversal allocation | Reversal (from invoice notes) reducing a target invoice's balance |
| `gir-` | **G**L — **I**nvoice **R**eceivable | GL entry: debit to Accounts Receivable |
| `gii-` | **G**L — **I**nvoice **I**ncome | GL entry: credit to Revenue |
| `glt-` | **G**L — **T**PS | GL entry: credit to TPS (GST) liability |
| `glq-` | **G**L — TV**Q** | GL entry: credit to TVQ (QST) liability |
| `gpb-` | **G**L — **P**ayment **B**ank | GL entry: debit to Bank |
| `gpr-` | **G**L — **P**ayment **R**eceivable | GL entry: credit to Accounts Receivable |

---

## Data Flow Diagram

```
┌─────────────────────────────────────────────────────────────────────┐
│                        LEGACY (MariaDB)                             │
│                                                                     │
│  account ──────────────────────────────────┐                        │
│  invoice ──┬── invoice_item                │                        │
│            └── invoice_tax (TPS/TVQ rows)  │                        │
│  payment ──┬── payment_item (allocations)  │                        │
│            └── type='credit' (memo→#NNN)   │                        │
└────────────────────────┬───────────────────┘                        │
                         │                                            │
                         ▼                                            │
┌─────────────────────────────────────────────────────────────────────┐
│                        ERPNext (PostgreSQL)                          │
│                                                                     │
│  ┌──────────┐    ┌─────────────────┐    ┌──────────────────────┐   │
│  │ Customer │◄───│ Sales Invoice   │───►│ GL Entry (4 per inv) │   │
│  │          │    │  ├─ SI Item     │    │  gir- receivable     │   │
│  │          │    │  ├─ SI Tax(TPS) │    │  gii- income         │   │
│  │          │    │  └─ SI Tax(TVQ) │    │  glt- TPS            │   │
│  │          │    │                 │    │  glq- TVQ             │   │
│  │          │    │                 │───►│ PLE (1 per invoice)   │   │
│  │          │    │                 │    │  ple-SINV-            │   │
│  │          │    └─────────────────┘    └──────────────────────┘   │
│  │          │                                                       │
│  │          │    ┌─────────────────┐    ┌──────────────────────┐   │
│  │          │◄───│ Payment Entry   │───►│ GL Entry (2 per pmt) │   │
│  │          │    │  └─ PE Ref      │    │  gpb- bank           │   │
│  │          │    │    (allocations)│    │  gpr- receivable     │   │
│  │          │    │                 │───►│ PLE (1 per alloc)    │   │
│  │          │    │                 │    │  ple-PER-            │   │
│  └──────────┘    └─────────────────┘    └──────────────────────┘   │
│                                                                     │
│  Credit Allocations ──────────────────►  PLE (plc-)                │
│  (return inv → target inv)              against_voucher = target    │
└─────────────────────────────────────────────────────────────────────┘
```

---

## Entity Relationships (FK Map)

Every entity references others by `name` (ERPNext primary key). The rename scripts update all these FKs in a single pass.

```
Customer (C-{legacy_account_id})
├── Sales Invoice.customer ─────────────► Customer.name
│   ├── SI Item.parent ─────────────────► Sales Invoice.name
│   ├── SI Tax.parent ──────────────────► Sales Invoice.name
│   ├── GL Entry.voucher_no ────────────► Sales Invoice.name  (4 per invoice)
│   ├── PLE.voucher_no ─────────────────► Sales Invoice.name  (1 per invoice)
│   ├── PLE.against_voucher_no ─────────► Sales Invoice.name  (from payments/credits)
│   ├── Payment Entry Reference.reference_name ► Sales Invoice.name
│   ├── Comment.reference_name ─────────► Sales Invoice.name  (invoice notes)
│   └── return_against ─────────────────► Sales Invoice.name  (credit note → original)
│
├── Payment Entry.party ────────────────► Customer.name
│   ├── PE Reference.parent ────────────► Payment Entry.name
│   ├── GL Entry.voucher_no ────────────► Payment Entry.name  (2 per payment)
│   ├── PLE.voucher_no ─────────────────► Payment Entry.name  (1 per allocation)
│   └── Comment.reference_name ─────────► Payment Entry.name
│
├── Subscription.party ─────────────────► Customer.name
│   ├── Subscription Plan Detail.parent ► Subscription.name
│   └── Subscription.service_location ──► Service Location.name
│
├── Service Subscription.customer ──────► Customer.name
│   └── Service Subscription.service_location ► Service Location.name
│
├── Issue.customer ─────────────────────► Customer.name
│   ├── Communication.reference_name ───► Issue.name
│   ├── Comment.reference_name ─────────► Issue.name  (ticket messages)
│   ├── Dispatch Job.issue ─────────────► Issue.name
│   └── Issue.service_location ─────────► Service Location.name
│
├── Comment.reference_name ─────────────► Customer.name  (memos)
│
├── Service Location ───────────────────► (linked via Subscription/Issue)
│   └── Service Equipment.service_location ► Service Location.name
│
└── Employee ───────────────────────────► (linked via Dispatch Technician)
    └── Dispatch Technician.employee ───► Employee.name
```

### GL Entry ↔ PLE Linking

```
Sales Invoice SINV-{id}
  ├── GL: gir-SINV-{id}  → Receivable  (debit grand_total)
  ├── GL: gii-SINV-{id}  → Income*     (credit net_total)
  ├── GL: glt-SINV-{id}  → TPS         (credit tps)
  ├── GL: glq-SINV-{id}  → TVQ         (credit tvq)
  └── PLE: ple-SINV-{id} → voucher_no=SINV-{id}, against_voucher_no=SINV-{id}

Payment Entry PE-{id} allocated to SINV-{target}
  ├── GL: gpb-PE-{id}    → Bank        (debit paid_amount)
  ├── GL: gpr-PE-{id}    → Receivable  (credit paid_amount)
  └── PLE: ple-PER-{id}-{idx} → voucher_no=PE-{id}, against_voucher_no=SINV-{target}

Credit Note SINV-{credit} → SINV-{target}
  └── PLE: plc-{serial}  → voucher_no=SINV-{credit}, against_voucher_no=SINV-{target}
```

*Income account = SKU-mapped account from `product_cat.num_compte` (dominant per invoice)

---

## Naming Series & New ID Allocation

Migrated records use legacy IDs. New records use autoincrement series set **above** the max legacy ID to avoid collisions.

| DocType | Legacy Format | Zero-padded | New Format (post-migration) | Series Counter |
|---------|--------------|-------------|----------------------------|----------------|
| Customer | `C-{legacy_account_id}` | `C-{id:014d}` | `C-10000000034941+` | `C-` → 10000000034940 |
| Sales Invoice | `SINV-{legacy_id}` | `SINV-{id:010d}` | `SINV-0000100001+` | `SINV-` → 100000 |
| Payment Entry | `PE-{legacy_id}` | `PE-{id:010d}` | `PE-0000100001+` | `PE-` → 100000 |
| Issue | `ISS-{legacy_ticket_id}` | `ISS-{id:010d}` | `ISS-0000250001+` | `ISS-` → 250000 |
| Service Location | `LOC-{legacy_delivery_id}` | `LOC-{id:010d}` | `LOC-0000100001+` | `LOC-` → 100000 |
| Service Equipment | `EQP-{legacy_device_id}` | `EQP-{id:010d}` | `EQP-0000100001+` | `EQP-` → 100000 |
| Service Subscription | `SUB-{legacy_service_id}` | `SUB-{id:010d}` | `SUB-0000100001+` | `SUB-` → 100000 |
| Subscription (native) | `ASUB-{legacy_service_id}` | `ASUB-{id:010d}` | `ASUB-0000100001+` | `ASUB-` → 100000 |
| Employee | `HR-EMP-{seq}` | — | `HR-EMP-{N+1}+` | `HR-EMP-` → max legacy |
| SI Item | `SII-{inv_id}-{idx}` | — | (child, no series) | — |
| PE Reference | `PER-{pmt_id}-{idx}` | — | (child, no series) | — |
| GL Entry | `gir-/gii-/glt-/glq-/gpb-/gpr-` | — | (child, no series) | — |
| PLE | `ple-/plc-/plr-` | — | (child, no series) | — |

### Rename Process (rename_all_doctypes.py)

Two-phase rename to avoid PK collisions:
1. **Phase A**: `SINV-638567` → `_TEMP_SINV-0000638567`
2. **Phase B**: `_TEMP_SINV-0000638567` → `SINV-0000638567`
3. **FK cascade**: All child tables updated via temp mapping table + `UPDATE ... FROM` JOIN before rename

FK tables updated per doctype:
- **Sales Invoice** → SI Item, SI Tax, GL Entry, PLE, PE Reference, Comment, Version, Dynamic Link
- **Payment Entry** → PE Reference, GL Entry, PLE, Comment, Version, Dynamic Link
- **Issue** → Dispatch Job, Comment, Version, Communication, Dynamic Link
- **Service Location** → Service Subscription, Subscription, Service Equipment, Issue, Dispatch Job
- **Service Equipment** → Comment, Dynamic Link
- **Service/Native Subscription** → Subscription Plan Detail, Comment, Dynamic Link

---

## Entity Mapping

### Customer

```
Legacy (account)              ERPNext (Customer)
─────────────────             ──────────────────
id                      →     legacy_account_id
first_name + last_name  →     customer_name
email                   →     (Contact)
                               name = CUST-{hash}
```

### Sales Invoice

```
Legacy (invoice)              ERPNext (Sales Invoice)
─────────────────             ──────────────────────
id                      →     legacy_invoice_id
                               name = SINV-{legacy_id}
                               Example: 638567 → SINV-638567
account_id              →     customer (via account→Customer map)
total_amt               →     grand_total (TAX-INCLUSIVE)
total_amt - tps - tvq   →     net_total
date_orig (unix ts)     →     posting_date (YYYY-MM-DD)
total_amt < 0           →     is_return = 1
billed_amt              →     (not stored — derived from PLE)
                               outstanding_amount = SUM(PLE against this inv)
```

**Key**: Legacy `total_amt` is TAX-INCLUSIVE. ERPNext stores both `grand_total` (with tax) and `net_total` (without tax).

### Invoice Items

```
Legacy (invoice_item)         ERPNext (Sales Invoice Item)
─────────────────────         ──────────────────────────
invoice_id              →     parent = SINV name
product_name            →     item_name, description
quantity                →     qty
unitary_price           →     rate
quantity × unitary_price →    amount
                               name = SII-{legacy_id}-{idx}
                               item_code = 'SVC'
                               income_account = mapped from SKU → product_cat.num_compte → ERPNext account
                                 (fallback: "Autres produits d'exploitation - T" if SKU unmapped)
```

### Invoice Taxes

```
Legacy (invoice_tax)          ERPNext (Sales Taxes and Charges)
────────────────────          ─────────────────────────────────
invoice_id              →     parent = SINV name
tax_name = 'TPS'        →     description = 'TPS à payer - T'
  amount                →       tax_amount (5% GST, #834975559RT0001)
  tax_rate = 0.05       →       rate = 5.0
tax_name = 'TVQ'        →     description = 'TVQ à payer - T'
  amount                →       tax_amount (9.975% QST, #1213765929TQ0001)
  tax_rate = 0.09975    →       rate = 9.975
                               name = stc-tps-{legacy_id} / stc-tvq-{legacy_id}
```

**Note**: Legacy stores TPS and TVQ as **separate rows** per invoice. ERPNext also uses separate rows (idx=1 for TPS, idx=2 for TVQ).

### Payment Entry

```
Legacy (payment)              ERPNext (Payment Entry)
────────────────              ──────────────────────
id                      →     legacy_payment_id
                               name = PE-{legacy_id}
account_id              →     party (via account→Customer map)
amount                  →     paid_amount = received_amount
date_orig (unix ts)     →     posting_date
memo                    →     remarks
type != 'credit'        →     (only non-credit payments imported as PE)
                               payment_type = 'Receive'
                               paid_from = 'Comptes clients - T'
                               paid_to = 'Banque - T'
```

### Payment Allocations

```
Legacy (payment_item)         ERPNext (Payment Entry Reference)
─────────────────────         ───────────────────────────────
payment_id              →     parent = PE name
invoice_id              →     reference_name = SINV name
amount                  →     allocated_amount
                               name = PER-{legacy_pmt_id}-{idx}
                               reference_doctype = 'Sales Invoice'
```

### Credit Notes (Return Invoices)

```
Legacy — Three linking mechanisms:

  1. Credit payment:
     payment (type='credit', memo='credit created by invoice #NNN')
       → payment_item.invoice_id = target invoice

  2. Invoice notes:
     invoice.notes = 'Renversement de la facture #NNN'
       → #NNN = target legacy invoice ID

  3. Reversement payment memo:
     payment (type='reversement', memo='create by invoice #CREDIT for invoice #TARGET')
       → payment_item.invoice_id = target invoice

ERPNext:
  Sales Invoice (is_return=1, return_against=target SINV)
    → PLE (plc-{id})   — from credit payments (mechanism 1)
    → PLE (plr-{id})   — from reversal notes + reversement memos (mechanisms 2+3)
        voucher_no = source return SINV
        against_voucher_no = target SINV
        amount = credit invoice's grand_total (negative)
```

---

## Accounting Entries (Double-Entry)

### Per Sales Invoice (4 GL entries)

```
                              Debit         Credit
                              ─────         ──────
Comptes clients - T (AR)      grand_total
Income account* (Revenue)                   net_total
TPS à payer - T (Tax)                       tps_amount
TVQ à payer - T (Tax)                       tvq_amount

* Income account = dominant SKU-mapped account for the invoice
  (from product_cat.num_compte). Falls back to "Autres produits d'exploitation - T".
  Note: GL uses ONE income account per invoice (the dominant/highest-amount one).
  Individual SI Items keep their per-SKU accounts for drill-down reporting.
                              ─────────     ─────────
                              grand_total = grand_total  ✓
```

For **return invoices** (negative amounts), debit/credit are swapped.

### Per Payment Entry (2 GL entries)

```
                              Debit         Credit
                              ─────         ──────
Banque - T (Bank)             paid_amount
Comptes clients - T (AR)                    paid_amount
```

### Payment Ledger Entry (PLE) — Outstanding Tracking

```
Type                    amount              against_voucher
────                    ──────              ───────────────
Invoice posting         +grand_total        self (SINV)
Payment allocation      -allocated_amount   target SINV
Credit allocation       -credit_amount      target SINV
Unallocated payment     -paid_amount        self (PE)

Outstanding = SUM(PLE.amount WHERE against_voucher = this invoice)
```

---

## ERPNext Chart of Accounts

```
TARGO (Company, abbr: T)
├── Comptes clients - T          Receivable    (debit_to for all invoices)
├── Banque - T                   Bank          (paid_to for all payments)
├── Income accounts (mapped from legacy product_cat.num_compte):
│   ├── Autres produits d'exploitation - T  Income (fallback for unmapped SKUs)
│   ├── [4020] Mensualite fibre - T         Income (fibre monthly)
│   ├── [4xxx] Other mapped accounts...     Income (per product category)
│   └── (run fix_income_sql.py to see full distribution)
├── TPS à payer - T              Liability/Tax (5% GST)
└── TVQ à payer - T              Liability/Tax (9.975% QST)
```

---

## Naming Conventions

### Migrated Data (Legacy IDs)

| Entity | Pattern | Example |
|--------|---------|---------|
| Sales Invoice | `SINV-{legacy_id}` | `SINV-638567` |
| SI Item | `SII-{legacy_id}-{idx}` | `SII-638567-0` |
| SI Tax (TPS) | `stc-tps-{legacy_id}` | `stc-tps-638567` |
| SI Tax (TVQ) | `stc-tvq-{legacy_id}` | `stc-tvq-638567` |
| Payment Entry | `PE-{legacy_id}` | `PE-76531` |
| PE Reference | `PER-{legacy_id}-{idx}` | `PER-76531-0` |
| GL (inv receivable) | `gir-SINV-{id}` | `gir-SINV-638567` |
| GL (inv income) | `gii-SINV-{id}` | `gii-SINV-638567` |
| GL (inv TPS) | `glt-SINV-{id}` | `glt-SINV-638567` |
| GL (inv TVQ) | `glq-SINV-{id}` | `glq-SINV-638567` |
| GL (pmt bank) | `gpb-PE-{id}` | `gpb-PE-76531` |
| GL (pmt receivable) | `gpr-PE-{id}` | `gpr-PE-76531` |
| PLE (invoice) | `ple-SINV-{id}` | `ple-SINV-638567` |
| PLE (pmt alloc) | `ple-PER-{id}-{idx}` | `ple-PER-76531-0` |
| PLE (unallocated) | `ple-PE-{id}` | `ple-PE-76531` |
| PLE (credit alloc) | `plc-{serial}` | `plc-1234` |
| PLE (reversal alloc) | `plr-{serial}` | `plr-567` |

### Post-Migration (New Documents)

| Entity | Pattern | Example |
|--------|---------|---------|
| Sales Invoice | `SINV-YYYY-NNNNN` | `SINV-2026-700001` |
| Payment Entry | ERPNext autoname | `PE-2026-00001` |

The different naming patterns between migrated and new documents ensure no collision if a reimport is needed after new documents have been created.

---

## Legacy Database Schema

### `invoice`
| Column | Type | Notes |
|--------|------|-------|
| `id` | bigint PK | → `legacy_invoice_id` in ERPNext |
| `date_orig` | bigint | UNIX timestamp |
| `account_id` | bigint FK | → `account.id` |
| `total_amt` | double(20,2) | **TAX-INCLUSIVE** |
| `billed_amt` | double(20,2) | Amount paid |
| `due_date` | bigint | UNIX timestamp |
| `correction` | tinyint | 1 = correction invoice |
| `notes` | mediumtext | |

### `invoice_item`
| Column | Type | Notes |
|--------|------|-------|
| `id` | bigint PK | |
| `invoice_id` | bigint FK | → `invoice.id` |
| `product_name` | varchar(512) | NOT `description` |
| `quantity` | double | NOT `qty` |
| `unitary_price` | double | NOT `price` |
| `sku` | varchar(128) | |

### `invoice_tax`
| Column | Type | Notes |
|--------|------|-------|
| `id` | bigint PK | |
| `invoice_id` | bigint FK | → `invoice.id` |
| `tax_name` | varchar(128) | `'TPS'` or `'TVQ'` — **separate rows, not columns** |
| `tax_rate` | double | 0.05 or 0.09975 |
| `amount` | double(20,2) | Tax amount for this type |

### `payment`
| Column | Type | Notes |
|--------|------|-------|
| `id` | bigint PK | |
| `account_id` | bigint FK | → `account.id` |
| `date_orig` | bigint | UNIX timestamp |
| `amount` | double | |
| `type` | varchar(25) | `'payment'`, `'credit'`, etc. |
| `memo` | varchar(512) | For credit: `"credit created by invoice #NNN"` |
| `reference` | varchar(128) | Stripe/processor transaction ID (e.g., `pi_3Sad...`) |

### `payment_item`
| Column | Type | Notes |
|--------|------|-------|
| `id` | bigint PK | |
| `payment_id` | bigint FK | → `payment.id` |
| `invoice_id` | bigint FK | → `invoice.id` |
| `amount` | double | Allocated to this invoice |

### `account` (customer)
| Column | Type | Notes |
|--------|------|-------|
| `id` | bigint PK | → `legacy_account_id` in Customer |
| `first_name` | varchar | |
| `last_name` | varchar | |
| `email` | varchar | |
| `company_name` | varchar | |

---

## Fiscal Year

Canadian fiscal year: **July 1 – June 30**

```
posting_date month >= 7  →  fiscal_year = "YYYY-(YYYY+1)"   e.g. "2025-2026"
posting_date month <  7  →  fiscal_year = "(YYYY-1)-YYYY"    e.g. "2024-2025"
```

---

## Legacy Non-Compliance Changelog

The legacy PHP/MariaDB system uses several non-standard accounting practices that were corrected during migration to ERPNext.

### 1. Reversal Invoices — No Credit Link

**Legacy behavior**: To cancel an invoice, the system creates a negative invoice (same amount, opposite sign) on the same account. It sets `billed_amt = total_amt` on both invoices to mark them as "settled," but **does not create a credit payment or any explicit reference between them**. There is no `return_against`, no `credit_of`, and no payment linking the two.

**Problem**: With no link, there's no audit trail showing which invoice was cancelled by which credit note. The negative invoices appear as unallocated credits, creating phantom overpayment.

**ERPNext fix**: Three matching mechanisms were developed to reconstruct the links:

1. **Credit payment allocations** (15,694 matches) — `payment.type='credit'` with `memo='credit created by invoice #NNN'` → `payment_item.invoice_id` points to target. Zero overlap with mechanism 2.
2. **Invoice notes field** (5,095 matches) — `invoice.notes` contains `'Renversement de la facture #NNN'` referencing the target legacy invoice ID.
3. **Reversement payment memos** (143 unique matches) — `payment.type='reversement'` with `memo='create by invoice #CREDIT for invoice #TARGET'` → `payment_item.invoice_id` = target. Only applied to invoices not already matched by mechanisms 1-2.

Total: **16,830 credit notes linked** via `return_against` + PLE allocation entries.

### 2. Fake "Reversement" Payments

**Legacy behavior**: When a reversal invoice is created, the system also generates a `payment.type = 'reversement'` record allocated to the original invoice. This is **not a real customer payment** — it's an internal bookkeeping entry that marks the original invoice as "paid."

**Problem**: If imported as a real Payment Entry in ERPNext, the original invoice appears double-settled (once by the fake payment, once by the credit note PLE), creating ~$955K in phantom overpayment.

**ERPNext fix**: Excluded all ~5,000 "reversement" payments from import. These are not real financial transactions — the credit note relationship (mechanism 1/2/3 above) replaces them. The reversement memos are still parsed for matching purposes (mechanism 3).

### 3. Payment Types in Legacy

| Legacy `payment.type` | Real Payment? | ERPNext Treatment |
|------------------------|---------------|-------------------|
| `paiement direct` | Yes | Payment Entry |
| `carte credit` | Yes | Payment Entry |
| `ppa` | Yes (pre-authorized) | Payment Entry |
| `credit` | No — credit note allocation | PLE (plc-) only, memo parsed for matching |
| `cheque` | Yes | Payment Entry |
| `reversement` | **No — fake reversal payment** | **Excluded from PE**, memo parsed for matching |
| `comptant` | Yes | Payment Entry |
| `credit targo` | No — internal credit | **Excluded** |

### 4. Duplicate Payments (Double Form Submission)

**Legacy behavior**: The customer portal sometimes processes the same credit card charge twice when the legacy PHP backend is slow to respond. The submit handler fires again, creating a duplicate `payment` record. Both payments have the **same `reference`** field (Stripe transaction ID, e.g., `pi_3SadKtAU3HUVhnM10KjBOebK`) but different `payment.id` values.

**Problem**: Both payments are allocated to the same invoice via `payment_item`, causing the invoice to appear overpaid. The GL bank balance is also overstated.

**ERPNext fix**: Deduplicate payments by `(account_id, reference)` during import — keep only the first payment per unique reference. **178,730 duplicate payments** removed (from 522K raw to 343K unique).

### 5. Tax-Inclusive Totals

**Legacy behavior**: `invoice.total_amt` is **tax-inclusive** (includes TPS 5% + TVQ 9.975%). Individual tax amounts are stored in separate `invoice_tax` rows, not as columns on the invoice.

**ERPNext**: Stores both `grand_total` (with tax) and `net_total` (without tax). Tax amounts are back-calculated from the `invoice_tax` table. When no tax record exists, taxes are estimated at 14.975% of total.

### 6. No Due Date Tracking

**Legacy behavior**: Many invoices have `due_date = 0` (UNIX epoch) or NULL. The system doesn't enforce payment terms.

**ERPNext fix**: Uses `posting_date` as fallback when `due_date` is missing or invalid (before 2000).

### 7. Credit Notes with Both Payment and Reversal PLE

**Legacy behavior**: Some credit invoices have BOTH a `credit` payment allocation (mechanism 1) AND a separate reversal invoice reference (mechanism 2 or 3). If both are processed, the target invoice gets double-credited.

**Problem**: Creates phantom overpayment (negative outstanding) on target invoices. Originally caused $975K in overpaid balances.

**ERPNext fix**: Apply matching mechanisms in priority order. Track `already_linked` set — if a credit invoice was matched by credit payment (mechanism 1), skip it for reversal notes (mechanism 2) and reversement memos (mechanism 3). This prevents double-counting.

### 8. Orphaned Invoices (No Customer Account)

**Legacy behavior**: 9 invoices reference `account_id` values that don't exist in the `account` table (deleted customers or test data).

**ERPNext fix**: Skipped during import. Logged as unmapped: IDs 2712, 5627, 15119, 15234, 195096, 216370, 272051, 277963, 308988.

### 9. Invoice Naming for CRA Compliance

**Legacy behavior**: Invoices have integer IDs with no prefix. No formal naming convention.

**Problem**: CRA requires sequential invoice numbering for audit trail. Hex-encoded IDs (e.g., `SINV-000009BE67`) are not human-readable and can cause hash collisions on child tables.

**ERPNext fix**: Use `SINV-{legacy_id}` format (e.g., `SINV-638567`). Legacy IDs are already sequential integers. Post-migration invoices will use `SINV-YYYY-NNNNN` format (starting at `SINV-2026-700001`) to avoid collision with legacy IDs on reimport.

### 10. Customer Portal Credentials

**Legacy behavior**: 15,305 customer accounts with username/password (hashed, likely MD5/SHA1). Password reset tokens in `client_pwd` table (9,687 tokens). Stripe customer IDs stored in `account.stripe_id`.

**ERPNext fix**: Legacy password hashes are incompatible with Frappe's bcrypt auth. After migration, ERPNext Website Users will be created with customer emails, and bulk password reset emails will be sent. Stripe IDs will be linked to ERPNext's payment integration.

---

## Migration Results (2026-03-29, full historical import)

**Migration time: ~16 minutes** (966 seconds) for full reimport including cleanup, data load from legacy MariaDB, bulk SQL inserts, GL entries, PLE entries, and outstanding recalculation. This compares to an estimated ~120 hours if using Frappe ORM.

| Metric | Count |
|--------|-------|
| Sales Invoices | 629,935 |
| Payment Entries | 343,684 (excl. reversement + credit + credit targo) |
| Payment References | 426,018 |
| GL Entries | 3,135,184 |
| PLE Entries | 1,060,041 |
| **GL Balance** | **$130,120,226.76 = $130,120,226.76 (diff $0.00)** |

### Credit Note Matching
| Mechanism | Matches |
|-----------|---------|
| Credit payment allocations (plc-) | 15,508 |
| Reversal notes + reversement memos (plr-) | 5,238 |
| **Total linked** | **20,746** |

### Invoice Status Breakdown
| Status | Count |
|--------|-------|
| Paid | 415,861 |
| Overdue | 197,190 |
| Return | 16,868 |
| Credit Note Issued | 15 |
| Unpaid | 1 |

### Outstanding
- **Owed: $20,500,562.62**
- **Overpaid: -$3,695.77** (15 invoices — unmatched credit notes with no reference to original in any of the 3 mechanisms)

### Payment Deduplication
- Raw payments from legacy: 522,416
- After dedup by `(account_id, reference)`: 343,686
- Duplicates removed: **178,730**

---

## How to Re-run

The migration is **idempotent** — the script deletes all existing data and reimports from scratch.

```bash
# From facturation.targo.ca:
ssh root@96.125.196.67 'docker cp /path/to/clean_reimport.py erpnext-backend-1:/home/frappe/frappe-bench/clean_reimport.py'
ssh root@96.125.196.67 'docker exec erpnext-backend-1 /home/frappe/frappe-bench/env/bin/python /home/frappe/frappe-bench/clean_reimport.py'
```

**Important**: Update the password in the script before running (redacted as `*******` in git).

---

## Complete Migration Phases & Script Inventory

### Execution Order

The migration is split into phases. Each phase can be re-run independently (most scripts are idempotent or nuke-and-reimport).

| Phase | Script | Description | Status |
|-------|--------|-------------|--------|
| **0** | `nuke_data.py` | Delete all migrated data except Users, Items, Plans | Run before reimport |
| **1a** | `clean_reimport.py` | **Master accounting import**: 630K invoices, 344K payments, 3.1M GL, 1M PLE | **DONE** |
| **1b** | `migrate_direct.py` | Legacy account → Customer (15,303) | **DONE** (via migrate_all.py) |
| **1c** | `import_items.py` | Legacy product → Item (833) + Item Groups (41) | **DONE** |
| **2a** | `migrate_phase3.py` | Subscription Plans + Subscriptions from active services (cats 4,9,17,21,32,33) | **DONE** |
| **2b** | `import_missing_services.py` | Services from excluded categories (non-standard) | **DONE** |
| **2c** | `fix_subscription_details.py` | Populate actual_price, custom_description, item_code, item_group, billing_frequency from hijack data | **DONE** |
| **2d** | `fix_annual_billing_dates.py` | Fix annual billing dates from legacy date_next_invoice | **DONE** |
| **2e** | `fix_sub_address.py` | Link Subscription → Service Location via delivery_id | **DONE** |
| **3a** | `migrate_locations.py` | Legacy delivery → Service Location + device → Service Equipment | **DONE** |
| **3b** | `import_devices_and_enrich.py` | Import missing equipment + enrich locations with fibre OLT data | **DONE** |
| **3c** | `import_fibre_sql.py` (/tmp on server) | Direct SQL: Add OLT fields to Service Equipment (4,930 records from fibre table) | **DONE** |
| **4a** | `migrate_tickets.py` | Legacy ticket → Issue (98,524) + ticket_msg → Communication | **DONE** |
| **4b** | `fix_issue_cust2.py` | Link Issue → Customer via legacy account_id | **DONE** |
| **4c** | `fix_issue_owners.py` | Fix Issue owner/assignee from legacy staff_id | **DONE** |
| **4d** | `import_memos.py` | Legacy account_memo → Comments on Customer | **DONE** |
| **5a** | `import_customer_details.py` | Add 12+ custom fields to Customer (billing, contact, commercial flags) | **DONE** |
| **5b** | `import_services_and_enrich_customers.py` | Enrich Customer with phone, email, stripe_id, PPA, notes | **DONE** |
| **5c** | `cleanup_customer_status.py` | Disable Customers with no active subscriptions | **DONE** |
| **5d** | `import_terminated.py` | Import terminated customers (status 3,4,5) | **DONE** |
| **6a** | `import_employees.py` | Legacy staff → Employee (155), maps group_ad → Department | **DONE** |
| **6b** | `import_technicians.py` | Link Employee → Dispatch Technician | **DONE** |
| **6c** | `add_office_extension.py` | Add office_extension field to Employee from legacy staff.ext | **DONE** |
| **7a** | `setup_user_roles.py` | Create Role Profiles + assign Users (admin, tech, support, etc.) | **DONE** |
| **7b** | `setup_scheduler_toggle.py` | API endpoints for scheduler control (scheduler_status, toggle) | **DONE** |
| **8a** | `fix_customer_links.py` | Fix customer references in SINV, Subscription, Issue (name → CUST-xxx) | **DONE** |
| **8b** | `fix_invoice_outstanding.py` | Correct outstanding_amount from legacy billing_status | **DONE** |
| **8c** | `fix_reversals.py` | Link credit invoices → originals via return_against + PLE | **DONE** |
| **8d** | `fix_reversement.py` | Delete incorrectly imported reversement Payment Entries | **DONE** |
| **8e** | `fix_dates.py` | Fix creation/modified timestamps from legacy unix timestamps | **DONE** |
| **8f** | `fix_and_invoices.py` | Fix Subscription.party + import recent invoices | **DONE** |
| **8g** | `fix_no_rebate_discounts.py` | Restore catalog prices on deliveries (rebate handling) | **DONE** |
| **9a** | `rename_to_readable_ids.py` | Rename hex IDs → human-readable (CUST-xxx, LOC-addr, EQ-dev) | **DONE** |
| **9b** | `geocode_locations.py` | Geocode Service Locations via rqa_addresses (Quebec address DB) | **DONE** |
| **9c** | `update_item_descriptions.py` | Update Item descriptions from legacy French translations | **DONE** |
| **10a** | `fix_invoice_customer_names.py` | Fix customer_name on 630K invoices + 344K payments (was CUST-xxx, now real names) | **DONE** |
| **10b** | `import_invoice_notes.py` | Import 580,949 legacy invoice.notes as Comments on Sales Invoice | **DONE** |
| **10c** | `import_ticket_msgs.py` | Import legacy ticket_msg as Comments on Issue (784,290 messages) | **DONE** |
| **10d** | `update_assigned_staff.py` | Update assigned_staff on Issues from legacy ticket.assign_to | **DONE** |
| **10e** | `update_opened_by_staff.py` | Update opened_by_staff on Issues from legacy ticket.open_by | **DONE** |
| **11a** | `fix_income_accounts.py` | Fix income_account on SI Items: SKU → product_cat.num_compte → ERPNext account (row-by-row ORM) | **DONE** |
| **11b** | `fix_income_sql.py` | Same fix as 11a but via bulk SQL temp table JOIN (faster) | **DONE** |
| **11c** | `fix_gl_entries.py` | Update GL entries to match updated SI Item income accounts (dominant account per invoice) | **DONE** |
| **12a** | `create_portal_users.py` | Create ~11,800 Website Users with legacy MD5 password hashes | **DONE** |
| **12b** | `setup_portal_auth_bridge.py` | Server Script: MD5 → pbkdf2 conversion on first login | **DONE** |
| **12c** | `setup_invoice_print_format.py` | Custom Print Format for Sales Invoice (Gigafibre branding) | **DONE** |
| **12d** | `setup_subscription_api.py` | Fix Subscription DocField restrictions for REST API | **DONE** |
| **12e** | `add_missing_custom_fields.py` | Add ~73 custom fields to ERPNext doctypes | **DONE** |
| **12f** | `migrate_provisioning_data.py` | Migrate WiFi + VoIP provisioning data into Service Equipment | **DONE** |
| **12g** | `fix_olt_port_ip.py` | Replace OLT port data with actual OLT IP from legacy fibre table | **DONE** |

### Earlier Migration Scripts (superseded by clean_reimport.py)

These were used during development. `clean_reimport.py` now handles all accounting in one pass.

| Script | Purpose |
|--------|---------|
| `migrate_all.py` | Original phase orchestrator (7 phases). Superseded by clean_reimport.py for accounting |
| `migrate_direct.py` | Direct PG insert for Customers — now embedded in migrate_all.py |
| `migrate_phase3.py` | Subscription Plans + Subscriptions — still used standalone |
| `migrate_phase5.py` | Opening Balance via Journal Entry — superseded by PLE approach |
| `migrate_missing_data.py` | Populate custom fields from legacy — superseded by targeted fix scripts |
| `migrate_users.py` | Create ERPNext Users from legacy staff + Authentik SSO | 
| `import_invoices.py` | Import invoices via Frappe ORM — superseded by clean_reimport.py (bulk SQL) |
| `import_payments.py` | Import payments via Frappe ORM — superseded by clean_reimport.py (bulk SQL) |
| `reimport_subscriptions.py` | Clean reimport of native Subscriptions from Service Subscription |
| `reconcile_subscriptions.py` | Reconcile Service Subscription vs Subscription for discrepancies |

### Rename Scripts (run once, in order)

| Script | Purpose |
|--------|---------|
| `rename_all_doctypes.py` | Rename all doctype IDs to zero-padded 10-digit numeric format |
| `rename_customers.py` | Rename Customer IDs to CUST-{legacy_id} |
| `rename_customers_c_prefix.py` | Prepend C- to customer names (visual distinction) |

### Analysis/Debug Scripts (read-only, not part of migration)

| Script | Purpose |
|--------|---------|
| `analyze_pricing_cleanup.py` | Pricing analysis: catalog vs hijack, rebate absorption |
| `check_items.py` | Check item_code/item_name values in Sales Invoice Items |
| `check_naming.py` | Verify invoice naming and legacy_invoice_id mapping |
| `check_gl_dates.py` | Check GL entry date distribution by income account |
| `check_missing_cat26.py` | Identify missing services in non-imported categories |
| `debug_match.py` | Debug credit/reversal matching logic for income accounts |
| `debug_match2.py` | Debug specific unupdated invoices and test match logic |
| `debug_idx.py` | Debug idx matching between legacy and ERPNext rows |
| `explore_expro_payments.py` | Compare legacy vs ERPNext payments for Expro Transit |
| `explore_expro_services.py` | Show active services for Expro Transit with pricing |
| `simulate_payment_import.py` | DRY RUN for Expro payments: timeline of invoice vs balance |
| `import_expro_payments.py` | Import missing Expro Transit payments (account 3673) |

### Helper Scripts (in parent /scripts/)

| Script | Purpose |
|--------|---------|
| `bulk_submit.py` | Bulk submit drafted Sales Invoices (docstatus 0 → 1) |
| `fix_ple_groupby.py` | Fix PostgreSQL GROUP BY errors in PLE queries |
| `fix_ple_postgres.sh` | Shell script to apply PostgreSQL patches |
| `server_bulk_submit.py` | Server-side bulk submit with progress tracking |

---

## Legacy Tables → ERPNext Mapping (Complete)

### Core Data

| Legacy Table | ERPNext DocType | Script | Notes |
|---|---|---|---|
| `account` | Customer | migrate_direct.py | 15,303 records |
| `account` (contact data) | Contact | import_services_and_enrich_customers.py | Phone, email, cell |
| `delivery` | Service Location | migrate_locations.py | Delivery addresses |
| `service` | Subscription | migrate_phase3.py + import_missing_services.py | Active services |
| `product` | Item | import_items.py | 833 items, 41 groups |
| `product` (plans) | Subscription Plan | migrate_phase3.py | Pricing plans |
| `device` | Service Equipment | migrate_locations.py + import_devices_and_enrich.py | ~7,241 devices |
| `fibre` + `fibre_olt` | Service Equipment (OLT fields) | import_fibre_sql.py | 4,930 with OLT data |

### Accounting

| Legacy Table | ERPNext DocType | Script | Notes |
|---|---|---|---|
| `invoice` | Sales Invoice | clean_reimport.py | 629,935 records |
| `invoice_item` | Sales Invoice Item | clean_reimport.py | Line items |
| `invoice_tax` | Sales Taxes and Charges | clean_reimport.py | TPS/TVQ rows |
| `payment` | Payment Entry | clean_reimport.py | 343,684 (deduped) |
| `payment_item` | Payment Entry Reference | clean_reimport.py | 426,018 allocations |
| — | GL Entry | clean_reimport.py | 3,135,184 generated |
| — | Payment Ledger Entry | clean_reimport.py | 1,060,041 generated |

### Support & HR

| Legacy Table | ERPNext DocType | Script | Notes |
|---|---|---|---|
| `ticket` | Issue | migrate_tickets.py | 98,524 tickets |
| `ticket_msg` | Communication | migrate_tickets.py | Ticket replies |
| `ticket_dept` | Issue Type | migrate_tickets.py | Department → type |
| `account_memo` | Comment | import_memos.py | Internal notes |
| `staff` | Employee | import_employees.py | 155 employees |
| `staff` | User | migrate_users.py | Active staff → ERPNext users |

### Service Enrichment Fields

| Legacy Source | ERPNext Field | Script |
|---|---|---|
| `service.hijack_price` | Subscription.actual_price | fix_subscription_details.py |
| `service.hijack_desc` | Subscription.custom_description | fix_subscription_details.py |
| `service.product_id` → product.sku | Subscription.item_code | fix_subscription_details.py |
| `service.date_next_invoice` | Subscription.current_invoice_start | fix_annual_billing_dates.py |
| `service.billing_frequency` | Subscription.billing_frequency (M/A) | fix_subscription_details.py |
| `fibre.sn` | Service Equipment.serial_number | import_fibre_sql.py |
| `fibre.info_connect` → `fibre_olt` | Service Equipment.olt_name/ip/slot/port | import_fibre_sql.py |
| `account.stripe_id` | Customer.stripe_id | import_services_and_enrich_customers.py |
| `account.password` | Customer.legacy_password_hash | Not yet migrated |

---

## Remaining Migration Tasks

| Task | Priority | Notes |
|------|----------|-------|
| ~~Migrate legacy password hashes~~ | ~~P1~~ | **DONE** — `create_portal_users.py` + `setup_portal_auth_bridge.py` |
| ~~Customer portal users~~ | ~~P1~~ | **DONE** — ~11,800 Website Users created with MD5 bridge |
| ~~Income account mapping~~ | ~~P1~~ | **DONE** — `fix_income_sql.py` + `fix_gl_entries.py` (2026-04-09) |
| Investigate 15 overpaid invoices ($3,695.77) | P2 | Unmatched credit notes |
| QR code on invoice PDFs | P2 | Stripe payment link for customer portal |
| Scheduler reactivation | P0 | **PAUSED** — need Louis-Paul approval before enabling |
| Password reset campaign | P3 | Email/SMS to portal users who haven't logged in |