gigafibre-fsm/scripts/migration/MIGRATION_MAP.md

# Legacy → ERPNext Migration Map

## Overview

Migration from legacy PHP/MariaDB billing system (`gestionclient`) to ERPNext v16 on PostgreSQL.
- **Source**: MariaDB at `10.100.80.100`, database `gestionclient`
- **Target**: ERPNext at `erp.gigafibre.ca`, company **TARGO**, currency **CAD**
- **Scope**: All historical data (no date cutoff)
- **Method**: Bulk SQL INSERT (bypasses Frappe ORM for speed — 4 min vs ~120 hours)

---

## Key Accounting Problems Solved

The legacy system had several non-standard practices that broke standard accounting. Here is what was fixed during migration:

| # | Legacy Problem | Impact | ERPNext Solution |
|---|---------------|--------|------------------|
| 1 | **Credit notes not linked to original invoices** — negative invoices created with no reference back to the invoice they cancel | No audit trail; credits appear as free-floating | 3 matching mechanisms reconstruct the links (16,830 credit notes linked via `return_against`) |
| 2 | **Fake "reversement" payments** — internal bookkeeping entries recorded as real payments when cancelling invoices | $955K phantom overpayment | Excluded from import; replaced by proper credit note allocation |
| 3 | **Duplicate payments from portal** — slow legacy backend causes same credit card charge to be submitted twice (same Stripe reference) | Invoices appear double-paid; bank balance overstated | Deduplicated by `(account_id, reference)` — 178,730 duplicates removed |
| 4 | **Invoices both paid and reversed** — customer pays, then invoice also gets a credit note reversal | Invoice shows negative outstanding (overpaid) | Extra reversal entries delinked; invoice marked as settled |
| 5 | **Tax-inclusive totals** — legacy stores total with tax included, no separate net amount | ERPNext needs both net and gross | Tax amounts back-calculated from `invoice_tax` table |
| 6 | **No due dates** — most invoices have no due date | Cannot determine overdue status | `posting_date` used as fallback |

---

## Glossary — Internal Prefixes

These prefixes are used in ERPNext document names to identify records created during migration:

| Prefix | Stands for | Description |
|--------|-----------|-------------|
| `SINV-` | **S**ales **INV**oice | Invoice document (e.g., `SINV-638567`) |
| `PE-` | **P**ayment **E**ntry | Payment received from customer |
| `PER-` | **P**ayment **E**ntry **R**eference | Allocation of a payment to a specific invoice |
| `SII-` | **S**ales **I**nvoice **I**tem | Line item on an invoice |
| `stc-tps-` | **S**ales **T**ax **C**harge — TPS | GST tax row (5%) |
| `stc-tvq-` | **S**ales **T**ax **C**harge — TVQ | QST tax row (9.975%) |
| `ple-` | **P**ayment **L**edger **E**ntry | Tracks what's owed per invoice (ERPNext outstanding system) |
| `plc-` | **PL**E — **C**redit allocation | Credit note reducing a target invoice's balance |
| `plr-` | **PL**E — **R**eversal allocation | Reversal (from invoice notes) reducing a target invoice's balance |
| `gir-` | **G**L — **I**nvoice **R**eceivable | GL entry: debit to Accounts Receivable |
| `gii-` | **G**L — **I**nvoice **I**ncome | GL entry: credit to Revenue |
| `glt-` | **G**L — **T**PS | GL entry: credit to TPS (GST) liability |
| `glq-` | **G**L — TV**Q** | GL entry: credit to TVQ (QST) liability |
| `gpb-` | **G**L — **P**ayment **B**ank | GL entry: debit to Bank |
| `gpr-` | **G**L — **P**ayment **R**eceivable | GL entry: credit to Accounts Receivable |

---

## Data Flow Diagram

```
┌─────────────────────────────────────────────────────────────────────┐
│                        LEGACY (MariaDB)                             │
│                                                                     │
│  account ──────────────────────────────────┐                        │
│  invoice ──┬── invoice_item                │                        │
│            └── invoice_tax (TPS/TVQ rows)  │                        │
│  payment ──┬── payment_item (allocations)  │                        │
│            └── type='credit' (memo→#NNN)   │                        │
└────────────────────────┬───────────────────┘                        │
                         │                                            │
                         ▼                                            │
┌─────────────────────────────────────────────────────────────────────┐
│                        ERPNext (PostgreSQL)                          │
│                                                                     │
│  ┌──────────┐    ┌─────────────────┐    ┌──────────────────────┐   │
│  │ Customer │◄───│ Sales Invoice   │───►│ GL Entry (4 per inv) │   │
│  │          │    │  ├─ SI Item     │    │  gir- receivable     │   │
│  │          │    │  ├─ SI Tax(TPS) │    │  gii- income         │   │
│  │          │    │  └─ SI Tax(TVQ) │    │  glt- TPS            │   │
│  │          │    │                 │    │  glq- TVQ             │   │
│  │          │    │                 │───►│ PLE (1 per invoice)   │   │
│  │          │    │                 │    │  ple-SINV-            │   │
│  │          │    └─────────────────┘    └──────────────────────┘   │
│  │          │                                                       │
│  │          │    ┌─────────────────┐    ┌──────────────────────┐   │
│  │          │◄───│ Payment Entry   │───►│ GL Entry (2 per pmt) │   │
│  │          │    │  └─ PE Ref      │    │  gpb- bank           │   │
│  │          │    │    (allocations)│    │  gpr- receivable     │   │
│  │          │    │                 │───►│ PLE (1 per alloc)    │   │
│  │          │    │                 │    │  ple-PER-            │   │
│  └──────────┘    └─────────────────┘    └──────────────────────┘   │
│                                                                     │
│  Credit Allocations ──────────────────►  PLE (plc-)                │
│  (return inv → target inv)              against_voucher = target    │
└─────────────────────────────────────────────────────────────────────┘
```

---

## Entity Mapping

### Customer

```
Legacy (account)              ERPNext (Customer)
─────────────────             ──────────────────
id                      →     legacy_account_id
first_name + last_name  →     customer_name
email                   →     (Contact)
                               name = CUST-{hash}
```

### Sales Invoice

```
Legacy (invoice)              ERPNext (Sales Invoice)
─────────────────             ──────────────────────
id                      →     legacy_invoice_id
                               name = SINV-{legacy_id}
                               Example: 638567 → SINV-638567
account_id              →     customer (via account→Customer map)
total_amt               →     grand_total (TAX-INCLUSIVE)
total_amt - tps - tvq   →     net_total
date_orig (unix ts)     →     posting_date (YYYY-MM-DD)
total_amt < 0           →     is_return = 1
billed_amt              →     (not stored — derived from PLE)
                               outstanding_amount = SUM(PLE against this inv)
```

**Key**: Legacy `total_amt` is TAX-INCLUSIVE. ERPNext stores both `grand_total` (with tax) and `net_total` (without tax).

### Invoice Items

```
Legacy (invoice_item)         ERPNext (Sales Invoice Item)
─────────────────────         ──────────────────────────
invoice_id              →     parent = SINV name
product_name            →     item_name, description
quantity                →     qty
unitary_price           →     rate
quantity × unitary_price →    amount
                               name = SII-{legacy_id}-{idx}
                               item_code = 'SVC'
                               income_account = "Autres produits d'exploitation - T"
```

### Invoice Taxes

```
Legacy (invoice_tax)          ERPNext (Sales Taxes and Charges)
────────────────────          ─────────────────────────────────
invoice_id              →     parent = SINV name
tax_name = 'TPS'        →     description = 'TPS à payer - T'
  amount                →       tax_amount (5% GST, #834975559RT0001)
  tax_rate = 0.05       →       rate = 5.0
tax_name = 'TVQ'        →     description = 'TVQ à payer - T'
  amount                →       tax_amount (9.975% QST, #1213765929TQ0001)
  tax_rate = 0.09975    →       rate = 9.975
                               name = stc-tps-{legacy_id} / stc-tvq-{legacy_id}
```

**Note**: Legacy stores TPS and TVQ as **separate rows** per invoice. ERPNext also uses separate rows (idx=1 for TPS, idx=2 for TVQ).

### Payment Entry

```
Legacy (payment)              ERPNext (Payment Entry)
────────────────              ──────────────────────
id                      →     legacy_payment_id
                               name = PE-{legacy_id}
account_id              →     party (via account→Customer map)
amount                  →     paid_amount = received_amount
date_orig (unix ts)     →     posting_date
memo                    →     remarks
type != 'credit'        →     (only non-credit payments imported as PE)
                               payment_type = 'Receive'
                               paid_from = 'Comptes clients - T'
                               paid_to = 'Banque - T'
```

### Payment Allocations

```
Legacy (payment_item)         ERPNext (Payment Entry Reference)
─────────────────────         ───────────────────────────────
payment_id              →     parent = PE name
invoice_id              →     reference_name = SINV name
amount                  →     allocated_amount
                               name = PER-{legacy_pmt_id}-{idx}
                               reference_doctype = 'Sales Invoice'
```

### Credit Notes (Return Invoices)

```
Legacy — Three linking mechanisms:

  1. Credit payment:
     payment (type='credit', memo='credit created by invoice #NNN')
       → payment_item.invoice_id = target invoice

  2. Invoice notes:
     invoice.notes = 'Renversement de la facture #NNN'
       → #NNN = target legacy invoice ID

  3. Reversement payment memo:
     payment (type='reversement', memo='create by invoice #CREDIT for invoice #TARGET')
       → payment_item.invoice_id = target invoice

ERPNext:
  Sales Invoice (is_return=1, return_against=target SINV)
    → PLE (plc-{id})   — from credit payments (mechanism 1)
    → PLE (plr-{id})   — from reversal notes + reversement memos (mechanisms 2+3)
        voucher_no = source return SINV
        against_voucher_no = target SINV
        amount = credit invoice's grand_total (negative)
```

---

## Accounting Entries (Double-Entry)

### Per Sales Invoice (4 GL entries)

```
                              Debit         Credit
                              ─────         ──────
Comptes clients - T (AR)      grand_total
Autres produits d'expl. (Rev)               net_total
TPS à payer - T (Tax)                       tps_amount
TVQ à payer - T (Tax)                       tvq_amount
                              ─────────     ─────────
                              grand_total = grand_total  ✓
```

For **return invoices** (negative amounts), debit/credit are swapped.

### Per Payment Entry (2 GL entries)

```
                              Debit         Credit
                              ─────         ──────
Banque - T (Bank)             paid_amount
Comptes clients - T (AR)                    paid_amount
```

### Payment Ledger Entry (PLE) — Outstanding Tracking

```
Type                    amount              against_voucher
────                    ──────              ───────────────
Invoice posting         +grand_total        self (SINV)
Payment allocation      -allocated_amount   target SINV
Credit allocation       -credit_amount      target SINV
Unallocated payment     -paid_amount        self (PE)

Outstanding = SUM(PLE.amount WHERE against_voucher = this invoice)
```

---

## ERPNext Chart of Accounts

```
TARGO (Company, abbr: T)
├── Comptes clients - T          Receivable    (debit_to for all invoices)
├── Banque - T                   Bank          (paid_to for all payments)
├── Autres produits d'exploitation - T  Income (all invoice revenue)
├── TPS à payer - T              Liability/Tax (5% GST)
└── TVQ à payer - T              Liability/Tax (9.975% QST)
```

---

## Naming Conventions

### Migrated Data (Legacy IDs)

| Entity | Pattern | Example |
|--------|---------|---------|
| Sales Invoice | `SINV-{legacy_id}` | `SINV-638567` |
| SI Item | `SII-{legacy_id}-{idx}` | `SII-638567-0` |
| SI Tax (TPS) | `stc-tps-{legacy_id}` | `stc-tps-638567` |
| SI Tax (TVQ) | `stc-tvq-{legacy_id}` | `stc-tvq-638567` |
| Payment Entry | `PE-{legacy_id}` | `PE-76531` |
| PE Reference | `PER-{legacy_id}-{idx}` | `PER-76531-0` |
| GL (inv receivable) | `gir-SINV-{id}` | `gir-SINV-638567` |
| GL (inv income) | `gii-SINV-{id}` | `gii-SINV-638567` |
| GL (inv TPS) | `glt-SINV-{id}` | `glt-SINV-638567` |
| GL (inv TVQ) | `glq-SINV-{id}` | `glq-SINV-638567` |
| GL (pmt bank) | `gpb-PE-{id}` | `gpb-PE-76531` |
| GL (pmt receivable) | `gpr-PE-{id}` | `gpr-PE-76531` |
| PLE (invoice) | `ple-SINV-{id}` | `ple-SINV-638567` |
| PLE (pmt alloc) | `ple-PER-{id}-{idx}` | `ple-PER-76531-0` |
| PLE (unallocated) | `ple-PE-{id}` | `ple-PE-76531` |
| PLE (credit alloc) | `plc-{serial}` | `plc-1234` |
| PLE (reversal alloc) | `plr-{serial}` | `plr-567` |

### Post-Migration (New Documents)

| Entity | Pattern | Example |
|--------|---------|---------|
| Sales Invoice | `SINV-YYYY-NNNNN` | `SINV-2026-700001` |
| Payment Entry | ERPNext autoname | `PE-2026-00001` |

The different naming patterns between migrated and new documents ensure no collision if a reimport is needed after new documents have been created.

---

## Legacy Database Schema

### `invoice`
| Column | Type | Notes |
|--------|------|-------|
| `id` | bigint PK | → `legacy_invoice_id` in ERPNext |
| `date_orig` | bigint | UNIX timestamp |
| `account_id` | bigint FK | → `account.id` |
| `total_amt` | double(20,2) | **TAX-INCLUSIVE** |
| `billed_amt` | double(20,2) | Amount paid |
| `due_date` | bigint | UNIX timestamp |
| `correction` | tinyint | 1 = correction invoice |
| `notes` | mediumtext | |

### `invoice_item`
| Column | Type | Notes |
|--------|------|-------|
| `id` | bigint PK | |
| `invoice_id` | bigint FK | → `invoice.id` |
| `product_name` | varchar(512) | NOT `description` |
| `quantity` | double | NOT `qty` |
| `unitary_price` | double | NOT `price` |
| `sku` | varchar(128) | |

### `invoice_tax`
| Column | Type | Notes |
|--------|------|-------|
| `id` | bigint PK | |
| `invoice_id` | bigint FK | → `invoice.id` |
| `tax_name` | varchar(128) | `'TPS'` or `'TVQ'` — **separate rows, not columns** |
| `tax_rate` | double | 0.05 or 0.09975 |
| `amount` | double(20,2) | Tax amount for this type |

### `payment`
| Column | Type | Notes |
|--------|------|-------|
| `id` | bigint PK | |
| `account_id` | bigint FK | → `account.id` |
| `date_orig` | bigint | UNIX timestamp |
| `amount` | double | |
| `type` | varchar(25) | `'payment'`, `'credit'`, etc. |
| `memo` | varchar(512) | For credit: `"credit created by invoice #NNN"` |
| `reference` | varchar(128) | Stripe/processor transaction ID (e.g., `pi_3Sad...`) |

### `payment_item`
| Column | Type | Notes |
|--------|------|-------|
| `id` | bigint PK | |
| `payment_id` | bigint FK | → `payment.id` |
| `invoice_id` | bigint FK | → `invoice.id` |
| `amount` | double | Allocated to this invoice |

### `account` (customer)
| Column | Type | Notes |
|--------|------|-------|
| `id` | bigint PK | → `legacy_account_id` in Customer |
| `first_name` | varchar | |
| `last_name` | varchar | |
| `email` | varchar | |
| `company_name` | varchar | |

---

## Fiscal Year

Canadian fiscal year: **July 1 – June 30**

```
posting_date month >= 7  →  fiscal_year = "YYYY-(YYYY+1)"   e.g. "2025-2026"
posting_date month <  7  →  fiscal_year = "(YYYY-1)-YYYY"    e.g. "2024-2025"
```

---

## Legacy Non-Compliance Changelog

The legacy PHP/MariaDB system uses several non-standard accounting practices that were corrected during migration to ERPNext.

### 1. Reversal Invoices — No Credit Link

**Legacy behavior**: To cancel an invoice, the system creates a negative invoice (same amount, opposite sign) on the same account. It sets `billed_amt = total_amt` on both invoices to mark them as "settled," but **does not create a credit payment or any explicit reference between them**. There is no `return_against`, no `credit_of`, and no payment linking the two.

**Problem**: With no link, there's no audit trail showing which invoice was cancelled by which credit note. The negative invoices appear as unallocated credits, creating phantom overpayment.

**ERPNext fix**: Three matching mechanisms were developed to reconstruct the links:

1. **Credit payment allocations** (15,694 matches) — `payment.type='credit'` with `memo='credit created by invoice #NNN'` → `payment_item.invoice_id` points to target. Zero overlap with mechanism 2.
2. **Invoice notes field** (5,095 matches) — `invoice.notes` contains `'Renversement de la facture #NNN'` referencing the target legacy invoice ID.
3. **Reversement payment memos** (143 unique matches) — `payment.type='reversement'` with `memo='create by invoice #CREDIT for invoice #TARGET'` → `payment_item.invoice_id` = target. Only applied to invoices not already matched by mechanisms 1-2.

Total: **16,830 credit notes linked** via `return_against` + PLE allocation entries.

### 2. Fake "Reversement" Payments

**Legacy behavior**: When a reversal invoice is created, the system also generates a `payment.type = 'reversement'` record allocated to the original invoice. This is **not a real customer payment** — it's an internal bookkeeping entry that marks the original invoice as "paid."

**Problem**: If imported as a real Payment Entry in ERPNext, the original invoice appears double-settled (once by the fake payment, once by the credit note PLE), creating ~$955K in phantom overpayment.

**ERPNext fix**: Excluded all ~5,000 "reversement" payments from import. These are not real financial transactions — the credit note relationship (mechanism 1/2/3 above) replaces them. The reversement memos are still parsed for matching purposes (mechanism 3).

### 3. Payment Types in Legacy

| Legacy `payment.type` | Real Payment? | ERPNext Treatment |
|------------------------|---------------|-------------------|
| `paiement direct` | Yes | Payment Entry |
| `carte credit` | Yes | Payment Entry |
| `ppa` | Yes (pre-authorized) | Payment Entry |
| `credit` | No — credit note allocation | PLE (plc-) only, memo parsed for matching |
| `cheque` | Yes | Payment Entry |
| `reversement` | **No — fake reversal payment** | **Excluded from PE**, memo parsed for matching |
| `comptant` | Yes | Payment Entry |
| `credit targo` | No — internal credit | **Excluded** |

### 4. Duplicate Payments (Double Form Submission)

**Legacy behavior**: The customer portal sometimes processes the same credit card charge twice when the legacy PHP backend is slow to respond. The submit handler fires again, creating a duplicate `payment` record. Both payments have the **same `reference`** field (Stripe transaction ID, e.g., `pi_3SadKtAU3HUVhnM10KjBOebK`) but different `payment.id` values.

**Problem**: Both payments are allocated to the same invoice via `payment_item`, causing the invoice to appear overpaid. The GL bank balance is also overstated.

**ERPNext fix**: Deduplicate payments by `(account_id, reference)` during import — keep only the first payment per unique reference. **178,730 duplicate payments** removed (from 522K raw to 343K unique).

### 5. Tax-Inclusive Totals

**Legacy behavior**: `invoice.total_amt` is **tax-inclusive** (includes TPS 5% + TVQ 9.975%). Individual tax amounts are stored in separate `invoice_tax` rows, not as columns on the invoice.

**ERPNext**: Stores both `grand_total` (with tax) and `net_total` (without tax). Tax amounts are back-calculated from the `invoice_tax` table. When no tax record exists, taxes are estimated at 14.975% of total.

### 6. No Due Date Tracking

**Legacy behavior**: Many invoices have `due_date = 0` (UNIX epoch) or NULL. The system doesn't enforce payment terms.

**ERPNext fix**: Uses `posting_date` as fallback when `due_date` is missing or invalid (before 2000).

### 7. Credit Notes with Both Payment and Reversal PLE

**Legacy behavior**: Some credit invoices have BOTH a `credit` payment allocation (mechanism 1) AND a separate reversal invoice reference (mechanism 2 or 3). If both are processed, the target invoice gets double-credited.

**Problem**: Creates phantom overpayment (negative outstanding) on target invoices. Originally caused $975K in overpaid balances.

**ERPNext fix**: Apply matching mechanisms in priority order. Track `already_linked` set — if a credit invoice was matched by credit payment (mechanism 1), skip it for reversal notes (mechanism 2) and reversement memos (mechanism 3). This prevents double-counting.

### 8. Orphaned Invoices (No Customer Account)

**Legacy behavior**: 9 invoices reference `account_id` values that don't exist in the `account` table (deleted customers or test data).

**ERPNext fix**: Skipped during import. Logged as unmapped: IDs 2712, 5627, 15119, 15234, 195096, 216370, 272051, 277963, 308988.

### 9. Invoice Naming for CRA Compliance

**Legacy behavior**: Invoices have integer IDs with no prefix. No formal naming convention.

**Problem**: CRA requires sequential invoice numbering for audit trail. Hex-encoded IDs (e.g., `SINV-000009BE67`) are not human-readable and can cause hash collisions on child tables.

**ERPNext fix**: Use `SINV-{legacy_id}` format (e.g., `SINV-638567`). Legacy IDs are already sequential integers. Post-migration invoices will use `SINV-YYYY-NNNNN` format (starting at `SINV-2026-700001`) to avoid collision with legacy IDs on reimport.

### 10. Customer Portal Credentials

**Legacy behavior**: 15,305 customer accounts with username/password (hashed, likely MD5/SHA1). Password reset tokens in `client_pwd` table (9,687 tokens). Stripe customer IDs stored in `account.stripe_id`.

**ERPNext fix**: Legacy password hashes are incompatible with Frappe's bcrypt auth. After migration, ERPNext Website Users will be created with customer emails, and bulk password reset emails will be sent. Stripe IDs will be linked to ERPNext's payment integration.

---

## Migration Results (2026-03-29, full historical import)

**Migration time: ~16 minutes** (966 seconds) for full reimport including cleanup, data load from legacy MariaDB, bulk SQL inserts, GL entries, PLE entries, and outstanding recalculation. This compares to an estimated ~120 hours if using Frappe ORM.

| Metric | Count |
|--------|-------|
| Sales Invoices | 629,935 |
| Payment Entries | 343,684 (excl. reversement + credit + credit targo) |
| Payment References | 426,018 |
| GL Entries | 3,135,184 |
| PLE Entries | 1,060,041 |
| **GL Balance** | **$130,120,226.76 = $130,120,226.76 (diff $0.00)** |

### Credit Note Matching
| Mechanism | Matches |
|-----------|---------|
| Credit payment allocations (plc-) | 15,508 |
| Reversal notes + reversement memos (plr-) | 5,238 |
| **Total linked** | **20,746** |

### Invoice Status Breakdown
| Status | Count |
|--------|-------|
| Paid | 415,861 |
| Overdue | 197,190 |
| Return | 16,868 |
| Credit Note Issued | 15 |
| Unpaid | 1 |

### Outstanding
- **Owed: $20,500,562.62**
- **Overpaid: -$3,695.77** (15 invoices — unmatched credit notes with no reference to original in any of the 3 mechanisms)

### Payment Deduplication
- Raw payments from legacy: 522,416
- After dedup by `(account_id, reference)`: 343,686
- Duplicates removed: **178,730**

---

## How to Re-run

The migration is **idempotent** — the script deletes all existing data and reimports from scratch.

```bash
# From facturation.targo.ca:
ssh root@96.125.196.67 'docker cp /path/to/clean_reimport.py erpnext-backend-1:/home/frappe/frappe-bench/clean_reimport.py'
ssh root@96.125.196.67 'docker exec erpnext-backend-1 /home/frappe/frappe-bench/env/bin/python /home/frappe/frappe-bench/clean_reimport.py'
```

**Important**: Update the password in the script before running (redacted as `*******` in git).

---

## Complete Migration Phases & Script Inventory

### Execution Order

The migration is split into phases. Each phase can be re-run independently (most scripts are idempotent or nuke-and-reimport).

| Phase | Script | Description | Status |
|-------|--------|-------------|--------|
| **0** | `nuke_data.py` | Delete all migrated data except Users, Items, Plans | Run before reimport |
| **1a** | `clean_reimport.py` | **Master accounting import**: 630K invoices, 344K payments, 3.1M GL, 1M PLE | **DONE** |
| **1b** | `migrate_direct.py` | Legacy account → Customer (15,303) | **DONE** (via migrate_all.py) |
| **1c** | `import_items.py` | Legacy product → Item (833) + Item Groups (41) | **DONE** |
| **2a** | `migrate_phase3.py` | Subscription Plans + Subscriptions from active services (cats 4,9,17,21,32,33) | **DONE** |
| **2b** | `import_missing_services.py` | Services from excluded categories (non-standard) | **DONE** |
| **2c** | `fix_subscription_details.py` | Populate actual_price, custom_description, item_code, item_group, billing_frequency from hijack data | **DONE** |
| **2d** | `fix_annual_billing_dates.py` | Fix annual billing dates from legacy date_next_invoice | **DONE** |
| **2e** | `fix_sub_address.py` | Link Subscription → Service Location via delivery_id | **DONE** |
| **3a** | `migrate_locations.py` | Legacy delivery → Service Location + device → Service Equipment | **DONE** |
| **3b** | `import_devices_and_enrich.py` | Import missing equipment + enrich locations with fibre OLT data | **DONE** |
| **3c** | `import_fibre_sql.py` (/tmp on server) | Direct SQL: Add OLT fields to Service Equipment (4,930 records from fibre table) | **DONE** |
| **4a** | `migrate_tickets.py` | Legacy ticket → Issue (98,524) + ticket_msg → Communication | **DONE** |
| **4b** | `fix_issue_cust2.py` | Link Issue → Customer via legacy account_id | **DONE** |
| **4c** | `fix_issue_owners.py` | Fix Issue owner/assignee from legacy staff_id | **DONE** |
| **4d** | `import_memos.py` | Legacy account_memo → Comments on Customer | **DONE** |
| **5a** | `import_customer_details.py` | Add 12+ custom fields to Customer (billing, contact, commercial flags) | **DONE** |
| **5b** | `import_services_and_enrich_customers.py` | Enrich Customer with phone, email, stripe_id, PPA, notes | **DONE** |
| **5c** | `cleanup_customer_status.py` | Disable Customers with no active subscriptions | **DONE** |
| **5d** | `import_terminated.py` | Import terminated customers (status 3,4,5) | **DONE** |
| **6a** | `import_employees.py` | Legacy staff → Employee (155), maps group_ad → Department | **DONE** |
| **6b** | `import_technicians.py` | Link Employee → Dispatch Technician | **DONE** |
| **6c** | `add_office_extension.py` | Add office_extension field to Employee from legacy staff.ext | **DONE** |
| **7a** | `setup_user_roles.py` | Create Role Profiles + assign Users (admin, tech, support, etc.) | **DONE** |
| **7b** | `setup_scheduler_toggle.py` | API endpoints for scheduler control (scheduler_status, toggle) | **DONE** |
| **8a** | `fix_customer_links.py` | Fix customer references in SINV, Subscription, Issue (name → CUST-xxx) | **DONE** |
| **8b** | `fix_invoice_outstanding.py` | Correct outstanding_amount from legacy billing_status | **DONE** |
| **8c** | `fix_reversals.py` | Link credit invoices → originals via return_against + PLE | **DONE** |
| **8d** | `fix_reversement.py` | Delete incorrectly imported reversement Payment Entries | **DONE** |
| **8e** | `fix_dates.py` | Fix creation/modified timestamps from legacy unix timestamps | **DONE** |
| **8f** | `fix_and_invoices.py` | Fix Subscription.party + import recent invoices | **DONE** |
| **8g** | `fix_no_rebate_discounts.py` | Restore catalog prices on deliveries (rebate handling) | **DONE** |
| **9a** | `rename_to_readable_ids.py` | Rename hex IDs → human-readable (CUST-xxx, LOC-addr, EQ-dev) | **DONE** |
| **9b** | `geocode_locations.py` | Geocode Service Locations via rqa_addresses (Quebec address DB) | **DONE** |
| **9c** | `update_item_descriptions.py` | Update Item descriptions from legacy French translations | **DONE** |

### Analysis/Exploration Scripts (read-only)

| Script | Purpose |
|--------|---------|
| `analyze_pricing_cleanup.py` | Pricing analysis: catalog vs hijack, rebate absorption |
| `check_missing_cat26.py` | Identify missing services in non-imported categories |
| `explore_expro_payments.py` | Compare legacy vs ERPNext payments for Expro Transit |
| `explore_expro_services.py` | Show active services for Expro Transit with pricing |
| `simulate_payment_import.py` | DRY RUN for Expro payments: timeline of invoice vs balance |
| `import_expro_payments.py` | Import missing Expro Transit payments (account 3673) |

### Helper Scripts (in parent /scripts/)

| Script | Purpose |
|--------|---------|
| `bulk_submit.py` | Bulk submit drafted Sales Invoices (docstatus 0 → 1) |
| `fix_ple_groupby.py` | Fix PostgreSQL GROUP BY errors in PLE queries |
| `fix_ple_postgres.sh` | Shell script to apply PostgreSQL patches |
| `server_bulk_submit.py` | Server-side bulk submit with progress tracking |

---

## Legacy Tables → ERPNext Mapping (Complete)

### Core Data

| Legacy Table | ERPNext DocType | Script | Notes |
|---|---|---|---|
| `account` | Customer | migrate_direct.py | 15,303 records |
| `account` (contact data) | Contact | import_services_and_enrich_customers.py | Phone, email, cell |
| `delivery` | Service Location | migrate_locations.py | Delivery addresses |
| `service` | Subscription | migrate_phase3.py + import_missing_services.py | Active services |
| `product` | Item | import_items.py | 833 items, 41 groups |
| `product` (plans) | Subscription Plan | migrate_phase3.py | Pricing plans |
| `device` | Service Equipment | migrate_locations.py + import_devices_and_enrich.py | ~7,241 devices |
| `fibre` + `fibre_olt` | Service Equipment (OLT fields) | import_fibre_sql.py | 4,930 with OLT data |

### Accounting

| Legacy Table | ERPNext DocType | Script | Notes |
|---|---|---|---|
| `invoice` | Sales Invoice | clean_reimport.py | 629,935 records |
| `invoice_item` | Sales Invoice Item | clean_reimport.py | Line items |
| `invoice_tax` | Sales Taxes and Charges | clean_reimport.py | TPS/TVQ rows |
| `payment` | Payment Entry | clean_reimport.py | 343,684 (deduped) |
| `payment_item` | Payment Entry Reference | clean_reimport.py | 426,018 allocations |
| — | GL Entry | clean_reimport.py | 3,135,184 generated |
| — | Payment Ledger Entry | clean_reimport.py | 1,060,041 generated |

### Support & HR

| Legacy Table | ERPNext DocType | Script | Notes |
|---|---|---|---|
| `ticket` | Issue | migrate_tickets.py | 98,524 tickets |
| `ticket_msg` | Communication | migrate_tickets.py | Ticket replies |
| `ticket_dept` | Issue Type | migrate_tickets.py | Department → type |
| `account_memo` | Comment | import_memos.py | Internal notes |
| `staff` | Employee | import_employees.py | 155 employees |
| `staff` | User | migrate_users.py | Active staff → ERPNext users |

### Service Enrichment Fields

| Legacy Source | ERPNext Field | Script |
|---|---|---|
| `service.hijack_price` | Subscription.actual_price | fix_subscription_details.py |
| `service.hijack_desc` | Subscription.custom_description | fix_subscription_details.py |
| `service.product_id` → product.sku | Subscription.item_code | fix_subscription_details.py |
| `service.date_next_invoice` | Subscription.current_invoice_start | fix_annual_billing_dates.py |
| `service.billing_frequency` | Subscription.billing_frequency (M/A) | fix_subscription_details.py |
| `fibre.sn` | Service Equipment.serial_number | import_fibre_sql.py |
| `fibre.info_connect` → `fibre_olt` | Service Equipment.olt_name/ip/slot/port | import_fibre_sql.py |
| `account.stripe_id` | Customer.stripe_id | import_services_and_enrich_customers.py |
| `account.password` | Customer.legacy_password_hash | Not yet migrated |

---

## Remaining Migration Tasks

| Task | Priority | Notes |
|------|----------|-------|
| Migrate legacy password hashes (`account.password`) | P1 | Needed for customer portal auth bridge (see project_portal_auth.md) |
| Investigate 15 overpaid invoices ($3,695.77) | P2 | Unmatched credit notes |
| Customer portal users (Website User creation) | P1 | 15,305 accounts with email — send password reset |
| QR code on invoice PDFs | P2 | Stripe payment link for customer portal |
| Scheduler reactivation | P0 | **PAUSED** — need Louis-Paul approval before enabling |