louis/gigafibre-fsm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
main
gigafibre-fsm/docs/archive/MIGRATION.md
louispaulb 41d9b5f316 feat: flow editor, Gemini QR scanner with offline queue, dispatch planning v2 
Major additions accumulated over 9 days — single commit per request.

Flow editor (new):
- Generic visual editor for step trees, usable by project wizard + agent flows
- PROJECT_KINDS / AGENT_KINDS catalogs decouple UI from domain
- Drag-and-drop reorder via vuedraggable with scope isolation per peer group
- Chain-aware depends_on rewrite on reorder (sequential only — DAGs preserved)
- Variable picker with per-applies_to catalog (Customer / Quotation /
  Service Contract / Issue / Subscription), insert + copy-clipboard modes
- trigger_condition helper with domain-specific JSONLogic examples
- Global FlowEditorDialog mounted once in MainLayout, Odoo inline pattern
- Server: targo-hub flow-runtime.js, flow-api.js, flow-templates.js
- ERPNext: Flow Template/Run doctypes, scheduler, 5 seeded system templates
- depends_on chips resolve to step labels instead of opaque "s4" ids

QR/OCR scanner (field app):
- Camera capture → Gemini Vision via targo-hub with 8s timeout
- IndexedDB offline queue retries photos when signal returns
- Watcher merges late-arriving scan results into the live UI

Dispatch:
- Planning mode (draft → publish) with offer pool for unassigned jobs
- Shared presets, recurrence selector, suggested-slots dialog
- PublishScheduleModal, unassign confirmation

Ops app:
- ClientDetailPage composables extraction (useClientData, useDeviceStatus,
  useWifiDiagnostic, useModemDiagnostic)
- Project wizard: shared detail sections, wizard catalog/publish composables
- Address pricing composable + pricing-mock data
- Settings redesign hosting flow templates

Targo-hub:
- Contract acceptance (JWT residential + DocuSeal commercial tracks)
- Referral system
- Modem-bridge diagnostic normalizer
- Device extractors consolidated

Migration scripts:
- Invoice/quote print format setup, Jinja rendering
- Additional import + fix scripts (reversals, dates, customers, payments)

Docs:
- Consolidated: old scattered MDs → HANDOFF, ARCHITECTURE, DATA_AND_FLOWS,
  FLOW_EDITOR_ARCHITECTURE, BILLING_AND_PAYMENTS, CPE_MANAGEMENT,
  APP_DESIGN_GUIDELINES
- Archived legacy wizard PHP for reference
- STATUS snapshots for 2026-04-18/19

Cleanup:
- Removed ~40 generated PDFs/HTMLs (invoice_preview*, rendered_jinja*)
- .gitignore now covers invoice preview output + nested .DS_Store

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-22 10:44:17 -04:00

136 lines
8.0 KiB
Markdown
Raw Permalink Blame History

# Legacy to ERPNext Migration
## 1. Legacy System
Source database: **gestionclient** (MariaDB on 10.100.80.100). Monthly recurring revenue ~$742K across fibre, wireless, TV, and telephony.
| Table | Records | Description |
|---|---|---|
| account | 15,303 | Customers (6,510 active, 8,594 terminated) |
| delivery | 17,114 | Installation addresses with GPS |
| service | 66,879 | Subscriptions (39,624 active) |
| product / product_cat | 833 / 34 | Catalogue items and categories |
| invoice / invoice_item | 630K / 1.86M | Invoices and line items |
| payment / payment_item | 541K / 685K | Payments and allocations |
| compta_comptes / ecritures | 48 / 1.2M | Chart of accounts and journal entries |
| ticket / ticket_msg | 243K / 784K | Support tickets and messages (8K with base64 images) |
| device / fibre | 10,377 / 16,057 | Network equipment and fibre connections |
| staff / bon_travail | 155 / 14,472 | Employees and work orders |
## 2. Legacy to ERPNext Mapping
| Legacy Table | ERPNext DocType | Records | Status |
|---|---|---|---|
| account | Customer + Contact + Address | 6,667 + 6,600 + 6,700 | Migrated |
| delivery + fibre | Service Location | 17,114 | Migrated |
| product | Item (type=Service) | 833 | Migrated |
| product_cat | Item Group | 34 | Migrated |
| service (active) | Subscription | 21,876 | Migrated (scheduler paused) |
| invoice + invoice_item | Sales Invoice | 630K | Opening balance only |
| payment + payment_item | Payment Entry | 99K imported | Migrated (recent) |
| compta_comptes | Account (Chart) | 48 | Mapped to ERPNext CoA |
| compta_journal_ecriture | Journal Entry | 1.2M | Opening balance only |
| ticket + ticket_msg | Issue + Communication | 243K + 784K | Migrated |
| device | Service Equipment | 10,377 | Migrated |
| staff | Employee / User | 45 users | Migrated |
| bon_travail | Dispatch Job | 14,472 | Pending |
| tax | Sales Taxes Template | 4 | Migrated (TPS 5% + TVQ 9.975%) |
## 3. Key Architecture Differences
- **Naming**: Legacy uses auto-increment IDs; ERPNext uses naming series (CUST-.YYYY.-, LOC-.#####)
- **Customer/Address**: Legacy stores everything in `account`; ERPNext separates Customer, Address, and Contact as linked doctypes
- **Subscriptions**: Legacy `service` is a static row; ERPNext `Subscription` auto-generates Sales Invoices each billing cycle
- **Custom Pricing**: Legacy `hijack_price` override on service; ERPNext uses `additional_discount` on Subscription or Pricing Rules
- **Taxes**: Legacy calculates TPS/TVQ in PHP code; ERPNext applies Sales Taxes Templates automatically
- **Accounting**: Legacy writes journal entries via PHP; ERPNext auto-creates GL Entries from submitted Sales Invoices
- **Permissions**: Legacy uses `staff.rights` (PHP serialized); ERPNext uses Role-based DocType permissions
- **Files**: Legacy stores images as base64 in mediumtext columns; ERPNext uses File doctype with `/files/` directory
- **Dates**: Legacy uses Unix timestamps (bigint); ERPNext uses ISO dates (YYYY-MM-DD)
- **RADIUS**: Legacy stores credentials on `service`; ERPNext uses custom fields on `Subscription`
## 4. Migration Phases
1. **Foundation** -- Done (2026-03-28)
- 34 Item Groups, 833 Items with ISP custom fields (speeds, quotas, fibre profiles)
- Tax Templates: TPS 5% + TVQ 9.975%
- 15 custom fields created across Item, Customer, and Subscription doctypes
2. **Customers & Addresses** -- Done (2026-03-28)
- 6,667 Customers, ~6,600 Contacts, ~6,700 Addresses via direct PostgreSQL (~30s)
- Customer-Contact-Address links established with GPS coordinates
3. **Subscriptions & Items** -- Done (2026-03-28)
- 92 Subscription Plans created from active products
- 21,876 Subscriptions imported with RADIUS data
- **Scheduler PAUSED** to prevent automatic invoice generation
4. **Invoices & Payments** -- Done (2026-03-29)
- Outstanding invoice analysis and AR reconciliation
- 99,000 payments imported with invoice references and mode mapping
5. **Opening Balances** -- Done (2026-03-29)
- Customer balance reconciliation (invoices minus payments)
- Opening balance journal entries created
6. **Tickets** -- Done (2026-03-29, 242K)
- 242,618 tickets migrated as Issues with parent/child hierarchy
- 784,290 ticket messages imported as Communications
- Issue types mapped from 21 legacy departments
- Staff assignment mapped from legacy IDs to ERPNext User emails
7. **Staff & Memos** -- Done (2026-03-29)
- 45 ERPNext Users created from legacy staff table
- 29,000 customer memos imported as Comments with original creation dates
## 5. Field Gap Analysis
~73 custom fields needed across 5 doctypes to reach zero data loss.
| DocType | Gaps | Key Missing Areas |
|---|---|---|
| Customer | ~23 | PPA banking (8 fields), Stripe (2), termination reason/date (4), VIP/flags (3), portal login (2), other (4) |
| Subscription | ~10 | Custom pricing flag/desc (3), suspension/active-until dates (3), quotas (2), static IP (1), other (1) |
| Service Equipment | ~17 | Management access (6: URL, port, protocol for GUI+CLI), provisioning from GenieACS (8: WiFi, VoIP, GPON/CWMP serials), ACS integration (4) |
| Service Location | ~18 | Fibre infrastructure (10: ONT serial, OLT IP/name, ONT ID, distance, spans), individual VLANs (4), legacy service IDs (4) |
| Dispatch Job | ~5 | Second technician + arrival/departure timestamps (5) |
Already migrated fields per doctype: Customer ~18, Subscription ~14, Equipment ~10, Location ~9.
## 6. Execution Log
- **2026-03-28**: Frappe Assistant Core v2.3.3 installed, legacy MariaDB access granted. 34 Item Groups + 833 Items imported. 15 custom fields created (Item 7, Customer 4, Subscription 3). 6,667 Customers + Contacts + Addresses via direct PG in ~30s. 92 Subscription Plans + 21,876 Subscriptions with RADIUS data. Scheduler paused.
- **2026-03-29**: Opening balance + AR analysis. 242K tickets + 784K messages migrated as Issues/Communications. 45 Users created. 29K customer memos imported. 99K payments imported.
- **2026-03-30**: Ops App V2 with ClientDetailPage, Dispatch module, Equipment page, OCR. Field Tech App (PWA with barcode scanner, offline support).
- **2026-03-31**: Inline editing (InlineField component), client search with legacy ID + or_filters, SMS/Email notifications via n8n/Twilio/Mailjet, ticket reply via Communication docs, Authentik federation (auth.targo.ca to id.gigafibre.ca).
## 7. Data NOT Migrated
- **Journal entries** (1.2M) -- opening balance only, no line-by-line import
- **Invoices > 2 years** -- covered by opening balance
- **Closed tickets** (241K) -- available in legacy read-only archive
- **RADIUS consumption** (`conso_*`) -- operational data, not financial
- **Phone call logs** (`phonecall_log_*`) -- historical records
- **Temp/backup tables** (`tmp*`, `*_bk`, `*_archive`)
- **Camping tables** (`camping_*`) -- confirm if still in use
- **Password manager** (`passwords_manager*`) -- never migrate third-party credentials
- **Work orders** (`bon_travail`, 14K) -- historical, Dispatch Job doctype ready for future import
## 8. Risks & Decisions
**Risks**:
- Incorrect balances after migration -- mitigated by before/after reconciliation reports
- PPA payment disruption during cutover -- mitigated by parallel operation (legacy bills, ERPNext takes over gradually)
- Missing subscriptions -- validation script compares active legacy services against ERPNext Subscriptions
- Lost base64 images -- batch extraction scheduled before cutover
- ERPNext performance with 630K invoices -- only 2 years imported, rest as opening balance
- **Scheduler is PAUSED** -- reactivation would trigger billing for 21,876 subscriptions
**Open Decisions**:
- Invoice history depth: currently 2 years, rest as opening balance
- PPA (AccesD Desjardins): ERPNext has no native AccesD support, middleware needed
- Legacy ID retention: `legacy_*` fields on all doctypes for traceability
- Customer portal: 659 portal profiles use unsalted MD5 passwords, force reset via email/SMS OTP required
- RADIUS integration: webhook from Subscription to FreeRADIUS via n8n
- Camping module: confirm if `camping_*` tables are still active
Reference in New Issue View Git Blame Copy Permalink