louis/gigafibre-fsm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
9fda9eb0b0
gigafibre-fsm/docs/archive/MIGRATION.md
louispaulb 41d9b5f316 feat: flow editor, Gemini QR scanner with offline queue, dispatch planning v2 
Major additions accumulated over 9 days — single commit per request.

Flow editor (new):
- Generic visual editor for step trees, usable by project wizard + agent flows
- PROJECT_KINDS / AGENT_KINDS catalogs decouple UI from domain
- Drag-and-drop reorder via vuedraggable with scope isolation per peer group
- Chain-aware depends_on rewrite on reorder (sequential only — DAGs preserved)
- Variable picker with per-applies_to catalog (Customer / Quotation /
  Service Contract / Issue / Subscription), insert + copy-clipboard modes
- trigger_condition helper with domain-specific JSONLogic examples
- Global FlowEditorDialog mounted once in MainLayout, Odoo inline pattern
- Server: targo-hub flow-runtime.js, flow-api.js, flow-templates.js
- ERPNext: Flow Template/Run doctypes, scheduler, 5 seeded system templates
- depends_on chips resolve to step labels instead of opaque "s4" ids

QR/OCR scanner (field app):
- Camera capture → Gemini Vision via targo-hub with 8s timeout
- IndexedDB offline queue retries photos when signal returns
- Watcher merges late-arriving scan results into the live UI

Dispatch:
- Planning mode (draft → publish) with offer pool for unassigned jobs
- Shared presets, recurrence selector, suggested-slots dialog
- PublishScheduleModal, unassign confirmation

Ops app:
- ClientDetailPage composables extraction (useClientData, useDeviceStatus,
  useWifiDiagnostic, useModemDiagnostic)
- Project wizard: shared detail sections, wizard catalog/publish composables
- Address pricing composable + pricing-mock data
- Settings redesign hosting flow templates

Targo-hub:
- Contract acceptance (JWT residential + DocuSeal commercial tracks)
- Referral system
- Modem-bridge diagnostic normalizer
- Device extractors consolidated

Migration scripts:
- Invoice/quote print format setup, Jinja rendering
- Additional import + fix scripts (reversals, dates, customers, payments)

Docs:
- Consolidated: old scattered MDs → HANDOFF, ARCHITECTURE, DATA_AND_FLOWS,
  FLOW_EDITOR_ARCHITECTURE, BILLING_AND_PAYMENTS, CPE_MANAGEMENT,
  APP_DESIGN_GUIDELINES
- Archived legacy wizard PHP for reference
- STATUS snapshots for 2026-04-18/19

Cleanup:
- Removed ~40 generated PDFs/HTMLs (invoice_preview*, rendered_jinja*)
- .gitignore now covers invoice preview output + nested .DS_Store

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-22 10:44:17 -04:00

8.0 KiB
Raw Blame History

Legacy to ERPNext Migration

1. Legacy System

Source database: gestionclient (MariaDB on 10.100.80.100). Monthly recurring revenue ~$742K across fibre, wireless, TV, and telephony.

Table Records Description
account 15,303 Customers (6,510 active, 8,594 terminated)
delivery 17,114 Installation addresses with GPS
service 66,879 Subscriptions (39,624 active)
product / product_cat 833 / 34 Catalogue items and categories
invoice / invoice_item 630K / 1.86M Invoices and line items
payment / payment_item 541K / 685K Payments and allocations
compta_comptes / ecritures 48 / 1.2M Chart of accounts and journal entries
ticket / ticket_msg 243K / 784K Support tickets and messages (8K with base64 images)
device / fibre 10,377 / 16,057 Network equipment and fibre connections
staff / bon_travail 155 / 14,472 Employees and work orders

2. Legacy to ERPNext Mapping

Legacy Table ERPNext DocType Records Status
account Customer + Contact + Address 6,667 + 6,600 + 6,700 Migrated
delivery + fibre Service Location 17,114 Migrated
product Item (type=Service) 833 Migrated
product_cat Item Group 34 Migrated
service (active) Subscription 21,876 Migrated (scheduler paused)
invoice + invoice_item Sales Invoice 630K Opening balance only
payment + payment_item Payment Entry 99K imported Migrated (recent)
compta_comptes Account (Chart) 48 Mapped to ERPNext CoA
compta_journal_ecriture Journal Entry 1.2M Opening balance only
ticket + ticket_msg Issue + Communication 243K + 784K Migrated
device Service Equipment 10,377 Migrated
staff Employee / User 45 users Migrated
bon_travail Dispatch Job 14,472 Pending
tax Sales Taxes Template 4 Migrated (TPS 5% + TVQ 9.975%)

3. Key Architecture Differences

  • Naming: Legacy uses auto-increment IDs; ERPNext uses naming series (CUST-.YYYY.-, LOC-.#####)
  • Customer/Address: Legacy stores everything in account; ERPNext separates Customer, Address, and Contact as linked doctypes
  • Subscriptions: Legacy service is a static row; ERPNext Subscription auto-generates Sales Invoices each billing cycle
  • Custom Pricing: Legacy hijack_price override on service; ERPNext uses additional_discount on Subscription or Pricing Rules
  • Taxes: Legacy calculates TPS/TVQ in PHP code; ERPNext applies Sales Taxes Templates automatically
  • Accounting: Legacy writes journal entries via PHP; ERPNext auto-creates GL Entries from submitted Sales Invoices
  • Permissions: Legacy uses staff.rights (PHP serialized); ERPNext uses Role-based DocType permissions
  • Files: Legacy stores images as base64 in mediumtext columns; ERPNext uses File doctype with /files/ directory
  • Dates: Legacy uses Unix timestamps (bigint); ERPNext uses ISO dates (YYYY-MM-DD)
  • RADIUS: Legacy stores credentials on service; ERPNext uses custom fields on Subscription

4. Migration Phases

  1. Foundation -- Done (2026-03-28)

    • 34 Item Groups, 833 Items with ISP custom fields (speeds, quotas, fibre profiles)
    • Tax Templates: TPS 5% + TVQ 9.975%
    • 15 custom fields created across Item, Customer, and Subscription doctypes

  2. Customers & Addresses -- Done (2026-03-28)

    • 6,667 Customers, ~6,600 Contacts, ~6,700 Addresses via direct PostgreSQL (~30s)
    • Customer-Contact-Address links established with GPS coordinates

  3. Subscriptions & Items -- Done (2026-03-28)

    • 92 Subscription Plans created from active products
    • 21,876 Subscriptions imported with RADIUS data
    • Scheduler PAUSED to prevent automatic invoice generation

  4. Invoices & Payments -- Done (2026-03-29)

    • Outstanding invoice analysis and AR reconciliation
    • 99,000 payments imported with invoice references and mode mapping

  5. Opening Balances -- Done (2026-03-29)

    • Customer balance reconciliation (invoices minus payments)
    • Opening balance journal entries created

  6. Tickets -- Done (2026-03-29, 242K)

    • 242,618 tickets migrated as Issues with parent/child hierarchy
    • 784,290 ticket messages imported as Communications
    • Issue types mapped from 21 legacy departments
    • Staff assignment mapped from legacy IDs to ERPNext User emails

  7. Staff & Memos -- Done (2026-03-29)

    • 45 ERPNext Users created from legacy staff table
    • 29,000 customer memos imported as Comments with original creation dates

5. Field Gap Analysis

~73 custom fields needed across 5 doctypes to reach zero data loss.

DocType Gaps Key Missing Areas
Customer ~23 PPA banking (8 fields), Stripe (2), termination reason/date (4), VIP/flags (3), portal login (2), other (4)
Subscription ~10 Custom pricing flag/desc (3), suspension/active-until dates (3), quotas (2), static IP (1), other (1)
Service Equipment ~17 Management access (6: URL, port, protocol for GUI+CLI), provisioning from GenieACS (8: WiFi, VoIP, GPON/CWMP serials), ACS integration (4)
Service Location ~18 Fibre infrastructure (10: ONT serial, OLT IP/name, ONT ID, distance, spans), individual VLANs (4), legacy service IDs (4)
Dispatch Job ~5 Second technician + arrival/departure timestamps (5)

Already migrated fields per doctype: Customer ~18, Subscription ~14, Equipment ~10, Location ~9.

6. Execution Log

  • 2026-03-28: Frappe Assistant Core v2.3.3 installed, legacy MariaDB access granted. 34 Item Groups + 833 Items imported. 15 custom fields created (Item 7, Customer 4, Subscription 3). 6,667 Customers + Contacts + Addresses via direct PG in ~30s. 92 Subscription Plans + 21,876 Subscriptions with RADIUS data. Scheduler paused.
  • 2026-03-29: Opening balance + AR analysis. 242K tickets + 784K messages migrated as Issues/Communications. 45 Users created. 29K customer memos imported. 99K payments imported.
  • 2026-03-30: Ops App V2 with ClientDetailPage, Dispatch module, Equipment page, OCR. Field Tech App (PWA with barcode scanner, offline support).
  • 2026-03-31: Inline editing (InlineField component), client search with legacy ID + or_filters, SMS/Email notifications via n8n/Twilio/Mailjet, ticket reply via Communication docs, Authentik federation (auth.targo.ca to id.gigafibre.ca).

7. Data NOT Migrated

  • Journal entries (1.2M) -- opening balance only, no line-by-line import
  • Invoices > 2 years -- covered by opening balance
  • Closed tickets (241K) -- available in legacy read-only archive
  • RADIUS consumption (conso_*) -- operational data, not financial
  • Phone call logs (phonecall_log_*) -- historical records
  • Temp/backup tables (tmp*, *_bk, *_archive)
  • Camping tables (camping_*) -- confirm if still in use
  • Password manager (passwords_manager*) -- never migrate third-party credentials
  • Work orders (bon_travail, 14K) -- historical, Dispatch Job doctype ready for future import

8. Risks & Decisions

Risks:

  • Incorrect balances after migration -- mitigated by before/after reconciliation reports
  • PPA payment disruption during cutover -- mitigated by parallel operation (legacy bills, ERPNext takes over gradually)
  • Missing subscriptions -- validation script compares active legacy services against ERPNext Subscriptions
  • Lost base64 images -- batch extraction scheduled before cutover
  • ERPNext performance with 630K invoices -- only 2 years imported, rest as opening balance
  • Scheduler is PAUSED -- reactivation would trigger billing for 21,876 subscriptions

Open Decisions:

  • Invoice history depth: currently 2 years, rest as opening balance
  • PPA (AccesD Desjardins): ERPNext has no native AccesD support, middleware needed
  • Legacy ID retention: legacy_* fields on all doctypes for traceability
  • Customer portal: 659 portal profiles use unsalted MD5 passwords, force reset via email/SMS OTP required
  • RADIUS integration: webhook from Subscription to FreeRADIUS via n8n
  • Camping module: confirm if camping_* tables are still active