Table Export System

Reusable backend-driven export for DataTable modules. A single GET endpoint serves both list and export; column definitions live in a registry; sync and async modes are chosen automatically by row count.

Reference implementation: COD Guides (cod.guides)

Section integration guide: Integrate export in a new section — step-by-step for backend + frontend (6 tasks).

Architecture

Design principles

Principle	Implementation
No duplicate export routes	Export via existing `GET` with `report=true`
Minimal definition files	Only `queryUtil.buildQueryContext`, `columns`, optional `mapRow`
Generic iteration	`tableExport/index.js` handles count + chunked ID pagination for all exports
Single orchestrator	`backend/libraries/tableExport/index.js` (registry, jobs, iteration, CSV)
Lazy definition load	Definitions load on first export request — a broken export does not block server boot
Read-only queries	Always `orm.READ_ONLY_QUERY_OPTIONS` for export reads
Performance	Frontend sends `recordsTotal` to skip redundant COUNT
Async jobs	Bull queue `table_export` processed by `worker.js`

Prerequisites

1. Database migration

Run once per environment:

sql

-- backend/scripts/admin-table-exports.sql
CREATE TABLE IF NOT EXISTS admin_table_exports ( ... );

2. Environment variables

Variable	Default	Purpose
`TABLE_EXPORT_SYNC_MAX_ROWS`	`5000`	Max rows for inline CSV response
`TABLE_EXPORT_TTL_HOURS`	`72`	Job/file expiration
`TABLE_EXPORT_LOCK_DURATION_MS`	`1800000` (30 min)	Bull job lock for long exports; renewed per chunk via `updateProgress`
`ENVIA_ADMON_HOSTNAME`	—	Base URL for Slack download links (`/dashboardV2?exportID={uuid}`)

3. Worker process

Async exports require the Bull worker:

bash

# worker.js registers:
# QueueWrapper('table_export') → jobs/tableExport.jobs.js
node worker.js

Debugging failed jobs: the processor writes progress to Bull via job.log() (visible in Redis/Bull UI or queue inspection). Search for [Table Export] / [tableExport] log lines tied to the job uuid.

Each job logs:

Job metadata: uuid, resource, userId, localeId, expiresAt, parsed filtersJson
Estimated chunks when row_count was stored at enqueue time (knownRowCount from the client)
Export context: column count, filters, filterLocale
IDs query config (select, from, structure, order) and per-chunk idsSQL / rowsSQL
Per chunk: offset, IDs fetched, rows fetched, cumulative row count; on failure, which chunk and error message
Final summary: total chunks, total rows, CSV byte size, S3 path, elapsed time

Step-by-step: Add a new export

Example: exporting My Module records under resource key my.module.

Integration requires 3 tasks only:

Create the definition file in definitions/
Add buildQueryContext to the section util
In the controller, if export → tableExportUtil.handleExportRequest

Count and chunked row iteration are generic — built into tableExport/index.js. The frontend sends recordsTotal so the API usually skips COUNT when choosing sync vs async.

Step 1 — Export definition (columns + queryUtil reference)

Create backend/libraries/tableExport/definitions/my.module.js (filename = registry key, auto-registered):

javascript

"use strict";

const myModuleUtil = require("../../myModule.util");
const generals = require("../../generals.util");

module.exports = {
    queryUtil: myModuleUtil,
    mapRow: (row, ctx) => ({ ...row, _exportLang: ctx.exportLang }), // optional
    columns: [
        { headerKey: "datatable.column.id", format: (r) => r.id || "" },
        { headerKey: "datatable.column.name", format: (r) => r.name || "" },
        {
            headerKey: "datatable.column.createdAt",
            format: (r) => generals.formatExportDate(r.created_at),
        },
    ],
};

No manual registration in definitions/index.js — any *.js file in that folder (except index.js) is auto-imported; key = filename without extension.

Column shape:

Field	Required	Description
`headerKey`	Yes	i18n key for CSV header (resolved with user language)
`format`	Yes	`(row, translator) => string` — cell value

Use helpers from generals.util.js: formatExportDate, formatExportAmount, etc.

Step 2 — Query util (`buildQueryContext` only)

Add one method to the section util. It must return the same query configs the list endpoint already uses:

Return field	Purpose
`queryConfigIDs`	ORM config to paginate IDs (used for list + export chunking)
`buildQuery` or `buildQueryGuides`	`(ids) => orm config` for full row data
`empty`	`true` when filters match nothing (skip queries)

COD reference: backend/libraries/codGuides.util.js

javascript

async function buildMyModuleQueryContext(
    data,
    filterLocale = "",
    pagination = {},
    queryOptions = orm.READ_ONLY_QUERY_OPTIONS,
) {
    // Build queryConfigIDs + buildQuery from filters (same as list handler)
    return { queryConfigIDs, buildQuery, empty: false };
}

module.exports = {
    buildQueryContext: buildMyModuleQueryContext,
};

What iterateForExport does (generic, not per-section)

The export system paginates IDs in chunks of 5000 (tableExport/index.js):

Run queryConfigIDs with LIMIT offset, chunkSize → get page of IDs
Call buildQuery(ids) → fetch full rows for those IDs
Repeat until no more IDs

This avoids loading all IDs into memory on large exports. Section utils do not implement this — only buildQueryContext.

Count is optional for sync/async decision

When the frontend sends recordsTotal (DataTable always does), the backend uses it directly and does not run COUNT. A generic fallback count exists only when recordsTotal is omitted (e.g. direct API calls).

Step 3 — Controller integration (no new route)

In the existing list handler, delegate to table export when report=true:

javascript

const { isTableExport, TABLE_DEFAULTS } = require('../schemas/tableDefaults');
const tableExportUtil = require('../libraries/tableExport');

async getMyModuleList(request, h) {
    const data = request.query;
    const exportRequest = isTableExport(data.report);

    if (exportRequest) {
        data.start = TABLE_DEFAULTS.start;
    }

    // ... build query context, handle empty ...

    if (exportRequest) {
        return tableExportUtil.handleExportRequest(request, h, 'my.module');
    }

    // ... normal paginated list response ...
}

What is `handleExportRequest`?

tableExportUtil.handleExportRequest(request, h, resourceKey) is the single entry point controllers call to run an export. It lives in backend/libraries/tableExport/index.js and encapsulates everything the controller should not reimplement: parsing export params, resolving the user language, invoking the registry exporter, and shaping the HTTP response.

Signature:

javascript

handleExportRequest(request, h, resourceKey);
// request    — Hapi request (query, auth, pre handlers)
// h          — Hapi response toolkit (required to set status codes and headers)
// resourceKey — Registry key from definitions/index.js (e.g. 'cod.guides', 'my.module')

What it does internally:

Step	Action
1	Reads `report`, `format`, `recordsTotal` from `request.query` via `parseExportReport()`
2	Strips pagination/export params with `extractExportFilters()` — keeps only module filters (`company`, `status`, etc.)
3	Loads the requesting user's `languageId` for translated CSV headers
4	Calls `startExport()` with `resourceKey`, filters, `filterLocale` from route pre-handlers, and `knownRowCount` from `recordsTotal`
5	Returns the appropriate Hapi response (see below)

startExport() (called internally) resolves the registered definition, decides sync vs async, and either generates CSV inline or enqueues a Bull job:

Condition	Result
`rowCount === 0`	Throws `404` — no data
`rowCount <= SYNC_MAX_ROWS` and `format === 'csv'`	Sync — generates CSV in-process
`format !== 'csv'` (large dataset)	Throws `400` — async only supports CSV today
Otherwise	Async — inserts row in `admin_table_exports`, enqueues `table_export` job

HTTP responses returned by handleExportRequest:

Sync (200):

Content-Type: text/csv; charset=utf-8
Content-Disposition: attachment; filename="{filename}.csv"
Body: raw CSV string

Async (202):

json

{
    "async": true,
    "uuid": "...",
    "status": "pending",
    "row_count": 8500
}

Controller responsibility

The controller only needs to:

Detect export with isTableExport(report)
Optionally validate filters / empty context before delegating (see COD example below)
Call handleExportRequest(request, h, 'resource.key') and return its result

Do not call startExport, generateCsv, or queue jobs directly from the controller — that logic belongs inside tableExportUtil.

COD reference (cashOnDelivery.controller.js):

javascript

if (empty || !queryConfigIDs) {
    if (exportRequest) return false; // early exit when filters match nothing
    return { recordsTotal: 0, data: [] };
}

if (exportRequest) {
    return tableExportUtil.handleExportRequest(request, h, "cod.guides");
}

The early empty check avoids starting an export when the filter context is already known to be empty. handleExportRequest will still run its own row-count validation via the registered exporter if called.

Step 5 — Route validation

Spread shared table query fields from schemas/tableDefaults.js:

javascript

const { tableQueryFields } = require('../schemas/tableDefaults');

query: Joi.object({
    ...tableQueryFields,
    // module-specific filters
    status: Joi.string().optional(),
}),

Export query contract (flat params):

GET /my-module?report=true&format=csv&recordsTotal=1200&status=active&...

Param	Type	Purpose
`report`	`boolean`	`true` triggers export
`format`	`csv` \| `xlsx`	Export format (async supports CSV only)
`recordsTotal`	`number`	Known row count from UI (skips COUNT)
`start`, `length`	—	Ignored/stripped on export

Helpers:

javascript

isTableExport(report); // boolean
parseExportReport(report, query); // { format, recordsTotal } | null

Frontend integration

Step 6 — Service (same endpoint)

Do not create a separate export URL. Use the list endpoint with report=true:

javascript

import { isExportReport, resolveTableParams } from '../utils';

getMyModule(filters) {
    const params = resolveTableParams(filters);
    if (isExportReport(filters.report)) {
        return this.fetchReport('/my-module', params);
    }
    return this.axios.get('/my-module', { params });
}

BaseService.fetchReport() handles:

200 → { sync: true, blob, filename }
202 → { async: true, uuid, status, row_count }

Step 7 — DataTable `exportConfig`

In the module Table.vue:

vue

<DataTable
    :loadData="loadData"
    :exportConfig="{
        strategy: 'backend',
        format: 'csv',
        name: 'my-module',
        resourceKey: 'my.module',
    }"
/>

Field	Description
`strategy`	`'backend'` for new system; `'legacy'` for client-side export
`format`	`'csv'` or `'xlsx'`
`name`	Download filename
`resourceKey`	Must match backend registry key (used in exports panel label)

DataTable.vue automatically:

Sends report=true, format, recordsTotal (from table total)
Downloads sync CSV immediately
Registers async jobs in useExportsStore (no table loading, no polling)

Step 8 — Exports panel (global, already wired)

Header button → GlobalExportsDrawer → GET /exports (loaded only when panel opens).

No per-module setup required. Ensure resourceKey matches for correct labels.

Step 9 — i18n keys

Add to remote i18n catalog (see openspec/changes/table-export-cod-pilot/TRANSLATIONS.md):

UI toasts / panel:

datatable.export.inProgress
datatable.export.asyncStarted
datatable.export.downloadReady
datatable.export.failed
datatable.export.slackHint
tableExport.panel.*
tableExport.status.*

Slack notification (backend):

tableExport.slack.header
tableExport.slack.download
tableExport.slack.resource
tableExport.slack.resource.{resource_key} — e.g. tableExport.slack.resource.cod.guides

Column headers: reuse existing datatable.column.* keys in your definition.

API reference

Export via module endpoint

GET /cod?report=true&format=csv&recordsTotal=8500&company=42&...

Sync response (200):

Content-Type: text/csv; charset=utf-8
Content-Disposition: attachment; filename="cod-guides.csv"

Async response (202):

json

{
    "async": true,
    "uuid": "550e8400-e29b-41d4-a716-446655440000",
    "status": "pending",
    "row_count": 8500
}

Slack deep link (frontend)

Slack download buttons point to the main SPA with a query param — not a dedicated download page:

/dashboardV2?exportID={uuid}

When the layout loads with exportID:

useExportDeepLink (wired in MainHeader) detects the param
Opens GlobalExportsDrawer and downloads if the job is completed
Removes exportID from the URL via router.replace (avoids re-trigger on refresh)

There is no dedicated frontend route for exports — only the query param on any dashboard page (typically /dashboardV2?exportID={uuid} from Slack).

Export job API endpoints

Method	Path	Description
`GET`	`/exports`	List current user's exports (`?limit=20`)
`GET`	`/exports/{uuid}`	Job status
`GET`	`/exports/{uuid}/download`	Download completed CSV

::: note API vs SPA /exports/{uuid} is a backend API route (JSON status). The Slack link uses the frontend query param exportID, not this path. :::

Sync vs async decision

if (rowCount === 0)           → 404 No data
if (rowCount <= SYNC_MAX_ROWS && format === 'csv') → sync CSV
if (format !== 'csv')          → async not supported for xlsx yet
else                           → async job + Slack notification

Row count resolution order:

recordsTotal from query (frontend table total)
Fallback: exporter count() query

File map

backend/
├── libraries/
│   ├── tableExport/
│   │   ├── index.js              # Registry, startExport, handleExportRequest, CSV builder
│   │   └── definitions/
│   │       ├── index.js          # Registry map
│   │       └── cod.guides.js     # Example definition
│   └── codGuides.util.js         # Example query util
├── schemas/tableDefaults.js      # TABLE_DEFAULTS, tableQueryFields, isTableExport
├── controllers/exports.controller.js
├── routes/exports.routes.js
├── jobs/tableExport.jobs.js
├── scripts/admin-table-exports.sql
└── worker.js                     # table_export queue processor

frontend/client/src/
├── components/interface/Table/DataTable.vue
├── components/layout/GlobalExportsDrawer.vue
├── stores/exports.js
├── services/exports.service.js
├── services/base.service.js      # fetchReport()
└── composables/useExportRealtime.js

Testing checklist

Backend

[ ] Definition registered and resolves without error on server start
[ ] GET ?report=true with small dataset returns 200 CSV
[ ] GET ?report=true with recordsTotal > SYNC_MAX_ROWS returns 202 + uuid
[ ] Worker processes job → completed + S3 file
[ ] GET /exports/{uuid}/download returns file for owner only
[ ] Export queries use read-only connection
[ ] Slack DM sent on async completion (if user has Slack linked)

Frontend

[ ] Export button triggers download (sync) or toast + panel entry (async)
[ ] Table does not show loading spinner during async export
[ ] Header exports panel lists jobs after opening
[ ] Download from panel works for completed jobs

Unit tests

backend/tests/tableExport/
backend/tests/schemas/tableDefaults.test.js
backend/tests/codGuides/codGuides.util.test.js

Migrating from legacy export

Legacy	New
Separate `/export` route	Same `GET` with `report=true`
`exportGuides()` in service	`getGuides()` + `fetchReport()`
`exportFormat` / `exportName` props	`exportConfig` object
Client-side `loadFromServer(true)`	`exportConfig.strategy: 'backend'`
Stream response in service	`BaseService.fetchReport()`

Keep legacy strategy for modules not yet migrated:

javascript

exportConfig: { strategy: 'legacy', format: 'xlsx', name: 'report' }

Real-time notifications (future)

There is no WebSocket integration in the layout today. Async completion is notified via:

Slack DM (backend, on job complete)
Exports panel (GET /exports when user opens it)
EventBus hook — useExportRealtime listens for exportCompleted / exportFailed

To add WebSocket push later, call useExportsStore().handleRemoteUpdate(job) from the socket handler when the backend emits a completion event.

Quick checklist (copy for PRs)

[ ] definitions/{resource.key}.js: queryUtil, columns (+ optional mapRow) — auto-registered by filename
[ ] Section util: buildQueryContext → { queryConfigIDs, buildQuery, empty }
[ ] Controller: isTableExport → handleExportRequest(request, h, 'resource.key')
[ ] Route: ...tableQueryFields in Joi schema
[ ] Frontend service: isExportReport → fetchReport(samePath)
[ ] Table.vue: exportConfig { strategy: 'backend', format, name, resourceKey }
[ ] i18n: tableExport.slack.resource.{resource_key}
[ ] Manual test: sync + async flows

UI Elements

Forms & Overlays

Tables

Analytics

Organization Settings

Payment Requests

Pricing & Costs

Additional Services & Charges

Table Export System

Architecture

Design principles

Prerequisites

1. Database migration

2. Environment variables

3. Worker process

Step-by-step: Add a new export

Step 1 — Export definition (columns + queryUtil reference)

Step 2 — Query util (`buildQueryContext` only)

Step 3 — Controller integration (no new route)

What is `handleExportRequest`?

Step 5 — Route validation

Frontend integration

Step 6 — Service (same endpoint)

Step 7 — DataTable `exportConfig`

Step 8 — Exports panel (global, already wired)

Step 9 — i18n keys

API reference

Export via module endpoint

Slack deep link (frontend)

Export job API endpoints

Sync vs async decision

File map

Testing checklist

Backend

Frontend

Unit tests

Migrating from legacy export

Real-time notifications (future)

Quick checklist (copy for PRs)

Table Export System ​

Architecture ​

Design principles ​

Prerequisites ​

1. Database migration ​

2. Environment variables ​

3. Worker process ​

Step-by-step: Add a new export ​

Step 1 — Export definition (columns + queryUtil reference) ​

Step 2 — Query util (buildQueryContext only) ​

Step 3 — Controller integration (no new route) ​

What is handleExportRequest? ​

Step 5 — Route validation ​

Frontend integration ​

Step 6 — Service (same endpoint) ​

Step 7 — DataTable exportConfig ​

Step 8 — Exports panel (global, already wired) ​

Step 9 — i18n keys ​

API reference ​

Export via module endpoint ​

Slack deep link (frontend) ​

Export job API endpoints ​

Sync vs async decision ​

File map ​

Testing checklist ​

Backend ​

Frontend ​

Unit tests ​

Migrating from legacy export ​

Real-time notifications (future) ​

Quick checklist (copy for PRs) ​

Table Export System

Architecture

Design principles

Prerequisites

1. Database migration

2. Environment variables

3. Worker process

Step-by-step: Add a new export

Step 1 — Export definition (columns + queryUtil reference)

Step 2 — Query util (`buildQueryContext` only)

Step 3 — Controller integration (no new route)

What is `handleExportRequest`?

Step 5 — Route validation

Frontend integration

Step 6 — Service (same endpoint)

Step 7 — DataTable `exportConfig`

Step 8 — Exports panel (global, already wired)

Step 9 — i18n keys

API reference

Export via module endpoint

Slack deep link (frontend)

Export job API endpoints

Sync vs async decision

File map

Testing checklist

Backend

Frontend

Unit tests

Migrating from legacy export

Real-time notifications (future)

Quick checklist (copy for PRs)