Case Study
·2019
Bank Muamalat Malaysia
Processing 10TB+ of Bank Statements With 50 Parallel Workers
High-throughput PDF statement generation system for Bank Muamalat — ingesting terabytes of mainframe spool data, parsing by document type, and rendering millions of formatted PDF statements within tight regulatory windows using 50 concurrent workers on Linux infrastructure.
10
TB+ Storage Processed
50
Concurrent Workers
99.9%
Generation Success Rate
48h
Peak Processing Window
Bank Muamalat
Statement PDF Generation System
The Challenge
Terabytes of statements. Hours to deliver. Zero margin for error.
Bank Muamalat's statement generation process relies on mainframe-produced spool files — raw text data containing customer account information, transaction histories, notices, and regulatory correspondence. During peak periods such as year-end cycles, the volume of spool data reaches terabytes, with an estimated total storage footprint exceeding 10TB and growing. Every document must be transformed into a properly formatted PDF — with the correct template, branding, and data mapping for its document type — and delivered to customers within days. The bank needed a system capable of processing this volume within a narrow regulatory window while guaranteeing that not a single statement is missed.
Terabyte-Scale Volume
End-of-year cycles produce terabytes of raw spool data that must be converted to formatted PDFs within a 48-hour regulatory window — any delay risks compliance violations.
Multiple Document Formats
Bank statements, notices, letters, and regulatory documents each require distinct PDF templates with different layouts, headers, footers, and data mapping rules.
Zero Tolerance for Missing Statements
Every single customer statement must be generated and verified. A single missing document could trigger regulatory scrutiny — the system must guarantee 100% completion.
The Solution
50-worker parallel engine with Redis-backed job orchestration
Advisory Apps architected a high-throughput document processing pipeline hosted on Linux infrastructure. The system operates in three stages: first, an ingestion layer reads raw spool files from the bank's internal mainframe and breaks them down into individual records loaded into MySQL. Second, parsed records are indexed in Redis for sub-millisecond job distribution across a pool of 50 concurrent worker threads. Third, each worker applies the correct PDF template based on document type, renders the formatted statement, and writes it to storage. A real-time job tracker continuously monitors every file from ingestion to completion, guaranteeing zero missing statements before the delivery deadline.
Processing Architecture
Three-stage pipeline: spool ingestion, Redis-orchestrated parallel processing, and verified PDF output — with 50 concurrent workers processing terabytes within the regulatory window.
Bank Mainframe
Raw spool files • Text data
Admin Panel
Template config • Monitoring
Stage 1: Spool Ingestion & Parsing
Read spool files → Break down by document type → Load records into MySQL
Stage 2: 50 Concurrent PDF Workers
Pull job from Redis → Apply template by doc type → Render PDF
50 parallel threads processing simultaneously
Job Tracker
Expected vs generated count verification
PDF Storage
10TB+ generated statements
Archive System
Bank's document archive & retrieval
MySQL
Parsed records • Metadata
Redis
Job queue • Fast lookup
Linux Server
On-premise • Bank infrastructure
Processing Pipeline
How raw mainframe spool data transforms into verified, formatted PDF statements — from file ingestion through to archive delivery.
Step 1
Ingest Spool Files
Raw text data received from bank mainframe via internal file system mount
Step 2
Parse & Load to DB
Break down spool files by document type, load structured records into MySQL
Step 3
Index in Redis
Link records in Redis for sub-ms job distribution to 50 worker threads
Step 4
Render PDFs
50 workers apply templates per doc type, generate formatted PDF statements
Step 5
Verify & Archive
Job tracker confirms 100% completion, PDFs pushed to bank archive system
Parallel PDF Generation Engine
50 concurrent worker threads processing spool files simultaneously — ingesting raw text, parsing by document type, applying templates, and rendering production-grade PDFs at scale.
Admin Template Management
Web-based admin panel for configuring PDF templates per document type — bank staff can modify layouts, branding, and data fields without developer intervention.
Job Tracking & Verification
Real-time job tracker monitors every file loaded into the system — cross-referencing expected vs generated counts to guarantee zero missing statements before the delivery deadline.
Implementation Timeline
Phase 1
Architecture & Prototyping
Phase 2
Core PDF Engine
Phase 3
Admin Panel & Templates
Phase 4
Integration & Load Testing
Phase 5
UAT & Production Deployment
Methodology
Engineered for regulatory-grade reliability
The system was designed from the ground up for the specific constraints of banking document processing: zero tolerance for missing statements, terabyte-scale throughput, and strict regulatory delivery windows. Every architectural decision — from Redis-backed job queues to 50-worker parallelism — was driven by the requirement to process peak volumes within hours, not days.
Infrastructure Design
Architected Linux-hosted multi-worker system with Redis-backed job queues, MySQL metadata store, and file system mount points for bank spool ingestion.
Engine Development
Built parallel PDF generation engine with 50 concurrent workers, spool file parser, template renderer, and document-type routing logic.
Integration & Security
Integrated with bank's internal mainframe for spool file delivery and external archiving system for long-term document storage and retrieval.
Load Testing & Hardening
Simulated peak end-of-year volumes with terabytes of test data to validate throughput, error recovery, and job completion guarantees under production conditions.
Key Features Delivered
50-Worker Parallel Processing
Concurrent worker pool that scales spool-to-PDF conversion horizontally — each worker independently parses, renders, and writes PDFs without blocking others.
Spool File Ingestion & Parsing
Automated intake of raw mainframe spool files, breaking them down by document type and customer, loading structured records into MySQL for processing.
Redis-Backed Job Queue
All parsed records indexed in Redis for sub-millisecond lookup — workers pull jobs from the queue with atomic dequeue operations ensuring no duplicate processing.
Template Management Admin Panel
Web-based interface for bank staff to configure and preview PDF templates per document type — statements, notices, letters — without code changes.
Job Tracker & Completion Verification
Real-time dashboard tracking every job from spool ingestion to PDF output — cross-referencing expected file counts against generated files to guarantee zero gaps.
Archive System Integration
Generated PDFs automatically pushed to the bank's archiving system — enabling document browsing, search, and retrieval for compliance and customer service.
The Results
Regulatory-grade performance at terabyte scale
10+
TB Storage Processed
Terabytes of raw spool data ingested, parsed, and converted to formatted PDF statements — volume growing annually.
50
Concurrent Workers
Parallel processing threads running simultaneously on Linux infrastructure to meet tight regulatory windows.
99.9%
Generation Success Rate
Job tracker verification ensures near-perfect completion — every statement accounted for before delivery deadline.
48h
Peak Processing Window
End-of-year terabyte-scale volumes processed within the regulatory compliance window for timely customer delivery.
Modules Used
Built with production-ready modules
This project leverages our library of battle-tested modules — each independently scoped, tested, and deployed across multiple client projects.
Document Management
File storage with folder hierarchy, version control, access permissions, and preview
Data Export & Reporting
One-click export to Excel, CSV, and PDF with custom templates and scheduled reports
Email Template Engine
Visual email builder with dynamic variables, SMTP integration, and delivery tracking
User Management
Unified user profiles, role-based access, and secure authentication
Dashboard & Analytics Builder
Drag-and-drop dashboard with charts, KPIs, real-time widgets, and role-based views
Activity & Audit Log
Immutable system-wide activity logging with user attribution, filtering, and compliance exports
Conclusion
Banking-grade document processing at scale
The Bank Muamalat statement generation system demonstrates Advisory Apps' capability in high-throughput, mission-critical document processing. By parallelising the workload across 50 concurrent workers with Redis-backed job orchestration, the system transforms terabytes of raw mainframe spool data into formatted, template-driven PDF statements — all within the tight regulatory windows that banking compliance demands.
The job tracking and verification layer ensures zero tolerance for missing statements — every file ingested is cross-referenced against generated output before the delivery deadline. The admin panel gives bank staff direct control over PDF templates without developer dependency, while integration with the bank's archiving system provides long-term document retrieval for compliance and customer service.
Future Outlook
- Horizontal scaling to 100+ workers for growing statement volumes as the bank expands
- Digital delivery channel integration for e-statements via email and mobile banking app
- Additional document types including regulatory correspondence and personalised marketing inserts
Services Used
Capabilities behind this project
Want similar results for your business?
Let's discuss how we can build a custom solution tailored to your needs.
Get a Free Consultation