“We need to migrate our virtual tape library to AWS S3. How long will it take?”
“How many files are we talking about?”
“About 200,000 datasets. Maybe more.”
This conversation happens more often than you might expect. Enterprises with decades of mainframe operations have accumulated massive amounts of archived data — millions of datasets across thousands of tapes, all needing to move to modern storage platforms as part of cloud migrations or cost optimization initiatives.
Standard PropelZ workflows handle individual datasets efficiently, but at this scale, a different approach becomes necessary. Managing hundreds of thousands of separate operations — even streamlined ones — presents unique operational challenges.
PropelZ Multiplexer addresses these scale-specific challenges by automating the coordination and execution of massive parallel migrations.
When Scale Requires Specialized Tools
For typical data migrations involving dozens or even hundreds of datasets, PropelZ’s standard workflow provides an efficient, manageable approach. The Multiplexer becomes valuable when dealing with very large-scale projects — typically 10,000+ datasets — where the coordination overhead becomes significant.
Large-scale data migrations face several practical problems unique to their scale:
- Operational Complexity: Nobody wants to manually configure 100,000 individual file transfer jobs. The administrative overhead would require a dedicated team for months.
- Resource Management: Running migrations one at a time is painfully slow. Running too many simultaneously can overwhelm network bandwidth, storage systems, or processing capacity.
- Progress Tracking: How do you monitor the status of hundreds of thousands of transfers? Which ones succeeded? Which failed and need retry? Where are you in the overall process?
- Error Handling: At scale, some transfers will fail due to network glitches, temporary resource constraints, or data issues. How do you identify and retry failed transfers without starting from scratch?
- Scheduling Coordination: Large migrations need to run during specific time windows to avoid impacting production systems. How do you coordinate hundreds of thousands of transfers within those constraints?
These scale-specific challenges require specialized tooling, which is where the Multiplexer provides value for massive migration projects.
How PropelZ Multiplexer Works
The Multiplexer changes the operational model entirely. Instead of managing individual file transfers, you manage an inventory-driven migration process:
- Step 1: Generate Inventory: Use your existing storage management tools — CA Vantage, Broadcom storage reporting, or custom queries — to generate a list of datasets to migrate. This could be 100,000 entries or more.
- Step 2: Configure Parallelism: Tell the Multiplexer how many transfers to run simultaneously. This depends on your infrastructure capacity — might be 50, 100, or even 200 parallel transfers on a well-configured modern mainframe.
- Step 3: Start Migration: The Multiplexer takes care of everything else — scheduling transfers, managing the parallel execution queue, monitoring progress, handling failures.
- Step 4: Monitor Progress: A single status file shows you exactly where you are in the migration, which transfers succeeded, which failed, and what’s currently running.
The customer experience goes from “manage 100,000 individual jobs” to “provide a list and monitor one status file.”
Intelligent Queue Management
The Multiplexer’s queue management handles the complexity of coordinating massive parallel operations:
- Dynamic Scheduling: As transfers complete, new ones start automatically. You always have the configured number of transfers running without manual intervention.
- Resource Awareness: The system monitors infrastructure utilization and can throttle transfer rates if resource constraints develop.
- Priority Management: Critical datasets can be prioritized to migrate first, while less important data migrates during off-peak periods.
- Failure Isolation: If individual transfers fail, they don’t impact other operations. Failed transfers are queued for retry while successful transfers continue.
Comprehensive Status and Monitoring
Traditional batch operations often provide minimal visibility into progress. The Multiplexer maintains detailed status information for every transfer:
- Real-Time Progress: See exactly how many transfers have completed, how many are running, how many are queued, and how many failed.
- Individual Transfer Details: For each dataset, track start time, completion time, bytes transferred, any error conditions, and retry attempts.
- Performance Metrics: Monitor transfer rates, identify bottlenecks, optimize parallel execution based on actual performance data.
- Failure Analysis: Failed transfers are logged with detailed error information, making troubleshooting straightforward.
This visibility transforms large-scale migrations from black-box operations into transparent, manageable processes.
Enterprise Integration
The Multiplexer doesn’t operate in isolation — it integrates with your existing mainframe automation infrastructure:
- Console Message Integration: Status updates can be written to the z/OS console, triggering existing automation tools like Ops/MVS for alerting, escalation, or corrective actions.
- Help Desk Integration: Failed transfers can automatically open help desk tickets with detailed error information, ensuring operational issues get proper attention.
- Notification Systems: Success and failure notifications can integrate with existing messaging systems — email, SMS, Slack, or whatever your operations team uses.
- Retry Logic: Automated retry policies can attempt failed transfers multiple times before escalating to human intervention.
This integration means large-scale migrations work within your existing operational processes rather than requiring entirely new procedures.
Performance and Throughput
Performance varies based on infrastructure, but typical enterprise configurations see impressive throughput:
- Modern Mainframe with High Bandwidth: 100-200 parallel transfers, processing several thousand datasets per hour.
- Network-Constrained Environments: 20-50 parallel transfers, still achieving migration rates that would take years with standard individual transfer management.
- Mixed Workloads: The Multiplexer can run during business hours at reduced parallelism, then increase throughput during off-peak periods.
A typical virtual tape library migration of 200,000 datasets that might take months with standard individual transfer management can complete in weeks with the Multiplexer’s parallel processing and automated coordination.
Cost Justification
The Multiplexer’s value becomes clear when you calculate the operational costs of alternative approaches:
- Manual Approach: Configuring and managing 100,000+ individual transfers would require dedicated staff for months, with high error rates and extensive troubleshooting.
- Custom Scripting: Building custom automation for large-scale transfers requires significant development effort, testing, and ongoing maintenance.
- Vendor Services: Migration services from consulting companies can cost hundreds of thousands of dollars for large-scale projects.
The Multiplexer typically pays for itself within the first few weeks of a large migration project through reduced labor costs and accelerated project completion.
Beyond Migration: Additional Use Cases
While virtual tape library migrations are the most common use case, the Multiplexer enables other large-scale operations:
- Application Modernization: Move thousands of datasets from legacy applications to modern storage as part of application retirement or modernization projects.
- Compliance Archiving: Bulk migration of historical data to cloud storage for long-term retention and compliance requirements.
- Disaster Recovery: Replicate large volumes of critical data to geographically distributed storage for disaster recovery purposes.
- Cost Optimization: Move infrequently accessed data from expensive mainframe storage to cost-effective cloud alternatives.
- Data Center Consolidation: Transfer data between mainframe systems as part of data center consolidation or hardware refresh projects.
Getting Started with Multiplexer
Organizations new to large-scale automated migrations can start with smaller projects to build confidence:
- Phase 1: Use the Multiplexer for a subset of data — perhaps 1,000-5,000 datasets — to validate the process and optimize parallelism settings.
- Phase 2: Expand to larger dataset collections, fine-tuning automation integration and monitoring procedures.
- Phase 3: Execute full-scale migrations with confidence in the process, automation, and monitoring capabilities.
- Phase 4: Leverage the Multiplexer for ongoing operational tasks — regular archiving, compliance data management, or routine data movement.
Conclusion
If your organization is planning large-scale mainframe data migrations, the Multiplexer changes the economics and timeline of these projects:
- Faster Completion: What might take months with individual transfers can complete in weeks with parallel processing.
- Lower Risk: Comprehensive status tracking and automated retry logic reduce the risk of migration failures or data loss.
- Reduced Operational Overhead: Inventory-driven processing requires minimal ongoing management compared to individual transfer coordination.
- Better Resource Utilization: Parallel processing makes efficient use of your infrastructure capacity rather than leaving resources idle.
The result: large-scale migrations become routine operational tasks rather than major project undertakings.
Next Steps
- Learn more about PropelZ 2.0.
- Schedule a PropelZ briefing.
- Watch a demo of PropelZ.
- Try PropelZ.
Learn More
- Visit our Customer Briefing Center.
- Review all VirtualZ Use Cases, Thought Leadership papers, and Solution Briefs.
- Explore our YouTube channel, podcasts, and blog.
- Still have questions? Contact us.
