How Database-Driven Integration Eliminates the Need for COBOL Copybooks
What if the biggest barrier to mainframe data integration wasn’t technical complexity, but the hours spent creating and maintaining COBOL copybooks? PropelZ’s JDBC input connector eliminates that bottleneck entirely.
Ask any mainframe professional about their biggest PropelZ adoption challenge, and you’ll hear the same story over and over: “Getting the COBOL copybook right is a nightmare.”
It shouldn’t be surprising. COBOL copybooks are finicky, environment-specific, and require deep knowledge of both the source data structure and COBOL syntax. Get one field definition wrong, and your entire data pipeline produces garbage. Miss a recent schema change, and you’re troubleshooting mysterious data corruption issues.
But what if you didn’t need copybooks at all?
The COBOL Copybook Problem
For those unfamiliar with mainframe data integration, here’s the challenge: PropelZ needs to understand the structure of your source data to transform it properly. With traditional VSAM and sequential files, that metadata comes from COBOL copybooks — text files that describe field names, data types, lengths, and positions.
Creating accurate copybooks requires:
- Deep Source Knowledge: You need to understand exactly how your source system structures its data, including any quirks or special formatting requirements.
- COBOL Expertise: Even if you understand the data, you need to express that understanding in proper COBOL copybook syntax.
- Ongoing Maintenance: When source schemas change (and they always do), your copybooks need updating too.
- Environment Coordination: Getting copybooks from development teams, validating them against production data, ensuring they match your specific data extraction requirements.
The result? What should be a straightforward “move this data from here to there” project becomes a complex copybook engineering exercise. IT teams spend weeks creating and testing copybooks before they can even start moving data.
The Database Revolution: Self-Describing Data
Databases changed everything about metadata management. Unlike flat files, databases inherently know their own structure. When you connect to any modern database — Oracle, PostgreSQL, SQL Server, even DB2 — the database can tell you:
- Every table and column name
- Data types and lengths for each field
- Constraints and relationships
- Indexes and keys
- Default values and nullability rules
This metadata is always current, always accurate, and always accessible through standard database APIs.
PropelZ’s JDBC input connector leverages this reality to eliminate copybook creation entirely.
How Automatic Metadata Discovery Works
Here’s what happens when you configure a JDBC input connector:
- Connect to Your Database: Provide connection details for any JDBC-compatible database — DB2 on z/OS, Oracle in the cloud, PostgreSQL on Linux, whatever your source happens to be.
- Write Your Query: Tell PropelZ what data you want with a standard SQL SELECT statement. This can be as simple as “SELECT * FROM CUSTOMER_TABLE” or as complex as a multi-table join with aggregations and filters.
- Automatic Metadata Extraction: PropelZ connects to the database and asks, “What’s the structure of the data this query will return?” The database responds with complete metadata — column names, types, lengths, everything PropelZ needs to understand the data.
- Pipeline Configuration: PropelZ uses that metadata to configure the transformation pipeline automatically. No copybook creation, no field mapping, no guesswork about data types or lengths.
The customer workflow goes from “spend weeks creating copybooks” to “write a SQL query and run the pipeline.”
Two Operational Modes for Maximum Flexibility
The JDBC input connector supports two different approaches, depending on your operational preferences:
- Metadata Generation Mode: PropelZ generates a COBOL copybook automatically based on the database metadata. If you have downstream processes that still expect copybooks, or if you want to review the field definitions before proceeding, this gives you a traditional copybook without the manual creation work.
- Direct Processing Mode: PropelZ uses the database metadata directly to process the data. No copybook generation step, no intermediate files — just straight from database to your configured output targets.
Both modes ensure the metadata is accurate, current, and automatically maintained. When your database schema changes, PropelZ adapts automatically.
Real-World Impact: Database Migration Made Simple
Consider a typical scenario: migrating a DB2 table to Snowflake for analytics.
Traditional Approach:
- Analyze the DB2 table structure
- Create a COBOL copybook describing every field
- Test the copybook against sample data
- Debug field alignment and data type issues
- Configure PropelZ with the copybook
- Run the migration pipeline
- Validate results and troubleshoot any data issues
JDBC Input Connector Approach:
- Write a SQL query: “SELECT * FROM DB2_TABLE”
- Configure Snowflake as the output target
- Run the migration pipeline
That’s it. PropelZ handles all the metadata discovery automatically. No copybook creation, no manual field mapping, no guessing about data types or formats.
Beyond Simple Migration: Complex Query Support
The real power emerges when you need more than basic table copying. Because the metadata discovery works with any SQL query, you can:
- Join Multiple Tables: Create a single PropelZ pipeline that combines data from multiple database tables, with metadata automatically derived from your JOIN query.
- Apply Business Logic: Include calculated fields, aggregations, and filters in your SQL. PropelZ will understand the metadata for your computed columns just as well as your source columns.
- Handle Complex Transformations: Use SQL to do initial data transformations at the database level, then let PropelZ handle format conversions and routing to multiple targets.
The metadata discovery adapts to whatever your SQL query produces, regardless of complexity.
The Strategic Advantage: Any Database, Anywhere
This approach works with any JDBC-compatible database, which means virtually every modern database system:
- Mainframe: DB2 for z/OS, IMS databases
- Traditional Enterprise: Oracle, SQL Server, PostgreSQL, MySQL
- Cloud Native: Amazon RDS, Google Cloud SQL, Azure SQL Database
- Analytics Platforms: Snowflake, Databricks, Teradata
- Specialized Systems: Any database with a JDBC driver
Customers can use the same PropelZ skills and operational procedures regardless of their source database platform. A team comfortable with DB2 migrations can apply the same approach to Oracle or PostgreSQL migrations.
Operations Benefits: Less Maintenance, More Reliability
Automatic metadata discovery isn’t just about easier initial setup — it fundamentally changes the operational characteristics of your data pipelines:
- Self-Maintaining Pipelines: When source schemas change, PropelZ adapts automatically. No more surprise pipeline failures because someone added a column and forgot to update the copybook.
- Faster Troubleshooting: Metadata mismatches are eliminated as a potential cause of data issues. If PropelZ is processing the data, you know the metadata is correct.
- Easier Environment Promotion: Moving from test to production becomes trivial. The same SQL query works in any environment, with metadata automatically adapted to the actual database schema.
- Reduced Expertise Requirements: Teams don’t need deep COBOL copybook knowledge to create PropelZ pipelines. If they can write SQL, they can configure PropelZ.
The Future: Extending Beyond Databases
The automatic metadata discovery model opens doors to other self-describing data sources. JSON APIs, cloud storage with schema registries, message queues with structured formats — any data source that can describe its own structure becomes a candidate for similar treatment.
We’re not just eliminating COBOL copybooks. We’re establishing a pattern for metadata-driven integration that can expand to whatever data sources your enterprise needs to connect.
What This Means for Your Team
If COBOL copybook creation has been a barrier to PropelZ adoption in your environment, that barrier just disappeared. Your database migration projects can start with SQL queries instead of copybook engineering sessions.
Your team can focus on the business logic of data transformation instead of the mechanical details of field definitions and data type mappings.
And perhaps most importantly, your data pipelines become more reliable and easier to maintain, because the metadata they depend on stays current automatically.
Ready to see automatic metadata discovery in action? Schedule a demonstration of PropelZ’s JDBC input connector and discover how eliminating copybooks can accelerate your database integration projects.
Next Steps
- Schedule a PropelZ briefing.
- Learn more about PropelZ 2.0.
- Watch a demo of PropelZ.
- Try PropelZ.
Learn More
- Visit our Customer Briefing Center.
- Review all VirtualZ Use Cases, Thought Leadership papers, and Solution Briefs.
- Explore our YouTube channel, podcasts, and blog.
- Still have questions? Contact us.
