SSL Support in IPv6/VSE

SSL Support in IPv6/VSE

Barnard Software, Inc. is pleased to announce that their IPv6/VSE product now provides SSL (Secure Socket Layer) support! Our new SSL support became available with GA Build 252. GA Build 252 is now available on our download page.

The initial release of IPv6/VSE did not contain any cryptographic facilities. During the 2011 GSE conference last October, in Berlin Germany,. BSI made a Statement of Direction indicating the SSL and IPSec features would be added to IPv6/VSE. 

IPv6/VSE SSL facilities will be released in two phases. 

Phase One will provide the BSI SSL Proxy Server that runs in a z/VSE partition. The BSTTPRXY server application will provide support for telnets (TN3270E over SSL), SMTPS (SMTP over SSL), HTTPS (HTTP over SSL) and FTPS (Implicit FTP over SSL) and more. IPv6/VSE also provides the SSL GSK API for use by customer and 3rd party vendor software. The GSK API is available using EZASOKET/EZASMI or LE/C interfaces. The only restriction is that this API is currently only available to LE batch applications. 

For IPv6/VSE users running on z/VSE 5.1 (or higher) support is provided for CPACF hardware cryptographic instructions and Crypto-Express adapters to improve performance.

Phase Two will remove the LE requirement providing support for both batch and CICS applications We also plan to provide explicit (Built in) SSL support for BSTTVNET (TN3270E server), BSTTMTPC (SMTP Client), BSTTFTPS (FTP server) and BSTTFTPC (batch FTP client).

Phase One of IPv6/VSE's SSL support is now available!

System Requirements

IPv6/VSE's Secure Sockets Layer (SSL) support uses the IBM z/VSE OpenSSL1.0.0 port. The IBM OpenSSL port was done using z/VSE C/VSE compiler permitting support for back level versions of z/VSE and VSE/ESA. The OpenSSL port became available with z/VSE 5.1. BSI IPv6/VSE customers have special options to enable IPv6/VSE's SSL support on back level releases of VSE/ESA and z/VSE. 

CPACF

The IBM OpenSSL port will take advantage of the System z's CP Assist for Cryptographic Function (CPACF) when running under z/VSE 5.1 (or higher). These CP Assists dramatically reduce CPU overhead involved when using cryptographic functions. Users of the z890, z990, z9, z10, z114, z196 (and newer processors) should have the no-charge enablement feature installed. This feature must be ordered to enable CPACF. z/VSE systems without CPACF will use system CPU resources to perform cryptographic functions.

Crypto Express Adapters

The IBM OpenSSL port will take advantage of the System z Crypto Express adapters supported by z/VSE 5.1 (or higher). Usage of Crypto Express adapters dramatically reduces the amount of CPU required to establish a secure socket connection. z/VSE 51 (or higher) systems without Crypto Express adapters  (or back level systems) will use system CPU resources to establish a secure socket connection.

Restrictions

Currently the IPv6/VSE SSL suport is in Phase One. Any application can make use the IPv6/VSE SSL Proxy server. This includes both batch and CICS, LE and non-LE applications. Also, any user or 3rd party LE conforming batch application can use the EZASOKET/EZASMI or LE/C GSK SSL API provided by IPv6/VSE. Non-LE based batch applications, CICS/VSE and CICS TS applications are currently not supported by the Phase I GSK SSL API. This restriction will be removed in a future build.

BSTTPRXY SSL Proxy Server

The BSTTPRXY SSL proxy server allows non-SSL server applications running in z/VSE to accept connections from SSL based client applications. Or, BSTTPRXY will allow non-SSL client applications to connect to SSL based server applications.

Proxy Types

Source	Destination
Accept clear text connection	Proxy to clear text connection
Accept SSL connection	Proxy to SSL conneIPv6/VSE SSL Supportction
Accept clear text connection	Proxy to SSL connection
Accept SSL connection	Proxy to clear text connection
Accept IPv4 or IPv6 connection	Proxy to IPv4 or IPv6 connection

Proxy Examples

Accept a clear text connection from BSTTMTPC on port 25 and proxy the connection to an SMTP server listening on SSL port 465. This type of connection is commonly called smtps.

Accept an SSL connection on port 443 and proxy the connection to CICS TS Web Services on clear text port 80. This type of connections commonly called https.

Accept an SSL connection on port 992 and proxy the connection to the BSTTVNET TN3270E server on clear text port 23. This type of connection is commonly called telnets.

Accept a clear text connection from BSTTFTPC on port 21 and proxy the connection to an FTP server listening on SSL port 990. This type of connection is commonly called ftps.

Accept an SSL connection on port 990 and proxy the connection to the BSTTFTPS FTP server on clear text port 21. This type of connection is commonly called ftps.

IPv6/VSE SSL Support

There you have it. IPv6/VSE SSL support is now available and supports SSL sockets for virtually any z/VSE application.

IPv6/VSE

IPv6/VSE

IPv6 news is everywhere these days. The availability of z/VSE 4.2.2 (and newer releases) provides the foundation of z/VSE's IPv6 support. Combine this with IBM's new TCP/IP solution, IPv6/VSE, which provides a single product solution for both IPv4 and IPv6 TCP/IP communications and you have a powerful way to take z/VSE into the future.


You are probably already hearing about IPv6 on z/VSE. Why? Your ISP is converting to IPv6 and makes it available to you. With the IPv4 address block exhaustion, new blocks of IPv4 addresses are now almost impossible to get. Your customers or business partners require IPv6 connections. Government organizations may only allow IPv6 capable devices and applications to participate in bidding. The US Department of Defense (DoD), for example, only allows products that are on the “Unified Capabilities Approved Products List” (UC APL) for its advertised bidding. This list is used by procurement offices in the DoD and the U.S. Federal agencies for ongoing purchases and acquisitions of IT equipment. Also, keep this in mind, as new standards are developed IPv6 is usually mandated. For example, 4G mobile devices are mandated to use IPv6 and IPv4 support is deprecated.


IPv6 support in z/VSE began with z/VSE 4.2.2.

IBM IPv6/VSE Version 1 Release 1

IBM licensed IPv6/VSE from Barnard Software, Inc.for worldwide distribution as their TCP/IP product solution for the future (IPv6/VSE is a registered trademark of Barnard Software, Inc.). IBM's IPv6/VSE announcement (210-066) was made on April 6, 2010. The availability of IBM's IPv6/VSE allows z/VSE users to participate in an IPv6 network and bring the benefits of IPv6 functionality to z/VSE users.


While the product is named IPv6/VSE, the product supports both IPv4 and IPv6 communications. IPv6/VSE provides a full-function IPv4 stack and applications as well as a full-function IPv6 stack and applications. Both TCP/IP stacks (IPv4 and IPv6) can be run together, individually or even stand alone. IBM's IPv6/VSE supports both IPv4 and IPv6 protocols and has been available since May 28, 2010.



IPv6/VSE IPv4 and IPv6 Support

IPv6/VSE Application	IPv4 Support	IPv6-Enabled	Description
BSTTFTPS	Yes	Yes	FTP Server
BSTTFTPC	Yes	Yes	FTP Client
BSTTVNET	Yes	Yes	TN3270E Server with DIRECT/LPR printing
BSTTNTPS	Yes	Yes	Network Time Protocol Server
BSTTNTPC	Yes	Yes	Network Time Protocol Client
BSTTSLOG	Yes	Yes	System Logger Client
BSTTMTPC	Yes	Yes	Mail Transport Protocol Client
BSTTREXC	Yes	Yes	Remote Execution Client
BSTTPING	Yes	Yes	Batch PING Utility
BSTTGZIP	Yes	Yes	GZIP Data Compression
REXX Automation	Yes	Yes	REXX Automation Support

IPv6/VSE Support for IBM Applications

Application	Supported by IPv6/VSE
CICS TS Web Services	Yes
CICS TS Listener	Yes
VTAPE	Yes
Connector Server	Yes
DB2 Server and DB2 Client	Yes
MQSeries	Yes
MQ Client	Yes
PSF/VSE	Yes
Linux Fast Path	Yes
VSAM Redirector	Yes
VSE Health Checker	Yes
VSE Script Server	Yes
VSE Navigator	Yes

Virtually all 3^rd party software vendors are now working with IPv6/VSE and support their customers using IPv6/VSE with their products.

IPv6 Programming Interfaces

Existing APIs have been extended. The Assembler SOCKET macro API has been extended to support IPv6 addressing and 64-bit virtual storage addresses. The EZASOKET and EZASMI API have also been extended. EZASOKET/EZASMI support for GETADDRINFO, FREEADDRINFO, NTOP and PTON with the new address family of AF_INET6 is available. Of course, the LE/C Socket API has been IPv6-enabled too.


The IBM IPv6/VSE Programming Guide has all the details on the IPv6-enablement features in these API's.

SSL and IPSec

During the 2011 GSE conference (October 2011 in Berlin Germany) Barnard Software, Inc. (BSI) issued a Statement of Direction that indicated BSI would make available both SSL and IPSec solutions for IPv6/VSE in a future release. At the time of the writing of this article the SSL support was undergoing beta test. Availability of these additions to IPv6/VSE is expected to become generally available (GA) during 2012.

Summing it up

There you have it. z/VSE is IPv6 ready with IBM's IPv6/VSE. IBM's IPv6/VSE provides both a full-function IPv4 stack and applications as well as a full-function IPv6 stack and applications. IPv6/VSE is a single product providing a complete solution, TCP/IP stacks, IPv6-enabled applications and programming interfaces. With IPv6/VSE your z/VSE system is IPv4 and IPv6 ready to go.

Optimizing Hipersockets in z/VSE

Let's review a bit about the z/VSE Hipersockets network interface ...

Hipersockets is a synchronous method of transferring data. The sending data transfer does not complete until the data is received by the destination. This is a big factor. If the destination system is busy this can have a large impact on throughput.

Hipersockets is very much is a CPU function. In general, the faster the CPU, the faster the Hipersocket interface can operate. Limiting CPU on an LPAR or virtual machine (VM) limits throughput.

Are all Hipersocket hosts using the same large MTU? The maximum MTU size of a Hipersockets network interface is determined by the frame size specified in the IOCP Configuration of the IQD CHPID through the OS parameter.

OS Frame Size vs. Maximum MTU:
OS = 00 : MTU =  8KB
OS = 40 : MTU = 16KB
OS = 80 : MTU = 32KB
OS = C0 : MTU = 64KB

Choose the OS parameter carefully. The value you choose will be transferred for every transfer even if there is only 1 byte to transfer.

And, now to continue ...

If you read my last posting about TCP/IP throughput rates you know that the first limitation to throughput is the size of the TCP/IP stack's transmit buffer. BSI customers can adjust this value using the SBSIZE stack startup parameter. Setting the SBSIZE to 65024 (SBSIZE 65024, the maximum value) may help throughput using a Hipersocket network interface. Warning: Changing this value may actually reduce throughput. Your mileage will vary and testing is required to optimize this value. Do not change this value without contacting BSI first.

Set the MTU size of the Hipersocket network interface to 57216. This is the maximum value allowed and is specified on the LINK statement in the stack startup commands.

Now we have ...

A 64K transmit buffer

A 56K MTU

The TCP Receive Window and Maximum Segment Size are not directly controllable by customers. However, when using Hipersockets, the TCP Receive Windows should be 64K and the MSS will be slightly less than the MTU size. All of this provides the framework for having the maximum possible data sent in each packet.

In my posting on 'Care and Feeding of Your z/VSE TCP/IP Stack' I talk about keeping the stack fed. You keep the stack fed by making sure that when it wants data to send, there is data in the transmit buffer. One way to do this is to use large application buffers (for example 1MB in size). Fill a large buffer with data and issue a single socket send request to transfer the data. When you are using a Hipersocket network interface with a 64K transmit buffer and a 56K MTU, using a 64K buffer to send data into the stack is not as efficient as using a 1M send buffer.

When you are using the BSI BSTTFTPC batch FTP application, you can tell BSTTFTPC to use large buffer socket send operations by specifying
// SETPARM MAXBUF=YES in the BSTTFTPC JCL.

Are there any other limitations on throughput? Maybe.

The IBM IJBOSA driver is used to access the Hipersocket network interface. Is IJBOSA a limitation to performance? In general, no. IJBOSA's design is good. However, understanding some of the workings of the IJBOSA routine can help you to understand overall Hipersocket performance picture.

The TCP/IP stack can send multiple packets into the IJBOSA routine in a single call. Currently the BSI TCP/IP stack supports sending up to 8 packets in a single call. Why was the number 8 chosen? Because IJBOSA provides for a default of 8 buffers (I know of no way to change the number of IJBOSA buffers used).

If the stack attempts to send packets to IJBOSA and all of the its buffers are in use, the IJBOSA routine returns a return coding indicating the Hipersocket device is busy. At this point the TCP/IP stack must wait and send the packets again. The BSI TCP/IP stack does this by simply dropping the packets and allowing normal TCP retransmission to resend the packets in 10ms to 20ms. This retransmission delay allows the buffer busy condition to clear. You can see increasing the number of IJBOSA buffers may help throughput by reducing retransmission delays. However, this is only true if the machine receiving the data does so in an efficient and timely fashion. If the receiving machine is busy for some reason increasing the number of IJBOSA buffers available will not help transfer rates.

Because TCP throughput is limited by the size of the TCP Receive Window (64K), only 2 buffers are likely being used for each bulk data socket. The IJBOSA's 8 buffers provides for 4 high speed bulk data transfer sockets operating at the same time. I suspect that for most customers, 8 buffers is plenty.

Still, the question remains, how can I tell if Hipersocket busy conditions are a problem for me?

Well, first, look for this message on the z/VSE console ...
R1 0497 0S39I ERROR DURING OSA EXPRESS PROCESSING,REASON=0046 CUU=xxxx
This message is output by the IJBOSA routine the first time a busy condition is encountered. Remember, this message is output only once and will let you know a busy condition occurred.

Since the BSI TCP/IP stack's track TCP retransmission activity, the TCP retransmit counter can be used as a proxy for determining the number of Hipersocket busy conditions. To display the TCP/IP stack statistics you can use the IP LOGSTATS command.

For example, 

- MSG BSTTINET,D=IP LOGSTATS
- Run your job that uses the Hipersocket network interface
- MSG BSTTINET,D=IP LOGSTATS
- MSG BSTTINET,D=SEGMENT * $$ LST CLASS=...

Look in the BSTTINET SYSLST log output and locate these messages ...

From the 1st IP LOGSTATS command ...
01-Oct-2010 10:33:15 F6 BSTT613I   TcpOutSegs:          0
01-Oct-2010 10:33:15 F6 BSTT613I   TcpRetransSegs:     0

From the 2nd IP LOGSTATS command ...
01-Oct-2010 10:33:15 F6 BSTT613I   TcpOutSegs:    1563218
01-Oct-2010 10:33:15 F6 BSTT613I   TcpRetransSegs:   4493 

The TCP retransmission rate is about 0.28%. Any rate under 1% is probably acceptable. Having 4493 segments retransmitted results in a delay of about 45 and 90 seconds. This probably sounds like a lot of time but to achieve this level of retransmission activity I had to run 4 batch FTP jobs concurrently, each running 18 minutes. Eliminating the retransmission would reduce the run time of each batch FTP by only about 15 to 20 seconds.

Well, there you have it, an optimized Hipersockets network interface.

The Care and Feeding of Your z/VSE TCP/IP Stack

I recently suggested to a customer that they might get better throughput with their application by changing the priority so that the application was running higher priority than the TCP/IP stack partition. I received a questioning response. Really? Doesn't the stack always run higher priority than the applications it services? My answer was, nice rule but not always true.

By way of explanation, we can look at throughput as the care and feeding of your application or of the TCP/IP stack.

Feeding the TCP/IP stack is something you do when your application is sending data. Your application feeds the TCP/IP stack data. The data is queued into the stack's transmit buffer. Keeping the transmit buffer full makes sure the stack always has data to transmit. Once the stack has taken all the data sent and queued it into the transmit buffer, the application's send request is posted.

You can see that if the stack is busy and your application is running lower priority than the stack, your application may not get dispatched to send more data until a later time. It is possible that the stack will transmit all the data in the transmit buffer before your application can send more data. This leaves the the transmit buffer empty and can reduce throughput.

One way to ensure the stack has data available to queue into the transmit buffer is to use large send buffers. Large send buffers (for example, 1MB send buffers) can help keep the stack's transmit buffer full of data to send. Using large send buffers is most helpful when you are using network interfaces that have large MTU sizes, like a Hipersockets network interface.

Feeding your application is something the TCP/IP stack does. When data arrives from the network it is placed in the stack's receive buffer and the application is posted that data is available. The application must then issue a read for the data. If the application is running lower priority than the stack it may be some time before the application actually gets to run to read and process the data. In fact, in the worst case, the stack's receive buffer may actually become full forcing the stack to close the TCP window and stopping the data flow from the remote host.

Wow, it sounds like I should run all my applications higher priority than the TCP/IP stack. No, not at all. In practice, only bulk data transfer applications run into these types of problems.

The general rule of running the stack higher priority than the applications it services would apply to almost all applications. For example, interactive and multi-user applications like CICS TS, your TN3270E server and even DB2 (as examples) actually benefit from having the TCP/IP stack running at a higher priority.

In addition, applications that are primarily sending data out into the network generally show little throughput increase by running higher priority than the TCP/IP stack. Keeping the stack's transmit buffer full is generally pretty easy even if your application is running lower priority than the TCP/IP stack. However, this assumes that the application has access to the CPU when it is needed to allow it to keep the stack's transmit buffer full.

What does this leave? Perhaps a batch FTP retrieving data from the network, your FTP server partition or the IBM VTAPE (TAPESRVR) partition might benefit from running higher priority than the TCP/IP stack partition.

So there you have it. You can make a case for running some applications higher priority than the TCP/IP stack.

z/VSE TCP/IP Throughput Rates

I receive a number of queries about TCP/IP transfer (or throughput) rates every year. So, I thought I would make some comments about the issue.

At a high level, the concept is simple. The sending application sends data to a receiving application. Great! So, how much data can we send and how fast?

Basically, we would like to send as much data as possible in every request. Sounds simple, so lets start there. I allocate a large buffer, perhaps 1MB, in my application, fill it with data and send it all to the TCP/IP stack in a single request.

OK, the TCP/IP stack probably can not handle all that data at once, so lets look at how the request is broken down. First, the application's data is queued into a transmit buffer within the TCP/IP stack partition. The TCP/IP transmit buffer is probably limited to 32K to 64K in size. So, the size of the transmit buffer is the 1st limitation.

Once the data has been queued into the transmit buffer, the stack can begin the process of creating a packet to transmit the data. The 2nd limitation is the MTU (Maximum Transmission Unit) of the network interface being used. On a typical Ethernet network this is probably 1500 bytes. If you have z/VSE connected to a Gb (Gigabit) network you can take advantage of jumbo Ethernet frames which have an MTU size of 9000 bytes. If you are using a Hipersocket interface then the MTU size can move up to as much as 64K.

Well, assuming a 32K transmit buffer and an OSA Express QDIO Gb network interface, the stack will take about 9000 bytes (less headers, etc.) from the transmit buffer to create a packet. But, wait, can we really send 9000 bytes? Maybe, there are 2 more factors to consider.

The 3rd limitation is the MSS (Maximum Segment Size) negotiated by the local and remote host's TCP/IP stack when the socket was created. For example, if the sending TCP/IP stack supports an 8K MSS and the receiving TCP/IP stack supports only a 1500 byte MSS ... Guess what? The 1500 byte MSS wins.

The 4th limitation is the amount of space (number of bytes) available in the remote host's TCP Receive Window. TCP uses a 64K window to manage data transmission. Up to 64K of data can be transmitted to the remote host without waiting for an acknowledgement. Each byte of data sent must be acknowledged by the remote host. When an ACK packet is sent the sequence number of the last byte of data received and size of the current TCP Receive Window included. The sending TCP/IP stack can not send more data than will fit into the currently advertised TCP Receive Window.

Wow. OK, we started with 1MB of data being sent to the stack, 32K was queued into the transmit buffer. Now, of the 32K available in the transmit buffer the amount of data sent in a single packet is the smaller of the MTU size, Maximum Segment Size and the TCP Receive Window size.

As an example, if the MTU is 9000, TCP Receive Window 64K and MSS 1500, guess what? The amount of data sent in a single packet is 1500 bytes (less headers, etc.). Our 32K transmit buffer full of data will take around 22 packets to transmit.

As the packets are transmitted, the local TCP/IP stack is also watching for ACK packets from the remote host TCP/IP stack. The sooner the ACK packet arrives from the remote host, the sooner the local stack can begin or continue transferring data. The big factor here? Network speed and the number of hops needed to get the packet to its destination. In a word? Latency.

Having a high speed network is wonderful but latency can kill performance. You are always at the mercy of the slowest link.

I will discuss this more in another posting when I revisit using Hipersockets in z/VSE and consider some of the performance issues involved in optimizing throughput over a Hipersocket network interface.