Expanded Storage Concepts
This section introduces concepts common to expanded storage.
micsrm140
Originally, this was a high-speed semiconductor memory, initially made available for the IBM 3090 processor series. In later systems, this became a redefinition of central storage performed during IPL. It was provided to maintain compatability with existing applications. Additionally, with WLM Goal mode, many of the metrics discussed in the following sections and the control mechanisms used were eliminated. The information is presented here in a historical context only, and is no longer used for managing system performance.
The following sections discuss expanded storage and its impact on the job of Real Storage Management.
Overview of Expanded Storage
Expanded storage is optional, high-speed semiconductor storage initially made available for the IBM 3090 processor series. The storage is installed in the central processor complex; that is, it is an integrated part of the complex and not a stand-alone piece of equipment.
In some respects, expanded storage can be thought of as a slower speed extension of central storage. The two differ in that expanded storage is not directly addressable and can be accessed only by moving 4K blocks of data from expanded storage into central storage. All data movement to and from expanded storage must be through central storage. There is no way to directly transfer data between expanded storage and auxiliary storage.
In other respects, expanded storage is similar to high performance paging devices such as the 2305-2 or 4305. However, the system control program knows more about expanded storage than it does about solid state paging devices. It uses this intelligence to optimize storage access, including paging and swapping operations and special kinds of application and system control data in ESA systems.
A primary use of expanded storage is as a substitute for auxiliary storage (page/swap data sets on DASD) in situations demanding better performance. The time it takes to read a page from auxiliary to central storage in a well-tuned system may range from a few milliseconds (with cached DASD controllers or solid-state channel-attached devices on dedicated paging channels) to 20 to 30 milliseconds for non-cached 3380 or 3390 DASD on non-dedicated channels. Indirect measurements suggest that typical times for page transfers from expanded to central storage take something less than 100 microseconds. This represents a reduction in page resolution time of approximately two orders of magnitude.
When expanded storage was first introduced, the system control program used it exclusively for paging and swapping. New MVS versions have introduced additional uses of expanded storage, such as virtual I/O (VIO) and hiperspaces.
Expanded storage, therefore, added a new level to the storage hierarchy between central and auxiliary storage. Now, page movement within the storage hierarchy can be divided into the following categories:
From | To | Terminology |
Central | Auxiliary | Page-out or swap-out |
Auxiliary | Central | Page-in or swap-in |
Central | Expanded | Expanded storage write |
Expanded | Central | Expanded storage read |
Expanded | Auxiliary | Page migration |
System Control of Expanded Storage
The Real Storage Manager (RSM) handles the transfer of pages between central and expanded storage. RSM determines both whether a page will be written to expanded storage, and when a page in expanded storage should be migrated to auxiliary storage. It bases these decisions on four values:
- Criteria age
- The installation-specified time value that is used by RSM to determine whether a page should be transferred from central to expanded storage. Criteria ages are specified for various categories of pages and workloads in the SYS1.PARMLIB member IEAOPTxx. IBM supplies default values that are reasonable for most normal processing environments.
- System-high unreferenced interval count (UIC)
- The time since the least recently referenced page in central storage was referenced. This value is therefore an inverse measure of central storage contention (a low UIC indicates a high demand for central storage). It is not possible to explicitly define a minimum value for the UIC, but it is influenced by several parameters in the IEAOPTxx member of SYS1.PARMLIB.
- Migration age
- The time that a page has remained unreferenced in expanded storage. It is an inverse measure of expanded storage contention (a low migration age indicates a high demand for expanded storage). In periods of contention, RSM moves pages in expanded storage that have remained unreferenced for this period of time into central storage, and the auxiliary storage manager (ASM) then pages them out to auxiliary storage. This process is referred to as page migration.
- Think time
- The average delay between the time when the system is ready for new input from a terminal user and the time when the terminal user actually provides such input by pressing the "enter" key or equivalent. This value is used primarily to control logical swapping, but is also used in terminal input and output wait swapping to expanded storage.
Criteria age, is specified for individual types of pages and address space by the installation (or allowed to default to IBM-specified values). The others are measured by the system control program and will vary with the nature and volume of work being performed and with system configuration. All are specified or measured in seconds.
In paging situations, RSM sends changed and unchanged stolen pages, virtual fetch pages, and page-out requested pages to expanded storage when the migration age is greater than the criteria age.
In swapping situations, RSM normally sends both swap-out trim and working set pages to expanded storage when the migration age is greater than the criteria age for the type of address space in question. However, TSO terminal wait, steal, and pageout users' working sets are sent to expanded storage when the sum of migration age and UIC is greater than the sum of criteria age plus think time.
When a page is required in central storage, RSM first checks to see if the page is in expanded storage. If not, a page fault is generated and the page is retrieved asynchronously from auxiliary storage by the ASM with a normal page-in operation. If the page is in expanded storage, however, no page fault is generated and RSM immediately transfers the page into central storage. This transfer happens synchronously with program execution. It is important to emphasize that a page fault is not generated in this case, and the expanded-to-central transfer is not considered a page-in. Indeed, RMF Version 3 does not even report page movement from expanded to central storage.
Swapping to expanded storage differs from swapping to auxiliary storage. When an address space is swapped out to expanded storage, RSM and the System Resources Manager (SRM) divide the working set pages into primary and secondary working set groups. The primary group, which is managed by RSM as a single entity, contains all LSQA and fixed pages, plus one page from each virtual storage segment included in the whole working set. The secondary group comprises all working set pages not included in the primary group.
Once the address space has been swapped out to expanded storage, RSM recognizes the difference between the primary and secondary working sets when it makes its decisions about migration. Secondary pages are migrated first, and only after all secondary pages have been migrated will the primary working set be migrated.
If the address space is swapped back in before it has been migrated, the primary working set is swapped in first, at which time the address space becomes dispatchable. Secondary working set pages are then treated the same as all other pages in expanded storage. This means that if they are migrated, they will be sent to local page data sets rather than swap data sets. If you do a substantial amount of swapping to expanded storage, this could be an argument in favor of an auxiliary storage subsystem configuration consisting only of local page data sets with no swap data sets.
Users of Expanded Storage
The RSM categorizes expanded storage use by the type of paging group and by the status of the page being considered for placement in expanded storage. The paging group can be a:
- Type 0
- Privileged or non-swappable (including all common areas) address space,
- Type 1
- Any other type of page not included in types 0 or 2, or
- Type 2
- A page belonging to a TSO user waiting for terminal I/O to complete, or pages paged-out or stolen from a TSO user.
The page's status is categorized as a:
Changed paged-out page Changed swapped-out page Changed stolen page Unchanged paged-out page Unchanged swapped-out page Unchanged stolen page Working set page ready for swap-out Virtual fetch page VIO swapped-out page Hiperspace page
Note:
Hiperspaces were introduced with MVS/ESA, and are not available in MVS/XA. The use of expanded storage for VIO was standardized in MVS/ESA, but is available in MVS/XA systems by means of applying IBM APAR YA09186.Different criteria ages can be specified for each combination of paging type and page status, with the exceptions of virtual fetch, VIO, and hiperspace pages. The exceptions have a single criteria age specification without paging type classification differences. This allows the installation to tailor the configuration so that expanded storage usage is prioritized by the type of address spaces that are most important in the specific processor complex. It is possible to set the parameters so that some of the combinations are eligible and others ineligible to use expanded storage.
Measurement, Analysis and Management
RMF measures and reports on many data elements concerning paging, swapping, and memory use. Beginning with RMF 3.3.0 and continuing through RMF Version 4 (introduced with ESA), much information has been added that differentiates between paging or swapping to expanded versus auxiliary storage, and between the use of central versus expanded storage for CSA, LPA, SQA, etc., pages. Expanded storage has added to the complexity of understanding and managing storage hierarchy.
Even though well-tuned systems or systems with low storage contention are likely to experience expanded storage-related delays so small as to be relatively unimportant when compared to other sources of delay, installations with expanded storage should now monitor its utilization and activity as well as auxiliary storage activity. Expanded storage delay can become substantial to the point of significantly reducing the benefit from your investment in it. The System Storage Usage Report described in section 2.2.3.2 can aid in tracking the utilization of expanded storage.
In a system with expanded storage, the degradation due to page movement delay from expanded storage to central storage is not directly measurable. Indeed, the rate of movement from expanded to central is not measured by RMF in MVS/XA, it is only measured in MVS/ESA systems. While the system control program maintains (and RMF records) the average time that it takes to read a page from auxiliary to central storage, no such estimate is made of the time it takes to transfer a page from expanded to central storage.
Page migration should be a primary concern because, if excessive, it represents ineffective page placement and ineffective use of expanded storage. If there is contention for expanded storage, it makes no sense to move a page from central to expanded storage if that page will eventually be moved back into central and then paged out to auxiliary. It would be much better to page it out directly to auxiliary storage.
Minimizing such ineffective page placement depends on how accurately an address space's future activity and storage references can be predicted. RSM makes these predictions based on the criteria age values specified in the IEAOPTxx parameters for the various categories of expanded storage use.
Expanded storage can be used to improve the response time of online systems, to increase the capacity of the processor complex, or a combination of the two. The best way to use expanded storage depends on the business and technical environment in which it is used.
Unfortunately, the mechanisms used to control expanded storage allocation do not provide for workload assignments, such as performance group or domain. It is therefore necessary to understand the storage utilization characteristics of the individual workloads and the system as a whole before attempting to modify the criteria ages that affect expanded storage control.
There are some general guidelines that apply to all systems, however. To improve the response times of online systems running as non-swappable address spaces, set low criteria ages for the type 0 paging group and high ages for types 1 and 2 paging groups. This favors the onlines over other work, but only if there is other work in the same system. In a system used for application development and testing with ISPF/PDF users submitting batch compiles and executions, you might want to favor type 2 over types 0 and 1. If you have a processor complex that is performing a wide variety of tasks and with changing workload characteristics over time, then to achieve both somewhat better response time and more throughput use the default values for all criteria ages.
When making criteria age changes, keep in mind the following:
- Assigning a low criteria age to a particular page type and status is saying, in effect, that you want RSM to move such pages to expanded storage before other page type/status classifications even if the contention for expanded storage is high. You would do this if- you regard the workload owning the page as important and deserving of a performance improvement,or- you expect the page to be referenced quickly and want to lower the load on ASM by making it unnecessary to page-out and then page-in the same page in a brief time period.
- Assigning a high criteria age to pages has the opposite effect: such pages are sent to expanded storage by RSM after page types with lower criteria ages. In periods of contention, such pages tend to go to auxiliary storage instead. You would do this if- you want to use your expanded storage resources for more important work,or- expect that such a page will not be referenced for a relatively long time and would be migrated to auxiliary storage anyway.
Although IBM has established reasonable default values, it is ultimately the responsibility of individual installations to set these values for best use of expanded storage. The performance analyst must monitor page movement to ensure that the specified values are optimizing the use of expanded storage for the installation's requirements. If criteria ages are set too low, expanded storage will tend to be over- utilized. This will result in a low migration age, large values for both central-to-expanded page movement, and page migration. If the difference between these two measurements is small, it could indicate that too many pages are being migrated.