Further Clarification of Year 2000 Testing Issues
December 28, 1998
Circular No. 11123

To the Chief Executive Officers of All State Member Banks, Bank Holding Companies, Edge and Agreement Corporations, and State-Chartered Branches and Agencies of Foreign Banks in the Second Federal Reserve District, and to Related Vendors Serving These Organizations:

Many banking organizations have raised questions regarding supervisory guidance on Year 2000 testing because guidance provided by different regulators does not always appear entirely consistent. This concern is particularly acute for internationally active institutions whose Year 2000 programs are supervised at an enterprise level by the home country supervisor and by host country supervisors for local activities within each jurisdiction.

This circular provides clarification on how the various guidance documents should be viewed in the context of devising an effective Year 2000 testing program and carrying out effective testing, particularly as the time and resources available for testing grow short. In particular, it focuses on lessons learned from the Year 2000 testing experience to date and how these lessons affect the design of effective test strategies. While it is primarily directed at internationally active institutions, the basic principle of focusing scarce resources on areas where they can be used most effectively to reduce Year 2000 risks applies to all organizations. When examiners review Year 2000 programs, this basic principle will be paramount.

Year 2000 Testing -- Experience to Date and Evolving Thinking

Testing or the validation phase of Year 2000 projects has been widely recognized as the most resource-intensive phase of overall Year 2000 programs. As organizations have advanced into or moved out of this phase, two observations have repeatedly been made:

Designing effective tests for Year 2000 business risks is more complicated than originally thought; and

Conducting external tests requires a level of coordination and cooperation that is highly resource-intensive and difficult to manage.

The result has been that many organizations have had to rethink their approach to testing in order to minimize Year 2000 business risks.

Guidance on testing, provided while most organizations were still in the assessment or remediation phase, tended to catalog various types of tests that might be considered in developing an effective testing program. While language was usually included that suggested some flexibility in designing the testing strategy, there often appeared to be an implicit assumption that all mission-critical applications should be tested with all business partners using the full range of tests described. In particular, there was special significance placed on the more complex end-to-end and, in some cases, industry-wide tests.

More recent guidance has focused more extensively on the appropriate use of proxy testing and developing testing strategies with a clear vision of minimizing business risks. This evolution is directly the result of lessons learned on testing and new cost-benefit analyses for mitigating Year 2000 business risks through tests of different types. In particular, internal testing is increasingly seen as providing more effective risk mitigation than external, end-to-end testing.

Internal testing is controlled by the organization itself with little or no need to coordinate with others outside the organization. Business managers are able to develop their own approaches to testing that align testing resources to the perceived business risks. Internal tests -- particularly internal integrated tests where applications are tested in conjunction with other applications with which they interact -- provide a very high degree of confidence that any Year 2000 problems remaining after remediation will be identified. To date, external testing has identified few problems for applications already thoroughly tested internally on an integrated basis.

As time grows ever shorter and resources become more scarce, business managers increasingly realize that they need to make tough choices between more rigorous internal testing regimes or more external tests that may be more elegant but also resource-intensive. While most business managers still see the need for some type of external end-to-end testing process for all mission-critical applications, they are looking for efficient ways to achieve this result. Typically, this means:

Decomposing complex end-to-end tests into elemental units that can be tested separately;

Using tests conducted by another party as a proxy for conducting the test themselves; or

Combining the two (i.e. testing only certain elements of the end-to-end test and relying on proxy tests for the other elements).

Supervisory Guidance

In the United States, guidance for Year 2000 testing for banking organizations has been provided by the FFIEC in its April 10, 1998 paper entitled " Guidance Concerning Testing for Year 2000 Readiness"3 ("Guidance") and its August 31, 1998, "Questions and Answers Concerning FFIEC Year 2000 Policy."4 ("Q&As") At the international level, the Joint Year 2000 Council has provided guidance on testing in its September 1998 paper, "Testing for Year 2000 Readiness" ("Joint Council").

In all three instances, the guidance was not intended as binding regulation. The documents emphasize flexibility in applying any specific guidance. In particular, in the Q&As, the FFIEC says:

The FFIEC recognizes that each financial institution is unique. Management should determine the best testing strategies and plans for its organization taking into account the size of the institution, the complexity of its operation, and the level of business risk exposure to the Year 2000. The FFIEC also recognizes that there is no single approach to testing for the Year 2000. Options range from testing within a financial institution's own environment to proxy testing."

The Joint Council makes the same point even more strongly:

Testing strategies for an individual institution or market is ultimately a business decision -- what are the business risks faced if a [business] application does not work, and what resources are appropriate to apply to reduce these risks to an acceptable level. For organisations starting late in their Year 2000 preparations, sufficient time and resources necessary to complete optimal testing may not be available. Business decisions and hard choices will have to be made on what is tested and how it is tested. Testing strategies need to be realistic and feasible and to make business sense for the organisation and for the market. Regulators and participants need to adopt strategies that are pragmatic and achievable lest resources be used in suboptimal ways.

Both the FFIEC and the Joint Council then go on to discuss proxy testing and point out that there are trade-offs between an institution executing a well designed test and relying on a proxy test conducted by others. Both strongly encourage institutions to conduct their own tests where business risks warrant and available resources permit because reliance on proxy test results will pose some modest level of added risk. Ultimately, however, it becomes a business judgment by the business manager and the organization as to whether the added risk is acceptable.

As organizations review their own testing strategies and those of key counterparties, customers and service providers -- and as regulators and external auditors and consultants review test plans of individual organizations -- several principles should be borne in mind:

Business managers are responsible for identifying Year 2000 business risks and establishing the priorities for addressing and mitigating the risks within each business line.

Resources should be allocated across the entire organization in a way that aligns resources to the risks faced.

Testing strategies should balance the needs of all parties to a transaction. That is, those providing services to others need to devise testing strategies that provide a reasonable degree of assurance that results of proxy tests are transferable to institutions not actually testing with them. Additionally, users of services need to recognize that testing with every user may not be an effective use of the service provider's resources and that proxy tests are a reasonable substitute.

Because time and resources are scarce and both will continuously diminish as we approach the end of 1999, trade-offs and tough business decisions are inevitable. Institutions that are lagging should leverage off the good work of other institutions that are well along if this makes good business sense.

Requiring universal or specific tests in a particular market or product may misallocate resources, resulting in unintended consequences. While such an approach may be appropriate for a very few key products or markets whose failure could have systemic implications, every effort should be made to devise testing strategies that allow businesses to make their own strategic decisions.

As organizations move further into the testing or validation phase, everyone will be facing new Year 2000 challenges and need to develop contingency plans to mitigate perceived business risks. Cooperative efforts, including the development of sound testing strategies and the sharing of test plans and test results, should help everyone in minimizing any possible disruptions arising from the Year 2000 issue.

Questions pertaining to this circular and other supervisory issues on the Year 2000 can be directed to Sarah Dahlgren or George R. Juncker, Vice Presidents, Bank Supervision Group, or to Ira Adler or Joseph Galati, Examining Officers, Advisory and Technical Services.

Of course, organizations dependent upon vendors or service providers may need to adjust test plans around product availability, but once the product is available, considerable flexibility exists on the conduct of the test.
This is especially true in cases where the format of the message remains unchanged. Even where the file format is changed, simple point-to-point tests are likely to achieve the same results as the more complicated end-to-end tests. Also, where external tests have revealed problems, there is usually a significant embarrassment factor for the organization identified as experiencing the problem, an embarrassment that can be avoided by more rigorous internal testing.