A/B Testing - A372

 

A/B Testing
Foundational knowledge article
Document Type	Article
Document Identifier	372
Themes	Residual stress and dimensional control management Quality/inspection management
Tags	Mechanical properties Outcome parameters Part parameters Structural analysis Functional Requirements
Prerequisites	Composite properties

Introduction

A and B-Basis testing is a statistical approach used to characterize material properties across various industries. This method involves testing a substantial number of specimens and applying statistical analysis to derive A or B-Basis values (also referred to as allowables). These allowables account for batch-to-batch and part-level variability, which is often inherent in material and component manufacturing. Incorporating these statistical limits supports quality assurance, process control, and design as it allows users to set more reasonable tolerance limits. This could mean reduced part scraping, design optimization for reduced weight, and processing optimization for reduced time without compromising product performance.

Definition

A-Basis and B-Basis values (often referred to as allowables) are statistically derived property limits used to ensure the reliability and safety of structural components. The A-Basis value is defined as the material property below which no more than 1% of the population is expected to fall, with a 95% confidence level. Similarly, a B-Basis is a property value below which no more than 10% of the population is expected to fall, also with 95% confidence . The 95% confidence level means there is a 5% chance that the actual value could be lower than the reported allowable. Basis value determination occurs at level 1 of the building block approach, where mechanical testing is performed on coupons - small, standardized specimens designed to represent the material used in the final part. Once a sufficient number of test results is collected, the data for each group can be analyzed. This is done through first evaluating which probability distribution best matches the result, and then following the analysis recommended by CMH-17 to calculate the desired basis values. Although this method is most commonly used to determine the material strength properties, the process can be applied to any measurable property at the coupon or component level.

More on the building block approach can be found here: Composites design

Analysis

The procedures for identifying the appropriate statistical distribution and computing A and B-Basis values are outlined in CMH-17 . It is recommended that the user create a histogram of their data to visually assess the underlying distribution.

Figure 1 Example histogram of the number of samples failed at a given strength value

As seen in Figure 1, the example histogram is skewed to the left, meaning that the Weibull Distribution method may be suitable for this data set. With a limited number of specimens, it can be challenging to confidently determine the most appropriate distribution model. Below are some examples of commonly used distributions for strength-based allowables and the equations used to calculate the B-Basis value from the corresponding distribution .

Weibull Distribution

The Weibull distribution is a highly versatile, unimodal, continuous probability function that is marked by its unique ability to model heavily skewed data. The ability to represent skewed data is especially an asset when analyzing strength-based allowables, where the data is frequently skewed due to multiple failure modes . It can also model both normal and exponential distributions. The equation for the B-Basis value using a two-parameter Weibull distribution is as follows :

\( B = \alpha \left(0.10536\right)^{\frac{1}{\beta}} \exp\!\left(\frac{-V}{\beta \sqrt{n}}\right) \)

Where \(\alpha\) and \(\beta\) are maximum-likelihood estimates (see respective calculations in MIL-HDBK-17 [2]), and \(V\) is a tolerance factor which is dependent on the sample size \(n\) .

Figure 2 Example of a possible Weibull Distribution

Normal Distribution

The normal distribution is a unimodal, continuous probability function. This distribution follows the shape of the common bell curve. The equation for the B-Basis value under a normal distribution is as follows [2]:

\( B = \bar{x}-k_bs \)

Where \(\bar{x}\) is the arithmetic mean, \(k_b\) , is the tolerance factor (a constnat which can be found in the table of) MIL-HDBK-17 [2]), and s is the standard deviation.

Figure 3 Example of a possible Normal Distribution

Lognormal Distribution

The lognormal distribution is a continuous probability function capable of handling skewed data. Though less flexible than the Weibull distribution, the lognormal distribution is commonly used for analysis of strength, especially when the data spans over several orders of magnitude. To calculate the B-Basis value using a lognormal distribution, one simply takes the log of the dataset and calculates the B-Basis value using the same formula as the Normal Distribution [2].

Figure 4 Example of a possible lognormal distribution

Advantages

A and B-Basis testing offers several advantages over traditional material characterization methods, such as reduced cost, improved reliability of data, and improved opportunities for variable isolation. As testing is performed at a lower level (coupon and sometimes component level), samples are cheaper and easier to make. This allows the user to test larger sample sizes more efficiently. Data collected at a lower level can be analyzed and used to predict performance at the subcomponent or full-structure level, supporting a multiscale modelling and certification strategy.

Variable Isolation

An important advantage of basis testing is the ability to isolate the effects of individual variables on material behaviour. Following the MSTEP approach, there is an iterative process of selecting the ideal material, shape, tooling, and equipment parameters. These parameters influence the manufacturing process of the part, which involves more parameter selection, especially surrounding the cure cycle. Decisions made using MSTEP will have large effects on the final component’s properties. For example, decisions made on part material will have a great effect on the component's strength, as well as how it performs under different environmental conditions, such as high temperature and high humidity. To best predict behaviour at a higher structural level, it is recommended to gather data on any parameters influencing the response of the final part. This includes but is not limited to processing parameters, tooling geometry, and temperature and humidity of the final part. To analyze the effect of each variable, a control group, with fixed processing and test conditions, must first be created to establish a baseline. Additional groups can then be fabricated with only one variable modified at a time. By comparing the calculated basis values of each group, the impact of individual parameters can be assessed. This approach allows for faster iteration, improved root-cause identification, and increased confidence in the structural response of the final part when supplemented with additional non-destructive testing and monitoring.

Statistical Significance and Reproducibility

Many composite materials exhibit inherent variability due to their manufacturing processes, material anisotropy, and sensitivity to environmental and process-induced effects. Basis testing helps distinguish statistically significant variations from expected scatter. Variability between specimens may arise from processing defects in manufacturing or variations in the processing parameters or properties of the constituent materials . Whether the source of variations is known or unknown, A and B-Basis testing accounts for this variability and helps identify the outliers. With a sufficient sample size, distributions can reveal meaningful trends, even in the presence of high variance. This leads to improved reproducibility. In small-batch testing, there are frequently too few data points to determine whether observed variance is due to process noise or defects. With a large number of data points, the expected range becomes clearer, and the performance metrics will be more consistent.

Decision Making

A/B-Basis testing provides valuable data for informed decision-making in design, process optimization, and part validation. When combined with other tools such as risk assessments and FEA, it allows engineers to define the manufacturing process and part designs to ensure robust performance under expected service conditions.

Limitations

While A and B-Basis testing reduces the need for full-scale structural testing, it remains a resource-intensive process. The statistical rigour required by standards such as CMH-17 demands a large number of specimens, even at the coupon level, which can be costly and time-consuming. With the rapid development of new composite materials, the cost and time required to generate sufficient data to meet the robust testing requirements specified by CMH-17 can be prohibitive. In such cases, manufacturers may opt for reduced testing protocols, which typically yield more conservative basis values, leading to less efficient material use. To address these challenges, various research groups have begun exploring virtual testing and machine learning techniques to process this data . Notably, the Society for Advancements in Materials Processing Engineering (SAMPE) has developed a framework combining reduced physical testing with simulation-based validation, adhering to the CMH-17 guidelines for reduced testing . This data is used to verify simulations of the behaviour at a higher level. This is in combination with Finite Element Analysis (FEA) to model behaviour from lamina to laminate level.

Best Practices

As described in , A and B-Basis testing should begin at the coupon level. Testing should be conducted using controlled groups following the robust testing guidelines set by CMH-17 for each desired variable. Following the building block approach, specimens should incrementally be made more complex until they have reached the level of complexity required to accurately represent the final component. For example, specimens would begin as flat test coupons, then would have added geometry changes and gradually become more similar to their final assembly.

Test Design

Specimens should closely match the manufacturing method, thermal history, and post-processing conditions of the final part. A test matrix to cover all relevant environmental conditions, load cases, and manufacturing process parameters must be considered. For each variable under investigation, a dedicated sample group should be analyzed with only the condition under test being varied relative to an established baseline. To create the baseline, a batch of samples should be created using the nominal (recommended) processing parameters and testing conditions.

Selection of Basis Values

The selection between A and B-Basis values is typically based on the load path after part failure :

• The A-Basis value is used as an allowable when part failure would result in the critical failure of a larger part or assembly . Since the A-Basis value is set at the 1st percentile with 95% confidence, it is also appropriate for applications where tight tolerances are required.
• The B-Basis value is more commonly used due to its higher acceptance rate, representing the 10th percentile with 95% confidence. This allowable is typically applied to parts where failure will not result in the critical failure of a larger component or assembly and still has a safe load distribution after failure . The selection of which Basis value best suits an application is up to the user and the risk assessments they have performed for the parts under test.

Applications In Composites

Composites are particularly well-suited for basis testing due to their orthotropic nature, process sensitivity, and batch-to-batch variability. The properties of composite materials are highly sensitive to variability due to the orthotropic nature of the material, effects of the manufacturing process (tooling, layup method, curing), temperature, humidity and pressure effects (both during manufacturing and during the operation of the final part), as well as composition and geometry. Each of these parameters, as well as the constituent materials themselves, causes the final parts to have a high degree of batch-to-batch variability. A and B-Basis testing is a useful tool for analyzing this variability. In the MSTEP approach, where the goal is the select the best parameters while analyzing manufacturing feasibility, data on the causes of batch-to-batch variability is crucial. Through performing robust testing over many batches, trends can be identified, and the expected range of variability becomes much clearer. This then influences parameter selection, which in turn influences the process, leading to improved part quality and more optimal manufacturing.

Case Study

An application of this testing is to study the effects of processing conditions. A common study in composites is the effect of pressure on the porosity of a laminate. With all other conditions maintained constant (test environment, processing temperature, tooling, etc), only the pressure is varied. A test matrix is then created with a range of pressures to be tested, along with the number of repeat tests to be conducted for each pressure. Porosity is then measured for each specimen and plotted on a histogram. The most appropriate distribution (see analysis section) is found, and the corresponding equation is used to find the A or B-Basis value depending on the part requirements.

References

About	Help

An approach where risk is incrementally assessed in the scale-up of part size and complexity. Material and process (M&P) variabilities are evaluated at lower scales; While at higher scales, load paths and structural designs are verified. This approach involves three complementary engineering design activities:

• Material qualification
• Structural certification, and
• Production approval.

The use of multiphysics models to predict the outcome of real-world scenarios. May be analytical (closed form, "hand calculations") or computational (implementation on computers is required due to the large number of calculations involved. e.g. finite element method, finite difference method)

Most often in composite materials engineering, modelling refers to either:

• Process modelling (predicting manufacturing outcomes, or the inverse problem of predicting manufacturing parameters to achieve a desired outcome), or
• Performance modelling (predicting the stiffness/strength/suitability of a structure, or the inverse problem of the material properties and dimensional requirements to achieve a desired stiffness/strength/suitability of a structure)

(same as "Simulation")

Material and process (M), Shape (S), Tooling and consumables (T) & Equipment (E) - all have an interlinked effect on the Process step (P). (See MSTE factory ontology)

Engineered materials (designed to have specific properties) made from two or more constituent materials with different physical or chemical properties. The constituents remain separate and distinct on a macroscopic level within the finished structure.

The individual materials that combine to form the composite material. The constituent materials are separate and distinct on a macroscopic level.

With regards to modelling & simulation, validation demonstrates the accuracy of a computational model — in the context of its intended use — to check that it represents the physics of the problem and the ‘reality of interest’ (needs are satisfied).

With regards to manufacturing, risk is the combination of the probability and consequences of undesirable manufacturing outcomes. Manufacturing risk can lead to technical issues, program/schedule delays, and cost overruns.

The Society for the Advancement of Material and Process Engineering (SAMPE).

The use of multiphysics models to predict the outcome of real-world scenarios. May be analytical (closed form, "hand calculations") or computational (implementation on computers is required due to the large number of calculations involved. e.g. finite element method, finite difference method)

Most often in composite materials engineering, simulation refers to either:

• Process simulation (predicting manufacturing outcomes, or the inverse problem of predicting manufacturing parameters to achieve a desired outcome), or
• Performance simulation (predicting the stiffness/strength/suitability of a structure, or the inverse problem of the material properties and dimensional requirements to achieve a desired stiffness/strength/suitability of a structure)

(same as "Modelling")

The continuous material phase that binds the reinforcement together, maintains shape, transfers load, protects the reinforcement from environment and damage, and provides the composite support in compression.

Desirable characteristics:

• Moisture/chemical resistance
• Low density
• Processability