danesabo/Obsidian
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

vault backup: 2025-04-30 11:07:29

Browse Source
This commit is contained in:
Dane Sabo 2025-04-30 11:07:30 -04:00
parent 6465afcbc1
commit c909e692e3
1 changed files with 42 additions and 27 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

69
300s School/ME 2150 - High Assurance Cyber-Physical Systems/ME 2150 - High-Assurance Cyber-Physical Systems Final Report.md
View File

@ -1,39 +1,54 @@
## Wheelchair Cushion Testing Rig Destroys Itself
### Introduction
Recently I have begun hosting a poker game. Poker is a beautiful game, as it connects people in conversation that is fulfilling and informative, as people discuss their life experiences. At my poker game, I learned about a somewhat comical cyber-physical system failure. A player at my poker game is a bioengineer who interned at [Pitt's Wheelchair and Cushion Standards Group](https://www.wheelchairstandards.pitt.edu/). She was responsible for configuring an immersion testing fixture to perform evaluations for a set of wheelchair cushions. She did not engineer the control program for the automated testing fixture, but soon found out one of her first days that the control program had a destructive mode of operation that was not intended.
## Recap of the Wheelchair Cushion Testing Rig Failure
Alex sought out the position at the Wheelchair and Cushion Standards Group as an summer internship that was conveniently located in the East Liberty neighborhood. Her job would be multifaceted: as part of a small laboratory group, he responsibilities would change upon the day and she would be expected to wear many hats. One of her first tasks was running experiments on sets of wheelchair cushions using an expensive test fixture.
### Wheelchair Cushion Standards
Wheelchair cushions are a surprisingly sophisticated device, and far from only a piece of foam. Wheelchair cushions are critical for users who spend a large amount of time in their wheelchair, and an effective cushion can play a significant role in the overall health of the user. Cushions redistribute the weight of a wheelchair user evenly across their buttocks, while an insufficient cushion can create problems for wheelchair-bound individuals including ulcers, posture issues, and blood flow restrictions. These problems are even further magnified for individuals who have loss of feeling in their legs, as they can not detect the development of injuries until they are visibly apparent or manifest in greater health issues. For these reasons, regulatory standards exist for wheelchair cushions. Standards such as [ISO 16840-2](https://www.iso.org/standard/84862.html) exist to protect wheelchair users by providing a standard to which cushions can be tested. By using wheelchair cushions that meet these standards, users can have a degree of assurance that the product they're using will minimize risks associated with prolonged wheelchair use.
For a wheelchair cushion to pass this standard, it must be experimentally tested. This testing is commonly done by companies that specialize in standards testing. The sensors and equipment required to do ISO testing can be very expensive, and prohibitive for individual manufacturers of items like wheelchair cushions to create test fixtures of their own. Pitt's Wheelchair and Cushion Standards group does such testing for this particular wheelchair cushion standard using a testing rig that is described in the following image.
### Testing Setup
**Context**
During a summer internship at Pitt’s Wheelchair and Cushion Standards Group, a friend of mine operated an ISO-16840-2 immersion test fixture designed to evaluate wheelchair cushions. The rig uses a hydraulic press fitted with a CNC-machined wooden “buttocks” model, instrumented with an array of pressure sensors, to measure both pressure distribution and press displacement as cushions are compressed.
![[press.png]]
This testing fixture consists of a hydraulic press with a specific CNC-machined wooden buttocks model attached. These buttocks integrate an array of pressure sensors at a speckling of locations in order to gather a holistic understanding of the distribution of pressure on the buttocks surface. A cushion is fixed to the lower part of the hydraulic press frame. The whole system measures two main quantities: the pressure distribution on the wooden buttocks, and the total displacement of the hydraulic press.
**Fatigue Test Incident**
In a cyclic fatigue test, the press descends until a target pressure is reached, holds, then retracts to zero, repeating many times. When testing a cheap, air-mattress-style cushion, the material never achieved the setpoint. The controller attempted to compensate by increasing displacement, triggering an over-pressure safety retraction. Because the control logic was not designed for cushions that cannot reach the target pressure, it then aggressively re-applied the previous displacement. This rapid cycle of over-loading and retraction violently rammed the wooden model into the steel frame, quickly breaking the lower sensors and, after only a few seconds, splitting the wooden buttocks apart under what was later estimated to be well over 1,000 lbf (far above the intended 400 lbf). The test only stopped when the emergency-stop button was finally engaged.
The positions of sensors on the wooden buttocks are of particular interest. These sensors are spread throughout the surface of the contoured shape. Ideally, these sensors should have similar values of pressure for a given load if the cushion is doing a good job supporting the load. These sensors have different heights relative to the displacement of the press. As a result, the sensors on the very bottom of the buttocks model first experience pressure, while the sensors on the sides of the buttocks eventually catch up in pressure measurement as displacement is increased.
**Key Cyber-Physical Failures**
### The Failure
1. **Unvalidated Operating Envelope:** The pressure-based controller was never formally verified for use with cushions that fail to meet the target pressure, allowing it to drive the system into an unsafe region.
2. **Faulty Assumptions About Sensor Integrity:** The over-pressure protection assumed sensors would continue to report accurately under extreme loading; as sensors degraded, the controller applied even more force.
The testing procedure Alex was specifically performing when the failure occurred was a fatigue failure test. In this test, the buttocks would descend upon the cushion until a proscribed pressure is achieved on all sensors, hold for a certain amount of time, and then ascend until zero pressure is achieved again. This cycle is then repeated for a very large number of cycles. Performance of the cushion is then determined as to how the pressure distribution changes over time--more performant cushions will retain their cushioning properties for longer than less performant cushions.
**Consequences**
No one was injured, but the incident incurred high costs: replacing the CNC-machined buttocks model, purchasing new pressure sensors, and dozens of hours spent recalibrating and re-testing. This failure highlights the importance of high-assurance methods—such as formal proof of safe operating bounds and rigorous digital engineering—to guarantee that a CPS cannot drive itself into a destructive mode, even under sensor faults or unexpected material behavior.
One of the first cushions Alex was tasked with testing was a cushion that was described as being similar construction of a cheap air-mattress. Alex was instructed on how to start the test, where the E-Stop button lived, and what should be expected. Once the person instructing her left, the test began.
## So, how could rigorous digital engineering solve this problem?
The press began depression the cushion with the wooden buttocks when the air-mattress-like construction gave only a whimper of support. The press continued to descend until the wooden buttocks were separated from the lower steel frame by only the thin rubber membrane of the cushion. At this point, the force on the lower sensors skyrockets while the circumferential sensors slowly accumulate load. The controller realizes that there is an extreme load on some of the sensors, however, and retracts the buttocks from the cushion. This safety feature was designed to protect the sensors in this exact event. Whoever designed this controller did not account for the fatigue testing mode, however, and once the buttocks were raised momentarily, the controller decided to return to the previous displacement. To Alex as an observer, the resulted in a violent ramming of the wooden buttocks into the steel frame, which repeated with cacophony as the air-mattress cushion sabotaged the testing fixture.
The fundamental problem with this testing set up was not a implementation
problem. The controller as it was programmed performed the fatigue test
exactly as it was asked. Instead, this cyber-physical system failure starts
before implementation and instead at the design stage. The designers of
this system never properly examined the possible states this system could
experience, or provided an opportunity in their design for their assumptions
to be violated.
After the first repetition, the lower sensors in the buttocks were broken. Instead of accurately reading the pressure on the wooden model, the pressure sensors instead read a much lower value. This results in the test fixture applying more displacement to continually try to reach the optimal test pressure. The sensors continue to read nonsensically low values as they are repeated slammed into the metal lower frame as they are slowly disintegrating. Each time they make contact with the frame they return with an even lower reported value, resulting in an even higher force applied to the buttocks model with each cycle.
Here's a simple example. One of the main causes of the failure was that
the control system could not reconcile that different pressure sensors may
have wildly different readings. This makes sense for cushions that are
operating properly where load would be evenly distributed, but cases
where this assumption is violated were never examined. In the case of this
failure, this assumption was violated by a subpar cushion, but what if a
sensor was never connected to the system? Presumably, the testing fixture
would behave this way regardless of the cushion being tested, and situations
where sensors may disconnected briefly for cleaning or moving the fixture
may be extremely common scenarios.
This whole process has happened in a time span of about 5 seconds. At this point, Alex realizes this is definitely not supposed to be happening and reaches for the E-Stop button. Before she could activate the E-Stop, the press descended the wooden buttocks with such ferocity that the wooden model splits in half as if the cushion itself were an axe. The ideal testing force is supposed to be around 400 lbf, but it was estimated that the fixture applied over a thousand pounds of force when the buttocks were destroyed. Finally, the E-Stop was activated before further damage was created.
After taking the HACPS class, I think the designers of this testing fixutre
could have made good use of a model checker like TLA+. An analysis of
testing system through a series of TLA+ modules could avoid these
disaster scenarios where sensors do not prompt correct control. One could
do this analysis by examining what 'correct' behavior is:
### Reflection
The part of this failure that makes it a cyber-physical failure is the fact that while a pressure based controller was implemented, it was never validated for a use-case where the cushion does not achieve a sufficient pressure on all of the buttock sensors. This case was realized when the buttocks made contact with the steel frame with this particular low quality cushion.
Another control was implemented in case of extreme pressure readings on the buttocks sensors, but that control also experienced a cyber-physical failure. The controller assumed a model of the sensors that would continuously report correct values even in the case of extreme loading. This assumption proved to be incorrect, which essentially nullified the over-pressure control once the model of the sensors was invalidated.
Fortunately, this comedy of errors did not injure anyone, but this failure incurred significant business costs. The wooden buttocks had to be replaced with a new model, which required expensive CNC machining. The damaged sensors had to also be replaced which were never cheap to begin with. And finally, the group paid Alex dozens of hours to recalibrate the system when the new sensors and buttocks model were installed, and to reperform all previous testing that was now invalidated by the broken sensors.
Cyber-physical system failures can be sneaky. It is not always clear while coding if a control will actually prevent a mitigated failure from actually occurring. Because of this opacity, other tools such as proof-based methods have to be used to ensure high-assurance. In this case, a proof could have been developed to determine that an unsafe force could not be applied to the lower sensors regardless of their reported value. By perhaps investigating this possible mode, the controller designer could be informed that only relying on the lower sensors as protection would have been insufficient to protect them. These proofs require extra effort to create, but if they prevent such expensive failures, it is worth it to create them.
For the testing fixture as described, a couple things should ALWAYS
happen:
1. The sensors on the bottom of the buttocks should ALWAYS experience
more load than the sensors on the side of the buttocks
2. The sensors on the bottom of the buttocks should never have a difference
in pressure from the side buttocks sensors greater than some value $\Delta P$.
3. Sensors across symmetries (left vs. right buttock) should