Leveraging Envelope Control to Unlock Capabilities for Future Vehicle Safety Systems
TRI’s Approach to Shared Control and Autonomy
By Dr. Carrie Bobier-Tiu, Dr. Sarah Koehler
In comparison to peers, Toyota Research Institute (TRI) has a unique perspective on autonomous vehicles. One of the many reasons we were both drawn to working at TRI was the focus on applications of new technologies, particularly those being developed for autonomy, to continually improve driver safety. This focus on driver safety and autonomy has led to Toyota’s Guardian and Chauffeur concepts.
Toyota Guardian is a vehicle concept that works alongside a driver in parallel autonomy, amplifying the human capabilities for enhanced accident prevention by communicating with the driver, and augmenting their commands under emergent circumstances. Toyota Chauffeur, on the other hand, is a vehicle concept for full autonomy where the human is removed from control, offering advances in mobility to those who need it. Full autonomy is a complex endeavor, requiring system level solutions that may not be available to production vehicles in the short term. The goal of Guardian is to leverage the technology and hardware being developed for Chauffeur, building up the infrastructure for a holistic approach that begins with providing new safety technologies to the driver as they are available and ends with a two-mode system that can provide the highest levels of safety to either a driver or an autonomous system.
Having a background in control and dynamics, our team has focused on developing control solutions for Guardian especially, although we have applied the same principles to our autonomous applications while striving to build one system without a duplication of efforts. In simple terms, the control system is the “hands and feet” of the vehicle, the last line of defense to maintain the vehicle within the limits of safety. In the autonomous state, the controller provides full steering and acceleration commands to the chassis. In parallel autonomy, or shared control, the controller acts as an extension of the driver’s intent, augmenting their commands only in emergent and critical scenarios. At a system level, the goal for Guardian is to improve the safety of the system by working with the driver to proactively avoid accident situations and improve safety. At the control level, we have investigated technologies that allow for shared control, a holistic approach that promotes coordinated, proactive actuation, and the ability to incorporate higher fidelity context about the environment as technology geared for autonomous driving improves.
In choosing a control algorithm to support Guardian research, we focused on key capabilities that the controller should implement:
- understand and use the full dynamic capabilities of the vehicle to maintain safety;
- enforce limits of the vehicle platform and coordinate actuation; and
- allow the driver to maintain agency as much as possible.
The approach we are taking to incorporate all of these capabilities is what we call envelope control.
What is Envelope Control?
Envelope control was historically applied to aircraft to help the pilot keep the aircraft body within a safe region of the flight state space, allowing the pilot to maneuver up to the limits of handling without risking instability. In simple terms, the controller keeps the plane from stalling or otherwise falling out of the sky due to a command requested by the pilot. A wide variety and number of limitations exist for different aircraft, including restrictions on angle of attack, pitch, bank angle, and speed. Generally, the pilot is afforded uninhibited operation within the safe regime, but is limited upon approaching or passing the edges of the safe envelope.
In the case of a car, the control layer should, like in the aircraft, maintain stability and enforce actuator limitations, but there is a much richer and more complex interaction between the vehicle and the environment than is seen in aircraft. Using the algorithms, sensors, and computing structures that are being developed for autonomy provides context about the environment and the ability to build up envelopes, or motion constraints, based on the physical surroundings of the vehicle. We are envisioning a future ADAS system that can process all of the constraint information for actuation, dynamics, and physical environment into one holistic approach with envelope control.
In our research, we chose Model Predictive Control (MPC) as the algorithmic backbone for envelope control. MPC is a model-based approach that predicts the vehicle motion over a finite time horizon. MPC allows us to incorporate dynamic models, system and environment predictions, and constraints on the system into one optimization problem. The constraints in particular describe the envelopes within which the vehicle must remain; it is the design and incorporation of these envelopes that make our approach unique.
For some examples of other research on envelope control and MPC for vehicles, check out the great work from the Dynamic Design Laboratory run by Professor Chris Gerdes at Stanford University (which is where Carrie’s exploration into envelope control began), or the Model Predictive Control Lab run by Professor Francesco Borrelli at UC Berkeley (where Sarah’s exploration into MPC began).
The Flexibility of Envelopes Versus Path Tracking
To better understand the importance of these envelopes, consider a classical control technique called Proportional Integral Derivative (PID) control. A PID controller is designed to track a reference by taking an error signal and computing a feedback control value that is composed of values proportional to the error, the integral of the error, and the derivative of the error. Technical jargon aside, one common application of PID for vehicle control is to track a path (this is the reference) given the information about where the vehicle is currently positioned relative to the path (this is the error). A PID controller will not know anything about envelopes — that is, where the edges of the road are, how much steering it can achieve, or what maneuvers may cause the vehicle to spin out. While a PID controller can be tuned to track a path quite well, it does not know anything about this safety critical information. Instead, the generated path must be designed to avoid those constraints, and the PID controller is assumed to be able to track that path perfectly — an assumption that is not always possible to be met.
For an autonomous vehicle, there may be a planner module that provides a path for the controller to track. Unless the planner can create the path with an unachievable amount of fidelity — including models of dynamics, delays, changes in road conditions, etc. — the controller will always have to correct for some model mismatch and/or disturbances, which can make tracking more difficult or impossible. When considering a driver safety system, the path tracking model breaks down even more. Predicting a driver’s future commands is not only difficult, but overcorrecting the driver during safe maneuvers leads to distrust and dislike for the system.
With envelope control, we can move toward letting the driver, or autonomous planner, drive freely in a region of trust or safety instead of enforcing the path tracking paradigm. The assumption that the vehicle will perfectly track the path can be dropped. When it comes to safety, better context allows for a more informed response. Envelope control can incorporate the constraint information directly into the controller — in other words, the algorithm that is steering and accelerating the vehicle now knows about the edges of the road, how much steering it can achieve, and what maneuvers to avoid that may cause the vehicle to spin out.
How is the math different between PID and envelope control?
To make this a bit less abstract, PID control is like a ball bearing sitting in the bottom of a bell-shaped teacup. Gravity (the PID controller) keeps the ball (the car), either sitting at or moving toward the center of the bottom of the cup (on the path). Envelope control is more like a flat-bottomed bowl or bucket, where the ball bearing (car) can move freely around the bottom, but rides along or is stopped by the edge of the bowl (edge of the road envelope). The second concept allows the driver or planner freedom of movement as long as it stays within the envelope, like the ball in the bottom of the bowl. PID, on the other hand, is always intervening, like the gravity keeping the ball in the apex of the cup, trying to keep the car doing a specific action — for example, staying on a set path, or following a set speed.
The ball in cup metaphor can also help explain how each type of control handles modeling error. In the case of PID, an error in modeling can be represented by the tipping of the cup. As the cup tips, the ball moves away from the only point it knows is safe on the path: the bottom of the cup. The PID controller has to work harder, against “gravity” to get the ball to recenter. With envelope control, model mismatch can be accounted for more gracefully. As the cup tips, the ball remains in the safe area, just maybe in a different position than expected. It may eventually hit the edge of the safe area, but it will still be bound by the edge of the envelope. Because we can differentiate between what is safe, or allowed, and what is not, there is more capability to adjust for errors than when the car is constrained to a specific or singular trajectory.
Enveloping the Environment
There are many envelopes that can define regions of safety for the vehicle in terms of position, velocity, acceleration, and chassis limitations. With cars, understanding the context of the road is paramount to safety. We can encode that context into, as one example that’s simple to explain, an environmental envelope. The environment is the physical space within which the vehicle is operating — essentially the road and the various agents (cars, motorcycles, bicycles, pedestrians, etc) around the vehicle. The environmental envelope outlines the physical space in which the vehicle is safe to drive or operate. For example, in the video below, the vehicle is constrained to drive within a cone corridor, which is similar to an environmental constraint that might be seen in a construction zone.
The convenience of the environmental envelope is that it is able to essentially “cut out” the dangerous areas — this flexible definition can then apply to applications as simple as emergency lane keeping, to keep the vehicle from running off the road, or to more complex situations that prevent the vehicle from colliding with another agent.
A Holistic Approach to ADAS
The ability to shape the environmental envelope to the current situation contributes to the holistic approach we are pursuing as the future of Advanced Driving Assistance Systems (ADAS). State of the art ADAS generally invokes a completely separate algorithm for lane keeping and a completely separate algorithm for obstacle avoidance (e.g. emergency braking); we can combine the envelopes for these systems into one system that considers all the constraints at the same time. The advantages to this are immense. For example, braking can be applied to support a lateral maneuver. Stability control can be activated while knowing to avoid the road barrier on the right and the vehicle on the left.
This is just one example of envelope applications with the environmental envelope. As part of the holistic systems approach to ADAS, we envision a control layer with a library of envelopes that give it full context for the vehicle’s safety in terms of position, velocity, and acceleration. The safety layer provided by this type of control structure is ideal for both driver safety and autonomous driving applications. In relation to this work, we are also proud to share that alongside Avinash Balachandran, TRI Head of Human Centric Driving Research, we were issued a patent on March 22, 2022 describing a shared control algorithm that unifies both system modes for Guardian and Chauffeur.
