The Quest for Bugs:
“Shift-left, right?”
​
​
​
​
​
​
​
​
​
​
​
7 minute read
Joe Convey, Acuerdo Ltd.
Bryan Dickman, Valytic Consulting Ltd.,
​
(Virtual Prototyping finds software and hardware bugs earlier and faster)
Shift-left, why?
Shift-left testing[1] is an approach to software and system testing which is performed earlier in the lifecycle (that is, moved left on the project timeline). It is the first half of the "Test early and often" maxim that was coined by Larry Smith in 2001.
It’s now an established idea, much talked about in both software and hardware engineering as a philosophy to drive testing as far to the left as possible on the project timeline, thus, removing bugs earlier, when they are cheaper and faster to find, and the bug impact costs are much lower.
So, shift-left has the potential to magnify these cost savings by reducing the number of bugs that are propagated downstream only to be found and then fixed at a much later time when the rework costs may be much higher and associated failures cause serious risks to life, or function.
We are going to discuss shift-left in the context of using virtual prototyping at the front-end of the development lifecycle. We will explain how shift-left with virtual prototyping will find software bugs earlier, and also find hardware bugs earlier. In the “quest for bugs”, finding bugs earlier is a significant win for the developers and users of ASIC.
​
Shift-left, how?
Virtual Prototyping enables a shift-left of the overall product development time by allowing development of the system software and firmware in parallel, even ahead of the design RTL bring-up. The Virtual Prototype (VP) decouples the system software development (BIOS, firmware, device drivers), OS porting/development, and application software development, from the RTL hardware development. VP Kits are easy and inexpensive to deploy to both internal and 3rd party software developers. This shift-left of the software bring-up enables the software developers to flush out most of the software bugs ahead of the hardware development. It’s essentially frontloading the software bug quest. The following diagram illustrates the measurable benefits of this shift-left for software development. However,Virtual Prototypes fulfil several other demands that will reduce the overall effort for RTL verification. By eliminating most of the software bugs ahead of the point where software-driven verification begins, the debug effort is significantly reduced, and the engineering effort can be focused on finding RTL bugs more efficiently.
Bringing up new hardware and new software/firmware together for the first time, leads to a double-blind situation which means debugging both in parallel – difficult!
​
Virtual prototyping enables a "find and fix" approach for most of the software/firmware bugs ahead of RTL verification. Further, the use of VP models in hybrid verification workflows (simulation, emulation and FPGA prototyping), reduces the mean-time to find bugs, by significantly accelerating these environments, especially when replacing a major RTL component such as a processor, with a fast VP model.
Source – Acuerdo and Valytic Consulting
Better Software. Faster![2]
​
In the above diagram, without Virtual Prototyping, System-software development (i.e. firmware/BIOS, RTOS porting, device drivers, and also possibly compilers and other software tools) is dependent on hardware development to provide a stable software execution environment. System-software developers will need to develop and validate code by running on either RTL emulation models, or FPGA prototypes. This presents a number of challenges in distribution, performance, and availability of the software execution environment. As mentioned in “System Bug Quest” and “Deep Cycles”, this can be accomplished with a centralized and scaled-out FPGA prototyping farm (or cluster).
This may be achievable for your internal software team (the system-software team), but what about internal or external Application-software developers, say for example an automotive company SW team developing a new autonomous driving chip? This community of software developers may need to wait for prototype silicon in the form of development boards, which can push application-software development far out to the right.
Earlier system-software
When a VP Kit (VPK) is developed up-front, the system-software is decoupled from the RTL development and can be developed in parallel. Thus, we have a shift-left for the delivery of the system-software.
Earlier application-software
Application-software is also decoupled from the HW development and is no longer dependent on the availability of hardware in the form of development boards. VPKs are easy to distribute to software development teams, since they are software tools. However, the application-software development still has a dependency on the availability of the system-software in most cases, especially for vertical end-to-end integration testing of the entire software stack.
DevOps enabler
CI/CD[3] (continuous integration/continuous delivery) is a standard best practice for modern software developers who are aligning to a DevOps approach. For the CI part, the cycle of build-test-merge can be enabled by a fast VP model, allowing fast regression testing of software code changes, as the gate-keeper for codebase repository commits. This continuous testing of the software ensures that bad commits are avoided, the codebase is “always-working” and software bugs are minimized even under high levels of code churn.
​
Better Hardware. Faster!
Hardware tape-out is generally dependent on completion of software-driven system validation. This phase is normally performed on hardware acceleration platforms (emulators, or FPGA prototyping systems). Software-driven system validation (the clue is in the name), requires the software, so cannot begin until sufficient system-software is available as validation payloads.
​
A VPK enabled workflow yields several realisable benefits for the hardware development lifecycle.
Less bugs in the hardware/software interface
Although additional investment is required for the VPK development, arguably the RTL development will be shorter as the hardware/software interface has been pre-validated and is no longer part of the RTL development process. Otherwise, the hardware-software dependencies might not be exposed until later in the system-software development phase, and this could lead to more impactful re-working of the hardware leading to an overall higher hardware development cost. Co-development of the hardware and system-software would allow such issues to be identified sooner and more easily absorbed.
Software is ready
With a VPK based workflow, system-software arrives much earlier thus enabling a system validation shift-left and earlier tape-out. Since the system software has been pre-validated against the VPK, it will be more stable, and you avoid the double-blind debugging situation mentioned previously.
Verification shift-left
VP models can be used to shift-left the development of RTL testbench environments thanks to co-simulation. For example, the VP can be used as a “Virtual-DUT” so that testbenches and test-cases can be developed, and verification bugs can be reduced in advance of RTL. Further, system-level test-cases can be developed in advance of RTL to speed up system integration testing.
“Accelerated acceleration”
VP models written in SystemC/TLM can be mixed and matched with RTL components to build hybrid- emulation, or hybrid-FPGA-Prototyping environments. The VP components are simulated on the host workstation while the RTL components are mapped to the hardware acceleration platform. This can be done early on in the development lifecycle to enable prototype development in advance of RTL, or when building a VP model of the complete system where one or more components do not exist as VPs but RTL is available. Alternatively, it can be used to swap out larger off-the-shelf RTL components (e.g. processors) with fast VP models, leaving only the RTL-under-development to run on the hardware acceleration platform. For example, replacement of a processor RTL model with a processor cycle approximate VP model will speed up code execution runtimes by several orders of magnitude (e.g. OS boot). Further, a reduction in the footprint of the hardware accelerated RTL model may increase the hardware acceleration system clock speed (e.g. less FPGAs required to map the design, reduces design partitioning overhead and effect on clock speed), and free up hardware acceleration resources. Modern emulation and FPGA prototype systems are designed to connect to VP models through libraries of ‘transactors’. These transactors allow the efficient mapping of virtual I/O transactions onto the pin-level activity of the RTL model. They are abstraction-level converters.
These benefits for hardware acceleration yield a further shift-left effect for the software-driven system validation phase of hardware development.
Source – Acuerdo and Valytic Consulting
Power validation feed-forward
When your virtual prototyping environment supports power exploration, these findings can be used to feed-forward into emulation power analysis workflows (see “Power Bug Quest” for more details).
Shift-left, right!
Recent reports show the global market for EDA Tools, estimated at US$10.5 Billion in the year 2020, is projected to reach a revised size of US$18.7 Billion by 2027[4]. Over a similar period, virtual prototyping is predicted to rapidly grow to a market size in excess of $860M by 2025[5]. It is perhaps surprising that this figure isn’t a larger proportion of tool spend, given the clear advantages it offers as a method to accelerate bug avoidance and avoid spending disproportionately more on other technologies, later in the design cycle.
​
Time to shift investment left?
In the semiconductor business the pressures of time to market and for products to be correct-by-design, the motivation is slightly different. Many companies have a verification philosophy that goes something like, “spend more on everything to avoid disappointment”. In other words, shift investment left to VP and “right” to emulation and FPGA simultaneously. However, not all companies can afford to do that, so it is well worth asking the “balance” question; what issues could we avoid by investing more in VP technology, as well as the necessary investments in emulation and FPGA and what are the likely returns?
While the most obvious and widely acknowledged advantage is the shift-left effect on software development, there are many other benefits that apply to hardware development with an overall multiplying effect on the final product delivery shift-left outcome!
Where do you and your team sit on the shift to virtual prototyping line…?
References
[1] https://en.wikipedia.org/wiki/Shift-left_testing
[2] Although published in 2014, this book gives a very good overview of Virtual Prototyping use-cases and is still highly recommended. https://www.synopsys.com/company/resources/synopsys-press/better-software-faster.html
[3] https://www.redhat.com/en/topics/devops/what-is-ci-cd
[4] https://www.reportlinker.com/p05817912/Global-EDA-Tools-Industry.html
[5] https://www.grandviewresearch.com/industry-analysis/virtual-prototype-market
Read the full whitepaper "Power Bug Quest"
