Debugging on-board issues

After giving ourselves confidence that the FPGA board is fully functional and  configured correctly, and also having checked the implementation and timing  reports for common errors, we can consider that our platform is implemented  correctly. From here on, any functional faults in the operation of the design will  probably be bugs in the design itself that are becoming visible for the first time. The  prototype is starting to pay us back for all our hard work. 

The severity and number of the bugs discovered will depend upon the maturity of  the RTL and how much verification has already been performed upon it. We shall  explore in the remainder of this chapter some common sources of faults.

Sources of faults

Every design is different and we cannot hope to offer advice in this manual on  which parts of the design to test first or which priority to place on their debug.  However, we can offer some guidance on ways to gain visibility into the behavior  and some often-seen on-board problems. 

Assuming the design was well verified before its release to use, we should be looking for new faults to become evident because the design on the bench is  exposed to new stimulus, not previously provided by the testbench during  simulation. The cause of such faults can be found in three main areas, as listed in  Table 29.

Type 1 bugs should be caught by the normal RTL verification plan and as  mentioned previously, an FPGA-based prototype is not the most efficient way to  discover RTL bugs, especially when compared to an advanced verification  methodology like VMM. Nevertheless, RTL bugs do creep into the prototype either  because there is an unforeseen error exposed by real-world data or because the RTL  issued to the prototyping team is not fully tested. 

Type 2 bugs are related to the SoC’s interface with external hardware. All SoC’s are  specified to be used in a final product or system but there is a diminishing return on  making specifications ultra-complete. Eventually there may be some combination of  external components that do not fit within the spec and the design needs to be  altered to compensate. This is often the case when extensive use of IP means that  the SoC is the first platform in which a certain combination of IP has ever been used  together. Alternatively, the design itself may be an IP block that is being specified  to run in a number of different SoC designs for different end-users. Predicting every  possible use of the IP is hard and corner cases are often discovered during the  prototyping stage. 

Type 3 bugs are the most valuable for the prototyping team to find. The prototype is  the first place where the majority of the embedded software runs on the hardware.  The interface between software and hardware is very complex having time  dependency, data dependency and environmental dependency. These dependencies  are difficult to model properly in the normal software development and validation  flows, therefore on introduction to real hardware running at (or near) real-speed, the  software will tend to display a whole new set of behaviors. 

Any particular fault may be a combination of any two or indeed all three types of  bug, so where do we start in debugging the source of a fault?

Logical design issues

Assuming that our newly found bug is not a known RTL problem, already discovered by the verification team since delivering the RTL for our prototype  (always worth checking), we now need to capture the bug. We need to trace enough  of the bug-induced behavior in order to inform the SoC designers of the problem  and guide their analysis. This means progressively zeroing-in on the fault to isolate  it in an efficient and compact form for analysis, away from the prototype if  necessary.  

The first step in identifying the fault is to isolate it to the FPGA level. To  accomplish this, we need visibility into the design as it is running in the FPGAs. As  mentioned in chapters 5 and 6, it is good practice to provide physical access to as  many FPGA pins as possible so that they can be probed with normal bench  instruments such as scopes and logic analyzers. Test points or connectors at key  pins will hopefully have been provided by the board’s designers for checking key  FPGA-Based Prototyping Methodology Manual 335  signals, resets, clocks, etc. However, most FPGA pins will usually be unreachable,  being obscured as they are by the FPGAs ball grid array (BGA) package and  therefore we need some other form of instrumentation.

The simplest type of instrumentation is to sample the pins of the FPGA using their  built-in boundary scan chains, often referred to by the name of the original industry  body that developed boundary scan techniques, JTAG (Joint Test Action Group).  FPGAs have included JTAG chains for many years, primarily to assist test during  manufacture and ensuring sound connection for BGAs. JTAG is most commonly  known to FPGA users as one of the means for configuring the FPGA devices via a  download cable. JTAG allows the FPGA’s built-in scan chains to be used to drive  values from the FPGA pins internally to the device and externally to the  surrounding board. Through intelligent use of the JTAG chains and intelligent  triggering of the sample by visible on-board events, a JTAG chain can capture a  series of static snapshots of the pin values, helping to add some visibility to a debug  process. 

Greater visibility at FPGA boundaries or at critical internal nodes is provided by  instrumentation tools such as ChipScope and Identify as described in chapter 3. As with any of these tools, there is an inverse correlation between how selective we are  in our sampling ‘vs’ the amount of sample data. Since data is usually kept in any  unused RAM resources available in the FPGA, we should expect that we will not be  able to capture more than a few thousand samples of a few thousand nodes in the  FPGA. Therefore we need to use some of our own intelligence and debug skills in  order focus the instrumentation on the most likely sites of the fault and its causes.  

A good Design-for-Prototyping technique is for the RTL writers to create a list of  the key nodes in their part of the design i.e., a “where would you look first” list. This list would be a useful starting point for applying our default instrumentation.

Logic debug visibility

There is a traditional perception that FPGA-based prototyping has low productivity  as a verification environment because it is hard to see what is happening on the  board. Furthermore, the perception has been that even when we can access the  correct signals, it is difficult to relate that back to the source design.  

It is certainly true that FPGA-based prototypes have far lower visibility than a pure  RTL simulator, however, that may be the wrong comparison. The prototype is  acting in place of the final silicon and as such it actually offers far greater visibility  into circuit behavior than can be provided from a test chip or the silicon itself.  Furthermore the focus of any visibility enhancement circuits can be changed,  sometimes dynamically. As a short recap on the debug tools explored in chapter 3, we can gain visibility into  the prototype in a number of ways; by extracting internal signals in real-time and  also by collecting samples for later extraction and analysis.  

Real-time signal probing: in this simplest method of probing designs’  internal nodes, we directly modify the design in order to bring internal  nodes to FPGA pins for real-time probing on bench instruments such as  logic analyzers or oscilloscopes. This is a common debugging practice and  offers the benefit that, in addition to viewing signals’ states, it is also easier  to link signal behavior with other real-time events in the system.  

Embedded trace extraction: This approach generally requires EDA tool  support to add instrumentation logic and RAM in order to sample internal  nodes and store them temporarily for later extraction and analysis. This  method consumes little or no logic resource, very little routing resource  and only those pins that are used to probe the signals.

We shall also look at two other ways of expanding upon debug capability,  especially for software:  

Bus-based instrumentation: some teams instantiate instrumentation elements into the design in order to read back values from certain key areas  of the design, read or load memory contents or even to drive test stimulus  and read back results. These kinds of approaches are often in-house  proprietary standards but can also be built upon existing commercial tools. 

Custom debuggers: parts of the design are “observed” by extra elements  which are added into the design expressly for that purpose. For example, a  MicroBlaze™ embedded CPU is connected onto the SoC internal bus in  order to detect certain combinations of data or error conditions. These are  almost always user-generated and very application-specific.

Bus-based design access and instrumentation

The most commonly requested enhancements to standard FPGA-based prototyping platforms are to increase user visibility and access to the system, including remote  access. We have seen how tools like Identify® and Xilinx® ChipScope tools offer  good visibility into the prototype but these communicate with their PC-hosted  control and analysis programs via the FPGA’s JTAG port. As mentioned, the  bandwidth of the JTAG channel can limit the maximum rate that information can be  passed into or out of the prototype. If we had a very high bandwidth channel into  the prototype, what extra debug capability would that give us?

Debug situations where higher bandwidth would be useful include:  

• High speed FPGA configuration  

• High performance memory access 

• High speed download and upload of software images  

• On-the-fly capture and pre-setting of memories 

• High-performance data streaming 

• Remote configuration and management 

We can provide a higher bandwidth interface by using a faster serial connection or  by using a parallel interface, or even a combination of the two. Very fast serial  interfaces from a host directly into FPGA pins on a board is a difficult proposition  but we could envisage a USB 2.0 connection and embedded USB IP in the design  which might be used for faster communication, replacing the standard JTAG  interface.

A simpler approach used by a number of labs is the instantiation in the design of  blocks which can receive data in a fast channel and distribute it directly into parts of  the design via dedicated fast buses. The instantiated block could be a simple register  bank into which values can be written which then override existing signals in the  design. Another use of a bus-based debug block might be to act as an extra port into  important RAMs in the design, for example, changing a single-port RAM into a  dual-port RAM so that the extra port can be used to pre-load the RAM.

The advantages of this kind of approach are clear, but even with the use of our  chosen hardware description language’s cross-module references this might mean  some changes to the RTL. However, much of this change could be automated, or  inserted after synthesis by a netlist editor. It might even be adopted as a companywide default standard, much as certain test or debug parts are added into SoC design  for other purposes during silicon fabrication.

These advanced communication and debug ports are traditionally the reserve of  tools which work on emulator systems but are starting to become more common in  FPGA-based prototypes as well. One example of this is the Universal Multi  Resource bus (UMRBus®) interface originally developed by one of the authors of  this book, René Richter, along with his development director at Synopsys, Heiko  Mauersberger (in fact, their names were the original meaning of the M and the R of  UMRBus).

UMRBus, as the new name suggests, is a multi-purpose channel for high-bandwidth  commutation with the prototype. It works by placing extra blocks into the design  and linking them together via a bus-based protocol which also communicates back  to PC-host. There are a number of blocks and other details which we will not cover  338 Chapter 12: Breaking out of the lab: the prototype in the field here, but at the heart of the UMRBus is a simple block called a client application  interface module, or CAPIM, a schematic of which is shown in Figure 140.

As we can see in the diagram, a CAPIM appears as a node on a ring communication  channel, the UMRBus itself, which carries up to 32 bits of traffic at a time. We can  choose a width which best suits the amount of traffic we want to pass. For example,  for fast upload of a many megabytes of software image, we might use a wider bus  but for setting and reading status registers in the design a 4-bit bus might suffice,  saving FPGA resources. Each CAPIM is connected into the design, often using  XMRs to save boundary changes, to any point of interest.

In Figure 141, we can see the use of three CAPIMs on the same UMRBus, each  offering access and control of a different part of the prototype. In this example,  UMRBus is allowing read and load of a RAM, of some simple registers and test  points or to allow reprogramming of the book code in the design.  

These functions might all be in the same FPGA or spread across the board across  different segments of a UMRBus. Similar in-house proprietary bus-based access  systems should also be designed to allow for multi-chip access and cross-triggering.

Benefits of a bus-based access system

Some labs have developed their own variations on this bus-based approach and each  will have its own details of operation, however, the general aims and benefits are  the same as those listed above, so let’s explore how this extra capability can help the  FPGA-based prototyping project. 

• High speed FPGA configuration: higher bandwidth access to the  prototype significantly speeds up the speed at which designs can be  downloaded onto the board compared to serial methods typically used.  This is especially valuable early in the design cycle when hardware-related  design changes most often occur and we are debugging the design’s first  Figure 141 : UMRBus with three CAPIMs connected into various design blocks 340 Chapter 12: Breaking out of the lab: the prototype in the field runs on the board. The use of a GUI or a command-line interface for  configuration and programming benefits greatly from “instant” access and  fast configuration, avoiding those irritating delays for a few minutes  configurations time. • An example of a configuration approach which uses a proprietary busbased interface is the CONFPRO unit from Synopsys which uses the  UMRBus protocol to send large amounts of data over USB to an embedded  supervisor on the FPGA board which configures the FPGAs in one of their  faster parallel modes (see chapter 3) rather than via a JTAG cable. 

• High performance memory access: the ability to read and write directly  to the memory on the FPGA-based prototype can dramatically reduce  bring-up time. On a prototype there can be many megabytes of the FPGA’s  internal RAM in use at any time. Memory pre-load and read-back  functions can be more easily implemented if an extra bus is placed into the  prototype for that purpose rather than trying to employ the SoC’s own  CPUs and buses to achieve the same result. We can avoid software rework  or scheduling issues involved in having the SoC CPUs simply listen to a  host-controlled port and pass data to a RAM. 

• Direct access to memory via something like the UMRBus enables us to  view memory contents and also rapidly download, upload and compare  large quantities of memory content under script control or via TCL  commands. A lab library of pre-defined interface objects might be  available for connection to our debug bus, such as memory wrappers,  which provide a second port into a RAM. This minimizes the need for onthe-spot modeling of many components and speeds access to the system  during debug. For example, Synopsys keeps pre-defined IP in the form of a  UMRBus-to-SDRAM component, which enables direct access to SDRAM  for programming, pre-load and read-back without re-synthesis and/or place  & route changes. 

• High speed upload of software images: a specific use of the fast-memory  access is for loading software images. Since the major use of the prototype  may be for enabling a fast and direct platform for the software team, we  should enable their normal fast and iterative working methodology. A  software image ready for loading into the CPU might be very quickly  generated using the normal compile and linking tools. It would then be  irritating if it took far longer to load the result into the platform in order to  run in. Estimates by colleagues using JTAG-based interfaces tell of 30  minutes to upload a typical software image. This could be cut to  considerably less than a minute using a higher-bandwidth interface. 

• High-performance data streaming: another use of the faster access into  the prototype might be to input data streams from the host at a rate fast  enough to fool the SoC design into thinking that it is coming from the real  FPGA-Based Prototyping Methodology Manual 341  world. Some types of data input might not be suitable for this approach, for  example, interactive data or data with a high-rate of delivery such as raw  network traffic. Other data, such as imaging or audio data would fit  particularly well into this approach. We could, for example, deliver  recorded video to the SoC from the host via a bus-based interface to test  the software’s capability in a certain video processing task. 

•It is a short step from this proprietary data delivery to adopting an industry  standard for passing data and even transactions into and out of the  prototype. We shall explore this area further in chapter 13. 

• Remote configuration and management: with advanced bus-based  access, we can remotely access and program the prototype via a host workstation. Board types and configurations could be scanned and their  setup requirements automatically detected. Board initialization,  configuration and monitoring could then be handled remotely, lightening  the burden for a non-expert end-user. For this, it may be necessary to drive  the bus-based access from a standard interface port, such as USB. This  could be performed via a hardware adaptor which runs the bridge from  PCIe (in this case) to the on-board bus access. In the HAPS-60 series of  boards, a hardware interface module called CONFPRO connects the PCIe interface on the host workstation to the UMRBus interface on the  prototyping system.

In very advanced cases, the bus-based access can become a transport medium for  protocol layers but this might be a large investment for most project-based labs,  therefore we might expect these types of advanced use modes to be bought in as  proven solutions from commercial board and tool vendors. 

Once we start to employ the most sophisticated methods for debug and  configuration of the system, we might begin to explore other use modes including: 

• Direct link to RTL simulator 

• Support for transaction-based interface via SCE-MI  

• Hybrid system prototyping with virtual platforms We shall explore these in chapter 13.

Custom debug using an embedded CPU

Most of the SoC designs which have embedded processors would also have built-in  custom debuggers. These custom debuggers would be used to connect to the  processor bus present in the SoC designs, usually to load software programs to be  run on the processor, debug the software programs by single stepping or breakpoints  and access all the memory space of the processor bus. These custom debuggers will  342 Chapter 12: Breaking out of the lab: the prototype in the field be present inside the design which will be connected to the external world through  some dedicated IO ports. Usually a debugging software utility running on a laptop  will be connected to these FPGA IO ports using some hardware connected through  USB or other IO ports as shown in Figure 142. Using the software utility we can  load programs into the program memory, debug the software and access the  memory space in the SoC.

If the design to be prototyped has this debugger, then it can be used to debug the  design if it doesn’t work on the board. We should take the dedicated IO ports  coming from the inbuilt debugger out of the prototyping board using IO boards or  test points. Then the debugging software utility can be connected to these IO ports.  After programming the FPGAs, the software utility can be asked to connect to the  internal debugger present in the design. Upon successful connection to the internal  debugger, we can try to read and write to all the different memory spaces in the  processor bus. Then a simple program can be loaded and run on the processor  present in the design. Finally the actual software debugging can happen over the  real design running on the prototyping board.