Debugging tools

Some thought needs to be given to how the prototype is to be used in the lab. What debug or test tools will be needed in order to provide the necessary visibility? There are a number of tools which provide debugging capabilities such as access to the design’s internal nodes for probing, event triggers etc. 

The debugging tools add some logic to the FPGA’s design that captures selected signals into unused memory blocks based on programmable events. After capture, the signals are read out from the FPGA using the JTAG utility and are displayed in a variety of ways such as waveforms or showing values in the source code. While some tools require the user to instantiate the capture logic in the RTL prior to synthesis, other tools add the instrumentation logic at the netlist level during the FPGA implementation phase leaving the RTL intact. In addition, some of the tools allow quick tool configuration changes without needing to go through the often lengthy place & route process. 

Design instrumentation for probing and tracing

During the prototype bring-up process, often the design does not work right away and there is a need to diagnose its state and understand its behavior. Even after the initial bring-up, during active use of the working prototype, the design may exhibit unexpected behavior. In either case, the visibility provided by the FPGA and other component IO pins is not likely to be sufficient to properly debug the situation. 

The prototype debug process requires visibility beyond that available only at the IO pins and this section describes common techniques that help instrument the FPGAs that comprise the FPGA-based prototype. We will also give an overview of some tools which can greatly aid the debug process and further tool detail is offered in the appendices. 

Techniques for probing internal signals of a design implemented in FPGA fall into two general categories: real-time signal probing and non-real time trace capture. 

Real-time signal probing: test points

Viewing nodes in real-time is a common debugging practice, as in addition to signals states, real-time probing can uncover race conditions and unexpected “runt” signals. In real-time probing, signals from the design are taken to bench instruments such as logic analyzer or an oscilloscope for design analysis. Probed signals are either normally available at FPGA boundaries, or specifically brought from the design’s internal nodes to FPGA test pins. 

In this simplest method of probing designs’ internal nodes, the user directly modifies the design and brings internal nodes to FPGA pins for real-time probing. This method consumes little or no logic resources and only a few routing resources plus, of course, the actual IO pins that bring the probed signals to the outside world. 

In general, the RTL would need to be altered in order to add test points to the design, so this may simply not be an option for many projects. However, some tools support the inference of test points without changing the code. For example, Synopsys FPGA synthesis tools support an attribute called syn_probe which will do exactly that. 

If RTL changes do become necessary (and are allowed) then we should at least try to minimize the impact and scope of the changes. Thankfully, both major HDL languages support remote linking between different parts of a design hierarchy. This is achieved in VHDL using global signals and in Verilog-HDL using XMRs (cross module reference). More information and examples of using remote referencing is given in chapter 7. 

One disadvantage of making RTL changes to bring out test points is the long turn-around time to view a different signal or set of signals. To view a new signal, the FPGA design RTL source would need to be modified and then the design is passed through the synthesis and place & route process. To mitigate this process, which might be rather long, it is recommended to consider in advance a superset of those signals that might be required for probing and bring as many to visibility points as possible. However, FPGA IO pins are a precious resource in most FPGA- based prototyping projects so the number of spare IO pins on the FPGA available to act as test points is likely to be low. 

One simple and low-cost method for increasing the visibility into internal signals is to create in the design a “test header” port at the top-level to which we might connect the various signals and make changes with ease. To further minimize potential design spins or when the number of pins for signal viewing is limited, a slightly more sophisticated scheme where signals are multiplexed as shown in the following drawing: 

Figure 36: Test pin muxing

As shown in Figure 36, an eight-signal wide 4-to-1 multiplexer (MUX1) is added to the design and is statically controlled with two select bits from outside the FPGA. Such a multiplexer allows probing of eight signals at the time selected from a set of 32 signals.

A second eight-signal wide 4-to-1 multiplexer (MUX2) is shown but this is controlled by an internal processor or state-machine. This arrangement saves the two select pins and simplifies the multiplexer control in some situations. If possible, we should use XMRs or global signals to connect lower-level signals for observation to the multiplexer inputs (more about the use of XMRs in chapter 7). 

Note: probing using Xilinx ® FPGA Editor

Another method of modifying the design quickly is to edit the design at the netlist level using FPGA tools such as the Xilinx ® FPGA Editor. Using this tool the user can add design elements such as pins, and connect them to the nodes that need probing. It’s important to note that a tool such as Xilinx ® FPGA Editor is very powerful but complicated to use and requires a very detailed knowledge of the FPGA’s resources. We therefore only recommend the use of Xilinx ® FPGA Editor for experts only. There is further information in the references .

Real-time signal probing: non-embedded

This method of probing real-time signals is often provided as part of a commercial prototyping system, such as the CHIPit® systems from Synopsys, but could also be provided by in-house boards. The idea is to reserve a fixed number of pins from each FPGA and to route them on the board to a probing header, which acts as a debug port to which we can connect our logic analyzer. We can select the signals to be viewed on each FPGA, perhaps using a hierarchical design browser. The tool then directly modifies the already implemented FPGA netlist using one of the back- end sub-tools called Xilinx ® FPGA Editor, which then connects the desired signals to the probing header. 

It takes only a short time to implement the connection from internal FPGA logic to the debug port. This is possible using a tool like Xilinx ® FPGA Editor because we do not need to re-run synthesis or place & route. 

Care should be taken with interpreting the information given with such an approach because it is possible that signals may take widely different times to reach the debug port from their sources inside the different FPGAs. As this is all taking place on a completely implemented FPGA, the signals must take whatever paths remain available inside the FPGAs. Therefore, some of these paths will be long and via non-optimal interconnect resources, especially if the FPGA utilization is high. As a result, the timing relationship observed between the signals will not necessarily represent the real relationship at the sources. However, it is possible to use Xilinx ® FPGA Editor to measure the time delay of the path for any particular signal and then to use some logic analyzers to compensate for the known delay. 

In a perfect environment, the design modification process and any timing compensation is transparent to the user but even in the lab, it is very useful to be able to quickly extract a signal and observe it on a scope. 

Non real-time signal tracing

A shortcoming of direct FPGA signal probing is the limited number pins available for probing. A non real-time signal tracing uses a different probing mechanism and gets around this limitation. The non real-time design probing and debugging tools are available from FPGA and other EDA vendors. These tools comprise a mix of hardware and software, and provide a logic analyzer style capture in the FPGA where FPGA resources are used to implement and add to the design modules to monitor and capture a set of signals selected by the user. 

Using the vendor’s software tools, the user then configures the trigger condition and capture type relative to the trigger condition. Leveraging the FPGA “read back” feature, in which each FF and embedded memory in the FPGA can be read through the JTAG port. The content of the captured signal values are then transferred from the capture buffer in the FPGA to the software application running on a host computer using the FPGA JTAG pod, usually the same one used to configure the FPGA in the first place. The capture buffer content can either be displayed on the vendor’s application or by other waveform display tools. 

JTAG has some speed limitations, however, so some tools will make use of a faster FPGA configuration channel, if available. For example, Xilinx ® devices have a facility called SelectMap which is used for fast parallel configuration. Synopsys’ CHIPit debug tools use SelectMap to quickly read back the captured register information in this way. 

These tools are extremely useful for an individual FPGA debug, and in addition to their use as an “after the fact” analysis, some tools have ways to cross-trigger in order to synchronize external bench tools such as signal generators or software debuggers and this helps to correlate captured events across the system. 

While there are a number of similar tools available in the market place, the most common FPGA probing tools for the Virtex FPGAs are the Xilinx ® ChipScope series of tools from Xilinx and the Identify ® tools from Synopsys and we shall give an overview of these in the following paragraphs before moving on to a debug strategy. 

Signal tracing at netlist level

The ChipScope tool is an FPGA debugging tool from Xilinx for general purpose use, but with some special-purpose analysis versions available, for example for linking to embedded CPUs or analyzing fast serial IO. ChipScope tool works by adding extra instrumentation resources to the design in the device for the purpose of communication, triggering and trace capture. The instrumentation is synchronous to the design and uses the same system clocks as the logic being sampled, thus avoiding setup and hold issues. It should be noted that the trace capture is in actuality a sample of activity on an internal node of the FPGA. The timing resolution is of the trace data will be only the same as the capturing clock, therefore this is not a tool for analyzing absolute timing of internal signals. 

Figure 37 describes the ChipScope tool implementation flow, showing two ways to add the instrumentation items to an FPGA design. 

Of these two methods, the typical implementation flow is: 

• Using Xilinx’s CORE Generator™ tool the user defines and generates the communication core and logic analysis cores to be embedded into the FPGA. 

• The user then combines the cores with the FPGA design, either by instantiation into the RTL or directly into the synthesized design where the cores are merged at the netlist level so there is no need to modify the RTL design. 

• Design must go through the synthesis place & route process before the tool is ready to be used. 

The alternative is to insert the instrumentation at the netlist level: 

• Use the ChipScope tool core inserter to open the netlist (either the edif from synthesis or Xilinx internal netlist, ngc). 

• Select the clocks(s) and signals to be traced. 

• The core inserter builds a new netlist and there is no need to use the CORE Generator™ tool. 

• Pass results to place & route. 

• In either case, once the instrumented design is downloaded into the FPGA, the use for debug is the same. The user communicates with the logic analysis cores via the FPGA’s JTAG download cable in order to set up the trigger conditions, capture mode etc. When the trigger and capture conditions are met, the tool transfers the captured data from the FPGA and displays it in the tool’s application running on the host PC. 

As shown in Figure 38, multiple ILAs can be instantiated in one FPGA while using only one communication core (ICON). An interesting feature available with ILA is the ability to generate a trigger output signal. This configurable signal reflects the trigger condition and can be routed to an FPGA pin and used to trigger other bench instruments. Although there is latency of ten clocks between the trigger event and the trigger output, it can be still used to correlate the trigger events with other parts of the system. 

Figure 37: ChipScope tools design flow

Figure 38: ChipScope™ Pro system block diagram

To view signals captured with LA1, Xilinx provides a custom graphical signal browser as part of the ChipScope™ software, a screenshot is shown in Figure 39. 

Here we can see in the central horizontal panel an area for setting trigger values, which can be simple events or combinations of events over time using counters. There are two ways shown for viewing captured samples, one is showing a traditional logic analyzer style of view while the view to the bottom right is of the variation of a bus value over time, represented as an analog waveform. These signal viewers also allow a variety of features such as signal renaming, regrouping, reordering etc. so we are in a very familiar environment like a logic analyzer or scope, even though the “engine” of it all is embedded in the FPGA. 

Figure 39: Screenshot of ChipScope™ debug tool

In addition to displaying captured data using the signal browser, ChipScope™ software can export the captured data in a variety of formats, including VCD (Value Change Dump), ASCII and Agilent’s FBDF (Fast Binary Data Format). Exported data can then be further viewed or analyzed by user scripts or by a wide variety of third-party tools. 

Signal tracing at RTL

Identify ® is a tool supplied by Synopsys which works in a similar way to the Xilinx ® ChipScope tool except that instrumentation takes place at the source RTL rather than at the netlist level. Like ChipScope software, Identify instruments the design by the inclusion of signal monitoring logic, trace capture and buffering, and communications to a host PC. In addition, Identify tools provide signal selection and monitoring mechanisms which enable the user to more easily trigger the trace capture and to correlate the captured information back to the source RTL. 

Identify tool is comprised of two sub-tools, called the RTL Instrumentor and the RTL Debugger: The key part of the instrumentation logic itself is called an IICE (pronounced “eye-ice”) or intelligent in-circuit emulator, which is embedded into the user’s design. The IICE contains the probing logic, the trigger logic and trace buffers plus the runtime communication to the host running the RTL debugger. 

Figure 40 gives an overview of the concept behind an RTL debugger. This is actually a screen shot and block diagram of the Synopsys Identify tool in its basic configuration. The tool inserts an IICE probing block that samples internal node information and gathers it together into a sample buffer, generally using one or more of the available FPGA BlockRAMs. 

Figure 40: Synopsys Identify ® tool overview

The IICE also includes triggering and sequencing logic for deciding when samples are captured. The IICE communicates over JTAG to the application where breakpoints and the state of the captured signals are shown and highlighted in the design’s RTL source. 

Figure 41 describes the general usage flow for Identify, showing a two-pass strategy to focus in quickly to “zoom in” on data required for a specific debug task. As shown in the left process (first pass), Identify cores are added to the design before synthesis, followed with the place & route processes, but subsequent changes (second pass) to the instrumented signals are implemented incrementally where the synthesis and place & route steps are by-passed. 

Figure 41: Identify use model and flow

The following describes in more detail Identify’s main usage flow: 

• Using Identify’s Instrumentor, the user selects signals to monitor and trigger conditions directly in the design from a hierarchical RTL source viewer. 

• Instrumentation automatically generates the IICE core. 

• After synthesis and place & route, the IICE core is controlled by the Identify Debugger where the user sets the trigger condition and arms and monitors the running hardware via the FPGA JTAG facility. 

• After trigger, data captured in FPGA memory is uploaded to the Identify Debugger application software over the FPGA JTAG facility. 

• Captured data can be displayed in a variety of ways: either directly annotated into the RTL code, or using common third-party waveform viewing tools. 

• Small changes to the signal list and trigger conditions can be quickly made incrementally without going through the synthesize place and route processes. 

A design can contain multiple IICEs, each configured in different clock domains and allowing different trace and trigger conditions. IICEs can cross-trigger between themselves so as to track a complex sequence of events. Multiple IICEs in an FPGA share the same communication mechanism with the JTAG interface. 

In addition, the IICE provides the option of exporting the trigger condition, where a copy of the trigger signal is brought to the design top level and used to sync to bench instruments or software debuggers for enhanced system-level debugging. Linking to a software debugger is covered in more detail in chapter 11. 

The Identify RTL Instrumentor is a sub-tool running on the host by which the user browses the design RTL in order to select the signals to be traced and if desired added to various cones of trigger logic. During instrumentation, the IICE is 

Figure 42: RTL Instrumentor source code view

configured and added. The most useful aspect of making such instrumentation at the RTL is that we can easily keep track of what visibility we are adding. Figure 42 is an excerpt from an Identify screen shot which shows the RTL source code appears during an instrumentation process. The available signals for sampling are underlined and the cartoon spectacles change color if that signal is sampled, or part of the trigger logic, or both. The sphere icons on the left show where breakpoints might be inserted adding a useful feature to allow us to trap when the design reaches a certain line of code in much the same way as software engineers might debug their code. 

Let us now look at the in-lab debugger part of Identify. The RTL Debugger is the sub-tool that provides an interactive interface that we use in the lab to control the triggering, capture, and various flow-control features such as breakpoints. The debugger also provides the interface to waveform viewers. 

Like ChipScope software, the Identify debugger provides a number ways to view the captured sample data and signals including: 

• State annotation: when using break points, it annotates the signals’ state directly into the source code using the supplied design navigator tool. Figure 43 shows a screen shot of Identify RTL Debugger displaying the RTL source annotated with breakpoints and captured data states. In reality, the captured data is overlaid on the source code, highlighted in yellow and users can scroll through sampled data and see the values on the source code screen change. This is roughly analogous to the stepping through a software debugging tool. 

• Waveforms: in addition to state annotation, Identify Debugger can interface with popular waveform viewers such as the freeware GTKWave viewer or the DVE environment provided with Synopsys simulators. The trace samples can also dump the signals into a standard VCD (Value Change Dump) file that can also be displayed on a wide variety of available waveform viewers. 

Figure 43: RTL debugger source code view showing sampled data

FPGA resources used by Identify consist of the capture logic and the capture buffer that is usually implemented in on FPGA BlockRAM. The size of the analysis resources depend on the number of signals to be captured, the trigger condition and to a greater extent, the desired capture buffer depth. 

Summarizing debugging tool options

One of the traditional complaints against FPGA-based prototyping was that visibility into the design once inside FPGAs was very poor and that debug was non- intuitive because even when we had visibility, we didn’t really know what we were seeing. After this section, we have seen many different ways to add debug visibility into our prototype and there are many other tools available which are variations on the instrumentation approach taken by Identify and ChipScope software. We have summarized the approaches in Table 6. 

Table 6: Comparing debugging technique and tools explored in this chapter