On-board clock resources

Using a modern FPGA fabric as an example, we see a remarkable variety of clock resources inside the device as shown in chapter 3. If the SoC design is small enough to be placed inside a single FPGA then as long as the FPGA has the required number of clock domains then the SoC clock network will probably be accommodated by the sophisticated clocking networks in the FPGA, including PLL, clock scaling and synchronization between local clocks and global clocks. 

Having claimed that, if SoC clock networks are small enough they can be accommodated by a single FPGA, once the SoC design must be partitioned across multiple FPGAs then an extra layer of complexity is involved. In effect, the top- layer of the clock network hierarchy needs to be accommodated by the board, rather than inside the FPGA devices. One useful approach is to consider the board as a “Super-FPGA” made from the individual FPGAs plus top-level resources joining them together into the overall platform. 

To replicate an FPGA’s full hierarchy of clock functionality at board level would require similar resources to those found inside each FPGA. So, for example, we would need board-level PLLs, clock multiplexing, clock dividers and multipliers, clock synchronization circuits and so forth. 

In many cases, we can simplify this requirement by using tools to simplify the SoC clocking into something that we can handle on our boards. For example, converting some of the SoC design’s gated clocks into equivalent global FPGA clocks, plus relevant clock enables, in order to reduce the total number of clock domains. This gated-clock conversion can be automated without altering the RTL as will be shown in detail in chapter 7. 

Building fully flexible clock resources into a board to meet the requirements of a variety of SoC designs requires a great deal of experience and expertise and so it may be tempting to create our board for a single project only. The board would be made with only those top-level clock and other resources necessary to map the specific SoC design. This is certainly much easier than planning for a fully flexible arrangement for all possible future FPGA-based prototypes. However, a board with inadequate flexibility in the clock resources will place greater constraint on the partition decisions should the design change to some degree during the project. It will also heavily reduce the ability for the board to be used across multiple projects because the top-level clock resources for one SoC project may not closely match those for subsequent projects. In the authors’ experience, it is limitation of the top- level clock resources that is a common reason that in-house boards are not often re- usable across many projects. 

Matching clock delays on and off board

A clocked sub-system in the SoC will be implemented in silicon with tight control and analysis of clock distribution delays. When such a sub-system is split across multiple FPGAs, we must not introduce skew into our clock networks. The partitioning of an SoC design into multiple FPGAs will require control of inter- FPGA delays and especially inter-board delays, if multiple boards are to be used. Some of this can be done by properly constraining the FPGA tools and the task might be made easier by reducing the overall target clock speed for the prototype. However, regardless of the clock speed the best way to ease this task is to design our boards with matched and predictable clock distribution resources. 

As an example of delay matching, we look no further than the Synopsys ® HAPS ® family of boards. These are laid out with quanta of delay which are repeated at key points in clock distribution paths. In the example in Figure 51, we see two quanta of 

Figure 51: Delay matching on and off board

delay, X and Y, which are used as fundamental values during the board design. Y is the delay of typical clocks along a particular standard length co-axial cable with high-quality shielding and co-axial connectors which is mass produced and used widely in HAPS-based platforms. Having a single type and length of cable with relatively constant delay meant that the on-board traces can be designed to the same value Y. It is a small matter for PCB designers to delay match traces on the board, although the zig-zag trace routing employed may increase the need for extra board routing layers. The other quantum of delay, X, is the stub delay to and from FPGA pins and nearby connectors and is relatively simple to keep constant. The values for X and Y will be characteristic for a given family of boards, for example, for one family of boards, X= 0.44ns and Y = 1.45ns. With care, the values of X and Y can be maintained across new board designs in order to allow easier mixing of legacy and new boards in the same prototype. 

We can now see from the diagram that clock delay between the clock source and any of the FPGA clock pins will be the same value, 2X+Y, i.e., 2.33ns for our HAPS example, and thus we have minimized clock skew across the partition. 

Note, however, that the received FPGA clocks, although mutually synchronized, will not be in sync with the master clock input, F. If we wish to use F and its distributed derivatives, then we will need to employ phase-locked loops, which we will now discuss. 

Phase-locked loops (PLL)

PLLs are extremely useful for removing insertion delay in cross-board clock

Figure 52: PLL used to re-align FPGA clocks

networks, for example, to reduce clock skew between different parts of the prototype. Figure 52 shows a board-level PLL driving four FPGAs. As above, with the use of trace-length matching, the stubs at each FPGA clock input and the stubs at the PLL outputs can be made equal to the value X. Over many board manufacturing runs, the absolute value of X may vary but will be consistent enough for any given board. 

We see also that the FPGAs are linked to the PLL outputs by equal length cables or delay Y. Co-axial cables and high-fidelity clock connections are highly recommended for best reliability and performance. We can see that the on-board delays from PLL to each FPGA are matched. 

For providing the necessary clock resources at the top-level of the “Super-FPGA,” the board should include PLL capabilities, which are useful for a number of tasks. Although the FPGAs used may include their own PLL features, the boards should include discrete PLL devices such as those available commercially from numerous vendors. 

System clock generation

The clock generation and its flexibility are critical to the reliable operation of a multi-FPGA Prototyping system. Some of these capabilities must be implemented at the board level, but others must make use of the available clock resources within the FPGAs. It is therefore important that from the beginning we understand the available clocking resources on the chosen FPGA. As mentioned in chapter 3, FPGA devices today have very capable clocking capabilities including DLLs, PLLs, clock multiplexers and numerous global and regional clocks across various regions of the FPGA fabric. 

Once the available clocking resources in the FPGAs are understood, we should determine what additional external clock sources and routing will properly exploit them and maintain the most flexibility for all future applications. 

The following is a list of recommended considerations: 

• Where is the clock sourced? 

o Generated locally on main board. 

o External source to the main board. 

o Generated in FPGA. 

• What clock rates are required? 

o Estimate the range of possible FPGA clock frequencies. 

o Plan to generate an arbitrary clock rate, with fine granularity. 

• What clock skew can be tolerated? 

o Inter-FPGA synchronization: make sure all FPGAs receive the clock sources at an acceptable skew. 

o Inter-board synchronization: in large systems, make sure all clocks arrive at all FPGAs with an acceptable skew.

In addressing the above considerations, an in-house board might typically include most or all of the following elements:

• On-board clock synthesis: typically a PLL driven by a crystal oscillator reference with configurable parameters to select the desired clock frequency. To increase the flexibility the crystal oscillator may be removable. Multiple clocks generators may be needed to support systems with multiple clocks. 

• Input clock source selector: given the multiple sources from which clocks can be sourced e.g., local (on board, FPGAs), or external, a clock source multiplexer should be implemented. Management of the multiplexer could be by manual switches or programmed by a separate software utility (see section 5.4 below). 

• Clock distribution: regardless of the clock source, clock distribution must ensure the clock is distributed to local and external consumers with an acceptable skew. Excessive skew may result in improper logic propagation and decrease the timing margins between two or more entities passing synchronous signals to each other. On-board and connector delays must be accounted for and equalized while maintaining acceptable signal quality. 

• External clock source: propagation delay from one board to another must be accounted for and propagation delays should be equalized by appropriate phase shifting. Special attention must be paid to a situation where multiple PLLs are cascaded between the source and destination of the clock path, as instability and loss of lock may occur if not properly designed. 

• Special high-speed clocks: in addition to the application clocks, there may be a need for high-speed clocks to multiplex multiple signals on one pin. This is typically used when the number of signals between two FPGAs is greater than the number available pins between them. Using a high-speed clock the signals can be time-multiplexed over a single pin at the source and are then de-multiplexed at the receiving end. For this scheme to work properly both sides of the interface must have the same high-speed clock with minimal skew. Some prototyping tools such as Synopsys’ CPM (Certify Pin Multiplier) assist the use of this technique. 

• Clock scaling: Flexibility in clocking requires that the top-level clocks can be scaled to the correct frequencies for various parts of the design. Although dividers and other logic resources can be used for this inside the FPGA devices, PLLs also have their role to play.