Multiplexing schemes

The partitioning tools insert multiplexing based on built-in proprietary multiplexing  IP elements. We normally do not know, or probably need to know, every detail of  these elements but we shall explore some typical implementations in the rest of this  section. At the end of the section a comparison of the different schemes is shown.

Schemes based on multiplexer

The simplest scheme is a mux in the sending FPGA and a dmux on the receiving  FPGA, much as we saw in Figure 119 but with the source FFs omitted. In the source  FPGA there is a 100MHz transfer clock which drives a small two-bit counter which  cycles through the select values for the mux. Every 10ns a fresh data value starts to  traverse the mux, over the board trace and into the dmux in time to be clocked into  the correct destination FF. Meanwhile, the dmux is switched in sync with the mux  as both are driven by a common clock, or more likely two locally generated clocks  which are synchronized as mentioned in section 8.5.3 above. 

On each riding edge of the transfer clock, new data ripples through to the mux  output and propagates across the trace from the source FPGA to the destination  FPGA in time for the receiving register to capture it. This is an extra FF driven by  the transfer clock rather than the design FF that would have captured the signal if no  multiplexing scheme had been in place. 

To use this simple scheme, we need to select candidate signals that will propagate  across the mux and dmux in order to meet the set-up timing requirement of the  receiving FF. If the timing of the direct inter-FPGA (i.e., non-multiplexed) path was  already difficult for the receiving FF to meet, then this is not a good candidate for  multiplexing. In normal usage, we would select signals with a good positive slack  and these can be estimated after a trial non-partitioned synthesis.  

Best candidate IOs for this simplest kind of multiplexing scheme are those directly  driven by a FF, which normally could map into IO FFs in the FPGA if there were enough pins available. These would have the maximum proportion of the clock  period available to propagate to the destination, assuming that the transfer clock is  in synchronization with the design clock driving those FFs. Using IO FFs, the  timing of inter-FPGA connections is more predictable and generally faster.  Therefore a multiplexing scheme should use IO FFs if possible and we should use  synthesis attributes to ensure that a boundary FF is mapped into an IO FF if  physically possible.  

Another multiplexing scheme, which is very similar to the one described above, has  additional sampling FFs in the source FPGA driven by the transfer clock exactly as  seen in Figure 119. Now the whole mux-dmux arrangement is in sync with the  transfer clock and it is easier to guarantee timing. In fact we need not even have a  transfer clock that is synchronous with the design clock but double-clocking  synchronizers may be necessary to avoid metastability issues.

Note: qualification criteria for multiplexing nets

There are different types of signal in the SoC design, on different kinds of FPGA  interconnections. Some types are suitable for multiplexing and other types should  not be multiplexed. Table 19 summarizes these different types and their suitability  for multiplexing.  

To get the highest performance in case of multiplexing the user should carefully  select the FPGA interconnection which should be multiplexed. For designs with  different design clocks the user should multiplex signals coming from a low-speed  clock with a higher ratio than signals from a high-speed clock. This keeps the  performance of the design high.

Schemes based on shift-registers

Another solution is based on shift-registers, as seen in Figure 120. Here the data  from the design is loaded in parallel into the shift-register on the rising edge of the  transfer clock and shifted out with the same clock.  

In the receiving FPGA, a shift-register samples the incoming data on the transfer  clock and provides the data in parallel to the design. The first sample (in this case  sig 4) is available at the shift register output from the sample clock edge but an extra  edge of the transfer clock may be necessary in some versions of this scheme in  order to latch in the data after it has been fully shifted into the destination registers  for finally clocking into the design FFs in the destination FPGA. Once again, the  sending and receiving shifters need to start up and then remain in sync.

This type of scheme is well suited for boards with longer than average flight time on  the inter-FPGA traces because there is no extra combinatorial delay in the path and  we obtain maximum use of the transfer clock period. In particular, it is not  acceptable to have new data being sampled onto the trace if the previous sample has  not yet been clocked into the receiving logic. In some lab situations we can get  lucky, but in others, the transmission line characteristic of the trace or a slight  discontinuity in a connection can make the transfer unreliable. Therefore we have an upper physical limit on transfer clock speed and if we hit that then the only way  to increase multiplexing ratio further is by reducing the overall system speed.  Having done so, then even when using mux ratios of 10:1 or higher, we just have to  operate the prototype at a lower clock rate.

Worked example of multiplexing

It is important to understand mux timing so here is a worked example of a  multiplexing solution that uses a mux followed by a sampling FF. 

Considering the Figure 121, in FPGA A we have a design with some flip-flops which we call FFs followed in this example by some combinatorial design logic (in  some other examples it might also be possible that there are only design flip-flops).  The values are fed to the send stage which contains a multiplexer which selects each  design signal in turn and an output FF, which we shall call FFMO. FFMO could be  placed into an IO FF of FPGA A in order to improve the output timing as mentioned  above.

Between the two FPGAs we use a single-ended connection for the multiplexed  signals (shown as mux on our diagram). As mentioned above, to guarantee signal  integrity we must ensure that a multiplexed sample is received and latched into the  destination FF within one transfer clock cycle between the sending FFMO and the  receiving FF, which we shall call FFMI.  

As the multiplexed samples are clocked one by one into FFMI in FPGA B, they are  stored in a bank of capture FFs, which we call FFMC. These capture FFs ensure that  the samples are all stable before the next clock edge of the design clock. Our  example also shows some combinatorial design logic in the receive side (but again,  this might only be design FFs). The aim is to show that TDM can be used on a wide  variety of candidate signals.

Now let’s see how we can calculate the maximum transfer frequency and the ratio  between transfer clock and design clock.  

Our constraint is to transfer a data value within on period of the transfer clock cycle  between FFMO and FFMI. The delays on the path are as follows: 

• The delay through the output buffer of FPGA A (𝑇𝑜𝑢𝑡)  

• The delay on board (𝑇𝑏𝑜𝑎𝑟𝑑)  

• The delay of the input buffer at FPGA B (𝑇𝑖𝑛)  

The maximum delay on the multiplexing connection is therefore: 

𝑇𝑀𝑈𝑋𝑚𝑎𝑥 = 𝑇𝑜𝑢𝑡 + 𝑇𝑏𝑜𝑎𝑟𝑑 + 𝑇𝑖𝑛 

If we assume typical values of 𝑇𝑜𝑢𝑡 = 5𝑛𝑠, 𝑇𝑏𝑜𝑎𝑟𝑑 = 2𝑛𝑠 and 𝑇𝑖𝑛 = 1 𝑛𝑠 we get a  maximum delay of: 

𝑇𝑀𝑈𝑋𝑚𝑎𝑥 = 5𝑛𝑠 + 2𝑛𝑠 + 1𝑛𝑠 = 8𝑛𝑠 

This sets the upper limit on transfer clock frequency of: 

𝐹𝑀𝑈𝑋𝑚𝑎𝑥 = 1/ 𝑇𝑀𝑈𝑋𝑚𝑎𝑥 =  1 /8 𝑛𝑠 = 125𝑀𝐻

That’s the theoretical rate that signals can pass between the FPGAs, but we should  also respect that we are using single-ended signaling and that there can be some  clock uncertainty, or jitter, between the FPGAs and we should also give some room  for tolerances. Therefore, based on our experiences, we should add 𝑇𝑡𝑜𝑙𝑒𝑟𝑎𝑛𝑐𝑒𝑠 = 1 − 2𝑛𝑠 to provide some safety margin, depending how confident we are in the  quality of the clock distribution on our boards. For our example let’s assume that  𝑇𝑡𝑜𝑙𝑒𝑟𝑎𝑛𝑐𝑒𝑠 = 2𝑛𝑠. This results in the following calculation:

𝑇𝑀𝑈𝑋𝑚𝑎𝑥 = 𝑇𝑜𝑢𝑡 + 𝑇𝑏𝑜𝑎𝑟𝑑 + 𝑇𝑖𝑛 + 𝑇𝑡𝑜𝑙𝑒𝑟𝑎𝑛𝑐𝑒𝑠 

If we assume the same values as above for the other delays we get: 

𝑇𝑀𝑈𝑋𝑚𝑎𝑥 = 5𝑛𝑠 + 2𝑛𝑠 + 1𝑛𝑠 + 2𝑛𝑠 = 10𝑛𝑠 

and a maximum transfer clock frequency of: 

𝐹𝑀𝑈𝑋𝑚𝑎𝑥 = 1/ 𝑇𝑀𝑈𝑋𝑚𝑎𝑥 =  1 /10 𝑛𝑠 = 100𝑀𝐻

The maximum clock frequency of 100MHz (or period of 10ns) must be given as a  constraint on the transfer clock during FPGA synthesis and place & route.  

Let’s now consider a little more closely how the mux and dmux components are  working in order to calculate the ratio between transfer clock and design clock. We  have to consider two possible use cases. The first case is that that the transfer clock  and the design clock are mutually synchronous i.e., they are derived from one clock  source and they are phase aligned. The second case is that the transfer clock and the  design clock are asynchronous, in which case, we don't know on which transfer  clock cycle the transfer of the data values starts and we have to set the right  constraints to make sure that it works. 

Starting with the synchronous case, we see from the block diagram that there are  some transfer clock cycles required to bring the data from the sending design FFS through the multiplexing registers FFMO, FFMI, FFMC to the receiving design register  FF_R.  

In addition, even though the two clocks are synchronous, we have to respect the  delays between the design clock and the transfer clock on the sending and receiving  side. These delays are marked in the block diagram with Tdesign−to−mux for the  sending side and Tdmux−to−design for the receiving side. For the following  calculation of the clock ratios we assume that these delays have a constant value and  we have to give these assumptions as constraints to synthesis and place & route. For  our example here we shall assume that these delays are a maximum of one transfer clock cycle, which is extreme, and a maximum of 10ns. 

Table 20 shows how the design signals are transferred through the multiplexing  based on our assumptions above.  Consider new data DA, which is valid in the design registers FFS. One transfer clock  cycle later, the first bit, DA1, is captured into FFMO. This is using our assumption  that the delay between the design and transfer clocks is a maximum one transfer  clock cycle and that the clocks are phase synchronous.

The shaded entries in the table show how the captured data bit is transported  through the mux and dmux and is clocked into the receiving FFR. We have  highlighted in capitals where each FF in the chain has new data.  

As we can see from the first column of our tab, the ratio between the design clock  and transfer clock is seven, which means that the design clock has to be seven times  slower than the transfer clock to guarantee correct operation.

Now it is trivial to calculate the maximum design clock frequency for our synchronous multiplexing example:  

𝐹𝐷𝐸𝑆𝐼𝐺𝑁𝑚𝑎𝑥𝑆𝑌𝑁𝐶 = 𝐹𝑀𝑈𝑋𝑚𝑎𝑥 / 𝑅𝐴𝑇𝐼𝑂 =  100𝑀𝐻𝑧 /7 = 14.28𝑀𝐻𝑧 

So for this design, which multiplexes signals using a 4:1 mux ratio at 100MHz, we  can run our design at over 14.28 MHz worst case, not the 25MHz that we might  have guessed from the 4:1 ratio. 

We have now seen the case where the transfer clock and the design clock are  synchronous but let us consider the difference in an asynchronous multiplexing scheme where the design clock and the transfer clock are not phase aligned. The  maximum transfer clock frequency is the same but we don’t know the skew  between the active edges of the design and transfer clock. Therefore we have to add  additional synchronization time on the send and receive sides to guarantee that we  meet set-up and hold time between design and transfer clocks. This adds an  additional transfer clock cycle on both send and receive sides. The calculation of the  maximum design frequency for the asynchronous multiplexing is:  𝐹𝐷𝐸𝑆𝐼𝐺𝑁𝑚𝑎𝑥𝐴𝑆𝑌𝑁𝐶 = 𝐹𝑀𝑈𝑋𝑚𝑎𝑥 /𝑅𝐴𝑇𝐼𝑂 + 2 =  100𝑀𝐻𝑧 /7 + 2 = 11.11𝑀𝐻𝑧 

We can see that the asynchronous case runs with a lower design frequency but the  advantage of asynchronous multiplexing is that we don’t need to synchronize the  design and transfer clock and we have greater freedom from where to source the  transfer clock.

To summarize the example, the important things we should keep in mind to constrain a design with multiplexing are: 

• Calculate the correct maximum frequency of the transfer clock based on  FPGA IO technologies, delays on board and tolerances. 

• Calculate the correct ratio between design and transfer clock, respecting  the difference between synchronous or asynchronous multiplexing. 

• Give correct constraints in synthesis and FPGA place & route for the  transfer clock and the design clock. 

• Give correct clock-to-clock constraints in synthesis and FPGA place &  route for the delay between transfer clock and design clock. 

Having considered different kinds of multiplexing using normal single-ended  signaling between the FPGAs, what can we do to have higher mux ratios but still  maintain a high system speed? The answer lies in raising the maximum transfer clock speed, using the FPGA’s built-in serial IP and a more robust signaling  technology.

Scheme based on LVDS and IOSERDES

We can further improve the transfer of data by sending the clock on a parallel path  to the data. This method of sending clock and data together is called a sourcesynchronous interface. This makes it easier to meet timing because the clock has the  same off-chip/on-chip skew as the data, especially if a well-designed board is used  upon which there are matched delay traces with the same flight time.  

Having the clock travel from the source, rather than be generated locally and kept in  sync, would not work for driving partitioned logic in general but it is very useful for  unidirectional serial data transfer.  

Trace flight time works in our favor but it still places a physical limit on the  maximum system speed and the way to overcome this is to use differential signaling  between the FPGAs.

In this advanced case, we use the FPGA’s built-in support for LVDS, which can  increase transfer rates up to 1GHz. This allows much higher mux ratios without  having to reduce the overall prototype clock speed. Certify’s HSTDM scheme,  briefly mentioned earlier, supports LVDS signaling but also uses another of the  FPGA’s built-in resources, the IOSERDES blocks. 

As introduced in chapter 3, modern FPGAs have dedicated serial-to-parallel  converters called IOSERDES which have specific clocking and logic features for  implementing high-speed serial transfers. Using IOSERDES avoids the timing and  layout complexities of implementing the similar functionality within the FPGA  fabric.  

We can also double our transmitted data rate by using both edges of the transfer  clock using the FPGA’s built-in support for double data rate (DDR) operation of the  IOSERDES blocks. Figure 122 gives an overview of the implementation of a highspeed TDM scheme which combines all of these advanced built-in features of a  Virtex-6 FPGA. 

LVDS guarantees the highest transfer speed on-board but the slight disadvantage is  that there are two pins required for each serialized data stream. So, while a singleended multiplexing scheme with a ratio of 8:1 needs only one inter-FPGA trace to  transfer eight data signals, a differential IO standard needs two, so we call this a  mux ratio of 8:2. Therefore the differential solution reduces the interconnections  only by factor four and not by factor eight. However, when the significantly higher  speed of LVDS is taken into account then multiplexing ratios of 128:2 can be  considered, which gives a far greater data transfer bandwidth than is possible with a  single-ended scheme.

As shown in the diagram, this IOSERDES is being run at a transfer rate of 400MHz and needs two clocks for operation. The clocks are generated from a PLL within the  source FPGA based on a 100MHz clock arriving from an external source. The  OSERDES is configured as an 8:1 ratio but because it operates in DDR mode, we  need only have a transfer clock of 400MHz rather than 800MHz. This means that  the 400MHz clock transfers 8 bits during one 100MHz period. To provide  multiplexing ratios higher than eight we can use an additional mux at the input of  the OSERDES but this only needs to work at the slower internal clock rate. 

The data and the transfer clock are both passed to the destination FPGA via LVDS  signaling but at the destination FPGA only the data goes through an IODELAY  element to adjust its timing relative to the clock. The data and the transfer clock  then drive the ISERDES block. Received clock is passed through BUFR block to  divide it by four to create the local version of the 100MHz, which is also required  for the ISERDES and the dmux control. 

The diagram gives only an overview and implementations may be more complex  and include so-called link training of all HSTDM multiplexing channels in order to  guarantee optimal alignment between clock and data. It is important that user data is  not transferred over the link while training occurs and it would be lost in any case,  so only after the link training is complete is a ready signal generated. This should be included as another input to the reset condition tree discussed in section 8.5 above.  If such a training is not implemented the user has to manually adjust the IDELAY  components to achieve highest speed and to guarantee the right operation.

Which multiplexing scheme is best for our design?

Given the number of options, it may be difficult to decide which multiplexing  scheme is the best to use in our prototyping project. To help us, Table 21 shows a  comparison of the multiplexing schemes described in the above sections.

The comparisons are shaded with the least preferable under each criterion shaded  darker. As we can see in the comparison, the shift register solution is a flexible, easy  to handle solution and can be used on all prototyping platforms. However, let’s not lose sight of our goal of enabling the highest transfer speed so that the prototype  speed is not compromised too much. Therefore the IOSERDES solution may be  offer a greater reward, but the effort may be higher and the board/system needs to  support LVDS signaling across matched-delay traces.