How to Design a Printed Circuit Board

I spend a lot of time in parts of the Internet where hobbyist electrical engineers gather. There are a few questions that come up over and over again in these spaces. One of the most common is also one of the hardest to answer:

How do I design a circuit board?

The reason it's hard to answer is because not everyone asking it does so for the same reason. Some people simply want to understand the mechanics, and can be helped by something like Hackaday's "Creating a PCB in Everything" series. Others want a beginner project to work on to start developing their skills, in which case they might check out Digikey's introductory series of videos. But for the purposes of this article, I'm going to focus on people who, when they ask this question, are really asking:

How do I solve a problem using electronics?

This is Part 1 of a blog series which will seek to answer that question through a tour of one of my own projects - Blue Oni, an open source Gigabit Ethernet Network Interface Card. I will talk you through not only every design decision that I make, but more importantly the process of making those decisions. Hopefully at the end, you'll have a good understanding of how to approach solving your own problems using electronics.

A Rough Outline

Here's a (rough, subject to change) outline of what I intend to cover with this series. As the posts are written, I’ll update this outline to contain the actual titles as well as direct links to the posts themselves.

  1. How to Design a PCB - This post

  2. Defining Requirements - Defining success for your PCB project

  3. Choosing a Technical Approach - Evaluating and selecting a technical strategy

  4. Key Component Selection - Choosing the core components of your project

  5. High-Level Design - Blocking out the project to guide you during schematic capture and layout

  6. Component Creation - How to avoid spending hours (or days) creating components in your EDA tool

  7. Schematic Capture - Finalizing the design and capturing it into a functional, readable schematic

  8. Layout - How to avoid stressing about layout (and those areas where some stress may be a good idea)

  9. Review - How to review your design yourself, and how to get other people to do it, too

Again, I reserve the right to adjust this outline as things go along. If there's something on here you're looking forward to, or something you'd really like to hear about that isn't present in this list, let me know! I'd love to hear about it.

Stop Reading Here

Honestly, you probably don't need to read the rest of this post. All that’s left is a description of my problem, and presumably you already know what your problem is. But if you're interested in how the Blue Oni project came to be, or just want a bit more context on it, feel free to read on. Otherwise, just skip to the conclusion - I promise I won't be offended.

Step Zero: The Problem

Before anything else, you need a problem to solve. Your problem will be your own, but here's the problem which led to the Blue Oni project.

"As long as it's unobservable, firmware is the enemy. We need open firmware. I don't know if we're ever going to get there." -Bryan Cantrill, CTO @ Joyent

I very much enjoyed Bryan Cantrill's talk at Uptime 2017, "Zebras All the Way Down". In it, Cantrill discusses the problems caused by firmware bugs at Joyent's datacenters. This talk inspired me to think about where we could attack the problem of closed-source firmware. But before that, let's back up a step and ask ourselves:

What is the real pain here?

In this case, Cantrill's actual problem is that software further up the stack is failing in bizarre and novel ways. However, this failure is the result of many more failures (of both technology and process) at many levels. In order to enumerate some of these additional failures, I find it helpful to use the popular Five "Why"s technique. Cantrill starts us off:

  • The software is failing in bizarre ways

    • Why?

  • Because it's encountering conditions that are supposed to be impossible.

    • Why?

  • Because the hardware is misbehaving and not honoring its contracts with the software

    • Why?

  • We don't know why

    • Why?

  • Because the problems are occurring in firmware and we can't observe the firmware

    • Why?

  • Because the firmware doesn't have good monitoring, it's closed source, and we can't replace it

However, I find that five is sometimes not enough "Why"s to get to a true root cause. Let's go a bit deeper:

  • Why?

  • Because the firmware runs on proprietary microcontrollers and ASICs

    • Why?

  • Because those are the parts designed into the various system components

    • Why?

  • Because only those parts provide the features, performance, and cost which make the components feasible

Now we have ourselves a nice deep tree of problems, and we can start discussing ways to solve them and whether a solution at that level will solve our original pain point.

I'll skip a lengthy deliberation here because I'm not actually Cantrill and I don't work for Joyent, or indeed for any large datacenter provider. I chose the bottom three problems to tackle because, frankly, they sounded like the most fun! Thus Blue Oni aims to implement a NIC which does not use proprietary ASICs, but which nonetheless delivers a useful set of features at an acceptable performance and cost.

Next Time: Defining Requirements

Thanks for reading! The next post in this series will explain how to derive technical and business requirements (in other words, define success) for your PCB project. After that, we'll get into the technical nitty-gritty and talk about choosing a broad technical approach to the problem. For the more business-minded, I'll also talk a little bit about how to determine whether a PCB is even a good solution to your problem, and if it is, whether it's worth your time to develop it yourself.

Exploring the Design Space of Open Source NICs

Ever since watching Bryan Cantrill's talk at Uptime 2017, I've been thinking about open source firmware. Of the three major types of firmware he discusses - storage, networking, and what I'd loosely call "system" firmware - I believe that networking is the most amenable to potential open-sourcing (although the OpenSSD Project may disagree). As such, I've been considering the possibilities for a while, and a recent Twitter thread prompted me to write down my thoughts on open source NICs.

The Approach

So what does "open source" mean in this context? Most network interface cards are designed based on proprietary ASICs. The ideal would be to develop our own, open-source ASICs, but the capital expenditure would be prohibitive. It's difficult even to interface with most networking ASICs in an open source way because the datasheets are not available without signing an NDA.

Fortunately, FPGAs occupy a very interesting niche. They allow full control over the digital design of the NIC, and many families come with high-speed serial links of the sort used for PCIe, 10G Ethernet, and other high speed networking technologies. I believe that this is the most open a network card can be in the present climate, so I'll take an FPGA based approach for the rest of this discussion.

Some Targets of Comparison

What features and price point are we targeting for our designs? For software features of course we can support anything eventually, but hardware features like numbers of ports and price points are still interesting. Cantrill was the inspiration for this project, and his company Joyent helpfully publishes a list of network interface cards used in their datacenters. Among them we have:

  • the HP361T, a 2-port Gigabit Ethernet NIC priced at $160
  • the HP560SFP+, a 2-port 10GbE NIC priced at $488
  • the Intel XL710, a 10G/40G controller whose 1-port 40GbE variant has an MSRP of $381

This gives us an idea of the ballpark we're playing in. We'll look at designs for a 1G, 10G, and 40G NIC with these in mind.

Design Proposals

Dual Gigabit Ethernet

Dual-port Gigabit NICs require only a 1x PCIe 1.1 link to fully saturate, so any FPGA with a transceiver capable of operating at 2.5 Gbps or higher should be sufficient. On the Ethernet side, the cheapest PHY I'm aware of is the KSZ9031 from Microchip, whose RGMII variant is priced at $2.98 in QTY 1 on Digikey. RGMII requires 14 I/O pins, so any FPGA with more than 24 I/O pins is sufficient.

The cheapest FPGA I found which meets our requirements is the Lattice ECP5 LFE5UM-25F-6BG381C, which has two 3.2Gbps transceivers and 197 I/O pins and sells for $12.78 in QTY 1 on Mouser. Alternatives include the Xilinx Artix-7 XC7A15T-1CPG236C, sold for $29.89 on Digikey, and the Altera Cyclone IV GX EP4CGX15BF14C8N, sold for $25.16 on Digikey.

Another option would be to use the TI XIO1100, a discrete PCIe 1.1 PHY, and an FPGA without transceivers. However, the XIO1100 sells for $11 on Digikey and therefore represents a more costly solution than the ECP5 above when paired with an FPGA costing more than $1.78.

Ultimately the ECP5 solution makes the most sense. Not only does it meet our requirements at the lowest price point, but it also has footprint-compatible upgrades should we find that we need more room for logic. The addition of the Lattice DDR3 Controller, Tri-Speed Ethernet MAC, and PCIe Endpoint cores takes up slightly less than 12,000 LUTs, leaving half of the device free for our logic. What's more, the EPC5 Versa board contains everything we'd need to prototype our solution for only $200.

It's worth talking about software licensing at this point. The ECP5 chips with transceivers require a license to the pro version of Lattice Diamond, although a one-year license comes with the Versa board. The Xilinx and Altera chips are supported on the free versions of their respective tools. For this analysis, I'm assuming that such things amortize out, but it's worth bearing in mind.

In conclusion, is this feasible? YES - with the Lattice ECP5 LFE5UM-25F-6BG381C.

Dual 10 Gigabit Ethernet

For 10GbE we have two options - a discrete PHY chip or driving the SFP+ directly from the FPGA's transceiver. XAUI allows us to use 4x 3.125 Gbps lanes to produce one 10 GbE lane, while RXAUI allows us to use 2x 6.25 Gbps lanes to produce one 10 GbE lane. Unfortunately, the only RXAUI chips I'm aware of, from Microsemi, require NDAs to access the datasheet, making them unusable for our purposes. This leaves us with XAUI or direct modulation. We also need to increase our PCIe bandwidth, to 12x PCIe 1.1, 8x PCIe 2.0, or 4x PCIe 3.0, in order to maintain line rate.

Lattice drops out of the running at this point - the ECP5-5G has enough speed for XAUI or PCIe 2.0, but the largest only has 4 transceivers. The Artix-7 series from Xilinx is an interesting option. The XC7A200T-2FFG1156C supports 16 6.25 Gbps transceivers, enough for 8x PCIe 2.0 and 8x XAUI giving dual 10GbE links. It is sold for $293.80 on Digikey. Altera provides a better solution though, with the Cyclone 10 GX line. The 10CX105YF780I6G, sold for $146.56 at Digikey, provides twelve 12.5 Gbps transceivers - enough to drive an 8x PCIe 3.0 link and four 10 GbE links. The hard PCIe block supports 4x PCIe 2.0, enough for a single 10 GbE link, and higher speeds can be supported in soft logic. Even better, designs for the Cyclone 10 GX can be made in Quartus Pro without paying for an additional license.

The Xilinx Kintex-7 series may seem appealing here, but read the datasheet carefully - the only packages which support >10 Gbps transceivers are either prohibitively expensive or only come with four transceivers.

In conclusion, is this feasible? Somewhat - with the Altera Cyclone 10 GX 10CX105YF780I6G. While it's technically feasible, and even within the reach of a hobbyist, the cost of $146.56 limits our achievable margin to about 1x, using the retail prices we discussed above. It's not clear whether we could make a sustainable product line at that price point, even after volume discounts on components.

40 Gigabit Ethernet

40 Gigabit Ethernet is implemented as a combination of four 10 Gigabit Ethernet lanes in a QSFP+ module. Thus, our PHY choices are the same as they are for 10 GbE. We must also increase our PCIe bandwidth again, to 16x PCIe 1.1,  12x PCIe 2.0, or 8x PCIe 3.0 for a single NIC and to 12x PCIe 3.0 for a dual NIC.

As mentioned above, the Altera Cyclone 10 GX 10CX105YF780I6G already meets the requirements for a single 40 GbE link! That was easy!

What about a dual 40 GbE NIC? Well, unfortunately this looks less rosy. There are some FPGAs available which would support such a design, such as the Kintex-7 XC7K325T-1FFG900C from Xilinx, but even the cheapest suitable part I was able to find, the Microsemi PolarFire  MPF500T-FCG784E, costs $551.785 on Digikey - in QTY 24. It seems that a single 40 GbE link is the farthest we can go with currently available technology.

In conclusion, is this feasible? Somewhat - with the Altera Cyclone 10 GX 10CX105YF780I6G. For a single 40 GbE NIC, we're in about the same position we were in with the dual 10 GbE NIC, economically speaking. A dual 40 GbE NIC looks to be infeasible.

Could We Go Further?

What about 25 GbE, or 100 GbE? These are essentially beyond our reach at the moment. 25 GbE is a single-lane solution, so it requires transceivers capable of operating faster than 25 Gbps. To reach these speeds in a Xilinx part we have to go to the highest end Virtex-7s or to a Kintex UltraScale+, the cheapest of which costs $1251.90. Altera is even worse - we need an Arria 10 GT, a Stratix V GT, or a Stratix 10, and the cheapest option is ludicrously expensive. 100 GbE is typically 4x 25 GbE, and while a 10x 10 GbE mode does exist, I'm not aware of a source for the PHY that would be needed for that scenario. If one could be found, the PolarFire FPGA above could almost saturate a 100 GbE link - the 12x PCIe 3.0 link would top out at 96 Gbps.

Ultimately, while the highest heights of high-speed Ethernet networking are unavailable to us, there is actually a fairly solid opportunity for open-source networking hardware at the 1G, 10G, and 40G levels. To further explore these possibilities, I've purchased an ECP5 Versa dev board and will be experimenting with it in the coming months - if that sounds interesting to you, watch this space.