I am working on a deep learning server that will have 8 Nvidia 4080 GPUs. The server will draw quite a bit of power and I want redundancy because our power will be 415/240 VAC three-phase system it made sense to use three 2000 W paralleling power supplies. This will give me about 4000 watts / 330 Amps of 12V DC and with 3 the server will still be up and happy with 1 dead supply. In the past, I have had good success with Supermicro so I started looking at PWS-2K04A-1R supplies. This is a nice auto-ranging supply that delivers 2000W of 12V DC power on a 240 VAC input.
The supply is low cost and has a 96% efficiency and supports PMBus 1.2 for remote management. I struggled to find the 54-pin card edge connector. After looking at an old server I found that Supermicro was using an ALL TOP connector, but I had trouble sourcing them. After digging around a bit on google I found that Compuware makes most if not all Supermicro power supplies and that this supply was Supermicro branded Compuware CPR-2021-3M6. With that information, I was able to get the datasheet and discovered the recommended connector, Amphenol GPCP442628110HR. The cool part was that Mouser had them in stock!
|5||I Share||32||RS GND|
|7||A2||34||12V RS GND|
|8||I2C SCL||35||Wakeup Bus|
|9||I2C SDA||36||DC Return|
|10||PS On/Off||37||5Vsb CO|
|11||SMB Alert||38||5Vsb CO|
|12||DC Good||39||DC Return|
I was much further along on my quest, the operation of these supplies is fairly straightforward. To turn on the supply you bring pin 12 PS On/Off to ground. Making sure three supplies can talk to each other and share the load is also straightforward. All you need to do is connect together the three supply pins.
pin 1 – 4 —- +12V
pin 5 —- I share
pin 8 —- I2C SCL
pin 9 —- I2C SDA
pin 11 —- SMB Alert
pin 13 – 27 —- +12V
pin 28 – 31 —- DC Return
pin 35 —- Wake up Bus
pin 39 – 54 —- DC Return
I wanted to use the I2C PMBus, but you could leave that off if you don’t care about PMBus.
With my newfound information, I started designing a PCB for this. I used an 8-pin ribbon cable to connect the 3 boards together, an SMBus connector to connect back to my Supermicro motherboard and a red LED tied to ping 12 DC Good so I could see if the supply was happy.
Since I wanted to use PMBus I had to give each supply a unique address, I could not find any information on how to set this so I connected the SMBus to a Raspberry Pi I2C port and ran i2cdetect as I changed the pins.
The pins were pulled high by default, so I just added 2 jumpers to pull them to ground when I applied a jumper.
I designed the PCB in KiCad and sent them off to JLCPCB to get fabricated. I had 10 boards made for $22 including the $17 upcharge for 2oz copper. Now before I get all sorts of flame mail, 2000 Watts of power is more than 160 Amps! I know that my little 2oz copper board is NOT going to handle that much current, but it was a low-cost way to test. Once the design is finalized I can have them made with 10oz copper.
I decided to use ½ copper threaded rod and nuts to carry current between the 3 boards and dual 2/0 welding cable to carry the load to my bus bars.
I connected the SMBus to a Supermicro H12SSL motherboard and went into IPMI and I was able to see stats on all 3 power supplies. I then connected my load bank to the unit and was able to pull 500 amps at 12V (the max of my load bank) with no problem.
Overall I think it worked out ok, one task down, and many more to go.