Nathan Stratton’s Homepage

Author Archive

Optimizing for Memory Intensive Workloads

by on Aug.20, 2013, under Hardware

Processor speed is an important factor when deciding on a new server spec, however with virtualization and other memory intensive workloads many times the memory system has a far greater impact on performance then even CPU speed. Xeon e5-2600 CPUs support 3 basic types of third generation dual data rate (DDR3) memory via 4 channels in up to 3 banks for each CPU. How or if those slots are filled and with what is very important to understand.

Types of Memory
The basic element of each type of DIMM is the DRAM chip that provides 4 or 8 bits of data. When ECC is used the DRAM chips provide 72 bits allowing many errors to be corrected without loss. Since ECC functions differently with 4 bit and 8 bit chips, different DIMMs types should never be mixed. The DRAMs on the DIMMs are arranged in groups called ranks, groups of chips that can be access simultaneously by the same chip select (CS) control signal.

UDIMM – With unregistered DIMMs each chip on the DIMM has its data and control lines directly tied to the integrated over the memory bus to the memory controller off the QPI ring integrated into each CPU. Each DRAM on this bus adds to the electrical load, because of this load UDIMM support is limited to only 2 dual-rank UDIMMs per channel. However this direct access to the DRAMs by the memory controller allows UDIMMs to provide the fastest and lowest latency memory access of all types.

RDIMM – Registered DIMMs are the most common in use in servers today. RDIMMs have an extra chip, a buffer that isolates the control lines between the memory controller and each DRAM chip. This buffer slightly increases latency, but allows a RDIMMs to support up to quad ranks and fill all 3 DIMM banks.

LRDIMM – Load Reduced DIMMs are a relatively new type of DDR3 memory that buffers all control and data lines from the DRAM chips. This isolation decreases the electrical load on the memory controller and allows the highest memory configuration possible. Since the DRAM chips are hidden by the buffer, LRDIMMs are able to implement rank multiplication offering the memory controller virtual ranks that may be less then the physical ranks on the DIMM. This hiding of physical ranks allows more rank support then the DDR3 memory architecture naively supports by the CPU. This increased capacity does come at the price of not only speed and latency, but also increased power consumption.

Memory speed
The clock frequency of the memory bus used to access DIMMs on the e5-2600 series is 1600, 1333, 1066, or 800 MHz (up to 1866 MHz on the new e5-2600 v2). This memory bus speed is controlled by the BIOS and is set per system. It is not possible to access memory in different banks at a different speeds. The maximum memory speed on the e5-2600 series is limited by the number of banks, ranks used, and the speed of the QPI ring. To support full 1600 MHz memory a 8.0 GT/s QPI is required, something that is not available on the standard or basic e5-2600 processors.

The e5-2600 supports up to 8 physical ranks per channel, DIMMs using single, dual, or quad ranks can be used, however quad rank DIMMs lower the clock frequency of the memory bus. The more ranks that are available to a channel the more parallelism can be preformed by the memory controller increasing memory performance, thus dual ranked DIMMs should be used if possible.

While the e5-2600 can physically support 3 banks of memory, only 2 banks can be used at 1600 MHz. If all 3 banks are used, the maximum clock frequency supported on the memory bus is 1066 MHz. Fully populating channels is not required on the e5-2600, it is highly recommended that all 4 are populated and if possible with two DIMMs in each channel increasing the ranks available on each channel.

Latency
Column Address Strobe, or CAS latency with DDR3 is the amount of clock cycles it takes between the moment the memory controller requests access to a DRAM and when that data is available on the DRAM chip on the DIMM. In searching for memory, particularly with lower cost UDIMMs, pay attention to the CAS latency and stay away from anything over a CAS of 9.

Maximum Performance
Bottom line… If you want the maximum memory performance use 16 sticks of 1600 MHz 1.5 volt dual rank UDIMMs with the lowest CAS latency as possible populating first two banks of all 4 channels. ECC DIMMs are also preferred by not required by the e5-2600.

Leave a Comment :, , , , , more...

Hello World, the current temp is:

by on May.02, 2010, under Hardware, Software

I have wanted to start playing with micro controllers for a while now, I ended up selecting the Parallax Propeller chip because of its ease of use and I liked it’s COG design with 8 32 big cores working together.

My first test was connecting a 4×20 line LCD and a few DS18S20 1-wire temp sensors to the propeller chip. Everything was very easy to learn the LCD was interfaced with no external components and the 1-wire bus only required a 4.7K pull up resistor.

Propeller PDB with LCD and 1-wire bus

Leave a Comment : more...

Central Air Pool Heat-Cool

by on Apr.24, 2010, under Projects

Worked today on my central air pool heat/cool system. The goal for the new system is to be able to operate in normal house cool mode, pool heat mode, and pool cool mode all in one HVAC system. The new system uses a reversing valve to pump down unused portions dramatically cutting down on the amount of refrigerant needed.

hvac-house_cool

Check out the project page for more info:

Leave a Comment :, more...

Infiniband

by on Jul.30, 2009, under Hardware

Infiniband is an often overlooked technology outside of the supercomputer / clustering space. I think that is a shame given some of the amazing aspects of this technology. Infiniband is a serial connection with a raw full duplex data rate of 2.5 Gbit/s known as 1x single data rate (SDR) mode. In addition to a double data rate (DDR) and a quad data rate (QDR) mode, links can be aggregated in units of 4 or 12 paths yielding up to 120 Gbit/s in 12X QDR mode. In a day where server motherboards are just starting to see 10 Gibt/s ethernet cards, the most common “low speed” infiniband options is 10 Gbit/s 4X SDR cards. Infiniband uses remote direct memory access (RDMA) for data transfer allowing data to be moved between hosts directly without any CPU cycles. All of this happens in about 1/4th the port to port speed of 10 Gbit/s ethernet!

The part I like best about Infiniband is the price, especially the used market. Lets take a look at a common setup on eBay. There are lots of switch options, but I like the TopSpin 120 also know as the Cisco 7000p. This is a 24 port 4X SDR 10 Gbit/s switch that runs for $750 – $1500 depending on the used source. There are even more options for Infiniband cards, I tend to stick with Mellanox chip set based cards and they can be found for as little as $40 for PCI-X and around $125 for PCI express. The only thing that is going to cost you more with Infiniband is the cables, they will run you $20 – $50 each.

Applications that support native infiniband RDMA are going to get the best performance, but with the Infiniband over IP (IPoIB) you can use standard TCP/IP! With IPoIB your infiniband card shows up as a normal interface and you can run DHCP or static IP on it.

Cisco 7000P

Cisco 7000P

Mellanox MHEL-CF128-T

Mellanox MHEL-CF128-T

GORE 4X Infiniband Cable

GORE 4X Infiniband Cable

1 Comment more...

APC Matrix 5000 Hack

by on Jul.19, 2009, under Hardware

I use a lot of power in my office, so much that the four 1500 VA UPS units I have only last me a few min. I needed something bigger so I went on eBay and found two APC Matrix 5000 UPS units for $450 each including shipping. There was only one downside, there were no batteries and new batteries would have cost me several thousand dollars.

The solution? I picked up 8 marine batteries at the auto parts store and wired them up (yes with fuses) into two 48 volt strings connected in parallel.

IMG_0324

Note: If you try this, you want to use Marine or better yet Deep-Cycle batteries rather then car starting batteries. Car batteries are designed to give very hight bursts of current and should only be discharged to about 5%. The very thin plates would destroyed over a few hundred discharges rather then the thousands you would get from deep cycle.

P.S. Yes, I built a cover for it!

3 Comments :, more...

BlinkMind This Week at InfoComm

by on Jun.16, 2009, under Uncategorized

BlinkMind, will be at InfoComm 09 in Orlando, Fl. If your in the area, stop by and check us out in both 3792 in the Video Conferencing Pavilion. Showing off our 16 active party video conferencing system, SIP Video Server, and IPTV system. We also are showing interoperability with all of systems at Grandstream GXV3006, Polycom VVX 1500, Creative InPerson, Tandberg E20, and Linphone soft client with the BlinkMind contributed H.264 code.

InfoComm Booth Under Construction

InfoComm Booth Under Construction

If you need help finding our booth, check out (click to expand):

Hall B Floor Plan

Hall B Floor Plan

Leave a Comment more...

Converting Citrix .xva to Xen.org .img

by on Jun.06, 2009, under Software

Xen is one of the coolest virtualization technologies out there. It comes in may flavors, the two largest being the bleeding edge xen.org open source project and the commercial (Citrix) version. There are things I love about the commercial version, but they lost me only supporting windows in their XenCenter administration interface.

The file formats of the commercial and open source Xenare totall different. The open source is a standard image file, you can mount it, fdisk it, whatever you would like. The Citrix Xen Virtual Appliance .XVA file is quite different. It is actually a tar file with ova.xml meta data and directories starting with Ref full of 1M files that make up the drive volumes of the virtual image.

To convert .xva to an xen .img file you first untar the image:

tar -xvf {image}.xva

Then grab this handy utility and run it on your untared data, as an example:

python xenmigrate.py –convert=Ref:3 {image}.img

This will paste all of those files back together, starting at 00000000. Note I have had problems running this script on Centos 5.x.

10 Comments : more...

Portable 3 Watt 3G Video Phone and HotSpot

by on Jun.02, 2009, under Hardware

I love my BlinkMind Video Phone service, but one problem has been being able to make calls when I am camping. I started by looking for a 3G access point that was already supported by OpenWRT, a Linux distribution for embedded devices. Since I run the Linksys WRT54G at home, the WRT54G3G was a logical choice.

Get Linux Running

Getting Linux Running on the WRT54G3G can be a pain since it’s PCMCI implementation does not work on the 2.6 kernel series. To make matters worse, Sierra Wireless only wrote and supports drivers for the 2.6 kernel. You can grab a copy of OpenWRT White Russian here for the BCM47xx chip set, next grab a hex editor (I used shed on Fedora) and change the 4 bytes to W3GA, once that is done you should be able to fire up the unit and upgrade the firmware with the edited image. If your lazy you can just download this.

Add a LCD

I selected a 4 line X 20 Character LCD display that could be used to show the IP address, upload / download speed, and signal strength. Modern Device made a nice little serial to LCD board that makes it VERY easy to connect a LCD to any service device.

lcd117

Modern Device LCD117 Board

RT204-1 20x4 LCD

RT204-1 20x4 LCD

Download PDF Instructions

The WRT45G3G does not have an external serial port, but internally it does have pads for a 3.3V serial. If you wanted to drive a computer serial port you would need a level converter such as MAX232. However, the Modern Device board is able to accept 3.3V without a problem. The only catch was finding the right pin. I broke out a logic probe and send some data out the port in pulses to eventually find the pin.

Receive and Transmist pads

Receive and Transmist pads

Power

I wanted the system to be able to run off batter for at least a few hours so I decided to go with two 6V 6.5AH batteries in series rather then a single 12 volt battery. The size allowed them to lay down on the bottom of my case. The LCD runs off 5 Volts, the easiest way to make this work with parts on hand was to use a +5 Volt regulator that fit nicely on one of the 4 mounting screwed for the router. Current draw is low enough that no heat sink is needed. I also added a 12 volt LCD voltage meter to the mix so I could tell when my batteries were running low.

Two 6 Volt 6.5 AH Batteries

Two 6 Volt 6.5 AH Batteries

3G Wireless

3G is via Sierra Wireless 881 PCMCI 3G card with AT&T service. I quickly realized that the default signal strength was not going to cut it and an amplifier would be needed. After some digging I selected Wilson Electronics 801101 3 watt cellular amp with in conjunction with a ARC Wireless Solutions ARC-FR0803R30 antenna. With the antenna I was able to buy a cable to connect to the Sierra card, but it required a 6 foot FME Female – FME Female cable that I replaced with a 1 inch FME Female coupler.

Putting Parts Together

imgp1549

Portable 3 Watt 3G Video Phone and HotSpot

Back of router board

Back of router board

Closeup of status display and voltate meter

Closeup of status display and voltate meter

Software

I wanted the display to show some useful, current script runs at startup and displays IP address, upload and download avg bandwdith in kb/s, and Signal strength in dBm.

Download Script

imgp1546-2

Leave a Comment :, more...


Home Automation – Pool Fill

by on May.25, 2009, under Projects

I now can fill my pool without wasting water!

http://www.robotics.net/projects/home-automation/pool-fill

-Nathan

Leave a Comment : more...

Looking for something?

Use the form below to search the site:

Cool Links!

A few highly recommended links...