This information is compiled from the archive at:
FreeBSD-amd64 Archive
By Willem Jan Withagen on
23 february 2004,
last update
27 september 2004.
I started this summary because I wanted to build a dual-opteron
system. So I asked the obvious question on the mailing list, and people told
me to read the archive. :-) Within 2 days I
saw several questions of the same nature. And I just started summarizing what
was there in this document.
Answers on the mailing list were given by the people in the list at the bottom.
And I took the liberty to submerse, compress, interpret, obfuscate and what
more. As such the information in this document could be inaccurate or outdated,
so check and double check.
Note: This is information is "as is", and the author(s) should not be hold responsible for any errors and/or consequences of possible errors. In the remainder of this FAQ "I" should not be read as the author, but as someone asking or answering a question.
| Athlon 64 CPU | Opteron CPU | dual Opteron CPU | |
| Tyan | K7X Pro, MPX | K8W, K8S Pro, K8SR, K8S, K8W | K8WS, K8S |
| Asus | KV8 deluxe, | SK8V, SK8N | none |
| MSI | K8T Neo-FIS2R | K8T
Master2-FAR K8D Master-F(T) |
|
| none | none | HDAMA, HDAMB | |
| Iwill | K7*, KK400* | none | DK8X |
| K8N*, K8V | none | none |
State per 24 february 2004
The onboard sATA on the S2885 isn't supported. The GX28 uses a Promise
controller chip that is more likely to work. I have no idea if the LSI
SCSI controller is supported. It's a BCM5704c ethernet: mine is a 5705.
Both the LSI SCSI and BCM5704c are supported.
I'm using a Tyan S2885, which appears somewhat similar to this. FreeBSD has run flawlessly for me - no reboots, crashes or other headaches. I'm using FreeBSD 5.2 RC2.
The onboard sATA on the S2885 isn't supported. The GX28 uses a Promise controller chip that is more likely to work. I have no idea if the LSI SCSI controller is supported. It's a BCM5704c ethernet: mine is a 5705.
I'm using 8 GB of memory (8x Crucial part #CT12872Y335) and a HighPoint 1542 sATA controller.
On the Tyan 2880 motherboard.
I know you've already bought it... but to everyone else I strongly recommend not buying this motherboard. The 4+2 DIMM configuration[*] is sub-optimal and you can get 4+4 DIMM configuration from Tyan (K8S Pro) and other makers.
[*] I fully know the bad history of Opteron 4+2 DIMM configurations... but can't speak publically about it. :-(
But many people have been highly successful with the Tyan boards (I have been running a K8S Pro-based server in production for more than two weeks now, and the mysqld process is approaching 100 hours of CPU time.)
So I am only suggesting them as an option ...
I'd love to get hold of a SK8V if only I could find one. :-) I have two K8V deluxes (one at home, the other at work, both with ECC non-Registered PC3200 memory from crucial.com). My SK8N with my (then) $750 cpu and $350 of ECC/REG PC3200 ram is on a shelf gathering dust because I lost my trust in the board.
Personally, I prefer the ASUS boards over the tyans because of the flexibility in the bios and the vastly superior active fan speed control system. But I have a slight preference for the AMD 8xxx chipset over the VIA K8T800, but its only slight. Both are (IMHO) way ahead of the nVidia nForce3-150. If you have to listen to the machine next to your desk, I suggest an asus - the tyan fans run at full speed all the time, with no thermal based fan throttling. If you don't have to listen to it, and want something slightly more server-oriented then the tyans are probably a slightly better bet because the AMD chipset has had longer to shake out the bugs.
I downloaded the 5.2-RC2 AMD64 miniinst ISO to try on a MSI K8T800 based dual opteron system today.
What is the board model number? I am not aware of any K8T800 based SMP boards. The only MSI 2P board I know if is the K8D Master MS-9131 which is AMD 8100.
If you used an Opteron 240 on an MSI K8T Master2-FAR board you would spend ~$600. That board is ATX-sized so it doesn't require any special case.
Your RAM is too slow, and I don't know if you can run the memory bus asynchronously, so if it were me I would figure on buying more RAM. I think you can get 1GB of Corsair Registered ECC DDR400 in a 'matched pair' of 512MB modules for less than $400, so that would put you under your spending limit.
On the K8T you can see the conductor paths of HyperTransport and V-Link With VIA's V-Map architecture makes bandwidths possible up to 1066 MB/s. The memory interface -- all four slots -- are directly connected to the first CPU socket. The second CPU accesses the memory through the HyperTransport of the first CPU. Theoretically eight slots (4 for each CPU like on the Tyan K8W) would be possible but through the restrictions of the ATX form there is no room left for the additional four slots. The VIA K8T800 chipset is passively cooled, the VIA 8237 chip doesn't need any cooling.
Which does not really sound like the 2nd processor is going to be
well connected to main memory???
And thus this sounds like an unbalanced architecture with respect to memory
access times....
Yes it is correct. The Master2-FAR has a "4+0" configuration
where one processor must conduct memory transactions through the other.
I don't know how much of a performance penalty this imposes. I'm sure
it depends on the application
I'm happy with mine, although I might buy a different board if I were
to purchase a new one today.
I really like both of the ASUS boards for personal use and the Rioworks for servers. I have not seen any of the MSI boards so I can't vouch for them. The MSI 2xCPU board only has 2 dimm slots on one of the cpus so I'm not that interested in their server board. Their consumer board appears to have similar specs to the asus K8V but I haven't tried it.
All of these boards work with ACPI, minus some quirks with ordering the serial ports. Some of them list the COM2 port first in the tables which means we assign 'sio0' to it. Thats only a minor annoyance though.
The onboard nVidia ethernet is annoying because they only give out a binary driver blob that you write an OS shim to use. This works for i386, but obviously isn't going to work too well for an amd64 kernel. The only board affected by this is the Asus SK8N, but ethernet cards are not a big deal.
I really really dislike the bios on the Gigabyte, and really like the ones on the ASUS boards. The Gigabyte bios interface is almost worthless and gives you very little control over anything. If it had to be named, I'd call it a 'BIOS for dummies'. On the other hand, the ASUS bios' give you more than enough rope. For example, Gigabyte have 'PNP OS: Yes' by default and dont give you the ability to turn it off. ASUS do let you turn it off.
Lots of people have that board; but it was sold to them as the "Arima HDAMA". It is a solid board. Being the first 4+4 DIMM configuration, a lot of vendors of 1U clusters use it.
Note the Hurricane64 (Model: ATO2163) is the same as the Iwill DK8X.
I really like the AGP slot location on this board.
> I should only consider AMD-8000 based boards?
For a workstation, you would probably prefer the VIA K8T800 chipset as you'll get things such as USB 2.0, and other niceities. The AMD 8100 chipset is server oriented and is a little more "plain".
nForce has issues - at least, that's what I'm led to believe by posts on this list.
nVidia nForce3 [for AMD64] chipsets are very problematic for Unix(BSD)/Linux.
I would avoid them if you want to run a non-MS-Windows operating system.
nForce3 systems have crappy BIOS's, stay away from them. (note, the crappiness
isn't any worse than many typical 32-bit x86 BIOS's) With the memory controller
being in the CPU, there is no benefit to the nVidia nForce chipset over
other chipsets. In fact, the Via K8T800 chipset is benchmarking the fastest.
> Note 5.2.1 also adds the Silicon Image 3114 SATA support needed to install on a SATA drive in many Opteron systems.
Is anyone else having trouble with this?
The 3114 is found no problem but the drive is not seen. The culprit may be this code in ata_reset().
My Tyan K8W S2885 works fine with its onboard Broadcom. Will's problems appear caused by ACPI problems, which is a property of the particular BIOS the motherboard interacting with whatever bugs exist in FreeBSD's ACPI interpreter. Your Tyan probably works as-is.
> The motherboard has the same Sil3114 SATA controller as the Tyan Thunder K8W (S2885). I have seen other posts to this list that indicate success in installing FreeBSD on the Thunder K8W, so I'm wondering why I can't see the drives on my system.
I made a simple modification to the driver which essentially maps the 3114 as a 3112, which works, sort of (not with the recent changes though, I'm getting those "TIMEOUT - ATAPI_IDENTIFY retrying messages" that others had mentioned as well). The drives do not go into SATA DMA mode (150Mbps), but I have the slight suspicion the code doesn't really support those yet with the SiL chipsets.
On very high load on the drives, I sometimes get ad4: WARNING - WRITE_DMA UDMA ICRC error (retrying request) style errors, but all in all, it's working very nicely.
At least with my board (an 'awful' 4+0 MSI K8T Master2-FAR board) the onboard broadcom NIC does not work correctly with bge. It does work with ndis though. Also for whatever reason -current as of last weekend would lock the machine up hard after ~ 30 minutes. Haven't had time to work on why.
I would prefer ddr non registered memory capability and sata drive support.
ALL Opterons require registered memory.
ECC memory is optional
> You might be able to save some money by using an Athlon64 CPU and motherboard instead of Opteron, but registered DIMM might not work.
Registered(buffered) DIMM's will not work in Athlon64 motherboards.
you want to populate memory in pairs per CPU. This is so you can use both channels of the DDR memory controller. If you only put one DIMM per CPU, you're missing 1/2 your potential bandwidth.
Well, my early Tyan Athlon MBs (Tiger S2460) didn't like Crucial DIMMs much, and my later Tyan boards (Tiger s2466's) won't work with the Crucial DIMMs at all (I have about 8 of them on a shelf that I can't use at all now). I never have seen any ECC errors (there registered ECC DIMMs) with these, just varying degrees of system unstability (random memory erors/panics). A couple of years ago I spent quite a bit of time on the phone with Crucial's tech support folks (shortly after I bought the memory) with no real resolution (the tech support guy tried to be helpful but couldn't really help).
Following this experience, I have come to worry much more than I previously did about memory compatibility issues; and I began to wonder just how much good ECC memory really does in these systems when you wind up with all kinds of system instability. I now worry more about compatibility than do ECC vs. non-ECC memory.
Note, that my newer Tyan dual MB systems (those with the S2466 MBs that can run with unbufferred/non-ECC DIMMs) have been running with unbufferred/non-ECC DIMMs for some time (over a year) and I have had much better stability with these systems than when I tried the Crucial reg/ECC DIMMs (generally, they either wouldn't boot at all or only ran for a short time, depending on how many DIMMs I installed--more meant shorter time to crash). Note that I never did try other brands of reg/ECC memory on these systems since they were stable with the cheaper stuff and I was growing tired of spending money on them.
At work we're pretty sloppy about what random no-name DIMM's we use. Only the Shuttle box has been picky. That said, Micron (MT) / Crucial DIMM's are probably the best quality. Samsung DIMM's also. These are the two brands AMD uses in the development systems it sends out.
I've got a case load of Kingston 1GB ValueRAM in which 1 out of 4 DIMM's is bad.
> Your RAM is too slow, and I don't know if you can run the memory bus asynchronously, so if it were me I would figure on buying more RAM.
Huh? I don't follow, why is running PC2100 in any AMD64 machine running the memory bus asynchronously??
PC1600 memory will work fine any any AMD64 mobo, so will PC2100 and PC2700. You'll need the latest BIOS's and a rev.C CPU to run DDR400.
On Sat, Feb 14, 2004 at 11:26:06PM +0100, Adriaan de Groot wrote:
Take a look at the archives of this list (via
Not just SK8N's. Just about every AMD64 mobo will "loose" physical memory from 3.5-4GB. So one would get 17.5GB in your example of 8x2GB DIMM's. The reasons have been explained in this list before.
It turns out that AMD has published its Opteron errata sheet and errate
item 101 appears to be the issue: a bug in the Opteron means you can't have
both "node interleave" and ECC scrubbing on at the same time.
I've turned off node interleave. I have no idea what are reasonable values
for ECC scrubbing in ROM setup but it seems more useful to leave scrubbing
on than to enable node interleave, even without a NUMA-aware kernel.
(this only applies to multiprocessor systems)
Oh my, thats bit of a stinker. Do you recall which steppings this applies to?
BTW; I suspect you might find that node interleave is more useful (speed wise) than ecc background scrubbing. But I guess that depends on what you want.. If you're trying to wring every bit of performance out of it, pick node interleave over scrubbing. On the other hand, if you'd perfer to have the system constantly checking that the ECC ram is ok and you're not so worried about speed, then pick scrubbing.
> Bug 101 affects both B3 and C0 steppings.
Well tested are Matrox G4[05]0 and nVidia GeForce{3,4}.
I have no ATIexperience on FreeBSD/amd64.
nVidia nForce3 [for AMD64] chipsets are very problematic for
Unix(BSD)/Linux.
I would avoid them if you want to run a non-MS-Windows operating system.
> Is anyone using an Adptec 29320? Mine is not working. I suspect is may be a BIOS problem on the Tyan K8W S2885. I also have a Highpoint 1820 that does not work right.
Not anymore. It worked for awhile and after a few days it just stopped. Next i was given a PERC 320/DC Adaptec FSA RAID (aac) This was much better but still an Adaptec. I'm using a 29160 now and it is going to go! These cards just don't seem to be up to a busy Unix environment. In all cases the firmware was the latest. (Surprisingly FreeDOS runs quite well on the amd64, and needed to run the flash.exe stuff.) I'm getting a real Cirrus Logic card. These cost a third what Adaptec costs and my experience has only been good with these cheap (smb) cards. I work with servers and most BOFH's swear by them and deplore Adaptec. (Therefore all those 'free' cards to try. ;-) Gladly stump up a hundred Euros or so for piece of mind.
Presently I'm using an Asus K8 motherboard for an Opteron-100 marketed as "Athlon-FX" or something. Not bad for Taiwan, but apparently Taiwan thought amd64 was going to be Windows and therefore some really goofy features.
All AMD products seem to work better slightly overclocked. 2250MHz for a 2200MHz chip is perfectly inline with the 3 - 5% rule. When increasing the bus speed, Adaptec cards give out first, then SATA and then the video and so on. Sorry, this is a bit offtopic.
I don't quite follow you. You are using the 2015S ZCR in 32-bit or 64-bit FreeBSD? It should work with FreeBSD/i386 on Opteron HW. asr(4) isn't 64-bit clean and thus asr(4) devices aren't supported in the FreeBSD/amd64 OS.
All right, so I gave up on my Adaptec 2100S RAID controller, and managed to locate a used HP NetRAID 3Si (aka MegaRAID 438). The NetRAID is only U2W (80 MB/s, as I recall?), but FreeBSD/amd64 seems to be much happier with it.
The only major chunk of hardware that I know of that has big issues is the Promise SX6000. The driver tries to use a 32 bit field in a hardware data structure to store a 64 bit pointer.
I've been asked a few times what case one can use in building a dual-proc Opteron workstation. So I thought I'd answer it publically for all.
AMD uses the InWin Q500 www.pcstats.com with the internal 3" drive cage removed. This case has tons of room and even with the 3" drive cage removed there are still 5 drive bays left. This case is US$69.32 from www.pricetool.com
If you want a shorter case, I have also installed both Tyan K8W (s2885) and MSI-9131 mobos into the Antec SX835II Performance Series II mid-Tower case www.antec-inc.com . Again with the internal 3" drive cage removed.
The "Antec" cases are also available under a variety of other names, they are actually manufactured by: www.chieftec.com