20020307: Support for 6.9/7.4 firmware, big I/Os

Support for the 6.9 (6xxx) and 7.4 (7xxx) firmware upates has been committed to both CURRENT and STABLE. The code is the same for both versions, and should build cleanly on 4.5-RELEASE as well. Some extra work may be required for systems prior to the introduction of d_thread_t; a tweak to twe_compat.h will be forthcoming at some point.

Note that there appears to have been a problem with the 7.3 and possibly early 7.4 firmware (7xxx cards shipping in the first quarter of 2002) where large I/Os (128kb or larger) fail or proceed extremely slowly. This is reported by testers to have been resolved in the 7.4 firmware due to be released this week on 3ware's site. If you are seeing problems that might be related to this, you're urged to update your firmware.

       20020218: Experimental patch available for 6.9 firmware

There are experimental patches for the twe driver for both STABLE and CURRENT to address issues with the 6.9 firmware release. These patches have been lightly tested on a 6410 running a RAID5 arry; there are no known issues with them at this time.

       20020210: Back in touch

I'm back in touch with 3ware's engineering folk again. They're interested in solving some customer issues with the 6.9 firmware (see the next note) and also working out anything needed for "large drive" support.

A snippet for those of you with 7xxx series controllers that aren't sure about the recall; check the '700' label on the back of your boards. The label should end in "05 A" or higher; if not, check with 3ware about an RMA.

       20020209: 6xxx series, 6.9 firmware issues

The twe driver in FreeBSD-4.5-RELEASE has reported issues with the 6.9 firmware release. Stay at 6.8 with this release; an updated driver will be available shortly.

       20011109: 3ware has a change of heart

In a press release describing more consolidation at 3ware, the company also announced that strong customer demand has resulted in a decision to start manufacturing the Escalade controllers again.

The power supply problems affect the 7850, 7450 and 7810 adapters. You should contact your supplier to arrange a recall. There is a firmware update for the 7410 adapters available from 3ware's site which should be applied to older 7410 adapters.
Thanks to ravi pina for this information.

I have not yet established contact with 3ware subsequent to this decision, but I hope to find a technical contact there again.

       20011003: 3ware abandons RAID controllers

3ware have abandoned their Escalade family of RAID controllers, and recalled the 7000 series products entirely. This effectively ends any support I might expect from 3ware, and the 7000 series controllers are now known to have power supply problems, so they should be avoided in high-load situations.

       20010804: 3dmd updated

3ware have provided an updated 3dmd package; this version features some updates from other Escalade products and slightly better error handling.

The 7000 family of controllers is also generally available now. These controllers are supported by the driver already, no update is required.

       20010213: 3dmd release candidate available

3ware's latest 3dmd binary is available from the RAID page. Note that a number of users have been having trouble with this because they don't have the /dev/twe0 device node. You can create this as follows:

    cd /dev
    ./MAKEDEV twe0

       20001028: improved support, firmware fixed

The problem plaguing users loading up controllers has been identified as a firmware off-by-one bug, and resolved by 3ware. If you're having trouble with the 'MCERR' diagnostic, upgrade your firmware!

The driver has also been significantly upgraded for the 4.2 release. CPU utilisation has been reduced, and an ioctl interface for management tools has been implemented. 3ware plan to release their '3dm' management and monitoring utility shortly.

       20000720: 4.1-RELEASE supports 3ware

With some last-minue help from the release engineering team, support for the 3ware Escalade 5x00-series controllers is present in the FreeBSD 4.1 release. Installation to, and booting from these controllers is fully supported.

In other news, 3ware have just announced a new controller. I've been informed by my helpful contacts there that they plan to ship me a sample and reference source sometime next week, so keep your eyes open for updates here soon.

       20000524: Committed to 4.x/5.x

The driver is now in the FreeBSD 4.x and 5.x codebases, and maintenance will continue there. Some feedback from 3ware indicates that we're doing pretty well in the performance stakes.

There has been some discussion regarding management utilities, but as yet nothing has been forthcoming, and it looks like it may require a firmware upgrade to get the sort of functionality that people are expecting. In the meantime, I think I'll have to pull the confusing RAID level messages from the driver.

At this point, the driver appears to be entirely stable, and I have a number of positive reports of use in production environments.

       20000515: running on stable and Alpha too

Well, the backport to -stable was easier than expected; it's up and has received some light testing. Again, feedback gratefully accepted. Likewise, building and running on the Alpha was a snap. Note that, at this point in time, you have to install the controller and drives in an x86 system in order to configure or repair the array, so this is very much minimal support.

It's worth noting that the backport was achieved with the help of a small heap of macros which hide the buf -> bio changes and a couple of inline version tests. This means that the code doesn't directly read as an example of how to do it 'right' under 4-stable, and also means that if -current diverges a lot more it may be necessary to branch the two properly.

The code also compiles on the Alpha, but I have no way of testing there yet, due to lack of a configuration tool (and it's a little hard to move my disk farm around, though I will have to do that if I'm going to do this at all). Some pictures of the setup:

  • Drives in their 'hotswap' trays.
  • The controller up close. One of the AccelerATA chips has been de-labelled to discover it's really a Xilinx FPGA.
  • A shot of the development system.

       20000514: running cleanly

Somewhat later than expected, the second beta is out for general consumption. At this point, the driver appears to be entirely stable; there are no expected failure modes (although I reserve the right to be unpleasantly surprised).

The big challenge now appears to be coaxing the folks at 3ware to give me some more data; there are a few things that the driver really needs to know that I simply can't implement for lack of information. Specifically, the driver can't tell what the organisation of an array is, nor what its current state is (eg. degraded RAID1). I also don't know how to instigate a rebuild operation, nor how to create new arrays, both of which would be useful to add to a userland configuration utility. In particular, we need this information if the controller is to be useful on the Alpha (since you can't otherwise configure it...).

There's some suggestion that 3ware's next generation management/monitoring interface might well be ported to FreeBSD; more details as they're established.

Items on the to-do list now include:

  • verify operation on the Alpha
  • backport to 4-stable
  • add more management intelligence

       20000510: pushing bytes

After a couple of false starts, we're up and moving data. Basic functionality is there (detect array, read, write, etc.). The controller has a few quirks, and it's quite unforgiving; its typical response to you telling it something wrong is for it to simply lock up. Not much fun.

Another minor nuisance is that you can only do I/O to buffers that are 512-byte aligned. This doesn't hurt the Linux folks, but the BSD physio interface can easily fall foul of this. We work around the problem by allocating an alignment buffer and copying back and forth, but creating filesystems, and some disk benchmark programs like rawio may give misleading performance results.

On a more cheerful note, it turns out that we don't need anything like as much wired memory as was expected. Instead, command and parameter buffers are mapped in and out as required, with the usual behaviour of retaining old command buffers for reuse. Performance looks pretty good; a 4-disk RAID0 stripe using 8GB Fujitsu drives yields ~40MB/sec filesystem throughput; results with better drives (eg. IBM DPTA) will have to wait until someone lends me a set.

       20000421: the hardware's here!

After quite a bit of chasing around (I first talked to 3ware in February), I'm in touch with the right people and there's a 4-channel controller here. We have the disks for it, so it's time to start work on the driver.

Unfortunately, nothing but the Linux source for documentation. It's easy enough to read, but not very clear and no comments to speak of. It looks like the driver architecture may have to digress quite a bit from the Mylex/AMI drivers - probably more so even than the ICP driver.

On the plus side; there's very little I/O direct to the controller; everything is handled via shared regions in host memory.
On the minus side; with support for 256 commands and 256 general-purpose buffers, each of 512 bytes, we're going to need a _lot_ of controller-visible memory permanently wired in. The most efficient way to do this is going to be to peg down a single 512k segment, but this will make loading the module pretty hard once physical memory is fragmented...