Fun with parallax

[Joe Durnavich]

Craig, that's brutal!

Jay, just out of curiosity, looking at all the water-cooling contraptions showing up on consumer PCs these days, I wonder, are the supercomputers these days water cooled at all? Or is air cooling the most efficient?

[craiglamson]

joedurnavich said:

Craig, that's brutal!

Jay, just out of curiosity, looking at all the water-cooling contraptions showing up on consumer PCs these days, I wonder, are the supercomputers these days water cooled at all? Or is air cooling the most efficient?

Yea, but I'm too cheap to air condition a 6000 sq ft concrete block building with 20 ft ceilings. I can stand to lose the weight.

It used to be worse. When I was still usign film you had to stand inside the unit while you ran the film brackets. Now with digital I just sit at the computer workstation with a nice fan blowing on me and control the camera and the shutter with my mouse.

[JayUtah]

Liquid cooling is more efficient in terms of heat transfer, of course. Cray still uses liquid cooling techniques, both circulating coolant and sprays to generate evaporative cooling. This works fine for them because they lean on the integrated side of supercomputer design. Unfortunately their approach is very expensive and SGI (which owns Cray) is perpetually hovering at the point of bankruptcy.

Air cooling is still the norm, mostly because it's relatively easy to engineer and (so far) provides enough heat transfer. If you get 100 cfm through a computational node with 70 F on the inlet you can run it at full power indefinitely. We did a design study using cold plates and some dripless disconnects. It worked, but it wasn't cost-effective for various reasons. The second evolution of our new product line will probably have the option of water cooling.

Gamer types who overclock their CPUs and thus benefit from extra cooling usually have lots of room in their computer cases. We don't. Our vertical-mount cases are only about two inches thick, not big enough for plumbing and cold blocks without significant redesign. The joy of air cooling is that the coolant medium is readily available and requires only a minimum of ducting.

Producers like Cray have custom cabinets in order to maximize the benefits of their physical design. We use standard 72U enclosures, around which much other equipment and design traditions have evolved. The architects who specialize in scientific and engineering facilities know how to design for them, and there are products for cooling and for raised flooring that presume those form factors.

As has been the tradition in server room design, the cabinets are arranged in "hot" and "cold" aisles. The rows of cabinets are arranged front-to-front and back-to-back, with vented tiles on the floor of the cold aisle and vented ceiling tiles on the ceiling of the hot aisle. Hot air exhausted from the rear of the cabinets is drawn upward into the ceiling where it travels transverse to the rows to the edges of the room. The air conditioners draw it downwards and chill it, discharging it under the floor. The pressure from this discharge forces it upward from the floor in the cold aisles, where it is drawn into the front of the cabinets.

A certain amount of spill is acceptable. Where thermal tolerances are tighter, the inlet and exhaust from the cabinets can be ducted directly. But the use of a raised floor and a suspended ceiling as plenums is fairly common.

[Joe Durnavich]

Hot air exhausted from the rear of the cabinets is drawn upward into the ceiling where it travels transverse to the rows to the edges of the room. The air conditioners draw it downwards and chill it, discharging it under the floor. The pressure from this discharge forces it upward from the floor in the cold aisles, where it is drawn into the front of the cabinets.

I am finding that the typical person's conception (including our building's management, unfortunately) of air conditioning overlooks this aspect of recirculating the air through the air conditioners where they incrementally extract heat from it. Our office building is down to just one failed unit which means that on our floor, half has air conditioning, the other half doesn't.

(My office is in that other half, alas. My mind doesn't lend itself easily to conspiracy theory, but that just has to be a plot against me...)

Several fans have been set up in a hallway that connects to two halves of the office to blow air from the cooler side to the hotter side, with the idea that "cold" on one side will move to the other side. I have to admit, it sounds like a good idea, but in practice, it makes little-to-no difference. When the air conditioned side starts to heat up in the afternoon, the explanation is that we have taken all their cold air and brought it over to our side (trust me, we haven't). Complaints to building management only bring more fans to put in the hallway. I believe I am only one fan away from being able to rent that hallway to NASCAR teams so that they can blow their cars and tweak aerodynamics.

[JayUtah]

The fundamental rule of cooling is that your equipment must be able to suck heat out of the air as fast or faster than the rate at which heat is being added to the air. Nothing you do with fans will alter than inescapable fact. If your building was spec'ed to require two air-handling units per floor, one unit will not suffice. Moving the air around does not compensate for the inability to remove sufficient heat from the air.

Now in practice you really do have to pay attention to the fluid dynamics. In my case we use Fluent to model the airflow in large installations and the rooms/buildings in which they live. But you pay attention to the fluid dynamics because bad flow requires more raw cooling in order to compensate for inefficiencies introduced by flow problems (e.g., columns, changes in ceiling height). There is no way to use fluid dynamics to improve the cooling solution beyond its theoretical limit. If you have one cooling unit where two are required, there is no way you're going to win.

[sts60]

Sorry for the thread necromancy, but I finally got around to rehosting the images I previously had on the now-defunct iuploads and fixed my page 2 post with the Denali pics. Anyway, it's handy bumping up Jay's gold-standard penguinaut series.