Tim Sweeney News entries for 1999

Happy New Year

The team is now getting back together from the holiday and we'll get back to Unreal 221 development first thing tomorrow (Saturday).  We were on the verge of releasing the patch right before Christmas, but we're now back to being a few days away, since I added some new features that need re-testing, and some additional bugs have been found.

For me, the holidays were fun and productive.  I hauled my computer up to Maryland for a one-week visit with my parents, and that provided a good excuse to work on engine improvements and new R&D in a zero-distractions environment.  Without getting to deep into the details, here are some of the things I worked on:

• Unicode Unreal.  I finished up 16-bit Unicode support; now the Unreal code base can be recompiled for the ANSI (8-bit) or UNICODE (16-bit) character sets.  This has several advantages.  First of all, localizing a game for non-Roman languages (such as Japanese, Chinese, Korean, Hebrew, Arabic) is easy with Unicode.  Second, we're really interested in Windows CE, and that is a pure Unicode OS.  Windows CE is the native operating system of the Dreamcast, so Unicode is a natural stepping-stone along the Dreamcast porting path which we're exploring.   Windows CE is also used in tons of palm-top computers.  Now, none of these palm-top computers are currently interesting from a gamer's perspective, but if you extrapolate their grown--in terms of video resolutions, RAM, and processing power--along a Moore's Law curve, they're not that many years away from being comparable to low-end Pentium II PC's today.  I like to think that the GameBoy's of the future won't be a closed platform, but rather run a standardized OS like Windows CE.

  All Unreal data files are "character set neutral", meaning they can contain any mix of ANSI and Unicode characters, and they're automatically converted/truncated at load time.  Network play similarly "just works" between the Unicode and non-Unicode versions.

  Regarding Unicode font issues ("who's going to paint all those Japanese characters?"), Jack Porter has built a TrueType font importer into UnrealEd, which converts any Windows TrueType font to an Unreal-compatible font bitmap.  This is an essential part of our Unicode efforts, but also ties into the new user interface he and Brandon are building for Unreal Tournament.  There are lots of Unicode fonts with support for nearly all Windows-supported languages, so localizing games into all languages should be fairly painless.

• Data compression.  I spent a couple days experimenting with various compression schemes and found some interesting results.  First of all, it's fairly easy to achieve PKZIP-quality compression--the algorithms are well documented and easy to implement.  Second, it's nearly impossible to beat PKZIP by much of a margin.   Beyond the standard algorithms (Huffman and LZH compression), I experimented with many variations, including:
  • "Double" Huffman-compressing each character in a file then recursively merging compressed regions together into spans, with the goal of compressing "different types of data" with different encoding tables.  This approach came out neutral: there were huge gains in compression, but they were offset by having to store lots of Huffman tables.
  • Trying Huffman compression in bases other than 2 (such as 3, 5, and 7): no overall gains.
  • "Infinite sliding window": An LZH-style scheme that spends a huge amount of CPU time on compression by analyzing an entire file for differences.  Here, the gain was typically 10%, but it was unbearably slow to work with.
  • Arithmetic encoding.  This is a Huffman-style scheme that employs some basic number theory to generate a compression table using arbitrary bases (other than base 2).   This gets real gains of around 10% compared to Huffman, but is much slower to decode because it relies on arbitrary-precision integer math.

  I didn't get the compression code into the Unreal master source (it was standalone R&D).  I'm considering supporting compression for client-downloadable files in Unreal Tournament, to make it easier to go onto servers running custom maps.  My most usable compression scheme got 4:1 compression on .unr and .u files (nice!) but no significant compression on textures and sounds.

  Unreal's textures compress poorly for an interesting reason: they are already stored using palettes, which are a form of compression.  Furthermore, the process of generating an optimal palette for an image yields an array of bytes which are distributed near-uniformly, which is the worst possible case for variable-code-word compression.   I think the state-of-the-art in texture compression will be S3's real-time S3T/DXT scheme, which is fairly lossy, but optimal for real-time decompression while rendering.   And it's 6:1 for all possible kinds of data.

  One interesting technique I played around with for texture compression is wavelet compression.  The 2D wavelet scheme for textures is quite similar to mipmaps: to oversimplify a lot, you represent an nxn image as a sequence of 1x1, 2x2, 4x4, 8x8 ... nxn.  Each bitmap is stored as "signed differences" relative to the lower-res versions of the bitmap.  Wavelets are a good starting point for lossy compression schemes, because the "most visually noticeable" details are in the low-res bitmaps, and in the high-frequency portions of the high-res ones.  So, it's fairly easy to filter out unimportant details and compress what remains.  Wavelets are also nice because you can generate all your texture's mipmaps from a single wavelet without any overhead.  While I don't think wavelets will play a role in our future texture-mapping plans, they look more useful as a way of storing and manipulating height maps and displacement maps.  Wavelets provide a very efficient, "level of detail"-aware form of storing 2D surface geometry.

• 3D API's.  I spent a few days learning the basics of curved surface rendering, which turned out to be very interesting.  I was going to do my research with Direct3D (under Windows NT 5.0 Beta 3 RC0) but the 3D hardware support isn't quite there yet, so I took the opportunity to learn OpenGL using the software rasterizer and 3dfx's OpenGL minidriver.  Both renderers were extremely stable and easy to use.   To my amazement, I never experienced a "blue screen of death" or had to reboot--quite a difference from my past work with Glide and Direct3D!  (Note: I haven't been very closely involved in the Unreal Direct3D and OpenGL support, which has been developed by our partners, so this is my first really in-depth experience).

  OpenGL's approach to window management turns out to be extremely simple and reliable.  I went from zero to having a spinning cube up and running in under 30 minutes.  The tools GL provides for polygon drawing are extremely simple and powerful.  For just drawing a few lone polygons, glVertex() and its associated functions are super-easy.  For drawing complex meshes, GL's approach to providing separate client-side arrays for vertices, colors, and texture coordinates is very easy and efficient for multi-pass rendering, since you can swap out one table (say, texture coordinates) without affecting the others.

  OpenGL's texture approach (glTexImage2D and glBindTexture) are also very simple and powerful.  I remember back under DirectX5, trying to upload a texture to hardware.   I spent 4 hours trying to figure out the code and rebooting my computer as it repeatedly locked up.  I spent less than 15 minutes figuring out OpenGL texturing and implementing it.  Comparing both API's texturing approaches, OpenGL's texture management is absolutely the right way to go.

  • It's extremely foolproof--you call 3 textures to create, upload, and delete textures.   Nothing can possibly go wrong!
  • There is no bizarre/mysterious emulation going on.  For example, in Direct3D you can modify a texture by Lock()ing it and writing to "video memory" directly.   But, on some hardware (like the Riva TNT), the hardware stores textures internally in a "swizzled" format you can't access directly.  So some emulation thing has to go on behind the scenes where multiple copies of the textures are maintained and copied around.
  • OpenGL hides the inner details of texture management from the app, enabling the driver to be optimized for whatever style of texture management is best for a particular 3D card.   In OpenGL, a driver writer has total control over texturing and can optimize the hell out of it.  In Direct3D, the API hides the application from the hardware and vice-versa, which makes it impossible for a game or a hardware driver to implement good texture caching.

  Here's a link to OpenGL information and DirectX information.

• Curved Surfaces. I became familiar with a bunch of different techniques.  Curved surfaces pose the same kind of dilemma I felt when writing Unreal's software renderer: there are lots of mutually incompatible approaches to choose from, but no really clear favorite.  Each technique has some desirable strengths, but no technique combines them all.

  My first experimentation was with bicubic bezier patches, which are a great primitive for building curvy shapes without holes between patches (as with polygons, continuity can be guaranteed by having adjacent patches share vertices and control points).  However, they require a fair amount of math for tessellation and, though continuity can be guaranteed, it's hard to join patches together smoothly (with a continuous normal vector across adjacent patches), especially during animation.  I'm pretty sure these will be a key primitive in the next-generation engine.  Beziers are very intuitive to model with.

  The tensor-product math for bicubic bezier quad patches can easily be modified to enable "bicubic bezier triangles", which can share edges continuously with quads.  I haven't been able to find any references on this, but I derived the math, and it seems to work.

  Then, a totally different approach can be used to generate smoothly curved surfaces of arbitrary topology, using a technique known as "subdivision surfaces".   This looks promising: approximately the same overhead as Beziers, but a bit more general.

  All of the above approaches can also be modified to support displacement maps, for creating more organic geometry.  While it will be a long time before 3D hardware supports sufficient triangle densities for displacement maps to be practical for fine surface detail (such as rocks and bricks in walls), they're very applicable to terrain and coarse surface detail.

  There are two even wilder curved techniques I experimented with and found to be quite promising, but I don't want to get too far off on that tangent.

  No, Unreal Tournament will not contain any curved surfaces.  While curves are cool and we'd love to have them, we've been totally focused on core game play, networking, and user-interface enhancements for UT.

  One thing that has become extremely clear in my curved-surface rendering research is that editing tools will be a make-or-break factor in the success of next-generation level design.  UnrealEd is a pretty feature-filled polygon based editing tool, but we're going to need a lot more than that for the next generation.  So many features that were trivial with polygon engines--such as texture alignment, light map placement, primitive building tools, object alignment, and freeform object editing--become more complex when curves are involved.  The editing tools will need to be more powerful and intuitive in order to compensate.  Editing tools will be a huge focus for us going into the next project.

-Tim Sweeney

Other Updates

• We updated and coolified the Unreal Technology Partners page, which lists all the announced Unreal engine licensees. 
• Alexander Brandon updated the Music composition page.
• Dave Carter's Nerve Damage contains lots of cool resources for Unreal modelers and animators.

Latest News

Progress on Unreal 221 and Unreal Tournament continues.  We are making changes and improvements at such a fast pace that it has been hard/impossible to stabilize the code for the 3-5 days required to fully test a new patch.  I apologize for the earlier false start we had with 221.

What we've been programming

I've been juggling work on the installer, OpenGL, and some core UnrealScript improvements.  The installer is coming together, and is complete in terms of installing the retail game and patches, but more work remains to be done on uninstallation issues, and improved support for installing/distributing mods.  Some of the latest UnrealScript improvements include a command-line script compiler, which can be used from any programmer's editor (or Visual C++) with standard output of error messages; UnrealScript execution optimizations; and support for dynamic strings.

UnrealScript's string support has been pretty solid all along, as a result of using BASIC style functions (Left, Right, Mid, InStr, Chr, Asc...) instead of C's error-prone string handling functions (strcpy, strcat, strcmp, sprintf).  The one annoying limitation, which has been the source of some bugs in the past, is that strings were fixed length.  221 supports dynamically sized strings with no practical size limitation (just available memory).  This makes it easier to write text-processing routines, which the new in-game server browser is really stressing.

Steve Polge has been improving the AI and the flexibility of the game code.   Because Unreal Tournament's gameplay can be far more complex than Unreal 1's (from a bot's perspective) as a result of team play and Capture The Flag, Steve has been teaching the bots a lot of new techniques for coordinating together and better understanding the environments.  It's sooo cool watching Cliff and Myscha playing CTF with a few bots on their teams.

Jack Porter is hammering away on the new user interface code and the in-game server browser.  In the past, first-person action games haven't been known for having good user interfaces.  We're hoping to change that, and go for the quality level of an EA Sports interface.  Sticking to our general-purpose roots, Jack isn't just creating user-interface code for Unreal Tournament, he's creating a flexible, object-oriented UI framework which will be very suitable for licensees and mod makers.

Brandon Reinhart is working hard, implementing a lot of the Unreal Tournament game code, the single-player progression, stats tracking, and much more.  The stats tracking is going to be a very cool feature for Internet play; perhaps I can convince Brandon to write a note about it here.

Erik de Neve, who created Unreal's algorithmic fire and water effects and optimized much of the rendering code, is working on a next-generation project that will make its debut in Unreal 2.

Linux Notes

We now have a command-line (windowless, console based) script compiler and server for Unreal, which has minimal dependencies on Windows.  This is a first step towards a Linux port, and it also runs very well under Wine, with minimal overhead.  This might be a good interim solution for Linux server administrators, before the full port is done.   I just noticed in GameSpy that the top server on the list is titled "CHiX Unreal NT Server (220) PORT UNREAL TO LINUX".  Hint taken. :)

We actually have the Unreal 221 Core source compiling under EGCS 2.90.29, C++ templates and all.  Getting the thing linking and running successfully will require a lot more work, which we won't be able to focus on until UT ships, but the experience thus far has been very positive.

-Tim Sweeney

3dfx Voodoo3 Totally Rocks!

I just picked up a carload of the new Voodoo3 3000's; we've put them to good use, as the team is now testing the upcoming 224 patch.

The performance and graphical quality of the Voodoo3 are simply amazing.  Though we've had a board for over a month, beta drivers and pre-release board problems prevented us from realizing just how far ahead of the pack this hardware is.

On the PC, most technological improvements come in little incremental improvements, such that we seldom have a chance to experience a single major leap.  Back when 3dfx introduced the Voodoo 1, that was one of those rare leaps.  Since I've been off working on "OpenGL land" on the Riva TNT, ATI Rage 128, and other accelerators, I haven't had a 3dfx card in my machine since soon after Unreal shipped.  So, for me at least, getting a Voodoo3 and being able to play at those ultra high resolutions at a great frame rate, is another one of those leaps.

While the Voodoo3's fill rate is outstandling, where the card really clobbers all others is its texture management performance.  This is a very important characteristic, because it determines how smooth the performance is from frame-to-frame.   Unreal, though it shipped nearly a year ago, still pushes texture limits harder than other 3D action games, and Unreal Tournament pushes them even harder. 

Unreal 224 performance is also being helped by some new lighting optimizations, and a major new engine feature, which Erik de Neve (Unreal optimization and algorithmic texture guru) has been working on.  But I'm not allowed to talk about that since Mark Rein is writing a press release about it now.

Note: Unreal 220 is incompatible with Voodoo3's due to some changes that were made to Glide; people who have Voodoo3 2000's will need the upcoming 224 patch. 

-Tim Sweeney

Getting closer...

But we're not quite there with 224 yet.  The guys have been beating on internal test versions for the past few days, and we've been polishing the code.  It's both a blessing and a curse that the Unreal code is evolving by leaps and bounds, gaining major new features all the time.  This approach makes testing each new patch a major effort.  Though, 224 will be especially worth the effort because of the in-game server browser and level-of-detail optimizations.

224 will probably happen late this week.

Here is a Next-Generation article where Mark Rein shamelessly plugs the patch.

Network Cross-Compatibility

(warning: nitty-gritty technical info)

An interesting technology feature in 224 is a new method of evolving .u files in a way that is backwards-compatible for network play.

One of the things that has always bothered me about past versions of Unreal is that clients and servers had to have exactly-matching .u files in order to play together.   This has meant that every recent Unreal patch has broken network compatibility, leaving some players orphaned.  The reason behind this is that Unreal's network code (described in detail here) uses the positions of objects within .u files as a "common point of reference" which clients and servers use to map objects between each others' address spaces.   Some scheme of this nature is necessary when network code is generalized like Unreal's is, with the same scripts executing on the client and server, passing arbitrary data back and forth, with the network code transparently coordinating the game world.

I've been aware of some solutions to this problem all along (such as "send all object names as strings"), but they all sacrificed bandwidth.

Recently, I found a no-compromises way to enable .u files to evolve incrementally without losing compatibility.  Now, when we release a new version, we copy the .u files into a new developers-only subdirectory \Unreal\SystemConform.  Then we can make minor changes to our scripts and recompile new versions that are network-compatible with the old ones.  When recompiling and saving a .u file such as \Unreal\System\Engine.u, the script compiler now examines the existing file \Unreal\SystemConform\Engine.u, and makes the new version "conform to the layout of the original".  In addition, when a client connects to a server, they both negotiate to figure out who has the earliest version of each .u file, and both talk about objects "in relation to that old .u file".  This enables compatibility without any loss in performance or bandwidth usage.

Really major updates will always break code assumptions and require downloading an update.  This approach just makes minor incremental updates possible, enables us to test spot-fixes on the net, and generally makes our programming lives easier.

STAT NET: Diagnosing your connection to the server

When playing a multiplayer game of Unreal, press TAB and type STAT NET to bring up the network stats display.  You'll see something like this:

(IN/OUT):
   137         PING
    38         CHANNELS
     0       0 UNORDERED/SEC
   12%     14% PACKET LOSS
    11      32 PACKETS/SEC
   126      32 BUNCHES/SEC
  1395    2031 BYTES/SEC
  2600    2600 NETSPEED

The numbers on the left represent incoming statistics (for example, how many packets from the server to you were lost), and the right outgoing statistics (for example, how many packets from you to the server were lost).  There interesting stats are:

• PING: Round-trip time between you and the server, in milliseconds.  Modem users will typically see 200-350 for in-country gameplay.  ISDN/T1 users will be in the 50-200 range.  Playing overseas results in enormous times, 600-1000 or more.   This number represents the quality of the connection between you and the server, and really should be 300 or less for good gameplay performance.    A bad connection on either end can hurt the ping time.
• CHANNELS: Roughly, the number of objects that the server is actively updating to you.   Proportional to how much combat is happening around you.
• PACKET LOSS: Percentage of packets lost between the client and server.  The higher the packet loss, the worse the performance will be, and the more you'll see objects skipping around rather than moving smoothly.  High packet loss can be caused by:
  • Using too aggressive a network speed on the client side.  Try typing "NETSPEED 2600" on the console to reign in bandwidth usage. 
  • The server's connection is overtaxed.  This will result in heavy packet loss and a bad gameplay experience for everyone.  Servers will limited bandwidth available need to set their play limit conservative (expect 5000 bytes per client per second), and also lower the MaxClientRate variable in Unreal.ini.  A value of 5000 should be good for everyone.
  • A bad connection anywhere between you and the server.
  • A bad connection between you and your ISP.
• PACKETS/SEC: Number of packets being sent per second.  The number on the left should be 13 or higher.  If it's significantly less, that means the server is either running too slowly, or experiencing high packet loss.  Gameplay will be bad at 10, and unbearable at 5.  The number on the right is proportional to your frame rate and you don't need to worry about it.
• BUNCHES/SEC: Number of objects being updated per second.  Not important for tweaking.
• BYTES/SEC: Bytes being sent per second.
• NETSPEED: The network speed setting you set using the console NETSPEED command.   Recommended settings are:
  • 2600 for a typical modem connection.
  • 2400 for a poor modem connection.
  • 5000 for an ISDN connection.
  • 20000 for a T1 connection.

Server Alert

Unreal 224 servers are popping up very rapidly, and becoming filled to capacity rather quickly.  From the last couple hours of cruising the net and playing on various servers, we're seeing wildly varying performance.  As with all games, the quality of Unreal netplay is a function of how well the server is performing and how good your connection is (bandwidth, packet loss, and latency).

If you're having a bad experience, please try out a few other servers.  This can affect performance hugely.  In my game sessions here, I had some great performance with a 300 msec ping and 28.8Kbps bandwidth, and some astronomically bad performance with a 100 ping and 50Kbps bandwidth.  The bad performance fell into two categories.

Some servers have more users than bandwidth available, so they are dropping packets, experiencing 50%+ packet loss and escalating ping.  Some are just running very slowly, around 5 fps (I'm not sure why, perhaps other processes were running?).  This tends to happen whenever we release a patch, as people rush to get into gameplay, flooding the servers that have upgraded, users setting up servers on cable modems out of desperation to get into a game... It can get chaotic.  In the future, the new backwards-compatible network code should mean there are always lots of servers available when a patch is released.

So, bottom line:  Please don't bitch if netplay performance is erratic over the next couple days as servers get set up and stabilized.  If netplay performance sucks after a couple of days, THEN commence bitching. :-)

-Tim

Unreal 224v C++ headers released!

For hardcore mod authors who want to experiment with Unreal's C++ interface, here it is!  This code requires Microsoft Visual C++ 6.0 with no service packs or SP1.   Read the Public Source ReadMe for more information and distribution terms.

Now it's time for a disclaimer about the C++ code.  For mod authors, UnrealScript is the best way to write game code.  The fact that 100% of the Unreal 1 and Unreal Tournament game-specific code is in UnrealScript illustrates our strategy here, that UnrealScript is a fully-featured language designed to simplify game programming compared to C++.  The C++ code is significantly more complex than UnrealScript, and is only necessary for programming things that just aren't possible under UnrealScript, such as algorithmic textures,  performance-critical AI decision making,  and that kind of low-level work.  Furthermore, our distributing just the headers (and not the C++ engine source), means that many key aspects of the engine are hidden in implementation details, so it's going to be hard to understand what's happening "under the hood".  Another major downside to C++ is that C++ mods will break with almost every upcoming patch, whereas UnrealScript is more stable (having broken only 3 times in 11 months and approximately 12 public betas released).  Summary: This C++ code is for very, very hardcore mod makers only.

That said, there is some cool stuff here.  I've included all of the core and engine header files and .lib link libraries in this release, so you can compile your own DLL's and link them to Unreal.  I've also included the C++ code for some modules which we don't feel are confidential: Unreal's OpenGL support layer (OpenGLDrv), the engine startup code (Launch), and the installer/uninstaller (Setup).

The OpenGL code will be of interest to OpenGL driver writers who want to see what we do in Unreal, in order to optimize their drivers, and possible to enthusiasts who want to experiment with rendering.

We will probably be able to publically release the Direct3D driver code soon too, also to aid driver writers and experimenters, but our Direct3D development partner is going to first spend some time cleaning up that code.

This code is provided as-is: we don't have much time for answering everyone's questions about it, so you're mostly on your own.

The C++ code can be downloaded here. Unzip it into your \Unreal directory, and the proper subdirectories will be created (described in the ReadMe).  The C++ code compiles there and links to DLL's in the \Unreal\System directory, read to run.  Enjoy!

-Tim Sweeney

Unreal 224v Released to the public!

The Unreal 224v patch is now available for download from our downloads page, along with the release notes.

-Tim Sweeney

Maps that broke since 220

Some user maps such has DmHellShrap broke with version 224. This was caused by a little bug in the file backwards-compatibility code, that I've fixed for 225.  Map authors don't need to do anything, this will automatically be fixed with 225.  The specific problem I fixed was an exit with an assertion:

    Assertion failed: Field [File:..\Inc\UnClass.h] [Line: 139]

If there are any user maps which don't require .u file mods, and the maps refuse to load with a different error message than this, please email the map (or a URL where we can download the map) to unreal224@epicgames.com.

-Tim

Quick Notes

We've been going through the Unreal 224 bug reports and addressing lots of issues people have pointed out.  Here's the latest status.

• Creative Labs EAX 3D sound support is fixed for 225.
• We're working with Aureal to get A3D 2.0 support into 225, which is very likely to happen.
• I fixed the cause of the "out of memory" errors that tended to happen when the console was pulled down on some machines. There was a problem in UnrealScript with switch statements operating on strings, which caused this and some similar problems reported by licensees.  Thanks to Mark Poesch of Legend for tracking this one down.
• Under some conditions, alt-tabbing between fullscreen and windowed play would cause the in-a-window game window to be collapsed to zero size. Fixed.
• I optimized the network visibility code by about 25%.  Server admins are reporting notably more CPU usage per player with 224 than 220.  I haven't tracked this down, but I'm going through a lot of code to measure and improve the server performance.

Many other fixes, tweaks, and incremental improvements are in the works. See the Known Bugs page for more info.

-Tim

Unreal 225 musings

The recent work on 225 has been focused on improving server performance and stability, and it has been paying off really well.  It turns out that 224's server code was noticeably slower than 220, as a result of some code improvements I made but didn't test and time rigorously.  The two bottleneck routines in the server are the visibility tracer and the actor replicator.  I've been making the visibility tracer faster by processing all actors simultaneously and optimizing the code for a pure "yes/no" visibility test, rather than using the more general (and slower) line trace routine built into the engine.  The actor replicator has gained significant speedups from a combination of low-level optimization (to the bit-stream writer, and replication condition caching code), and a new time-stamping system that speeds up replication by up to 50% on servers with high player counts.

Now, the 225 server is faster than all previous versions, and I'm working on some further improvements over the next few days.

As many server admins have pointed out to us, if we can double Unreal's server performance, admins will be able to run twice as many servers on their machines!

"Make Something Unreal" Contest

Check out the Contest site for the latest info and news on the contest for Unreal map makers, mod authors, artists, and TC developers!

Things That Are Cool

• Microsoft Visual SourceSafe:   Anybody who is programming on a large project without using a source-control system is nuts!  SourceSafe provides a central repository for all code, and enables programmers to check out files and work on them without their changes "colliding" with those of other programmers.  SourceSafe also tracks all changes to all files since the beginning of a project, which has been incredibly useful in developing Unreal, because it enables us to examine our past code changes when tracking down bugs.  Finally, SourceSafe is excellent for managing "branched" projects: when we split Unreal 224 apart from the Unreal Tournament codebase, SourceSafe managed the files' shared history automatically.

• GLsetup:  Now that Microsoft has publicly committed to dropping the ball on 3D hardware driver distribution with DirectX7, GLsetup is the only hope for game developers who want their game to work on users' computers out-of-the-box.  The PC is already plagued by driver problems, and the classic symptom is users having to go to the web and download new drivers to get new games working.  The world would be a much better place if Microsoft took a leadership role in driver updates, by keeping the DirectX redistribution continually refreshed with new QA-certified drivers, thus assuring that new games always work "out of the box" -- if that happened, PC gaming would be a lot closer to the level of reliability of console games.  But Microsoft isn't doing that, and GLsetup is a grassroots effort to solve the problem.

-Tim

P.S. Check out Unreal Nation, the cool in-depth Unreal news site.

Latest News

• 4 out of 5 users have reported success with the experimental UnrealEd fix -- it made those nasty "Runtime Error" messages go away.  If you're having problems with UnrealEd not starting up, grab this file!
• The other one reported an error message relating to Threed32.ocx which I fixed in UnrealEd Experimental Fix 3.
• There was a master server outage/problem today.  If you weren't seeing servers in Unreal's new server browser or GameSpy today, don't worry, the problem was at the master server and it has been fixed.
• Aussies, check out the OzUnreal Contest for Australian level designers!

Holiday?

Here at Epic HQ, we suspect today is some type of holiday.  The tell-tale sign is that the parking lot was empty when we all came to work this morning.  That usually gives it away.  However, we haven't figured out the identity of the occasion -- it seems too warm outside for Easter, and our publisher isn't calling us three times a day about shipping our game so it's probably not Christmas.

New 225f Patch for server administrators only

This should fix the UActorChannel::ReplicateActor crash and some smaller problems.   It's backwards-compatible with Unreal 224-225 clients.  If you have any server crashes, please email your Unreal.log file to unreal225@epicgames.com.

Only download this if you are running an Unreal server: UnrealPatch225f.exe.

-Tim

Server 225e Patch Feedback

Server admins are reporting much better performance, but 90% of the bug reports point to a crash in UActorChannel::ReplicateActor.  I've been fixing this (and a few other things) and will have a new server-only patch later tonight.  Thanks for everyone's feedback!

UnrealEd Experimental Fix 2

Based on feedback, Saturday's experimental UnrealEd fix solved the problems for only about 20% of users (wild guesstimate) who were previously unable to run UnrealEd.  On further investigation, we've found a set of Visual Basic runtime DLL's that are more likely to work for most users.  If UnrealEd still isn't working for you, please try Experimental Fix 2.  Please let us know what happens by emailing unreal225@epicgames.com with a subject of "Experimental Fix 2". 

IMPORTANT: After installing the fix, reboot your machine before running UnrealEd.   The proper files won't be refreshed until you reboot.

Webmasters: Please only post this info on Unreal news sites.  I don't want it to go on general gaming sites until we have feedback on patch success/failure from users.

-Tim

Direct3D Improvements

Since Riva TNT users are reporting that Direct3D is more than 30% faster than OpenGL on their cards, I've been spending some more time improving the D3D code, adding better support for video mode switching, and optimizing the texture management code a bit more.

Another motivation for the Direct3D improvements is the Matrox G400.  I just received one, and it's an awesome video card--fast fill rate, great graphics quality, and decent texture management performance.  The G400 has a great Direct3D driver, but their OpenGL is hideously bad.

One interesting new feature I managed to implement in Direct3D is recursive, fractal detail textures.  With this feature enabled in Direct3D, you never see any individual texels, no matter how close you get to a surface.  It's a really interesting contrast to the blurry bilinear filtering you see near surfaces in most games.  Considering how fast current 3D hardware has become, there's no reason for games to reveal any individual texels anywhere.

-Tim

Direct3D Is Happening

We're testing 226a internally now.  The results of are good so far.  If no significant "non-driver" problems are found, we'll have Unreal 226a out later this week.  Here is the kind of feedback we're seeing on D3D Unreal performance.

First client impressions, after zonking it with a couple of really nasty, detailed, clippy, high polycount maps: incredible improvement! I got through DMDowntown, my personal UnrealRavager(tm) with nary a clip, pop or tick. Switching resolutions works well now, but you have to have the latest Hercules driver (now using 1.70, dated 6-28-99) for it to work, or you get a critical error. ALT-ENTER works fine too, thanks Tim (now I can get multiple screenshots without exiting and restarting Unreal). DMBreakfast, a new map which uses a great deal of UnrealEd rendered graphics, played very well too.

This is a quantum leap for the 'in game' video speed on the Hercules TNT2. Most impressive. I'll try to get this patch on the Nali City game server (unreal://207.53.187.28) tomorrow so we can all beat it up.

-Pete, PlanetUnreal

This isn't a miracle improvement, but TNT, G400, and Rage 128 owners should see clear speed and smoothness gains. Just make sure you get the latest drivers available for your card.

-Tim

On Our World Domination Plans

Ok, the headline is exaggerating a bit.  Still, there are some cool things happening with the Unreal engine that we find very exciting--uses of the technology that we never dreamed would exist a couple years ago.

For example, as the latest PlanetUnreal news story describes, the Digitalo Design team is building a "virtual reconstruction" of the Notre Dame Cathedral.  And it looks jaw-droppingly amazing!   The team here, busy working on Unreal Tournament, is very jealous about the polygon counts these guys are using (the lucky bastards don't have to worry about the CPU impact of 12 players deathmatching in their creation!)

Also, the rapidly-growing independent news site, UnrealEngine.com, is covering lots of other Unreal engine projects that fall way outside the "first person action" stereotype, such as Hasbro's Nerf Arena Blast (see AVault preview) -- as well as the other well-known projects from our partners.

There are other interesting things happening with the tech, behind the scenes.  A number of schools (ranging from high schools to universities) are using UnrealScript to each real-time object-oriented programming.  Several university masters-degree thesis projects are centered around projects involving the engine.  Some architects are using the editor to generate real-time architectural walkthroughs.  At least two Silicon Valley startups are pitching completely non-game projects incorporating the Unreal engine to investors.

What's really happening here?

What we're seeing is part of a larger trend that will continue to grow.  Remember ten years ago, when PC's were very weak compared to the state-of-the art: SGI workstations, and the Unix workstations from HP, Sun, and Apollo?

Now, a $2000 Pentium III PC with a Voodoo3 or TNT2 card eclipses the performance of a $30,000 SGI for real-time rendering.  The CPU is faster, the fill rate is faster. When I first saw an SGI Reality Engine, my impression was, "Holy cow, I can't believe how much better this is than my '286!"  But nowadays, the best 3D games look far cooler than anything you see running on a Reality Engine.

That entire segment of the high tech industry, the one containing non-PC workstations, SGI's high end hardware, simulation tools vendors, and so on, is falling off a cliff and disappearing.  One-time leaders like SGI and Intergraph are turning into niche players, who are losing out badly to the new guys like Dell, Gateway, NVidia, and 3dfx who were born and bred in the consumer, high volume, low-price PC era.

The result is that projects, like the Notre Dame reconstruction, architectual walkthroughs, and training simulations--which were once the realm of high-end SGI's--are now coming to the PC in droves.  Soon, they'll all be here.

The Economics

What we're seeing in processors, 3D accelerators, and games is a fundamentally different level of technological progress than happens in other markets. Industrial era businesses, like car manufacturing, and limited by raw materials and labor.  Most of what you pay for a car goes to materials, labor, and the other economic components of the industrial pipeline (like transportation, insurance, and so on).  Years of tweaking the process have made manufacturing near optimally efficient, so future progress goes very slowly, gaining perhaps 2%-3% productivity per year.  That's why there is no Moore's law for car performance.

But in our business, the cost of raw materials is insignificant, so innovation is driven by volume and R&D investment.  A 500 MHz Pentium III now costs the same to manufacture as a 60 MHz Pentium cost in its day, or a 386-16 long ago.  A retail game contains around $2.50 of raw materials (box and CD) and sells for around $49.95.  So, manufacturing isn't a limiting factor here.

Still, games, processors, and 3D cards require a tremendous up-front investment in R&D and facilities.  It takes lots of people and a lot of capital to develop a game, design a chip, or design and build a manufacturing plant. The difference is, this investment is a fixed cost--you pay it once, then you have the ability manufacture as many games or chips as you desire at an (approximately) insignificant cost.

The remaining piece of the puzzle is the "winner take all" nature of technology markets.  Since manufacturing costs are small, price differences are minor, leadership goes to the companies that have the best high-volume products.   Therefore, everyone invests heavily in R&D, and attempts to produce the best product in hopes of selling tons of chips or games or engines.  When they do this successfully, they make a profit and plow it back into the business to get further ahead.   Most R&D and capital-investment hungry markets work this way; it's why you only see a handful of prominent CPU makers, 3D card makers, and engine developers.  The ones who delivered were successful early on and reinvested in their continued growth; and those who didn't deliver either out-of-business or are falling apart now.

The Result: General-Purpose Solutions Win

The result is that high-volume general-purpose hardware and software is quickly gaining the lead in absolute performance.  This trend will only accelerate as the PC market becomes larger and the R&D investments grow.  The ramifications are:

• Specialized software and hardware loses out to general-purpose.  The company that focuses on generality (for example, a CPU, or a 2D+3D graphics accelerator, or a general-purpose game engine) has the expectation of higher sales, and can make so much more of an R&D investment that special-purpose solutions can't compete (for example, 3D-only accelerators, or a once-off engine powering a single game).

• Volume and increasing R&D investment enable 3dfx and NVidia to sell millions of $150 graphics cards that outperform SGI's $50,000 solution which sells 1000 units.  That has the effect of crushing SGI's business model, even though companies like 3dfx aren't even consciously trying to compete with SGI.

• When a company like Epic or id Software builds a 3D engine, we do it with the expectation of it powering games that sell a few million copies, between our games and licensee projects.  We make money from each one, and that realization drives investment in making more and better tools.  This gives our engines a price, performance, and feature advantage over "in-house" engines made specifically for one game, and over less-capable engines designed for niche markets.  We amortize our R&D investment over ~10X more units sold.  This licensing model has already proven successful with mainstream game developers, but now it's starting to overflow into non-game markets too.  We have this great 3D engine, why not use it for your architectural walkthrough?

• As a result of widespread interest, thriving ecosystems develop around successful general-purpose products.  For example, there are hundreds of web sites dedicated to 3dfx hardware, and lots of developers optimizing their games for 3dfx.  There are many hundreds of web sites covering Unreal and licensee projects; thousands of kickass aspiring level designers building maps and making them available online; many licensees building games; and lots of other cool projects that tie into the engine, such as mods and TC's, and the research projects like Notre Dame.  The community has a multiplicative effect on growing the platform.

Where we go from here

When you look at the big picture, what's happening now with 3D graphics on the PC is just the tip of the iceberg.  In the past, we've been limited to the realm of hardcore gamers, but now 3D acceleration is becoming much more mainstream, and 3D engines are becoming recognized as a viable development tool for a wide range of projects.   The coming years will be interesting.

-Tim

New Downloads & Links Pages

By popular request, the links on the tech page have been updated and cleaned up.   I added:

• New Unreal Downloads page: An index of all the official files here on the tech page, plus links to dedicated Unreal download sites.  If you're looking for the latest PC or Mac patch, UnrealEd patch, Fusion maps, or whatever, look here.

• New Unreal Links page. Links to some of the major "hub" web sites related to the Unreal community, Epic, our partners.

I also wanted to mention the supercool new download site, Unrealfiles.com, check it out.

-Tim

Unreal Tournament development note

In response to the pirate punk who has been claiming to have a "final" version of UT...

Unreal Tournament isn't finished yet, so there isn't a final version. We're still polishing, fine-tuning, and optimizing the game and will 'till it's done.

Believe me, when UT goes gold, we'll be announcing it here prominently.

Until the demo and release version are available, any copy of UT you see is an unfinished beta version; unless you're a reviewer who received such a copy direct from Epic or GT, having such a copy is illegal. These versions are clearly labeled "confidential beta version" or "release candidate - not for distribution" in at least 5 places in the program. In addition, they have time-bomb logic which will cause them to stop functioning after a certain amount of time. So if anybody gets hold of a version of UT before release, it should be pretty clear that it's not legitimate.

DirectX7

Microsoft has released DirectX7.  You can read about it and download it from Microsoft's DirectX Page and download it on their Download Page.

Now that DirectX7 is available, we're doing final testing and tweaking on the upcoming Unreal Tournament demo with Direct3D and OpenGL support, coming in the next few days.

One thing we've found on DirectX7 is that NVidia TNT2 performance is not fill-rate limited until you get to very high resolutions.  During testing, performance at 640x480 seemed a bit slow, but as we increased the resolution, the frame rate was hardly impacted at all.  So 1024x768 seems to yield the best overall experience on this card.

I've enabled support for 32-bit color textures under Direct3D, which significantly improve the graphical quality, with a 10-15% frame rate impact.

The Direct3D code now uses vertex buffers, which speeds up mesh and text drawing a bit.

What about OpenGL?

We're still maintaining OpenGL sypport, though it's not as much of a priority at the moment because texture management performance in GL on Windows is significantly behind Direct3D.  It's still useful in NT4 and Linux of course.  I've been advocating a GL extension which would enable the fine-grained control over texture management that has led to such an improvement in Unreal performance under Direct3D.  Here is a suggestion I sent to the OpenGL ARB.

I've been maintaining the Unreal / Unreal Tournament rendering code under both Direct3D and OpenGL simultaneously for a while. They used to have approximately equal performance back when I was using Direct3D's "driver-managed textures" (similar to the one and only OpenGL option for texture management, which is transparent to the application). Unfortunately, the performance wasn't very good, and the memory usage was nuts because our game uses a huge amount of textures, and the automatic texture management in Direct3D and OpenGL had to keep duplicate copies around as "backing store" to the copies in video memory.

However, I recently rewrote my Direct3D code to manage textures explicitly. I create a limited number of textures of all possible sizes, forced to be in video memory, and then at runtime swap my actual game textures into those "video memory textures". This way, I have complete control over texture management, and can make optimal decisions about where to put textures, based on the constraints of my app, such as:

• Texture usage patterns.
• Decompressing textures at glTexImage2D time using my own internal formats.
• Having a background thread load them off disk speculatively and stick them in video memory when needed.\
• Other nonlinear, possibly time-variant priority factors.

When I replaced D3D's default texture manager with my own code, Unreal's performance and memory usage under Direct3D improved very significantly, to the point where it's not worth bothering playing the game in OpenGL anymore.

So, my question is, does anybody have plans to add an "extended version of glBindTexture" which lets the application tell the OpenGL driver, "allocate this texture in video memory, keep it there, and never swap it out"? If an option like this existed, I could do the same kind of high-level optimizations as I did in Direct3D, and probably get a comparable increase in performance. Obviously, video memory is a limited resource, so at some point those glBindTextures will fail if video memory fills up, and have to return an error code.

This is a bit lower-level than GL's existing glBindTexture mechanism, but I think it's justified by the need to support apps which use huge quantities of textures and have to manage them carefully in order to maintain realtime frame rates.

For example:

If the GL_EXT_BIND_TEXTURE_2D_VIDEOMEMORY string is exposed, then you can replace a call to glBindTexture( GL_TEXTURE_2D, MyBindId ) with glBindTexture( GL_TEXTURE_2D_VIDEOMEMORY, MyBindId ) and the texture is guaranteed to be allocated in video memory and never moved around by the GL driver's texture manager.

For my application, glPrioritizeTextures(?) isn't a viable solution, because it still requires extra backing-store copies of textures to be kept in system memory so they can be swapped in and out. In addition, implementing my own texture manager on top of the existing, unextended glBindTexture isn't at all efficient, because I don't have any way of knowing how many textures I can allocate before they start spilling out of video memory, and in addition most GL drivers do lots of extra internal memcpy's in the process -- and with big textures, having lots of unnecessary 256K memcpys (for 256x256x32-bit textures) just kills performance.

Such a solution would benefit Unreal engine games on Windows, Linux, and possibly Mac such as Unreal Tournament, Duke Nukem Forever, Deus Ex, Wheel of Time, etc etc etc. As well as other developers' texture-intensive games that use OpenGL, such Starsiege: Tribes (the developers have been running into this same issue).

Experiencing poor Internet play on your  Voodoo3 3500TV?

As Voodoo Extreme reports, the Voodoo3 3500TV's WebTV installer does some evil things to your Internet setting, causing many games (including Unreal Tournament and Quake 3 Arena) to experience poor Internet play:

Hello, We are aware of the problems that the Voodoo3 3500TV is experiencing with online games. Many of these problems can be cured by removing WebTV, however that will disable the Visual Reality software. The problem usually can also be cured by removing the Internet Explorer 5 upgrade from Windows98. While these are just a work around, and not really fixes, both options usually will cure the problem until 3dfx releases a real fix. We are currently working with Microsoft to try and figure out whether the problem is due to Microsoft software (WebTV and IE5) or 3dfx software (3500TV drivers and Visual Reality).

Thank you for your patience, Aaron D. Patton 3dfx Interactive Email Support

-Tim

TNT Users, Try This!

Tweaks:

• Play in 16-bit color, which is significantly faster than 32-bit color.
• Turn the detail options down in the Preferences/Video menu.
• Use the "-" key to reduce the HUD size.
• In Preferences/Game, turn off the display of your weapon.
• If you have enabled Advanced Options / Audio / Use3DHardware audio support, TURN IT OFF for now! Big driver performance problems.

Experimental improvement for the brave:

• Try NVidia's Beta Detonator 3.34 drivers (thanks to Anandtech's site for the link).
• Try putting this new D3DDRV.DLL into your \TournamentDemo\System directory. Please back up your existing one first!

These two downloads are beta and haven't been rigorously tested yet, but in our gameplay sessions here, they're a huge improvement on low-RAM machines with the TNT, TNT2, and GeForce256.  

Please let us know how these work for you, by emailing utbugs@epicgames.com

Latest Findings

We've been benchmarking UT performance and memory usage on many 3D cards, and we've found the source of the performance problems on 64-96 meg machines with TNT/TNT2 cards.

Our measurements indicate that the TNT/TNT2 drivers consume up to 25 megabytes of page-locked system memory with UT, while other Direct3D cards don't.  On machines with 64-96 megs, that kills performance, as one would expect.  The game runs, but data is swapped to the hard drive so frequently that playability is compromised.   We've been comparing with other 3D cards and aren't seeing the issue on the Rage 128 or Voodoo3 under Direct3D.

System-wide memory allocations measured with SYSMON.EXE on a 64-meg Celeron running Windows 98 SE:

• TNT D3D: 51 MB baseline, 122 MB playing UT in 16-bit color, 134 MB playing UT in 32-bit color.
• Rage 128 D3D: 55 MB baseline, 107 MB playing UT in 16-bit color, 109 MB playing UT in 32-bit color.
• Voodoo3 D3D: 51 MB baseline, 100 MB playing UT in 16-bit color.

We're working with the NVidia guys to track down the source of the extra 25 megabytes of memory consumption.  I'm going to hold off on posting our benchmarks till we've had a good chance to deal with the memory consumption issue, so everyone is on a level playing field.   The NVidia cards already perform well on PC's with lots of RAM, so they should be in good shape once the memory issues are solved.

Links

Unreal Technology in the news:

• The Little Engine That Can Do More - MSNBC
• Playing With The Big Kids - Newsday

Cool Sites:

• Unreality.dk - Cool Unreal site & community host in Denmark.
• GameSpy.com - The GameSpy empire's new portal site.  Check out their live stats page, where Unreal Tournament is #6 and rising quickly.  Hopefully someday we'll overtake that 3-year-old QuakeWorld thing. :-)

-Tim

Windows 2000 RC2 Rocks!

I recently upgraded to Windows 2000 Release Candidate 2, and wanted to mention that it's extremely solid!  With NVidia's latest Windows 2000 TNT/GeForce drivers, DirectX7's Direct3D and OpenGL both work flawlessly.  What's cool about the new OS:

• High-performance DirectX7 Direct3D and OpenGL support.
• USB mouse support -- mouse movement is very smooth.
• Much more stable than Windows 98, especially for software development.
• Good administration tools, more powerful than NT4 and far easier to use.
• Great international support.
• Dual processor support.
• Plug and play.

Windows 2000 Internationalization

This is a topic we're very interested in, because we're always trying to make our games and engine as widely accessible as possible.  As a matter of fact, today I'm working out the font management issues for the Japanese version of Unreal Tournament.

The Windows 95/98 OS had very weak international support, and a confusing array of incomplete features such as supporting Unicode in a few API's but not all, weird and undocumented multibyte character set handling, and mixed ANSI/OEM file system conventions.  Localizing Windows 9X apps is a very difficult undertaking.

On the other hand, Windows 2000 is by far the most thoroughly internationalized piece of software I've ever seen.  All international versions sharing the same binaries, fonts, and input support. Everything is 100% Unicode.  So, you can open up a Japanese language document on your English computer, visit Arabic web sites, use an Input Method Editor to type in Chinese text, and so on.  While most people might not care about that, it's infinitely beneficial for developers to have that power at their fingertips, because it enables you to do all of your international development and most of your testing without jumping through hoops and installing multiple OS's.

When Windows 2000 makes its transition into the consumer OS market, it will be a great day for worldwide software development.

-Tim