Putting the 'role' back in role-playing games since 2002.
Donate to Codex
Good Old Games
  • Welcome to rpgcodex.net, a site dedicated to discussing computer based role-playing games in a free and open fashion. We're less strict than other forums, but please refer to the rules.

    "This message is awaiting moderator approval": All new users must pass through our moderation queue before they will be able to post normally. Until your account has "passed" your posts will only be visible to yourself (and moderators) until they are approved. Give us a week to get around to approving / deleting / ignoring your mundane opinion on crap before hassling us about it. Once you have passed the moderation period (think of it as a test), you will be able to post normally, just like all the other retards.

"How game sizes got so huge, and why they'll get even bigger"

Infinitron

I post news
Staff Member
Joined
Jan 28, 2011
Messages
97,228
Codex Year of the Donut Serpent in the Staglands Dead State Divinity: Original Sin Project: Eternity Torment: Tides of Numenera Wasteland 2 Shadorwun: Hong Kong Divinity: Original Sin 2 A Beautifully Desolate Campaign Pillars of Eternity 2: Deadfire Pathfinder: Kingmaker Pathfinder: Wrath I'm very into cock and ball torture I helped put crap in Monomyth
Informative article for Codexers who complain about number of GBs: https://www.pcgamer.com/how-game-sizes-got-so-huge-and-why-theyll-get-even-bigger/

How game sizes got so huge, and why they'll get even bigger
It's the textures above all, and nothing is going to stop them.

5kKS38G9f2A89TGg4Vhu5G-650-80.jpg


'Crunch,' in modern parlance, is a period of unhealthy overwork during the lead up to a game's release. But in the not-too-distant past it could have also described the nature of the work itself: a crunching and culling of excess data, enough to squeeze the whole game onto a DVD or two and comfortably onto a 100GB (or smaller) hard drive.

Game sizes have always been limited by their delivery medium—Myst was famously made possible by the introduction of the CD-ROM drive—but in 2004, when dual-layer DVDs were introduced, it was already too late for the PC game shelves at retailers. Steam launched around the same time, and as broadband snaked further into the suburbs and beyond, no physical medium could keep up with downloading. Games were allowed to grow, and grow, constrained only by users' bandwidth and hard drive space. That's resulted in 100GB-plus games that wouldn't even fit on a Blu-ray disc, never mind a dual-layer DVD.

That's causing grief for players who don't have access to speedy internet connections, or who haven't recently upgraded their storage in a pricey SSD market. It's especially bad for those stuck with data caps, a maximum data allotment per month that, if exceeded, results in extra charges. Broadband Now has catalogued 210 internet service providers that impose data caps.

For PC gaming to get better for those with poor internet service, either games will have to get smaller (or at least stop growing), or our selection of ISPs will have to get better, providing quality, uncapped service to more customers. After speaking to a few game developers, I can say confidently that the former isn't going to happen. The upper limit of game sizes is only going to increase.

yaiTTRHaQDiFuejWPYGxmH-650-80.jpg


What's taking up all the space?

Audio contributes a great deal to file size, but it hasn't been the primary contributor to the growth of game sizes in general. The resolution (sample rate) of audio will depend on what sounds the developer is reproducing—Tripwire, for instance, uses 44kHz audio for weapon sounds, and 22kHz audio for speech—but that was already the case back when it released Red Orchestra: Ostfront 41-45, one of the first third-party games to release on Steam.

The use of 5.1 surround sound audio over mono or stereo audio has been one reason games are getting bigger, but the simple inclusion of more audio has probably had a greater effect. We expect voiced characters and high fidelity audio far more than we once did.

Video, on the other hand, has broadly increased in resolution—from 640x480 cutscenes way back when all the way up to 4K—which has notably increased its footprint on game sizes. Video can be heavily compressed, though. "1500x compression factors are not unheard of," says Stardock lead developer Nathan Hanish, pointing to the H.264 codec. So while high-quality audio and high-resolution video have helped bulk up games, they alone can't account for the leaps in size we've taken since 2000.

One element does check all the boxes for exponential growth: textures. Like video, textures are increasing in resolution, except unlike video they aren't fond of being compressed. Images can be heavily compressed—as with jpgs—but artifacting would be noticable. And regarding Hanish's specific work on Stardock games, "[DirectDraw Surface] images have to use a GPU friendly memory layout," he writes. "They only use local block compression, and thus yield at best an 8-to-1 compression."

On top of that, textures are also growing in complexity. "In 2005 you had a texture, just a texture, which is what later people would call a diffuse texture, but it's just a texture," says Tripwire Interactive president John Gibson. "And then in the next generation, you have a diffuse texture, a normal map texture, and usually a specular—like a 'shininess' texture. So not only are you using a higher-resolution texture, because video card memory increased, and computer memory increased, but you also have three of them for every object."

NUBmueSgQpDLZwM8xMacPX-650-80.png

A visualization of Killing Floor 2's elements by size, captured with Stardock's SpaceMongerapplication. Note the large .TEX files.

Those objects—their meshes—are increasingly complex, too. A breakdown of Tripwire's games tells the story. In the mid-2000s, Red Orchestra 1 released at 2.6GB, which players thought was awfully big at the time. ("People would complain if we did a 100 megabyte patch," says Gibson.) Sound only accounted for 327MB of that release, and environmental meshes and textures weighed in at 1.4GB. Jump to Killing Floor 2, which released last year, and sound now accounts for 1.1GB of the whole. Environmental meshes and textures? 17.4GB.

"One of the quickest ways to make your game look better is to up the texture resolution," says Gibson. "Content will be authored at a very high resolution no matter what game you're making, and then you res it down to what will fit into memory. And as memory expands, people keep doing higher and higher resolution textures."

The same trend can be seen in Stardock games from 2003 to the present, beginning with Galactic Civilization 1: Ultimate Edition, which was largely bulked up by Bink video files. In 2011's Galactic Civilization 2: Ultimate Edition, textures grew to snag about 15 percent of the total filesize on their own. In 2015, Galactic Civilization 3's textures grabbed another five percent to become 20 percent of the total size. And in Stardock's latest, Ashes of the Singularity: Escalation, textures account for 60 percent of the total file size, says Hanish.

Textures aren't the only culprit, of course. Some games have no 'textures' in the 3D rendering sense in the first place, and some are very heavy on high-res video and can attribute the bulk of their size to that. Everything else has been growing, too, including executables, geometry, redistributables, and so on. But "textures have grown exponentially and represent (by far) the largest absolute and proportional area of growth—at least for Stardock products," writes Hanish, so if you want something to blame, look to the mossy rocks, the peeling wallpaper, and the freckled faces.

5kE5VqCyo756k9tjoxEakP-650-80.jpg


Why are some games so much bigger than others?

Game sizes can be confusing. Why, for instance, are The Witcher 3 and Rainbow Six: Siege both 50GB despite one being an open world RPG and the other being a multiplayer shooter? Such comparisons indicate that the 'scope' or 'size' of a game, as we perceive it in terms of 'a world' or 'hours of story,' has little bearing on install size. Vast plains built of terrain geometry and repeating grass textures appear bigger in our heads than they actually are on disk, while a complex gun model that's physically small in-game may weigh more than we know.

NBA2K17 is 66GB, for instance, a hefty install size. Why would a game that takes place within NBA arenas be bigger than the entire world of The Witcher 3? For one thing, every player from every team is modeled and textured to a high degree of fidelity, whereas The Witcher 3 probably contains far fewer unique bodies and faces. And as Hanish pointed out when I brought up the topic, there's likely a heap of motion-captured animation data in NBA2K17.

There are other questions. How much video is included? And audio? And is it compressed? Titanfall was a big game after install—48GB—because it included uncompressed audio to lessen CPU load. We can assume that from Respawn's point of view, accommodating dual-core CPUs would increase the number of satisfied customers more than a smaller install would have.

qnmknFtQxLrvDdMmoA6GnP-650-80.jpg


Why not just compress everything?

That 48GB Titanfall install was 'only' a 21GB download, though, meaning it was heavily compressed. So why don't more games do that? An educated guess: it was only the fact that Titanfall's large install size was mostly audio that made it possible to compress it to 21GB in the first place.

It's those pesky textures again. We've established that textures have been the biggest contributor to the overall increase of game sizes, and as Hanish said, textures can't be too heavily compressed. Unfortunately, they also can't be heavily compressed during download and then transcoded at install time, because that process "is extremely CPU and memory intensive and must really be performed in the studio," says Hanish (adding later that "some flavors of dds are much more intense than others to transcode").

That's not to say that compression doesn't occur (Steam itself compresses games and decompresses them after download, Gibson tells me). Nor does it mean developers don't take pains to decrease download sizes. It just isn't practical for every type of data, and isn't nearly as vital as it used to be.

Oculus VR coder Tom Forsyth, who previously worked on games for the PC, Dreamcast, and Xbox, recalls the crunching required to squeeze games onto discs back in the late '90s and early 2000s. He built utilities to compare compressed and uncompressed textures, to determine which looked acceptable in their compressed state and which had to be left uncompressed. He also hunted down duplicate textures and other unnecessary bits and pieces. While working on StarTopia, Forsyth even invented his own audio compression format to solve a problem with the music—with a week before shipping. Solving these problems, sometimes at the last minute, was necessary, as "you either fit on the disc or you didn't."

The same space-saving efforts happen today (Hanish described more-or-less precisely the same processes), but because an extra 200MB no longer blocks a release, and Steam won't take a bigger cut for a bigger game, there's far less incentive now to set a hard size limit during development. If shaving off a gig means a delay, or means resources can't be allocated to pressing bug fixes and improvements, it probably isn't going to happen.

"It's an infinite rabbit-hole," writes Forsyth. "You could spend years on compressing this stuff. But nobody has the time. Especially as you only really appreciate the problem once you're close to shipping. So do you delay shipping by a month to reduce the download size by 20 percent? That's a terrible trade-off—nobody would do that."

No developer I've spoken to denies that, with enough time and work, bytes can probably be taken off of any final build—at least until you get to some theoretical lower limit. But does cutting 50MB matter for a 30GB release? That's especially questionable when, as Gibson points out, the game is probably going to grow after release anyway. Free Killing Floor 2 updates have added new maps, modes, skins, items, and so on since its release.

2D games and intentionally low-poly, flatly-textured games will continue keeping the low bar low. Stardew Valley, for instance, is only a 462MB download. But Forza Motorsport 7 is a 96.5GB download, and there's no sign that the upper bar is going to stop rising.

"The moral of the story is: buy stock in ISPs and hard drive companies," jokes Gibson. And that's how it is. As our hardware gets better, developers are going to want to use more and more of our memory for higher quality audio, video, geometry, and textures—which means bigger games. We have little choice but to rely on our internet service providers to make fast, uncapped connections affordable. Unfortunately, ISPs, many of which pushed to repeal Net Neutralityregulations in the US, are not great allies. But at least our big, big games are going to look shiny (or specular) as hell.

Sidebar:

HIDDEN DATA
Why is it that people always seem to be finding unused assets in games? Cut content often makes it into full releases simply because of how complex game development is. Forsyth mentions, for instance, that artists may bundle a texture with a model for another artist to work with, and that texture might make it into the game even though it's duplicated elsewhere.

And as games get more and more complex, and take longer and longer to develop, the probablility increases that unused data will make the cut.

"Because these projects are so big, there are so many assets built up by so many people over a number of years, it can actually become very difficult to know which assets are or are not actually running in a game," says Hanish. "…So I imagine that, obviously not in any of our games, but in other people's games, there are probably a fair number of art assets that make it out into the distribution that aren't actually used anywhere."

(I'll take Hanish at his word that Stardock's games are free of excess data.)
 

lightbane

Arcane
Joined
Dec 27, 2008
Messages
10,156
Shoehorning every cutscene in different languages, instead of using the one used for installation, inflates the game's size, an unfortunately common tendency. That, and optimization having become a lost art.
 

Freddie

Savant
Joined
Sep 14, 2016
Messages
717
Location
Mansion
I don't really follow AAA space, but at least Activision and EA should be in position to licence h.256 and HE-AAC / xHE-AAC in bulk. I wonder if those are used.
 

Infinitron

I post news
Staff Member
Joined
Jan 28, 2011
Messages
97,228
Codex Year of the Donut Serpent in the Staglands Dead State Divinity: Original Sin Project: Eternity Torment: Tides of Numenera Wasteland 2 Shadorwun: Hong Kong Divinity: Original Sin 2 A Beautifully Desolate Campaign Pillars of Eternity 2: Deadfire Pathfinder: Kingmaker Pathfinder: Wrath I'm very into cock and ball torture I helped put crap in Monomyth
It's not cutscenes men, this isn't the 1990s anymore.

I first realized the impact of TEXTURE BLOAT around 2014 when games like Blackguards started coming out with 20+ gigabyte footprints. You look through their data files and it's just piles and piles of high-res textures.
 

Freddie

Savant
Joined
Sep 14, 2016
Messages
717
Location
Mansion
Yeah, it's says that much in the article, but it also says why textures are difficult to compress. So that leaves video on audio and h.256 and xHE-AAC are both in ISO which can help a lot and by no means I meant that xHE-AAC would be used only in cut scenes. It's purpose is to work for speech and music in very good compression ratio.
 

sser

Arcane
Developer
Joined
Mar 10, 2011
Messages
1,866,661
What I'm reading between the lines here is that some devs have just gotten lazy with compressing because hard drives have gotten better and internet speeds are faster.

Company of Heroes: 1gig.


Six years later...

Company of Heroes 2: 30gigs.
 
Self-Ejected

unfairlight

Self-Ejected
Joined
Aug 20, 2017
Messages
4,092
It honestly doesn't matter to me. Physical is dead and I have a unlimited connection, I don't care if it's 5 or 80 gigs but typically the only things that are above 50 are just AAA shit I don't care about. I think we're sorta at it's peak at this point and we won't be going above what we have today for a while
 

Freddie

Savant
Joined
Sep 14, 2016
Messages
717
Location
Mansion
sser

It's between the lines in bold font or something. Perhaps I'm cynical but big bureaucracies are ineffective, then some folks come up with standardisation to make management easier and work flow / production more effective sometimes customers may see something of these benefits. New stuff comes around, over time possibilities lead to ineffective use of resources. Rinse and repeat.
 

Roguey

Codex Staff
Staff Member
Sawyerite
Joined
May 29, 2010
Messages
35,653
Fucking 4k is the reason why I can't get the PC ports for Bayonetta and Vanquish (well I could, I just don't wish to endure the bandwidth tightening to do so at this time). The least they could have done is left those as an optional download, like Skyrim's high-res textures. I don't have nor will I ever get a 4k monitor so I have no use for those even taking up space on my hard drive.

Unfortunately, ISPs, many of which pushed to repeal Net Neutralityregulations in the US, are not great allies.

Oh boo hoo, the thing we never actually needed for decades got repealed.
 

PorkBarrellGuy

Guest
This is stupid. Graphics don't make games more fun. Zork, NetHack, etc. are fun. None of those multi-gig monsters are anywhere near as fucking fun as an old-ass game that fits on a couple of floppies tops. Why are they doing this? Why do they think it's going to make anything better? Fuck sake.
 

Wirdschowerdn

Ph.D. in World Saving
Patron
Joined
Nov 30, 2003
Messages
34,458
Location
Clogging the Multiverse with a Crowbar
This is stupid. Graphics don't make games more fun. Zork, NetHack, etc. are fun. None of those multi-gig monsters are anywhere near as fucking fun as an old-ass game that fits on a couple of floppies tops. Why are they doing this? Why do they think it's going to make anything better? Fuck sake.

It's not about fun, it's about keeping the industry's gears smeared. Marketing, hardware vendors and a surplus of artists demand perpetual "advancements" in graphics and other costly guff.

If you want real innovation, I'll stick with the smaller studios who are still willing to take risks.
 

SCO

Arcane
In My Safe Space
Joined
Feb 3, 2009
Messages
16,320
Shadorwun: Hong Kong
There are as many reasons as there are CEOs. My personal theory is that all of these companies watch each other products for the next cargo cult or marketing point and when dedicated graphic cards power got over the '100 fps' marker they hopped into the '4k' train.

Nothing very strange just a deliberate strategy of 'faux luxury' that has benefits on marketing and shutting down upstarts to the indie ghetto even if they deliver a good game. Fortunately, 3d storage is just around the corner, so it's quite likely we'll not give a shit soon (if you were tasteless enough to care already).
 

ShadowSpectre

Arbiter
Joined
Mar 11, 2017
Messages
333
Location
Limbo
Since most PC games are downloaded now anyways and aren't usually bought off store shelves anymore, developers might as well give the option to check off which resolution of textures you want so you don't have to download a bunch of stuff needlessly. It would also help if you wanted to jump into a game faster and download the higher-res textures at a later date. Although seriously, 4k? Pretty sure some games are packed with 8k textures too with the sheer size of them.

The worst part is we could probably find some games made a few years back that were a fraction of the size of current ones that look better. Sounds like a lot of graphical bloatware included with the games as well.
 

Stavrophore

Most trustworthy slavic man
Patron
Vatnik
Joined
Aug 17, 2016
Messages
12,604
Location
don't identify with EU-NPC land
Strap Yourselves In
For every asset you can have up to 6-8 textures/maps -normal, occlussion, displacement, metallic, curves and more. A good modeller/texturer meticulously create model, then uv unwrap it, making sure every UV piece won't be doubled.
Now we have automatic uv unwrapping that bloat textures. Things that could look as well with optimized 2k textures, needs 4k textures because of lazy devs. Then there are raw cutscenes. I mean i don't see much problem, unless the game takes like 60+ GB.
 

Zed Duke of Banville

Dungeon Master
Patron
Joined
Oct 3, 2015
Messages
11,756
Daggerfall (1996) had several options for installation size, with the largest occupying about half a gigabyte of hard drive space. In terms of game size relative to typical hard drive size, even the most bloated games of today can't compare. :M
 
Joined
Apr 3, 2013
Messages
2,071
Location
Siberia
I remember deleting some inessential stuff from system folder to make space for both HoMM3 and Fallout on my 1.6gb drive. Good times.

You can do quite a bit in terms of optimising space and memory usage as far as models and geometry go. You can't really do much about the textures though, aside from maybe making the Ultra HD stuff a separate download for people who have rigs that can handle it.
 

Unkillable Cat

LEST WE FORGET
Patron
Joined
May 13, 2009
Messages
27,088
Codex 2014 Make the Codex Great Again! Grab the Codex by the pussy
Does this mean that big installation size = literally sexual harassment?

Don't be silly, ALL OF THEM are literally rape.

As for the article - unless graphical engines start to move away from textures into other rarely explored territories (they exist) this is not going to change anytime soon. That said, trimming the fat from your code sounds like a good idea until you realize that no one cares about doing that anymore... that's the more troubling sign for me.

By sheer coincidence I was sorting through data on a hard drive last night and came across my Thief 1+2 Fan Mission archive. For Thief 1 it's 1.47 Gb in size, for Thief 2 it's 32.6 Gb. Remembering an old post here about those FM's being full of redundant data, I started cleaning them up. After 20 minutes I had cleaned the Thief 1 archive of all of its 50 temp files and it still stood at 1.47 Gb. Another 40 minutes later I had scratched the surface of the 2000+ temp files in the Thief 2 archive and reclaimed 200 Mb of space, most of it from FM releases of the last two years. Keep in mind that the Thief 1 archive mostly consists of files from 1999 to 2002 and has little growth today, while the Thief 2 archive mostly has files from 2003-2009 but still receives constant additions to this day. Somewhere in the 00s the idea seeped in that bloated file sizes are OK. The article just happens to mentions a specific event in that time period to explain that...
 

As an Amazon Associate, rpgcodex.net earns from qualifying purchases.
Back
Top Bottom