SteamPipe/SteamCMD conversions?

[Powerlord]

How are the SteamPipe/SteamCMD conversions going to be handled?

[Edge100x]

We've handled conversions for DoD:S and HL2MP servers already, and they were relatively smooth. We convert servers by switching to the new file set, moving custom content to the new place Valve wants it, and updating Metamod:Source/Sourcemod for the customer.

[Brewskii187]

What are the new directories? (for us VPS users) it seems to be hard to find good information on this.

[Edge100x]

All custom content, except for addons and configuration files, goes into sub the "custom" folder on a server that has been switched to the SteamPipe/SteamCMD version of the game. Any .vpk files would be placed directly into "custom" while loose files would need to be placed into subfolders of "custom". Valve talks more about how this works on the Steampowered website.

For any standalone game server or managed game server that we run, we will post an event to the log describing the changes at the time that we make them.

[Powerlord]

All custom content, except for addons and configuration files, goes into sub the "custom" folder on a server that has been switched to the SteamPipe/SteamCMD version of the game. Any .vpk files would be placed directly into "custom" while loose files would need to be placed into subfolders of "custom". Valve talks more about how this works on the Steampowered website.

For any standalone game server or managed game server that we run, we will post an event to the log describing the changes at the time that we make them.

It's not required to move all custom content there, though... and what subdirectory under custom are you using?

While custom/maps will work, it works because someone (which may have been me) on the server mailing list was suggesting that people move them into custom/maps and Valve updated the server accordingly... it's actually supposed to be something like custom/custommaps/maps

[Edge100x]

It's not required to move all custom content there, though...

Nearly all custom content must be moved in order for the server to see it. Valve explains in the post that I linked you to.

Our system handled the changeover for our customers, so the change is seamless. I don't believe that other companies are doing the same thing we are; most are requiring that customers start from scratch.

and what subdirectory under custom are you using?

You can use any directory name. We use the one that we posted about in the "Events log" that customers can read.

If you need more help on how to manually convert a server not in our system to SteamCMD, please post in our dedicated server section.

While custom/maps will work, it works because someone (which may have been me) on the server mailing list was suggesting that people move them into custom/maps and Valve updated the server accordingly

Odd. I have not heard this and I'm not sure that it's true.

Maps actually work in the original place (cstrike/maps) as well. They are one of the few exceptions to the custom content location requirement. However, you should still put them in the correct place inside the custom folder, as that is undocumented behavior that Valve may change. Additionally, for our customers running managed game servers, the Websync tool will not look for files that are not in "custom".

... it's actually supposed to be something like custom/custommaps/maps

Yes, that is what is documented.

[Powerlord]

It's not required to move all custom content there, though...

Nearly all custom content must be moved in order for the server to see it. Valve explains in the post that I linked you to.

This is why being in hlds/hlds_linux is an advantage. You see posts like this one from Fletcher Dunn on Feb 20:

Re: Where to put custom maps and assets. Putting them directly in the tf/maps folder should still work, but it sort of negates some of the advantage of the new search path flexibility, and I don't think it's the recommended thing to do. You might have noticed that under SteamPipe, we've choosen to ship all the .bsp files loose on disk, rather than inside a VPK. (There is no advantage in putting them in VPK, they are already basically big package files themselves.) So if you put custom maps directly in tf/maps, they will be all mixed up with the standard maps. (Similar comments apply to any sort of custom asset. It's best to avoid mixing them up with the official game assets.) Probably most in line with the intentions of the new filesystem changes would be to make one or more new directories for all custom server stuff. You could put them in VPK if you are not using the fast download server. But fast download server needs them loose on disk.

It was implied that this also covered custom models, materials, and sounds.

[Edge100x]

This is why being in hlds/hlds_linux is an advantage. You see posts like this one from Fletcher Dunn on Feb 20:

I am a subscriber. I saw the email. I read the email, and all other emails on this topic. I also read all of the SteamCMD-related pages, and the readme.txt files being distributed with the games. I recommend that you read those pages and readmes, as they are easier to understand than a chain of emails, and more up-to-date.

... Putting them directly in the tf/maps folder should still work...

That's what I said. I must again stress that you should not do this, though.

You said earlier that maps could go in "custom/maps", which is not correct.

It was implied that this also covered custom models, materials, and sounds.

You can certainly try not putting them in "custom", but it probably won't work in the future, if it does work now.

It has been quite awhile since Fletcher sent the email that you quoted, and since then he has made clarifications on what needs to be done, as they have continued to develop the system. For instance, on March 14, he said:

To clarify: each subfolder under custom becomes its own search path.

So this is an example of the right thing to do:

dod/custom/[customization_name]/models
dod/custom/[customization_name]/materials
dod/custom/[customization_name]/maps

Note that:
* the custom folder goes under the game folder, not the "orangebox" folder. (There is no longer an orangebox folder or any shared content or binaries. Each game is totally distinct.)
* you need to make a subfolder under "custom". Putting "materials" or "maps" directly under custom is probably not what you want.

Read gameinfo.txt for more details on this.

The DoD:S beta just released had some old readmes in the wrong folder. I will fix that shortly with a beta update today.

And here's a quote from the wiki that I previously linked and suggested you review:

Q. Can I still customize any game file?

Yes. Create a folder such as dod/custom/my_custom_stuff/ and put your custom files there.

For example, you might customize the scoreboard by extracting the resource/ui/scoreboard.res file from the appropriate VPK (using the VPK tool in the bin folder, or a free tool such as GCFScape) as dod/custom/my_custom_stuff/resource/ui/scoreboard.res, and then modifying it. When the game boots, it will see the subdirectory my_custom_stuff and add that folder as a search path. Then your version of resource/ui/scoreboard.res will take priority over the official shipping version of the file.

You can have more than one such subfolder under dod/custom, (for example, you might be working on two different customization projects and don’t want to get the files mixed up), but each folder is another search path added in front of the VPKs will slow down load times a little more. When you’re done tweaking your customization, you can always pack it up into a VPK and delete the subfolder.

Q. Why can't I just edit files directly in the game folder like I did before?

Modifying files directly in game installation folders (dod/resource, dod/materials, etc) won't always work anymore. Here's why.

Previously, when the engine looked for a file, it would first search folders on your hard disk, and then look inside the Steam GCF files. This meant that custom files in the dod folder would take priority over the files in the GCF.

This method had two disadvantages:

It slowed down loading, because the vast majority of files were in the GCFs. Booting the game or loading a level would cause many thousands of operating system calls to attempt to open files, most of which would fail.
Files from many different sources were mixed together. (Files you intentionally customized, files downloaded from gameservers you visited, files from third party mods, and any official shipping games files that were not in the GCF.) Removing a mod was difficult because the files were scattered across several directories. This problem will be exacerbated with SteamPipe, because more game files are shipped loose on disk than was the case when GCF's were used.

The new system searches VPK files first, and then search folders on your hard disk. This reduces load times significantly. But it means that the file in the VPK's we ship will always take priority over loose files in the dod folder.

Users interested in understanding the filesystem search paths in more detail should take a look at dod/gameinfo.txt.

We have tested this and everything that we tell customers in the "Events log" update is accurate. Custom content for all customers was moved successfully and it's being detected properly.

[Powerlord]

It was implied that this also covered custom models, materials, and sounds.

You can certainly try not putting them in "custom", but it probably won't work in the future, if it does work now.

Posted by Fletcher Dunn two minutes ago:

If there are no filename conflicts with the official game files, then you can just drop them directly in the tf/maps, tf/materials folder as before and it works.

[Edge100x]

I do not understand why this matters to you. We moved everything to the preferred location in "custom" already and that is the best place to put the resources. Per Valve's documentation, it is where they want them to be put, and as I mentioned, this should continue to work, whereas the other location may not.

You are posting in the forum for regular game server customers and your initial question implied that you were a game server customer wondering how we were planning the conversion. I have described what we did, why it's the best course of action, and where resources need to go from this point forward.

If you aren't actually a game server customer, then you can do things in your own way and perform the conversion yourself in whatever manner you see fit. Moving files to "custom" is the ideal solution, but you may prefer to do something else. The forum to use for discussions of software questions related to unmanaged services is: http://www.nfoservers.com/forums/viewforum.php?f=44

[Powerlord]

I do not understand why this matters to you. We moved everything to the preferred location in "custom" already and that is the best place to put the resources. Per Valve's documentation, it is where they want them to be put, and as I mentioned, this should continue to work, whereas the other location may not.

You are posting in the forum for regular game server customers and your initial question implied that you were a game server customer wondering how we were planning the conversion. I have described what we did, why it's the best course of action, and where resources need to go from this point forward.

If you aren't actually a game server customer, then you can do things in your own way and perform the conversion yourself in whatever manner you see fit. Moving files to "custom" is the ideal solution, but you may prefer to do something else. The forum to use for discussions of software questions related to unmanaged services is: http://www.nfoservers.com/forums/viewforum.php?f=44

I'm in an odd position in that I'm your customer on two different fronts: My game group leases a standard TF2 server from you and I personally lease a VDS from you.

[Edge100x]

Ah, ok.

[Powerlord]

Are maps being moved to be under custom/ as well? One of the SourceMod devs noted that the map list functions, used by SourceMod's mapchooser and its variants, doesn't support the virtual filesystem.

[kraze]

Are maps being moved to be under custom/ as well? One of the SourceMod devs noted that the map list functions, used by SourceMod's mapchooser and its variants, doesn't support the virtual filesystem.

Yes, the maps were moved into custom into a folder called old_custom_files/maps.

[Powerlord]

There seems to be permissions issues moving files from old_custom_files to other locations.