tools: add repo-version to print a package version from git references

[illwieckz]

This is an extraction and a port to Python of the printVersion function from the release-scripts/build-release script, the produced version string is usable for dpk archives, but also fr other things like unizip or the symbols zip for which we expect to have the same version string of the game dpk.

This is a standalone re-implementation of what is done in Urcheon for computing dpk version strings.

This is usable in two ways:

Without argument:

tools/repo-version/repo-version

This prints the version string for the current directory.

With argument:

tools/repo-version/repo-version 'path/to/some/other/repository'

This prints the version string for the given repository path passed as argument.

This script is stored in Dæmon repository because I want a common place for it. We may even use it when doing nightly builds of the engine only, or things like that.

[slipher]

Without argument:

tools/git-version/git-version

This prints the version string for the repository the script is stored,

With argument:

tools/git-version/git-version 'path/to/some/other/repository'

This prints the version string for the given repository path passed as argument.

I would find the no-argument behavior surprising, expecting it to use the current directory rather than Daemon's directory.

[illwieckz]

I would find the no-argument behavior surprising, expecting it to use the current directory rather than Daemon's directory.

Hmm, right, this would also remove the need for a wrapper in Unvanquished repo.

[illwieckz]

I would find the no-argument behavior surprising, expecting it to use the current directory rather than Daemon's directory.

Now using the current directory.

[illwieckz]

My original code was purposed to also handles the cases when:

• the directory is not a Git repository yet
• there is no commit yet
• there is no tag yet

And to provide fallbacks to them.

The reason why I want that is that such permissivity is useful in Urcheon, and at some point I want this code to be exactly the same in Urcheon (I'll replace current Urcheon version code with this one, as it is more mature and less convoluted).

So I finally made the script erroring on Git errors, and made it not sending Git error message to /dev/null, by default. But I also added options to ignore and silence errors, so I can merge this back in Urcheon later.

We may want to check for the directory always being a repository in the Unvanquished release script, so having this code aborting on error and printing errors is fine. Same if one day we wire it into cmake to embed the result as internal game version string.

[slipher]

Why would you want to ignore/silence Git errors? It would seem the script can't do anything useful if Git doesn't work.

[illwieckz]

If Git doesn't work, and if the script is allowed to continue, it produces a version string using current date with same syntax (using 0 as fallback tag and commit).

By using similar code in Urcheon I can generate versioned dpkdir of a work-in-progress folder even before committing the very first changes, and the fallback version string is compatible with what will be produced once there is a git repository, and once there is a git tag.

Example:

• 0-20240408-050944+dirty: not a git repository yet, or git repository but no commit yet (date is world date).
• 0-20240408-051101-2bdab49: git repository with committed data but not tag (date is latest commit date).
• 0-20240408-051101-2bdab49+dirty: git repository with committed data but not tag, with uncommitted change (date is latest commit date).
• 0.54.1: git repository, tag is current commit.
• 0.54.1+dirty: git repository, tag is current commit, with uncommitted change.
• 0.54.1-20240408-051101-2bdab49: git repository, untagged commits since latest tag (date is latest commit date).
• 0.54.1-20240408-051101-2bdab49+dirty: git repository, untagged commits since latest tag, with uncommitted change (date is latest commit date).

[illwieckz]

I forgot to mention that “Git not working“ is assumed to mean “This directory is not a git repository”, not that Git is not installed. I may add a dedicated check for when Git is not installed at all (like by calling git -v).

[illwieckz]

The script now tests if Git is available and working. The --permissive option can only continue on the fallback compute if Git is failing because the directory is not a git repository yet (the reason why there is a fallback).

[illwieckz]

I renamed the tool to dir-version, because it prints a version string for a given directory, either being a git repository or not, either being a subdir of the repository or not.

I finally ported and improved the complete dirt detection code from Urcheon.

• It reports the version as dirty if there are uncommitted and untracked files in the directory.
• It doesn't report the version as dirty if the modified file is outside of the directory the script is computing a version from, even if the directory is from the same git repository.

The second feature means doing this dir-version Unvanquished/pkg/unvanquished_src.dpkdir doesn't report the version as dirty if there is a change outside of this directory, despite this directory being part of the same Unvanquished/ repository.

I added a --whole option to force looking for dirt in the whole repository the directory is part of. For example that will report dirt from Unvanquished/src and other places fromUnvanquished when doing dir-version Unvanquished/pkg/unvanquished_src.dpkdir.

[slipher]

Wouldn't it be easier to omit the git diff final argument instead of having to look up the repository root?

[slipher]

I think it is a bad default to depend on the current directory within the repository. What is the use case for that anyway?

[illwieckz]

Use case: https://github.com/UnvanquishedAssets/UnvanquishedTestAssets/tree/master/pkg

I can make it non-default, but I need the feature to be there.

[slipher]

OK, please make it non-default. The subdirectory use case seems second-class because tags are only made at the whole-repository scope, so it may be common to have different tags with identical content.

[illwieckz]

Behavior is reversed. By default it now uses the whole repository for computing dirtiness, but can only look for subdirectory dirtiness when using --local.

I'm not very happy with the local name, but current would be confusing as it may give the false idea it's for the folder the script is called from, not the given folder on command line. Suggestions welcome.

[slipher]

remove

[illwieckz]

Done.

[illwieckz]

Some things I'm still thinking about:

• I'm not entirely happy with the name, I may rename it git-dir-version/GitDirVersion.

I'm not happy with the original git-version/GitVersion because it was confusing it may provide the version of Git itself, but I'm not happy with repo-version/RepoVersion or dir-version/DirVersion because this is a tool to compute a Git-based version (with a non-Git optional fallback) for a source directory. Git is always required even when the directory is not tracked by Git (to know it's not tracked by Git).

• I will probably implement the integration of this new lib in some Urcheon branch before merging it there to make sure it works in Urcheon too to avoid some back and forth merge roundtrips.

• I may add an option to provide a custom/explicit Git binary path (may be useful when integrating things like CMake to make sure we use the same Git CMake uses).

[slipher]

I would name the tool version-string

[illwieckz]

I would name the tool version-string

Now renamed version-string.

[illwieckz]

Well, in fact I dislike version-string, as it doesn't tell the version string of what.

Actually my initial name of repo-version wasn't bad.

[illwieckz]

I renamed it repo-version back.

[slipher]

Should throw an exception if the return code is not 0 or 1. Same goes for tag subcommand

[slipher]

Just check if the output is nonempty instead of splitting it?

[slipher]

There's one more way for the repo to be dirty: having changes in the staging area.

[slipher]

Is permissive just for the non-repository case? If the script will be called repo-version, non-repository handling probably shouldn't be part of the script at all. If that case will be included in the script, it doesn't make sense to run 15 failing git commands. It should just check whether the target directory is in a repository and if not, use a separate code path. Interleaving it with the repository path makes the code hard to follow and means you have to disable error handling completely if you want to allow a non-repository.

[illwieckz]

A repository is a folder containing sources, being tracked by Git yet or not.

[slipher]

I think there is a way to check for all kinds of dirtiness at once: check for empty output of git status --porcelain. It produces outputs for regular diffs, staged diffs, and untracked non-ignored files.

[illwieckz]

This script is in a kind of identity crisis by the way. This tool is meant to be used by Dæmon's (and then Unvanquished's) cmake, by Urcheon (for any dpkdir), by our release scripts, I may start using it for NetRadiant as well.

Because we encourage scripts used by Dæmon's cmake to be in python, I rewrote it in Python. But then if it's written in Python, it better be a python module that can be imported straight by Urcheon.

So I made it a RepositoryVersionString.py module that can be executable as a script. The repo-version script is just a demo of how some can make an installable runner, but one can just rename RepositoryVersionString.py as repo-version and it works as well. I'll likely copy it with such repo-version name or so in the DaemonGitTools repository, while it will be imported as RepositoryVersionString.py in Urcheon.

[illwieckz]

The +dirty string is now only used for dirty Git repositories, versions from repositories not yet tracked by Git now use the +stray string. The need for supporting stray repositories comes from Urcheon: we want Urcheon to compute a meaningful version whatever the dpkdir being built.

[illwieckz]

Now that I figured out the “stray” terminology, it may allow us to rename the string differently.

For example, if we rename the --permissive option as --stray, then we may put the git string in the tool name.

So for example if we name it git-repo-version, it wouldn't be that wrong to do git-repo-version --stray for a non-git repository, while git-repo-version --permissive was telling nothing about the fact it may not print some git stuff.

[illwieckz]

Actually, looking at other names from DaemonGitTools like git-checkout-url and git-commit-modules, it looks better to name this tool git-compute-version.

It follows the usage of wording a verb after git, and this makes it obvious it is not meant to print the version of git, but that it uses git to compute a version.

Also, git version already exists and the man page is already named git-version, for printing the version of git itself.

[slipher]

See if you like this version. Mostly I improved error handling: now there is only one command where Git is allowed to return an error, and that's only if "allow stray" is set. (You may wish to change the fact that it's an error to not have Git installed in stray mode). Also:

• Changed version pattern from v[0-9].* to v[0-9]* because it's a glob not a regex.
• Got rid of an unnecessary show toplevel command.

#! /usr/bin/env python3

# Daemon BSD Source Code
# Copyright (c) 2024-2026, Daemon Developers
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#  * Redistributions of source code must retain the above copyright
#    notice, this list of conditions and the following disclaimer.
#  * Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#  * Neither the name of the <organization> nor the
#    names of its contributors may be used to endorse or promote products
#    derived from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT HOLDER> BE LIABLE FOR ANY
# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

import datetime
import os.path
import subprocess
import sys
import time

class GitComputeVersion():
	git_short_ref_length = 7

	def __init__(self, source_dir, allows_stray, is_quiet, is_local):
		if not os.path.isdir(source_dir):
			raise(ValueError, "not a directory")

		self.process_stderr = None

		if is_quiet:
			self.process_stderr = subprocess.DEVNULL

		self.is_local = is_local

		self.source_dir_realpath = os.path.realpath(source_dir)

		self.git_command_list = ["git", "-C", self.source_dir_realpath]

		# Test that Git is available and working.
		self.runGitCommand(["-v"])

		self.allows_stray = allows_stray

	def runGitCommand(self, command_list):
		command_list = self.git_command_list + command_list

		process = subprocess.run(command_list,
			stdout=subprocess.PIPE, stderr=self.process_stderr, text=True, check=True)

		return process.stdout.rstrip()

	def getDateString(self, timestamp):
		return datetime.datetime.fromtimestamp(timestamp, datetime.UTC).strftime('%Y%m%d-%H%M%S')

	def isGitDirty(self):
		if self.is_local:
			lookup_dir_arg = [self.source_dir_realpath]
		else:
			lookup_dir_arg = []

		git_status_porcelain_string = self.runGitCommand(
			["status", "--porcelain", "--"] + lookup_dir_arg)

		return git_status_porcelain_string != ""

	def getVersionString(self):
		# Fallback version string for stray repositories (not tracked by git yet).
		tag_string="0"
		date_string="-" + self.getDateString(time.time())
		ref_string=""
		dirt_string="+stray"

		# Git returns an error if the directory is not a Git repository or it has no
		# commits. It also may return an error if the repo is broken somehow. We fail
		# to distinguish these cases and assume it is not a Git repository.
		try:
			git_last_commit_string = self.runGitCommand(
				["rev-parse", "HEAD", "--"])
		except subprocess.CalledProcessError:
			if not self.allows_stray:
				raise
			return tag_string + date_string + ref_string + dirt_string

		# Now we know we must use a Git-based version string.
		dirt_string = ""

		# Git prints the current commit reference.
		git_last_commit_short_string = git_last_commit_string[:self.git_short_ref_length]
		ref_string = "-" + git_last_commit_short_string

		# Git prints the current commit date.
		git_last_commit_timestamp_string = self.runGitCommand(
			["log", "-1", "--pretty=format:%ct"])
		date_string = "-" + self.getDateString(int(git_last_commit_timestamp_string))

		# Git prints the most recent tag, or a commit hash if there is no tag at all.
		git_closest_tag_string = self.runGitCommand(
			["describe", "--always", "--tags", "--abbrev=0", "--match", "v[0-9]*"])

		# If a tag is found:
		if git_closest_tag_string[0] == "v":
			git_closest_tag_version_string = git_closest_tag_string[1:]
			tag_string = git_closest_tag_version_string

			# Git prints a version string that is equal to the most recent tag
			# if the most recent tag is on the current commit.
			git_describe_tag_string = self.runGitCommand(
				["describe", "--tags", "--match", "v[0-9]*"])
			git_describe_version_string = git_describe_tag_string[1:]

			if git_closest_tag_version_string == git_describe_version_string:
				# Do not write current commit reference and date in version
				# string if the tag is on the current commit.
				date_string = ""
				ref_string = ""

		if self.isGitDirty():
			# Write the dirty flag in version string if not everything in
			# the Git repository is properly committed.
			dirt_string = "+dirty"

		return tag_string + date_string + ref_string + dirt_string

def getVersionString(source_dir, allows_stray=False, is_quiet=False, is_local=False):
	return GitComputeVersion(source_dir, allows_stray, is_quiet, is_local).getVersionString()

def main():
	import argparse

	def existing_dir(path):
		if not os.path.isdir(path):
			raise argparse.ArgumentTypeError(f"{path} is not an existing directory")
		return path

	parser = argparse.ArgumentParser(description="Print repository version string")

	parser.add_argument("-s", "--stray", dest="allows_stray", help="allows stray repositories not tracked by Git", action="store_true")
	parser.add_argument("-q", "--quiet", dest="is_quiet", help="silence Git errors", action="store_true")
	parser.add_argument("-l", "--local", dest="is_local", help="look for dirtiness in given directory only, not in whole repository", action="store_true")
	parser.add_argument(dest="source_dir", nargs="?", metavar="DIRNAME", type=existing_dir, default=".", help="repository path")

	args = parser.parse_args()

	print(getVersionString(args.source_dir, allows_stray=args.allows_stray, is_quiet=args.is_quiet, is_local=args.is_local))

if __name__ == "__main__":
	main()

[illwieckz]

I pushed your changes, that seems to work, thanks.

[illwieckz]

We need to make sure Git is installed, because it's the only safe way to return the extected string, as we should only produce a stray string if that's not a repository, so the git tool is required to know it's stray.