Epic Games announces Lore version control system

Git is bad at such volumes of binary assets, textures, models, sounds, etc.

[1] https://github.com/debarshibasak/assets

Git-annex ?

In fairness, p4 predates most of the VCSes we consider "modern", so I empathize with a lot of the underlying architecture decisions. However, it has and continues to utterly fail at improving at a reasonable pace.

For example:

  - p4 tracks file metadata of client workspaces on the server (sync'ed locally, opened for edit, file revision, etc) and uses this as the basis to avoid doing unneeded work.  If this becomes desync'ed, a reconcile or force sync must be used.  A reconcile can take hours, potentially days; it tries do detect file moves by default, so likely at least O(c^n) for some c>1. I have never personally seen a default reconcile operation _complete_ over any modestly large game code base, and in practice, people accumulate a litany of workarounds and scripts to fix this for themselves.

  - Scripting p4 is a nightmare. Documentation is poor, schemas do not exist, and all the language-specific libraries are just thin wrappers over its C++ API.

  - By default, p4 "helps" you with text files by "correcting" line endings on sync or even converting between encodings. This works until you have a mixed-OS environment, and discover a part of the pipechain that _must_ have a certain style.  There are various levers to pull to make this better, but I've yet to find something fool proof.

  - By default, p4 keeps flies read-only, only unlocking them when explicitly marked as being edited.  This means, to avoid having to do this manually, every tool you use needs to be p4-aware.  Or, you can turn this off, and choose to contend reconcile instead. (See above)

  - Branching a modest game project, with, say, Unreal source code, can take hours.  And this is the quick version where you ask the server to simply create new metadata, with no file transfer to a client.

  - p4 is licensed by the user-account. Every user entity in p4 not intended exclusively for performing backups and maintenance operations counts toward this, including users required to integrate with other services.  Plus, often times, these integration users must have admin access to be useful. The security posture is horrific.

If perforce is the best there is out there for large binary asset management, then there is a blue ocean worth of potential improvement for git.

Perforce is a piece of crap, a relic of the 20th century that must die in a fiery inferno.

The objects are referenced by trees. A tree is just a directory.

The trees are then referenced by commits and/or tags into a DAG with named pointers into various parts of it (which are your branch and tag references):

https://git-scm.com/book/en/v2/Git-Internals-Git-Objects

Because it would be terribly in-efficient to have a bunch of loose objects, git periodically groups them together into packs. To save space, the objects are compressed against one another (delta compression) within the packs.

https://git-scm.com/docs/git-pack-objects

https://github.com/git/git/blob/master/Documentation/technic...

When pushing or pulling, the git transfer protocol basically enumerates what objects each side has so that it only needs to transfer the difference. On top of that, it delta compresses the objects on each side that aren't already grouped into packs against each other to save space.

https://github.com/git/git/blob/master/Documentation/technic...

Because git is an open-source project written by nerds, it shows you all of this information. Feel free to ignore it!

But if you really want to know, it's all documented both in the git book and git documentation directory, both linked above.

(Caveat: I'm working from memory and surely got some detail at least slightly wrong.)

The git support for new employees drops basically to zero.

    gritzko@spot ~/beagle $ be get
     19:07  get ?#0ac49e6a
     16:58  post ?0ac49e6a#POST-018 put:/post: banner on stdout
     19:07  new beagle/test/be-post-put-banner.sh
     19:07  upd dog/INDEX.md
     ...more stuff...
     19:07  del test/post/01-bare-msg/01.put.err.txt
     19:07  del test/post/01-bare-msg/02.post.err.txt
     19:07  get abc?4222dfab

I’d be a bit worried if git didn’t heave that particular contented sigh when I ask it to push

Not saying Lore's approach is bad, but sometimes "worse is better".

Having this output is useful for when it does break and you need to copy-paste your terminal output to someone who does understand it to explain it to you or explain how to fix it, but you're correct that 90% of this is effectively debug output that is almost never useful or relevant.

In most cases I would say they should remove any output that isn't necessary, but given that some git operations can be extremely long-running it's beneficial to have some kind of output so the user knows what's going on.

Case in point, this is the output I get when I try to clone the Linux kernel:

    Cloning into 'linux'...
    remote: Enumerating objects: 11623749, done.
    remote: Counting objects: 100% (396/396), done.
    remote: Compressing objects: 100% (189/189), done.
    Receiving objects:   1% (181683/11623749), 90.11 MiB | 19.17 MiB/s

Also, I assume you're not but in case anyone is interested in the answers to these questions:

> What the hell is "delta compression"?

The 'delta' is the difference between one thing and another - usually one version of a file and another. Git does some fancy thinking to figure out which files are which other files but with changes, so that it can store just the changes from one version to the next.

For example, a 100 KB file where we only changed 500 bytes ten times would be 1000 KB, but because Git can store the deltas from one to the next it can be 100 KB (the original) plus ten 500-byte changes, for a total of about 105 KB.

> Why do I care how many threads it's using?

Because it directly affects how fast the process works; using 16 threads is 16x faster than using 1 thread (on average). Git automatically detects how many CPU threads are available and uses as many as it can, but if it's being very slow you might look and see 'oh, right, this VM only has two CPUs'.

> What is an 'object' and what does it mean when it's 'local'?

Uh, this one is deliberately vague I guess. An object is a thing that Git keeps track of. Usually this will refer to a blob, which is 'a bunch of bytes that make up a file', or a 'tree', which is a list of files and other trees - basically a directory structure, or a commit's information, but anything that Git keeps track of is an object.

Local just means that you already have a copy on your system. in the 'remote:' line you see output from the other end (where you're pushing to), so that's the server saying that it's using the files it already has.

> What does 'pack-reused' mean?

To be efficient, Git can take all the 'objects' and smush them into one big packfile (rather than having to keep track of hundreds or thousands of separate files). Since Git keeps track of files based on their contents, two identical files are just stored as one copy referenced twice, so it's possible that the file that you're pushing already exists in a pack file and can just be reused rather than having to push another copy.

That already warrants different storage architectures.

Don't be too hard on Lore.

I'm not just picking nits here. And this is not cynicism.

so there you go.

> Open-source the desktop client so the community can build on its full graphical experience, not just download it. An early desktop client already exists as a binary download, but it isn't open source yet — it depends on some proprietary components, including Epic's internal design system. We're working to make all of it available in the open so that the client can ship as source alongside the rest of Lore. Lore is an open project, so it is important that the desktop client — which will be one of the main ways many people will interact with Lore — is also fully open so that the community is free to review, extend, and shape it.

they do say they will open source it, but who knows:

"It isn’t open source yet—it currently depends on some proprietary components, including Epic’s internal design system—but we’re committed to open-sourcing it in the future"

EDIT: and then an example for the merge stuff I couldn't find while typing before: https://mergiraf.org/ and HN discussion a few years ago: https://news.ycombinator.com/item?id=42093756

Anyway it's probably fine software and I am genuinely going to give it a shot for a usecase I have involving large image files. But the LLM-generated docs don't inspire confidence.

[1] https://epicgames.github.io/lore/explanation/system-design/

[2] They literally have a section header "10.1 Revision state as a 320-byte fragment". The byte size isn't even relevant in the code as an implementation detail, much less belongs in a design doc. No one read this doc before publishing it.

On the flip side, I expect the project itself will be workable - well, assuming they're actually using it themselves! UE is a big pile of Stuff on its own, and Fortnite must have god knows how much additional crap in there, so if this is (or will be) their replacement for Perforce internally, then it'll be getting a good deal of testing. (If they're just chucking it over the wall, though... well, sheesh, you first...)

(Perforce is the standard thing for games, and pretty well it works too, and hopefully this will deliver it a well-needed kick. It was sold to private equity about 10 years ago, and it feels like they've been coasting ever since. (Perhaps users in other sectors are happier though?))

Lore is meant for situations where your repository is going to contain gigabytes of binary files, such as art assets for games. Git is technically great at everything but that, and even the external solutions for that situation still kind of suck.

I am not sure it is much of an issue to implement a centralised system to solve domain-specific problems that are caused by or are in the context of inevitable centralisation.

I can see a bunch of media companies liking this, maybe even small design shops. Quite a lot of CMS and change control work for media is not much more than asset management; version control is often a bit of an afterthought.

Web interfaces to this might end up interesting for non-profits and charities with significant media outreach.

> 3.2 Explicit non-goals¶

> Peer-to-peer decentralization. Lore is centralized by design. Two clients communicate through the remote, not directly.