Mercurial vs. Bazaar speedtest clone and log (unscientific)

I just did a test with the provided Python 2.x repos from the DVCS PEP [14] for Python to check the performance of Bazaar and Mercurial.

(this is a slightly changed version of a mail posted to the mercurial list: http://selenic.com/pipermail/mercurial/2008-November/022199.html [15] )

All these tests are done only once with some mostly constant load, so they don't qualitfy as scientific tests, but they give a good impressing of the differences between Bazaar (bzr) and Mercurial (hg).

Versions:
- Bazaar 1.5
- Mercurial 1.0.2

These are the ones which are marked as stable in my Gentoo tree (amd64).

First test: initial cloning from the web.

With repositories containing the same changesets (roughly, since bzr tracks dir name changes) Mercurial is about 9 times faster than Bazaar.

$ time bzr branch http://code.python.org/python/trunk [16]
No handlers could be found for logger "bzr"
[13868] 2008-11-05 11:36:44.358 INFO: Branched 40626 revision(s).
Branched 40626 revision(s).

real 24m0.446s
user 12m47.623s
sys 0m15.842s

$ time hg clone http://code.python.org/hg/trunk/ [17] python-hg-trunk
requesting all changes
adding changesets
adding manifests
adding file changes
added 40556 changesets with 86253 changes to 8166 files
updating working directory
3922 files updated, 0 files merged, 0 files removed, 0 files unresolved

real 2m40.237s
user 1m34.721s
sys 0m13.027s

>>> (24.0 + 0.44 / 60 ) / ( 2 + 40.237 / 60 )
8.989434400294563

And hg only uses about half as much space as bzr for the repository data (with bzr 1.5 that is - that was the stable version in my Gentoo tree as of 2008-10-31):

$ du -hs python-bzr-trunk/.bzr python-hg-trunk/.hg
214M python-bzr-trunk/.bzr
111M python-hg-trunk/.hg

Second test: a full log.

A full bzr log takes about 2.5 times as long as a full hg log in the cloned python 2.x repositories (so hg log is about 2.5 times faster than bzr log):

$ time bzr log
...
real 1m10.258s
user 0m37.236s
sys 0m2.250s

$ time hg log
...
real 0m27.886s
user 0m12.737s
sys 0m0.482s

>>> (1 + 10.258/60) / (27.886 / 60)
2.519472136555978

As I understand it, the new revlog in C should give Mercurial another nice speedup here (am I right in that?).

Third test: local clones.

A local clone in Mercurial is about 11 times faster than a local clone in bzr.

$ time bzr branch python-bzr-trunk/ python-bzr-trunk2
...
real 11m36.265s
user 8m38.400s
sys 0m11.145s

$ time hg clone python-hg-trunk python-hg-trunk2
...
real 0m59.759s
user 0m23.321s
sys 0m5.455s

A second run (backwards, with hot filesystem caches) gave even stronger results:
$ rm -r clone python-hg-trunk
$ time hg clone python-hg-trunk2 python-hg-trunk
real 0m38.779s
user 0m23.394s
sys 0m4.635s

With bzr I first cloned backwards to heat up the filesystem caches and then cloned again:
$ rm -r python-bzr-trunk
$ bzr branch python-bzr-trunk2 python-bzr-trunk
$ rm -r python-bzr-trunk2
$ time bzr branch python-bzr-trunk python-bzr-trunk2
...
real 12m8.374s
user 8m44.747s
sys 0m10.782s

Result: bzr is unusable for quick local clones of not-so-small projects.

Sidenote:
$ #cold copy of .hg
$ time cp -r python-hg-trunk/.hg tmp-hg
real 0m53.981s
user 0m0.077s
sys 0m3.486s

# hot copy of .bzr
$ time cp -r python-bzr-trunk/.bzr tmp-bzr
real 0m25.738s
user 0m0.012s
sys 0m1.368s

# hot copy of .hg
$ rm -r tmp-hg
$ time cp -r python-hg-trunk/.hg tmp-hg
real 0m14.702s
user 0m0.085s
sys 0m6.241s

To alleviate some doubts about system load: this is what top gave while I did the first local clone of the bzr repo:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23202 arne 21 1 359m 255m 3412 R 70.1 12.7 1:39.33 bzr

CPU load due to bzr moved between 60% and 80% while cloning.

The general load on the system while running bzr commands was about the same as while running hg commands (I compiled stuff with niceness 11 = low priority).

Result: So Mercurial is far faster than bzr, while being a lot more newbie friendly than git (especially when the newbies are former svn users - Python currently uses svn).

Mercurial vs. Bazaar speedtest clone and log - update: hg 1.1 vs. bzr 1.10

I repeatet my test [18] with the provided Python 2.x repos from the DVCS PEP [14] for Python to check the performance of Bazaar and Mercurial.

All these tests are done only once with some mostly constant load, so they don't qualify as scientific tests, but they give a good impression of the differences between Bazaar (bzr) and Mercurial (hg).

Versions:
- Bazaar 1.10
- Mercurial 1.1

This comparision should be fair since Bazaar 1.10 is more recent, but Mercurial 1.1 is a major release.

You can do all these tests yourself using the script from the hg-vs-bzr-unscientific repository:

hg clone http://www.bitbucket.org/ArneBab/hg-vs-bzr-speedtest-unscientific/ [19]
cd hg-vs-bzr-unscientific
./test.sh

Results

First test: Initial cloning from the web.

With repositories containing the same changesets (roughly, since bzr tracks dir name changes and hg doesn't) Mercurial is more than 6 times as fast as Bazaar. But there might be a better way to serve bzr repositories, so this isn't perfectly decisive. But since local cloning in Mercurial is 6.7 times faster, the reason for the higher speed is likely to be in the client.

Second test: Repository size

A Mercurial repository needs about half the space of a Bazaar repository, even though they contain roughly the same data. Manual repacking in Bazaar doesn't change this size significantly (213M instead of 214M).

Third test: A full log.

Here Mercurial is about 2.2 times as fast as Bazaar: 13s instead of 30s.

Fourth test: Local clones

Local cloning in Mercurial is about 6.7 times faster than in Bazaar: about 38s instead of 4min 14s.

Fifth test: Local clones with shared repository and forced hardlinks (bzr)

As suggested in the Mercurial Mailinglist, I reran the local cloning test for Bazaar using a shared repository and forced hardlinks. With this Bazaar needed just 4.5s for a local clone, without hardlinks about 12s, so by using shared repositories the local cloning becomes quite fast in Bazaar (as long as you restrict yourself to the shared repository).

Data

This is a clean run of the script plus two reruns of the last test: hot copy of .bzr and .hg, since there was an error in the script.


$ ./test.sh
bzr --version
No handlers could be found for logger "bzr"
Bazaar (bzr) 1.10
Python interpreter: /usr/bin/python 2.5.2
Python standard library: /usr/lib64/python2.5
bzrlib: /usr/lib64/python2.5/site-packages/bzrlib
Bazaar configuration: /home/arne/.bazaar
Bazaar log file: /home/arne/.bzr.log

Copyright 2005, 2006, 2007, 2008 Canonical Ltd.
http://bazaar-vcs.org/ [20]

bzr comes with ABSOLUTELY NO WARRANTY. bzr is free software, and
you may use, modify and redistribute it under the terms of the GNU
General Public License version 2 or later.

hg --version
Mercurial Distributed SCM (version 1.1)

Copyright (C) 2005-2008 Matt Mackall and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

initial clone bzr
time bzr branch http://code.python.org/python/trunk [16] python-bzr-trunk
No handlers could be found for logger "bzr"
[18778] 2008-12-13 12:39:35.629 INFO: Branched 40761 revision(s).
Branched 40761 revision(s).

real 8m52.962s
user 5m52.733s
sys 0m10.519s

initial clone hg
time hg clone http://code.python.org/hg/trunk/ [17] python-hg-trunk
requesting all changes
adding changesets
adding manifests
adding file changes
added 40690 changesets with 86495 changes to 8167 files
updating working directory
3921 files updated, 0 files merged, 0 files removed, 0 files unresolved

real 1m23.699s
user 1m0.614s
sys 0m9.797s

comparing repository sizes
du -hs python-bzr-trunk/.bzr python-hg-trunk/.hg
214M python-bzr-trunk/.bzr
111M python-hg-trunk/.hg

time for a full bzr log
time bzr log >/dev/null
No handlers could be found for logger "bzr"

real 0m29.207s
user 0m28.022s
sys 0m0.378s

time for a full hg log
time hg log >/dev/null

real 0m12.907s
user 0m12.385s
sys 0m0.244s

bzr local clone
time bzr branch python-bzr-trunk/ python-bzr-trunk2
No handlers could be found for logger "bzr"
| [=============================================== ] Copying content texts 3/5^[18796] 2008-12-13 12:45:55.641 INFO: Branched 40761 revision(s).
Branched 40761 revision(s).

real 4m14.102s
user 3m23.507s
sys 0m6.860s

hg local clone
time hg clone python-hg-trunk python-hg-trunk2
updating working directory
3921 files updated, 0 files merged, 0 files removed, 0 files unresolved

real 0m37.642s
user 0m21.634s
sys 0m5.216s

clone with hot filesystem buffers, hg
rm -r python-hg-trunk
time hg clone python-hg-trunk2 python-hg-trunk
updating working directory
3921 files updated, 0 files merged, 0 files removed, 0 files unresolved

real 0m32.744s
user 0m21.820s
sys 0m4.431s

clone with hot filesystem buffers, bzr
rm -r python-bzr-trunk
bzr branch python-bzr-trunk2 python-bzr-trunk
No handlers could be found for logger "bzr"
[18802] 2008-12-13 12:51:30.879 INFO: Branched 40761 revision(s).
Branched 40761 revision(s).
rm -r python-bzr-trunk2
time bzr branch python-bzr-trunk python-bzr-trunk2
No handlers could be found for logger "bzr"
[18816] 2008-12-13 12:55:16.111 INFO: Branched 40761 revision(s).
Branched 40761 revision(s).

real 3m44.461s
user 3m22.856s
sys 0m5.401s

hot copy of .bzr
cp -r python-bzr-trunk/.bzr /tmp/tmp-bzr
rm -r tmp-bzr
rm: Entfernen von „tmp-bzr“ nicht möglich: Datei oder Verzeichnis nicht gefunden
time cp -r python-bzr-trunk/.bzr /tmp/tmp-bzr

real 0m2.979s
user 0m0.012s
sys 0m1.271s

hot copy of .hg
cp -r python-hg-trunk/.hg /tmp/tmp-hg
rm -r tmp-hg
rm: Entfernen von „tmp-hg“ nicht möglich: Datei oder Verzeichnis nicht gefunden
time cp -r python-hg-trunk/.hg /tmp/tmp-hg

real 0m8.213s
user 0m0.079s
sys 0m2.476s
rm -r python-bzr-trunk python-bzr-trunk2 python-hg-trunk python-hg-trunk2 /tmp/tmp-hg /tmp/tmp-bzr

# testing the hot copy of .bzr and .hg again, but this time really deleting /tmp/tmp-*

$ rm -r /tmp/tmp-* ; time cp -r python-hg-trunk/.hg /tmp/tmp-hg ; time cp -r python-bzr-trunk/.bzr /tmp/tmp-bzr ;

real 0m3.183s
user 0m0.079s
sys 0m2.250s

real 0m2.139s
user 0m0.014s
sys 0m1.593s

$ rm -r /tmp/tmp-* ; time cp -r python-hg-trunk/.hg /tmp/tmp-hg ; time cp -r python-bzr-trunk/.bzr /tmp/tmp-bzr ;

real 0m3.921s
user 0m0.059s
sys 0m2.691s

real 0m3.350s
user 0m0.013s
sys 0m1.632s

An additional test with a bzr optimized workflow gives the following results:


bzr optimized local cloning test using a shared repo.
time bzr init-repo bzr
No handlers could be found for logger "bzr"
Shared repository with trees (format: rich-root-pack)
Location:
shared repository: bzr

real 0m0.442s
user 0m0.328s
sys 0m0.069s
cd bzr
time bzr branch http://code.python.org/python/trunk [16] python-bzr-trunk
No handlers could be found for logger "bzr"
[26234] 2008-12-15 11:55:14.502 INFO: Branched 40773 revision(s).
Branched 40773 revision(s).

real 10m39.557s
user 5m28.317s
sys 0m9.968s

check the size after repacking
bzr pack
No handlers could be found for logger "bzr"
rm .bzr/repositry/obsolete_packs/*
rm: Entfernen von „.bzr/repositry/obsolete_packs/*“ nicht möglich: Datei oder Verzeichnis nicht gefunden
du -hs .bzr
213M .bzr
time bzr branch python-bzr-trunk python-bzr-trunk2 --hardlink
No handlers could be found for logger "bzr"
[26879] 2008-12-15 13:07:07.532 INFO: Branched 40773 revision(s).
Branched 40773 revision(s).

real 0m4.546s
user 0m3.682s
sys 0m0.737s
cd ..
rm -r python-* /tmp/tmp-hg /tmp/tmp-bzr bzr/

Notes

To alleviate some doubts about system load, I stopped all other operations besides this test, konqueror and kmail. During the inital bzr clone top gave me the following (on my system everythign I start from an X session has niceness 1 so X doesn't bog down core processes):

top - 12:31:07 up 1 day, 2:36, 5 users, load average: 0.70, 0.99, 0.95
Tasks: 146 total, 3 running, 143 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 2.0%sy, 97.7%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
Mem: 2060384k total, 1146140k used, 914244k free, 103944k buffers
Swap: 5111796k total, 4048k used, 5107748k free, 112700k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18778 arne 21 1 209m 89m 3960 R 94.7 4.4 0:14.24 bzr
15922 arne 21 1 416m 134m 42m S 3.3 6.7 4:40.40 konqueror
7591 root 21 1 651m 310m 6568 S 1.0 15.4 21:54.84 X

Bazaar once managed to get a local clone in a shared repository with forced hardlinks done in just 2.598 seconds, but every subsequent run gave results in the range of 4.3s to 4.8s, so that speed wasn't repeatable and I used a medium value for evaluation (about 4.5s).

This definitely shows the danger of unscientific tests: You don't know how unsure the results are - the difference in this case was about 50%, with the variance yet to be determined.

Result

So Mercurial is far faster than bzr when it comes to cloning (except when using bzr with a shared repository), especially for the initial clone, and a good deal faster in generating the log, while being a lot more newbie friendly than git (especially when the newbies are former svn users - Python currently uses svn).

Update: Python decided to switch to Mercurial [21].

Mercurial vs. GIT, Leistungstest für lokale Operationen

English: A speed test for Mercurial and git (in german). -> Links to other tests [23]. Update: I wrote a new test for use in server applications, since they are the only place where a difference of 30-60ms really matters: hg vs. git for server applications [1].

Ich habe ein paar Testsskripte geschrieben, um die lokale Leistung von Mercurial und GIT vergleichen zu können.

Update: Ich habe einen neuen Test für die Nutzung in Serverprogrammen geschrieben, weil dort der Unterschied um 30-60ms wirklich etwas ausmacht: hg vs. git for server applications [1].

Was ich wissen wollte ist, wie schnell die Systeme jeweils bei lokalen commits sind (andere Operationen prüfe ich noch nicht), und wie sich ihre Leistung ändert, wenn ich die Anzahl der Dateien oder die Art der Änderungen variiere.

Meine Anregung dafür war der Mercurial-vs-Bazaar [24] Test

Ich nutze selbst fast ausschließlich Mercurial, daher kann es sein, dass ich ein paar Optimierungsmöglichkeiten von git übersehen habe. Ich habe bei Mercurial allerdings auch keine genutzt.

Meine Skripte dazu sind alle in einem Mercurial repository im Netz verfügbar: - http://freehg.org/u/ArneBab/python_tests/

Die Testergebnisse genügen nicht wissenschaftlichen Standards, weil

• Mein Rechner teilweise Last hatte, und
• Ich keine Standardabweichung der Mittelwerte hier berechne, so dass nicht sicher ist, wie ungenau die Daten sind.

Bei den Tests war git in den meisten Fällen schneller als Mercurial, vor allem mit wenigen Dateien.

Bei vielen Dateien kam die Geschwindigkeit von Mercurial näher an git heran.

Die Tests, die ich gemacht habe sind:

• init: Einfach nur ein Repository initialisieren
• initial_commit: Ein Repository erstellen und die Dateien hinzufügen.
• commit_after_append_small: Eine kurze Zeile anhängen (eine Fließkommazahl als String) und committen.
• commit_after_append_of_many_lines: An jede Datei eine Anzahl Zeilen gleich der Anzahl von Dateien anhängen.
• commit_after_append_of_one_long_line: An jede Datei eine Zeile anhängen, die für jede Datei 2 Fließkommazahlen enthält.

Bisher hat sich git in Tests mit 2 Dateien als um den Faktor 4 bis 5 schneller erwiesen.

Mit 200 Dateien war es noch um den Faktor 2.5 bis 4 schneller.

Gleichzeitig haben die Daten im Mercurial Ordner nur etwa halb so viel Platz eingenommen, wie in git (ohne repack).

Mit 2000 Dateien war git nur noch um den Faktor 2 bis 2,6 schneller. Das git repo hat dabei knapp 100 MiB belegt hat und das Mercurial repo nur knapp 80 MiB.

Struktur der Ergebnisse: = Für 2 Dateien =

init: - git: 0.01423661708831787 s - Mercurial: 0.13704051971435546 s

initial_commit: - git: 0.086095404624938962s - Mercurial: 0.32350056171417235 s

commit_after_append_small: - git: 0.084930634498596197 - Mercurial: 0.31660361289978028

commit_after_append_of_many_lines: - git: 0.086296844482421878 - Mercurial: 0.31294920444488528

commit_after_append_of_one_long_line: - git: 0.08749730587005615 - Mercurial: 0.32180678844451904

= Für 2000 Dateien =

[git, hg], time in seconds, mean value.

init [0.063668489456176758, 0.14365851879119873] - Factor: 2.25635192571

initial_commit [1.5152170658111572, 3.1502538919448853] - Factor: 2.07907762064

commit_after_append_small [1.4661256074905396, 3.7556029558181763] - Factor: 2.56158335727

commit_after_append_of_many_lines [16.113877415657043, 42.500172019004822] - Factor: 2.63748885031

commit_after_append_of_one_long_line [15.808947086334229, 38.482058048248291] - Factor: 2.43419487953

Anmerkungen:

• Ein Problem von GIT ist, dass ich Repositories regelmäßig manuell packen muss. So wie ich mich kenne, würde ich das prinzipiell vergessen und mich dann wundern, warum meine Platte voll ist.

Versionsverwaltungen testen - Mercurial (hg) und Bazaar (bzr)

Ich habe gerade einen sehr schönen Vergleich der Versionsverwaltungssysteme Mercurial und Bazaar gefunden.

Für alle, die sich nicht durch einen langen Artikel wühlen wolllen:

Mercurial ist in den meisten Tests 2-5x schneller als Bazaar.

Der volle Artikel:

http://sayspy.blogspot.com/2006/11/bazaar-vs-mercurial-unscientific.html [4]

Falls ihr gerne eine Übersetzung der kritischen Teile hättet, schreibt es bitte.

Nebenbei: Ich nutze Mercurial, um meine Programme und meine Texte (Rollenspielzeug, Lieder, Gedichte und Geschichten) zu verwalten, und es ist ein tolles Gefühl jederzeit zurückgehen oder einfach schauen zu können, was ich wann gemacht habe.

Es gibt mir ein Gefühl der Sicherheit.

Egal, was ich ändere, solange meine Platten bleiben, kann ich es jederzeit rückgängig machen.

Und eine Sicherheitskopie anlegen ist so einfach wie

hg clone . {Zielordner}

Sie zu aktualisieren und nur rüberzukopieren ist ein einfaches

hg push {Zielordner in dem eine Sicherheitskopie ist}

--- Nachtrag ---

Ich habe jetzt auch eigene Tests durchgeführt und viele andere gesammelt:

Leistungstests und Vergleiche, DVCS: Mercurial (HG), Git, Bazaar(bzr), ... [23]