Freenet Communication Primitives: Part 1, Files and Sites

Basic building blocks for communication in Freenet.

This is a guide to using Freenet [1] as backend for communication solutions - suitable for anything from filesharing over chat up to decentrally hosted game content like level-data. It uses the Python interface to Freenet [2] for its examples.

TheTim from Tim Moore, licensed under cc by

TheTim
from Tim Moore,
License: cc by [3].

This guide consists of several installments: Part 1 (this text) is about exchanging data, Part 2 [4] is about confidential communication and finding people and services without drowning in spam and Part 3 ties it all together by harnessing existing plugins which already include all the hard work which distinguishes a quick hack from a real-world system. Happy Hacking and welcome to Freenet, the forgotten cypherpunk paradise [5] where no one can watch you read!

1 Introduction

The immutable datastore in Freenet provides the basic structures for implementing distributed, pseudonymous, spam-resistant communication protocols. But until now there was no practically usable documentation how to use them. Every new developer had to find out about them by asking, speculating and second guessing the friendly source (also known as SGTFS).

We will implement the answers using pyFreenet. Get it from http://github.com/freenet/pyFreenet [6]

We will not go into special cases. For these have a look at the API-documentation of fcp.node.FCPNode().

1.1 Install pyFreenet

To follow the code examples in this article, install Python 2 with setuptools and then run

easy_install --user --egg pyFreenet==0.4.0

2 Sharing a File: The CHK (content hash key)

The first and simplest task is sharing a file. You all know how this works in torrents and file hosters: You generate a link and give that link to someone else.

To create that link, you have to know the exact content of the file beforehand.

import fcp
n = fcp.node.FCPNode()
key = n.put(data="Hello Friend!")
print key
n.shutdown()

Just share this key, and others can retrieve it. Use http://127.0.0.1:8888/ as prefix, and they can even click it - if they run Freenet on their local computer or have an SSH forward for port 8888.

The code above only returns once the file finished uploading. The Freenet Client Protocol (that’s what fcp stands for) however is asynchronous. When you pass async=True to n.put() or n.get(), you get a job object which gives you the result via job.wait().

To generate the key without actually uploading the file, use chkonly=True as argument to n.put().

Let’s test retrieving a file:

import fcp
n = fcp.node.FCPNode()
key = n.put(data="Hello Friend!")
mime, data, meta = n.get(key)
print data
n.shutdown()

This code anonymously uploads an invisible file into Freenet which can only be retrieved with the right key. Then it downloads the file from Freenet using the key and shows the data.

That the put and the get request happen from the same node is a mere implementation detail: They could be fired by total strangers on different sides of the globe and would still work the same. Even the performance would be similar.

Note: fcp.node.FCPNode() opens a connection to the Freenet node. You can have multiple of these connections at the same time, all tracking their own requests without interfering with each other. Just remember to call n.shutdown() on each of them to avoid getting ugly backtraces.

So that’s it. We can upload and download files, completely decentrally, anonymously and confidentially.

There’s just one caveat: We have to exchange the key. And to generate that key, we have to know the content of the file.

Let’s fix that.

3 Public/Private key publishing: The SSK (signed subspace key)

Our goal is to create a key where we can upload a file in the future. We can generate this key and tell someone else: Watch this space.

So we will generate a key, start to download from the key and insert the file to the key afterwards.

import fcp
n = fcp.node.FCPNode()
# we generate a key with the additional filename hello.
public, private = n.genkey(name="hello")
job = n.get(public, async=True)
n.put(uri=private, data="Hello Friend!")
mime, data, meta = job.wait()
print data
n.shutdown()

These 8 lines of code create a key which you could give to a friend. Your friend will start the download and when you get hold of that secret hello-file, you upload it and your friend gets it.

Hint: If you want to test whether the key you give is actually used, you can check the result of n.put(). It returns the key with which the data can be retrieved.

Using the .txt suffix makes Freenet use the mimetype text/plain. Without extension it will use application/octet-stream.

If you start downloading before you upload as we do here, you can trigger a delay of about half an hour due to overload protections (the mechanism is called “recently failed”).

Note that you can only write to a given key-filename combination once. If you try to write to it again, you’ll get conflicts – your second upload will in most cases just not work. You might recognize this from immutable datastructures (without the conflict stuff). Freenet is the immutable, distributed, public/private key database you’ve been phantasizing about when you had a few glasses too many during that long night. So best polish your functional programming skills. You’re going to use them on the level of practical communication.

3.1 short roundtrip time (speed hacks)

A SSK is a special type of key, and similar to inodes in a filesystem it can carry data. But if used in the default way, it will forward to a CHK: The file is salted and then inserted to a CHK which depends on the content and then some, ensuring that the key cannot be predicted from the data (this helps avoid some attacks against your anonymity).

When we want a fast round trip time, we can cut that. The condition is that your data plus filename is less than 1KiB after compression, the amount of data a SSK can hold. And we have to get rid of the metadata. And that means: With pyFreenet use the application/octet-stream mime type, because that’s the default one, so it is left out on upload. If you use raw access to FCP, omit Metadata.ContentType or set it to "". And insert single files (we did not yet cover uploading folders: You can do that, but they will forward to a CHK).

import fcp
n = fcp.node.FCPNode()
# we generate a key with the additional filename hello.
public, private = n.genkey(name="hello.txt")
job = n.get(public, async=True, realtime=True, priority=0)
n.put(uri=private, data="Hello Friend!", mimetype="application/octet-stream", realtime=True, priority=0)
mime, data, meta = job.wait()
print public
print data
n.shutdown()

To check whether we managed to avoid the metadata, we can use the KeyUtils plugin to analyze the key.

If it is right, when putting the key into the text field on the http://127.0.0.1:8888/KeyUtils/ [7] site, you’ll see something like this:

0000000: 4865 6C6C 6F20 4672 6965 6E64 21
         Hello Friend!

Also we want to use realtime mode (optimized for the webbrowser: reacting quickly but with low throughput) with a high priority.

Let’s look at the round trip time we achieve:

import time
import fcp
n = fcp.node.FCPNode()
# we generate two keys with the additional filename hello.
public1, private1 = n.genkey(name="hello1.txt")
public2, private2 = n.genkey(name="hello2.txt")
starttime = time.time()
job1 = n.get(public1, async=True, realtime=True, priority=1)
job2 = n.get(public2, async=True, realtime=True, priority=1)
n.put(uri=private1, data="Hello Friend!",
      mimetype="application/octet-stream",
      realtime=True, priority=1)
mime, data1, meta = job1.wait()
n.put(uri=private2, data="Hello Back!",
      mimetype="application/octet-stream",
      realtime=True, priority=1)
mime, data2, meta = job2.wait()
rtt = time.time() - starttime
n.shutdown()
print public1
print public2
print data1
print data2
print "RTT (seconds):", rtt

When I run this code, I get less than 80 seconds round trip time. Remember that we’re uploading two files anonymously into a decentralized network, discover them and then download them, and all that in serial. Less than a minute to detect an upload to known key.

90s is not instantaneous, but when looking at usual posting frequencies in IRC and other chat, it’s completely sufficient to implement a chat system. And in fact it’s how FLIP is implemented: IRC over Freenet.

Compare this to the performance when we do not use the short round trip time trick of avoiding the Metadata and using the realtime queue:

import time
import fcp
n = fcp.node.FCPNode()
# we generate two keys with the additional filename hello.
public1, private1 = n.genkey(name="hello1.txt")
public2, private2 = n.genkey(name="hello2.txt")
starttime = time.time()
job1 = n.get(public1, async=True)
job2 = n.get(public2, async=True)
n.put(uri=private1, data="Hello Friend!")
mime, data1, meta = job1.wait()
n.put(uri=private2, data="Hello Back!")
mime, data2, meta = job2.wait()
rtt = time.time() - starttime
n.shutdown()
print public1
print public2
print data1
print data2
print "RTT (seconds):", rtt

With 300 seconds (5 minutes), that’s more than 3x slower. So you see, if you have small messages and you care about latency, you want to do the latency hacks.

4 Upload Websites: SSK as directory

So now we can upload single files, but the links look a lot like what we see on websites: http://127.0.0.1:8888/folder/file. So can we just mirror a website? The answer is: Yes, definitely!

import fcp
n = fcp.node.FCPNode()
# We create a key with a directory name
public, private = n.genkey() # no filename: we need different ones
index = n.put(uri=private + "index.html",
      data='''<html>
  <head>
    <link rel="stylesheet" type="text/css" href="style.css">
    <title>First Site!</title></head>
  <body>Hello World!</body></html>''')
n.put(uri=private + "style.css", 
      data='body {color: red}\n')
print index
n.shutdown()

Now we can navigate to the key in the freenet web interface and look at our freshly uploaded website! The text is colored red, so it uses the stylesheet. We have files in Freenet which can reference each other by relative links.

4.1 Multiple directories below an SSK

So now we can create simple websites on an SSK. But here’s a catch: key/hello/hello.txt simply returns key/hello. What if we want multiple folders?

For this purpose, Freenet provides manifests instead of single files. Manifests are tarballs which include several files which are then downloaded together and which can include references to external files - named redirects. They can be uploaded as folders into the key. And in addition to these, there are quite a few other tricks. Most of them are used in freesitemgr which uses fcp/sitemgr.py.

But we want to learn how to do it ourselves, so let’s do a more primitive version manually via n.putdir():

import os
import tempfile

import fcp
n = fcp.node.FCPNode()
# we create a key again, but this time with a name: The folder of the
# site: We will upload it as a container.
public, private = n.genkey()
# now we create a directory
tempdir = tempfile.mkdtemp(prefix="freesite-")
with open(os.path.join(tempdir, "index.html"), "w") as f:
    f.write('''<html>
    <head>
    <link rel="stylesheet" type="text/css" href="style.css">
    <title>First Site!</title></head>
    <body>Hello World!</body></html>''')

with open(os.path.join(tempdir, "style.css"), "w") as f:
    f.write('body {color: red}\n')

uri = n.putdir(uri=private, dir=tempdir, name="hello", 
               filebyfile=True, allatonce=True, globalqueue=True)
print uri
n.shutdown()

That’s it. We just uploaded a folder into Freenet.

But now that it’s there, how do we upload a better version? As already said, files in Freenet are immutable. So what’s the best solution if we can’t update the data, but only upload new files? The obvious solution would be to just number the site.

And this is how it was done in the days of old. People uploaded hello-1, hello-2, hello-3 and so forth, and in hello-1 they linked to an image under hello-2. When visitors of hello-1 saw that the image loaded, they knew that there was a new version.

When more and more people adopted that, Freenet added core support: USKs, the updatable subspace keys.

We will come to that in the next part of this series: Service Discovery and Communication [4].

Anhang	Größe
thetim-tim_moore-flickr-cc_by-2471774514_8c9ed2a7e5_o-276x259.jpg [8]	19.79 KB

Freenet Communication Primitives: Part 2, Service Discovery and Communication

Basic building blocks for communication in Freenet.

Mirror,
Freenet Project,
License: GPL [9].

This guide consists of several installments: Part 1 [10] is about exchanging data, Part 2 [11] is about confidential communication and finding people and services without drowning in spam and Part 3 ties it all together by harnessing existing plugins which already include all the hard work which distinguishes a quick hack from a real-world system (this is currently a work in progress, implemented in babcom_cli [12] which provides real-world usable functionality).

Note: You need the current release of pyFreenet for the examples in this article (0.3.2). Get it from PyPI [13]:

# with setuptools
easy_install --user --egg pyFreenet==0.4.0
# or pip
pip install --user --egg pyFreenet==0.4.0

This is part 2: Service Discovery and Communication. It shows how to find new people, build secure communication channels and create community forums. Back when I contributed to Gnutella, this was the holy grail of many p2p researchers (I still remember the service discovery papers). Here we’ll build it in 300 lines of Python.

Welcome to Freenet, where no one can watch you read!

USK: The Updatable Subspace Key

USKs allow uploading increasing versions of a website into Freenet. Like numbered uploads from the previous article [10] they simply add a number to site, but they automate upload and discovery of new versions in roughly constant time (using Date Hints [14] and automatic checking for new versions), and they allow accessing a site as <key>/<name>/<minimal version>/ (never understimate the impact of convenience!).

With this, we only need a single link to provide an arbitrary number of files, and it is easy and fast to always get the most current version of a site. This is the ideal way to share a website in Freenet. Let’s do it practically.

import os
import tempfile

import fcp
n = fcp.node.FCPNode()
# we create a key again, but this time with a name: The folder of the
# site: We will upload it as a container.
public, private = n.genkey()
# now we create a directory
tempdir = tempfile.mkdtemp(prefix="freesite-")
with open(os.path.join(tempdir, "index.html"), "w") as f:
    f.write('''<html>
    <head>
    <link rel="stylesheet" type="text/css" href="style.css">
    <title>First Site!</title></head>
    <body>Hello World!</body></html>''')

with open(os.path.join(tempdir, "style.css"), "w") as f:
    f.write('body {color: red}\n')

uri = n.putdir(uri=private, dir=tempdir, name="hello",
               filebyfile=True, allatonce=True, globalqueue=True,
               usk=True)
print uri
n.shutdown()

But we still need to first share the public key, so we cannot just tell someone where to upload the files so we see them. Though if we were to share the private key, then someone else could upload there and we would see it in the public key. We could not be sure who uploaded there, but at least we would get the files. Maybe we could even derive both keys from a single value… and naturally we can. This is called a KSK [15] (old description [16]).

KSK: Upload a file to a password

KSKs allow uploading a file to a pre-determined password. The file will only be detectable for those who know the password, so we have effortless, invisible, password protected files [17].

import fcp
import uuid # avoid spamming the global namespace

n = fcp.node.FCPNode()
_uuid = str(uuid.uuid1())
key = "KSK@" + _uuid
n.put(uri=key, data="Hello World!",
      Global=True, persistence="forever",
      realtime=True, priority=1)
print key
print n.get(key)[1]
n.shutdown()

Note: We’re now writing a communication protocol, so we’ll always use realtime mode. Be aware, though, that realtime is rate limited. If you use it for large amounts of data, other nodes will slow down your requests to preserve quick reaction of the realtime queue for all (other) Freenet users.

Note: Global=True and

persistence="forever"

allows telling Freenet to upload some data and then shutting down the script. Use async=True and waituntilsent=True to just start the upload. When the function returns you can safely exit from the script and let Freenet upload the file in the background - if necessary it will even keep uploading over restarts. And yes, Capitcalized Global looks crazy. For pyFreenet that choice is sane (though not beautiful), because Global gets used directly as parameter in the Freenet Client Protocol (FCP). This is the case for many of the function arguments. In putdir() there’s a globalqueue parameter which also sets persistence. That should become part of the put() API, but isn’t yet. There are lots of places where the pyFreenet is sane, but not beautiful. It seems like that’s its secret how it could keep working from 2008 till 2014 with almost no maintenance

For our purposes the main feature of KSKs is that we can tell someone to upload to an arbitrary phrase and then download that.

If we add a number, we can even hand out a password to multiple people and tell them to just upload to the first unused version. This is called the KSK queue.

KSK queue: Share files by uploading to a password

The KSK queue used to be the mechanism of choice to find new posts in forums, until spammers proved that real anonymity means total freedom to spam: they burned down the Frost Forum System. But we’ll build this, since it provides a basic building block for the spam-resistant system used in Freenet today.

Let’s just do it in code (descriptions are in the comments):

import fcp
import uuid # avoid spamming the global namespace

n = fcp.node.FCPNode()
_uuid = str(uuid.uuid1())
print "Hey, this is the password:", _uuid
# someone else used it before us
for number in range(2):
    key = "KSK@" + _uuid + "-" + str(number)
    n.put(uri=key, data="Hello World!", 
          Global=True, persistence="forever",
          realtime=True, priority=1,
          timeout=360) # 6 minutes
# we test for a free slot
for number in range(4):
  key = "KSK@" + _uuid + "-" + str(number)
  try:
    n.get(key, 
          realtime=True, priority=1, 
          timeout=60)
  except fcp.node.FCPNodeTimeout:
    break
# and write there
n.put(uri=key, data="Hello World!",
      Global=True, persistence="forever",
      realtime=True, priority=1,
      timeout=360) # 6 minutes
print key
print n.get(key)[1]
n.shutdown()

Note that currently a colliding put – uploading where someone else uploaded before – simply stalls forever instead of failing. This is a bug in pyFreenet. We work around it by giving an explicit timeout.

But it’s clear how this can be spammed.

And it might already become obvious how this can be avoided.

KSK queue with CAPTCHA

Let’s assume I do not tell you a password. Instead I tell you where to find a riddle. The solution to that riddle is the password. Now only those who are able to solve riddles can upload there. And each riddle can be used only once. This restricts automated spamming, because it requires an activity of which we hope that only humans can do it reliably.

In the clearweb this is known as CAPTCHA. For the examples in this guide a plain text version is much easier.

import fcp
import uuid # avoid spamming the global namespace

n = fcp.node.FCPNode()
_uuid = str(uuid.uuid1())
_uuid2 = str(uuid.uuid1())
riddlekey = "KSK@" + _uuid
riddle =  """
What goes on four legs in the morning,                          
two legs at noon, and three legs in the                         
evening?
A <answer>
"""
# The ancient riddle of the sphinx
n.put(uri=riddlekey, data="""To reach me, answer this riddle.

%s

Upload your file to %s-<answer>
""" % (riddle, _uuid2),
      Global=True, persistence="forever",
      realtime=True, priority=1)

print n.get(riddlekey, realtime=True, priority=1)[1]
answer = "human"
print "answer:", answer
answerkey = "KSK@" + _uuid2 + "-%s" % answer

n.put(uri=answerkey, data="Hey, it's me!",
      Global=True, persistence="forever",
      realtime=True, priority=1)

print n.get(answerkey, realtime=True, priority=1)[1]
n.shutdown()

Now we have fully decentralized, spam-resistant, anonymous communication.

Let me repeat that: fully decentralized, spam-resistant, anonymous communication.

The need to solve a riddle everytime we want to write is not really convenient, but it provides the core of the feature we need. Everything we now add just makes this more convenient and makes it scale for many-to-many communication.

(originally I wanted to use the Hobbit riddles for this, but I switched to the sphinx riddle to avoid the swamp of multinational (and especially german) quoting restrictions)

Convenience: KSK queue with CAPTCHA via USK to reference a USK

The first step to improve this is getting rid of the requirement to solve a riddle every single time we write to a person. The second is to automatically update the list of riddles.

For the first, we simply upload a public USK key instead of the message. That gives a potentially constant stream of messages.

For the second, we upload the riddles to a USK instead of to a KSK. We pass out this USK instead of a password. Let’s realize this.

To make this easier, let’s use names. Alice wants to contact Bob. Bob gave her his USK. The answer-uuid we’ll call namespace.

import fcp
import uuid # avoid spamming the global namespace
import time # to check the timing

tstart = time.time()
def elapsed_time():
    return time.time() - tstart


n = fcp.node.FCPNode()

bob_public, bob_private = n.genkey(usk=True, name="riddles")
alice_to_bob_public, alice_to_bob_private = n.genkey(usk=True, name="messages")
namespace_bob = str(uuid.uuid1())
riddle =  """
What goes on four legs in the morning,                          
two legs at noon, and three legs in the                         
evening?
A <answer>
"""
print "prepared:", elapsed_time()
# Bob uploads the ancient riddle of the sphinx
put_riddle = n.put(uri=bob_private,
                   data="""To reach me, answer this riddle.

%s

Upload your key to %s-<answer>
""" % (riddle, namespace_bob),
                   Global=True, persistence="forever",
                   realtime=True, priority=1, async=True,
                   IgnoreUSKDatehints="true") # speed hack for USKs.

riddlekey = bob_public
print "riddlekey:", riddlekey
print "time:", elapsed_time()
# Bob shares the riddlekey. We're set up.

# Alice can insert the message before telling Bob about it.
put_first_message = n.put(uri=alice_to_bob_private,
                          data="Hey Bob, it's me, Alice!",
                          Global=True, persistence="forever",
                          realtime=True, priority=1, async=True,
                          IgnoreUSKDatehints="true")

print "riddle:", n.get(riddlekey, realtime=True, priority=1, followRedirect=True)[1]
print "time:", elapsed_time()

answer = "human"
print "answer:", answer
answerkey = "KSK@" + namespace_bob + "-%s" % answer
put_answer = n.put(uri=answerkey, data=alice_to_bob_public,
                   Global=True, persistence="forever",
                   realtime=True, priority=1, async=True)

print ":", elapsed_time()
# Bob gets the messagekey and uses it to retrieve the message from Alice

# Due to details in the insert process (i.e. ensuring that the file is
# accessible), the upload does not need to be completed for Bob to be
# able to get it. We just try to get it.
messagekey_alice_to_bob = n.get(answerkey, realtime=True, priority=1)[1]

print "message:", n.get(uri=messagekey_alice_to_bob, realtime=True, priority=1,
                        followRedirect=True, # get the new version
                        )[1]

print "time:", elapsed_time()
# that's it. Now Alice can upload further messages which Bob will see.

# Bob starts listening for a more recent message. Note that this does
# not guarantee that he will see all messages.
def next_usk_version(uri):
    elements = uri.split("/")
    elements[2] = str(abs(int(elements[2])) + 1)
    # USK@.../name/N+1/...
    return "/".join(elements)

next_message_from_alice = n.get(
    uri=next_usk_version(messagekey_alice_to_bob),
    realtime=True, priority=1, async=True,
    followRedirect=True) # get the new version

print "time:", elapsed_time()
# Alice uploads the next version.
put_second_message = n.put(uri=next_usk_version(alice_to_bob_private),
                           data="Me again!",
                           Global=True, persistence="forever",
                           realtime=True, priority=1,
                           IgnoreUSKDatehints="true",
                           async=True)

# Bob sees it.
print "second message:", next_message_from_alice.wait()[1]
print "time:", elapsed_time()

print "waiting for inserts to finish"
put_riddle.wait()
put_answer.wait()
put_first_message.wait()
put_second_message.wait()
print "time:", elapsed_time()

n.shutdown()

From start to end this takes less than 2 minutes minutes, and now Alice can send Bob messages with roughly one minute delay.

So now we set up a convenient communication channel. Since Alice already knows Bobs key, Bob could simply publish a bob-to-alice public key there, and if both publish GnuPG keys, these keys can be hidden from others: Upload not the plain key, but encrypt the key to Bob, and Bob could encrypt his bob-to-alice key using the GnuPG key from Alice. By regularly sending themselves new public keys, they could even establish perfect forward secrecy. I won’t implement that here, because when we get to the third part of this series, we will simply use the Freemail and Web of Trust plugin which already provide these features.

This gives us convenient, fully decentralized, spam-resistant, anonymous communication channels. Setting up a communication channel to a known person requires solving one riddle (in a real setting likely a CAPTCHA, or a password-prompt), and then the channel persists.

Note: To speed up these tests, I added another speed hack: IgnoreUSKDatehints. That turns off Date Hints [14], so discovering new versions will no longer be constant in the number of intermediate versions. For our messaging system that does not hurt, since we don’t have many intermediate messages we want to skip. For websites however, that could lead your visitors to see several old versions before they finally get the most current version. So be careful with this hack - just like you should with the other speed hacks.

But if we want to reach many people, we have to solve one riddle per person, which just doesn’t scale. To fix this, we can publish a list of all people we trust to be real people. Let’s do that.

Many-to-many: KSK->CAPTCHA->USK->USK which is linked in the original USK

To enable (public) many-to-many communication, we propagate the information that we believe that someone isn’t a spammer and add a blacklist to get rid of people who suddenly start to spam.

The big change with this scheme is that there is two-step authentication: Something expensive (solving a riddle) gets you seen by a few people, and if you then contribute constructively in a social context, they mark you as non-spammer and you get seen by more people.

The clever part about that scheme is that socializing is actually no cost to honest users (that’s why we use things like Sone or FMS), while it is a cost to attackers.

Let’s take Alice and Bob again, but add Carol. First Bob introduces himself to Alice, then Carol introduces herself to Alice. Thanks to propagating the riddle-information, Carol can directly write to Bob, without first solving a riddle. Scaling up that means that you only need to prove a single time that you are no spammer (or rather: not disruptive) if you want to enter a community.

To make it easier to follow, we will implement this with a bit of abstraction: People have a private key, can introduce themselves and publish lists of messages. Also they keep a public list of known people and a list of people they see as spammers who want to disrupt communication.

I got a bit carried away while implementing this, but please bear with me: It’ll work hard to make it this fun.

The finished program is available as alice_bob_carol.py [18]. Just download and run it with python alice_bob_carol.py.

Let’s start with the minimal structure for any pyFreenet using program:

import fcp

n = fcp.node.FCPNode() # for debugging add verbosity=5

<<body>>

n.shutdown()

The body contains the definitions of a person with different actors, an update step (as simplification I use global stepwise updates) as well as the setup of the communication. Finally we need an event loop to run the system.

<<preparation>>

<<person>>

<<update>>

<<setup>>

<<event_loop>>

We start with some imports – and a bit of fun :)

import uuid
import random
try:
    import chatterbot # let's get a real conversation :)
    # https://github.com/guntherc/ChatterBot/wiki/Quick-Start
    # get with `pip install --user chatterbot`
    irc_loguri = "USK@Dtz9FjDPmOxiT54Wjt7JwMJKWaqSOS-UGw4miINEvtg,cuIx2THw7G7cVyh9PuvNiHa1e9BvNmmfTcbQ7llXh2Q,AQACAAE/irclogs/1337/"
    print "Getting the latest IRC log as base for the chatterbot"
    IRC_LOGLINES = n.get(uri=irc_loguri, realtime=True, priority=1, followRedirect=True)[1].splitlines()
    import re # what follows is an evil hack, but what the heck :)
    p = re.compile(r'<.*?>')
    q = re.compile(r'&.*?;')
    IRC_LOGLINES = [q.sub('', p.sub('', str(unicode(i.strip(), errors="ignore"))))
                    for i in IRC_LOGLINES]
    IRC_LOGLINES = [i[:-5] for i in IRC_LOGLINES # skip the time (last 5 letters)
                    if (i[:-5] and # skip empty
                        not "spam" in i # do not trigger spam-marking
                    )][7:] # skip header 
except ImportError:
    chatterbot = None

The real code begins with some helper functions – essentially data definition.

def get_usk_namespace(key, name, version=0):
    """Get a USK key with the given namespace (foldername)."""
    return "U" + key[1:] + name + "/" + str(version) + "/"

def extract_raw_from_usk(key):
    """Get an SSK key as used to identify a person from an arbitrary USK."""
    return "S" + (key[1:]+"/").split("/")[0] + "/"

def deserialize_keylist(keys_data):
    """Parse a known file to get a list of keys. Reverse: serialize_keylist."""
    return [i for i in keys_data.split("\n") if i]

def serialize_keylist(keys_list):
    """Serialize the known keys into a text file. Reverse: parse_known."""
    return "\n".join(keys_list)

Now we can define a person. The person is the primary actor. To keep everything contained, I use a class with some helper functions.

class Person(object):
    def __init__(self, myname, mymessage):
        self.name = myname
        self.message = mymessage
        self.introduced = False
        self.public_key, self.private_key = n.genkey()
        print self.name, "uses key", self.public_key
        # we need a list of versions for the different keys
        self.versions = {}
        for name in ["messages", "riddles", "known", "spammers"]:
            self.versions[name] = -1 # does not exist yet
        # and sets of answers, watched riddle-answer keys, known people and spammers.
        # We use sets for these, because we only need membership-tests and iteration.
        # The answers contain KSKs, the others the raw SSK of the person.
        # watched contains all persons whose messages we read.
        self.lists = {}
        for name in ["answers", "watched", "known", "spammers", "knowntocheck"]:
            self.lists[name] = set()
        # running requests per name, used for making all persons update asynchronously
        self.jobs = {}
        # and just for fun: get real conversations. Needs chatterbot and IRC_LOGLINES.
        # this is a bit slow to start, but fun. 
        try:
            self.chatbot = chatterbot.ChatBot(self.name)
            self.chatbot.train(IRC_LOGLINES)
        except:
            self.chatbot = None


    def public_usk(self, name, version=0):
        """Get the public usk of type name."""
        return get_usk_namespace(self.public_key, name, version)
    def private_usk(self, name, version=0):
        """Get the private usk of type name."""
        return get_usk_namespace(self.private_key, name, version)

    def put(self, key, data):
        """Insert the data asynchronously to the key. This is just a helper to
avoid typing the realtime arguments over and over again.

        :returns: a job object. To get the public key, use job.wait(60)."""
        return n.put(uri=key, data=data, async=True,
                     Global=True, persistence="forever",
                     realtime=True, priority=1,
                     IgnoreUSKDatehints="true")

    def get(self, key):
        """Retrieve the data asynchronously to the key. This is just a helper to
avoid typing the realtime arguments over and over again.

        :returns: a job object. To get the public key, use job.wait(60)."""
        return n.get(uri=key, async=True,
                     realtime=True, priority=1,
                     IgnoreUSKDatehints="true",
                     followRedirect=True)

    def introduce_to_start(self, other_public_key):
        """Introduce self to the other by solving a riddle and uploading the messages USK."""
        riddlekey = get_usk_namespace(other_public_key, "riddles", "-1") # -1 means the latest version
        try:
            self.jobs["getriddle"].append(self.get(riddlekey))
        except KeyError:
            self.jobs["getriddle"] = [self.get(riddlekey)]

    def introduce_start(self):
        """Select a person and start a job to get a riddle."""
        known = list(self.lists["known"])
        if known: # introduce to a random person to minimize
                  # the chance of collisions
            k = random.choice(known)
            self.introduce_to_start(k)

    def introduce_process(self):
        """Get and process the riddle data."""
        for job in self.jobs.get("getriddle", [])[:]:
            if job.isComplete():
                try:
                    riddle = job.wait()[1]
                except Exception as e: # try again next time
                    print self.name, "getting the riddle from", job.uri, "failed with", e
                    return
                self.jobs["getriddle"].remove(job)
                answerkey = self.solve_riddle(riddle)
                messagekey = self.public_usk("messages")
                try:
                    self.jobs["answerriddle"].append(self.put(answerkey, messagekey))
                except KeyError:
                    self.jobs["answerriddle"] = [self.put(answerkey, messagekey)]

    def introduce_finalize(self):
        """Check whether the riddle answer was inserted successfully."""
        for job in self.jobs.get("answerriddle", [])[:]:
            if job.isComplete():
                try:
                    job.wait()
                    self.jobs["answerriddle"].remove(job)
                    self.introduced = True
                except Exception as e: # try again next time
                    print self.name, "inserting the riddle-answer failed with", e
                    return

    def new_riddle(self):
        """Create and upload a new riddle."""
        answerkey = "KSK@" + str(uuid.uuid1()) + "-answered"
        self.lists["answers"].add(answerkey)
        self.versions["riddles"] += 1
        next_riddle_key = self.private_usk("riddles", self.versions["riddles"])
        self.put(next_riddle_key, answerkey)


    def solve_riddle(self, riddle):
        """Get the key for the given riddle. In this example we make it easy:
The riddle is the key. For a real system, this needs user interaction.
        """
        return riddle

    def update_info(self):
        for name in ["known", "spammers"]:
            data = serialize_keylist(self.lists[name])
            self.versions[name] += 1
            key = self.private_usk(name, version=self.versions[name])
            self.put(key, data)

    def publish(self, data):
        self.versions["messages"] += 1
        messagekey = self.private_usk("messages", version=self.versions["messages"])
        print self.name, "published a message:", data
        self.put(messagekey, data)

    def check_network_start(self):
        """start all network checks."""
        # first cancel all running jobs which will be replaced here.
        for name in ["answers", "watched", "known", "knowntocheck", "spammers"]:
            for job in self.jobs.get(name, []):
                job.cancel()
        # start jobs for checking answers, for checking all known people and for checking all messagelists for new messages.
        for name in ["answers"]:
            self.jobs[name] = [self.get(i) for i in self.lists[name]]
        for name in ["watched"]:
            self.jobs["messages"] = [self.get(get_usk_namespace(i, "messages")) for i in self.lists[name]]
        self.jobs["spammers"] = []
        for name in ["known", "knowntocheck"]:
            # find new nodes
            self.jobs[name] = [self.get(get_usk_namespace(i, "known")) for i in self.lists[name]]
            # register new nodes marked as spammers
            self.jobs["spammers"].extend([self.get(get_usk_namespace(i, "spammers")) for i in self.lists[name]])

    def process_network_results(self):
        """wait for completion of all network checks and process the results."""
        for kind, jobs in self.jobs.items():
            for job in jobs:
                if not kind in ["getriddle", "answerriddle"]:
                    try:
                        res = job.wait(60)[1]
                        self.handle(res, kind, job)
                    except:
                        continue

    def handle(self, result, kind, job):
        """Handle a successful job of type kind."""
        # travel the known nodes to find new ones
        if kind in ["known", "knowntocheck"]:
            for k in deserialize_keylist(result):
                if (not k in self.lists["spammers"] and
                    not k in self.lists["known"] and
                    not k == self.public_key):
                    self.lists["knowntocheck"].add(k)
                    self.lists["watched"].add(k)
                    print self.name, "found and started to watch", k
        # read introductions
        elif kind in ["answers"]:
            self.lists[kind].remove(job.uri) # no longer need to watch this riddle
            k = extract_raw_from_usk(result)
            if not k in self.lists["spammers"]:
                self.lists["watched"].add(k)
                print self.name, "discovered", k, "through a solved riddle"
        # remove found spammers
        elif kind in ["spammers"]:
            for k in deserialize_keylist(result):
                if not result in self.lists["known"]:
                    self.lists["watched"].remove(result)
        # check all messages for spam
        elif kind in ["messages"]:
            k = extract_raw_from_usk(job.uri)
            if not "spam" in result:
                if not k == self.public_key:
                    print self.name, "read a message:", result
                    self.chat(result) # just for fun :)
                    if not k in self.lists["known"]:
                        self.lists["known"].add(k)
                        self.update_info()
                        print self.name, "marked", k, "as known person"
            else:
                self.lists["watched"].remove(k)
                if not k in self.lists["spammers"]:
                    self.lists["spammers"].add(k)
                    self.update_info()
                    print self.name, "marked", k, "as spammer"


    def chat(self, message):
        if self.chatbot and not "spam" in self.message:
            msg = message[message.index(":")+1:-10].strip() # remove name and step
            self.message = self.name + ": " + self.chatbot.get_response(msg)

# some helper functions; the closest equivalent to structure definition
<<helper_functions>>

Note that nothing in here depends on running these from the same program. All communication between persons is done purely over Freenet. The only requirement is that there is a bootstrap key: One person known to all new users. This person could be anonymous, and even with this simple code there could be multiple bootstrap keys. In freenet we call these people “seeds”. They are the seeds from which the community grows. As soon as someone besides the seed adds a person as known, the seed is no longer needed to keep the communication going.

The spam detection implementation is pretty naive: It trusts people to mark others as spammers. In a real system, there will be disputes about what constitutes spam and the system needs to show who marks whom as spammer, so users can decide to stop trusting the spam notices from someone when they disagree. As example for a real-life system, the Web of Trust plugin uses trust ratings between -100 and 100 and calculates a score from the ratings of all trusted people to decide how much to trust people who are not rated explicitly by the user.

With this in place, we need the update system to be able to step through the simulation. We have a list of people who check keys of known other people.

We first start all checks for all people quasi-simultaneously and then check the results in serial to avoid long wait times from high latency. Freenet can check many keys simultaneously, but serial checking is slow.

people = []

def update(step):
    for p in people:
        if not p.introduced:
            p.introduce_start()
    for p in people:
        p.check_network_start()
    for p in people:
        if p.message:
            p.publish(p.name + ": " + p.message + "   (step=%s)" % step)
        p.new_riddle()
    for p in people:
        if not p.introduced:
            p.introduce_process()
    for p in people:
        p.process_network_results()
    for p in people:
        if not p.introduced:
            p.introduce_finalize()

So that’s the update tasks - not really rocket science thanks to the fleshed out Persons. Only two things remain: Setting up the scene and actually running it.

For setup: We have Alice, Bob and Carol. Lets also add Chuck who wants to prevent the others from communicating by flooding them with spam.

def gen_person(name):
    try:
        return Person(myname=name, mymessage=random.choice(IRC_LOGLINES))
    except:
        return Person(myname=name, mymessage="Hi, it's me!")

# start with alice
alice = gen_person("Alice")
people.append(alice)

# happy, friendly people
for name in ["Bob", "Carol"]:
    p = gen_person(name)
    people.append(p)

# and Chuck
p = Person(myname="Chuck", mymessage="spam")
people.append(p)

# All people know Alice (except for Alice).
for p in people:
    if p == alice:
        continue
    p.lists["known"].add(alice.public_key)
    p.lists["watched"].add(alice.public_key)

# upload the first version of the spammer and known lists
for p in people:
    p.update_info()

That’s it. The stage is set, let the trouble begin :)

We don’t need a while loop here, since we just want to know whether the system works. So the event loop is pretty simple: Just call the update function a few times.

for i in range(6):
    update(step=i)

That’s it. We have spam-resistant message-channels and community discussions. Now we could go on and implement more algorithms on this scheme, like the turn-based games specification [19] (ever wanted to play against truly anonymous competitors?), Fritter [20] (can you guess from its name what it is? :)), a truly privacy respecting dropbox or an anonymizing, censoriship resistant, self-hosting backend for a digital market [21] like (the in 2023 long defunct) OpenBazaar.

But that would go far beyond the goal of this article – which is to give you, my readers, the tools to create the next big thing by harnessing the capabilities of Freenet.

These capabilities have been there for years, but hidden beneath non-existing [22] and outdated documentation, misleading claims of being in alpha-stage even though Freenet has been used in what amounts to production for over a decade and, not to forget, the ever-recurring, ever-damning suggestion to SGTFS (second-guess the friendly source). As written in Forgotten Cypherpunk Paradise [23], Freenet already solved many problems which researchers only begin to tackle now, but there are reasons why it was almost forgotten. With this series I intend fix some of them and start moving Freenet documentation towards the utopian vision laid out in Teach, Don’t Tell [24]. It’s up to you to decide whether I succeeded. If I did, it will show up as a tiny contribution to the utilities and works of art and vision you create.

Note that this is not fast (i.e. enough for blogging but not enough for chat). We can make it faster by going back to SSKs instead of USKs with their additional logic for finding the newest version in O(1), but for USK there are very cheap methods to get notified of new versions for large numbers of keys (subscribing) which are used by more advanced tools like the Web of Trust and the Sone plugin, so this would be an optimization we would have to revert later. With these methods, Sone reaches round trip times of 5-15 minutes despite using large uploads.

Also since this uses Freenet as backend, it scales up: If Alice, Bob, Carol und Chuck used different computers instead of running on my single node, their communication would actually be faster, and if they called in all their alphabet and unicode friends, the system would still run fast. We’re harvesting part of the payoff from using a fully distributed backend :)

And with that, this installment ends. You can now implement really cool stuff using Freenet. In the next article I’ll describe how to avoid doing this stuff myself by interfacing with existing plugins. Naturally I could have done that from the start, but then how could I have explained the Freenet communication primitives these plugins use? :)

If you don’t want to wait, have a look at how Infocalypse [25] uses wot [26] to implement github-like access [27] with user/repo, interfaces [28] with Freemail [29] to realize truly anonymous pull-requests from the command line and builds [30] on FMS [31] to provide automated updates of a DVCS wiki over Freenet [32].

Happy Hacking!

PS: You might ask “What is missing?”. You might have a nagging feeling that something we do every day isn’t in there. And you’re right. It’s scalable search. Or rather: scalable, spam- and censorship-resistant search. Scalable search would be Gnutella. Spam-resistance would be Credence [33] on the social graph (the people you communicate with). Censorship-resistant is unsolved – even Google fails there. But seeing that Facebook just overtook Google as the main source of traffic [34], we might not actually need fully global search. Together with the cheap and easy update notifications in Freenet (via USKs), a social recommendation and bookmark-sharing system should make scalable search over Freenet possible. And until then there’s always the decentralized YaCy [35] search engine which has been shown [36] to be capable of crawling Freenet. Also there are the Library and Spider plugins, but they need some love to work well. Also there are the Library and Spider plugins, but they need some love to work well.

PPS: You can download the final example as alice_bob_carol.py [37]