Freenet Communication Primitives: Part 1, Files and Sites

The immutable datastore in Freenet provides the basic structures for implementing distributed, pseudonymous, spam-resistant communication protocols. But until now there was no practically usable documentation how to use them. Every new developer had to find out about them by asking, speculating and second guessing the friendly source (also known as SGTFS).

We will implement the answers using pyFreenet. Get it from http://github.com/freenet/pyFreenet

We will not go into special cases. For these have a look at the API-documentation of fcp.node.FCPNode().

The first and simplest task is sharing a file. You all know how this works in torrents and file hosters: You generate a link and give that link to someone else.

To create that link, you have to know the exact content of the file beforehand.

import fcp
n = fcp.node.FCPNode()
key = n.put(data="Hello Friend!")
print key
n.shutdown()

Just share this key, and others can retrieve it. Use http://127.0.0.1:8888/ as prefix, and they can even click it - if they run Freenet on their local computer or have an SSH forward for port 8888.

The code above only returns once the file finished uploading. The Freenet Client Protocol (that’s what fcp stands for) however is asynchronous. When you pass async=True to n.put() or n.get(), you get a job object which gives you the result via job.wait().

To generate the key without actually uploading the file, use chkonly=True as argument to n.put().

Let’s test retrieving a file:

import fcp
n = fcp.node.FCPNode()
key = n.put(data="Hello Friend!")
mime, data, meta = n.get(key)
print data
n.shutdown()

This code anonymously uploads an invisible file into Freenet which can only be retrieved with the right key. Then it downloads the file from Freenet using the key and shows the data.

That the put and the get request happen from the same node is a mere implementation detail: They could be fired by total strangers on different sides of the globe and would still work the same. Even the performance would be similar.

Note: fcp.node.FCPNode() opens a connection to the Freenet node. You can have multiple of these connections at the same time, all tracking their own requests without interfering with each other. Just remember to call n.shutdown() on each of them to avoid getting ugly backtraces.

So that’s it. We can upload and download files, completely decentrally, anonymously and confidentially.

There’s just one caveat: We have to exchange the key. And to generate that key, we have to know the content of the file.

Let’s fix that.

Our goal is to create a key where we can upload a file in the future. We can generate this key and tell someone else: Watch this space.

So we will generate a key, start to download from the key and insert the file to the key afterwards.

import fcp
n = fcp.node.FCPNode()
# we generate a key with the additional filename hello.
public, private = n.genkey(name="hello")
job = n.get(public, async=True)
n.put(uri=private, data="Hello Friend!")
mime, data, meta = job.wait()
print data
n.shutdown()

These 8 lines of code create a key which you could give to a friend. Your friend will start the download and when you get hold of that secret hello-file, you upload it and your friend gets it.

Hint: If you want to test whether the key you give is actually used, you can check the result of n.put(). It returns the key with which the data can be retrieved.

Using the .txt suffix makes Freenet use the mimetype text/plain. Without extension it will use application/octet-stream.

If you start downloading before you upload as we do here, you can trigger a delay of about half an hour due to overload protections (the mechanism is called “recently failed”).

Note that you can only write to a given key-filename combination once. If you try to write to it again, you’ll get conflicts – your second upload will in most cases just not work. You might recognize this from immutable datastructures (without the conflict stuff). Freenet is the immutable, distributed, public/private key database you’ve been phantasizing about when you had a few glasses too many during that long night. So best polish your functional programming skills. You’re going to use them on the level of practical communication.

So now we can upload single files, but the links look a lot like what we see on websites: http://127.0.0.1:8888/folder/file. So can we just mirror a website? The answer is: Yes, definitely!

import fcp
n = fcp.node.FCPNode()
# We create a key with a directory name
public, private = n.genkey() # no filename: we need different ones
index = n.put(uri=private + "index.html",
      data='''<html>
  <head>
    <link rel="stylesheet" type="text/css" href="style.css">
    <title>First Site!</title></head>
  <body>Hello World!</body></html>''')
n.put(uri=private + "style.css", 
      data='body {color: red}\n')
print index
n.shutdown()

Now we can navigate to the key in the freenet web interface and look at our freshly uploaded website! The text is colored red, so it uses the stylesheet. We have files in Freenet which can reference each other by relative links.