Each Syndie instance operates with its own archive of messages, storing
the data extracted from the signed messages
within its local database (by default stored under
$dataRoot/db/), an archive of those signed messages in the
$dataRoot/archive/ file hierarchy, and an archive of locally
created but not yet distributed messages in the
$dataRoot/outbound/ file hierarchy. The contents of
$dataRoot/archive/ can be wiped out without any loss of
functionality, though doing so prevents the Syndie instance from sharing
the authenticated messages with other people.
Within the $dataRoot/archive/ directory is a periodically
rebuilt file shared-index.dat, containing a listing of files
in the archive that should be reachable, some metadata surrounding each of
those channel and message files, and some general metadata for the archive
itself. This file is only accessed when serving up the archive on the web -
never for those not publishing an archive. In general, it contain a subset
of the available files underneath the archive directory, so as to hide some
anonymity-related facts (such as delaying the listing of locally created
messages until they have been published elsewhere, etc).
The shared-index.dat file uses the following format:
$archiveFlags // 2 bytes - see below for meaning
$archiveAdminChannel// 4 bytes - index into the channels where the archive admin posts
$numAltURIs // 1 byte
for (i = 0; i < $numAltURIs; i++)
$url // UTF-8 encoded SyndieURI of some other archives
$numChannels // 4 bytes
for (i = 0; i < $numChannels; i++)
$channelHash // 32 byte SHA256 value
$channelEdition // 8 byte unsigned integer
$channelFlags // 1 byte - see below
$numMessages // 4 bytes
for (i = 0; i < $numMessages; i++)
$messageId // 8 byte unsigned integer
$scopeChannel // 4 byte unsigned integer - index into $numChannels above
$targetChannel // 4 byte unsigned integer - index into $numChannels above
$msgFlags // 1 byte - see below
$archiveFlags:
first byte:
high bit: if true, the archive wants to receive only recent messages
bit 6 : if true, the archive wants to receive only messages in known channels
bit 5 : if true, the archive will accept passphrase encrypted messages
bit 4 : if true, the archive will accept private reply messages
bit 3 : if true, the archive requires a passphrase to post to it
bit 2-0 : along with bits 7 and 6 of the second byte, configures the hashcash requirements to post
second byte:
high bit: see above
bit 6 : see above
bits 5-4: index rebuild frequency
0 = hourly, 1 = every 6 hours, 2 = every 12 hours, 3 = every 24 hours
bits 3-0: maximum message size the archive will receive, in kilobytes, bitshifted
0 = 1KB, 1 = 2KB, 2 = 4KB, ..., 14 = 8MB, 15 = 16MB
$channelFlags:
high bit: if true, the channel info is passphrase encrypted
bit 6: if true, the channel info publishes its read key
bit 5: if true, the channel info has been updated "recently"
bit 4: if true, the archive wants to receive updates to this channel's metadata
bit 3: if true, the archive wants to receive new messages in this channel
bits 2-0: undefined
$messageFlags:
high bit: if true, the message is passphrase encrypted
bit 6: if true, the message is encrypted to the forum owner
bit 5: if true, the message publishes its read key
bit 4: if true, the archive considers this message "new"
bits 3-0: message size shift, in kilobytes, starting at 4KB
0 = 4KB, 1 = 8KB, 2 = 16KB, ..., 14 = 32MB, 15 = 64MB
Individual posts are found under
$dataRoot/archive/$scope/$messageId.syndie, and metadata under
$dataRoot/archive/$scope/meta.syndie. The externally referenced
posts are found under their original scope path, not the targetted channel
path - $dataRoot/archive/$scopeHash/$messageId.syndie and not
$dataRoot/archive/$channelHash/$messageId.syndie
Given the simple file-based archive hierarchy and index, running a public
Syndie archive is trivial - simply publish your $dataRoot/archive/
in a webserver and tell people the URL. They will then be able to load up
their Syndie instance and use the getindex and fetch
commands to pull posts from the archive into their local Syndie instance.
Syndie also includes a built-in HTTP server (run with the
httpserv command), which both serves up the
content and accepts HTTP posts from others (if allowed), limiting the data accessible
to those messages and channel metadata files published in the
shared-index.dat file.
The server accepts posts (uploaded through schedule and put), writing them to a temporary directory under the syndie root, and running bulkimport on them after they've all been read.
The HTTP post received is not a mime-encoded post, but a fairly simple concatenation of all of the data to be uploaded (with metadata messages first):
POST /import.cgi\r\n
Content-length: $total\r\n
\r\n
$headerSize // 2 bytes
$header // $headerSize bytes, for authorization/etc
foreach (msg)
$msgFlags // 1 byte. 0x0 for normal posts, 0x1 for metadata posts
$msgSize // 4 bytes
$msgData // $msgSize bytes