interstellar-distributed-file-system

The goal of this project is to develop a distributed file system suitable for use in unreliable network environments, such as in space. Code running on each node of the network (space stations, satellites, rovers, etc.) manages the sending/receiving of data via Delay-Tolerant Network (DTN) protocols, such that each node maintains a consistent view of the data available in the network. This means that although the data is physically stored on satellites/space stations scattered throughout space, each object has easy access to the data as if it was saved locally (with an access delay, of course). To help mitigate slow access speeds and limited bandwidth, the project will also incorporate controls that allow data to be automatically downloaded when available, or ignored if the data isn't of interest. There are other extensions possible as well, such as providing fault tolerance by automatically backing up data on different nodes of the network.

To produce a working rapid-prototype, we've broken the project into pieces that we'd like to accomplish as follows:

1. Get a test network environment up and running using the tools provided by one of the DTN libraries. This will allow us to get a feel for what parameters are involved, look at what tools already exist for the library and what APIs are exposed, and also have a place to test our prototype code.

2. Develop a basic application that looks for files placed in a given directory, sends them to a pre-configured server using the DTN library, and deletes the local copies afterward. This application is pretty cool in that it would provide a way for data, images, and other mission-data to be easily aggregated, queued, and sent off when the network is available. This would be especially useful for space craft to earth communications. This code will be needed by the more complex application, so it's a good stepping stone!

3. Add sending and receiving into the same application, making it possible to have two ends synchronize its copy of the file system with the others. This application could be useful in the same case as above, but where you may also want to upload new data to the space craft in addition to retrieving data from it. This code will be reusable as well!

4. Optimize the application to minimize needless data transfers. Instead of sending all file data to all ends, just send a list of available data and provide a way to have other applications selectively get the file data. We're thinking this will come in the form of an API where each application can express interest in individual files, which will cause the file system to pull it when possible.

5. Continue optimizing and support scenarios involving more than two-nodes. Things can get more complex when you throw in more nodes, so we'll save these challenges for last. :)

Project Information

Resources