torrential
| resources: | Home Mailing List Source Code Members Development Blog |
|---|
Development Blog
February 16, 2005Not much to report today. I've learned how to debug Mozilla using gdb, so I can finally get to tracking down stability issues. The two biggest problems were fixed today. Before today the two major issues were random crashes occuring during a download and the tendency for some .torrents to bring down the system. The random crashes were occuring when a connection to a peer was aborted, I wasn't properly checking the validity of a connection before referencing it, and a null pointer reference emerged. The .torrent bug was caused by the tendency for some loads to occur across multiple calls to OnDataAvailable.
With those two issues fixed, a new bug has emerged. Currently there is no validation of the incoming commands, specifically if the client receives a piece command with an invalid piece number or offset, a buffer overflow can occur. It appears as though other clients send faulty information quite regularly. Yippee. That should be pretty easy to fix, and I should have that working by the end of the day.
If all goes (very) well today, I might just be ready to release something before the end of the day. A note to those that are watching: after today I may have no time to even look at this until Tuesday of next week. So if I don't get anything out today, it will probably be at least a week until you hear anything new. It's possible I'll have some time over the weekend, but not likely.
February 15, 2005Some major milestones have been reached. Last entry listed some things that I thought were necessary for an initial release, I have completed many of those items. Specifically:
- Pieces are requested in order, without an endgame strategy (which isn't necessary fora first release).
- Pieces can be received, hashed, and saved if correct.
Additionally, a more efficient method of intercepting data of type application/x-bittorrent has been developed. Before I was using an nsIContentHandler, which required that any received application/x-bittorrent data be cancelled and requested again (due to an oversight in the design of nsIContentHandler). Now I'm using an nsIURIContentListener, which is working perfectly.
I also needed to replace the SHA1 hashing system. Previously I used nsISignatureVerifier, but that is very broken. Whenever a large amount of data was hashed the whole system crashed. The amount of data in a single piece of a file was too much. So now I'm using GPLed SHA1 code from http://www.cr0.net:8040/code/crypto/sha1/. Ultimately I'd like to actually fix nsISignatureVerifier, but for now the GPLed solution will work fine.
As the project currently stands, it's possible to completely download a file. I've done it, many times. Unfortunately, the software is somewhat unstable and I'm working on locating the source of the crashes. Now and then it just suddenly decides to lock up the browser, I suspect that I'm doing something not thread-safe. Or of course my lack of error-handling has gotten the best of me.
Only a few things remain before a first release:
- Outgoing notification of piece availablility (bitfield and have commands)
- Sending of pieces
- Visual feedback
- Stability fixes
Basically, I won't release anything that's going to create any leeches on the system. You have to share! Also, I'd like to get things more stable, it's not acceptable to have the browser crashing randomly while either sending or receiving torrents. A few minor stability problems would be OK (it's version 0.0.1 after all), but the current level of instability is too high.
February 11, 2005I've learned to love/hate XPCOM in the last few days. It's nice to have a method of automatically handling allocations and deallocations, but if you mess it up it can be impossible to track down the cause.
Here's what I have working:
- Management of the list of current downloads
- Peer connection management, including incoming connections
- Handshaking (both on initiated connections and incoming connections)
- Incoming commands (piece, request, and cancel not done)
- Contacting the tracker initially and on shutdown
So, in summary: when you click on a .torrent download, the .torrent is downloaded and decoded. The tracker is contacted and upon receiving a response a list of peers is created and associated with the download. A handshake occurs with each peer, and any commands they send (not related directly to data transfer) are interpreted and acted upon. Furthermore, incoming connections are accepted and associated with the appropriate download.
Given that functionality, all that's left to finish before I will feel comfortable releasing v0.0.1 is as follows:
- Some sort of central management needs to be written that determines which pieces to request from whom. I think I have this all planned out.
- Peers must be informed of which pieces are available.
- Peers need to be able to request pieces from us.
- We must be able to ask for a piece, hash it, and save it.
- Some sort of (very basic) visual feedback needs to be provided. Initally this will probably be a simple window with a progress bar.
Once all those things are done, I will release v0.0.1. It shouldn't take too long at this point, I've gotten through the ugly mess of managing the connections with peers (with one exception, see below). All the rest of the work really only requires that a few simple functions be written.
As for the connection management exception, I need some help. It appears as though some sort of object is not going away when it needs to. When you shutdown the browser you get a series of errors that read as follows:
###!!! ASSERTION: unable to post continuation event: 'Error', file nsStreamUtils.cpp, line 441
Break: at file nsStreamUtils.cpp, line 441
The number of times the error appears seems to be related to the number of peers you have connected to, but I can't find a 1-to-1 correlation between the errors and any specific case (such as peers that timed-out or peers that have open connections). I suspect that this error has to do with some an nsISocketTransport object not being destroyed correctly. If anyone knows what conditions might cause this error, please let me know.
February 7, 2005I've spent the last few days learning as much as I can about the architechure of the download manager in the Mozilla core code. I've been trying to determine if it would be wisest to write Torrential as an extension or a patch to the core code.
Each potential method has pros and cons:
Extension pros:
- Easy to install
- Available to people that aren't building from source
- Quicker to develop an early version with bare-minimum functionality
Extension cons:
- Must be carefully designed in order to facilitate future integration into the Mozilla core
- Will require multiple builds for each target OS and architecture
- Will be difficult to seemlessly integrate with the download manager
- Will require loads of duplicate code (code stolen from nsDownloadManager and friends)
Patch pros:
- Will enable simple integration into the main trunk
- Will allow for easy integration with the existing download manager
- Can leverage existing download manager code
Patch cons:
- Will be very difficult to install, requiring that the end user build from scratch
- Will be much more time consuming to develop, as a higher standard of quality will be required before any release is made
- Will bottleneck deployment, as less people will be willing to patch and rebuild
Given this list, I've decided to go with an extension. My goal is to quickly provide the core functionality (download torrents). Once an extension is developed and (mostly) stable, then I will take the existing code and integrate it into the existing download manager code so that it can be considered for inclusion in the main trunk. The biggest stumbling block that I see is the requirement that multiple builds be provided, for each OS and architecture. In order to address this, I plan on an initial release that only supports Linux on x86. I can also provide Windows binaries before too long. Anyone else is either going to need to build themselves or wait until a patch version is available.
I expect to have some sort of early release in a week or two. You should be able to download and seed single file .torrents, and that's about it. Error checking will nearly non-existent, and a UI will be basic if it exists at all. I've developed some of the necessary code for integration with the download manager, but without flushing it out a lot more it's rather useless, so it won't be in the first release.
February 3, 2005Since it's the first entry, today I plan on just spewing out the brief history of this project, up until now.
About two weeks ago I got the idea for this project. I was actually just looking through all the available extensions for Firefox and I noticed that no one had written a Bittorrent client yet. I was pretty amazed that I was the first one to think of it (I've subsequently learned that I wasn't the first). So, I started researching what would be required and got right to work. By the end of the week I had written an extension entirely in Javascript that successfully downloaded a file via Bittorrent.
Of course, it wasn't perfect. It had no UI, communicated status completely by printing to the console, and it was slow. Very slow. It brought the entire browser to a crawl and didn't download data particularly fast.
I tried addressing the speed issue by rewriting the code that handled the download to take advantage of multithreading, only to find out that many essential XPCOM components weren't thread-safe.
Long story short: Javascript simply isn't fast enough. It can't move around large amounts of information without bogging down the browser, and multi-threading wasn't going to work. Besides, I didn't really do a good job designing the software. So early this week I scrapped the Javascript code and started over in C++.
Currently the design and implementation of the decyphering of the .torrent and the contacting of the tracker are complete, and work beautifully. I plan on spending today and some time tomorrow designing and implementing the management of peer connections.