Archived

This topic is now archived and is closed to further replies.

Ultima

BEncode Editor

Recommended Posts

bencode-editor.th.png

Introduction
BEncoding is a data encoding scheme used primarily in the BitTorrent world. Because BEncoded files can contain binary data, and because of some of the intricacies involved in the way binary strings are stored, it is often not safe to edit such files in text editors. Many people have previously asked about where one might find an editor that can safely edit BEncoded files, but none have really existed in the general sense. The only ones I've seen were .torrent file editors, which (although they technically do edit BEncoded files) don't edit just ANY type of BEncoded file. Out of a bit of boredom and free time, this editor was born.

Example Uses

Edit .torrent files:
- the announce/announce-list keys (binary/list) store the tracker(s)
- the url-list key (binary/list) stores the webseed(s)
- the nodes key (list) stores the DHT bootstrap node(s)
 

WARNING: When editing .torrent files, any changes to the info dictionary will change the .torrent file's infohash. Unless you know what you're doing, you should refrain from doing this. If you aren't familiar with the .torrent file metadata structure, read this: http://wiki.theory.org/BitTorrentSpecification in particular, the "Metainfo File Structure" section)

 

Edit µTorrent's .dat files:
- corrupt resume.dat files can sometimes be salvaged simply by loading and saving the resume.dat file in this editor
- the paths stored in resume.dat can be edited en masse with Find/Replace
 

WARNING: Make sure you first exit µTorrent before editing these files, as µTorrent rewrites/updates the files on exit. Additionally, the .fileguard key should be removed, since µTorrent will consider its .dat file to be damaged if it is edited and no longer matches the stored .fileguard hash.

 

Final Warning
This utility is for advanced users. Read the warnings in the USES section, and read them again until you understand well what you're getting yourself into by using this editor. Its relatively intuitive UI can beguile most uninitiated/beginning users, as it has an inherent ability to invalidate many files through wrong and incorrect edits.

Consider this editor to be experimental. As such, you should exercise caution by backing up any files you plan on editing with this utility.

Download
 

x64 builds will run only on 64-bit versions of Windows. Unicode builds can display Unicode characters (like Asian characters) properly, but work only with Windows 2000 and above. ANSI builds can't display Unicode characters properly, but should work on Windows 9x and above.

Notes
- Binary data and integers are exported as raw data rather than BEncoded data
- Dictionary merging does not sort keys or resolve duplicate keys
- File recovery recovers only decodable parts of input files
- Finding "Value by Key" finds exact key names unless RegEx is used
- Holding Ctrl while reloading reloads data from disk
- Holding Shift while moving moves an item to top/bottom (direction-dependent)
- Holding Shift while pasting pastes the item below the currently focused item
- Holding Shift while sorting will recursively search for dictionaries to sort
- Only dictionaries can be sorted
- Only files containing dictionaries and lists can be edited directly
- Switching an item between dictionary/list will cause the item to be cleared
- The root of a dictionary (not its children) must be selected to sort it

Final Thoughts
Bug reports, suggestions, and the like are always welcome... Just be aware that there are no guarantees that I'll ever get around to doing anything about them. I might not do anything about them for a period of time, and then I might suddenly work on them in a "spur of the moment" kind of... uh, whatever.

 

History

v0.7.1.0 (2010-02-09)
+ Feature: Recovery fallback on decoding error

v0.7.0.0 (2009-12-31)
^ New: x64 build support
+ Feature: Item hashing
+ Feature: Item validation
+ Feature: Merge binary/dictionary/list data
+ Feature: Move item to top/bottom
+ Feature: Recursive sorting (hold Shift while clicking Item > Sort Keys)
+ Feature: Undo/redo changes
~ Change: Allow blank key names when searching using "Value by Key"
~ Change: Allow save if file no longer exists, even without changes
~ Change: Binary input/conversion made slightly more lenient/reliable
~ Change: Exporting integers exports a raw (rather than BEncoded) integer
~ Change: Indicate in the window title whether a file has been modified
~ Change: Move up/down keyboard shortcuts changed to Ctrl+Up/Down
~ Change: Reload the treeview on F5, reload from disk on Ctrl+R
~ Change: Select all text in focused dialog input control on Ctrl+A
~ Change: Shift+F3 searches in the opposite direction
~ Change: Show "Find" dialog if never shown before "Find Next" is used
~ Change: Store Find options only on search, not on dialog close
* Fix: Repeated successive conversions into Binary allowed in Find/Replace
* Fix: Switching data types not properly hiding/showing controls in dialogs
* Fix: UI "allows" child to be added to an integer, but crashes on attempt

v0.6.1.0 (2008-09-18)
* Fix: Paths with consecutive '%' characters fail to open

v0.6.0.0 (2008-06-20)
+ Feature: Cut/Copy/Paste items
+ Feature: Filter searches based on key
+ Feature: Holding Ctrl while reloading reloads data from disk
+ Feature: Log the number of replacements made during Replace All
+ Feature: Move item up/down
+ Feature: On import, use filename as suggestion for key name (if empty)
+ Feature: Regular expressions search/replace (PCRE engine)
+ Feature: Select all Logger tab items with Ctrl+A
+ Feature: Sort keys (for dictionaries only)
~ Change: Add accelerators for dialog buttons
~ Change: Center popup dialogs on display
~ Change: Disable dialog buttons on action
~ Change: Display Edit dialog on double-click only if text double-clicked
~ Change: Don't collapse/expand treeview item on double-click
~ Change: Faster BDecoding (~30% improvement over v0.5.1 in tests)
~ Change: Faster BEncoding (~85% improvement over v0.5.1 in tests)
~ Change: Faster Find/Replace operation (much improved on larger treeviews)
~ Change: Faster treeview filling (~50% improvement over v0.5.1 in tests)
~ Change: Focus main window on drag-and-drop
~ Change: Increase input limit on edit controls
~ Change: Searches for blank values are always treated as exact searches
~ Change: Select all text in Value field each time Find/Replace dialog shown
~ Change: Unicode support on treeview and listview
~ Change: Use accelerators instead of global hotkeys
~ Change: Visual feedback when busy performing find (busy mouse cursor)
* Fix: Inaccurate window resize limiting under various Windows themes
* Fix: Non-transparent background on "button" icons in certain situations
* Fix: Switching binary input type in Add/Edit dialogs causes data loss
* Fix: Sub-item count not displayed for dictionaries/lists added to a list

v0.5.1.0 (2008-03-06)
* Fix: Missing controls in the Find/Replace dialog

v0.5.0.0 (2008-02-14)
+ Feature: "Item" menu with associated keyboard shortcuts
+ Feature: Copy Logger tab items with Ctrl+C
+ Feature: Double-click to edit item
+ Feature: Export data from selected item
+ Feature: Import external file as binary data
+ Feature: Input raw BEncoded data directly or from an external file
~ Change: Add .torrent and .dat to dropdown list in Open/Save dialogs
~ Change: Add line breaks to the Logger tab between file opens
~ Change: Disable hotkeys when busy
~ Change: Don't allow saves when no changes have been made to data
~ Change: Faster BDecoding (~15% improvement over v0.4.1 in tests)
~ Change: Faster treeview filling (~20% improvement over v0.4.1 in tests)
~ Change: Miscellaneous Find/Replace tweaks, fixes, and polish
~ Change: Show type and item count for "[ ROOT ]" item
* Fix: "ITEM -1" when adding child list/dictionary to bottom of list
* Fix: "Reached beginning/end of document" message sometimes shown twice
* Fix: Combobox not showing options in dropdown on pre-XP systems
* Fix: Strings can replace integers in Find/Replace
* Fix: Typo in Logger entry for BEncoding
* Fix: Using "Find Next" sometimes returns focus to wrong window

v0.4.1.0 (2007-12-05)
* Fix: "Properize" bad integer input

v0.4.0.0 (2007-12-05)
~ Change: "File > Load" changed to "File > Open"
~ Change: Faster BDecoding (~30% improvement over v0.3 in tests)
~ Change: Faster treeview filling (~70% improvement over v0.3 in tests)
~ Change: Minor optimizations to adding/deleting/editing of items
~ Change: No longer clears window if file loading/decoding fails
~ Change: No longer switches to Structure tab on file open
~ Change: Scroll Logger when new entries are added
~ Change: Timings shown in Logger
* Fix: File locked if open/save fail
* Fix: File save error caused window to clear (possible data loss)
* Fix: Find/Replace on keys with dictionary values caused incorrect item count
* Fix: Integers in lists not displayed on treeview
* Fix: Integers larger than 32-bits not displayed properly on treeview
* Fix: Integers larger than 64-bits unusable
* Fix: Keyboard shortcut for file open not working
* Fix: Making binary value empty in lists doesn't update/show on treeview

v0.3.0.0 (2007-11-25)
+ Feature: Drag and drop support
+ Feature: Find/Find Next/Replace
+ Feature: Load files from commandline
~ Change: A bit more aware of disk read/write errors
~ Change: Don't close main window on Esc
~ Change: Faster BDecoding (~75% improvement over v0.2 in tests)
~ Change: Log some more events
~ Change: Smaller executable due to lessened dependence on standard library
~ Change: Visual feedback when busy (busy mouse cursor)

v0.2.0.0 (2007-10-25)
+ Feature: Create NEW BEncoded files
+ Feature: Edit "[ ROOT ]" item directly
+ Feature: Keyboard shortcuts
~ Change: Ask to save file before performing actions where changes get lost
~ Change: Don't display ellipses ("...") for empty binary strings
~ Change: Improve string/binary conversion to minimize chances of data loss
~ Change: Switch to Logger tab on error
~ Change: Use child dialogs for FileOpenDialog, FileSaveDialog, and MsgBox
~ Change: Use icons as buttons (Crystal Clear icon set by Everaldo)
~ Change: Warn when data loss imminent from switching views
* Fix: Saving appends data to end of file

v0.1.0.0 (2007-10-08)
^ New: Initial release

 

Share this post


Link to post
Share on other sites

Yeah, I forgot to mention that pre-XP OSes don't work with it I haven't tested this on anything besides Windows XP. Uh, I don't have any system with an old OS running at the moment, so I can't really test it, but AFAIK, fixing that would might break Unicode support...

I noticed some bug where saving to existing files would append data to the end of the existing file instead of overwriting it with the new data. So uh, be cautious of that, as that might kill the file you're trying to overwrite. I've fixed it in my build, though.

Share this post


Link to post
Share on other sites
Changelog

v0.2 (2007-10-25)

+ Ability to create NEW bencoded files

+ Ability to edit "[ ROOT ]" item directly

+ Added some keyboard shortcuts

+ Ask to save file before performing actions where changes get lost

+ Warn when data loss imminent from switching between binary/string view

~ Buttons moved and changed to icons (Crystal Clear icon set by Everaldo)

~ Don't display ellipses ("...") for empty binary strings

~ Fixed bug where saving saving would append data to the end of the file

~ Improve string/binary conversion to minimize chances of data loss

~ Switch to Logger tab on error

~ Use child dialogs for FileOpenDialog, FileSaveDialog, and MsgBox

Download

[2007-10-24] v0.2 (Unicode) - http://www.zshare.net/download/4430821f21d914/

[2007-10-24] v0.2 (ANSI) - http://www.zshare.net/download/4430841eacfd53/

Bah. As usual, no guarantees on anything, so if your computer blows up or you lose some super important/expensive piece(s) of data while messing with this, don't blame me :P

Share this post


Link to post
Share on other sites

Aaaargh, I guess I can stop working on my JAVA tool, as yours seems better than mine (no lovely and friendly GUI) ! Thanks for working on it anyway ;)

Chris

Share this post


Link to post
Share on other sites

Ultima

Moderator

I want thank you for your Bencode Editor. )))

...recently i got a power loss and my resume.dat with big bunch of torrent was corrupted. Utorrent only say that hash failed and clean torrent file!

So i searching for soft that can fix my resume.dat and nothing found. But i find uTorrentMonitor.js with decoding routine $))) i rewrote function BDecode_ to delphi and write small decoder to view structure of resume.dat, and ooh...i find place where something bad was writed to the resume.dat. i cuted out last torrent and resave file with Bencode Editor...moving file to utorrent and..finally got all my torrents back!!!! (except something last maybe) $)

Share this post


Link to post
Share on other sites

You're welcome; I'm just glad it was of use :)

Changelog

v0.3 (2007-11-25)

+ Drag and drop support

+ Find/Find Next/Replace

+ Load files from commandline

~ A bit more aware of disk read/write errors

~ Don't close main window on Esc

~ Improved decoding speed (dramatically in some cases)

~ Log some more events

~ Smaller executable due to lessened dependence on standard library

~ Visual feedback when busy (busy mouse cursor)

Just as a note, ANSI builds are absolutely untested, so if they don't work on pre-Win2K systems... well, there isn't much I can do about it, since I don't have an old Windows install to test it on :/

Find/Replace is erm... how would you say... highly experimental? I did not test it very extensively. From my semi-cursory tests, though, it does work for basic stuff. I'm not sure whether or not it'll cause data loss in large chunks of binary data, so be careful when using it (especially that Replace All button). Really, I think you should be making backups of any file before you edit them with this editor. Anyhow, it's possible to replace substrings in key, values, or keys and values. Replacing keys only makes it simple to replace paths in resume.dat en masse. w00t.

I actually wanted to try implementing some Undo feature, but got lazy-ish. Some groundwork was laid (and I'm not liking it very much), but in order to get this release out the door, I had to omit that feature (otherwise, this would have been sitting on my drive for weeks without getting touched).

Move Up/Down and Cut/Copy/Paste are some other stuff I'm looking at, but I feel shaky about implementing them, since there's no easy way to swap treeview items around without manually recursing/iterating through every item and its children to copy (or at least that's the only way I can think of). That'd probably end up very slow in some cases. Meh.

Suggestions and feedback are always welcome!

And here's the usual disclaimer:

This utility is provided as-is. I make no guarantees about its quality, so... use with caution?

Share this post


Link to post
Share on other sites

Nice tool, but there is a critical bug in it: it spoils all the integers bigger than 2^32 -- apparently, you're using DWORD to save the decoded integers.

Share this post


Link to post
Share on other sites

Thanks for the bug report! I missed this bug entirely because I didn't realize that anyone would need such large numbers :o The bug was actually just a tad bit different than you described. The treeview was limited to displaying a maximum of (2^31)-1, while internally, the editor was able to handle up to (2^63)-1. As an example, try creating a bencoded file like this:

d4:testi9223372036854775808ee

That's 2^63. You'll notice the treeview displays -1, but when you try editing the item, it displays (2^63)-1 = 9223372036854775807 (not the correct value, but still larger than 2^32).

Anyhow, I've fixed the bug in my copy (along with an assortment of other silly bugs I noticed), and the editor can now handle integers of unlimited size (assuming no other bugs crept in). Interestingly, from some tests, fixing this actually improved the decoding time by around 2-5%, but that's nothing to write home about, so bleh :P

Share this post


Link to post
Share on other sites
Changelog

v0.4 (2007-12-05)

~ Change: "File > Load" changed to "File > Open"

~ Change: Faster bdecoding (~30% improvement over v0.3 in tests)

~ Change: Faster treeview filling (~70% improvement over v0.3 in tests)

~ Change: Minor optimizations to adding/deleting/editing of items

~ Change: No longer clears window if file loading/decoding fails

~ Change: No longer switches to Structure tab on file open

~ Change: Scroll Logger when new entries are added

~ Change: Timings shown in Logger

* Fix: File locked if open/save fail

* Fix: File save error caused window to clear (possible data loss)

* Fix: Keyboard shortcut for file open not working

* Fix: Integers in lists not displayed on treeview

* Fix: Integers larger than 32-bits not displayed properly on treeview

* Fix: Integers larger than 64-bits unusable

* Fix: Making binary value empty in lists doesn't update/show on treeview

* Fix: Find/Replace on keys with dictionary values caused incorrect item count

YMMV may vary when it comes to performance improvements. I tested using some random .torrent file I found a while ago that took a long time for me to decode and fill the treeview with (because it contains lots of files). Rawr.

Edit: 0.4.1 released due to stupid bug.

Share this post


Link to post
Share on other sites

Oh, I'm not sure about allowing the user to cut/paste -- it's not implemented :P I had been sitting on this release for the past few days because I was trying to think of a good way to do the undo/copy/cut/paste/move up/move down, but it all came back to ugly and slow, so I decided to call it quits on those features (for now?). Maybe I'll think of something next time...

What those features all have in common is that they potentially mean removing treeview items and/or refilling parts of the treeview, which is horrendously slow in cases where there are lots of child nodes/items in the tree. Using the torrent I'm testing, it could mean up to 4 seconds of wait just to perform a move operation, or to undo a delete, or to paste some subtree. Ugh.

Edit: Eh, just realized that weird stuff can be inputted as "integers". Fixed and reuploading...

Changelog

v0.4.1 (2007-12-05)

* Fix: "Properize" bad integer input

Share this post


Link to post
Share on other sites

Thanks very much for this app. It expedited my move from Windows to Mac, thanks to find and replace. Runs fine in Darwine.

Share this post


Link to post
Share on other sites

I'm a bit confused about this tool wrt resume.dat

I have a resume.dat that utorrent doesn't like (after a system crash) ie - When starting, utorrent logs "Warning: file integrity check failed (hash doesn't match): ...\resume.dat", renames the file .bad and I loose ALL torrent info.

I pulled this resume.dat up in this editor. The editor does NOT report any hash error. (huh?)

I deleted the last torrent, then saved the file. The fileguard hash value did NOT change.

I started utorrent using the modified resume.dat.

Again, I got the SAME error in the utorrent log, BUT, utorrent went ahead and used the resume data anyway - ie I did NOT loose everything (just the one torrent I deleted with the editor).

Is this the way everything is SUPPOSED to work?

When does this editor recompute the hash?

Why does utorrent say the hash fails (both orig and modified), but the editor doesn't?

Why does utorrent go ahead and use the modified 'bad' file, but not the original?

???

Share this post


Link to post
Share on other sites

You forgot to say thank you to Ultima.

IN any case I"M surprised it loaded the file fine with the .fileguard attribute in there.

FILEGUARD is calculated by uT upon save... the fact it didn't change is normal.

Are you sure it didn't use the resume.dat.old when it found your edited resume.dat "Bad"?

Also log messages in your original thread would help :)

THis thread is (as i see it) the place to placate and place niceties on Ultima for making such a small easy to use bencoded file editor :)

BTW, Ultima RULES!!! :D :cool: B-) K:)

Share this post


Link to post
Share on other sites

I was under the impression that that was part of the feature set of this tool - ie that if you modified resume.dat that the tool recalculated the fileguard hash value so utorrent would be happy with it.

And yes, thanks Ultima - but am I missing something? People in this thread talk about using it to modify resume.dat, but noone else has asked about the hash issue?

IE - Am *I* doing something wrong? or am *I* misunderstanding what the tool does?

And further, if the hash isn't right, is it expected that utorrent will go ahead and use it anyway ?!

Oh and PS - I was using the 0.4.1 unicode version. FYI - It seemed to work on W2K except that the drop downs on the find/replace dialogue had no options (but I ended up doing the edits on an XP system)

jewelisheaven - re: your other questions (wrt the bug report) - I've added the msg to it, and yes I'm sure the .old wasn't involved as I moved everything out the directory when experimenting.)

Share this post


Link to post
Share on other sites

Not especially. From Ultima's presentation of the tool, I take it as an actual graphical way to edit generic bencoded files. The specific.fileguard key is a component of uTorrent's configuration and as such, you can just remove the key when giving it unauthenticated files. *I thought i said that above, but I must have had my m&c on my mind*

As far as you misunderstanding, I don't think so. You were quite a sport doing all that research, and using the forum, and finding this. I really only wanted to poke fun at you for not saying thank you. ;)

The other tool I use is torrentspy.sf.net but that doesn't allow one to edit, AND it only loads .torrent files. So when I wanted to check out changed keys in the 1.8 line for example, i had to rename files, then rename them back to use them.

With the editor i can File - Load and BOOM!

Again thanks for being an investigative person.

Share this post


Link to post
Share on other sites
The editor does NOT report any hash error. (huh?)

...

The fileguard hash value did NOT change.

...

Is this the way everything is SUPPOSED to work?

Yes, because this is a BEncode editor, not a resume.dat editor. So um, it'll never use/recompute the .fileguard hash, and I don't intend on ever making it do so (or at least I don't forsee it happening).

About the dropdown menu being "invisible" under pre-XP sytems... Heh, mistake. I knew about this problem going in, but picked incorrect sizes for the widgets, so it came out badly. Whoops! It's fixed for the next release (whenever that might be). Thanks for the report :)

Added to the first post:

Note: If you are editing µTorrent's .dat files with this utility, make sure you...

1) remove the .fileguard hash key in the respective file (unless you want µTorrent to complain and throw the file away...?)

2) exit µTorrent before actually saving the file

Share this post


Link to post
Share on other sites

Thanks for the response. However, in looking over the replies to this thread, you might notice that most people are interested in using it for resume.dat editting. As such, and as a feature request for a future release, please consider adding logic along the lines of:

If .fileguard present, then compute hash when saving the file.

In any event, thanks again for creating this tool.

PS - I'm still confused as to utorrent's use of the fileguard element, as it seems willing to accept the data in files that fail the hash check (or don't have the element at all? have not tried that yet).

PPS - Just noticed your edit - which makes the fileguard element seem even less useful ??

Share this post


Link to post
Share on other sites

The editor is, as jewelisheaven said, a generic BEncode file editor. µTorrent will accept .dat files without any issue if the .fileguard hash is missing, and it isn't that difficult to remove the hash if needed. The only reason the .fileguard hash is there is to detect a bad shutdown. If you're manually editing the .dat file, it's not a bad shutdown, so there's no reason µTorrent should require the .fileguard hash. Removing it makes perfect sense.

If the .fileguard hash is there, and the data doesn't match, µTorrent will simply rename the file as resume.dat.*.bad and start using from a blank resume.dat (or resume.dat.old if it exists).

Share this post


Link to post
Share on other sites

"bad shutdown" - hmmm...

My recent experience (see bug report forum) is that utorrent accepted the resume.dat data even though the hash check failed. Yes, it renamed it .bad when it was done, but it still accepted the contents.

(This was with 1.7.2 - don't know about other versions)

Anyway, thanks for the clarification. I'll give that a whirl next time...

Share this post


Link to post
Share on other sites

A really great Editor. Thanks for writing and sharing! :)

FeatureRequests:

- Editbutton on the left site (edit + -) bcs its an EDITor :)

- Doubleclick on item = edit item ;)

- Possibility to move entries in list (upward/downward)

- Possibility to include external binary code (e.g. a grafic or something)

pixel.gif

I also noticed that this editor is really great for php script websites!

I have a webspace with no mysql. So I use files as a database. But now I am able to use multiple informations in one file thanks to you! :) I use this bencode.php library in my php script and it works great (with the examples).

So it would be great if you could include upward/downward of list entries to your next release. ;)

Edit - BTW - Feedback:

Your editor works great under wine (linux) too (until yet). I am using ubuntu here (I dumped my windows partition).

Edit2: I am now linking to this thread on my signature. If others of you also like this editor pls to the same. Thank you.

Share this post


Link to post
Share on other sites

Thanks for the feedback and suggestions :)

A few remarks...

- I don't think the edit button should be moved over just because it's an editor. Adding an entry to the bencoded data is still considered editing it.

- Moving entries up/down has been previously addressed

- I'll see about double-clicking, but I wouldn't count on it

- Importing external files as values? I was considering something similar (setting/getting dictionary/list values directly as raw bencoded data by clipboard/text, or from file). We shall see...

Share this post


Link to post
Share on other sites

Feature Request:

Is it possible to add filetypes to the dropdown for the Ctrl-O open dialog? All files is nice, since it's a generic editor, but *.dat, and *.torrent would be a nice addition :D

OOOOH Raw data import... that certainly would help if say I wanted to ADD announce-list key. Right now it takes .. 3 clicks, 1, add announce-list, change to list. add-child list, add child, binary as string. :D

Edit: What kind of data is your test torrent populated with.

I tried the loading with a torrent whose info block had 1200 files in it... < 1.5 sec loadtime. Also had the main dictionary have over 200 keys (manually edited) and it still didn't take > 2 seconds... Are you sure it's not based upon processor speed? I have a (relatively old 1.6 GHz) computer, but I can understand if you're taking the approach that uT is based upon, where it should be able to run speedily even on a 14 MiB RAM computer... ;)

Share this post


Link to post
Share on other sites