Current stable release is 0.4.0 .
All releases, including interim (potentially unstable) releases can be found here: Release History
Version 0.4.0 includes numerous tweaks and patches introduced since 0.2.0.
Notable changes:
This will be the last version of the DeDuplicator that is built against Heritrix 1.10.0. Building against that version of Heritrix has made the DeDuplicator compatible with almost all 1.x versions of Heritrix. Note though that 0.4.0 is built with Java 1.5, unlike 0.2.0 which was built with Java 1.4.2.
In version 1.12.0 Heritrix added some useful features that the DeDuplicator should make use of, most notably marking content as 'not novel' (i.e. duplicate). Also in 1.14.0 there is rudimentary WARC support and the aim is to have the DeDuplicator support writing to WARC files. Therefor, any future versions will be built against Heritrix 1.14.0.
Support for Heritrix 2.0 is planned but there is no set timeframe for it. This requires considerable changes to the DeDuplicator and will likely not be implemented until Heritrix 2.x is sufficiently mature that it is used routinely instead of 1.x for large scale production crawls.
A new interim release has been uploaded to deal with the changed crawl.log format in Heritrix 1.12.0. 0.4.0 will be the final release for Heritrix up to version 1.12.0 and should be released soon.
Heritrix version 2.0.0, currently in development, will greatly change Heritrix's API and so will require significant changes to the DeDuplicator. Look for the first interim release built against the new Heritrix API as soon as the changes are moved into the trunk of the Heritrix project. Probably sometime this month.
The project has now been moved to Sourceforge.net. The code has been moved to SF's CVS and anonymous access is now possible. Initial commit was of version 0.2.0. Change history prior to 0.2.0 will be discarded except that we will keep the packaged PreRelease versions that were made.
Along with the public CVS, SourceForge also provides bug and RFE trackers . The project website has also been moved to http://deduplicator.sourceforge.net/ .
Stable releases will now be distributed via SourceForge while iterim builds will continue to be made available on the Release History page (until continous integration is set up).
On September 21, Kristinn Sigur�sson presented a paper on the DeDuplicator titled 'Managing duplicates across sequential crawls' at the 6th International Web Archiving Workshop held in conjunction with the 10th ECDL in Alicante, Spain.
The paper is available in the Workshop Procedings and can also be downloaded by itself directly from here: Managing duplicates across sequential crawls .
