diffoscope (137) unstable; urgency=medium * Also extract classes2.dex, classes3.dex etc. from .apk files. (Closes: reproducible-builds/diffoscope#88) * Add generalised support for "ignoring" returncodes and move special-casing of returncodes in the zip comparator to this. * Accommodate sng returning with a UNIX exit code of 1 even if there minor errors in the file (discovered via #950806). -- Chris Lamb Tue, 18 Feb 2020 17:15:59 +0000 diffoscope (136) unstable; urgency=medium [ Chris Lamb ] * Improvements: - Support external build tools. (Closes: reproducible-builds/diffoscope#87) - Fallback to the regular .zip container format for .apk files if apktool is not available. - Clarify that "install X" in difference comment messages actually refer to a system/distribution package. - Drop the --max-report-size-child and --max-diff-block-lines-parent options; both deprecated and scheduled for removal in January 2018. * Bug fixes: - No longer raise a KeyError exception if we request an invalid member from a directory container. * Logging improvements: - Log a debug-level message if we cannot open a file as container due to a missing tool in order to assist diagnosing issues. - If we don't know the HTML output name, don't emit an enigmatic "html output for" message. - Add the current PATH environment variable to the "Normalising locale..." debug-level message. - Print the "Starting diffoscope $VERSION" line as the first line. - Correct a debug-level message for compare_meta calls to quote the arguments correctly. * Refactoring: - Add support for alternative container types for a file, allowing for runtime (vs import time) control of fallbacks such as adding comments. and append a comment to a difference if we fallback to an inferior container format due to missing a tool. - Factor-out the generation of "foo not available in path" difference comment messages as a helper method in the exception that represents them. * Code improvements: - Tidy diffoscope.main's configure method, factoring out the set of the Config() global out of the run_diffoscope method and inlining the functionality of maybe_set_limit, etc. - Rename diffoscope.locale module to diffoscope.environ as we are modifying things beyond just the locale (eg. calling tzset(), etc.) - Drop unused "Difference" import from the APK comparator. - Drop an assertion that is guaranteed by parallel "if" conditional. - Add a "noqa" line to avoid a false-positive flake8 "unused import" warning. - Turn down the "volume" for a recommendation in a comment. * Release/source-code management: - Add a .git-blame-ignore-revs file to improve the output of git-blame(1) by ignoring large changes when introducing the Black source code reformatter and update the CONTRIBUTING.md guide on how to optionally use it locally. - Convert CONTRIBUTING.rst to CONTRIBUTING.md and include it in the PyPI.org release. * Test improvements - Refresh and update the fixtures for the .ico tests to match the latest version of Imagemagick in Debian unstable. [ Holger Levsen ] * Bump Standards Version to 4.5.0, no changes needed. [ Marc Herbert ] * Search for expected keywords in the output of cbfstool tests and not a specific output. (Closes: reproducible-builds/diffoscope!42) -- Chris Lamb Fri, 24 Jan 2020 15:38:57 +0000 diffoscope (135) unstable; urgency=medium [ Chris Lamb ] * Extract resources.arsc files (as well as classes.dex) from Android .apk files to ensure that we at least show differences there. (Closes: #916359) * Always pass a filename with a ".zip" extension to zipnote otherwise it will return with an exit code of 9 and revert to a binary diff for the entire file. * Don't exhaustively output the entire HTML report when testing the regression for #875281; parsing the JSON and pruning the tree should be enough. * Factor out running all of our zipinfo variants into a new method and use this routine when comparing .apk files, thus now also displaying any differences exposed by bsdtar. * Testsuite improvements: - Always allow fixtures called "output*". - Actually test the output of the test_html_regression_875281 test. * Add a note to the "Contributing" page to suggest enable concurrency when running the tests locally. [ Marc Herbert ] * Fix a misplaced debug "Loading diff from stdin" logging message. * Add new "Testing" section to CONTRIBUTING.rst. -- Chris Lamb Tue, 14 Jan 2020 14:26:07 +0000 diffoscope (134) unstable; urgency=medium [ Chris Lamb ] * Ensure that autopkgtests are run with our pyproject.toml present for the correct "black" source code formatter settings. (Closes: #945993) * Tidy some unnecessary boolean logic in the ISO9660 tests. * Rename the "text_option_with_stdiout" to "text_option_with_stdout". * Include the libarchive file listing for ISO images to ensure that timestamps (not just dates) are visible in any difference. (Closes: reproducible-builds/diffoscope#81) [ Eli Schwartz ] * Fix an exception in the progressbar handler. [ Vagrant Cascadian ] * Add an external tool reference for zstd on GNU Guix. -- Chris Lamb Fri, 27 Dec 2019 19:17:16 +0000 diffoscope (133) unstable; urgency=medium * Correct the substitution/filtering of paths in ELF output to avoid unnecessary differences depending on the path name provided on the commandline. (Closes: #945572) * Silence/correct a SyntaxWarning message due to incorrectly comparing an integer by identity (is) over equality (==). (Closes: #945531) -- Chris Lamb Thu, 28 Nov 2019 10:03:33 +0000 diffoscope (132) unstable; urgency=medium * Allow all possible .zip variations to return with non-zero exit codes, not just known types we can explicitly identify (eg. Java .jmod and .jar files). (Closes: reproducible-builds/diffoscope#78) * Also permit UTF-8 encoded .dsc and .changes files. (Re: reproducible-builds/diffoscope#78) * Rework a long string of "or" statements into a loop with a "break". -- Chris Lamb Tue, 26 Nov 2019 10:31:44 +0000 diffoscope (131) unstable; urgency=medium * Clarify in the HTML and text outputs that the limits are per output format, not global. (Closes: #944882, reproducible-builds/diffoscope#76) - Bump the previous "max_page_size" limit from 400 kB to 4 MB. * Update an Android manifest test to reflect that parsed XML attributes are returned in a sorted or otherwise novel manner under Python 3.8. * Update code to reflect version 19.10b0 of the black source code reformatter and don't run our self-test for versions earlier than this as it will generate different results and thus fail. * Limit .dsc and .buildinfo matching; don't attempt to compare them as ELF sections or similar. (Closes: reproducible-builds/diffoscope#77) * Add a comment that the text_ascii{1,2} test fixture files are used in multiple places so are not trivial to generate on the fly. -- Chris Lamb Sat, 23 Nov 2019 15:22:51 -0500 diffoscope (130) unstable; urgency=medium [ Chris Lamb ] * debian/tests/basic-command-line: Move from deprecated ADTTMP to AUTOPKGTEST_TMP. * Correct the matching of R .rds files by also detecting newer versions of this file format. * Drop unused BASE_DIR global in the tests. * Try and ensure that new test data files are generated dynamically, ie. at least no new ones are added without "good" reasons. * Truncate the tcpdump expected diff to 8KB (from ~600KB). * Refresh OCaml test fixtures to support OCaml >= 4.08.1. Closes: #944709 * Correct reference to the ".rdx" extension in a comment. * Update XML test for Python 3.8+. * Don't use line-base dbuffering when communucating with subprocesses in binary mode. (Closes: reproducible-builds/diffoscope#75) [ Jelle van der Waa ] * Add support for comparing .zst files are created by zstd. (Closes: reproducible-builds/diffoscope!34) -- Mattia Rizzolo Thu, 14 Nov 2019 11:22:20 +0100 diffoscope (129) unstable; urgency=medium * Call R's "deparse" function to ensure that we do not error out and revert to a binary diff when processing .rdb files with internal "vector" types as they do not automatically coerce to strings. * Add the ability to pass Python bytestrings to external commands and pass our long script to parse R .rdb files using this new method over a long command-line argument * Use Rscript's --vanilla option over --no-environ as this also enables --no-save, --no-restore, --no-site-file and --no-init-file. * Improve command-line error messages: - Split out formatting into a separate utility function. - Truncate very long lines when displaying them as an external source of data. - When printing an error from a command, format the command for the user. * Use "exit code" over "return code" when referring to UNIX error codes in displayed differences. -- Chris Lamb Mon, 28 Oct 2019 11:15:18 +0000 diffoscope (128) unstable; urgency=medium * Query the container for the full path of the parallel R .rdx file for a .rdb file as well as looking in the same directory. This ensures that comparing two .deb/.changes files shows a varying path introduced in version 127. -- Chris Lamb Fri, 25 Oct 2019 09:21:40 +0100 diffoscope (127) unstable; urgency=medium [ Chris Lamb ] * Move build-dependency on python-argcomplete to the Python 3.x version to facilitate Python 2.x removal. (Closes: #942967) * Overhaul the handling of GNU R .rdb files: - Rework and refactor the handling of .rdb files specifically with respect to locating the parallel .rdx file prior to inspecting the file to ensure that we do not add files to the user's filesystem in the case of directly comparing two .rdb files or, worse, overwriting a file in is place. - Use a ("""-formatted) docstring for our internal R script to dump variables. - Mask/hide standard errors; R will often produce noisy output that is not useful to us. - Don't include a useless and misleading "NULL". - Include all R object names are displayed, including ones beginning with a fullstop ("."). - Sort package fields when dumping data for output stability. - Format package contents as "foo = bar" rather than using ugly and misleading brackets. - Include the object's type when dumping package contents. - Never read the site or user's R location environment configuration to ensure output stability. - Expose absolute paths in the semantic/human-readable output, preventing falling back to a useless hexdump. * Improve the formatting of command lines: - Ensure newlines and other metacharacters appear escaped as "\n", etc. - Use the newline (etc.) escaped version of the commandline being executed in logging/debug output. - When displaying standard error, ensure use the escaped version too. * Add support for easily masking the standard error of commands and use this in the ffprobe comparator. * To match the libarchive container, raise a KeyError exception if we request an invalid member from a directory container. * Correct string representation output in the traceback when we cannot locate a specific item in a container. [ Mattia Rizzolo ] * Run Debian autopkgtests against all Python versions. -- Chris Lamb Thu, 24 Oct 2019 15:59:34 +0100 diffoscope (126) unstable; urgency=medium [ Chris Lamb ] * Track and report on missing Python modules. (Closes: reproducible-builds/diffoscope#72) * Drop some unnecessary control flow. * Drop explicit inheriting from 'object' class; unnecessary in Python 3. * Bump Standards-Version to 4.4.1. [ Mattia Rizzolo ] * Exit with 2 instead of 1 in case of no disk space. -- Chris Lamb Mon, 14 Oct 2019 12:29:47 -0700 diffoscope (125) unstable; urgency=medium * The test_libmix_differences ELF test requires xxd. (Closes: #940645) -- Chris Lamb Wed, 18 Sep 2019 11:44:57 +0200 diffoscope (124) unstable; urgency=medium [ Chris Lamb ] * Also conditionally skip the identification and "no differences" tests as we require the Ocaml compiler to be present when building the test files themselves. (Closes: #940471) [ Mattia Rizzolo ] * Permit all sorts of version suffixes when checking the Debian package version matches setup.py, not just for backports. (Closes: #939387) [ Vagrant Cascadian ] * Add external tools on GNU Guix for odt2txt, sng and pgpdump. [ Marc Herbert ] * Remove StaticLibFile in the ELF comparator -- ArFile.compare_details() is superior. (Closes: reproducible-builds/diffoscope#64) -- Chris Lamb Tue, 17 Sep 2019 12:19:07 +0200 diffoscope (123) unstable; urgency=medium [ Chris Lamb ] * Build OCaml test input files on-demand rather than shipping them with the package in order to prevent test failures with OCaml 4.08. (Closes: #939386) * Update test skipping messages: - When skipping tests due to the lack of installed tool, print the package that may provide it. - Update "requires foo module" messages to clarify that they are regarding Python modules, not packages. * Rebuild the test squashfs images to exclude the character device as they requires root or fakeroot to extract. (reproducible-builds/diffoscope#65) * Remove accidentally-committed test fixture generation code from tests. * Set long_description_content_type to 'text/x-rst' in setup.py to appease the PyPI.org linter. [ Mattia Rizzolo ] * Don't crash when the progress is requested but the Python module is missing. (Closes: #939085) [ Vagrant Cascadian ] * Add external tools for gifbuild, javap and kbxutil in GNU Guix. [ Marc Herbert ] * In the cbfs tests, add a "Comp" column test to support Coreboot utils > 4.6. -- Chris Lamb Sat, 07 Sep 2019 14:16:31 +0100 diffoscope (122) unstable; urgency=medium [ Chris Lamb ] * Apply patch from László Böszörményi to update the squashfs test output and bump the required version for the test itself. (Closes: #935684) * Skip calls to unsquashfs when we are not root or fakeroot. (Re: reproducible-builds/diffoscope#63) * Include either standard error or standard output (and not just the latter) if/when an external command fails. * Fix a few unicode/bytes issues: - Avoid a possible traceback caused by a str/bytes confusion when handling the output of failing external commands. - Ensure that all of our artificially-created subprocess.CalledProcessError instances have `output` instances that are bytes objects, not str. * Improve debugging output: * Add the containing module name to the (eg.) "Using StaticLibFile for ..." * Improve and condense output when creating our Comparator object types. * Correct a reference to `parser.diff` as `diff` in this context is a Python function in the module, not the actual output returned from diff(1). * Add the "wabt" Debian package to the test dependencies so that we run the wasm tests. [ Mattia Rizzolo ] * Now that we test-require wabt, expect that its tools to be available during autopkgtests. -- Chris Lamb Fri, 30 Aug 2019 08:52:25 +0100 diffoscope (121) unstable; urgency=medium * Don't fallback to a (useless) raw hexdump when readelf(1) reports an minor issue in a section in an ELF binary. For example, when the "frames" section is of the "NOBITS" type, its contents are apparently "unreliable" and thus readelf(1) exits with a returncode of 1. (Closes: #849407, #931962) * Add support to Difference.from_command_exc and friends to optionally ignore specified returncodes from the called program and treat them as "no" difference. * Simplify the parsing of the optional "command_args" argument to the from_command and from_command_exc methods in the Difference class. -- Chris Lamb Fri, 16 Aug 2019 09:49:07 -0700 diffoscope (120) unstable; urgency=medium * No-change sourceful after accidentally uploading binaries in order to migration to testing. -- Chris Lamb Mon, 29 Jul 2019 17:17:46 -0300 diffoscope (119) unstable; urgency=medium [ Chris Lamb ] * If a command fails to execute but does not print anything to standard error, try and include the first line of standard output in the message we include in the diff. This was motivated by readelf(1) returning its error messages on stdout. (Closes: #931963) * Add support for Java ".jmod" modules. Not all versions of file(1) support detection of Jmod files yet, so we perform a manual comparison instead. (Closes: #933308) * Re-add "return code" noun to "Command `foo` exited with X" error messages. * Factor out the ability to ignore the exit codes of "zipinfo" and "zipinfo -v" in the presence of non-standard headers. * Only override the exit code from our special-cased calls to zipinfo(1) if they are 1 or 2 to avoid potentially masking real errors. * Also add missing textual description entries for ZipFile and MozillaZipFile file types. * Skip extra newline in "Output:\n". * Cease ignoring test failures in stable-backports. * Bump debhelper compat level to 12. [ Marc Herbert ] * Catch failures to disassemble and rescue all other differences. (Closes: reproducible-builds/diffoscope!29) -- Chris Lamb Mon, 29 Jul 2019 15:22:24 -0300 diffoscope (118) unstable; urgency=medium * Don't fail in autopkgtests when, for example, we do not have sufficiently newer or older version of file. (Closes: #931881) * Also include python3-tlsh in test dependencies. * Tidy handling of DIFFOSCOPE_FAIL_TESTS_ON_MISSING_TOOLS: - Merge two previously-overlapping environment variables. - Use repr(..)-style output when printing test status. - Add some explicit return values to appease pylint. -- Chris Lamb Sat, 13 Jul 2019 10:23:29 -0300 diffoscope (117) unstable; urgency=medium [ Chris Lamb ] * Add support for file 5.37. Thanks again to Christoph Biedl for the heads-up in advance. (Closes: reproducible-builds/diffoscope/#57) * Apply patch from Gianfranco Costamagna to fix autopkgtest failures in Debian. (Closes: #931709) [ Holger Levsen ] * debian/control: Bump Standards-Version to 4.4.0. -- Chris Lamb Tue, 09 Jul 2019 10:24:54 -0300 diffoscope (116) unstable; urgency=medium [ Chris Lamb ] * Upload to unstable after the release of Debian "buster". * README.rst & manual page generation: - Strip out manpage-only parts of the README rather than using "only" reStructuredText directives in order to support the demands of the latest PyPI website. - Use "real" reStructuredText comments instead of using the "raw" directive. * Dockerfile: - Build the Docker image from the current Git checkout, not the Debian archive. (reproducible-builds/diffoscope#56) - Use the ENTRYPOINT directive with the JSON syntax instead so we pass all arguments to the underlying diffoscope executable. * Document that run_diffoscope should not be considered a stable API. [ Mattia Rizzolo ] * Rename a test function to prevent shadowing a previous one with the same name. * Add ffmpeg to the list of Debian build-dependencies for the testsuite. [ Vagrant Cascadian ] * Add support for known external tools in GNU Guix. -- Chris Lamb Sun, 07 Jul 2019 11:54:29 -0300 diffoscope (115) experimental; urgency=medium [ Chris Lamb ] * Fix execution of symbolic links that point to the "bin/diffoscope" entry point by fully resolving the location. * Add a Dockerfile. * Update contributing messages with instructions regarding the Docker image. [ Mattia Rizzolo ] * tests: + Allow specifying which tools are known missing using DIFFOSCOPE_TESTS_MISSING_TOOLS, to override a _FAIL_ON_MISSING_TOOLS. + With DIFFOSCOPE_TESTS_FAIL_ON_MISSING_TOOLS=1, actually fail only tests that are missing the required tools. * debian: + Add black to the test dependencies. + Enforce the "fail on missing tools" only when testing with all the recommended packages. + Ack some missing tools during autopkgtest. + Reinstate oggvideotools and procyon-decompiler test dependencies, now that are not buggy anymore. -- Mattia Rizzolo Tue, 21 May 2019 16:02:02 +0200 diffoscope (114) experimental; urgency=medium [ Chris Lamb ] * Add support for GnuPG "keybox" files. Thanks to Daniel Kahn Gillmor for the suggestion. (Closes: #871244, reproducible-builds/diffoscope#23) * Always warn if tlsh module is not available (not just if a specific fuzziness threshold is specified) to match the epilog of the --help output. This prevents missing support for file rename detection. (Closes: #888237, reproducible-builds/diffoscope#29) * Treat missing tools on Debian autopkgtests as individual test failures by checking whether a new DIFFOSCOPE_TESTS_FAIL_ON_MISSING_TOOLS environment variable is exported. (Closes: #905885, reproducible-builds/diffoscope#35) * Require that "-" is explicitly specified to read a single diff from standard input to avoid non-intuitive behaviour when diffoscope is called without any arguments. (Closes: reproducible-builds/diffoscope#54) * Make --use-dbgsym a ternary operator to make it easier to totally disable. Thanks to Mattia Rizzolo for the suggestion. * Consolidate on "e" as the aliased exception name. [ Milena Boselli Rosa ] * Various fixes to the HTML markup to prevent validation warnings/errors: - Prevent empty values for the "name" attribute name on HTML anchor tags, and add an "id" to its parent "div" container. - Fix "table column x established by element 'col' has no cells beginning in it" warnings. - Fix "Text run is not in Unicode Normalization Form C". - Remove the "type" HTML attribute from