Tensor.Art
Create

Dataset Tools for Imaging and Captioning

OTHER
Original

Updated:

v0.59
V0.58.5
v0.55

Dataset-Tools: A Simple Dataset Viewer for AI Art

Dataset Tools is a desktop application designed to help users browse and manage their image datasets, particularly those used with AI art generation tools (like Stable Diffusion WebUI Forge, A1111, ComfyUI) and model files (like Safetensors). Developed using Python and PyQt6, it provides an intuitive graphical interface for browsing files, viewing embedded generation parameters, and examining associated metadata.

This project is inspired by tools within the AI art community, notably stable-diffusion-prompt-reader by receyuki, and aims to empower users in improving their dataset curation workflow. We welcome contributions; feel free to fork the repository and submit pull requests!

Daily updates are here: https://github.com/Ktiseos-Nyx/Dataset-Tools

Features

  • Lightweight & Fast: Designed for quick loading and efficient metadata display.

  • Cross-Platform: Built with Python and PyQt6 (compatible with Windows, macOS, Linux).

  • Comprehensive Metadata Viewing:

    • Clearly displays prompt information (positive, negative, SDXL-specific).

    • Shows detailed generation parameters from various AI tools.

  • Intuitive File Handling:

    • Drag and Drop: Easily load single image files or entire folders. Dropped files are auto-selected.

    • Folder browsing and file list navigation.

  • Image Preview: Clear, rescalable preview for selected images.

  • Copy Metadata: One-click copy of parsed metadata to the clipboard.

  • Themeable UI: Supports themes via qt-material (e.g., dark_pink, light_lightgreen_500).

  • Extensible Parser System:

    • Utilizes a significantly adapted and enhanced version of sd-prompt-reader for robust parsing of many common AI image metadata formats.

    • New Custom Parsers: Includes dedicated parsers for:

      • RuinedFooocus (UserComment JSON).

      • Civitai ComfyUI (UserComment JSON with "extraMetadata").

    • Model File Support: Basic metadata viewing for .safetensors and .gguf model files.

  • Configurable Logging: Control application log verbosity via command-line arguments for easier debugging.

Supported Formats

Dataset-Tools aims to read metadata from a wide array of sources. Current capabilities include:

AI Image Metadata:

  • A1111 webUI / Forge: PNG (parameters chunk), JPEG/WEBP (UserComment).

  • ComfyUI:

    • Standard PNGs (embedded workflow JSON in "prompt" chunk).

    • Civitai-generated JPEGs/PNGs (UserComment JSON with "extraMetadata").

  • NovelAI: PNG (Legacy "Software" tag & "Comment" JSON; Stealth LSB in alpha channel).

  • InvokeAI: PNG (parsing "invokeai_metadata", "sd-metadata", or "Dream" chunks).

  • Easy Diffusion: PNG, JPEG, WEBP (embedded JSON metadata).

  • Fooocus: PNG ("Comment" chunk JSON), JPEG (JFIF comment JSON).

  • RuinedFooocus: JPEG (UserComment JSON).

  • Draw Things: PNG (XMP metadata containing JSON).

  • StableSwarmUI: PNG, JPEG (EXIF or "sui_image_params" in PNG/UserComment).

  • (Support for other formats may be implicitly included via the adapted sd-prompt-reader core.)

Model File Metadata (Header Information):

  • .safetensors

  • .gguf

Other File Types:

  • .txt: Displays content.

  • .json, .toml: Displays content (future: structured view).

Installation

Prerequisites:

  • Python 3.11 (as this was the version used during development and for dependency resolution). Other Python 3.9+ versions might work but are not extensively tested.

  • pip (Python package installer).

  • git (for cloning the repository).

Steps:

  1. Clone the repository:

    git clone https://github.com/Ktiseos-Nyx/Dataset-Tools.git
cd Dataset-Tools

  2. Create and activate a Python virtual environment (recommended):

    python3.11 -m venv .venv 
# Or: python -m venv .venv

# Activate:
# Windows: .venv\Scripts\activate
# macOS/Linux (bash/zsh): source .venv/bin/activate
# macOS/Linux (fish): source .venv/bin/activate.fish

  3. Install the package and its dependencies: The project uses pyproject.toml and can be installed using pip.

    # For users (standard install):
pip install .

# For developers (editable install, recommended for contributing):
pip install -e .

    This command will read pyproject.toml and install Dataset-Tools along with all libraries listed as dependencies (e.g., PyQt6, Pillow, qt-material, piexif, pyexiv2, toml, rich, pydantic).

    NOTE: uv users

    cd Dataset-Tools
uv pip install .

  4. Run the application with dataset-tools command:

    dataset-tools

You're in!

GUI Interaction

Loading Files:

  1. Click the "Open Folder" button or use the File > Change Folder... menu option.

  2. Drag and Drop: Drag a single image/model file or an entire folder directly onto the application window.

  3. If a single file is dropped, its parent folder will be loaded, and the file will be automatically selected in the list.

  4. If a folder is dropped, that folder will be loaded.

Navigation:

  1. Select files from the list on the left panel to view their details.

    • Image Preview: Selected images are displayed in the preview area on the right. Non-image files or files that cannot be previewed will show a "No preview available" message.

    • Metadata Display: Parsed prompts (Positive, Negative), generation parameters (Steps, Sampler, CFG, Seed, etc.), and other relevant metadata are shown in the text areas below/beside the image preview. The Prompt Info and Generation Info section titles will update based on the content found.

    • Copy Metadata: Use the "Copy Metadata" button to copy the currently displayed parsed metadata (from the text areas) to your system clipboard.

    • File List Actions: Sort Files: Click the "Sort Files" button to sort the items in the file list alphabetically by type (images, then text, then models).

    • Settings & Themes: Access application settings (e.g., display theme, window size preferences) via the "Settings..." button at the bottom or the View > Themes menu for quick theme changes.

Future Ideas (TODO)

  • Enhanced metadata display and editing for model files (.safetensors, .gguf).

  • Full metadata editing and saving capabilities for images.

  • Batch operations: e.g., export all metadata from a folder, rename files based on metadata.

  • Advanced search and filtering capabilities within loaded datasets based on metadata content.

  • Support for more image, text, and model metadata formats.

  • A plugin architecture to allow for easier addition of custom parsers or functionalities.

  • Improved UI/UX for text/schema file viewing (e.g., syntax highlighting for JSON/TOML, better text wrapping).

  • Packaging for PyPI for easier pip install dataset-tools.

  • Creation of standalone executables for Windows, macOS, and Linux.

  • Comprehensive automated test suite to ensure stability and prevent regressions.

Contributing

Your contributions are welcome! Whether it's bug reports, feature requests, documentation improvements, or code contributions, please feel free to get involved.

  • Issues: Please check the issues tab for existing bugs or ideas. If you don't see your issue, please open a new one with a clear description and steps to reproduce (for bugs).

  • Pull Requests: Fork the repository. Create a new branch for your feature or bugfix (git checkout -b feature/your-feature-name or bugfix/issue-number). Make your changes and commit them with clear, descriptive messages. Push your branch to your fork (git push origin feature/your-feature-name). Submit a pull request to the main branch of the Ktiseos-Nyx/Dataset-Tools repository. Please provide a clear description of your changes in the PR.

Lisc: https://github.com/Ktiseos-Nyx/Dataset-Tools/blob/main/LICENSE

Am available for commissions.

Recipe:

1 cup sass

3 cups WHOOPS.

5 cups WHERES THE. CHEETOS?

Bake at 450 for 24 hours and then call the nearest fire department.

Read a few memes while you're in the ER for severe burns

(KIDDING.)

Contact Us:

Our Discord: https://discord.gg/HhBSvM9gBY

Earth & Dusk Media https://discord.gg/5t2kYxt7An

Backups: https://huggingface.co/EarthnDusk

Send a Pizza: https://ko-fi.com/duskfallcrew/

WE ARE PROUDLY SUPPORTED BY: https://yodayo.com/ / https://moescape.ai/

JOIN OUR DA GROUP: https://www.deviantart.com/diffusionai

JOIN OUR SUBREDDIT: https://www.reddit.com/r/earthndusk/

Version Detail

SD 1.5
<p><strong>PR: Major Overhaul - Parsing, Stability, UI, Security, and Docs</strong></p><p>This pull request introduces a significant set of improvements and features, representing a substantial leap forward in the development of Dataset-Tools. It focuses on enhancing core stability, overhauling the metadata parsing pipeline, adding key usability features like drag-and-drop, integrating new custom parsers, and updating project documentation for clarity and accuracy.</p><p><strong>⚡ Key Highlights & New Features</strong></p><ul><li><p><strong>Drag and Drop Support:</strong> Users can now drag and drop image/model files or entire folders directly onto the main application window for easier loading. Dropped single files are auto-selected.</p></li><li><p><strong>New Custom Parsers:</strong></p><ul><li><p><code>RuinedFooocusFormat</code>: Added support for RuinedFooocus JSON metadata.</p></li><li><p><code>CivitaiComfyUIFormat</code>: Added support for Civitai's ComfyUI-based UserComment metadata, including mojibake decoding.</p></li><li><p><code>Enabled GGUFParser</code>: Basic metadata extraction from <code>.gguf</code> model files is now functional.</p></li></ul></li><li><p><strong>Major Stability Improvements:</strong> Addressed numerous critical bugs related to imports, attribute access, and type errors across the application.</p></li><li><p><strong>Enhanced Vendored Code:</strong> Significantly refactored the adapted <code>sd-prompt-reader</code> components for better status handling, error reporting, and integration.</p></li><li><p><strong>Security Hardening:</strong> Fixed XML parsing vulnerabilities by integrating <code>defusedxml</code>.</p></li><li><p><strong>Improved Code Quality:</strong> Applied extensive linting fixes using Ruff, adhering more closely to PEP 8 and improving maintainability.</p></li><li><p><strong>Updated Documentation:</strong> <code>README.md</code> and <code>SECURITY.MD</code> have been overhauled; <code>NOTICE.md</code> added for vendored code attribution.</p></li></ul><p><strong>🐛 Core Stability & Bug Fixes</strong></p><ul><li><p>Resolved critical import errors (<code>ModuleNotFoundError</code>, <code>NameError</code>) across <code>ui.py</code>, <code>logger.py</code>, <code>correct_types.py</code>, and <code>vendored_sdpr/*</code> by correcting import paths and resolving circular dependencies.</p></li><li><p>Fixed <code>AttributeError: 'str' object has no attribute 'value'</code> by refactoring <code>EmptyField</code>, <code>UpField</code>, <code>DownField</code> in <code>correct_types.py</code> to be proper <code>enum.Enum</code> classes. UI and parsers now correctly use <code>.value</code> and <code>get_ordered_labels()</code>.</p></li><li><p>Fixed <code>AttributeError</code> in <code>widgets.py</code> related to <code>ExtensionType</code> by using correct attribute names (e.g., <code>SCHEMA_FILES</code>, <code>MODEL_FILES</code>).</p></li><li><p>Resolved <code>TypeError</code> for parser <code>__init__</code> calls (e.g., for <code>A1111</code>) in <code>ImageDataReader</code> by ensuring passed arguments match parser signatures.</p></li><li><p>Fixed <code>ValueError: not enough values to unpack</code> in <code>A1111._sd_format()</code> by correcting regex match unpacking.</p></li><li><p>Addressed <code>NameError: name 'Path' is not defined</code> in <code>ImageDataReader</code> by adding the <code>pathlib.Path</code> import.</p></li><li><p>Eliminated <code>F811 Redefinition of unused __init__</code> in <code>access_disk.py</code>.</p></li><li><p>Corrected <code>F821 Undefined name traceback</code> in <code>access_disk.py</code> by importing <code>traceback</code>.</p></li></ul><p><strong>⚙️ Vendored SD-Prompt-Reader Enhancements & Fixes</strong></p><ul><li><p>Refactored <code>ImageDataReader._try_parser()</code> helper for clearer instantiation and more robust status checking of individual format parsers.</p></li><li><p>Ensured <code>ImageDataReader.status</code> is accurately updated based on the success/failure of sub-parsers, resolving previously misleading <code>FORMAT_ERROR</code> reports.</p></li><li><p>Improved status and error message handling within individual format parsers (e.g., <code>A1111._process()</code> now reliably sets its own success/failure status).</p></li><li><p>Standardized logger names within the <code>vendored_sdpr</code> components to use the <code>"DSVendored_SDPR"</code> prefix.</p></li><li><p>Pruned <code>vendored_sdpr/format/utility.py</code> to essential functions, removing unused GUI dependencies.</p></li><li><p>Integrated <code>defusedxml</code> for safer XML parsing in <code>ImageDataReader</code> (for DrawThings XMP data), addressing Bandit-flagged vulnerabilities (B408, B318). Added <code>defusedxml</code> as a project dependency.</p></li></ul><p><strong>✨ New Parser Integrations</strong></p><p>(Covered in Key Highlights)</p><p><strong>🎨 Code Quality & Linting (Ruff)</strong></p><p>Applied extensive linting fixes across the codebase using Ruff (<code>ruff format .</code> and <code>ruff check . --fix</code>):</p><ul><li><p>Resolved numerous <code>E701</code>, <code>E702</code>, <code>E703</code> style issues (multiple statements per line, semicolons).</p></li><li><p>Removed many <code>F401</code> (unused imports) and <code>F841</code> (unused local variables) after careful review.</p></li><li><p>Corrected <code>F403</code>/<code>F405</code> related to star imports.</p></li><li><p>Fixed <code>E741</code> (ambiguous variable names).</p></li><li><p>Resolved <code>E722</code> (bare <code>except</code>).</p></li><li><p>Corrected <code>F541</code> (f-strings without placeholders).</p></li></ul><p>These changes significantly improve code readability, maintainability, and adherence to PEP 8.</p><p><strong>🔊 Logging System Improvements</strong></p><ul><li><p>Resolved the <code>WARNING</code> in <code>main.py</code> regarding logger reconfiguration by aligning the function call to <code>reconfigure_all_loggers</code>. CLI log level setting is now fully effective.</p></li><li><p>Enhanced debug logging in key areas for better diagnostics.</p></li></ul><p><strong>📚 Documentation & Project Files</strong></p><ul><li><p><code>README.md</code>: Overhauled with details on new features (drag & drop, new parsers), revised installation instructions (reflecting <code>pyproject.toml</code>), updated license information, and improved overall structure and formatting.</p></li><li><p><code>SECURITY.md</code>: Updated to be more user-friendly, clearly state the project's GPL-3.0 license, define support for the current main branch, and list security tools now in use (<code>pip-audit</code>, Bandit, Ruff).</p></li><li><p><code>NOTICE.md</code>: Created to provide proper attribution and license information for vendored code, particularly the MIT-licensed components from <code>sd-prompt-reader</code>.</p></li></ul><p><strong>🔬 Testing Notes (Brief)</strong></p><ul><li><p>Manual testing performed for A1111/Forge PNGs, showing correct parsing and status reporting.</p></li><li><p>Drag and drop functionality tested with single files and folders.</p></li><li><p>Civitai ComfyUI JPGs and RuinedFooocus JPEGs now correctly identified by their respective parsers.</p></li><li><p>(Add any other key tests you performed).</p></li></ul><p><strong>👀 Areas for Particular Review (Optional)</strong></p><ul><li><p>Verification of <code>__init__</code> signatures for <code>NovelAI(info=...)</code> and <code>ComfyUI(info=...)</code> calls within <code>ImageDataReader.py</code> (regarding <code>width</code>/<code>height</code> arguments) to ensure they match the actual definitions in those parsers.</p></li><li><p>The robustness of the regex in <code>metadata_parser.py</code>'s <code>make_paired_str_dict</code> for diverse A1111 settings strings (this is an ongoing refinement).</p></li><li><p>The refactored ComfyUI graph traversal logic in <code>vendored_sdpr/format/comfyui.py</code> for correctness and completeness with varied workflows.</p></li></ul><p><a target="_blank" rel="ugc" href="https://github.com/Ktiseos-Nyx/Dataset-Tools/releases/tag/0.58.5">https://github.com/Ktiseos-Nyx/Dataset-Tools/releases/tag/0.58.5</a></p>

Project Permissions

    Use Permissions

  • Use in TENSOR Online

  • As a online training base model on TENSOR

  • Use without crediting me

  • Share merges of this model

  • Use different permissions on merges

    Commercial Use

  • Sell generated contents

  • Use on generation services

  • Sell this model or merges

Related Posts