malscraper v1.4

a modular tool that streamlines the process of scraping and managing lists of payload domains, ioc's & c2 ips from various feeds for easy blacklisting.

github .zip .tar.gz
malScraper Screenshot malScraper Image

table of contents

contents

introduction

malscraper is a modular tool that streamlines the process of scraping and managing lists of payload domains, ioc's & c2 ips from various feeds. it simplifies the task of blacklisting for security and threat intelligence purposes.

features

  • modular design: easily extendable with additional modules.
  • version checking: verify if you are using the latest version on startup.
  • reopen functionality: ability to reopen a previously composed report.
  • download progress bars: clean, color-coded progress bars for all downloads.
  • clear file save feedback: after each download, the full path of the saved file is displayed.
  • payload report obfuscation/zip: choose to obfuscate or zip the payload report before it is written to disk, to help avoid antivirus false positives.

⚠️ antivirus warning

some reports generated by malscraper (especially PayloadReport.txt) may be flagged or quarantined by antivirus software (such as windows defender) because they contain known malware indicators. these files are for research and defensive use only.

  • you will be prompted to obfuscate or zip the payload report to help avoid false positives.
  • if you need to retain these files, consider adding an exclusion for the report directory, or use the obfuscate/zip options.

current release notes

malscraper 1.4 - python expansion

overview

complete rewrite in python, making it fully cross-platform compatible with windows, macos, and linux environments. maintains all the key functionality of the original bash script while adding new features, improving performance, and enhancing the user experience.

key features

cross-platform compatibility
  • now works seamlessly on windows, macos, and linux
  • platform-specific optimizations for file paths and system operations
  • no external dependencies beyond python and the requests library
enhanced user interface
  • improved ascii art header with pyfiglet support
  • consistent color-coded output across all platforms
  • better formatted menus and command outputs
  • progress bars for downloads
  • clear file save locations after each download
  • user prompt for obfuscating/zipping payload report
improved update system
  • detailed version checking with github api integration
  • visual download progress indicators
  • automatic backup of previous version before updating
  • clean restart process after updates
  • display of release notes within the application
robust error handling
  • graceful exit handling (ctrl+c, ctrl+d)
  • comprehensive exception management
  • network connectivity error handling
  • file operation error recovery
performance enhancements
  • faster data processing
  • more efficient memory usage
  • improved file i/o operations

installation

requirements
  • python 3.6 or higher
  • requests library pip install requests
  • optional: pyfiglet for enhanced ascii art pip install pyfiglet
windows
python -m pip install requests
python -m pip install pyfiglet  # optional
python malScraper.py
macos/linux
pip3 install requests
pip3 install pyfiglet  # optional
python3 malScraper.py

upgrading from previous versions

users of previous bash-based versions can simply download the new python script and run it. the application will maintain the same folder structure and file naming conventions.

known issues

  • some terminal emulators may not fully support color codes
  • character encoding issues may occur in certain environments with emoji display
  • antivirus software may flag or quarantine reports containing known malware indicators (see antivirus warning)

version history

  • 1.4.5 - enhanced download experience & error handling
    • download experience:
    • download output is now clean and professional: only the download name, a progress bar, and a single success/failure line per file
    • after each successful download, the full path of the saved file is displayed for user clarity
    • payload report handling prints both the .txt and .zip file paths as appropriate
    • the number of lines in the payload report is now included in the summary
    • error handling:
    • improved error handling for all downloads, with clear user feedback for failures
    • fixed method ordering and linter errors
    • user experience:
    • the summary at the end of downloads is clearer, and the user is prompted to review before continuing
  • 1.4.4 - home menu experience & navigation improvements
    • home menu experience:
    • the malscraper banner and social/application info are now always visible on initial launch and when returning to the home menu
    • the help menu is displayed together with the banner and socials on the home screen for a welcoming, branded experience
    • help menu behaviour:
    • when the help menu is called directly (e.g., by typing help), the screen is cleared and only the help menu is shown, for clarity and focus
    • navigation & cleanliness:
    • improved screen clearing logic for all menus and prompts, ensuring a clean and consistent user interface
    • removed unnecessary debug output from the initial launch
    • bug fixes:
    • fixed an issue where the banner and socials would disappear when showing the help menu
    • fixed an issue where the home menu was not always shown after certain actions
  • 1.4.3 - atomic self-update & user experience improvements
    • atomic self-update:
    • updates are now downloaded and staged in a temporary location
    • the actual script replacement occurs safely on the next launch, avoiding issues with replacing a running script
    • the previous version is always backed up before replacement
    • user experience:
    • after downloading an update, the user is prompted to exit, ensuring the update is applied immediately on restart
    • improved and clarified user messaging throughout the update process
    • eliminated double update prompt, users are only asked once per update cycle
    • robustness:
    • handles update failures gracefully and cleans up temporary files
    • maintains compatibility with single-script update flow
  • 1.4.2 - dependency management & banner rendering
    • dependency management:
    • the script now checks for required third-party packages (requests, pyfiglet) before running
    • if any are missing, the user is prompted to install them automatically
    • missing package messages are now highlighted in yellow and bold for better visibility
    • all dependencies are listed in requirements.txt for easy setup
    • banner rendering:
    • the malscraper banner now adapts to the terminal width and is left-aligned for consistent display across environments
    • code quality:
    • refactored the location of the colours class to support early use in the script
  • 1.4.1 - code improvements & user experience enhancements
    • code improvements:
    • completely restructured the codebase using object-oriented programming
    • replaced string-based path handling with python's pathlib for more reliable cross-platform operation
    • fixed semantic versioning comparison to handle updates correctly
    • improved error handling with proper try/except patterns
    • user experience:
    • fixed non-functional "home" menu option
    • enhanced progress visualization during downloads with proper progress bars
    • added more descriptive status messages during operations
    • improved terminal ui with better formatting and visual indicators
    • added new splash text messages for variety
    • performance:
    • better file handling with proper context managers
    • optimized update process with cleaner file operations
    • improved resource cleanup during error conditions
    • enhanced overall stability and error recovery
    • cross-platform compatibility:
    • improved path handling for windows, macos, and linux
    • better file operation compatibility across different operating systems
    • fixed platform-specific command execution
  • 1.4 - complete rewrite in python
    • fully cross-platform compatible with windows, macos, and linux environments
    • maintains all the key functionality of the original bash script
    • added new features, improving performance, and enhancing the user experience
  • 1.3 - initial python conversion
    • the tool has been converted to python for improved functionality
  • 1.2 - version checking
    • malscraper now verifies if you are using the latest version on startup
  • 1.1 - reopen functionality
    • added the ability to reopen a previously composed report
  • 1.0 - initial implementation
    • core functionality was implemented in the first release

installation

1. clone the repository.
   git clone https://github.com/rynmon/malscraper

2. navigate to the project directory.
   cd malscraper

3. install dependencies.
   pip install -r requirements.txt

usage

to use malscraper, follow these steps:

1. run the tool.
   python malscraper.py

2. follow the on-screen instructions to configure and use the tool.
   - after each scan, you will see a summary and the save locations for all reports.
   - for the payload report, you will be prompted to obfuscate or zip the file before it is written to disk.

contributing

contributions are welcome! if you find a bug or have an enhancement in mind, feel free to open an issue or submit a pull request.

version checking

malscraper will automatically check for updates on startup. make sure you are using the latest version to benefit from new features and improvements.

license

this project is licensed under the mit license.

Nomu
Derry