qufe.interactionhandler module
Screen interaction and automation utilities for capturing, analyzing, and interacting with screen content.
This module provides comprehensive tools for: - Screen capture and image processing - Color detection and analysis - Mouse automation and region-based clicking - Image comparison and difference detection - Progress tracking in Jupyter environments
- Required dependencies:
pip install qufe[vision]
This installs: opencv-python>=4.1.0, matplotlib>=3.1.0, pyautogui>=0.9.48, mss>=4.0.0
- Optional Jupyter dependencies:
pip install qufe[jupyter]
This installs: ipython>=6.0.0
- class qufe.interactionhandler.ProgressUpdater(initial_text: str = '', display_id: str = 'Progress')[source]
Bases:
objectA progress updater for Jupyter notebooks that updates text in place.
Uses display handles to update progress information without creating multiple output cells.
Note: Requires Jupyter/IPython environment. Install with: pip install qufe[jupyter]
- __init__(initial_text: str = '', display_id: str = 'Progress')[source]
Initialize progress updater.
- Parameters:
initial_text – Initial text to display
display_id – Unique identifier for the display handle
- Raises:
ImportError – If IPython is not available (not in Jupyter environment)
- qufe.interactionhandler.analyze_color_codes(x: int = 0, y: int = 0, size: int = 0, print_result: bool = True, display_result: bool = False, display_original: bool = False, img_path: str = '', thickness: int = 1, img=None)[source]
Analyze color codes in a specified square region of the screen or image.
- Parameters:
x – Left coordinate of analysis region
y – Top coordinate of analysis region
size – Size of square region to analyze
print_result – Whether to print individual pixel colors
display_result – Whether to display the analyzed region
display_original – Whether to display original image with region box
img_path – Path to image file (alternative to screen capture)
thickness – Box border thickness for display
img – Image array to analyze (alternative to screen capture or file)
- Returns:
Array of unique colors found in the region
- Return type:
- Raises:
ImportError – If required vision dependencies are not installed
- qufe.interactionhandler.build_region(x: int, y: int, w: int, h: int) Dict[str, int][source]
Build a region dictionary for screen capture.
- qufe.interactionhandler.click_random_in_position(top_left_x: int, top_left_y: int, bottom_right_x: int, bottom_right_y: int) Tuple[int, int][source]
Click randomly within a rectangular area defined by two corner points.
- qufe.interactionhandler.click_random_in_region(region: Dict[str, int], margin: float = 0.4) Tuple[int, int][source]
Click randomly within a specified region with margin.
- Parameters:
region – Region dictionary with ‘left’, ‘top’, ‘width’, ‘height’ keys
margin – Margin ratio to avoid edges (0.0 to 0.5)
- Returns:
Click coordinates (x, y)
- Return type:
- Raises:
ImportError – If required vision dependencies are not installed
- qufe.interactionhandler.display_image(img, jupyter: bool = True, is_bgra: bool = False) None[source]
Display image in Jupyter notebook or OpenCV window.
- Parameters:
img – Image array to display (numpy.ndarray)
jupyter – Whether to display in Jupyter notebook (default: True)
is_bgra – Whether image is in BGRA format (default: False)
- Raises:
ImportError – If required vision dependencies are not installed
- qufe.interactionhandler.draw_boxes(image, boxes: List[Tuple[int, int, int, int]], color: Tuple[int, int, int] = (0, 0, 255), thickness: int = 2)[source]
Draw bounding boxes on an image.
- Parameters:
image – Input image array (numpy.ndarray)
boxes – List of bounding boxes as (x, y, width, height)
color – Box color in BGR format (default: red)
thickness – Box border thickness
- Returns:
Image with drawn boxes
- Return type:
- Raises:
ImportError – If required vision dependencies are not installed
- qufe.interactionhandler.find_image_differences(img_1, img_2, min_size: int = 5, print_result: bool = False, display_result: bool = False, thickness: int = 1, is_bgra: bool = False) List[source]
Find differences between two images and return contours of different regions.
- Parameters:
img_1 – First image array (numpy.ndarray)
img_2 – Second image array (numpy.ndarray)
min_size – Minimum size for valid difference regions
print_result – Whether to print difference regions
display_result – Whether to display result with boxes
thickness – Box border thickness for display
is_bgra – Whether images are in BGRA format
- Returns:
Contours of different regions
- Return type:
List
- Raises:
ImportError – If required vision dependencies are not installed
- qufe.interactionhandler.find_largest_box(boxes: List[Tuple[int, int, int, int]]) Tuple[int, int, int, int] | None[source]
Find the box with the largest area from a list of boxes.
- qufe.interactionhandler.get_color_boxes(image, rgb_color: Tuple[int, int, int], tolerance: float = 0.1, min_size: int = 5) List[Tuple[int, int, int, int]][source]
Find bounding boxes of regions matching a specific color within tolerance.
- Parameters:
image – Input image array (numpy.ndarray)
rgb_color – Target RGB color tuple
tolerance – Color matching tolerance (0.0 to 1.0)
min_size – Minimum size for valid boxes
- Returns:
List of bounding boxes as (x, y, width, height)
- Return type:
- Raises:
ImportError – If required vision dependencies are not installed
- qufe.interactionhandler.get_resolution() Tuple[int, int][source]
Get screen resolution.
- Returns:
Screen width and height in pixels
- Return type:
- Raises:
ImportError – If required vision dependencies are not installed
- qufe.interactionhandler.get_screenshot(x: int = 0, y: int = 0, w: int = 0, h: int = 0)[source]
Capture a screenshot of specified region or full screen.
- Parameters:
x – Left coordinate (default: 0)
y – Top coordinate (default: 0)
w – Width (default: 0, captures full width)
h – Height (default: 0, captures full height)
- Returns:
Screenshot as numpy array in BGRA format
- Return type:
- Raises:
ImportError – If required vision dependencies are not installed