Skip to content

file_reader_refactoring.md

Latest commit

 

History

History
179 lines (137 loc) · 5.61 KB

file_reader_refactoring.md

File metadata and controls

179 lines (137 loc) · 5.61 KB

DefaultFileReader Class Refactoring

Overview

The DefaultFileReader class has been refactored to improve testability, readability, and maintainability. This document outlines the changes made and how to use the new implementation.

Key Improvements

1. Separation of Concerns

  • File path resolution is now handled by dedicated methods
  • File opening is separated from path resolution
  • Configuration management is centralized and configurable

2. Enhanced Testability

  • Dependency injection through constructor parameters
  • Mockable methods for unit testing
  • Clear interfaces between different responsibilities
  • Comprehensive test coverage with isolated test cases

3. Better Error Handling

  • Custom exception hierarchy for different error types
  • Descriptive error messages with context
  • Proper exception chaining for debugging

4. Improved Configuration

  • Configurable defaults that can be overridden
  • Environment-specific settings support
  • Clear configuration contract

5. Enhanced Readability

  • Comprehensive docstrings for all methods
  • Clear method names that describe their purpose
  • Logical method organization from public to private
  • Type hints throughout the codebase

Class Structure

DefaultFileReader

The main class that provides the file reading framework:

class DefaultFileReader(BaseDataAccessLayer):
    # Configuration constants
    DEFAULT_CODE_PACKAGE = 'payload'
    DEFAULT_FILE_FOLDER = 'files'
    DEFAULT_CONFIG_FILE = 'config.json'

    def __init__(self, code_package=None, file_folder=None, config_file=None):
        # Initialize with custom or default configuration

    def file_open(self, file_name: str) -> io.TextIOWrapper:
        # Main public method for opening files

    def get_search_locations(self) -> list[Path]:
        # Get all possible search locations

Exception Hierarchy

FileReaderError (base)
├── FileNotFoundError (file not found in any location)
└── FileAccessError (permission, I/O errors, etc.)

Usage Examples

Basic Usage

from datacustomcode.file.reader.default import DefaultFileReader

# Use default configuration
reader = DefaultFileReader()
with reader.file_open('data.csv') as f:
    content = f.read()

Custom Configuration

from datacustomcode.file.reader.default import DefaultFileReader

# Custom configuration
reader = DefaultFileReader(
    code_package='my_package',
    file_folder='data',
    config_file='settings.json'
)

Error Handling

try:
    with reader.file_open('data.csv') as f:
        content = f.read()
except FileNotFoundError as e:
    print(f"File not found: {e}")
except FileAccessError as e:
    print(f"Access error: {e}")

File Resolution Strategy

The file reader uses a two-tier search strategy:

  1. Primary Location: {code_package}/{file_folder}/{filename}
  2. Fallback Location: {config_file_parent}/{file_folder}/{filename}

This allows for flexible deployment scenarios where files might be in different locations depending on the environment.

Testing

Unit Tests

The refactored class includes comprehensive unit tests covering:

  • Configuration initialization
  • File path resolution
  • Error handling scenarios
  • File opening operations
  • Search location determination

Mocking

The class is designed for easy mocking in tests:

from unittest.mock import patch

with patch('DefaultFileReader._resolve_file_path') as mock_resolve:
    mock_resolve.return_value = Path('/test/file.txt')
    # Test file opening logic

Integration Tests

Integration tests verify the complete file resolution and opening flow using temporary directories and real file operations.

Migration Guide

From Old Implementation

The old implementation had these issues:

  • Hardcoded configuration values
  • Mixed responsibilities in single methods
  • Limited error handling
  • Difficult to test

To New Implementation

  1. Update imports: Use DefaultFileReader from datacustomcode.file.reader.default
  2. Error handling: Catch specific exceptions instead of generic ones
  3. Configuration: Use constructor parameters for custom settings
  4. Testing: Leverage the new mockable methods

Benefits

For Developers

  • Easier debugging with clear error messages
  • Better IDE support with type hints and docstrings
  • Simplified testing with dependency injection
  • Clearer code structure with separated responsibilities

For Maintainers

  • Easier to extend with new file resolution strategies
  • Better error tracking with custom exception types
  • Improved test coverage with isolated test cases
  • Clearer documentation with comprehensive docstrings

For Users

  • More reliable with proper error handling
  • More flexible with configurable behavior
  • Better debugging with descriptive error messages
  • Consistent interface across different implementations

Future Enhancements

The refactored structure makes it easy to add:

  • Additional file resolution strategies (URLs, cloud storage, etc.)
  • File format detection and automatic handling
  • Caching mechanisms for frequently accessed files
  • Async file operations for better performance
  • File validation and integrity checking

Conclusion

The refactored DefaultFileReader class provides a solid foundation for file reading operations while maintaining backward compatibility. The improvements in testability, readability, and maintainability make it easier to develop, test, and maintain file reading functionality in the Data Cloud Custom Code SDK.