00.overview

File Storage Module Overview

Overview Description

The File Storage module provides comprehensive functionality for managing file uploads to Google Cloud Storage (GCS) and tracking upload history. This module includes features for generating signed URLs for direct uploads, storing upload metadata, retrieving upload histories, and managing error reporting. The module is designed to support CSV file uploads with validation error tracking and provides both API responses and CSV export capabilities.

Activity Diagram

---
config:
  theme: base
  layout: dagre
  flowchart:
    curve: linear
    htmlLabels: true
  themeVariables:
    edgeLabelBackground: "transparent"
---
flowchart TD
    Client[Client Application]
    API[UploadGcsController]
    GCSService[GcsService]
    ImportService[ImportReviewService]
    ExportService[ExportService]
    Repository[ReviewUploadHistoryRepository]
    Database[(Database)]
    GCS[(Google Cloud Storage)]
    
    Client --- Step1[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #6699cc !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>1</span>
            <p style='margin-top: 8px'>Request Signed URL</p>
        </div>
    ]
    Step1 --> API

    API --- Step2[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #6699cc !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>2</span>
            <p style='margin-top: 8px'>Generate Signed URL</p>
        </div>
    ]
    Step2 --> ImportService

    ImportService --- Step3[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #6699cc !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>3</span>
            <p style='margin-top: 8px'>Request URL from GCS</p>
        </div>
    ]
    Step3 --> GCSService

    GCSService --- Step4[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #6699cc !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>4</span>
            <p style='margin-top: 8px'>Return Signed URL</p>
        </div>
    ]
    Step4 --> GCS

    GCS --- Step5[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #6699cc !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>5</span>
            <p style='margin-top: 8px'>Return URL to Client</p>
        </div>
    ]
    Step5 --> GCSService
    GCSService --> ImportService
    ImportService --> API
    API --> Client

    Client --- Step6[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #6699cc !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>6</span>
            <p style='margin-top: 8px'>Upload File Directly</p>
        </div>
    ]
    Step6 --> GCS

    Client --- Step7[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #99cc66 !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>7</span>
            <p style='margin-top: 8px'>Store Upload History</p>
        </div>
    ]
    Step7 --> API

    API --- Step8[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #99cc66 !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>8</span>
            <p style='margin-top: 8px'>Save to Database</p>
        </div>
    ]
    Step8 --> Repository

    Repository --- Step9[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #99cc66 !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>9</span>
            <p style='margin-top: 8px'>Store History Record</p>
        </div>
    ]
    Step9 --> Database

    Client --- Step10[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #cc66cc !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>10</span>
            <p style='margin-top: 8px'>Request History List</p>
        </div>
    ]
    Step10 --> API

    API --- Step11[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #cc66cc !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>11</span>
            <p style='margin-top: 8px'>Query Database</p>
        </div>
    ]
    Step11 --> Repository

    Repository --- Step12[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #cc66cc !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>12</span>
            <p style='margin-top: 8px'>Return History Data</p>
        </div>
    ]
    Step12 --> Database

    Client --- Step13[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #ff9966 !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>13</span>
            <p style='margin-top: 8px'>Download Error CSV</p>
        </div>
    ]
    Step13 --> API

    API --- Step14[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #ff9966 !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>14</span>
            <p style='margin-top: 8px'>Generate CSV File</p>
        </div>
    ]
    Step14 --> ExportService

    ExportService --- Step15[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #ff9966 !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>15</span>
            <p style='margin-top: 8px'>Return CSV Download</p>
        </div>
    ]
    Step15 --> API
    API --> Client
    
    style Client fill:#e6f3ff,stroke:#0066cc,stroke-width:2px
    style API fill:#e6f3ff,stroke:#0066cc,stroke-width:2px
    style GCSService fill:#f0f8e6,stroke:#339933,stroke-width:2px
    style ImportService fill:#f0f8e6,stroke:#339933,stroke-width:2px
    style ExportService fill:#f0f8e6,stroke:#339933,stroke-width:2px
    style Repository fill:#fff0f5,stroke:#cc6699,stroke-width:2px
    style Database fill:#ffe6cc,stroke:#ff9900,stroke-width:2px
    style GCS fill:#ffe6cc,stroke:#ff9900,stroke-width:2px
    style Step1 fill:transparent,stroke:transparent,stroke-width:1px
    style Step2 fill:transparent,stroke:transparent,stroke-width:1px
    style Step3 fill:transparent,stroke:transparent,stroke-width:1px
    style Step4 fill:transparent,stroke:transparent,stroke-width:1px
    style Step5 fill:transparent,stroke:transparent,stroke-width:1px
    style Step6 fill:transparent,stroke:transparent,stroke-width:1px
    style Step7 fill:transparent,stroke:transparent,stroke-width:1px
    style Step8 fill:transparent,stroke:transparent,stroke-width:1px
    style Step9 fill:transparent,stroke:transparent,stroke-width:1px
    style Step10 fill:transparent,stroke:transparent,stroke-width:1px
    style Step11 fill:transparent,stroke:transparent,stroke-width:1px
    style Step12 fill:transparent,stroke:transparent,stroke-width:1px
    style Step13 fill:transparent,stroke:transparent,stroke-width:1px
    style Step14 fill:transparent,stroke:transparent,stroke-width:1px
    style Step15 fill:transparent,stroke:transparent,stroke-width:1px

Module List

Name Overview Link Description
Generate Signed URL GCS Upload Generate signed URLs for direct file uploads to Google Cloud Storage
Store Upload History Store History Store metadata about file uploads including status and error information
Upload Histories Upload Histories Retrieve paginated list of upload history records with filtering
List Histories List Histories Get formatted history list with API resource transformation
Get History by ID Get by ID Retrieve detailed information about a specific upload history
Get Import Errors Import Errors Retrieve validation errors for a specific upload history
Download Error CSV Download CSV Download CSV file containing detailed error information

API Endpoints

Method Endpoint Description
GET /api/v1/general/upload-file/gcs/get-signed-url Generate signed URL for direct file upload
POST /api/v1/general/upload-file/gcs/upload-history Store upload history metadata
GET /api/v1/general/upload-file/gcs/histories/list Get paginated list of upload histories
GET /api/v1/general/upload-file/gcs/histories/{id} Get specific upload history by ID
GET /api/v1/general/upload-file/gcs/histories/{id}/import-errors Get import errors for specific history
GET /api/v1/general/upload-file/gcs/histories/{id}/download-errors Download error CSV file

Database Schema

erDiagram
    review_upload_histories {
        bigint id PK
        bigint user_id FK "Reference to users table"
        bigint group_id FK "Reference to groups table"
        string file_name "Name of the uploaded file"
        string gcs_path "Google Cloud Storage path where file is stored"
        tinyInteger status "Upload status: 0=Notyet, 1=processing, 2=success, 3=fail"
        string error_reason "Error message if upload failed (nullable)"
        timestamp validated_at "Timestamp when file was validated (nullable)"
        timestamp created_at
        timestamp updated_at
    }
    review_upload_errors {
        bigint id PK
        bigint review_upload_history_id FK "Reference to review_upload_histories table"
        string header "Field header that caused the error (nullable)"
        string value "Problematic data value (nullable)"
        integer error_line "Line number where error occurred (nullable)"
        json error_messages "Array of error messages for the field"
        timestamp created_at
        timestamp updated_at
    }
    users {
        bigint id PK
        string name "User's full name"
        string email "User's email address"
    }
    groups {
        bigint id PK
        string name "Group name"
    }

    review_upload_histories ||--o{ users : belongs_to
    review_upload_histories ||--o{ groups : belongs_to
    review_upload_histories ||--o{ review_upload_errors : has_many

Key Features

  • Direct Upload Support: Generate signed URLs for direct client-to-GCS uploads
  • Upload Tracking: Comprehensive history tracking with status and error information
  • Error Management: Detailed error tracking with field-level validation errors
  • CSV Export: Download error reports in CSV format for data correction
  • Pagination: Support for large datasets with configurable pagination
  • Group-Based Access: Automatic filtering by user's group for data security
  • Resource Transformation: Consistent API responses using Laravel resources
  • Group-Based Path Structure: Automatic path generation based on user's group and date

Security Considerations

  • Group Isolation: Users can only access upload histories from their own group
  • Signed URL Expiration: Generated URLs expire after 15 minutes
  • Input Validation: All endpoints include comprehensive input validation
  • Error Logging: Failed operations are logged for monitoring and debugging
  • File Path Security: Files are stored in group-specific directories with date-based organization