Google Cloud Storage Upload

Overview Description

The Google Cloud Storage (GCS) upload feature provides a secure and efficient way to upload, download, and manage files in Google Cloud Storage. This feature uses signed URLs to allow direct uploads from clients to GCS, which improves performance by bypassing the application server for file transfers. It also provides methods to list, delete, and get temporary access URLs for files stored in GCS.

Swagger Link

API: Gcs Upload

Activity Diagram

---
config:
  theme: base
  layout: dagre
  flowchart:
    curve: linear
    htmlLabels: true
  themeVariables:
    edgeLabelBackground: "transparent"
---
flowchart TD
    Client[Client Application]
    API[UploadGcsController]
    GCSService[GcsService]
    ImportService[ImportReviewService]
    GCS[(Google Cloud Storage)]
    
    Client --- Step1[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #6699cc !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>1</span>
            <p style='margin-top: 8px'>Request Signed URL</p>
        </div>
    ]
    Step1 --> API

    API --- Step2[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #6699cc !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>2</span>
            <p style='margin-top: 8px'>Generate Signed URL</p>
        </div>
    ]
    Step2 --> ImportService

    ImportService --- Step3[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #6699cc !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>3</span>
            <p style='margin-top: 8px'>Request URL from GCS</p>
        </div>
    ]
    Step3 --> GCSService

    GCSService --- Step4[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #6699cc !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>4</span>
            <p style='margin-top: 8px'>Return Signed URL</p>
        </div>
    ]
    Step4 --> GCS

    GCS --- Step5[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #6699cc !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>5</span>
            <p style='margin-top: 8px'>Return URL to Client</p>
        </div>
    ]
    Step5 --> GCSService
    GCSService --> ImportService
    ImportService --> API
    API --> Client

    Client --- Step6[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #6699cc !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>6</span>
            <p style='margin-top: 8px'>Upload File Directly</p>
        </div>
    ]
    Step6 --> GCS
    
    style Client fill:#e6f3ff,stroke:#0066cc,stroke-width:2px
    style API fill:#e6f3ff,stroke:#0066cc,stroke-width:2px
    style GCSService fill:#f0f8e6,stroke:#339933,stroke-width:2px
    style ImportService fill:#f0f8e6,stroke:#339933,stroke-width:2px
    style GCS fill:#ffe6cc,stroke:#ff9900,stroke-width:2px
    style Step1 fill:transparent,stroke:transparent,stroke-width:1px
    style Step2 fill:transparent,stroke:transparent,stroke-width:1px
    style Step3 fill:transparent,stroke:transparent,stroke-width:1px
    style Step4 fill:transparent,stroke:transparent,stroke-width:1px
    style Step5 fill:transparent,stroke:transparent,stroke-width:1px
    style Step6 fill:transparent,stroke:transparent,stroke-width:1px

Case Documentation

Case 1: Generate Signed URL for Direct Upload

Description

Client requests a signed URL to upload a file directly to Google Cloud Storage.

Sequence Diagram

sequenceDiagram
    participant Client
    participant API as UploadGcsController
    participant Service as ImportReviewService
    participant GCSService as GcsService
    participant GCS as Google Cloud Storage
    
    Note over Client,GCS: Generate Signed URL Flow
    
    rect rgb(200, 255, 200)
    Note right of Client: Happy Case Flow
    
    Client->>API: GET /api/v1/general/upload-file/gcs/get-signed-url
    
    rect rgb(200, 230, 255)
    Note right of API: Input Validation
    API->>API: Validate filename parameter
    end
    
    rect rgb(200, 255, 255)
    Note right of API: Business Logic
    API->>Service: getSignedUrl(filename)
    Service->>Service: Generate path with group and date structure
    Service->>GCSService: getSignedUrl(path, contentType, expiration)
    GCSService->>GCS: Generate signed URL with PUT method
    GCS-->>GCSService: Return signed URL
    GCSService-->>Service: Return URL data with path and expiration
    Service-->>API: Return URL data
    end
    
    API-->>Client: 200 OK (signed URL and metadata)
    end
    
    rect rgb(255, 200, 200)
    Note right of Client: Error Handling
    rect rgb(255, 230, 230)
    alt Validation Error
        API-->>Client: 422 Validation Error
    else GCS Error
        GCS-->>GCSService: Error response
        GCSService-->>Service: Error result
        Service-->>API: Error result
        API-->>Client: 400 Bad Request
    end
    end
    end

Steps

Step 1: Request Signed URL

Description: Client requests a signed URL for uploading a file
Request: GET /api/v1/general/upload-file/gcs/get-signed-url
Parameters:
- filename: The name of the file to upload (required)
Validation:
- Filename must be a string and not exceed 255 characters

Step 2: Generate Signed URL

Description: System generates a signed URL for direct upload
Action:
- Generate path structure: temp/{group_id}/{date}/{filename}
- Set content type to 'text/csv'
- Request a signed URL from Google Cloud Storage with PUT method
- Set the URL to expire after 15 minutes (900 seconds)

Step 3: Return URL to Client

Description: Return the signed URL and metadata to client
Response:
- Success: 200 OK with signed URL data including success flag, URL, path, expiration, and content type
- Error: 400 Bad Request with error message

Error Handling

Log
- URL generation failures logged to application log
Error Detail:

Status Code Error Message Description

422 Validation failed When filename parameter is invalid

400 Failed to generate signed URL When GCS returns an error or service fails

Status Code	Error Message	Description
422	Validation failed	When filename parameter is invalid
400	Failed to generate signed URL	When GCS returns an error or service fails

Case 2: Upload File via Server

Description

Client uploads a file to the server, which then uploads it to Google Cloud Storage.

Sequence Diagram

sequenceDiagram
    participant Client
    participant API as UploadGcsController
    participant Service as GcsService
    participant GCS as Google Cloud Storage
    
    Note over Client,GCS: Server-Side Upload Flow
    
    rect rgb(200, 255, 200)
    Note right of Client: Happy Case Flow
    
    Client->>API: POST /api/v1/general/upload-file/gcs/upload-file
    
    rect rgb(200, 230, 255)
    Note right of API: Input Validation
    API->>API: Validate file and path
    end
    
    rect rgb(200, 255, 255)
    Note right of API: Business Logic
    API->>Service: uploadFile(file, path)
    Service->>Service: Generate unique filename
    Service->>GCS: Upload file
    GCS-->>Service: Confirm upload
    Service-->>API: Return file path and URL
    end
    
    API-->>Client: 200 OK (file path and URL)
    end
    
    rect rgb(255, 200, 200)
    Note right of Client: Error Handling
    rect rgb(255, 230, 230)
    alt Validation Error
        API-->>Client: 422 Validation Error
    else Upload Error
        GCS-->>Service: Error response
        Service-->>API: Error result
        API-->>Client: 500 Internal Server Error
    end
    end
    end

Steps

Step 1: Upload File Request

Description: Client uploads a file to the server
Request: POST /api/v1/general/upload-file/gcs/upload-file
Parameters:
- file: The file to upload (required)
- path: The path where the file will be stored (required)
Validation:
- File must be present and not exceed 10MB
- Path must be a string

Step 2: Process Upload

Description: System uploads the file to GCS
Action:
- Ensure path uniqueness by adding a random string
- Set appropriate content type based on file MIME type
- Upload the file to GCS

Step 3: Return Result

Description: Return the upload result to client
Response:
- Success: 200 OK with file path and public URL
- Error: Appropriate error code with message

Error Handling

Log
- Upload failures logged to application log
Error Detail:

Status Code Error Message Description

422 Validation failed When file or path parameters are invalid

500 Failed to upload file: {error} When GCS returns an error

Status Code	Error Message	Description
422	Validation failed	When file or path parameters are invalid
500	Failed to upload file: {error}	When GCS returns an error

Case 3: List Files in Directory

Description

Client requests a list of files in a specific directory in Google Cloud Storage.

Sequence Diagram

sequenceDiagram
    participant Client
    participant API as UploadGcsController
    participant Service as GcsService
    participant GCS as Google Cloud Storage
    
    Note over Client,GCS: List Files Flow
    
    rect rgb(200, 255, 200)
    Note right of Client: Happy Case Flow
    
    Client->>API: GET /api/v1/general/upload-file/gcs/list-files
    
    rect rgb(200, 230, 255)
    Note right of API: Input Validation
    API->>API: Validate directory and recursive flag
    end
    
    rect rgb(200, 255, 255)
    Note right of API: Business Logic
    API->>Service: listFiles(directory, recursive)
    Service->>GCS: List files in directory
    GCS-->>Service: Return file list
    Service-->>API: Return file list and directory
    end
    
    API-->>Client: 200 OK (file list)
    end
    
    rect rgb(255, 200, 200)
    Note right of Client: Error Handling
    rect rgb(255, 230, 230)
    alt Validation Error
        API-->>Client: 422 Validation Error
    else GCS Error
        GCS-->>Service: Error response
        Service-->>API: Error result
        API-->>Client: 500 Internal Server Error
    end
    end
    end

Steps

Step 1: Request File List

Description: Client requests a list of files in a directory
Request: GET /api/v1/general/upload-file/gcs/list-files
Parameters:
- directory: The directory to list files from (optional, default: root)
- recursive: Whether to list files recursively (optional, default: false)
Validation:
- Directory must be a string
- Recursive must be a boolean

Step 2: Retrieve File List

Description: System retrieves the list of files from GCS
Action:
- Query GCS for files in the specified directory
- Include file metadata (size, last modified, etc.)
- Return formatted file list

Step 3: Return File List

Description: Return the file list to client
Response:
- Success: 200 OK with file list and directory information
- Error: Appropriate error code with message

Error Handling

Log
- File listing failures logged to application log
Error Detail:

Status Code Error Message Description

422 Validation failed When directory or recursive parameters are invalid

500 Failed to list files: {error} When GCS returns an error

Status Code	Error Message	Description
422	Validation failed	When directory or recursive parameters are invalid
500	Failed to list files: {error}	When GCS returns an error

Database Related Tables & Fields

erDiagram
    review_upload_histories {
        bigint id PK
        bigint user_id FK "Reference to users table"
        bigint group_id FK "Reference to groups table"
        string file_name "Name of the uploaded file"
        string gcs_path "Google Cloud Storage path where file is stored"
        tinyInteger status "Upload status: 0=Notyet, 1=processing, 2=success, 3=fail"
        string error_reason "Error message if upload failed (nullable)"
        timestamp validated_at "Timestamp when file was validated (nullable)"
        timestamp created_at
        timestamp updated_at
    }
    users {
        bigint id PK
        string name "User's full name"
        string email "User's email address"
    }
    groups {
        bigint id PK
        string name "Group name"
    }

    review_upload_histories ||--o{ users : belongs_to
    review_upload_histories ||--o{ groups : belongs_to

Key Features

Direct Upload Support: Generate signed URLs for direct client-to-GCS uploads
Group-Based Path Structure: Automatic path generation based on user's group and date
CSV File Support: Optimized for CSV file uploads with proper content type handling
Error Handling: Comprehensive error handling with detailed logging
Security: Signed URLs with configurable expiration times
File Management: Support for file listing and management operations

Security Considerations

Group Isolation: Files are stored in group-specific directories
Signed URL Expiration: Generated URLs expire after 15 minutes
Input Validation: All endpoints include comprehensive input validation
Error Logging: Failed operations are logged for monitoring and debugging

01.gcs-upload

Google Cloud Storage Upload

Overview Description

Swagger Link

Activity Diagram

Case Documentation

Case 1: Generate Signed URL for Direct Upload

Description

Sequence Diagram

Steps

Error Handling

Case 2: Upload File via Server

Description

Sequence Diagram

Steps

Error Handling

Case 3: List Files in Directory

Description

Sequence Diagram

Steps

Error Handling

Database Related Tables & Fields

Key Features

Security Considerations