Wishlist Dataset GCS Path CSV Download Module

Description

This module manages the process of downloading CSV data from wishlist datasets and storing files on Google Cloud Storage (GCS). When a user requests a CSV download, the system creates a record with status "new", then a batch job processes the request by fetching review sentences in chunks of 1000 records, updating the status to "completed" and sending a notification when finished. If there's an error, the system updates the status to "failed" and logs the error message. The batch job runs every 5 minutes.

Overview Activity Diagram

---
config:
  theme: base
  layout: dagre
  flowchart:
    curve: linear
    htmlLabels: true
  themeVariables:
    edgeLabelBackground: "transparent"
---
flowchart TB
    %% Main components
    User[User]
    API[API Gateway]
    Controller[ReviewDownloadController]
    Service[ProductAnalyzerService]
    Database[(wishlist_dataset_gcs_path)]
    BatchJob[Batch Job - Backend]
    GCS[Google Cloud Storage]
    Notification[Notification Service]
    
    subgraph Controllers
        ReviewDownloadController[ReviewDownloadController]
    end
    
    subgraph Services
        ProductAnalyzerService(ProductAnalyzerService)
    end
    
    subgraph Models
        WishlistDatasetGcsPath[[WishlistDatasetGcsPath]]
    end
    
    subgraph External
        GCS((Google Cloud Storage))
        Notification((Notification Service))
    end
    
    User --- Step1[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #6699cc !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>1</span>
            <p style='margin-top: 8px'>Request CSV Download</p>
        </div>
    ]
    Step1 --> API

    API --- Step2[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #6699cc !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>2</span>
            <p style='margin-top: 8px'>Route to Controller</p>
        </div>
    ]
    Step2 --> Controller

    Controller --- Step3[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #6699cc !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>3</span>
            <p style='margin-top: 8px'>Call Service</p>
        </div>
    ]
    Step3 --> Service

    Service --- Step4[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #6699cc !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>4</span>
            <p style='margin-top: 8px'>Create Record with Status New</p>
        </div>
    ]
    Step4 --> Database

    Database --- Step5[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #6699cc !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>5</span>
            <p style='margin-top: 8px'>Return Success Response</p>
        </div>
    ]
    Step5 --> Service

    Service --- Step6[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #6699cc !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>6</span>
            <p style='margin-top: 8px'>Return Success Response</p>
        </div>
    ]
    Step6 --> Controller

    Controller --- Step7[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #6699cc !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>7</span>
            <p style='margin-top: 8px'>Return HTTP Response</p>
        </div>
    ]
    Step7 --> API

    API --- Step8[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #6699cc !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>8</span>
            <p style='margin-top: 8px'>Send Response to User</p>
        </div>
    ]
    Step8 --> User

    %% Batch Job Process (Every 5 minutes)
    BatchJob --- Step9[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #99cc66 !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>9</span>
            <p style='margin-top: 8px'>Check New Records Every 5 min</p>
        </div>
    ]
    Step9 --> Database

    Database --- Step10[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #99cc66 !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>10</span>
            <p style='margin-top: 8px'>Return New Records</p>
        </div>
    ]
    Step10 --> BatchJob

    BatchJob --- Step11[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #99cc66 !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>11</span>
            <p style='margin-top: 8px'>Update Status to Processing</p>
        </div>
    ]
    Step11 --> Database

    BatchJob --- Step12[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #99cc66 !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>12</span>
            <p style='margin-top: 8px'>Fetch Review Sentences Chunk 1000</p>
        </div>
    ]
    Step12 --> Database

    Database --- Step13[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #99cc66 !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>13</span>
            <p style='margin-top: 8px'>Return Review Data</p>
        </div>
    ]
    Step13 --> BatchJob

    BatchJob --- Step14[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #99cc66 !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>14</span>
            <p style='margin-top: 8px'>Generate CSV File</p>
        </div>
    ]
    Step14 --> GCS

    GCS --- Step15[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #99cc66 !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>15</span>
            <p style='margin-top: 8px'>Return GCS Link</p>
        </div>
    ]
    Step15 --> BatchJob

    BatchJob --- Step16[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #99cc66 !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>16</span>
            <p style='margin-top: 8px'>Update Status to Completed</p>
        </div>
    ]
    Step16 --> Database

    BatchJob --- Step17[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #99cc66 !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>17</span>
            <p style='margin-top: 8px'>Send Success Notification</p>
        </div>
    ]
    Step17 --> Notification

    %% Error Handling
    BatchJob --- Step18[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #cc6666 !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>18</span>
            <p style='margin-top: 8px'>Handle Error</p>
        </div>
    ]
    Step18 --> Database

    Database --- Step19[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #cc6666 !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>19</span>
            <p style='margin-top: 8px'>Update Status to Failed</p>
        </div>
    ]
    Step19 --> BatchJob

    BatchJob --- Step20[
        <div style='text-align: center'>
            <span style='display: inline-block; background-color: #cc6666 !important; color:white; width: 28px; height: 28px; line-height: 28px; border-radius: 50%; font-weight: bold'>20</span>
            <p style='margin-top: 8px'>Send Error Notification</p>
        </div>
    ]
    Step20 --> Notification

    %% Styling
    style User fill:#e6f3ff,stroke:#0066cc,stroke-width:2px
    style API fill:#e6f3ff,stroke:#0066cc,stroke-width:2px
    style Controller fill:#e6f3ff,stroke:#0066cc,stroke-width:2px
    style Service fill:#f0f8e6,stroke:#339933,stroke-width:2px
    style Database fill:#ffe6cc,stroke:#ff9900,stroke-width:2px
    style BatchJob fill:#f0f8e6,stroke:#339933,stroke-width:2px
    style GCS fill:#fcd9d9,stroke:#cc3333,stroke-width:2px
    style Notification fill:#fcd9d9,stroke:#cc3333,stroke-width:2px
    style Controllers fill:#e6f3ff
    style Services fill:#f0f8e6
    style Models fill:#fff0f5
    style External fill:#fcd9d9
    style Step1 fill:transparent,stroke:transparent,stroke-width:1px
    style Step2 fill:transparent,stroke:transparent,stroke-width:1px
    style Step3 fill:transparent,stroke:transparent,stroke-width:1px
    style Step4 fill:transparent,stroke:transparent,stroke-width:1px
    style Step5 fill:transparent,stroke:transparent,stroke-width:1px
    style Step6 fill:transparent,stroke:transparent,stroke-width:1px
    style Step7 fill:transparent,stroke:transparent,stroke-width:1px
    style Step8 fill:transparent,stroke:transparent,stroke-width:1px
    style Step9 fill:transparent,stroke:transparent,stroke-width:1px
    style Step10 fill:transparent,stroke:transparent,stroke-width:1px
    style Step11 fill:transparent,stroke:transparent,stroke-width:1px
    style Step12 fill:transparent,stroke:transparent,stroke-width:1px
    style Step13 fill:transparent,stroke:transparent,stroke-width:1px
    style Step14 fill:transparent,stroke:transparent,stroke-width:1px
    style Step15 fill:transparent,stroke:transparent,stroke-width:1px
    style Step16 fill:transparent,stroke:transparent,stroke-width:1px
    style Step17 fill:transparent,stroke:transparent,stroke-width:1px
    style Step18 fill:transparent,stroke:transparent,stroke-width:1px
    style Step19 fill:transparent,stroke:transparent,stroke-width:1px
    style Step20 fill:transparent,stroke:transparent,stroke-width:1px

Module List