For handling large datasets in a Mule application, streaming is an effective strategy. Streaming allows the application to process large amounts of data in chunks, reducing memory usage and improving performance.
Using a file storage repeatable strategy for reading records from the database ensures that the data is read in manageable chunks and stored temporarily in files, which can be re-read if necessary, enhancing reliability.
The batch aggregator with streaming to write to an FTPS server ensures that data is written in chunks, aligning with the processing capabilities of the application running in CloudHub with 0.2 vCore and 8 GB storage. This configuration optimizes performance by balancing the load and managing the dataset size effectively, ensuring that the high rate of records can be processed and written to the FTPS server without overwhelming the system.
References:
* MuleSoft Documentation on Streaming
* MuleSoft Documentation on Batch Processing