开发者

Apache Camel pull data from ftp incrementally & periodically

开发者 https://www.devze.com 2023-02-26 05:40 出处:网络
I am very new to Apache camel and I am exploring how to create a rout which pulls data from ftp for instance each 15 minutes and pulls only new or updated files, so if some files were downloaded early

I am very new to Apache camel and I am exploring how to create a rout which pulls data from ftp for instance each 15 minutes and pulls only new or updated files, so if some files were downloaded early and still the same (unchanged) ftp loader should not load them to the destination folder.

Any advices are warmly appreciated.

UPDATE #1

I've already noticed that I need to look at the FTP2, and actually I've already made a progress, the las开发者_如何转开发t thing that I want to clarify: consumer.dealy defines delay between each download attempt, for instance consumer.delay = 5s and at the first attempt ftp contains 5 files, consumer pulls data to somewhere and waites 5s at the second attempt ftp still the same and camel just does nothing, after that to ftp arrives additional 5 files and after 5 seconds ftp consumer downloads these just arrived new files or consumer.delay just makes consumer wait between each download of file (file#1 -> 5s -> file#2 -> 5s -> etc...)

I want to achieve first scenario.

Also, I observed that once some files were downloaded to the destination folder, I mean from ftp to local file system, this files will be ignored in subsequent data loads, even if this files were deleted on the local file system, how I can tell to camel to download again deleted files, how it stores information about already loaded files? And it seems that it downloads all files each time even files were downloaded at first data pull. Do I need to write a filter to exclude already downloaded files?


there is FTP component for apache camel http://camel.apache.org/ftp.html

use "consumer.delay" property to pull data for delay in milliseconds between each poll.

for implementation details look here http://architects.dzone.com/articles/apache-camel-integration

0

精彩评论

暂无评论...
验证码 换一张
取 消