The instance in turn is within an ASG to be resilient against instance failure. So, this approach would also have required further workarounds.Īs a result, we had to run our own FTP server in an EC2 instance. But even if that would have been an option at the time, the FTP version is only available inside a VPC (including e.g. Later on Amazon has added “Transfer for FTP”. And sadly, the interface we must provide has to have a non-S FTP option. At the time of evaluating the service though, “Transfer for SFTP” was the only available option. Not only does that sound too good to be true, it would also make for a rather short and anti-climactic article. Options Considered TransferĪ fully managed FTP server that is seamlessly backed by an S3 bucket. We share these key lessons on this article. Making it an “AWS-native” to integrate it with the new HTTP API using AWS infrastructure was not exactly straightforward and many lessons were learnt. The challenges in this setup lie not merely in running a resilient FTP server in AWS. So, whilst the files are rather small, usually less than 10 kB, each transfer causes a considerable amount of compute, from handling the FTP connection to notifying several involved subsystems. Under the hood said FTP server should trigger transformation of uploaded files to the format of the new API and then, either push them to the new API, or use a notification mechanism like SQS to mark them for pickup from a shared location. At the same time, the system has to hold historical data, which spans across millions of files. These are comprised of over 50,000 uploads, usually in the form of listings, and up to 50,000 downloads, usually in the form of customer-facing logs. Performance-wise, the system has to handle an average of 100,000 transfers per day. To get out of the data centre and avoid running two systems, we decided to operate an FTP server in AWS. The legacy FTP solution however was still running in our data centre, which was about to shutdown. At the same time, we need to support the legacy alternative in the form of an FTP server.įor quite a while REA has been building its systems in AWS. In the case at hand, as of beginning of 2020 REA is providing a new HTTP API for uploading listings. When re-designing customer-facing APIs, having a smooth migration path is rarely an avoidable topic.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |