[AusNOG] Australian based cloud storage
Brad Eckert
brad at nimbus.net.au
Mon Oct 26 07:16:49 EST 2020
Morning,
Look at rclone <https://rclone.org/#about>
It supports 40+ cloud backends. Details below.
Rclone really looks after your data. It preserves timestamps and verifies
> checksums at all times. Transfers over limited bandwidth; intermittent
> connections, or subject to quota can be restarted, from the last good file
> transferred. You can check the integrity of your files. Where possible,
> rclone employs server side transfers to minimise local bandwidth use and
> transfers from one provider to another without using local disk.
>
> Virtual backends wrap local and cloud file systems to apply encryption,
> caching, chunking and joining.
>
> Rclone mounts any local, cloud or virtual filesystem as a disk on Windows,
> macOS, linux and FreeBSD, and also serves these over SFTP, HTTP, WebDAV,
> FTP and DLNA.
>
> Rclone is mature, open source software originally inspired by rsync and
> written in Go. The friendly support community are familiar with varied use
> cases. Official Ubuntu, Debian, Fedora, Brew and Chocolatey repos. include
> rclone. For the latest version downloading from rclone.org is recommended.
>
> Rclone is widely used on Linux, Windows and Mac. Third party developers
> create innovative backup, restore, GUI and business process solutions using
> the rclone command line or API.
>
> Rclone does the heavy lifting of communicating with cloud storage.
> What can rclone do for you?
>
> Rclone helps you:
>
> Backup (and encrypt) files to cloud storage
> Restore (and decrypt) files from cloud storage
> Mirror cloud data to other cloud services or locally
> Migrate data to cloud, or between cloud storage vendors
> Mount multiple, encrypted, cached or diverse cloud storage as a disk
> Analyse and account for data held on cloud storage using lsf, ljson,
> size, ncdu
> Union file systems together to present multiple local and/or cloud
> file systems as one
>
> Features
>
> Transfers
> MD5, SHA1 hashes are checked at all times for file integrity
> Timestamps are preserved on files
> Operations can be restarted at any time
> Can be to and from network, eg two different cloud providers
> Can use multi-threaded downloads to local disk
> Copy new or changed files to cloud storage
> Sync (one way) to make a directory identical
> Move files to cloud storage deleting the local after verification
> Check hashes and for missing/extra files
> Mount your cloud storage as a network disk
> Serve local or remote files over HTTP/WebDav/FTP/SFTP/dlna
> Experimental Web based GUI
>
>
>
Regards,
Brad
Regards,
Brad
On Sun, 25 Oct 2020 at 14:59, James Hodgkinson <yaleman at ricetek.net> wrote:
> Pretty sure the user stuff gets parsed, like google drive and photos - I'd
> be very surprised if they messed with the object/disk storage...
>
> James
>
> On 2020-10-25 14:42 Matthew Scutter wrote:
>
> Going to call a [citation needed] on that, because it reeks of FUD to me.
>
> On Sun, Oct 25, 2020 at 1:11 PM Kai <vk6ksj at westnet.com.au> wrote:
>
> Hi folks,
>
> Thank you for all the feedback, greatly appreciated.
>
> I read an article yesterday which said that in the same way that GMail
> parses email content for AI learning and targeting ads, that Google
> cloud storage may also index content, including facial recognition, with
> any photo's, for behaviour learning and targeted ads.
> That's one of the reasons I'm not keen to store data on Google cloud.
> They're on a need-to-know basis with my activities.
>
> Does anyone know if Microsoft, AWS or other providers may allow
> themselves access to stored files?
>
> On 25/10/20 10:25 am, Jacob Taylor wrote:
> > An important question to ask is whether you require file storage or
> > object storage.
> >
> > While the S3 portal provides a veneer of a filesystem-style hierarchy,
> > S3 is really a key-value object store. If you build an application on S3
> > but use filesystem-style queries ("list all files in a directory" as an
> > example), it can end up being very costly.
> >
> > If you just want a place to upload big files, such as backups, VMs,
> > images, and videos, then S3 is ideal.
> >
> > If you are looking for something that requires a file hierarchy, then it
> > might not be appropriate.
> >
> > To reiterate what Shaun says, the data you put in an S3 bucket mastered
> > in the Sydney region (ap-southeast-2) *will not be stored elsewhere*
> > unless you explicitly want it to (via cross-region replication or other
> > sync methods).
> >
> > To go into more detail on encryption options:
> >
> > * Server Side Encryption (SSE): Encryption is done in the S3 service
> > itself, you upload/download in plaintext. Comes in a few flavours,
> > but they all use the same algorithm (AES-256):
> > o SSE-S3: This is the simplest and easiest to use, basically
> > turnkey. S3 will use an AWS-managed key in KMS to encrypt your
> > files.
> > o SSE-KMS: Same as above, however it uses a custom key you manage
> > (could be generated on-prem and uploaded, as an example).
> > o SSE-C: Encrypts files with a key given to S3 by your application
> > at the time of upload, and you cannot download the file without
> > providing the same key at the time you request it.
> > * Client Side Encryption: Describes any scenario where your
> > application encrypts a file prior to uploading, and decrypts after
> > downloading.
> >
> > Disclaimer: I work for AWS
> >
> > On Sun, Oct 25, 2020 at 12:34 PM Shaun Ewing <shaun at shaun.net
> > <mailto:shaun at shaun.net>> wrote:
> >
> > Data uploaded to S3 will stay entirely within a region unless you
> > explicitly configure cross-region replication.____
> >
> > __ __
> >
> > There’s a bunch of encryption options including Amazon S3-Managed
> > Keys and customer provided keys.____
> >
> > __ __
> >
> > (Disclosure: I work for AWS)____
> >
> > __ __
> >
> > *From:*AusNOG <ausnog-bounces at lists.ausnog.net
> > <mailto:ausnog-bounces at lists.ausnog.net>> *On Behalf Of *Giles
> Pollock
> > *Sent:* Sunday, 25 October 2020 12:08 PM
> > *To:* Kai <vk6ksj at westnet.com.au <mailto:vk6ksj at westnet.com.au>>
> > *Cc:* Ausnog <ausnog at lists.ausnog.net <mailto:
> ausnog at lists.ausnog.net>>
> > *Subject:* Re: [AusNOG] Australian based cloud storage____
> >
> > __ __
> >
> > Amazon have a region, ap-southeast-2 which is Sydney based. Can't
> > comment whether stuff that goes into s3 gets replicated elsewhere, I
> > believe you can set the class so it doesn't, but you'd need to talk
> > to someone at AWS to confirm. ____
> >
> > __ __
> >
> > On Sun, 25 Oct 2020, 12:02 Kai, <vk6ksj at westnet.com.au
> > <mailto:vk6ksj at westnet.com.au>> wrote:____
> >
> > Hi folks,
> >
> > Happy weekend.
> > I'm searching for Australian based cloud storage.
> >
> > Google, Microsoft and the other big names might have cache
> > server here
> > but the data is also stored overseas, I'm looking for providers
> who
> > either allow you to choose your cloud storage location, or only
> > have
> > hosting within Australia, and have storage which is encrypted.
> >
> > Any feedback is welcome.
> >
> > Cheers
> > Kai
> > _______________________________________________
> > AusNOG mailing list
> > AusNOG at lists.ausnog.net <mailto:AusNOG at lists.ausnog.net>
> > http://lists.ausnog.net/mailman/listinfo/ausnog____
> >
> > _______________________________________________
> >
> > ____
> >
> > AusNOG mailing list
> >
> > ____
> >
> > AusNOG at lists.ausnog.net <mailto:AusNOG at lists.ausnog.net>
> >
> > ____
> >
> > http://lists.ausnog.net/mailman/listinfo/ausnog
> >
> > ____
> >
> > _______________________________________________
> > AusNOG mailing list
> > AusNOG at lists.ausnog.net <mailto:AusNOG at lists.ausnog.net>
> > http://lists.ausnog.net/mailman/listinfo/ausnog
> >
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
>
>
> _______________________________________________
> AusNOG mailing list
> AusNOG at lists.ausnog.net
> http://lists.ausnog.net/mailman/listinfo/ausnog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ausnog.net/pipermail/ausnog/attachments/20201026/26a1e5e4/attachment.html>
More information about the AusNOG
mailing list