The Flat Earth Society
Other Discussion Boards => Technology & Information => Topic started by: xasop on October 04, 2014, 04:01:07 PM
-
As some of you may remember (http://theflatearthsociety.org/forum/index.php?topic=51755.msg1338851#msg1338851), some time ago I bought a new computer with 2 TB of storage, plus another 2 TB for backups. Well, two and a half years after that fateful purchase decision, I am now almost out of space.
Just the other day, I bought two 4 TB drives (of the same brand and product line) to replace my existing 2 TB drives. I don't have any spare SATA cables, so I'll have to replace one at a time, but thanks to the magic of Linux LVM I can move data from one drive to the other while the system is running, so I only need to reboot briefly to exchange drives.
I'll also be taking this opportunity, in light of our recent Orwellian legislation (http://forum.tfes.org/index.php?topic=1947.0), to encrypt the new drive before I put any data on it.
Let's get this party started!
-
What in the name of code are you filling your drive up with?
-
What in the name of code are you filling your drive up with?
Lots of things.
The initial drive swap went mostly without a hitch. The only trouble I encountered was that when I yanked a SATA cable out of one of the old drives, the connector snapped clean in half. Nothing a bit of superglue couldn't fix, even if it cost me a few square millimetres of epidermis.
-
Capacity upgrades are so 2000's. In this decade it's all about speed upgrades. If your read/write speed is below 1 GB/s you're doing it wrong.
-
Capacity upgrades are so 2000's. In this decade it's all about speed upgrades. If your read/write speed is below 1 GB/s you're doing it wrong.
It's easy to have both. I have an SSD where low-latency access is required (mainly my OS, git repositories, and frequently-used application data), and HDDs for the bulk of my data which is not speed-critical.
Also, I am now migrating data between drives:
root@vader:~# cryptsetup luksFormat /dev/sdb
WARNING!
========
This will overwrite data on /dev/sdb irrevocably.
Are you sure? (Type uppercase yes): YES
Enter passphrase:
Verify passphrase:
root@vader:~# cryptsetup open --type luks /dev/sdb sdb
Enter passphrase for /dev/sdb:
root@vader:~# pvcreate /dev/mapper/sdb
Physical volume "/dev/mapper/sdb" successfully created
root@vader:~# vgextend data /dev/mapper/sdb
Volume group "data" successfully extended
root@vader:~# pvmove /dev/sdc1
/dev/sdc1: Moved: 0.0%
/dev/sdc1: Moved: 0.1%
Linux is so great.
-
Only plebs have data that is not speed critical. All of you data should be SSD based.
-
Only plebs have data that is not speed critical. All of you data should be SSD based.
You do understand that the primary advantage to solid-state storage is latency and not throughput, right? If you're doing sequential reads, such as for music and movies (which makes up the bulk of my data), an SSD is significantly more expensive for no real benefit.
On the other hand, booting an OS requires reads all over the place to load all the various executables and libraries your system needs, so it makes sense to put that on an SSD.
-
Done and done:
...
/dev/sdc1: Moved: 99.9%
/dev/sdc1: Moved: 100.0%
root@vader:~# vgreduce data /dev/sdc1
Removed "/dev/sdc1" from volume group "data"
I now have 2 TB of free space in my volume group, plus an unused 2 TB hard drive.
-
You do understand that the primary advantage to solid-state storage is latency and not throughput, right? If you're doing sequential reads, such as for music and movies (which makes up the bulk of my data), an SSD is significantly more expensive for no real benefit.
On the other hand, booting an OS requires reads all over the place to load all the various executables and libraries your system needs, so it makes sense to put that on an SSD.
It makes sense to put everything on an SSD. Quieter, less energy usage, less heat. The only reason you're not using an SSD for everything is because HDDs are cheaper. That is literally the only reason.
-
It makes sense to put everything on an SSD. Quieter, less energy usage, less heat. The only reason you're not using an SSD for everything is because HDDs are cheaper. That is literally the only reason.
Well, yes. Is there something wrong with that?
-
Well, yes. Is there something wrong with that?
Yes. You should be ashamed.
-
Imagine how many SSDs you could afford if you didn't waste all that money on whisky.
-
Well, yes. Is there something wrong with that?
Yes.
Do elaborate.
-
Do elaborate.
You can't get something that is objectively better than an HDD because you can't afford it. It's the system keeping you down, man.
Also, Tom Bishop informed me being too poor for things is your fault.
-
You can't get something that is objectively better than an HDD because you can't afford it. It's the system keeping you down, man.
Also, Tom Bishop informed me being too poor for things is your fault.
Even if I could afford it, I wouldn't, because this isn't a case of "it costs too much", it's a case of "a 1 TB SSD costs more than double what I paid for a 4 TB hard drive". That's more than an eight-fold difference in price per unit capacity. That's not worth minor noise, power and heat gains.
In terms of value for money, HDDs do win economically, unless you need low-latency I/O.
-
Do elaborate.
You can't get something that is objectively better than an HDD because you can't afford it.
What makes you think that SSDs are objectively better than HDDs for media storage?
-
What makes you think that SSDs are objectively better than HDDs for media storage?
I've already explained that in previous posts, in this thread, no less. If you're going to try to argue with me, at least bother reading the argument first. I look forward to your thoughts on HDD vs SSD.
In terms of value for money, HDDs do win economically, unless you need low-latency I/O.
I never disputed that. I am just saying you should be ashamed. So, there.
-
What makes you think that SSDs are objectively better than HDDs for media storage?
I've already explained that in previous posts, in this thread, no less. If you're going to try to argue with me, at least bother reading the argument first. I look forward to your thoughts on HDD vs SSD.
First of all, what makes you think I want to argue anything? Am I not allowed to ask you to clarify your assertions?
Secondly, what makes you think that noise, energy and heat are significant concerns for media storage?
-
First of all, what makes you think I want to argue anything? Am I not allowed to ask you to clarify your assertions?
You're really bad at this, Markjo.
Secondly, what makes you think that noise, energy and heat are significant concerns for media storage?
Really, really bad.
-
Guys, please take this to CN or wherever we talk about markjo being terrible these days. This thread is about how awesome Linux is.
-
Guys, please take this to CN or wherever we talk about markjo being terrible these days. This thread is about how awesome Linux is.
I can (and have) upgraded storage capacity rather easily using Windows, and oh, it didn't involve a bunch of command line junk. Windows has this (new?) thing called a GUI that allows users to perform actions without looking up or memorizing a set of arbitrary commands.
-
I can (and have) upgraded storage capacity rather easily using Windows, and oh, it didn't involve a bunch of command line junk. Windows has this (new?) thing called a GUI that allows users to perform actions without looking up or memorizing a set of arbitrary commands.
Does Windows allow you to move all your existing files from one drive to another while you're using them as normal?
-
Does Windows allow you to move all your existing files from one drive to another while you're using them as normal?
Yes, unless you're using a third party application (like Intel's drive migration utility) in which case Windows thinks the files are in use by another application.
-
Does Windows allow you to move all your existing files from one drive to another while you're using them as normal?
Yes[citation needed]
-
Yes[citation needed]
Considering I've done it before, no, one is not needed.
-
Yes[citation needed]
Considering I've done it before, no, one is not needed.
What is different about Intel's drive migration utility from any other application that might be using the files? Why will Windows let you move files while in use by one application, but not another?
-
What is different about Intel's drive migration utility from any other application that might be using the files? Why will Windows let you move files while in use by one application, but not another?
Windows allows applications (through some mechanism, I am unaware of the specifics) to lock out files as "in use by application." For example, when Microsoft office has an Excel file open and it doesn't want you touching it, it locks the file out. This is opposed to an Access database that can have hordes of people all accessing the same thing. Some applications lock out pretty much entire folders, some lock down just a few files, some don't lock anything at all.
-
Windows allows applications (through some mechanism, I am unaware of the specifics) to lock out files as "in use by application." For example, when Microsoft office has an Excel file open and it doesn't want you touching it, it locks the file out. This is opposed to an Access database that can have hordes of people all accessing the same thing. Some applications lock out pretty much entire folders, some lock down just a few files, some don't lock anything at all.
So you can't just move the entire filesystem from one device to another while using it as normal, if you happen to use an application that locks files. Good to know.
The migration I just did was done at the block level. Individual files were irrelevant, since the filesystem itself is above the layer at which data got migrated. Also, the paths to the files didn't change, since the files themselves stayed on the same filesystem. The physical location of the data is completely separate from logical file access.
-
So you can't just move the entire filesystem from one device to another while using it as normal, if you happen to use an application that locks files. Good to know.
Well, yes... I thought I've already addressed that. Were you confused by my original answer?
The migration I just did was done at the block level. Individual files were irrelevant, since the filesystem itself is above the layer at which data got migrated. Also, the paths to the files didn't change, since the files themselves stayed on the same filesystem. The physical location of the data is completely separate from logical file access.
I thought we were talking about drive migration (you even use the term "migrated"). It sounds like you just expanded your drive, not migrated it. I don't know if Linux treats those as separate things, but Windows can do either.
-
Well, yes... I thought I've already addressed that. Were you confused by my original answer?
I was, since you opened with "yes" and then clarified that the answer is "no".
I thought we were talking about drive migration (you even use the term "migrated"). It sounds like you just expanded your drive, not migrated it. I don't know if Linux treats those as separate things, but Windows can do either.
I migrated data from one drive to another. There was no expansion involved; at the end of the process, my original drive was empty and I could remove it.
-
I've just finished a first pass of wiping my original drive with random data, since it had unencrypted files on it. I'll do a second pass later this week, but for now, I'm going to swap in my new 4 TB backup drive and get backups configured.
-
I'm now doing an initial backup with dirvish, which is a far superior backup program to the one I used previously, rdiff-backup. rdiff-backup is basically a Python script that stores binary diffs between backup increments, so you have to wait approximately three billion years for it to unravel its own diffs in order to restore to anything but the most recent backup.
Dirvish uses filesystem snapshots instead (usually hard-linked trees, but I'm using a patch that allows it to make use of btrfs copy-on-write snapshots for better efficiency), which means that each backup increment just appears as a directory on my filesystem. I can browse each increment as I would any other filesystem tree, and even trivially grep across multiple historical snapshots if I'm looking for something specific.
The other really nice thing about dirvish is that it's very easy to specify custom snapshot expiry. Previously, I kept backups going back 90 days, but I think now that I have dirvish I'll keep some backups (say, one increment per month) forever. October 2014 will be the first month I always have backups of.
-
I've just finished a first pass of wiping my original drive with random data, since it had unencrypted files on it. I'll do a second pass later this week...
Have you considered using the ATA Secure Erase command? Seems a whole lot quicker and just as secure.
https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase
-
I was, since you opened with "yes" and then clarified that the answer is "no".
You asked if Windows can move files while they're in use [by Windows], the answer is yes. I then clarified that if a third party application is using the files, the answer is no.
I migrated data from one drive to another. There was no expansion involved; at the end of the process, my original drive was empty and I could remove it.
Well, since you use Linux I suppose it might treat it differently, but Windows has two options: you expand the original drive and move data between physical locations (e.g. you expand the C:\ drive by X amount) or you create a new drive D:\ and move the data between drive designations. The former option isn't something I'd refer to as a drive migration.
-
Have you considered using the ATA Secure Erase command? Seems a whole lot quicker and just as secure.
https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase
What makes you think that would be any quicker?
You asked if Windows can move files while they're in use [by Windows], the answer is yes. I then clarified that if a third party application is using the files, the answer is no.
No, I asked if Windows could move files while you're using them as normal. Is your normal usage pattern to boot up an OS and sit there staring at your desktop without starting any applications?
Well, since you use Linux I suppose it might treat it differently, but Windows has two options: you expand the original drive and move data between physical locations (e.g. you expand the C:\ drive by X amount) or you create a new drive D:\ and move the data between drive designations. The former option isn't something I'd refer to as a drive migration.
When I said "drive" I meant, well, a hard drive. I'm not surprised you're confused if you consider "drive" to mean "logical volume" and/or "filesystem".
-
Have you considered using the ATA Secure Erase command? Seems a whole lot quicker and just as secure.
https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase
What makes you think that would be any quicker?
For one thing, it only requires one pass.
-
No, I asked if Windows could move files while you're using them as normal. Is your normal usage pattern to boot up an OS and sit there staring at your desktop without starting any applications?
Like I said, it depends entirely on the application. Most don't lock out anything. If it is too much trouble you could always just copy and paste instead of directly moving the data.
When I said "drive" I meant, well, a hard drive. I'm not surprised you're confused if you consider "drive" to mean "logical volume" and/or "filesystem".
Words have multiple meanings. Shocking, I know.
-
For one thing, it only requires one pass.
I wouldn't trust any magnetic storage erasure mechanism that claims to require only one pass.
Like I said, it depends entirely on the application. Most don't lock out anything. If it is too much trouble you could always just copy and paste instead of directly moving the data.
So you lose any writes that are done by said applications after copying? What happens if they write something to the file while it's being copied? Do you get a consistent copy?
Words have multiple meanings. Shocking, I know.
Irrelevant, unless you're claiming to have invented a new one for "drive" without telling anyone, in which there's no point continuing to communicate with you.
-
So you lose any writes that are done by said applications after copying?
Yes.
What happens if they write something to the file while it's being copied?
Nothing.
Do you get a consistent copy?
Yes.
Irrelevant, unless you're claiming to have invented a new one for "drive" without telling anyone, in which there's no point continuing to communicate with you.
The drive is simply the file location in Windows. (e.g. C:\, D:\, E:\ etc.) it doesn't have to refer to a physical drive. A physical hard disk can have multiple partitions, each with their own drive mapping. At this point it just feels like you're being purposefully obtuse which is rather obnoxious. You're the only Linux person I know of that really, truly, doesn't know how Windows works, which leads me to believe you're just being a dolt on purpose.
-
Uhhhh... Rushy? I don't quite understand what you're talking about.
You can't expand a drive without free(unpartitioned) space on said drive and while windows does allow drive spanning, it requires all drives to be blank and unpartitioned first.
As for copying from one to another:
Yes its possible. I've done it myself though its not a windows function but a 3rd party program. Xcopy won't make your new HDD usable even if it does copy most of the data.
-
So you lose any writes that are done by said applications after copying?
Yes.
That's hardly equivalent, or even comparable.
What happens if they write something to the file while it's being copied?
Nothing.
Really? Nothing at all? I would expect, at the very least, that the OS would perform the write on the original copy of the file.
Do you get a consistent copy?
Yes.
How does the OS ensure consistency if an application writes to the file while it is being copied?
The drive is simply the file location in Windows. (e.g. C:\, D:\, E:\ etc.) it doesn't have to refer to a physical drive. A physical hard disk can have multiple partitions, each with their own drive mapping. At this point it just feels like you're being purposefully obtuse which is rather obnoxious. You're the only Linux person I know of that really, truly, doesn't know how Windows works, which leads me to believe you're just being a dolt on purpose.
You brought Windows into this discussion, not me. It is not reasonable to assume that I was using a Windows-specific definition of a word in a thread I made about how great Linux is.
-
For one thing, it only requires one pass.
I wouldn't trust any magnetic storage erasure mechanism that claims to require only one pass.
According to Internets, the whole "more than one pass is necessary" thing is based off of an unfounded hypothetical and one pass is all that's necessary.
-
The conspiracy can examine your hard drive and determine which magnetic highs and lows are stronger and weaker and then work out the actual data if you only do one pass. Please learn to conspiracy.
-
The only proper way to permanently delete files on your hard drive is to burn it. When it cools, the magnetism on the disk will align with the earth's magnetic field and permanently erase all the data.
-
According to Internets, the whole "more than one pass is necessary" thing is based off of an unfounded hypothetical and one pass is all that's necessary.
It's neither unfounded, nor a hypothetical. It's a simple statement of how magnetism works.
Depending on your goals, more than one pass may or may not be necessary. If you don't want me to be able to read your data with my laptop, one pass is easily good enough; but if I was a forensic scientist who wears a suit, a lab coat (simultaneously) and sunglasses, and if I had a budget of tens of thousands of dollars, it would no longer be good enough.
-
...but if I was a forensic scientist who wears a suit, a lab coat (simultaneously) and sunglasses, and if I had a budget of tens of thousands of dollars, it would no longer be good enough.
Citation please.
Daniel Feenberg, an economist at the private National Bureau of Economic Research, claims that the chances of overwritten data being recovered from a modern hard drive amount to "urban legend".[3] He also points to the "18½ minute gap" Rose Mary Woods created on a tape of Richard Nixon discussing the Watergate break-in. Erased information in the gap has not been recovered, and Feenberg claims doing so would be an easy task compared to recovery of a modern high density digital signal.
-
...but if I was a forensic scientist who wears a suit, a lab coat (simultaneously) and sunglasses, and if I had a budget of tens of thousands of dollars, it would no longer be good enough.
Citation please.
Daniel Feenberg, an economist at the private National Bureau of Economic Research, claims that the chances of overwritten data being recovered from a modern hard drive amount to "urban legend".[3] He also points to the "18½ minute gap" Rose Mary Woods created on a tape of Richard Nixon discussing the Watergate break-in. Erased information in the gap has not been recovered, and Feenberg claims doing so would be an easy task compared to recovery of a modern high density digital signal.
Ontrack makes a living doing this. They can easily recover data from a single pass. I've had to recover data form a drive wiped for HIPAA compliance, and it wasn't any trouble at all for them. They can also recover data from RAID sets using proprietary striping algorithms in 3rd party storage solutions (EMC, Netapp, etc.).
-
The real question is what the hell are you people doing that requires such thorough cleansing of your hard drives. If I gave my hard drive over to the NSA right now, completely intact, the most they would notice is I have a nigh inconceivable amount of science fiction games.
-
The real question is what the hell are you people doing that requires such thorough cleansing of your hard drives. If I gave my hard drive over to the NSA right now, completely intact, the most they would notice is I have a nigh inconceivable amount of science fiction games.
I keep logs of all my private IM conversations since 2008 archived permanently, as well as credentials for various online services, including FES. Do you really want the NSA reading all my chats with pizaaplanet about running this website and then using them to impersonate me with my own account?
For all you know, Barack Obama could be typing this post right now, trying to lead you off the trail.
-
For all you know, Barack Obama could be typing this post right now, trying to lead you off the trail.
Let me be clear: this is currently not the case.
-
For all you know, Barack Obama could be typing this post right now, trying to lead you off the trail.
Let me be clear: this is currently not the case.
Good job B. :]
-
I keep logs of all my private IM conversations since 2008 archived permanently, as well as credentials for various online services, including FES. Do you really want the NSA reading all my chats with pizaaplanet about running this website and then using them to impersonate me with my own account?
Ego much?
-
Ego much?
Given that widespread surveillance of random citizens has already been shown to be fact, it's hardly a matter of ego.
-
Ego much?
Given that widespread surveillance of random citizens has already been shown to be fact, it's hardly a matter of ego.
Should the NSA/conspiracy really care what you guys talk about?
-
Should the NSA/conspiracy really care what you guys talk about?
They shouldn't, but we already know they do.
-
What are you hiding, terrorist?