With the completion of the 1.0.1 release, we have pretty good code foundation on which to add additional features.  We also have a 1.1.0 open source release coming up which will address a number of issues – particularly performance on globally dispersed (i.e. high latency) networks.

As we continue add new features, I wanted to open up the conversation for suggestions.  So, let us know if you have any ideas or suggestions for new capabilities to add to the Dispersed Storage software.  We’re currently in the midst of a lot of future release planning, so the timing is good for input.

Chris

 As the size of Dispersed Storage Networks in operation grow larger, what is the optimum maximum number of Slices for each data element stored on the dsNet?

The “Width” of a “Vault” on a dsNet determines the number of dispersed Slices that are physically stored for each data element in that vault.  For example, a typical block interface typically uses 4K blocks, so the dsNet block interface presents an interface that stores and retrieves 4K blocks.  (The dsNet iSCSI target uses this block interface.)

When storing a 4K block, the dsNet creates a number of slices equal to the width set for the associated vault on the dsNet.  Each of these slices are then typically stored on different Slicestors.  Most of the initial dsNets used 8 Slicestors and set the width of each vault to 8.  Although the information dispersal algorithms we use can mathematically support any width (and our implementation can be configured with widths up to 256), the initial commercial release only supported 8 as the width since that was a width for which we had completed comprehensive testing.  We’re now beginning comprehensive testing with 16 wide dsNets and then expect to test wider widths:  32, 64, maybe 128 or higher.

In smaller dsNets, it is common for the width of the vault(s) on the dsNet to equal the number of Slicestors, but as dsNets grow the number of Slicestors will exceed the width of any vault on that dsNet.

Many dsNets will store and distribute at least Petabytes of data using more thousands or more Slicestors and will often use different widths (and threshold settings) for various vaults stored on those dsNets.  We expect that the widths for those vaults will vary from 8 to 64 or perhaps 128, but will be significantly less than the total number of Slicestors on the dsNet.

Chris

Wow it’s been a long time since our last release, and as you can imagine, we’ve been busy. With our release candidate 1.0, you should experience stable software. The core software comes from our commercial release, which has been tested extensively. We mainly tested a dsNet with a width of 8, and a threshold of 6, meaning, you can experience 2 simultaneous unavailable Slicestors, and still bit-perfectly recreate your data.

Two areas that are improved:

• iSCSI target stabilization.  We now have windows support as well.

• Adverse network handling. Your network is up, your network is down. We’ve worked diligently to elegantly handle unavailability scenarios while still providing seamless access to your data.

Personally, and someone who’s been here from the start (January 2005), it is very exciting to see our technology stabilize and become available through open source.

We recommend you try this release for yourself. Please let us know how it works for you.

Thanks!

As a comment to my post last month about hard drive speeds, Waldo made a comment about the potential impact of Solid State Drives (SSDs) to change the mainstream data storage paradigm.  I agree that SSDs have significant potential and will become increasingly important.  As computers become more ubiquitous:  more transportable, more hold-able, more wearable, etc., the benefits of solid state storage – lighter, smaller, more durable – will become more relevant.  In addition, SSDs are much faster that hard drives.

However, at the same time people will are using more and more data which will further expose the weakness of SSD:  their cost per unit storage is significantly higher that hard drives.

So how can you have the best of both worlds?  Imagine a mobile device, like a mobile phone or a laptop, with a SSD (or just a lot of solid state storage).  Now imagine that mobile device also has a high speed wireless data connection to a Dispersed Storage network.  In this approach, the SSD is a fast, huge storage cache while the dsNet provides massive, reliable secondary storage.  With this approach, you get fast access in a small, light and durable device while having access to unlimited storage on a dsNet.  And if you lost your mobile device, you could connect a new mobile device (or two or five) to you data on the dsNet and never lose a bit.

The pieces to put this together will all be available this year.  Anyone want to port the dsNet client to their phone?

Now that Cleversafe has announced its upcoming commercial products, we can talk more clearly about the relationship between the Dispersed Storage open source project (here at www.cleversafe.org) and Cleversafe’s commercial products. 

As we are doing with our technical approach, we are utilizing the outstanding and proven approach taken by the Internet for our business ecosystem model. 

The key protocols that power the Internet:  TCP, IP, UDP, LDAP, DNS, etc are genuinely open protocols in that they are in the public domain and available as open source.  As a result, the Internet really is an interoperable internetwork.  Way back in the 80’s, I was in the IT department of a large aerospace company responsible for setting networking standards and I remember when other networking protocols – such as IBM’s SNA, DECNET and Novell’s SPX/IPX – were more popular than TCP/IP.  Each of these proprietary protocols were driven by well-funded and capable R&D organizations; whereas, the R&D behind TCP/IP was not fueled by a large, well-funded technology company.  However 20 years later, TCP/IP has pretty much completely taken over as the networking and internetworking protocol of choice and the Internet absolutely dominates any networks still using these previously popular proprietary protocols (P4’s).

Yes, TCP/IP is a great protocol, but the P4’s protocols were pretty great, too.  I believe part of the reason that TCP/IP and the Internet emerged as the world’s network is based on human nature.  Simply put, most people didn’t want the world’s network to be owned or controlled by one company, so as the world built its internetwork, a genuinely open protocol had an inherent advantage and was ultimately propelled to become the protocol behind the Internet, the world’s network.

When we began to develop Dispersed Storage, we realized the technology had significant potential, namely to store the world’s data.  We also realized, based on the lesson of the Internet, that the protocols of a Storage Internet ultimately must also be open source, so we created the Dispersed Storage project  in order to develop and publish the Dispersed Storage protocol as open source.  In order to do this, we also created a lot of new open source code and incorporated a lot of open source code from other projects.

Even though the Dispersed Storage protocol stack and core features are available as open source, plenty of great opportunities exist for creating commercial products and services.  To use the Internet analogy again, network equipment companies, like Cisco and Juniper create commercial products (switches, routers, gateways, etc.) by taking the open TCP/IP protocol, adding proprietary features like management systems, integrating onto optimized hardware with an optimized OS and then selling those products through a trained channel that provides services like support and installation.  Cleversafe is using that same approach for its commercial products, so once again we are standing on the shoulders of the giants who build the technology and the business models for the Internet.

Chris

I previously wrote a blog post about how hard drive performance has been increasing slower than the rate of change of Moore's law which means that hard drives have become relatively much slower over the past 15 years.

Another related and potentially more significant trend is that fiber bandwidth speeds have been increasing faster than Moore's Law over the past several years.  Although the data on this trend is much harder to find, I did find this reference at the Moore's Law article on Wikipedia:

Data per optical fiber. According to Gerry/Gerald Butters,^[15][16] the former head of Lucent's Optical Networking Group at Bell Labs, there is another version, called Butter's Law of Photonics,^[17] a formulation which deliberately parallels Moore's law. Butter's Law ^[18] says that the amount of data coming out of an optical fiber is doubling every nine months. Thus, the cost of transmitting a bit over an optical network decreases by half every nine months. The availability of wavelength-division multiplexing (sometimes called "WDM") increased the capacity that could be placed on a single fiber by as much as a factor of 100. Optical networking and DWDM is rapidly bringing down the cost of networking, and further progress seems assured. As a result, the wholesale price of data traffic collapsed in the dot-com bubble

I ran into Marc Verdiell who is a world-class fiber data communication expert and was able to ask him whether he was seeing such a dramatic shift in fiber bandwidth and he confirmed that the bandwidth potential for fiber is effectively unlimited as there are many, many colors of light that you can theoretically transmit down a piece of fiber.  (Of course, the technology to make this happen gets extremely complex which will keep experts like Marc busy for many years.)

I am seeking to understand this further, but these initial pieces of information suggest that we are in for continued, dramatic increases in network bandwidth in the coming years -- at a rate faster than the doubling-every-two-years realized pace of Moore's law. 

When combined with the long term trend that hard drive performance is increasing at a rate slower than Moore's Law, the clear result is that high performance and large scale data storage systems will evolve to designs based on reading to and writing from large numbers of hard drives simultaneously over high speed network connections.

Chris

Performance benchmarks with the latest internal release (0.8.0)  show that the read/write performance of Dispersed Storage went up by 2x (again!).  We're now generally seeing realized throughput rates in the 20-30 MBps range (equal to 160-320 Mpbs) though a single Accesser (client) on a dsNet.  This level of performance is well beyond the theoretical maximum that we thought we'd get to in this initial release.  (So it is a good thing that our developers are better at performance improvements than they predicting the theotical maximum for performance!)

To put that in perspective, we ran some apple-to-apples comparisons between a dsNet and a local hard drive.  The test we ran was reading and writing a 1 GB file on both a local (desktop PC) hard drive and a dsNet over a 1 Gbps connection.  It turns out that the dsNet was a bit faster for the write and a bit slower for the the read vs. a local hard drive.  The results are in this chart:

Overall, this is a huge deal.  This level of performance is about 100x where we thought we'd end up for this release.  Because we are now providing hard drive level performance through regular hard drive interfaces (Block, iSCSI, etc.), we really are optomisic about the potential for Dispersed Storage.

Going forward, we are confident that we'll be able to increase the performance of dsNets even further and ultimately consistently exceed hard drive performance.

Chris

Our focus for Dispersed Storage performance has been overall system performance – which in a Dispersed Storage Network is driven by how quickly bytes travel up and down the client stack.  This stack performance is basically what determines the speed of getting data on or off the dsNet, i.e. the speed of reads and writes.  The sequence of events when going down the stack -- “writing” data is:

1. Source data integrity check

2. Compression

3. Encryption

4. Dispersal

5. Slice integrity check

6. Packetizing

7. Network transmission

When “reading” the data flow is the opposite – you just go up the stack.  Also, note that compression and encryption are optional.

So, our thinking about performance revolves around the overall throughput which means that the slowest layer has the greatest affect on throughput.  Our goal for the performance of the dispersal algorithms is to make sure they are not the slow link in the performance chain required to get data through the performance stack.

We’ve been running a variety of performance tests – reads, writes, different files sizes, etc. and are consistently seeing that the dispersal algorithms are now exceeding our goal for performance in this release.  Specifically, on a Xeon 5300 Quad Core running at 2.33 GHz, we are getting data through the Dispersed Storage stack (with compression and encryption turned off, so that we are mainly exercising the dispersal algorithms) generally around 100 Mbps (or 12 MBps) for reads or writes. 

We’ll have more test data in the future, but we’re very pleased that these results are looking so good so far.  And we still expect to realize further performance gains over time with further optimization.

In our previous tests of compression and encryption performance, we saw that either of these two functions are much more CPU-intensive than dispersal.  So, we are confident that our implementation of information dispersal algorithms won’t be the slow layer in the stack.  We are now re-testing with compression and/or encryption turned to confirm these prior results.

For even higher performance, you can run the Dispersed Storage stack on multiple machines or front-end the dispersal stack with other processes, like compression or de-duplication to increased the realized data throughput.  For example, if you had a process running on another server that provided 10:1 compression and you dispersed the compressed data, you could realize source data throughputs of 1+ Gbps through a single quad core box running the Dispersed Storage stack.  And if you ran the compression and Dispersal stacks processes on multiple servers in parallel, then overall performance just scales up.  I think you see where this is headed.

The summary idea is that performance in a Dispersed Storage network doesn’t have an inherent limitation – just like a packet switched network doesn’t have an inherent performance limitation.  We have a lot of work to do to fully realize this vision, but asking “how fast is a Dispersed Storage Network” should be like asking “how fast is packet switching.”

Chris

In preparing a presentation recently on long term trends in data storage, I was talking with Russ Kennedy and he mentioned that it was a known fact within the data storage industry that hard drive performance has been lagging.  So mentioned this to Dennis Roberson and he connected me with this article at Tom’s Hardware which I found very enlightening.

It turns out that increases in hard drive capacities have been keeping pace with Moore’s law (as observed) by doubling every 24 months.  But hard drive performance (reads and writes) hasn’t been keeping pace.  Even though hard drive performance has been increasing, it hasn’t been increasing as fast as hard drive capacities have been increasing. 

So if the amount of data that people want to read from and write to hard drives has been increasing with the rate of increase of CPU performance and hard drive capacities, then the realized performance of hard drives (on this “technology adjusted” basis) has been getting SLOWER!  This is a big part of why Windows never seems to load any faster.

The article at Tom’s Hardware really brought these diverging trends together in a compelling way by measuring the speed to read a single platter against the year when the hard drive was manufactured.  This chart shows that the speed to read a single platter of data has decreased by almost 10X over the past 15 years.  Wow. 

If these trends continue, hard drive speeds will become an increasingly limiting factor which will further limit the approach of a local hard drive as the primary data storage system.  Especially for high performance environments, the solution to lagging drive performance will be architectures like Dispersed Storage that write to or read from multiple drives in parallel.

Chris

You may have noticed that the open source release we posted last month was a complete re-write of the code based we published last year.  Not a single line of code was unchanged.  This change wasn’t just due to the change from C++ to Java, but was a result of a decision to completely re-architect, re-design and re-write the Dispersed Storage software. 

I’ve spoken a few times with Joe Jablonski of Acumence and others about the merits of re-writing software vs. enhancing an existing code base and Joe and I agree that software development organizations too often make the mistake of continuing to enhance weakening code base vs. re-writing from scratch.

I think this misjudgment comes from a fundamental misunderstanding of the nature of modern software development.  Modern software development tools in the hands of capable developers can quickly produce complex software.  (We are using SCRUM as our development methodology, by the way, and have been quite pleased with the results.)  But software development does not just mean the act of writing software.  And the time required to write complex software is not just the time required to type the code.  Completing a complex software development project requires dynamic coordination of requirements definition, architecture, design, development, testing, validation, tuning, and enhancement.  If done correctly, the act of writing code is only a portion of the time and effort required for software development, especially for complex software and especially for a new type of complex software.

Our goal in the initial production release of Dispersed Storage software is to create an outstanding software foundation on which we and others can build Dispersed Storage solutions.  The work we did in 2005 and 2006 provided many insights in how to build a Dispersed Storage system and that know-how enabled us around the beginning of this year to know that we needed to re-write our software.  That know-how also included knowing how to proceed toward an outstanding initial production release.

Whether we realize that goal will ultimately be determined by market acceptance of our software and specifically whether it provides the reliability, security, performance, scalability, longevity and cost-effectiveness benefits we envision.  But the preliminary results we are seeing now from our re-write over the past year so far exceed last year’s results that we know that re-writing was a necessary step.

Chris

Don’t get me wrong, getting old is not fun, but it does afford a certain perspective that only comes with being around for a while.  That’s sort of a good thing, I guess.  Being in one industry for a while you can definitely see innovation and progression, however, there are relatively few times when you get to witness revolutionary change.  Most of the time things tend to evolve…evolutionary change.

It would be hard to argue that the Information Technology and Data Processing has experienced significant change in the last 30 years.  There are countless examples of things we take for granted today that weren’t even conceived of a generation ago. 

The face of the IT industry has changed and changed again in that time frame.  When I entered the work force, all computing and data processing was performed by expansive systems that took up rooms of floor space, had to be cooled by water flow and had about the same processing capacity and memory as the laptop that I’m using today.  You had large expansive mainframe boxes that performed batch oriented jobs and very few humans interacted with computers at all. 

With the invention of the personal computer and client-server computing, IT and data processing broke out of the “glass house” and became the prevue of every human.  Networking advances connected all this computing power together and placed information, entertainment, and just about everything in our lives at our fingertips.  That’s revolutionary, it’s changed the world.

In storage, there have been significant changes as well, but one could argue again, that most of the changes have been evolutionary, not revolutionary.  Driven by areal densities, speed and recording capabilities of magnetic material and manufacturing efficiencies, the storage industry has seen its share of improvements but not at the level of other core elements of IT.

Look at today’s choices in the storage system market!  Probably the most significant invention in storage was RAID technology at the end of the last century.  Since then, most of the invention you’ve seen from the storage manufacturers is in the area of connectivity, capacity and performance.  Every storage vendor basically solves the problem the same way.  They build a box (an array) that is a collection of heads/controllers with interfaces (where most of the intelligence lies), a set of drives that operate collectively to provide capacity, performance and recovery from failure and some form of management software that is essential to operate and manage this stuff.  It’s not very exciting and not much invention - except some of the management techniques.

What we need is some revolutionary thinking, “outside the box” thinking similar to the days when data processing escaped the monolithic compute platforms and became accessible and usable by the majority.  Why should all critical information assets be stored on one box in one location, even if it’s copied to another box or tape cartridge for safe-keeping?  Why shouldn’t information be stored in such a way that it’s not only private and protected without replication but it’s easily accessible and distributable to humans that want to use it?

Hitachi recently announced that they will be shipping a 4 terabyte hard drive in 2011.  One terabyte hard drives are now available for $320, so it you can expect 4 terabyte drives will be in this same price range within a short number of years.  (4 terabytes = 4,000 gigabytes = 4,000,000 megabytes.)

Recently one of my friends sent me this link that tracks hard drive pricing over time.  In 1956, IBM was selling 5 megabytes for $50,000 which equates to $10,000 per megabyte.  So in 1956, four terabytes of hard drive storage would have cost $40 billion dollars.  Yes, $40 billion dollars for the storage that will be on a single hard drive in 2011.  ($10,000 x 1,000 x 1,000 x 4)

So if this rate of decreasing cost per unit of hard drive storage continues at the same rate through 2066, would be able to buy a 500,000,000 terabyte (500 exabyte) hard drive for around $320 if fifty years or so.  Five hundred million terabytes is a huge number when you think about it in 2007 – just like 4,000,000 megabytes (4 terabytes) seemed like a huge amount of storage in 1956.

To put a five hundred million terabyte hard drive in perspective, consider how much storage is available on all hard drives currently in use today.  According to Disk/Trend, hard drive factories produced between 450 million and 460 million hard drives in 2006.  If you assume the average size of a hard drive manufactured in 2006 was 100 GB, that hard drive factories will produce about 40% more capacity in 2007 and produced about 40% less capacity in 2006 (following the rate of change of Moore’s Law) and that hard drives remain in use 3 years, then the total capacity of all hard drives in use on the planet at the end of 2007 will be about 140,000,000 terabytes (140 exabytes).

So this means that if prior trends continue, a typical hard drive in 2066 would have a capacity equal to about 3 times all the storage capacity of all the hard drives in the world today.  I may be off by an order of magnitude or more and/or a decade or more, but the point is that that previous trends suggest that your grandchildren’s hard drive will be as big as all the hard drives currently on the planet.  What would you do with all this data storage?  Or maybe a better question is what will our grandchildren do with all this storage?

Chris

“How did you get the initial idea for Cleversafe?” is a question I am asked fairly often possibly since I don’t have a long career background in data storage systems or in coding algorithms. What lead me to that idea of building a geographically distributed Dispersed Storage grids was that I was looking for a way to store my personal data and I had been reading a lot about the history of cryptography.