Citrix Blogger

VHD versus NTFS alignment

February 7, 2010 · 1 Comment

This topic is guaranteed to bore most people.  Or, maybe I am wrong.  Are you the kind of person that loves to defrag your disk?  Are you always looking for new ways of speeding up your machine?  Are knobs and buttons something you love to tweak?  How about that registry cleaner?

No, I am not saying those things are bad.  Some people have the patience for this while others do not.  Usually the people that fiddle with the settings are eventually rewarded.  At the very least they can proclaim that they understand what the settings are really doing.

Thanks to a reader (thanks John!) I was made aware of the offset problem within VHDs.  Well, let me rephrase that.  I knew about it but I did not know its official name or that people had solved it.

John pointed me to this blog post about the offset problem.  There are already tools out there from NetApp to solve the issue but basically you need to be a NetApp customer in order to get access (at least for the tools that actually fix your images).  In truth, the problem affects most virtualisation users.

So, this is the part where the problem is dissected and understood.  Virtual disks still follow rules left over from physical disks.  Specifically, they have a “geometry” setting that indicates things like sectors per track.  Here is an example from my tool named vhddump of one the test VHDs laying around.

Geometry             3F10BA9E
Cylinders            9EBA
Heads                10
Sectors/Track        3F

This information is a field inside the VHD header.  The geometry value splits into the next three values based on interpreting the bytes.  Since VHDs are big endian (big bytes first instead the typical little endian on Windows).  vhddump is reporting the field backwards since it had not been important to preserve the byte order.

Anyways, you can see that our virtual disk has 3F sectors per track.  This translates to 63 sectors in decimal.  The geometry reported (also known as CHS) affects the alignment of partitions.  The rule with older versions of Windows is that the boot partition starts after the first track.  This is the output for the same VHD with vhddump:

Index(0) 80 01 01 00 07 FE FF FF 3F 00 00 00 B5 98 70 02
Index(0) BootFlag(80) Type(07) SectorStart(0000003F) SectorLength(027098B5)
Index(0) CHSStart(010100) CHSEnd(FEFFFF)
Start CHS  Head(01) Sector(01) Cylinder(0000)
End   CHS  Head(FE) Sector(3F) Cylinder(03FF)

Volume Count         1
Volume Index         0
FirstVolumeSector    3F

The master boot record starts at the first sector of the virtual disk.  This dump is from a Windows XP image.  The first sector of the first volume/partition is at 0×3f.

Why is this a problem?

Well the most obvious problem is that the sector 0×3f does not align with VHD blocks or even NTFS clusters.  Since it is an ‘odd’ sector, it is guaranteed never to align with anything inside the VHD.

This problem was first seen when exploring the clusters inside the VHD.  Instead of being neatly aligned with the VHD blocks, it was possible to have a cluster that spanned two blocks.  Even the sector bitmap showing the written clusters did not align on byte boundaries for a cluster.  Not only did it make it harder to correspond information, it was more wasteful to do more work for something that should have been aligned in the first place.

This kind of offset problem would show as a performance problem over time.  It is always more efficient to have alignment.

If people really understood this problem, they would probably insist that the partitions were aligned.  A simple example is a cluster that spans two blocks.  Not only is it a read/write hit to access two blocks, but it also potentially wastes space with the second block if there is nothing else there.

If clusters are aligned with VHD blocks, it is much easier to correlate the file data.  It makes sense that the disk should be aligned not with pretend physical settings but rather the VHD format itself.  Even though it is counter-intuitive, it might make sense to have the first partition start at a 2MB boundary.  Some space would be wasted before and after a given partition but the partition would be guaranteed to be isolated from the MBR area and the other partitions.

John had asked for a tool to fix this.  Unfortunately I do not have time right now to solve it.  There are other areas which are currently more important.  However, it would be fun to write such a tool.

→ 1 CommentCategories: NTFS · VHD
Tagged: , ,

Git Over It

November 13, 2009 · Leave a Comment

git-logo

Git is a tool used extensively for open source projects.  It is a way of distributing source control instead of the typical central repository.  History has it that Linus Torvalds himself developed the original tool.

Every Git working directory is a full-fledged repository with complete history and full revision tracking capabilities, not dependent on network access or a central server.

Why is it called git?  The theory is that Linus named it after himself. :)  However, no one seems to know for sure.

Why git?  It all started as a dispute between the Linux developers and BitKeeper.  The BitKeeper tools had been free to the community until the company charged the developers with reverse engineering their product.   The result was that developers would need to pay to continue using BitKeeper.  Given the “free” nature of Linux development this obviously did not go down well.  Linus led the project to develop an alternative and the result is git.

The overall good news is that git has been widely adopted even though it was only created starting in 2005.  There are several good features it has compared to the traditional source control solutions.

  • Offline changes
  • Free (always helps)
  • Fairly simple commands
  • Distributed source management
  • Open source (for the hardcore)

To get a better introduction, visit the git homepage.  As a set of examples of this homepage:

Git Is...There is also a section on quick start.  Git has two paths for creating a repository.  Either it is created from a clone from another server or it is created from scratch.

GitQuickStartAnything in parentheses is meant to be replaced with specific actions (like a directory or file).  To keep it simple, it is best to experiment with a locally created depot.  It is worth doing an easy walk through of the creation.

  1. git init
  2. git add .
  3. git commit -m “My message”

“git init” creates the initial depot for this current project.  It creates a subdirectory (usually hidden) called .git which contains all the information that git needs to keep to manage the source.  Think of it as being where everything is kept to keep git happy.

“git add .” instructs git to find every single file under the current directory and add it to the staging area to be prepared for committing. It will only add files that are new or changed when used with the “.” which is a great trick.  If there are files that you do not want to automatically add, you can always change the “.gitignore” file to skip over them.  Typically the object and binaries need to be skipped if the tree is actually built.  Doing the add is really just a sign of intent.  It does not actually change anything in the git tree.  Also of note, if you change any files that have been added, you need to add them again if you want to capture the changes in the staging area.

There is nothing to fear with commitment.  Commit with “git commit” just means that you want to capture all the changes into one thing.  This is where it updates the local git tree and it is seen as a kind of “snapshot” of where the code is at.  Keep in mind that a commit means nothing to anyone else.  A commit is just a local thing.  This enables git to be used offline and without the need for a central server ever.

However, git can also be used in a distributed team.  In fact, that was the original intent.  That is the point of the other path for creating a depot on the local machine.

Working off of a remote depot is in this order:

  1. clone a remote depot (git clone)
  2. make changes to the local files (git add)
  3. commit the changes (git commit)
  4. send the patch to someone that cares (git format-patch)

This is not the only way to do it.  It is also possible to submit the changes back to the remote depot assuming you have been authorized.

Coming from a world of PVCS and Perforce, git can take some getting used to.  It seems uncommon for git to be used in commercial products in Citrix (except for Xen products).  XenApp and XenDesktop are based on Perforce.

There are so many different places you can go to find out more about git.  Here are some examples:

Linus Torvalds (git creator) speaks about git at Google Talks.

Thanks goes to Michael Wookey in Citrix Labs Sydney for being such a great git advocate.

→ Leave a CommentCategories: Linux Tools · Source Control · Tools · Windows Tools
Tagged:

Linux Bridge

November 11, 2009 · 2 Comments

Having come from a Windows background, it can be very interesting to try catching up to where the Linux world is now.  It is more than likely that most Linux people feel the same way about Windows.  Well, more accurately they usually despise Windows and would rather leave that to someone else.  Generalities, of course, are not that accurate. sydney-harbour-bridge-nightThere have been many different paths to bring the two worlds together.  Most of this due to work done by open source projects.  Microsoft has semi-embraced working with Linux.  Typically Microsoft likes to position competitors into specific boxes.  Even though Linux is hard to contain, Microsoft likes to paint it as having no common thread for all the different distributions.  It also tends to show it as not being enterprise quality.

The ironic thing is that Microsoft has been heavily influenced by Unix and Linux over the years.  Many of its biggest core offerings are derived from ideas created by common research years ago.  No one company can really go it alone but it is possible to pretend that you thought of everything yourself.

Regardless of how Microsoft handles this today, the truth is that battle with Linux has been going on for many many years.  It appears that they have toned down the opposition for fear of offending key large accounts (typically governments).

So, what is the point of this blog post?  Really, it is just a place to sum up ways of using Linux tools in Windows. Unknown to most Windows users, there are Windows version of Linux tools.  Also, some of these tools are quite useful. As part of my current project, it has become very important to develop for Linux.  This is a blog post in itself.  All I intend to do here is list the more interesting tools.

wget – get a file from the Internet from the command prompt (either HTTP, HTTPS,  or FTP)

winscp – copy files between Windows and Linux

msysgit – source control (git for Windows)

cygwin – Linux shell on Windows

7zip – compression of just about every model

Instead of writing a huge list, it is better to focus on just these.  The current star is wget.  I am using it to download a 6GB file over the net.  Using standard browser download is unfriendly since it tends to get errors and does not having decent handling of retry.  Wget is expert in trying the download again and at the point that it left off.  In general, wget makes it easy to automate downloading just about anything.  Not only that, it is quite simple to start with.

7zip is attractive because it can do so much for so little.  It can register itself for a very long list of compression types.  Not only that, it is quick and easy.  Being able to browse an ISO without having to dig up a special program is so nice.

WinSCP is great for making copying easier between different systems.  Drag and drop of entire directories is just a click away.  It is stable and fairly fast.  Yesterday I used it to copy a 10GB file between Windows and a Linux server.  When I was done processing the file, I used it to copy back.  Now that is a great simplification of the process.

To me, the tools are better when compiled for Windows instead of running inside cygwin.  The authors have expanded the tools in some ways.  The best kind of expansion comes from using the Windows UI to drive it.

GIT is a tool to manage source and is very common for Linux-based projects.  Using git on Windows is possible thanks to a port that is available from Google code.  There is also an extension that more deeply integrates with Visual Studio.  The UI version is easier and more powerful (to the beginner) than the command line.  Learning the command line options and flows can be quite time consuming and frankly can be a waste of time for the average developer.

At some point, a future post will be about writing code for both Windows and Linux.  This has been my focus over the last year related to supporting VHDs in XenClient.

→ 2 CommentsCategories: Linux Tools · Windows Tools

Blog Evaluation

November 7, 2009 · 4 Comments

BlogStats

This blog has been going for three years with this being the 301st post.  Within the last year, things have slowed down considerably for a number of different factors.  Perhaps the biggest reason is being too busy with the current project (XenClient).  However, there are several other factors as well.

Blogging turns out to be a mostly solitary venture.  It takes a great deal of commitment and time to formulate a blog that gathers widespread adoption.  Even then, I suspect that success breeds pressure to always push harder.  It is difficult to gain that kind of momentum and keep it there.  This kind of stress is similar to anyone trying to hold on to success and fame.  Very few go the distance.  Perhaps very few should.

Originally I had big plans for doing this.  It seemed like a great idea for bringing in new customer requirements.  It also seemed like a good place to try out some ideas of my own.  Quietly, I hoped for a large following.  I tracked the visits per day and experimented with different blog postings to see what would make a difference.  This theory was flawed.  Any particular post might do better than others but overall it really did not make a big difference.

At one point the goal was to have about a hundred visits a day.  There was a blog post almost every day.  This artificially seemed to push up the numbers based on the number of posts but never seemed to bring any kind of long term growth.  It was probably similar to typical sales cycles.  The sales guy can always make things look better based on activity at the end of the month but the same person will never guarantee long term great results.

Later on, it just seemed a bit pointless.  The things I had set out to do just did not happen.  Also, ironically, my visit rate jumped up consistently with complete blog neglect.

It seems a bit like tracking the stock market.  The sooner it is realized that logic has nothing to do with it, the better.

Which brings us back to passion and purpose.  Success is truly an internal event.  It is the belief that one has a passion for doing something that really brings about the best results.  In fact, it is also the belief in letting go and being able to admit to mistakes which clears the way to a better future.

My passion is mostly about making things better.  I enjoy learning how things work (in true engineering fashion) and proceeding to think of ways of making it work even better.  Everything can be improved.  That alone is a major passion of the entire business world.  Some people focus on how much they can get.  Others do it for the thrill of making a difference.

So, what the heck does all this mean?  Well, either it means nothing or it means that I might have finally found new purpose to doing this blog.  I suspect it is the latter.  Time will tell.

For some, this might be seen as some kind of “middle-aged” blog crisis.  Perhaps it is.  At this point I do not even really care. :)

It is perhaps when I am most concerned about what people think that I am the most evasive.

To clear the air and breath fresh, is actually quite good.  Shadows of the mind dance more harsh than the path of the real daylight.  In other words, the thoughts overshadow action.  In fact, thoughts can easily confuse what path to choose.

With that in mind, it is time to start over.  The blog needs renewing and obviously refocus as well.

→ 4 CommentsCategories: Angst

Disposable Devices

August 16, 2009 · 2 Comments

We are entering a very different era from our parents.  Technology is shifting from a very individual experience into a worldwide exchange.  As each wave forms, the resulting push goes further and further inland.  The goal is to reach some kind of perfection that will satisfy all our possible wants.  The frontier continues to expand.

Devices (laptops and smart phones) which are currently the focus of a mobile and remote work force are becoming disposable.  This is due to a number of factors such as damage, theft, and becoming obsolete.  The point is that real devices are always going to have a limited lifespan. 

It has been interesting to watch the computer industry over the last thirty years.  The speed of which the changes have come has only made technology obsolete that much faster.  Using a computer from more than five years ago is often a questionable venture.  Laptops and phones are probably more around 2 or 3 years. 

Given the limited lifespan, what is the real value?  Much like human knowledge, the best things are passed down.  The information is the soul of the device.  It is this data that needs to be preserved and propagated to new devices.

Add the concepts of virtualization and the Internet, it is possible to build a model whereby the information will never die.  In fact, given enough focus, the environment will evolve and flourish.

The information can be safely saved on trusted servers.  If the device is lost, stolen, or dead, the environment can be brought forward to a new device.

It is similar to the idea of storing photos on the web for backup purposes.  In theory, your photos will never be discarded.  The same would be true of your standard computing environment.

This kind of strategy puts the focus on what is really valuable.  It certainly is not the disposable device.

Admit to yourself one thing.  Your laptop ages twenty times faster than a human (laptop years).  Your phone is closer to 30 years to one human year.  I would bet your phone or laptop is already older than you.

The information however, is priceless, ageless, and completely virtual. 

Doesn’t it seem we always focus on what we can see?  It’s the things that can’t be seen that really make the lasting impact.

→ 2 CommentsCategories: Uncategorized

CHM Confusion

June 25, 2009 · 2 Comments

Recently, I downloaded the Sysprep package from Microsoft for XP SP3.  It does not really matter why.  The package was expanded and the contents explored.  However, what happened next was unexpected.  See the screen capture below:

AddressNotValid“The address is not valid” !  How baffling is that?  Not only is it not using the Internet to get the content, there is no context information describing how this happened.  My first assumption was that I did not have the same version as the content.  This turned out to be untrue.

The truth is that Windows is blocking the content since it came from another computer.  If you look at the properties of the CHM file, it will show ‘Unblock’ as an option.  Select Unblock and the problem goes away the next time you run the CHM.

If you would like a more detailed explanation, check out this post.

→ 2 CommentsCategories: Windows XP
Tagged: ,

Knowing What Matters

May 26, 2009 · 2 Comments

There is nothing more important.  More important than what?  What current preoccupation is that important?  Well, it seems that important so therefore it is.

It’s a judgment call,  and clearly not all the facts are known.

Largely, the computer industry is driven from very important factors.  Much like life, there is plenty of activity both on the development and consumption.  The desire for more is always there.  The hunger for bigger and better goes on and on.

There is clearly a parallel with how automobiles were first developed.  Initially cars were very complicated and costly.  The user had to know every control and quirk.  The industry was fairly small with not much sympathy for the amateur.  Even today, this is still mostly true.

The truth is that computers are not important.  The same is true of automobiles.  It is what these tools can accomplish for the user that brings value.  Functionally, different cars only add value based on what the drivers can derive as a benefit.  

What I am getting at is that the computer experience needs to be much more transparent.  Sitting at a monitor with keyboard and mouse is not conducive to being easy.  The hardware needs to blend in with the environment to the point that the user/consumer does not even have to think about it.  This is the mantra of ubiquitous computing.  The difficult aspect is thinking outside the range of current thinking.

I was talking with my Dad over the weekend and we started talking about airplanes.  In the 60’s to 80’s, he worked as a pilot and navigator.  He pointed out that modern control systems on planes make it very difficult for a pilot to override potentially very dangerous automated decisions.  We talked about centralizing the power in the computers, it actually takes away from the experience and thinking of pilots.  From talking about this, it became clear to me that pilots should see the plane as an extension of themselves.  The computers should be used like how the brain controls the body.  The brain is the pilot and the body is the plane.  Ideally, this would be true except for cases for when the pilot cannot process the information fast enough.  Dad mentioned that planes have an automated system for landing.  It is well known and a perfect application for planes since there is actually more risk having the pilot do it.  He explained that a pilot could easily cause a “Dutch roll” in specific situations. If bad enough, this condition can be catastropic.

This comes back to an interesting idea.  Since our computers are still very dumb, it is unwise to put too much faith in them in unknown situations.  The lack of intelligence is often forgotten due to the speed of how computers figure out known problems.  It is fine for it to focus on something that can be digitized and programmed, but completely helpless with sketchy data and an unknown state.  Most faults in computer systems can either be tracked to hardware failures or software that could not recover from an unknown state.  Lately I would tend to think that most problems are in software.  It is very hard to write software that will work in all cases forever and ever.  Things change.  

So, this brings us back to what matters.  First of all, we should not put too much trust in our existing frameworks to carry us forward to the more casual computing model.  The hidden message is that basing computers solely on digital on/off switches limits their abilities considerably.  By now, we should be looking more seriously at technologies that better duplicate intelligence.  Perhaps the most revolutionary idea is that it is okay not to know things.  Much like life, we have to accept that we don’t know everything and that the computer should be allowed to learn.  This puts the computer more in the category of having a real brain.  Obviously a computer also needs to ability to link multiple locations together (or unlink) so that it can more easily find connections.  This should be done more like our neural networks and made to accept unknown states.

The best analogy is this.  You are walking down the street and suddenly something amazing happens.  For whatever reason, gravity no longer works.  You have not been programmed :) to handle this situation but you quickly try to find an idea that will help.  If you were a robot, there is a perfect chance that you were never programmed to handle this.  So, either your software faults or you just keep on doing what you were doing.

So, what would happen if gravity just stopped?  I would guess that you’d be thrown (slowly?) off the earth based on inertia and the earth’s rotation.  I don’t know.  The point is that if it did happen, at least I would be trying something.  At the very best, it is clear that I should not fault.   Even if I did fault (pass out), most likely I would reset just fine (perhaps floating off to space).

I like the idea that people usually don’t fault.  It implies that we are much too accepting of existing models and hardware.

Please excuse the tangents.  It’s hard to think clearly due to my fuzzy programming :) .

What really matters is that our faith should be shaken to usher in something new.  In fact, given the success of the web, we have largely solved the act of exchanging information.  What we have missed is realizing that many our designs are based on 1960’s (or even earlier) thinking.  

We’ve come a long way.  We still have a long ways to go.  A key goal is removing the importance of the computer in how separate it is from everthing else around us.  Things should be more specialized, simple, and transparent.  It is a process that should be much more evolutionary than it currently is.  Competing models should be allowed to win and lose based on what the environment allows.

→ 2 CommentsCategories: Angst
Tagged:

Latest Citrix Receiver for iPhone video

May 6, 2009 · 1 Comment

→ 1 CommentCategories: Uncategorized
Tagged: ,

Registry Details from Microsoft

April 16, 2009 · Leave a Comment

Microsoft obviously does not want to publish certain things even though it has published a number of other things (e.g. the VHD Spec).  One of those things is the format of the registry files.  However, they have taken the time to publish what they are willing to share.  It would be worthy of reading for anyone that has not tried to look under the covers yet.

You can find the page at the Microsoft support site:

http://support.microsoft.com/kb/256986/

→ Leave a CommentCategories: Registry

Citrix eDocs – online documentation

March 31, 2009 · Leave a Comment

This one is just for reference.  Citrix has published a much more interactive model for finding documentation.  On the support site is a web-hosted documentation system.

Navigation is much easier and it looks like all the products are included.  It should be easier than having to download pieces or looking at printed copies.  It even supports searching.

→ Leave a CommentCategories: Citrix Documentation