My questions answered

Problem:

I run a site called Videoly. It is a simple video emailing service. The site is hosted on Amazon EC2 instance and we wanted a simple backup solution for our user generated content. The criteria for the backup solutions were simple - cheap, simple, reliable. It should include incremental changes to user generated data, database changes, and important configuration files. Also, the process should generate reports and deliver them through emails.

How to backup MySQL database and files from Amazon EC2 instance ?

Solution:

Amazon EC2 service is neatly coupled with simple storage service also called as Amazon S3. We decided to use the S3 service to store our backups. The service is reliable and most importantly all transfers between S3 and EC2 instances are free.

Here is our current solution -

1. Use JungleDisk to map s3 to our EC2 instance
2. Backup our content and MySQL database using a utility called Backup Manager
3. Use Cron job to perform regular backups and send  emails

Here we go ... concise list of steps you need to perform in order to implement such solution -

1. Jungle Disk

Reference to a helpful post here

Install jungledisk, and FUSE. Check out your distribution instructions. During your installation remember the location where you store your settings file. In my case, I stored it in /root/.jungledisk/jungledisk-settings.xml

Edit jungledisk-settings.xml.

<?xml version="1.0" encoding="utf-8" ?>

<configuration>

<configEncryptionKey>PROTECTED:dgfdfgd34342fsd9f8fs90sfs</configEncryptionKey>

<GlobalConfig>

<proxyServer>

<enabled>0</enabled>

<proxyServer></proxyServer>

<userName></userName>

<password></password>

</proxyServer>

<listenPort>2667</listenPort>

<cacheDirectory>/root/.jungledisk/cache/</cacheDirectory>

<logDuration>30</logDuration>

<largeWindows>0</largeWindows>

<useSSL>1</useSSL>

<loginAccount>

<userName>kunal@videoly.com</userName>

<password>ENCRYPTED:sdfsdfN398sdfKNUsdf09343098sdfSHUIsdf</password>

<domain></domain>

</loginAccount>

<enableUpdateCheck>1</enableUpdateCheck>

<passwordPrompt>0</passwordPrompt>

<computerName></computerName>

<uploadLimit>

<enableLimit>0</enableLimit>

<rateLimit>200</rateLimit>

<limitTimeRange>0</limitTimeRange>

<startTime>0</startTime>

<endTime>0</endTime>

</uploadLimit>

<downloadLimit>

<enableLimit>0</enableLimit>

<rateLimit>200</rateLimit>

<limitTimeRange>0</limitTimeRange>

<startTime>0</startTime>

<endTime>0</endTime>

</downloadLimit>

<buckets>

<value>

<BucketConfig>

<bucketName>videoly-backup</bucketName>

<accessKey>1SDFAF2dfSKDFJ4R2</accessKey>

<secretKey>ENCRYPTED:SSFOQ3sdf:LIOIDSFhNDNlYjFiZjk5ZDlhYjsdufIOOsdff3A6Qi9Rwt3AZ//GOXCeZQyIsi8XtpOr8jYyYY411MqQ=</secretKey>

<cacheCheckInterval>120</cacheCheckInterval>

<maxCacheSize>1000</maxCacheSize>

<retryCount>3</retryCount>

<asyncOperations>1</asyncOperations>

<encryptionKey></encryptionKey>

<decryptionKeys></decryptionKeys>

<useFastCopy>1</useFastCopy>

<uploadResume>1</uploadResume>

<deltaUpdates>1</deltaUpdates>

<plusFallback>0</plusFallback>

<archiveFlag>1</archiveFlag>

<archiveConfig>

<archiveUpdated>1</archiveUpdated>

<archiveDeleted>1</archiveDeleted>

<maxArchiveSize>100</maxArchiveSize>

<maxArchiveCount>0</maxArchiveCount>

<enableExcludeTypes>0</enableExcludeTypes>

<enableExcludeFolders>0</enableExcludeFolders>

<archiveExcludeTypes></archiveExcludeTypes>

<archiveExcludeFolders></archiveExcludeFolders>

</archiveConfig>

<archiveCleanupConfig>

<minArchiveCount>0</minArchiveCount>

<archiveDuration>60</archiveDuration>

</archiveCleanupConfig>

<networkDrive>

<mapDrive></mapDrive>

<mountVolume></mountVolume>

<mountPath>/mnt/jungledisk</mountPath>

</networkDrive>

<loginTicket>VCsdfZWjL+1aWWCeQKwdXXDaE8g1YkDo+1aWWCeQKwdXQ/uzrVJXH1zFPJIq+1aWWCeQKwdX/4EsP1615Nh5DCO1yKVw2uWYZCw2SCzmzoNnVhMOvr</loginTicket>

<licenseData>Y+1aWWCeQKwdXMzgzODVjNWI2ZWI0MjFiNDE3ZjkxMWQ0ZFtr++1aWWCeQKwdX|qaJyURvY6hk86lGhpDHTtKBStO9YH9Z9qaUI7WMgg8ly3wo9zrSLvEaVfmKFPX3IvZvUuoNBsSRTeW23eCO/+1aWWCeQKwdX/LbxjJ00/Y=|fnYsPyUVCZWjLXDaIulYAjzf6BdQJYvP2kA0tvg2Kfx98YxKdRE8g1YkDoQ/uzrVJXH1zFPJIq+1aWWCeQKwdX/4EsP1615Nh5DCO1yKVw2uWYZCw2SCzmzoNnVhMOvr</licenseData>

<archivePath>/~VersionArchive</archivePath>

<backupJobs></backupJobs>

</BucketConfig>

</value>

</buckets>

</GlobalConfig>

</configuration>

Mount JungleDisk

/sbin/modprobe fuse

/usr/local/jungledisk/jungledisk /mnt/jungledisk -o config=~/.jungledisk/jungledisk-settings.xml

2. Backup Manager

Reference to a great article by Tim Linden here

wget http://www.backup-manager.org/download/backup-manager-0.7.7.tar.gz

gzip -d backup-manager-0.7.7.tar.gz

tar -xvf backup-manager-0.7.7.tar

cd backup-manager-0.7.7

make install

cp /usr/share/backup-manager/backup-manager.conf.tpl /etc/backup-manager.conf

Backup Manager 0.7.7 includes a feature which uploads the backup files to S3 location. However, when I read though the documentation and forums, I found this post. The author of the utility does not speak highly about the feature, so I stayed away from it. I just used the utility to backup my files. Here is my configuration file -

Edit /etc/backup-manager.conf. I left everything else to default

export BM_REPOSITORY_ROOT="/mnt/jungledisk/"

#otherwise you will get chmod errors

export BM_REPOSITORY_SECURE="false" 

export BM_ARCHIVE_METHOD="mysql tarball-incremental"

# dar slices backups, helps to keep objects below 5 GB - requirement of S3

export BM_TARBALL_FILETYPE="dar"

#Content folders

BM_TARBALL_TARGETS[0]="/usr/local/content1" 

BM_TARBALL_TARGETS[1]="/usr/local/content2" 

BM_TARBALL_TARGETS[2]="/usr/local/content3" 

# S3 allows file size to be max 5 GB so I set them to be 4.5 GB to be on safe side

export BM_TARBALL_SLICESIZE="4500M"

#here is an example of incremental backup setting

# once a week fullbackup on the day 1

export BM_TARBALLINC_MASTERDATETYPE="weekly"

export BM_TARBALLINC_MASTERDATEVALUE="1"

export BM_MYSQL_DATABASES="__ALL__"

export BM_MYSQL_ADMINLOGIN="root"

export BM_MYSQL_ADMINPASS="notgoingtotellyou"

#turned off upload feature

export BM_UPLOAD_METHOD="none"

3. Add a Cron Job

   1: crontab -e

   2: enter “45 1 * * * /usr/sbin/backup-manager –v“

   3: crontab -l

These are minimum steps to get you started. Lot of minor tweaks might be necessary to get it to work for you.

written by kunal

Premise:

I want rails method to respond differently if the request has originated from javascript or plain html post. Part of my site is designed by ExtJS framework and part of the pages include simple html forms. The rails response should be different for these two different requests.

Problem:

In Extjs I am using Ext.Ajax.Request to post the form. Rails differentiates requests using following code -

respond_to do |wants|
        wants.html {redirect_to :controller =>'dashboard'}
        wants.js { render :text => {:success => 'true' }.to_json, :layout => false }

end

However, rails was not differentiating these two calls.  My problem was how do I differentiate Ajax requests from plain html?

Solution:

I am using firefox plugins to view http headers. It turns out that rails decides request types from http header  'Accept' set by client. My headers were

Accept : text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5

This header accepts almost anything and everything from server. All I had to do is to set this header to 'text/javascript' when using Ajax

In Extjs, it was simple -

Ext.Ajax.request({
             url: '/user/login',
             headers: { 'Accept' : 'text/javascript' },
             params: { param1: "food" },
             success: function() { alert('success'); },
             failure : function(form, action) { Ext.Msg.alert('Error',action.result.text); }

})

written by kunal

On those days when I lack the motivation ... and gloomy clouds of desperation start lurking ... here is the solution to get rid of it!

Steve Job's commencement speech at Standford 2005

JKR's Commencement speech at Harvard

written by kunal

Premise:

I am working on a project where I need to integrate a small flash module into an Ajax application. I developed flash module with Flex builder; I tested it and then I copied the pre-generated html code into my Ajax application.

Problem:

Flash application would not render into my HTML page. It would only show be a gray box but no error. I would right click on the gray box and it would produce flash context menu telling me that flash file is found though for some reason it is not being loaded.

Solution:

My solution is just to get my development going. Please read the related document to find the best and most secure configuration for the production

1. Verify the paths : If you don't see any flash object being rendered, obviously what you should investigate is paths. Look at the html code and the flash object embedded in it. The flash object has all variable defined twice in the code. So make sure all the changes are done TWICE!

2. AllowScriptAccess: I set it to always

3. Go to Global security panel and select option to "Always allow"  http://www.macromedia.com/support/documentation/en/flashplayer/help/settings_manager04.html 

written by kunal

I am trying to get activeScaffolding working on Rails 2.1.0. The edge branch on active scaffolding and current release 1.1.1 both are broken on it. Whenever I run server the application errors out with following output -

c:/ruby/lib/ruby/gems/1.8/gems/activesupport-2.1.0/lib/active_support/core_ext/module/aliasing.rb:31:in `alias_method': undefined method `find_full_template_path' for class `ActionView::Base' (NameError)
        from c:/ruby/lib/ruby/gems/1.8/gems/activesupport-2.1.0/lib/active_support/core_ext/module/aliasing.rb:31:in `alias_method_chain'
        from c:/Users/kunal/Documents/Aptana Studio/vLetter/vendor/plugins/active_scaffold/lib/extensions/generic_view_paths.rb:33
        from c:/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `gem_original_require'
        from c:/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `require'
        from c:/ruby/lib/ruby/gems/1.8/gems/activesupport-2.1.0/lib/active_support/dependencies.rb:509:in `require'
        from c:/ruby/lib/ruby/gems/1.8/gems/activesupport-2.1.0/lib/active_support/dependencies.rb:354:in `new_constants_in'
        from c:/ruby/lib/ruby/gems/1.8/gems/activesupport-2.1.0/lib/active_support/dependencies.rb:509:in `require'
        from c:/Users/kunal/Documents/Aptana Studio/vLetter/vendor/plugins/active_scaffold/environment.rb:63
         ... 46 levels...
        from c:/ruby/lib/ruby/gems/1.8/gems/rails-2.1.0/lib/commands/server.rb:39
        from c:/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `gem_original_require'
        from c:/ruby/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `require'
        from script/server:3

so far google search has brought me to http://code.google.com/p/activescaffold/issues/detail?id=515 forum where it is evident that the problem has not been fixed yet!

written by kunal

Premise:

I was writing ruby on rails code for :has_may relationship. I have simple model - User :has_many Posts.

class User < ActiveRecord::Base
 has_many :posts
end

class Posts < ActiveRecord::Base
 belongs_to :user
end

Problem:

Minute I tried accessing posts method from User object I got following error

>> User.find_by_id(1).posts

NameError: uninitialized constant User::Post

 from c:/ruby/lib/ruby/gems/1.8/gems/activesupport-2.1.0/lib/active_support/dependencies.rb:492:in `const_missing'

 from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.1.0/lib/active_record/base.rb:1909:in `compute_type'

 from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.1.0/lib/active_record/reflection.rb:129:in `send'

 from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.1.0/lib/active_record/reflection.rb:129:in `klass'

 from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.1.0/lib/active_record/reflection.rb:137:in `quoted_table_name'

 from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.1.0/lib/active_record/associations/has_many_association.rb:84:in `construct_sql'

 from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.1.0/lib/active_record/associations/association_collection.rb:8:in `initialize'

 from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.1.0/lib/active_record/associations.rb:1128:in `new'

 from c:/ruby/lib/ruby/gems/1.8/gems/activerecord-2.1.0/lib/active_record/associations.rb:1128:in `posts'

Blunder:

All posts regarding this error told me to look at the NAMES. Marking 'has_many' takes pluralized name of the model name. My model class name is 'Posts' and has_many tag includes model name as Posts as well.

Solution:

Either change my model class name from Posts to Post, or change "has_many :posts" to indicate pluralized version of posts!

written by kunal

Problem:

A web application I am working on currently includes streaming and recording videos - Videoly.com. When I was testing the application, I needed to simulate slow internet connection. Naturally, I reverted to google and started searching for tools to do that. Here is my criteria

1. Simple tool to regulate network speed for my application
2. Variate the configuration easily
3. Easy interface to read the reports
4. Windows based

Solution

I found just the tool to do that - NetLimiter

The annoyances to this tool is that it is not free! But for my purposes the trial version was good enough. However, it served it purpose greatly and in no time I was able to test my application on various internet and intranet speed variations.

written by kunal \\ tags: netlimiter, network, Testing, Tools

ClickOnce is by design a per-user installation technology. However, ClickOnce can be combined with MSI to workaround some of the restrictions. Following steps will give you a quick and simple example of how to do it.

• Publish your ClickOnce application : If you are using Visual Studio to publish the application, then just follow these instructions. Otherwise, make sure that your deployment manifest points to location where the application will be published. Find <deploymentProvider  codeBase  /> tag in the deployment manifest, and codebase should point to location of your application/depapp.application
• Create an MSI Package: Visual studio makes an easy job to create a MSI setup. Here are the instructions for VS 2005. Steps should be similar for other versions as well. If you are using any other authoring tools, all you have to do is include the deployment manifest into your MSI package.

Visual Studio 2005 Setups to create MSI package:

1. Create a new Project -> Setup and Deployment -> Setup Project                                                                   

   

2. Select the  "User's Programs Menu"  node on the left and right click on the right pane. Select Add -> File...

   

3. Navigate to your publish location and add Deployment manifest file.                                                               

   

4. Build the Project. You will get a nice and clean MSI package in your build output location.

5. Run the installation. Select option "Everyone" when prompted.                                                              

   

6.  Activate Start->All Programs -> GeneraicApp.application to install your ClickOnce application.

Tips:

• Activating MSI DOES NOT install ClickOnce application. It will only copy deployment manifest. Each user will initiate application installation when the file is activated first time. Each user will have a separate copy of ClickOnce application. 

• Once ClickOnce installation is complete, the application will be installed into Publisher -> Application name folder.

• MSI installer can be further configured to add file associations and desktop shortcuts or any other advance pre-installation configuration steps.

written by kunal \\ tags: hack, Install, msi, Per-Machine, Per-User, shortcut, system-wide

Scenario:

You own a ClickOnce application which is installed on multiple client machines. All clients check updates from your server A. One day you get up and realize that you need to retire server located at location A and need to move the application on server B. No problem… you say! You push an update which changes Deployment Provider URL in the manifest and point it to server B. Once client machines are updated, all subsequent update checks go to server B.

Easy ? Well .. not really ClickOnce makes a boo-boo and your mailbox is flooded with complains from your customers!!

Issue:

If you update the publish location , also called as codebase, clients machines see following result -

• Update is installed as if it is a new application installation, showing duplicate shortcut entries for offline applications,
• Trust prompts is displayed before installing the update even if the application trust settings have not changed,
• User data is lost
• Multiple entries in the Add and Remove Programs for same application

Above behavior is a design bug in ClickOnce for .NET 3.0 and below. The good news is that the issue has been fixed for .NET 3.5 version!

Background:

ClickOnce internally uses what we call a “subscription identity” to find if prior versions of an application has been installed on a client machine. Subscription Identity is similar to application identity except it includes codebase as an additional attribute. When a new update with changed codebase if activated, ClickOnce cannot find installed versions on the client machine because subscription identity is different. Hence, the update is regarded as a fresh installation.

Fix:

In Orcas release, which is a synonym for .NET 3.5, this problem is fixed. It goes hand in hand with another change introduced in this same release. The requirement of including deployment provider URL (DPURL) in all offline applications is waved. Now, in case of codebase change, following variables decide ClickOnce behavior -

• Activation method -  clicking deployment manifest directly or clicking application shortcut,
• Type of Application – Offline or Online application
• Deployment Provider in incoming deployment manifest – deployment provider in the incoming deployment manifest is absent or pointed at itself.

Example

Scenario steps for a successful codebase rollover -

• Publish V1 of application at location LC1
• Install on Client A
• Publish V2 at location LC1 but DPURL in the deployment manifest at LC1 is pointing at a deployment manifest located at LC2. Also, Publish V2 at location LC2 but DPURL in the deployment manifest at LC2 is pointing at itself.
• Activate application on Client A
• ClickOnce will go to LC1 and detect that a new update is available, it will check DPURL which points to a manifest located at LC2.
• ClickOnce will download dep. manifest from LC2
• Checks if DPURL in dep. manifest located at LC2 is pointing at itself, then migrates data, trust decision, and shell shortcuts.

Note:

In a given update, at a time, maximum two attributes from the application subscription identity can change. Version always has to change, otherwise ClickOnce will block  the in-place update. Thus, an update can only change a public key token as explain here or change codebase. If both are changed in one update, then ClickOnce will regard the update as a new installation.

Here is the table of scenario and expected behavior for .NET 3.5:

Legend:

written by kunal

Scenario:

You have published a ClickOnce application which is installed on several client machines. One day you get up and realize that your publisher certificate has expired! Alright, so you go and get a new certificate. Now, you publish an update to your application and sign it with your shiny new certificate. BOOM… your mailbox is flooded with complains from your customers!!

Issue:

If an update is signed with a certificate different from the earlier versions of application deployments, clients machines see following result -

1. New update is installed as if it is a new application, showing duplicate shortcut entries for offline applications,
2. Trust prompts is displayed before installing the update even if the application trust settings have not changed,
3. User data is lost
4. Multiple entries in the Add and Remove Programs for same application

Above behavior is a design bug in ClickOnce for .NET 3.0 and below. The good news is that the issue has been fixed for .NET 3.5 version!

Background:

ClickOnce uses identities to uniquely identity each application. Identity is comprised of 5 attributes – Name, Version, PublicKeyToken, processorArch, Language. When a certificate with new public key token is used to sign an update, version number and publicKeyToken attributes change. ClickOnce only ignores version number to decide if any prior versions of the same application exist on the client machine. In this case, ClickOnce fails to detect the update due to publicKeyToken attribute change. Hence, a new shortcut is created, trust decision is not migrated, and client data is lost.

Fix:

ClickOnce considers an update to an installed application if and only if all attributes of installed application identity and the update are same except the version and publicKeyToken attributes.

Example: