Virtual Guru's Blog – Home of Virtualization Workshops

April 15, 2009

Amazon EC2 OpenSolaris re-bundling process trouble shutting

Filed under: amazon_aws, cloud, opensolaris, Virtualization — natiku @ 2:23 am

This entry is based on customer escalation, I hope it will help or at least inspire you in some extend too.

This entry is part of ‘OpenSolaris on Amazon EC2’ workshop


Customer wont to regularly backup running Amazon EC2 OpenSolaris instance, while APPs was running.


Re-bundling take a,ling time and it looks frozen (plus it was occupy lot of resources)

Monitor ZFS cloning process – ( script

Note: Rebundle script use ZFS mirror cloning mechanism implemented on system level, as such it has build it “share resources” capability.

Log with another connection and start this script

Paste it in BASH, some $ and \ is escape with ONE more \
cat >/tmp/mon.ksh <<EOF
echo "ZFS Cloning started"
echo "Waiting , so clone process really start "
while true
zpool status rpool | grep "resilver in progress" >/dev/null
if [ \$? -eq 0 ]
print -n -e "\b-"
sleep 1
print -n -e "\b\\\"
sleep 1
print -n -e "\b|"
sleep 1
print -n -e "\b/"
sleep 1
while true
zpool status rpool | grep "resilver in progress" >/dev/null
if [ \$? -eq 1 ]
status=\$(zpool status rpool | grep "resilver in progress"| gsed -e 's/ scrub: resilver in progress for/Elapsed/g' )
print -n -e "\r \$status  "
print -n -e "\b-"
sleep 1
print -n -e "\b\\\"
sleep 1
print -n -e "\b|"
sleep 1
print -n -e "\b/"
sleep 1
echo "ZFS Cloning ended"
exit 0
chmod 0777 /tmp/mon.ksh

Monitor and Control resources usage during ec2bundle command

  1. Try to add to params -v (or –verbose)
  1. ec2-bundle-image command is in ruby and simple use pipes with gzip, is possible then for some reason there is not enough cpu or mem to proccess it.

MEM: try to make couple “sync; sleep 10” commands so zfs cache cleans memory after cloning , monitor MEM, check, if you have swap and you don’t swap when executing ec2-bundle command.

We need keep in mind then ec2tools was designed on Linux, where /tmp is by default on disk, on OpenSolaris? /tmp is by default in MEMORY backuped by SWAP.

CPU: try just to export GZIP zariable with level 1 compression, this will lover cpu load (undocumented by Amazon, but can help isolate issue)

(export GZIP=’-1′; ec2-bundle-image …)

Of course from CPU point of view you can try to run AMI in Second more powerful 32bit profile where you have 2 virtual CPUS

What ec2-bundle-image pipe does

    cat /opt/ec2/lib/ec2/amitools/bundle.rb
# Bundle the AMI procedure:
# The image file is tarred - to maintain sparseness, gzipped
for compression and then encrypted with AES in CBC mode for
# To minimize disk I/O the file is read from disk once and
piped via several processes.     # The tee is used to allow a
digest of the file to be calculated without having to re-read it
from disk.
['tar', "#{openssl} sha1 < #{digest_pipe} & " + tar.expand],
['tee', "tee #{digest_pipe}"],
['gzip', 'gzip'],
['encrypt', "#{openssl} enc -e -aes-128-cbc -K #{key} -iv #{iv} > #{bundled_file_path}"]

1. You can inspect ruby bundle pipe with Solaris commands

1.1 Get rebundle process PID

# ps -ef | grep "ec2-bundle-image -c"
root  6205  2146   0 14:49:52 pts/1       0:00 /bin/bash /opt/ec2/bin/ec2-bundle-image -c

1.2 See command tree

# ptree 6205
6205  /bin/bash /opt/ec2/bin/ec2-bundle-image -c /mnt/keys/cert-GW---------------------CF
6206  ruby -I /opt/ec2/lib /opt/ec2/lib/ec2/amitools/bundleimage.rb -c /mnt/keys/cert
6210  /bin/bash -c /usr/sfw/bin/openssl sha1 < /tmp/ec2-bundle-image-digest-pipe & /u
6211  /usr/sfw/bin/openssl sha1
6212  /usr/sfw/bin/gtar -c -h -S -C /mnt Glassfish_2008.11_32_1.0.img
6213  tee /tmp/ec2-bundle-image-digest-pipe
6214  gzip
6215  /usr/sfw/bin/openssl enc -e -aes-128-cbc -K 48--keep-it-really-sercert--65

1.3 You can see whole constructed pipe as args of process next to ruby:

# pargs 6210

2 .You can try to monitor or even limit PIPE line bandwidth itself by using PV (Pipe Viewer)

Note: For this you need patch a Amazon EC2 ruby library, this is an supported, for debugging purposes only hack.

PV (Pipe Viewer)

With PV you can limit bandwidth so process will not be so resource intensive, for more see PV man here

# pkg set-authority -O contrib
# pkg install pv
#  pv -V
pv 1.1.4 - Copyright(C) 2008 Andrew Wood <>
# cp /opt/ec2/lib/ec2/amitools/bundle.rb /opt/ec2/lib/ec2/amitools/
# vim /opt/ec2/lib/ec2/amitools/bundle.rb
['tar', "#{openssl} sha1 < #{digest_pipe} & " + tar.expand],
['pv','pv -q -L 500k'],
['tee', "tee #{digest_pipe}"],
Hint: If you want to use pv just for progress monitoring use:
['pv','pv -N rebundle -Wpteb -s 10485761024 -B 500000 -f'],
You will see:
Rebundle:  753MB 0:02:36 [==>                           ]  7% ETA 0:31:56
# start ec2-bundle-image command
# Get pv pid
ps -ef | grep pv
root  6794  6791   0 15:38:46 pts/3       0:00 pv -N rebundle -Wptebf -s 10485761024 -B 68157440
Now you can limit bandwith with pv -R PID -L banwidth
# pv -R 6794 -L 100K
You can also play with -B buffer size param

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

Blog at

%d bloggers like this: