Virtual Guru's Blog – Home of Virtualization Workshops

July 15, 2009

Updated script for fully local IPS mirror for faster DC and AI experimenting

Filed under: cloud, jeos, opensolaris, virtualbox, Virtualization, vmware — natiku @ 4:50 am

Updated script for fully local IPS mirror of selected packages for faster OpenSolaris 200906 DC or AI install Prototyping.


This is updated IPS packages subset mirror script with support of OpenSoalris 200906 names with “/” like “system/ldoms/ldomsmanager” and also strip out most foreign arch data (For example SPARC on x86). I use this script to make a fast repeatable installations of already installed packages set of reduced OpenSolaris 200906. I create a fully local IPS repository on local virtual disk and I am installing from localhost, this fully eliminate any dependence on network reliability. I strip our opposite arch to save disk space, you can comment out this code, if you have enough disk space and you want to get 1:1 repo clone of selected packages. Provided script bellow is sample code and is valid only in context of current 200906 repository.

Good news is then in context of DEV (OpenSolaris 200906) we will not need this script any more, I will show it later.

See also OpenSolaris 200811 post: Home No-Network needed OpenSolaris Live-CD IPS REPO full mirror, about the origin of script and more info.

This script is sample of fully local (local disk, localhost net) one arch only repo clone.

cat repo-self.sh
#!/bin/sh
AUTHORITY=http://pkg.opensolaris.org:80/release
TARGET=/localrepos/ips/self
ISA_INFO=`/sbin/uname -p`
echo "INFO: We are on platform: $ISA_INFO"
# This is how I start the ips repository daemon after updating the mirror:
#
# Ideally create a separe zpool for repo
#
# 1. Create a dedicated zfs based system
# echo -y | format
# Searching for disks...done
#   0. c3t0d0 system
#   1. c3t1d0 instance data
#   2. c3t2d0 
#
# zpool create localrepos c3t2d0
# zfs create localrepos/ips
# zfs create localrepos/ips/self
#
# 2. Get repo clone from pre-prepared list
#
# rm -f /tmp/repo-self.log
# repo-self.sh ./OSOL0906-JeOS-pkgsfmris.lst 2>&1 | tee /tmp/repo-self.log
#
# 3. After clone is done, run this pkg.depmod comand to start repo:
#
# screen -L -d -m /usr/lib/pkg.depotd --rebuild -d /localrepos/ips/self -p 10000
#
# ifconfig -a to find global IP so you can check if repository FREEZE of selected packages is created correctly.
#
# fail() { rm -rf ${TDIR}; exit 1; }
if [ ! -d ${TARGET} ]; then
mkdir -p ${TARGET} 2>/dev/null
fi
TDIR=`mktemp -d`
if [ ! -d $1 ]; then
echo "Passed file wilh pkg list of full FMRIs" $1
cat $1 | grep -v "^#" |sed s%\pkg:/%%|sort -u >${TDIR}/packages.lst
else
echo "Geting installed package list from local pkg database ..."
pkg list -v | awk '{print $1}'|sed s%\pkg:/%%|sort -u | sed -e 's/\//%2f/g' >${TDIR}/packages.lst
fi
DIRS="catalog trans updatelog"
for d in $DIRS; do [ -d $d ] && rm -rf $d; done
FILES="search.dir search.pag"
for f in $FILES; do [ -f $f ] && rm -f $f; done
# I add there escaping for / as %2F for names like 'system/ldoms/ldomanager' packages
cat <${TDIR}/packages-download.sh
x() {
PKG_NAME=\`echo \$1|sed s/@.*//\| sed s/\//%2F/g`
PKG_VERS=\`echo \$1|sed s/.*@//|sed s/,/%2C/|sed s/:/%3A/\`
[ -d ${TARGET}/pkg/\$PKG_NAME ] || mkdir -p ${TARGET}/pkg/\$PKG_NAME
echo Downloading \$1
wget ${AUTHORITY}/manifest/0/\$1 -O ${TDIR}/package && mv ${TDIR}/package ${TARGET}/pkg/\$PKG_NAME/\$PKG_VERS
}
EOF
cat ${TDIR}/packages.lst |sed s/\^/x\ / >>${TDIR}/packages-download.sh
sh ${TDIR}/packages-download.sh
echo Generating list of files referenced by the packages...
# For now we exclude binaly files for oposite achitecture i386 and SPARC to save space
# Other stuff like Scripts or License have neglible size and they don't need to be ecluded
#
#
if [ $ISA_INFO = "sparc" ] ; then
for i in `ls ${TARGET}/pkg`; do
cat ${TARGET}/pkg/$i/*
done | grep -v "elfarch=i386"| egrep "^(file|license)\ "|awk '{ print $2 }'|sort -u >${TDIR}/files-to-download.lst
else
for i in `ls ${TARGET}/pkg`; do
cat ${TARGET}/pkg/$i/*
done | grep -v "elfarch=sparc"| egrep "^(file|license)\ "|awk '{ print $2 }'|sort -u >${TDIR}/files-to-download.lst
fi
mkdir ${TDIR}/files.d
split -l 100 -a 8 ${TDIR}/files-to-download.lst ${TDIR}/files.d/files-
# I'm using bash because of the "echo -n" bit
cat <${TDIR}/files-download.sh
for list in \`ls ${TDIR}/files.d\`; do
POST_TMP=\`cat ${TDIR}/files.d/\${list}|nl -s\=|sed s/\^\ */File-Name-/\`
POST=\`for i in \$POST_TMP; do echo -n \&\$i; done\`
echo Downloading...
wget  --post-data="\$POST" ${AUTHORITY}/filelist/0 -O ${TDIR}/files.d/\${list}.tar || fail
echo Extracting...
mkdir ${TDIR}/files.d/\${list}.d
(cd ${TDIR}/files.d/\${list}.d; tar xf ${TDIR}/files.d/\${list}.tar)
for file in \`ls ${TDIR}/files.d/\${list}.d\`; do
FILE_DIRL1=\`echo \$file|cut -c -2\`
FILE_DIRL2=\`echo \$file|cut -c 3-|cut -c -6\`
mkdir -p ${TARGET}/file/\${FILE_DIRL1}/\${FILE_DIRL2} 2>/dev/null && mv \
${TDIR}/files.d/\${list}.d/\$file ${TARGET}/file/\${FILE_DIRL1}/\${FILE_DIRL2} || exit 1
done
rmdir ${TDIR}/files.d/\${list}.d
rm ${TDIR}/files.d/\${list}.tar
rm ${TDIR}/files.d/\${list}
done
EOF
bash ${TDIR}/files-download.sh
#rm -rf ${TDIR}
exit 0
Advertisements

1 Comment »

  1. […] Blogs: Home No-Network needed OpenSolaris Live-CD IPS REPO full mirror Blogs: Updated script for fully local IPS mirror for faster DC and AI experimenting […]

    Pingback by OpenSolaris 200906 Virtualization Assistant Live-CD Proof of Concept « Virtual Guru's Blog – Home of Virtualization Workshops — February 1, 2010 @ 6:15 pm


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: