Virtual Guru's Blog – Home of Virtualization Workshops

December 9, 2009

Personal SourceJuicer in OpenSolaris JeOS Prototype

Filed under: cloud, jeos, opensolaris, virtualbox, Virtualization — natiku @ 8:03 am

OpenSolaris 200906 JeOS Prototype is PURE CLI enviroment, so its small true command line only enviroment, as such is ideal install BASE for compact porting environment.

OpenSolaris provide a Source Juicer project for porting mainly User Land programs, but many programs are still ported by ROBOPORTER with for example not ideal build dependencies.

This example allow you to use OpenSolaris 200906 JeOS Prototype PURE CLI enviroment to check real dependency need of ported applications in your private Source Juicer with pkgbuild like environment.

pkgbuild is tool which help you to port SW to OpenSolaris using a RPM spec like build receipts, this personal Source Juicer enviroment can help to learn such build process.

If you like to have full C devel environment for personal Source Juicer in OpenSolaris 200906 JeOS Prototype, see more in C compile enviroment in OpenSolaris JeOS Prototype


General Information – JeOS Prototype OpenSolaris project

Download OpenSolaris 200906 JeOS Prototype VM for one 10+ most popular Virtualization Formats:

JeOS: Just enough OS Project Homepage

OpenSolaris 2009.06 JeOS Prototype VM Images Download

OpenSolaris 2009.06 JeOS Prototype 10 Simple Usage Samples

Discussion jeos-discuss@opensolaris.org (Main page)

Discussion jeos-discuss@opensolaris.org (Archives)

Part 1: Check JeOS readiness as user root

1. Switch to root

pfexec su - root

2a. Check, if PKG will clean download cache on SUCCESS

root@osol-jeos:~# pkg property flush-content-cache-on-success
PROPERTY                       VALUE
flush-content-cache-on-success True

2b. Check, then we increase PKG download patience for slowest networks

root@osol-jeos:~# set | grep PKG
PKG_CLIENT_TIMEOUT=120
PKG_TIMEOUT_MAX=16

2c. Check, then we use right main repository for opensolaris.org

root@osol-jeos:~# pkg publisher
PUBLISHER                             TYPE     STATUS   URI
opensolaris.org          (preferred)  origin   online   http://pkg.opensolaris.org/release/
contrib                               origin   online   http://pkg.opensolaris.org/contrib/

2d. Check, then we can access these repositories

root@osol-jeos:~# pkg refresh --full

2e. Check, opensolaris.org download speed of whole repository catalog

root@osol-jeos:~# curl pkg.opensolaris.org/release/catalog/0 >/dev/null
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
100 1333k  100 1333k    0     0  93491      0  0:00:14  0:00:14 --:--:--  126k

2f. Check, if we have enought memory and activated swap (I recommend if possible to have VM with 768MB-1GB RAM configured)

root@osol-jeos:~# top | grep Memory
Memory: 1024M phys mem, 525M free mem, 512M total swap, 512M free swap

Part2: Install Source Builder like enviroment

Source Juicer run in zone, here is a diff between default iPKG zone installation and JeOS prototype jeos-discuss adding zones to osol jeos?

pkgbuilder need a C compiler, eve when you will post only non C stuff

1. Install OS default GCC compiler

root@osol-jeos:~# pkg install gcc-dev
DOWNLOAD                                    PKGS       FILES     XFER (MB)
Completed                                  21/21   3097/3097   37.99/37.99
PHASE                                        ACTIONS
Install Phase                              3946/3946
Deleting content cache

2. Optionally: Install CBE environment, this will needed for more complex stuff, it was not needed for porting Python stuff I am interesting in.

Install the [ CBE homeage] , [Desktop CBE download] on opensolaris.org

3. Install latest pkgbuild package, check pkgbuild downloads

root@osol-jeos:~# mkdir ~/pkgbuild
root@osol-jeos:~# cd ~/pkgbuild
root@osol-jeos:~/pkgbuild#
root@osol-jeos:~/pkgbuild# wget http://prdownloads.sourceforge.net/pkgbuild/pkgbuild-1.3.101.tar.bz2?download
root@osol-jeos:~/pkgbuild# bzcat pkgbuild-1.3.101.tar.bz2 | tar xf -
root@osol-jeos:~/pkgbuild# cd pkgbuild-1.3.101
root@osol-jeos:~/pkgbuild/pkgbuild-1.3.101# ./configure --prefix=/opt/pkgbuild
root@osol-jeos:~/pkgbuild/pkgbuild-1.3.101# make
root@osol-jeos:~/pkgbuild/pkgbuild-1.3.101# make install

4. Start local PKG server (by default on port 80) and set it as pkgbuild repository

svcadm enable pkg/server
sleep 10
pfexec pkg set-authority -O http://localhost:80/ pkgbuild

Part 3: Test you private Source Juicer on Python ZSI 2.1_a1 package

1. pkgbuild like to be run as non privileged user, so I will use a JeOS default osol user

root@osol-jeos:~#
root@osol-jeos:~# su - osol
osol@osol-jeos:~$

2. Get JDE specfiles so we have needed SPEC include files

mkdir ~/jds
cd ~/jds
svn co svn+ssh://anon@svn.opensolaris.org/svn/jds/spec-files/trunk/include spec-files/include
mkdir ~/spec-files
cp -v ~/jds/spec-files/include/*  ~/spec-files

For experimenting with VirtualBox 3.x WebSerevices API in Python I need a Python SOAP API ZSI library 2.1_a1 (ZSi2.0 is not enough) ZSI 2.1.a1 package – SOAP for Python

According Source Jucier logs ZSI 2.1.a1 Auto ported by Roboporter on 2009-08-08, but still not in contrib repository , there is only a version ZSI2.0 visible ?

So lets get create a local IPS package for ZSI 2.1._a1 with our Private Source Juicer enviroment using ROBOPORTER spec file.

1. Get zsi.spec from ZSI package link and create zsi.copyright file

root@osol-jeos:~# cd ~/spec-files
osol@osol-jeos:~/spec-files$ wget  http://jucr.opensolaris.org/files/1244/3814/specs/zsi.spec

We also need a license, no download link, so create one using content of original ZSI archive:

Create a zsi.copyright file

cat ~/spec-files/zsi.copyright
Most of ZSI is covered by an MIT/X Consortium copyright, as found
in doc/zsi.tex.
The wsdl2python tools are covered by an LBNL BSD-style license, as found
in ZSI/LBNLCopyright.
Most of the code in wstools is covered by the Zope Public License,
which is a certified open source license that is GPL compatible.
See ZSI/wstools/ZPL for details.

Copy zsi.copyright to build location

mkdir -p ~/packages/SOURCES
cp zsi.copyright ~/packages/SOURCES

2. Add pkgtools to path

osol@osol-jeos:~/spec-files$ export PATH=$PATH:/opt/pkgbuild/bin

3. Pre-download source in zsi.spec, its not on OpenSolaris for now.

osol@osol-jeos:~/spec-files$ cat zsi.spec | grep ".tar."
Source:          ftp://tarball.opensolaris.org/pub/zsi_2.1~a1.orig.tar.gz
Meta(info.repository_url): http://ftp.de.debian.org/debian/pool/main/z/zsi/zsi_2.1~a1.orig.tar.gz
osol@osol-jeos:~/spec-files$ gsed -i 's\ftp://tarball.opensolaris.org/pub/zsi_2.1~a1.orig.tar.gz\ftp.de.debian.org/debian/pool/main/z/zsi/zsi_2.1~a1.orig.tar.gz\g' zsi.spec
cd ~/packages/SOURCES
wget http://ftp.de.debian.org/debian/pool/main/z/zsi/zsi_2.1~a1.orig.tar.gz

4. Make first pkgbuild attempt to check depndecies

osol@osol-jeos:~/spec-files$  pkgtool build --download --ips zsi.spec
...
WARNING: skipping package zsi: required package SUNWpython-setuptools not installed
...

5. OK, lest get list of them

osol@osol-jeos:~/spec-files$  pkgtool build --download --ips zsi.spec | grep "not installed" | awk '{print $7}' | sort | uniq | tee /tmp/zsideps.lst
SUNWgnome-media-center
SUNWgnome-python-libs
SUNWgst-python
SUNWmysql-python
SUNWpython-imaging
SUNWpython-setuptools
SUNWpython-twisted
SUNWpython-twisted-web2
SUNWpython-zope-interface
SUNWpython24-cssutils
SUNWpython24-ctypes

6. WOW, why it wants to get Python GUI and MySQL stuff ???, Lets check dependencies with names

Installing Python GUI stuff will result into 100+MB donwnload , plus MySQL will need additional 80+MB

root@osol-jeos:~# df -h | grep -A 1 opensolaris
rpool/ROOT/opensolaris
7.0G  827M  6.2G  12% /
root@osol-jeos:~# pkg install SUNWgnome-media-center SUNWgnome-python-libs SUNWgst-python SUNWpython-imaging SUNWmysql-python
DOWNLOAD                                    PKGS       FILES     XFER (MB)
Completed                                  64/64 14952/14952 241.24/241.24
PHASE                                        ACTIONS
Install Phase                            18355/18355
Deleting content cache
root@osol-jeos:~# df -h | grep -A 1 opensolaris
rpool/ROOT/opensolaris
7.0G  1.7G  5.4G  24% /
osol@osol-jeos:~/spec-files$ cat zsi.spec | grep "BuildRequires:" | awk '{print $2}'| xargs -n1 pkg list -saH
SUNWPython                     The Python interpreter, libraries and utilities
SUNWPython-extra               Supplemental Python libraries and utilities
SUNWbash                       GNU Bourne-Again shell (bash)
SUNWbtool                      CCS tools bundled with SunOS
SUNWcs                         Core Solaris
SUNWcsl                        Core Solaris, (Shared Libs)
SUNWesu                        Extended System Utilities
SUNWgcc                        gcc - The GNU C compiler
SUNWggrp                       ggrep - GNU grep utilities
SUNWgnome-media-center         Media center written in Python
SUNWgnome-python-libs          Python support libraries for GNOME
SUNWgnu-coreutils              coreutils - GNU core utilities
SUNWgst-python                 Python bindings for the GStreamer streaming media framework
SUNWgtar                       gtar - GNU tar
SUNWgzip                       The GNU Zip (gzip) compression utility
SUNWinstall-libs               System install libraries
SUNWipkg
SUNWipkg                       Image Packaging System
SUNWlibms                      Math & Microtasking Libraries
SUNWlibsasl                    SASL v2
SUNWloc                        System Localization
SUNWlxml                       The XML library
SUNWmysql-python               A MySQL database adapter for the Python programming language
SUNWopenssl                    OpenSSL Commands
SUNWperl584core                Perl 5.8.4 (core)
SUNWpool                       Resource Pools
SUNWpr                         Netscape Portable Runtime
SUNWpython-cherrypy            CherryPy
SUNWpython-imaging             Python's own image processing library
SUNWpython-mako                Mako
SUNWpython-ply                 Ply
SUNWpython-pyopenssl           pyOpenSSL - Python interface to the OpenSSL library
SUNWpython-setuptools          Download, build, install, upgrade, and uninstall Python packages easily
SUNWpython-twisted             Event-based framework for internet applications
SUNWpython-zope-interface      A separate distribution of the zope.interface package used in Zope 3
SUNWpython24-cssutils          A Python 2.4 package to parse and build CSS Cascading Style Sheets.
SUNWpython24-ctypes            Python C data types
SUNWpython24-simplejson        JSON (Java Script Object Notation) encoder/decoder for Python 2.4
SUNWtls                        Network Security Services
SUNWxcu4                       XCU4 Utilities
SUNWzfs                        ZFS
SUNWzlib                       The Zip compression library
SUNWzone                       Solaris Zones
osol@osol-jeos:~/spec-files$ cat zsi.spec | grep -v "BuildRequires:" | grep "Requires" | awk '{print $2}'| xargs -n1 pkg list -saH
SUNWPython                     The Python interpreter, libraries and utilities
SUNWPython-extra               Supplemental Python libraries and utilities
SUNWpython-pyopenssl           pyOpenSSL - Python interface to the OpenSSL library
SUNWpython-setuptools          Download, build, install, upgrade, and uninstall Python packages easily
SUNWpython-twisted             Event-based framework for internet applications
SUNWpython-twisted-web2        A HTTP/1.1 Server Framework
SUNWpython24-ctypes            Python C data types

7. Lets remove some olpran build depndecies

osol@osol-jeos:~/spec-files$ gsed -i 's/BuildRequires:\tSUNWmysql-python/#BuildRequires:\tSUNWmysql-python/g' zsi.spec
osol@osol-jeos:~/spec-files$ gsed -i 's/BuildRequires:\tSUNWgnome-media-center/#BuildRequires:\tSUNWgnome-media-center/g' zsi.spec
osol@osol-jeos:~/spec-files$ gsed -i 's/BuildRequires:\tSUNWgnome-python-libs/#BuildRequires:\tSUNWgnome-python-libs/g' zsi.spec
osol@osol-jeos:~/spec-files$ gsed -i 's/BuildRequires:\tSUNWgst-python/#BuildRequires:\tSUNWgst-python/g' zsi.spec
osol@osol-jeos:~/spec-files$ gsed -i 's/BuildRequires:\tSUNWpython-imaging/#BuildRequires:\tSUNWpython-imaging/g' zsi.spec
osol@osol-jeos:~/spec-files$ gsed -i 's/BuildRequires:\tSUNWpython-zope-interface/#BuildRequires:\tSUNWpython-zope-interface/g' zsi.spec
osol@osol-jeos:~/spec-files$ gsed -i 's/BuildRequires:\tSUNWpython24-cssutils/#BuildRequires:\tSUNWpython24-cssutils/g' zsi.spec

8. Check rest of dependencies

osol@osol-jeos:~/spec-files$ pkgtool build --download --ips zsi.spec | grep "not installed" | awk '{print $7}' | sort | uniq | tee /tmp/zsideps.lst
SUNWpython-setuptools
SUNWpython-twisted
SUNWpython-twisted-web2
SUNWpython24-ctypes

9. Lets check rest of dependencies with IPS dry run

osol@osol-jeos:~/spec-files$ cat  /tmp/zsideps.lst | xargs pkg install -nv
Creating Plan / Before evaluation:
UNEVALUATED:
+pkg:/SUNWpython-setuptools@0.5.11,5.11-0.111:20090508T163114Z
+pkg:/SUNWpython-twisted@0.5.11,5.11-0.111:20090508T163116Z
+pkg:/SUNWpython-twisted-web2@0.5.11,5.11-0.111:20090508T163134Z
+pkg:/SUNWpython24-ctypes@0.5.11,5.11-0.111:20090508T163141Z
After evaluation:
None -> pkg:/SUNWpython-setuptools@0.5.11,5.11-0.111:20090508T163114Z
None -> pkg:/SUNWpython-twisted@0.5.11,5.11-0.111:20090508T163116Z
None -> pkg:/SUNWpython-twisted-web2@0.5.11,5.11-0.111:20090508T163134Z
None -> pkg:/SUNWpython24-ctypes@0.5.11,5.11-0.111:20090508T163141Z
None -> pkg:/SUNWpython-zope-interface@0.5.11,5.11-0.111:20090508T163137Z

10. Lets install them with IPS PKG command

You can see, now I need to install just 6MB, instead of nearly 200MB

osol@osol-jeos:~/spec-files$  cat  /tmp/zsideps.lst | xargs pfexec pkg install
DOWNLOAD                                    PKGS       FILES     XFER (MB)
Completed                                    5/5   1982/1982     5.89/5.89
PHASE                                        ACTIONS
Install Phase                              2178/2178
Deleting content cache

11. Try to build ZSI again

osol@osol-jeos:~/spec-files$ pkgtool build --download --ips zsi.spec
INFO: IPS packages will be installed by default from http://localhost:80/
INFO: Copying %use'd or %include'd spec files to SPECS directory
INFO: Processing spec files
INFO: Finding sources
INFO: Running pkgbuild -ba [...] zsi.spec (zsi)
INFO: zsi PASSED
INFO: Installing zsi from authority pkgbuild
Summary:
package |      status | details
---------------------------------+-------------+-------------------------------
zsi |      PASSED |

12. Make some checks

osol@osol-jeos:~/spec-files$ pkg list -s | grep zsi
zsi (pkgbuild)                 Zolera SOAP Infrastructure (ZSI)
osol@osol-jeos:~/spec-files$ pkg list -a zsi
NAME (PUBLISHER)                              VERSION         STATE      UFIX
zsi (pkgbuild)                                2.1.1-0.111     installed  ----
osol@osol-jeos:~/spec-files$ pkg list -av zsi
FMRI                                                             STATE      UFIX
pkg://pkgbuild/zsi@2.1.1,5.11-0.111:20091209T164138Z             installed  ----
osol@osol-jeos:~/spec-files$ pkg contents -m pkg://pkgbuild/zsi@2.1.1,5.11-0.111:20091209T164138Z
set name=fmri value=pkg://pkgbuild/zsi@2.1.1,5.11-0.111:20091209T164138Z
....
links http://locahost:80
osol@osol-jeos:~/spec-files$ wget -q http://localhost:80/en/catalog.shtml -O - | grep ">zsi<"
<td><a href="../info/0/zsi%402.1.1%2C5.11-0.111%3A20091209T164138Z">zsi</a></td>

Hinta: Debugging pkgbuild with -v -v -v

osol@osol-jeos:~/spec-files$ pkgtool build -v -v -v --download --ips zsi.spec
I see for example this output, but I was not able to debug it with '''-v -v -v''' options.
INFO: Processing spec files
basename: missing operand
Try `basename --help' for more information.
basename: missing operand
Try `basename --help' for more information.
basename: missing operand
Try `basename --help' for more information.
Advertisements

3 Comments »

  1. someone aleady tried running the http://tinyurl.com/bootstrap-sfe-latest to get a spec-files-extra and spec-files-jucr like environment by a single script run?

    Comment by Thomas Wagner — December 11, 2009 @ 3:16 am

  2. […] Personal SourceJuicer in OpenSolaris JeOS Prototype […]

    Pingback by OpenSolaris JeOS (Main Page) « Virtual Guru's Blog – Home of Virtualization Workshops — February 7, 2010 @ 3:49 pm

  3. […] In OpenSolaris 200906 a sample zone can be created on ZFS system and after it can be cloned after to working zones, this way we can play can always clean zone, for example as SW porting enviroment, see more in Personal SourceJuicer in OpenSolaris JeOS Prototype […]

    Pingback by JeOS like Zones in OpenSolaris JeOS Prototype « Virtual Guru's Blog – Home of Virtualization Workshops — February 7, 2010 @ 5:37 pm


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

%d bloggers like this: