SSH as a Reverse Proxy

With the Internet of things, we get used to all devices being able to communicate to each other. Recently, I was tasked with creating a virtual private cloud (VPC) on AWS that was secure and could not route traffic outbound which normally would be done by a NAT instance. In this design, each instance was deployed from an AMI and was accessed via SSH through an OpenVPN instance.

This was nice and secure, but what about periodic updates and installing software after the installation? These are in the “cloud” and it should be pretty straightforward to enable the instances to communicate outbound for these administrative duties on a temporary basis.

SSH comes to the rescue on this one. You can use the -D option for application forwarding over the secure channel via SOCKS protocol which is handled by SSH on the remote node. In this case, the only machine able to communicate outside the VPC to the Internet was the OpenVPN server.

Since the admin is able to access this node, the admin connects to the VPN, SSH to the guest and needs to connect back to the OpenVPN server to reverse proxy any data necessary. You do not need to establish a pty on the remote node to execute commands, so you can use the -N option. This is very convenient for port forwarding or proxying. You should run this in the background so you have access to the terminal for executing the necessary commands:

$ ssh -D localhost:3128 -N username@remote-server &

At this point, you have the ability to proxy using SOCKS via the remote-server. However, the application you use needs to support it. The Bash shell conveniently provides the http_proxy environment variable for this purpose.

$ export http_proxy=socks4://localhost:3128

You can test your reverse proxy by fetching a web page using `curl`  and writing the output to the stdout. In this test, we will use httpbin to output the headers as seen and returned by the remote web server:

$ curl -o- http://httpbin.org/headers
{
  "headers": {
    "Accept": "*/*", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.29.0"
  }
}

Success! But, what about `yum`? Thanks for asking. You can edit the configuration normally located at /etc/yum.conf and add the following line:

proxy=socks4://localhost:3128

Now, `yum` will fetch and install packages via the SSH reverse proxy. There is much more you can do with SOCKS since it is a lower level protocol than HTTP. These two examples are very straightforward and provide quite a bit of usability using a reverse proxy while your guests stay secure and are not able to route traffic outbound via a gateway.

Now, it is time to clean up and exit. You can find the process you’ve placed in the background, kill it and then logout:

$ jobs
[1]+  Running                 ssh -D localhost:3128 -N username@remote-server &
$ kill %1
$ 
[1]+  Done                    ssh -D localhost:3128 -N username@remote-server
$ logout

During this time, the hosts have not been remotely accessible, but local processes have been able to fetch remote data so you could get the job done!

Directories Greyed Out in OSX Mavericks’ Finder

I’ve been gone for a while working on a major project with a lot of pieces. While putting those pieces together, I came across a silly issue that rarely causes problems. In this case, I installed IntelliJ IDEA and the Go plugin to work on the biggest piece of this puzzle. I know you are curious, but I will explain the project in due time.

The problem was, that I needed to select the Go SDK in IDEA for the project. However, it would not let me navigate to the folder where it lived. The folder was in /opt/local using Macports. It would not display /opt as well as many other folders in /. I figured it was a permission, acl or extended attribute.

According to `man ls`, if the long listing of `ls` outputs a ‘@’ at the end of the file modes, then the file has extended attributes. If there is a ‘+’, then the file has extended security information such as access control lists. I used `xattr` and got this:

$ xattr /opt
com.apple.FinderInfo

So I then performed:

$ sudo xattr -d com.apple.FinderInfo /opt

Needless to say, that did not correct the problem. /opt was still greyed out in Finder. I then turned my attention to acl’s, but that did not show any information worth easter egging:

$ ls -le /
total 16453
drwxrwxr-x+ 94 root  admin     3196 24 Sep 09:59 Applications
 0: group:everyone deny delete
drwxr-xr-x+ 71 root  wheel     2414 28 Jul 17:58 Library
 0: group:everyone deny delete
drwxr-xr-x@  2 root  wheel       68 25 Aug  2013 Network
drwxr-xr-x+  4 root  wheel      136 27 Oct  2013 System
 0: group:everyone deny delete
lrwxr-xr-x   1 root  wheel       60  2 May  2012 User Guides And Information -> /Library/Documentation/User Guides and Information.localized
lrwxr-xr-x   1 root  wheel       49 10 Jul  2013 User Information -> /Library/Documentation/User Information.localized
drwxr-xr-x   6 root  admin      204 30 Nov  2013 Users
drwxrwxrwt@  4 root  admin      136 24 Sep 11:44 Volumes
 0: group:everyone deny add_file,add_subdirectory,directory_inherit,only_inherit
drwxr-xr-x   3 root  wheel      102 17 Sep  2012 asix_temp
drwxr-xr-x@ 39 root  wheel     1326  3 Jul 10:50 bin
drwxrwxr-t@  2 root  admin       68 25 Aug  2013 cores
dr-xr-xr-x   3 root  wheel     4309 24 Sep 11:40 dev
lrwxr-xr-x@  1 root  wheel       11 27 Oct  2013 etc -> private/etc
dr-xr-xr-x   2 root  wheel        1 24 Sep 11:46 home
drwxrwxrwt   2 root  wheel       68 28 Jul 22:43 lost+found
-rwxr-xr-x@  1 root  wheel  8394200 18 Aug 03:50 mach_kernel
dr-xr-xr-x   2 root  wheel        1 24 Sep 11:46 net
drwxrwxr-x   5 root  wheel      170 24 Sep 09:24 opt
drwxr-xr-x@  6 root  wheel      204 27 Oct  2013 private
drwxr-xr-x@ 62 root  wheel     2108 18 Sep 13:11 sbin
lrwxr-xr-x@  1 root  wheel       11 27 Oct  2013 tmp -> private/tmp
drwxr-xr-x@ 14 root  wheel      476  7 Nov  2013 usr
lrwxr-xr-x@  1 root  wheel       11 27 Oct  2013 var -> private/var

I then decided to look at file system flags which can contain the hidden flag:

$ man chflags
...
hidden  set the hidden flag [Hide item from GUI]
...

You can list your file system’s flags by using the -O option to `ls`:

$ ls -lO /
total 16453
drwxrwxr-x+ 94 root  admin  -         3196 24 Sep 09:59 Applications
drwxr-xr-x+ 71 root  wheel  -         2414 28 Jul 17:58 Library
drwxr-xr-x@  2 root  wheel  hidden      68 25 Aug  2013 Network
drwxr-xr-x+  4 root  wheel  -          136 27 Oct  2013 System
lrwxr-xr-x   1 root  wheel  -           60  2 May  2012 User Guides And Information -> /Library/Documentation/User Guides and Information.localized
lrwxr-xr-x   1 root  wheel  -           49 10 Jul  2013 User Information -> /Library/Documentation/User Information.localized
drwxr-xr-x   6 root  admin  -          204 30 Nov  2013 Users
drwxrwxrwt@  4 root  admin  hidden     136 24 Sep 11:44 Volumes
drwxr-xr-x   3 root  wheel  -          102 17 Sep  2012 asix_temp
drwxr-xr-x@ 39 root  wheel  hidden    1326  3 Jul 10:50 bin
drwxrwxr-t@  2 root  admin  hidden      68 25 Aug  2013 cores
dr-xr-xr-x   3 root  wheel  hidden    4309 24 Sep 11:40 dev
lrwxr-xr-x@  1 root  wheel  hidden      11 27 Oct  2013 etc -> private/etc
dr-xr-xr-x   2 root  wheel  hidden       1 24 Sep 11:46 home
drwxrwxrwt   2 root  wheel  -           68 28 Jul 22:43 lost+found
-rwxr-xr-x@  1 root  wheel  hidden 8394200 18 Aug 03:50 mach_kernel
dr-xr-xr-x   2 root  wheel  hidden       1 24 Sep 11:46 net
drwxrwxr-x   5 root  wheel  hidden     170 24 Sep 09:24 opt
drwxr-xr-x@  6 root  wheel  hidden     204 27 Oct  2013 private
drwxr-xr-x@ 62 root  wheel  hidden    2108 18 Sep 13:11 sbin
lrwxr-xr-x@  1 root  wheel  hidden      11 27 Oct  2013 tmp -> private/tmp
drwxr-xr-x@ 14 root  wheel  hidden     476  7 Nov  2013 usr
lrwxr-xr-x@  1 root  wheel  hidden      11 27 Oct  2013 var -> private/var

Well, what do you know? There was the culprit! As already mentioned, this issue rarely creeps up. But, this is easily solvable by removing the flag:

$ sudo chflags nohidden /opt
jcmbpro:js jcook$ ls -lO /
total 16453
drwxrwxr-x+ 94 root  admin  -         3196 24 Sep 09:59 Applications
drwxr-xr-x+ 71 root  wheel  -         2414 28 Jul 17:58 Library
drwxr-xr-x@  2 root  wheel  hidden      68 25 Aug  2013 Network
drwxr-xr-x+  4 root  wheel  -          136 27 Oct  2013 System
lrwxr-xr-x   1 root  wheel  -           60  2 May  2012 User Guides And Information -> /Library/Documentation/User Guides and Information.localized
lrwxr-xr-x   1 root  wheel  -           49 10 Jul  2013 User Information -> /Library/Documentation/User Information.localized
drwxr-xr-x   6 root  admin  -          204 30 Nov  2013 Users
drwxrwxrwt@  4 root  admin  hidden     136 24 Sep 11:44 Volumes
drwxr-xr-x   3 root  wheel  -          102 17 Sep  2012 asix_temp
drwxr-xr-x@ 39 root  wheel  hidden    1326  3 Jul 10:50 bin
drwxrwxr-t@  2 root  admin  hidden      68 25 Aug  2013 cores
dr-xr-xr-x   3 root  wheel  hidden    4309 24 Sep 11:40 dev
lrwxr-xr-x@  1 root  wheel  hidden      11 27 Oct  2013 etc -> private/etc
dr-xr-xr-x   2 root  wheel  hidden       1 24 Sep 11:46 home
drwxrwxrwt   2 root  wheel  -           68 28 Jul 22:43 lost+found
-rwxr-xr-x@  1 root  wheel  hidden 8394200 18 Aug 03:50 mach_kernel
dr-xr-xr-x   2 root  wheel  hidden       1 24 Sep 11:46 net
drwxrwxr-x   5 root  wheel  -          170 24 Sep 09:24 opt
drwxr-xr-x@  6 root  wheel  hidden     204 27 Oct  2013 private
drwxr-xr-x@ 62 root  wheel  hidden    2108 18 Sep 13:11 sbin
lrwxr-xr-x@  1 root  wheel  hidden      11 27 Oct  2013 tmp -> private/tmp
drwxr-xr-x@ 14 root  wheel  hidden     476  7 Nov  2013 usr
lrwxr-xr-x@  1 root  wheel  hidden      11 27 Oct  2013 var -> private/var

After you have removed the hidden flag, you can immediately kill Finder windows and it will reappear, showing the directory that was greyed out. Now, in your application — in this case IntelliJ IDEA — you can navigate to that directory and select the folder you so desired.

$ killall Finder

Good luck!

Netapp 8.2.1 Simulator in VirtualBox on OSX

Recently, I needed to interface with many Netapp appliances via the ZAPI over HTTP. I have development access to non-production devices, but surely there is a better way to do it on your local computer with a virtual device. Well, of course there is!

Downloads

If you head over to Netapp’s site for the simulator — you will need an account — there are several options available such as release version, 7-mode, Clustered-ONTAP and the VMware release. In this article, I will be using release 8.2.1 in 7-mode for VMware Fusion. However, the instructions should work for any of the devices.

To start, you need to have VirtualBox installed. I have the latest version, which as of this writing is 4.3.12. In VirtualBox, select Preferences > Network. Under Nat Networks, select “Add NAT Network”. I will call it “NetappNAT”. Select “Edit NAT Network” and place “10.0.100.0/24” for the network and CIDR address. Then, deselect “Supports DHCP”. Save and exit the menu.

Believe it or not, VirtualBox is now ready! But, we need to export the VMware guest into a format VirtualBox can use. The tools necessary are VMware’s `ovftool` and `vmware-vdiskmanager`. The link for the version of ovftool I use is for version 3.5.2. You will need a VMware account to access the download.

VMware Fusion provides vmware-vdiskmanager. Download the free trial and install. You now have all the tools you need to convert, finesse, and install the simulator.

Netapp Simulator

To save the trouble of configuration such as size of RAM, network interfaces and storage controllers, we’ll let the already-configured VMware guest do most of that for us. To start, untar the downloaded appliance. This will extract circa 130 files:

$ tar zxvf vsim_netapp-7m.tgz
x vsim_netapp-7m/
x vsim_netapp-7m/cfcard/
x vsim_netapp-7m/cfcard/env/
x vsim_netapp-7m/cfcard/env/env
x vsim_netapp-7m/nvram
x vsim_netapp-7m/DataONTAP.vmdk
...

OVFTool

The `ovftool` binary is in “/Applications/VMware OVF Tool”. Point to the vmx file and provide a path for the ovf files to be placed. The path provided will be a directory created. This will take several minutes to complete:

$ ovftool vsim_netapp-7m/DataONTAP.vmx vsim_netapp-7m_ovf
Opening VMX source: vsim_netapp-7m/DataONTAP.vmx
Opening OVF target: vsim_netapp-7m_ovf
Writing OVF package: vsim_netapp-7m_ovf/vsim_netapp-7m/vsim_netapp-7m.ovf
Transfer Completed 
Completed successfully

Merge VMDK Files

All of that trouble was for both the ovf and mf files. The vmdk’s it created are not necessary. I wish I knew why, but they would not import without errors. We have things to get done; therefore, go ahead and delete them:

$ rm vsim_netapp-7m_ovf/vsim_netapp-7m/*.vmdk

Now, recreate the vmdk files you just deleted. Use the same name for the destination; because, when you manually add them, you will add them in numerical order:

$ vmware-vdiskmanager -r vsim_netapp-7m/DataONTAP.vmdk -t0 vsim_netapp-7m_ovf/vsim_netapp-7m/vsim_netapp-7m-disk1.vmdk
$ vmware-vdiskmanager -r vsim_netapp-7m/DataONTAP-var.vmdk -t0 vsim_netapp-7m_ovf/vsim_netapp-7m/vsim_netapp-7m-disk2.vmdk
$ vmware-vdiskmanager -r vsim_netapp-7m/DataONTAP-nvram.vmdk -t0 vsim_netapp-7m_ovf/vsim_netapp-7m/vsim_netapp-7m-disk3.vmdk
$ vmware-vdiskmanager -r vsim_netapp-7m/DataONTAP-sim.vmdk -t0 vsim_netapp-7m_ovf/vsim_netapp-7m/vsim_netapp-7m-disk4.vmdk

 Edit OVF File

Open the vsim_netapp-7m.ovf file and remove all references to the disk items in the ovf file. For convenience, the diff is provided below:

$ diff -u vsim_netapp-7m.ovf.orig vsim_netapp-7m.ovf
--- vsim_netapp-7m.ovf.orig 2014-07-04 14:32:30.000000000 +0100
+++ vsim_netapp-7m.ovf 2014-07-04 14:33:16.000000000 +0100
@@ -1,19 +1,6 @@
 <?xml version="1.0" encoding="UTF-8"?>
 <!--Generated by VMware ovftool 3.5.2 (build-1880279), UTC time: 2014-07-04T12:58:28.250867Z-->
 <Envelope vmw:buildId="build-1880279" xmlns="http://schemas.dmtf.org/ovf/envelope/1" xmlns:cim="http://schemas.dmtf.org/wbem/wscim/1/common" xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1" xmlns:rasd="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData" xmlns:vmw="http://www.vmware.com/schema/ovf" xmlns:vssd="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_VirtualSystemSettingData" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
- <References>
- <File ovf:href="vsim_netapp-7m-disk1.vmdk" ovf:id="file1" ovf:size="432187392"/>
- <File ovf:href="vsim_netapp-7m-disk2.vmdk" ovf:id="file2" ovf:size="71168"/>
- <File ovf:href="vsim_netapp-7m-disk3.vmdk" ovf:id="file3" ovf:size="72704"/>
- <File ovf:href="vsim_netapp-7m-disk4.vmdk" ovf:id="file4" ovf:size="113152"/>
- </References>
- <DiskSection>
- <Info>Virtual disk information</Info>
- <Disk ovf:capacity="1057" ovf:capacityAllocationUnits="byte * 2^20" ovf:diskId="vmdisk1" ovf:fileRef="file1" ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized" ovf:populatedSize="440205312"/>
- <Disk ovf:capacity="1542" ovf:capacityAllocationUnits="byte * 2^20" ovf:diskId="vmdisk2" ovf:fileRef="file2" ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized" ovf:populatedSize="65536"/>
- <Disk ovf:capacity="5122" ovf:capacityAllocationUnits="byte * 2^20" ovf:diskId="vmdisk3" ovf:fileRef="file3" ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized" ovf:populatedSize="65536"/>
- <Disk ovf:capacity="250" ovf:capacityAllocationUnits="byte * 2^30" ovf:diskId="vmdisk4" ovf:fileRef="file4" ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized" ovf:populatedSize="0"/>
- </DiskSection>
 <NetworkSection>
 <Info>The list of logical networks</Info>
 <Network ovf:name="hostonly">
@@ -82,38 +69,6 @@
 <vmw:Config ovf:required="false" vmw:key="yieldOnPoll" vmw:value="true"/>
 </Item>
 <Item>
- <rasd:AddressOnParent>0</rasd:AddressOnParent>
- <rasd:ElementName>disk0</rasd:ElementName>
- <rasd:HostResource>ovf:/disk/vmdisk1</rasd:HostResource>
- <rasd:InstanceID>7</rasd:InstanceID>
- <rasd:Parent>4</rasd:Parent>
- <rasd:ResourceType>17</rasd:ResourceType>
- </Item>
- <Item>
- <rasd:AddressOnParent>1</rasd:AddressOnParent>
- <rasd:ElementName>disk1</rasd:ElementName>
- <rasd:HostResource>ovf:/disk/vmdisk2</rasd:HostResource>
- <rasd:InstanceID>8</rasd:InstanceID>
- <rasd:Parent>4</rasd:Parent>
- <rasd:ResourceType>17</rasd:ResourceType>
- </Item>
- <Item>
- <rasd:AddressOnParent>0</rasd:AddressOnParent>
- <rasd:ElementName>disk2</rasd:ElementName>
- <rasd:HostResource>ovf:/disk/vmdisk3</rasd:HostResource>
- <rasd:InstanceID>9</rasd:InstanceID>
- <rasd:Parent>3</rasd:Parent>
- <rasd:ResourceType>17</rasd:ResourceType>
- </Item>
- <Item>
- <rasd:AddressOnParent>1</rasd:AddressOnParent>
- <rasd:ElementName>disk3</rasd:ElementName>
- <rasd:HostResource>ovf:/disk/vmdisk4</rasd:HostResource>
- <rasd:InstanceID>10</rasd:InstanceID>
- <rasd:Parent>3</rasd:Parent>
- <rasd:ResourceType>17</rasd:ResourceType>
- </Item>
- <Item>
 <rasd:AddressOnParent>1</rasd:AddressOnParent>
 <rasd:AutomaticAllocation>true</rasd:AutomaticAllocation>
 <rasd:Connection>hostonly</rasd:Connection>
@@ -168,4 +123,4 @@
 </Item>
 </VirtualHardwareSection>
 </VirtualSystem>
-</Envelope>
\ No newline at end of file
+</Envelope>

Once you have done that, the sha1 sum needs to be recalculated and updated in the vsim_netapp-7m.mf file.

$ shasum vsim_netapp-7m.ovf
8819239ab5a81b2fa9e67b27e93333a4f088687d vsim_netapp-7m.ovf

Replace the sum in the mf file and remove the sums of the vmdk file entries. Otherwise, the import will fail!

Import to VirtualBox

Select File > Import Appliance in the VirtualBox menu. Navigate to the OVF file you just edited and select it. The display will populate with the configuration. You may want to change the name. Go ahead and do so if you wish, and then select Import.

There are now a few settings that need to be changed. Select the new guest and settings. Mainly, the disks need to be added in numerical order. A floppy controller needs to be created with an empty disk attached. If this is not present, the appliance will not boot. The host only networks need to be changed to internal network. Increase the display memory in Display > Video to 16MB. Finally, enable COM1 serial port. Port mode should be host pipe; select “Create Pipe”; provide a File Path such as /tmp/netapp1. Please see the attached images for clarity.

Start the simulator! After a few seconds, you will see “Press Ctrl-C for Boot Menu”. Please do so, press 4 and <enter>. After a few seconds, you will be prompted, “Zero disks, reset config, and install a new file system”. Type “yes” and press <enter>. You will then be prompted, “This will erase all the data on the disks, are you sure?” Again, type “yes” and press <enter>.

The appliance will be terminated. Shut it down and restart. Let it boot without interfering. You will see, “Wipe filer procedure requested” along with verbose output. You will finally see a series of ‘…..’ on the screen for several lines.

After several minutes, it will enter setup and you will be prompted with “Please enter the new hostname[]:”. Enjoy!

 

 

 

 

 

 

 

 

 

 

RHEL7 on VirtualBox with VBoxLinuxAdditions

RedHat Enterprise Linux 7 Released

Everyone be happy! After some delay, RHEL7 is finally with us. And, it has a significant amount of additional functionality such as XFS by default, Docker containers, active directory integration, OpenLMI — the list goes on. RHEL5 was a very large improvement over previous RHEL releases, and RHEL7 is that and more.

In my previous post, I shared with you how to install VirtualBox on Mac OSX Mavericks. Today, I decided to install a RHEL7 guest in order to play around with it. I have tested the beta release on VirtualBox, but did not install guest additions.

Installing RHEL7 is the same as the previous post. However, there is a difference when it comes to installing the VirtualBox guest additions. In my case, I do not have an installation registered with a repository; so, I mounted the CD and created a pointer to the repo by creating the following file with contents:

$ cat /etc/yum.repos.d/rhel7.repo 
[RHEL7]
name=Red Hat Enterprise Linux 7
baseurl=file:///media/RHEL/
enabled=1
gpgcheck=0
gpgkey=file:///media/RHEL/RPM-GPG-KEY-redhat-release

You can now install using `yum` with the local disk repository. And, you’ll need to install prerequisites to build the VBoxLinuxAdditions. I only performed a minimal install; so, I needed quite a few tools. Please note, the following command includes several dependencies such as Perl and `make`:

$ sudo yum install bzip2 kernel-devel gcc

Once that was complete — and it was fast — the normal installation of VirtualBox guest additions failed with a compilation error. After about 30 seconds of Google, I found this. To make a long story short, there is a backport from kernel 3.13 to the RHEL7 kernel, 3.10. Since this is RedHat specific, you will need to correct this.

There is a bug report to VirtualBox that someone linked the fix. It seems there is no fix committed to the VirtualBox installer, yet. After all, it was just released. I imagine the developers will correct this in due time.

In order to manually install the guest additions, you will need to extract the source from VBoxLinuxAdditions.run, edit the file, and then manually execute the installation. You can do so with the following:

$ sh /path/to/mounted/VBoxLinuxAdditions.run --noexec --target ~/vbox-extract
$ ls ~/vbox-extract/
deffiles  install.sh  vboxadd         VBoxGuestAdditions-amd64.tar.bz2
do_dkms   LICENSE     vboxadd-service VBoxGuestAdditions-x86.tar.bz2
installer routines.sh vboxadd-x11

Notice there are tarballs for each supported architecture. Untar the specific tarball, navigate to the affected file — src/vboxguest-4.3.12/vboxguest/r0drv/linux/memobj-r0drv-linux.c — and open it with your favorite editor. Make the corrections provided, recreate the tarball, and then manually execute the installer:

$ sudo ./install.sh 
VirtualBox Guest Additions installer
Removing installed version 4.3.12 of VirtualBox Guest Additions...
Copying additional installer modules ...
Installing additional modules ...
Removing existing VirtualBox non-DKMS kernel modules        [ OK ]
Building the VirtualBox Guest Additions kernel modules
Building the main Guest Additions module                    [ OK ]
Building the shared folder support module                   [ OK ]
Building the OpenGL support module                          [ OK ]
Doing non-kernel setup of the Guest Additions               [ OK ]
Starting the VirtualBox Guest Additions                     [ OK ]
Installing the Window System drivers
Could not find the X.Org or XFree86 Window System, skipping.

Once you have complete, reboot your guest and enjoy! You can perform a `dmesg` and ensure things are working as intended.

$ dmesg | grep vbox
[   15.294965] vboxguest: module verification failed: signature and/or required key missing - tainting kernel
[   15.299797] vboxguest: major 0, IRQ 20, I/O port d020, MMIO at 00000000f0400000 (size 0x400000)
[   15.299797] vboxguest: Successfully loaded version 4.3.12 (interface 0x00010004)
[   15.823884] [drm] Initialized vboxvideo 1.0.0 20090303 for 0000:00:02.0 on minor 0
[   19.082299] vboxsf: Successfully loaded version 4.3.12 (interface 0x00010004)

Command-line VirtualBox on OSX Mavericks

I’ve recently bought a Mac Mini so I will have access to a machine for development purposes, a remote home server and and taking advantage of my super-fast home connection. In this post, I will share the steps taken and my configuration in case anyone is interested. If I have missed something, please feel free to let me know.

Use your favorite SSH client to connect to your OSX server. Once you have done that, you are ready to begin. In the example below, please note, we are forwarding port 5000 to the server’s port 5000. This will be explained later.

$ ssh -L 5000:localhost:5000 your.home.server

First, OSX has quite optimal power-saving features enabled by default. When OSX sleeps, your guests will pause. So, you need to configure it not to sleep. This is quick and easy.

$ sudo pmset -a sleep 0

In this example, I use Scientific Linux as the guest operating platform since it is a RedHat Enterprise Server respin. If you do not have it, go ahead and download the image of your choice. I prefer the boot-only iso and perform a network install.

$ curl -o SL-65-x86_64-2013-12-05-boot.iso http://ftp1.scientificlinux.org/linux/scientific/6x/x86_64/iso/SL-65-x86_64-2013-12-05-boot.iso

Now, it is time to install VirtualBox. Head over to virtualbox.org and download the latest OSX package along with the extension pack. Once you have these two files, you need to install the application from the dmg. First, mount the dmg volume and execute the installer with root privileges. Once this is done, you should then detach the volume. You will now have VirtualBox installed.

$ hdiutil attach VirtualBox-4.3.12-93733-OSX.dmg
$ sudo installer -pkg /Volumes/VirtualBox/VirtualBox.pkg -target /Volumes/Macintosh\ HD
$ hdiutil info     # You may not need this but it lists the device path below
$ hdiutil detach /dev/disk1s2

At this point, you may install guests. However, for various reasons, the extension pack is preferred. If you would like to use the remote desktop connection, you will need it. I strongly recommend installation.

$ sudo VBoxManage extpack install Oracle_VM_VirtualBox_Extension_Pack-4.3.12-93733.vbox-extpack
$ VBoxManage extpack cleanup    # In case of failure, run this command and try again

You now have an optimal environment for running VirtualBox guests headless with complete command-line access and control along with remote desktop connections. However, there is no guests. As such, let’s install one!

You will need to instruct VirtualBox to register a guest. In this instance, the name of the guest will be SL6-1 of type RedHat 64 bit.

$ VBoxManage createvm --name SL6-1 --ostype RedHat_64 --register

As long as the previous command did not return an error, the registration should be successful. However, this is only the beginning. There is information required such as the amount of memory, storage and network configuration. The ‘modifyvm’ subcommand allows one to specify a plethora of information. In this example, I am alloting 512 MB RAM, enabling ACPI, configuring boot from DVD, bridging virtual Ethernet adapter one to the host’s ‘en0’ and enabling the remote desktop port on 5000.

$ VBoxManage modifyvm SL6-1 --memory 512 --acpi on --boot1 dvd --nic1 bridged --bridgeadapter1 en0 --vrdeport 5000

Now, create the virtual hard drive file of size ten gibibytes. Please note, this file will normally be placed in ~/VirtualBox VMs/SL6-1/ directory since the SL6-1 is the name of this guest. You can either create this file in that directory, or move it there after you have created it.

$ VBoxManage createhd --filename SL6-1_Main.vdi --size 10000

The storage disks need a controller. In this case, I will use SATA for the storage disks and IDE for the DVD. The DVD controller is necessary for installation of the platform and guest additions.

$ VBoxManage storagectl SL6-1 --name "SATA Controller" --add sata --controller IntelAhci
$ VBoxManage storagectl SL6-1 --name "IDE Controller" --add ide --controller PIIX4

Since the controller now exists, you can attach the storage medium to it. The SATA disk will be the file we have just created. The IDE disk, or DVD, will be the downloaded ISO file for installation of Scientific Linux. We are only using one disk on each type, so use port and device zero.

$ VBoxManage storageattach SL6-1 --storagectl "SATA Controller" --port 0 --device 0 --type hdd --medium "SL6-1_Main.vdi"
$ VBoxManage storageattach SL6-1 --storagectl "IDE Controller" --port 0 --device 0 --type dvddrive --medium ~/Downloads/SL-65-x86_64-2013-12-05-boot.iso

The guest is completely configured with RAM, network interface, storage devices and display console. It is configured to boot from DVD which has been “inserted”. Now, it is time to turn it on

$ VBoxManage startvm SL6-1 --type headless

Assuming no typo’s, or non-default configuration, you should now have a VM guest running and prompting on installation. If you recall, we ssh’d in with port 5000 forwarded. This is because we configured RDP on port 5000. You can open your favorite remote desktop connection client and connect to localhost:5000 now. The point of this article does not cover installation of Scientific Linux. However, you will need to provide the following URL to the installation medium if you do not have it. The download at the beginning does not.

http://ftp1.scientificlinux.org/linux/scientific/6.5/x86_64/os/

Once the installation is complete, it will ask you to reboot the machine. Instead of selecting that, you can simply power off the guest. You will need to remove the DVD by providing ‘none’ as the medium.

$ VBoxManage controlvm SL6-1 poweroff
$ VBoxManage storageattach SL6-1 --storagectl "IDE Controller" --port 0 --device 0 --medium none

Once this is complete, you have a fully functioning guest. However, there are VirtualBox guest additions that provide features you may desire in the future. The VirtualBox installation comes with a virtual DVD iso. You need to find the path and then mount it to the same device/controller as the Scientific Linux medium.

$ find /Applications/VirtualBox.app/ -iname '*.iso'
/Applications/VirtualBox.app//Contents/MacOS/VBoxGuestAdditions.iso
$ VBoxManage storageattach SL6-1 --storagectl "IDE Controller" --port 0 --device 0 --type dvddrive --medium /Applications/VirtualBox.app/Contents/MacOS/VBoxGuestAdditions.iso

Now, power on the guest with the same command used to start the installation. The guest will boot to login. You may now login with the credentials you provided. In order to install the guest additions, there are certain package necessary. The following command will install these.

# yum install gcc kernel-devel perl

The DVD was “inserted” and now needs mounted manually. Create the directory to mount to, mount the dvd and run the additions installer.

# mkdir /media/dvd
# mount /dev/dev /media/dvd
# /media/dvd/VBoxLinuxAdditions.run

Again, assuming all is well and no errors, you may reboot the guest and enjoy a fully-functioning VirtualBox system with all extensions and guest additions.

 

 

 

 

 

 

 

 

 

 

 

JavaScript Oh My!

Where has the time gone? It’s been a bit busy since I posted something useful, and you will not find this of much use at all. However, I’ve been quite deep in JavaScript lately. While I find the language full of some nasty gotchas, there is a portion of it that is usable without rolling your eyes. Well, not that much.

I’ve been using Rhino for an engine we use at work for the last month or so. So, it’s not my fault, ok! It is the default JavaScript engine interpreter for Java 6 & 7. I installed Rhino with Macports, and it works quite well as an interactive and exploratory programming environment. However, I use it for the sole purpose that it is our embedded engine and has been replaced by Nashorn in Java 8. I have not looked at it or other implementations such as V8 or node.js; because, well, it’s JavaScript.

There are good videos that are somewhat aged, but quite useful. You can start by finding them here. There are also two books that are valuable — JavaScript the Good Parts and JavaScript the Definitive Guide 6th Edition. The videos and those two books will be enough to make you as good a pro as any. Well, if you consider JavaScript pro worthy. I will save you the details, but needless to say, corporate politics have left the language in the undesired state it is.

Anyhow, after you have installed Rhino successfully, you can invoke it as such:

java -cp /opt/local/share/java/rhino/js.jar org.mozilla.javascript.tools.shell.Main

It is nice enough that you can quickly get comfortable behind your editor and develop solutions to real-world problems such as embedded file i/o and stand-alone command-line utilities. For example, if you would like to write a useful copy utility that can be solely command-line driven with provided file names or read from standard input (stdin) with error detection and graceful exit, you can do so in roughly 50 lines of code:

#!/usr/bin/env java -cp /opt/local/share/java/rhino/js.jar org.mozilla.javascript.tools.shell.Main -strict

importPackage(java.io);
importPackage(java.lang);

var Writer = function(inp) {
    return new BufferedWriter(new OutputStreamWriter(new FileOutputStream(inp),
                                                     'utf-8'));
}

var Reader = function(out) {
    if (out === '-') {
        out = new InputStreamReader(System['in']);
    } else {
        out = new FileReader(out);
    }
    return new BufferedReader(out);
}

try {
    var iname = arguments[0];
    var oname = arguments[1];
    if (!iname || !oname) { 
        throw new Error("copy.js <source> <target>");
    }
} catch (e) {
    print(e);
    quit();
}

try {
    var reader = new Reader(iname);
    var writer = new Writer(oname);
    var s = true;
} catch (e) {
    print(e);
    quit();
}

try {
    while (s) {
        s = reader.readLine();
        if (s) {
            writer.write(s);
            writer.newLine();
        }
    }
} catch (e) {
    print(e);
} finally {
    reader.close();
    writer.close();
}

Please note, while this code does not exhaustively cover all worst-case scenarios, it will work in a known good environment.

$ ./copy.js file.out nonexistent_file.in
JavaException: java.io.FileNotFoundException: nonexistent_file.in (No such file or directory)

As you can see, an interpreted environment that provides exploratory programming is a very attractive environment and should be used accordingly. Since programming is a “means to an end”, the environments increase productivity while allowing one to develop on the fly. If you are so inclined, I recommend the videos, books and quickly install one of these engines to see if it is something that would work for you.

Building Python3 on Old Environments

If you are using an environment with < Python2.5, you will need to build Python2.5 or later by simply following these commands and editing Makefile to point to the newer Python2 in order to build Python3. This is needed for RHEL4.

# yum groupinstall "Development Tools"
# yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel
# ./configure CXX=`which g++` --prefix=/opt/python3
# make

If you are using a 64 bit build environment and need to create a 32 bit python to run on 32 bit arch, ensure elfutils, glibc-devel.i386, compat-gcc-32 and compat-gcc-32-c++ are installed and create config.site with:

ac_cv_file__dev_ptmx=no
ac_cv_file__dev_ptc=no

and then:

# export READELF=`which readelf`
# ./configure CC="gcc -m32" CXX="g++ -m32" CONFIG_SITE=config.site --disable-ipv6 --build=x86_64-pc-linux-gnu --host=i386-pc-linux-gnu --prefix=/opt/pym32

PyQt5 on Mac OSX

I’ve been gradually trying to get in to Qt after playing with it, well, let’s just say a few years ago. Anyway, I am a Pythonista as a few people know. So, it’s really neat to see there are Python bindings for Qt. Tie that together with Qt’s aggressive cross-platform support, and it makes sense to have a look. There are plenty of videos on Youtube showing the use of Qt Designer along with `pyuic`. And, the ease of creating flexible and usable GUIs for one’s code is compelling. 

I started with PyQt4, since the latest book uses it as a reference, and all it’s sample code is in PyQt4. I found it in Macports with a `port search qt`. After installing PyQt4 on Python 3.4, reading a bit and watching a few vids, I was ready to dive in more, but wanted to focus on Qt5 since it supports Android and iOS. Why not learn the latest version?

However, Qt5 is not in Macports. So, after surfing around, it seemed all I needed was Qt5 installed, SIP for the version of Python I was using and build PyQt5. Since I already had PyQt4 installed, SIP was already there. I port uninstalled PyQt4 and Qt4, downloaded Qt5 and the PyQt5 source.

I simply installed Qt5 from the dmg into /Applications/Qt. Then, untar PyQt5. Enter the directory, and execute `python3.4 configure.py –qmake /Applications/Qt/5.2.1/clang_64/bin/qmake –sip /opt/local/bin/sip-3.4`, `make` and `make install`. 

Finally, I changed into the directory of the PyQt4 source code from the book, changed one to PyQt5 and voila! Qt Designer, Qt Creator and `pyuic5` are installed and work. I tested some code from a user on #pyqt from Freenode, and it worked. So, I am all set. Please note, you will need Xcode installed to perform this process.

Macports and Universal Builds

I’ve decided to give GUI utilities a go since everything is going mobile, and — whether we want to admit it or not — it is going global with hand gestures and a plethora of devices. While I’ve always used GUI utilities, I find myself far more productive with command-line utilities and pipes. The library I’ve decided to use is QT with Python. So, I got the book and `port install py33-pyqt4-devel`.

I was then ready to dive in and follow along with the source examples. I got to the first example in chapter four and while the program would compile and run, there was no output and control-c would not kill it. Something was wrong. I then discovered that QT4 needs a 32-bit build; however, the Python3.3 installation was not compiled with 32 bit support. 

After following some instructions, I uninstalled Python3.3 and QT4 from Macports and reinstalled them. I now have universal builds that support 32 and 64 bit. After about fifteen minutes with Google and these helpful posts, I am now up and running PyQT4 on my OSX 10.9 with a universal build. 

 

I didn’t even think about this until today when I saw Switzerland ranked number two with only six medals ahead of several other countries with ten or more in the official Sochi medal standings. It’s clear that the Sochi winter Olympics are using total gold medals followed by silver and then bronze to sort the rankings. I then came across Yahoo with medal count based on total followed by gold, silver and bronze in the same fashion. This makes a bit more sense. But, both systems are too rigid and do not provide a total realistic tally.

If you have a look here, you will see that in the past there have been these two variants of Olympic medal ranking, but, it seems with all the mathematicians at our disposal, a weighted system would be far better. Since countries like the US normally dominate, no one says anything. But, I simply weighed gold as three points, silver as two and bronze as one, did some multiplication and addition, and voila!

The numbers I got were clearly similar, but there was some difference. Please see below the official Sochi standings as of 16 Feb 2014:

Rank	Country	Gold	Silver	Bronze	Total
1	Germany	7	3	2	12
2	Netherlands	5	5	7	17
3	Norway	5	3	6	14
4	Switzerland	5	1	1	7
5	Russian Fed.	4	7	5	16
6	Canada	4	6	4	14
7	United States	4	4	8	16
8	Poland	4	0	0	4
9	China	3	2	0	5
10	Belarus	3	0	1	4

Yahoo’s Version:

	Country				Total
1	Netherlands	5	5	7	17
2	Russia	4	7	5	16
3	United States	4	4	8	16
4	Norway	5	3	6	14
5	Canada	4	6	4	14
6	Germany	7	3	2	12
7	Sweden	2	5	2	9
8	Switzerland	5	1	1	7
9	Austria	2	4	1	7
10	France	2	0	4	6

And, my calculations:

1. Netherlands
2. Russian Fed.
3. Germany
4. Canada
5. United States
6. Norway
7. Switzerland
8. China
9. Poland
10. Belarus

My calculations are based on the following code:

#!/usr/bin/env python3
"""Fetch the current Olympic medal count alloted to each country 
by gold, silver and bronze. Then, apply weight calculation and 
sort by total. The entry with the greatest total should return as
first place followed by sorting total. In event of a tie in
weight, sort by total number of medals.
"""

from sys import exit
from urllib import request
import json

# Thanks Will
# http://www.clearlytech.com/2014/02/08/building-simple-olympic-medals-api/
__url__ = "http://olympics.clearlytech.com/api/v1/medals/"

req = request.Request(__url__)
cnt = json.loads(request.urlopen(req).read(18192).decode('utf-8'))
cdict = {cntry['country_name']: (3*int(cntry['gold_count']) +
                                 2*int(cntry['silver_count']) +
                                   int(cntry['bronze_count']),
                                 int(cntry['medal_count'])) 
         for cntry in cnt}

sc = sorted(cdict.keys(), key=lambda x: cdict[x], reverse=True)

for i in range(10):
    print('{0: >2} {1: >20} {2}'.format(i+1, sc[i], cdict[sc[i]]))

The above code simply sorts the entries based on each gold medal having a value of three, silver two and bronze one. In this way, total medals along with the value of the medal all count toward standing versus using total medal count followed by or only gold, silver and bronze count in order.

Since this is all a media creation, it will come down to how nationalist the specific media organisation is. The Olympic committee has no preference nor care it seems. That is fine. But, at the end of the day, it is an interesting abstract math problem.