Upgrading PostgreSQL 9.0 to 9.1 with pg_upgrade
Recently, I updated all of the packages I have installed via Homebrew and ran into some issues with the PostgreSQL package. I was getting this error:
The data directory was initialized by PostgreSQL version 9.0, which is not compatible with this version 9.1.4
A quick search took me to this page, however the docs left a lot to be desired to say the least. Here's a step-by-step to how I eventually fixed it using pg_upgrade.
First, change directories to your Postgres data directory.
$ cd /usr/local/var
Next, create a new directory for the new data directory.
$ mkdir postgres9
Use initdb to initialize the new Postgres cluster in the new data directory you created above.
$ initdb /usr/local/var/postgres9
Run pg_upgrade with the following arguments:
$ pg_upgrade -d /usr/local/var/postgres/ -D /usr/local/var/postgres9 -b /usr/local/Cellar/postgresql/9.0.4/bin/ -B /usr/local/Cellar/postgresql/9.1.4/bin/ -v
If all goes without error, you can switch the data directories so Postgres will point to the right source.
$ mv postgres postgres9.0.1
$ mv postgres9 postgres
You can delete the script left behind by pg_upgrade:
$ rm delete_old_cluster.sh
I actually had my old Postgres instance running, so I had to stop it before restarting the new one:
$ pg_ctl -D /usr/local/var/postgres9.0.1 stop -m fast
Then I restarted the new Postgres instance:
$ launchctl unload -w homebrew.mxcl.postgresql.plist
$ launchctl load -w homebrew.mxcl.postgresql.plist
And you're done - this should fix the PostgreSQL incompatible data directory issue.
Installing OpenTera on a TeraStation
I have a Buffalo TeraStation that I bought in 2006 to store all my big media files. Through the years it has served me well, although with Mac OS X Lion, connectivity to the TeraStation stopped working altogether - mainly due to Lion's changed AFP connectivity. Usually with hardware of this age, it would be simply replaced; however, I've always found Buffalo's hardware to be of great quality which is a testament to how long it's lasted. Sadly, installing OpenTera has been on my TODO list for roughly a year now, so this fix is overdue. Buffalo officially responded that there will be a firmware fix for this in the future, but that was over a year ago. This is a quick walkthrough of how I got read/write access back on the TeraStation through Mac OS X Lion. Note: I did the download and firmware updates from my Windows PC, telnet from Mac OS X Lion.
1) Find out the firmware version of your TeraStation. At the time of writing, I have v2.14. Download the corresponding firmware version from this site and install it. This is essentially the same firmware but with telnet enabled. You'll need telnet enabled to log into the TeraStation to install OpenTera.
2) Download OpenTera Update v5 and Samba 3 from this site. Unzip and move this on the TeraStation where you can later access it. I moved this into my "Share" directory.
3) Telnet into the TeraStation. I believe the default user for the firmware is "myroot", which you should use to telnet in.
$ telnet 192.168.0.108
Trying 192.168.0.108...
Connected to 192.168.0.108.
Escape character is '^]'.
Password:
Login incorrect
TERASTATION login: myroot
root@TERASTATION:~#
4) Change directories into the OpenTera directory, the one you just placed on the TeraStation, and run the Update_OpenTera.sh script.
root@TERASTATION:/# cd mnt/array1/share/OpenTera_v5/
root@TERASTATION:/mnt/array1/share/OpenTera_v5# ./Update_OpenTera.sh
This script is an interactive install, so it will prompt you to say "y" a few times. Go ahead and do so. If all is successful, you should see:
[OpenTera] ============ OpenTera Install Completed ===============
5) Next, change directories into the Samba directory and run the Update_Samba.sh script.
root@TERASTATION:/mnt/array1/share/OpenTera_v5# cd /mnt/array1/share/samba-3.0.37_ppc/
root@TERASTATION:/mnt/array1/share/samba-3.0.37_ppc# ./Update_Samba.sh
When all is completed, you should see:
*********************************************************************
Completed samba-3.0.37_ppc.tgz install
*********************************************************************
And that's it! I was able to access the TeraStation from Mac OS X Lion. Hopefully this will be helpful to someone else out there, as I know I'm not the only one with this problem. If I missed any steps, let me know - good luck!
Applications I Use
About two years ago, I bought my dad a MacBook Pro as a gift and with him being a long-time Windows user, the transition wasn't seemless to say the least. I wanted to write an article on the applications I use to give him an idea of what's out there. So here are some common and not-so-common applications I use on a daily basis:
General
Alfred is a handy utility app that has a lot more functionality than what I use it for. It lets you search and open applications quickly with a few key strokes by simply typing the first few letters of the application name.
A password manager app. I don't know any of my passwords anymore. You keep one password that unlocks the "vault" and it gives you access to all of your passwords. Mine are randomly generated. Save your encrypted passwords file to Dropbox so you can have it on other devices (iPad, iPhone, or any other machine that can run 1Password).
For effortless syncing between machines and helpful for sharing files, pictures, etc.
Quick entry of appointments in your calendar. Uses natural language and parses it to extract the date, time, and title of meeting. You can type in: "Lunch with Lydia tomorrow afternoon at 1pm" and it will create an calendar event for you correctly.
An easier way to share screenshots. You can set it up auto-upload every time you take a screenshot and it automagically copies the sharable URL to your clipboard.
For when working late at night, you can turn down the brightness of your screen. It has presets, so once you activate it, it will dim to your custom setting without having to fiddle with the System Preferences each time you want to dim or turn back on your brightness.
Keeps your screen from dimming or shutting off. Useful when I'm unplugged and on the go - if I don't want the screen to dim after a few minutes of inactivity, I simply turn this on. Helpful when others are looking on at your laptop and you're discussing something. You won't need to keep touching the touchpad every minute or two so it won't dim. Sure, you can change this in your System Preferences, but the idea is to NOT have to keep modifying the presets so it's not intrusive.
A simple little app that sits in your top menu bar and tells you the internal temperature. I like to keep an eye on this when I work - it tells me if there's an application that's eating up a lot of CPU, because the fans would turn on and the temperatures would rise. Also lets you control the fan speed.
I use this app to make web apps seem like a native application. For example, instead of visiting my company's Basecamp URL, I can make the web application into a (fake) standalone application. Now, I can just launch "basecamp" in Alfred and it will open up a dedicated browser for Basecamp.
I use this for quick organization of my applications and windows. You can have presets for sizes that you like to repeat, such as half-screen, full-screen, etc. Useful when working with several applications, such as a browser, terminal, and emacs, and keeping a visual on them all at once.
Coding
Emacs
I've been using Emacs for a few years now and the muscle memory is definitely ingrained where I feel more productive in it than almost anything else.
Another editor that I used heavily before diving (back) into emacs. Out of habit, it's more of a crutch than anything else. I still find myself using it for viewing project tree structures and quick searching throughout the project.
An alternative to Terminal, much more feature-rich.
Made its way into my daily development arsenal as of recently. Has become extremely useful in seeing all the responses the applications I'm coding is making.
Chrome and developer tools, Firefox, Safari
My browsers of choice, in that order.
I use a fork of this and prefer it over some of the newer git GUIs out there.
Another good one, only reason why it's on here is because it's free (for now), so go ahead and grab a copy before they start charging! Great product.
I begrudgingly use this because I have one remaining project on SVN, which will soon be no more. It makes working with SVN somewhat bearable.
Great GUI for MySQL databases remote and local.
GUI for PostgreSQL databases remote and local. I use this in conjunction with pgphpmyadmin.
(S)FTP client.
Frontend/Design
Photoshop CS5
Need I say more?
For making sure everything is pixel perfect and colors match exactly in the websites I code. Has been an indespensible tool throughout the years.
Great alternative to Photoshop. I'm not a designer, but sometimes when Photoshop chokes on opening a specific file, Pixelmator always seems to come through in the end.
Great tool for quick wireframing and rapid prototyping.
Use this to additionally compress your images and get them ready for website use to ensure speed in loading the media assets.
Business
This is a great app that I use daily to track my time and send invoices. It also lets me see what payments are outstanding from each client. It has a handy timer in the menu bar that lets you track time at any given point with whatever you're doing.
Great tool that I started using last year to create estimates for clients. I mostly love any tool that Omni Group makes.
Notational Velocity with Simplenote integration
Great for quick, free form note taking. Notational Velocity lets you simply type in search terms in the top bar and it will search your notes for it.
Simple, fantastic tool for writing. Has Markdown support!
A neat little calculator that uses some natural language to give you the answer.
ScanSnap Manager Amazon link, since Fujitisu is more likely to change their URL than Amazon.
I scan all my physical documents with the ScanSnap S1500M and use OCR scanning to make the entire document searchable.
Social/Fun
I use Adium for instant messaging as it allows you to sign in with multiple types of accounts at once. I know iChat is getting better, but I've been a long time user of Adium and feels like home to me.
Great dedicated application for Campfire, so you don't need to use the browser to access it. Feels more like native IM now.
Great GUI for IRC. I used to just connect in my Emacs window, however it proved to be too distracting when working.
My Twitter client of choice.
Great Google Reader application, now with Readability support.
Great to making those always funny memes...and sending work stuff to people.
That's it!
Like many people in my profession, I consider myself to be a bit of a software junkie...and with good reason! Just like with programming languages - there's always the right tool for the right job. This was a fun little exercise to go over the tools I use on a daily basis and I hope this is helpful for some out there. In one of next articles, I plan to write more about my development workflow and the tools I use in greater detail.
2011 In Review
Since it seems that everyone and their mother is doing a "year in review" post, I find it fitting that I be no different and write one as well - except it's three weeks later than everyone else. 2011 was an exciting year for me for a variety of reasons, as I went through some big work and life changes, met some great new friends, and reunited with a few long lost ones.
Work
In February of 2011, I left my job at exhibit-E where I was a web developer making web sites for the art world. Over the past few years, I've always done freelance work on the side whether it be small contract jobs or building full social networking websites as a solo developer. My decision to leave exhibit-E seemed to make sense now - I wanted to see if I had what it took to survive as a full-time freelancer while working under my own company name before I got too old and perhaps couldn't take on so much risk. I wasn't too concerned with finding work, but more so about being able to GROW my own company; the latter obviously being the hardest part. Within a week or two, I managed to land myself a great gig as the lead developer for the RachaelRayShow.com...and now it was off to the races!
Conferences
In March, I went to PyCon 2011 in Atlanta, GA. This was my first programming conference and I met some fantastically smart people there - some of which I'm fortunate to actually work with right now on some of 3atmosphere's current projects, and some that I still keep in contact with via email. The talks were great and the "networking" at night (aka exploring ATL's nightlife) was the best part of the trip. I'm really looking forward to returning to PyCon again this year.
Nick holding up his new laptop that we bought at a mall in Atlanta during PyCon...during iPad 2 (insanity) launch day!
More Work!
Over the next few months into the early summer, the project load at 3atmospheres really started to pick up. I take pride in the fact that I never made one cold call or solicited ads for work - all the requests we got in were from word-of-mouth and I couldn't be more proud of our humble beginning. It picked up so much that the once lonely one-man show (me!) turned into (at the peak) six people working on several different projects. I found myself quickly transitioning from coding to project management, which I would say still is evenly split for me today. Out of necessity for a meeting spot, we bought some seats at General Assembly where we currently work.
On a personal note, I got engaged at the end of the summer to my lovely girlfriend of over four years, Patricia, which easily made 2011 the best year of my life thus far.
More Conferences!
In September, I was fortunate enough to get to go to three conferences:
- DjangoCon (Portland)
- PyGotham (NYC)
- Strata Conference (NYC)
And for the rest of the fall season into winter, it was just work, work, and more work - literally every free hour I had was devoted to meeting the deadlines that were set at the end of the year.
Let's Wrap It Up...
2011 was a great year and I've never worked harder, been more tired, and, at the same time, had as much fun as I've ever had in my professional career. I am so lucky to have gotten to meet and work with such smart people, clients and coworkers alike, and I'm extremely proud at how far we've come today. Admittedly, it wasn't always rainbows and butterflies with work, but for all the negative moments I've had this year, I've used it as a learning experience and I wouldn't trade it for a second. I get to program in languages and frameworks of my choosing (Python and Django!) and get to work with people that are smarter than me. Here's to a great 2011 and an even better 2012!
Virtualenv and pip Basics
When doing any kind of Python development, one tool I find indispensable is virtualenv. Virtualenv, along with virtualenvwrapper and pip, make for a great way to completely isolate your development environment.
When I first started out developing Django sites, I used to use easy_install to install all packages I needed to the system-wide site-packages directory. Even as a newbie to Django, I knew this wasn't good practice, but it ensured that commonly used libraries such as MySQL-python was available without any extra configuration with new projects. Regardless, completely isolating your environment with virtualenv ensures that a) you don't install conflicting packages and b) any bugs introduced in your project can be traced back directly to the packages you installed. Also, a huge benefit is that it makes installing multiple versions of Python super easy without having to create any symlinks.
Getting started with Virtualenv and pip
The first thing you will need to do is install pip. If you have setuptools installed, which you most likely will with most modern platforms, you can install pip through easy_install:
easy_install pip
Next, you'll need to install virtualenv with pip:
pip install virtualenv
Finally, I would highly recommend installing virtualenvwrapper as it makes it much easier to create and start virtual environments:
pip install virtualenvwrapper
As part of the install instructions for virtualenvwrapper, you need to add this to your .bash_profile
# virtualenv
export WORKON_HOME=$HOME/.virtualenvs
source /Library/Frameworks/Python.framework/Versions/2.7/bin/virtualenvwrapper.sh
*Please note that this path may differ depending on what version of Python you have. Also, I like to keep all my virtualenvs in a directory called
.virtualenvsin my home directory, but this may differ for you if you choose to keep your virtual environments in a different directory.
Make sure you source your new .bash_profile
source ~/.bash_profile
...and that's it! Now you're all set to start using virtual environments!
Creating a Virtual Environment
A few handy aliases I have in my .bash_profile are found on Doug Hellmann's blog and listed below:
# virtualenv aliases
# http://blog.doughellmann.com/2010/01/virtualenvwrapper-tips-and-tricks.html
alias v='workon'
alias v.deactivate='deactivate'
alias v.mk='mkvirtualenv --no-site-packages'
alias v.mk_withsitepackages='mkvirtualenv'
alias v.rm='rmvirtualenv'
alias v.switch='workon'
alias v.add2virtualenv='add2virtualenv'
alias v.cdsitepackages='cdsitepackages'
alias v.cd='cdvirtualenv'
alias v.lssitepackages='lssitepackages'
This saves some keystrokes, especially since I always create new virtual environments with the --no-site-packages switch to ensure a completely clean environment.
To create and start a new virtual environment with --no-site-packages, enter:
$ v.mk myvirtualenv
New python executable in myvirtualenv/bin/python
Installing setuptools............done.
(myvirtualenv) $
This creates and virtual environment and makes it active. To deactivate it, you can simply type:
(myvirtualenv) $ deactivate
$
So let's go ahead and start our virtual environment once again and install some packages to it.
$ v myvirtualenv
(myvirtualenv) $
We're going to install Python package Yolk as it is a useful command line utility that lists the packages installed for the environment.
$ v myvirtualenv
(myvirtualenv) $
(myvirtualenv) $ pip install yolk
Downloading/unpacking yolk
Downloading yolk-0.4.1.tar.gz (80Kb): 80Kb downloaded
Running setup.py egg_info for package yolk
Requirement already satisfied (use --upgrade to upgrade): setuptools in /Users/jonathan/.virtualenvs/myvirtualenv/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg (from yolk)
Installing collected packages: yolk
Running setup.py install for yolk
Installing yolk script to /Users/jonathan/.virtualenvs/myvirtualenv/bin
Successfully installed yolk
Cleaning up...
Now you can use yolk -l to list the packages installed for this virtual environment:
(myvirtualenv) $ yolk -l
Python - 2.7.1 - active development (/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload)
pip - 0.8.1 - active
setuptools - 0.6c11 - active
wsgiref - 0.1.2 - active development (/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7)
yolk - 0.4.1 - active
Here is a brief one-line example showing how to create a virtualenv and install Django, MySQL Python, South, Python Imaging Library (PIL), and ImageKit using pip:
$ v.mk newdjangoenv
(newdjangoenv) $ pip install django MySQL-python south pil django-imagekit
When you have your requirements installed, it's always good to take a snapshot of the requirements and the current versions. You can do this by typing freeze and specifying an output file:
(newdjangoenv) $ pip freeze > requirements.txt
And finally, you can use the requirements.txt file so that your environment is completely and easily replicable:
$ pip install -r requirements.txt
And there you have it -- you can now create and test your Python applications in completely isolated environments!
For more on pip and virtualenv, check out this great post by Salty Crane which got me started on all this.