I’m not going to lie and tell you that I’m great at task management. Frankly, I suck and I blame it on technology. If only there were a task system that could light a fire under my butt. We all know there isn’t, but the system that I’ve found to work best for me is Todoist. Todoist is really light and fast. Also, it has a simple API so it’s very extensible and already has a lot of tools surrounding it (quicksilver, launch, dashboard widget, mobile, gmail integration, etc).
The Problems
The problem that I have with task management is two fold. First, I need reminders. My programming brain is sharply trained to scoot information in and out, only keeping what is immediately necessary. It’s great to have a to do list but if it doesn’t remind me, it barely works. The good news is that todoist has a premium service that is only $3/month (yes only $3) that allows for reminders in any form you like (twitter, sms, email, jabber, etc). First problem solved.
The second problem that I have is that sometimes a reminder is not enough. I personally need to see my upcoming projects, tasks and calendar items in front of me anytime I can. It hit me the other day that I look at my desktop a lot. This got me wondering if I could use the todoist API to put my upcoming, date sensitive tasks right there, on the desktop, in front of me whenever I’m on my laptop.
The Solution
I did some googleing and found Geektool, which I had installed/uninstalled before but this time seemed to have a purpose. The end result is in the screen cap below.
To do this, I created a quick and dirty ruby script to hit the todoist api and put out a simple list of all time sensitive tasks. 1) I put the following script at ~/bin/todoist.rb.
Like I said, it’s quick and dirty. 2) Now setup geektool to run the script at a given interval (I chose 2 minutes).
3) I created a file in my home directory to store my todoist token and the number of days I want to show. The main reason I separate the token from the ruby script is in case I feel like using it somewhere else on my system with another script. If my token ever changes, I can update it in one place and all the scripts will still use correct one. Anyway…I put the file at ~/.todoist and it looked something like this:
token: foobarbaztoken
days: 4
Now every two minutes the script runs and updates based on how I am updating my tasks at todoist.com. My tasks are continually in front of me, which helps keep them in my mind so that I actually get stuff done.
Other Uses
Some other uses I could see valuable for this would be calendaring. Google Calendar has an API so you could hit that and show your upcoming events right by your tasks. Maybe there is some site that posts a lot of updates and it’s hard to keep up with by feeds? Just use feedtools and a tinye ruby script to show the latest posts for your perusal when you get a few seconds. Just some thoughts.
Anyone else do something along these lines? Have some code to share or thoughts on how to do it better? Let me know.
I have a dream. A dream wherein I can update a buddy’s iChat status remotely. It’s not a change the world dream, but it would be fricken sweet, right?. I mean imagine your fiend coding up a storm on his computer and all of a sudden noticing that his iChat status says ‘I have a man-crush on DHH.’ I know you are feelin’ me.
Anyway, I guess I should drop some history. The dream started a year back with Steve, Apache and PHP, but quickly faded as we lost interest. A few days back, the dream was rekindled, this time with Chas, Twitter and Ruby and it has now ended in success.
The Goal
The goal is very unselfish. I don’t want this power all to myself. I want to share it with all my friends, that is with the exception of who we shall now refer to as “the victim.” We wanted to be able to allow ourselves and a few friends the ability to update one of our other friend’s iChat status (“the victim”) from anywhere. What is a tool we all use, all the live long day? Twitter. The solution is actually pretty simple. It’s only 3 steps. Create a private twitter account that you can all direct message. Install a ruby script on “the victim’s” computer. Setup scheduled task to run ruby script every few minutes. I’ll now detail these steps with code samples but remember, with great power comes great responsibility (and laughter).
Step 1 – Create Private Twitter Account
Create a twitter account and make it private. All the friends that would like to have power over “the victim” should then request to follow the new account and you should approve them. This means that any friend who follows that account, and that you approve, can now direct message that account. The general public cannot. Now the group of controlling friends needs only to twitter ‘d newtwitteraccount some new ichat status’ and the next status for “the victim” is ready to be set.
Step 2 – Install Ruby Script
Ok, so I could care less about windows. Knowing that, I’m only dealing with Mac’s. What does every mac come with? Ruby. What language do I make love to every night? Ruby. Now beings that I wrote the twitter gem, you might assume that I used that in the script. You’d be wrong. You know what they say about assuming, right? It makes an ass out of ‘u’ and ‘ming’ or something like that.
More than likely you will only have a few bathroom break minutes to setup everything on your friends computer. This means we need the least amount of dependencies to slow us down. Below is the quick and dirty script I wrote to get the latest direct message from a twitter account using only core ruby libraries. Basically it grabs the latest direct message and if it is new, it executes some applescript to update the iChat status.
#!/usr/bin/env ruby
%w[open-uri rexml/document fileutils].each { |x| require x }
Debug = true
Username = 'newaccount'
Password = 'password'
dms = []
url = "http://twitter.com/direct_messages.xml"
lines = ['tell application "System Events"', 'if exists process "iChat" then', 'tell application "iChat"', 'set the status message to "#{latest}"', 'end tell', 'end if', 'end tell']
puts 'Doing some magic' if Debug
data = open(url, :http_basic_authentication => [Username, Password]).read
doc = REXML::Document.new(data)
doc.elements.each('direct-messages/direct_message/text') { |el| dms << el.text }
latest = dms.first
if latest
puts 'we have a latest message' if Debug
open('latest.txt', 'w+') do |f|
last = f.gets
if latest != last
puts 'we are changing ichat status now' if Debug
f.puts latest
cmd = "-e '#{lines.collect { |l| l.gsub('#{latest}', latest) }.join("' -e '")}'"
system "osascript #{cmd}"
end
end
end
This script should be located at ~/Library/TwitteriChatHack/twitter-dm.rb. Be sure to update Username and Password to the private account you just created in step 1.
Step 3 – Schedule Task to Run Ruby Script
We have our script, but it does nothing without being told to. That said, we have to schedule it to run on an interval. Mac OSX has a great scheduling tool called launchd. It appears to be a black art, but is actually really easy to use. Justin just wrote up a great article on how to automate rick-rolling your friends with launchd. Below is the sample launchd plist item that instead of rick-rolling, will run our ruby script from step 2.
This script should be located at ~/Library/LaunchAgents/com.addictedtonew.twitter.ichat.hack.plist. Be sure to change MACOSXUSERNAME with your friends max osx username.
Now the only thing left to do is to actually load the task into launchd. If you don’t do this, your script won’t work until “the victim” logs out and back in. Just run the following command:
That’s it. Once you create the private twitter account, add the ruby script onto “the vicitim’s” computer and setup the launchd task, you can can ‘d twitteraccount i love britney spears’ and within a minute your friend’s iChat status will be updated appropriately. Below is an example on my computer of the direct message and iChat status update.
Notes
The interval at which the status can be udpated at is set with the StartInterval number. Mine defaults to 60 but you might want to set it to 300 (every 5 minutes) so you aren’t continually checking for updates.
The ruby script checks to see if iChat is running and only attempts to udpate status if it is so it won’t pop open iChat on your friends computer every minute.
The ruby script also saves the last direct message that was used so if you direct message and your buddy changes his status back to something else, it will only change it again if you put through a new direct message.
So that is all. What kind of crazy things like this have you all done?
Jeremy, from my local ruby group, had a sweet idea. He was piping crontab commands into mail and thought “why not use twitter?” He hacked around a bit and sent me some code. I tweaked a couple things and now, as of version 0.2.6, the twitter gem can take stdin when posting from the command line. Below are a few examples from the changelog.
$ twitter post 'test without stdin'
# => twitters: "test without stdin"
$ echo 'test with stdin' | twitter post 'and an argv[1]'
# => twitters: "test with stdin and an argv[1]"
$ echo 'test with stdin without any argv[1]' | twitter post
# => twitters: "test with stdin without any argv[1]"
If you are curious about the implementation, see the bottom of the bin/twitter file.
A Recent Cool Use of the Twitter Gem
Also, while on the topic of twitter, some pals of mine are repurposing the twitter gem to do reviews (and they were even so kind as to update the gem and send a pull request). If you are a twitterer, check out microrevie.ws.
There are a few view related things of late I’ve been using to make my code a bit more readable. Trust me, they are nothing new, but for whatever reason I hadn’t jumped on the train.
div_for
This doesn’t do anything special but saves some typing and I tend to think it makes things a bit more aesthetically pleasing in TextMate. Using div_for, the following:
... removed for clarity ...
becomes..
<%- div_for attachment do -%>
... removed for clarity ...
<%- end %>
You can also use it’s companion, content_tag_for in the case where you are creating table rows or something:
<%- content_tag_for :tr attachment do -%>
... removed for clarity ...
<%- end %>
local_assigns
Want to pass in optional parameters to a partial rendering? local_assigns is the key to your problems.
<%- if local_assigns.has_key?(:show_project) -%>
<%= time_entry.project.name %>
<%- end -%>
If you did <%= render :partial => 'foo', :locals => {:time_entry => time_entry, :show_project => true} %>, the project heading would be shown but if you just did <%= render :partial => 'foo', :locals => {:time_entry => time_entry} %> without the :show_project hash key, the heading would not be printed. That might be a bad example, but you get the idea.
content_tag
Obie mentioned a good use for this in the Rails Way, which is in combo with an if statement for some one line goodness. Take the following for example:
<%- if current_user.authorized_to_edit?(user) -%>
<%= link_to 'Edit', edit_user_path(user) %>
<%- end -%>
That works ok, but using content_tag you can easily put it on one line like so:
The two examples put out the same html with the same permissions but the latter is much more succinct.
debug
In PHP, it was var_dump. In ColdFusion is was . Somehow, I overlooked debug in rails for almost a year. Simply call <%= debug @projects %> in your view and you will get a to_yaml representation of the variable wrapped in pre tags.
I’m sure there are tons of others that I’m not super aware of, but these are a few that I’ve found helpful of late. Anybody have other favorites? Maybe concat mixed with a block? Maybe some capture action?
Doesn’t it stink that google analytics doesn’t have an API? Yeah, I know. It seems nearly every post I mention conductor nowadays so I’ll bring it up again. We have about 15 sites live in conductor and darn it if I’m not interested in how they are doing traffic-wise.
The Original Sin
What I thought would be handy, is one simple dashboard where I could see all the pageviews, visits, etc. for each of the live sites in conductor, both separately and collectively. Naturally, I went searching for an analytics API and found nothing. I did a bit more digging and found that there were a few rogue hackers out there who had some form of an API working. It intrigued me so I started to check out the analytics source code. You know all those fancy charts and such? Yep, they’re powered by XML. Woohoo!
Log Me In Scotty
Now, the only thing between my glorious dashboard and me was a dainty browser session. For this, I reverse engineered Google’s login, discovered their cookie monster and created an authentication base class. This made it drop dead easy to make authenticated requests to google for data (I’m also using it in a GReader wrapper I’m working on ).
If you want to make an authenticated request to google it’s as simple as gem install googlebase and the following:
Ding, ding, ding. Dainty browser session down, it was now time to start wrapping up those xml responses into ruby objects that are more fun to play with. It’s pretty much just a lame start, but it is aptly named Statwhore and it’s on a github near you. For now it just has really basic stuff, like pageviews and visits for a given time period. Take the irb session below for example:
Cool stuff, eh? So I didn’t write this post to say go use this in production for a paying client. I just thought it was fun playing around and thought I would put it out there with some explanation on the off chance other people might feel like helping the project along.
The Future
Eventually, I see Statwhore wrapping a crap load of the analytics api’s that I’m sure are out there. Feedburner has an awareness api and a feed management api. Rest assured I’ll be wrapping those up eventually, as I have a dream of only Feedburner urls being spewed from Conductor’s news section. I guess my question is, what are some other good analytics services that also have api’s?
Testing Live Web Services
Also, if you are working on a web service gem and are wondering what is the best way to test it, I don’t have the answer. :) I have tried a few different ways. My recommendation is to mock or stub in some way the response and then let your class go to town. Below are a few different ways I have done it on previous gems.
Live. Don’t do this as your only tests. It sucks. Good idea to have around to see if API changes though. I now horribly regret the tests I put in for the twitter gem. They are horribly fragile.
Ok. Back on track for keeping your tag cloud running. We talked about backups and short/long term problem resolution, so now it’s on to resource reporting. I really see resource reporting more at a macro level. What I’m talking about are things like memory, CPU and filesystem usage, apache processes and volumes and mysql queries and throughput. These stats give you information on how your application is running and where possible rough spots are (spots that need tweaking).
Munin
There are a lot of tools out there to get the job done but the one that I picked is munin so that’s what I’ll talk about today. From the munin website:
Munin the monitoring tool surveys all your computers and remembers what it saw. It presents all the information in graphs through a web interface. Its emphasis is on plug and play capabilities. After completing a installation a high number of monitoring plugins will be playing with no more effort.
Server/Node
That pretty much explains it. What I did is I have one small VPS that I use for version control and as a munin server. I guess before I go to far into that I should explain some quick theory. Munin has a node/server setup. That means you can have all your servers act as nodes which simply make information available to a configured server. The server is the machine that goes out every minute (or how ever often you say), grabs the information from the nodes and collects it into one place. This is nice because I can get a quick daily, weekly, monthly or yearly overview of all the nodes at once on one page. I only keep track of about seven so the page is not ridiculously full of information.
Installation
My gauge for easy installation is, if I can do it, anyone can do it. I’m far from a linux guru, trust me. On Ubuntu, it’s nearly as easy as sudo apt-get install munin and then a few tweaks. On CentOS, once you get the RPMForge and Dag hooked up, installation is a simple sudo yum install munin and, again, a couple of tweaks and you are good to go. I’ll list some links at the end that got me going.
The Payday
So I’ll be the first to admit that Munin could use some UI love but the most important thing is seeing the data in a graph and it accomplishes that well enough. Below are a few samples of real graphs (all daily) pulled from Conductor, our home grown, rails, multi-site website management tool at the University of Notre Dame.
Apache Access: This is helpful just to see the wax and wane of traffic on your site. I find it interesting that most of the sites in Conductor go nearly dormant at night. Leads me to believe we have mostly U.S. traffic (which makes sense).
File System: Now I should have stuff set in to automatically warn me if I get up in file system usage but I’m so low right now, I don’t worry about it too much.
Memory Usage: This is really important for Rails apps. They are known to use a fair share of memory and you need to keep an eye on this for leaks. If you see steady growth, you probably have some issues. One thing to note from the spike above is that it is caused each night by a backup task that I have to shuffle my database, assets and themes off-site. I need to use rsync or something but it is working and who has time, right? Plus if things get out of control, I have monit to get things back in order.
MySQL Queries: Now obviously, more traffic equals more queries so this graph has tended to show the same trends as apache access. I think it’s pretty interesting to see the number of queries per second so I tend to check it out.
Real Life Ways Munin Helped Me
So I show this to give a valid example of how Munin helped me. I avoided premature optimization when building Conductor, and just focused on how the app worked. Once I got Munin tracking all this stuff, I noticed that it seemed like a lot of queries, for the amount of traffic we had.
I did some tailing of log files, tweaking of queries and some htaccess trickery, and you can see the dramatic drop. That drop was not a drop in traffic. In fact, traffic and the number of sites in conductor has only risen, but the queries are still down quite a bit, simply do to a bit of analyzation (which was prompted by the queries per second graph). Also, ignore the blank spots. :) We had some issues where server reboots were required and I forgot to set Munin to start on boot and also forgot to check to see if it was running. Like I said, I’m not a server admin. I’m like you (unless you are a server admin).
Another way that Munin helped out, was by showing me the number of slow queries. This particular slow query, again, happens when the memory jumps as ruby tries to FTP a crap load of backups off-site. How this was helpful, was it prompted me to do more research on indexes and analyzing queries. I was noticing a lot of slow queries as the tally of pages grew in conductor. I did some grepping of the slow query log, based on what Munin was telling me, then some EXPLAIN’in in MySQL, and quickly added two or three indexes which fixed all the issues (more on that in another post).
These are just two examples, and I’m sure I could come up with more but I think I got the point across. Munin is pretty easy to install, has lots of documentation/articles on the net and has come in really handy for me.
So I nearly threw my laptop across the room in a violent rage trying to figure this out. Thankfully I didn’t and rather scoured the googles for a solution. Upon finding it, beautiful music rang from the heavens and a solitary ray of sunshine beamed down on me.
I don’t know if it’s something people rarely try to do or if it’s just hard to find but I thought I would post it here in hopes that the next poor sap who attempts to do so has a bit easier going.
The Tricki-Tikki-Tavi
The trick (if it can be qualified as that) is to use !xml | and some proper spacing like so:
law_news_category:
theme_id: 1
filename: news.category.liquid
data: !xml |
{{ site.name }} // University of Notre Dame
So each time I go to code in a rails app I open up three tabs: one for script/server, one for autotest and one for general commands like script/generate and such. Let’s say I want to open up tinye, a project I’m working on at work. I have to do the following:
cd sites/nd/tinye.git
script/server
# new tab
cd sites/nd/tinye.git
a #that is my shortcut for autotest -rails
# new tab
cd sites/nd/tinye.git
I got a bit tired of this, so based on some scripts I saw around the interwebs, I put together the one below:
#!/bin/sh
if [[ $# == 0 ]]; then
PROJECT_DIR=$PWD
elif [[ $# == 1 && -d "$1" ]]; then
PROJECT_DIR="$@"
else
print "usage: iterm.sh [rails project directory]"
return 1
fi
# how to make new terminal (make new terminal)
osascript <<-eof
tell application "iTerm"
tell the last terminal
activate current session
tell the last session
set name to "server"
write text "cd \"$PROJECT_DIR\""
write text "ss"
end tell
launch session "Default Session"
tell the last session
set name to "autotest"
write text "cd \"$PROJECT_DIR\""
write text "a"
end tell
launch session "Default Session"
tell the last session
set name to "app"
write text "cd \"$PROJECT_DIR\""
write text "open http://localhost:3000/"
end tell
end tell
end tell
eof
I put this in /bin, named iterm.sh, and made sure it was executable (chmod u+x ~/bin/iterm.sh). Then I threw the following in as aliases (/.bash_profile for me):
alias r='iterm.sh $1'
alias ss='script/server'
alias sc='script/console'
alias a='autotest -rails'
This allows me to open the same project I talked about earlier by simply typing:
r sites/nd/tinye.git
Instantly, I get three tabs in iTerm (server, autotest and blank) and an open browser window on localhost:3000. Thought I would share it here to spark other ideas like this. Also, I’m curious if others have similar things (or cooler things). Below is quick and dirty screencast that shows it in action.
I held out as long as I could, but on Wednesday, I grabbed a ham sandwich, nestled into my office chair and hit play in Quicktime for a lunch filled with my three favorite G’s—Geoffrey Grosenbach and Git. An hour later, I was kind of blown away and feeling stupid for not checking it out sooner. I cashed in my GitHub invite and started gitifyin’ some of my svn projects.
Why?
First off, it’s clean. You know all those .svn folders inside every directory of your subversion project? Well, you can have one .gitignore file in the root of your project and you are good to go.
Second, you can use it offline. Ever been away from a net connection? Yeah, me neither. But if it did every happen, wouldn’t offline mode be handy? Git is made for distributed development. This means you can do it all locally but it doesn’t hold you to just that. If you want (and who wouldn’t), you can setup remote git repositories which allows you to push and pull your project and share easily amongst a team or open source community.
Third, straight from the git website, Git has “strong support for non-linear development.” You know that branch feature of svn that you never use because you can’t remember the command? Well, git makes it gosh darn easy. Did I mention I taught my cat Bob to do it? Just be sure to give your animals a treat after they make a branch for you. You can create a new branch…do some crazy experimental thing…and only merge your crazy code into your master branch once you have it all working and fully tested (like a good little programmer).
Fourth, sweet svn integration. Yep, anytime you make a new tool, you have to give your new cult following some way to import all the crap they have in their current tool. I liked git for the first three reasons but I love git for this fourth. cd into your local svn project, use drnic’s gitify and you have a git repo that can also commit to your svn repo (for all of those still kickin’ it old school). Branch it up, make changes and when you are done git committing, just git svn dcommit and it transfers all your git commits to the remote svn repository. But what if someone updates the svn repo while you are gittin’ your git on? Well, Linus thought of that too. You can run git svn rebase and your git repo will update with the remote changes in your svn repo. This dude even has some sake tasks to make it easier to fool your svn friends into thinking that you aren’t using git.
How can I get started?
I would highly recommend breaking the bank for the $9 git screen cast by Geoff. He covers all the stuff you need to git going, including installation. Also, there are a few places where you can host your git projects for free (repo.or.cz, gitorious, and github, which I recommend). Pretty cool stuff. All I’m trying to get at in this article is if you are thinking, “Learn something new? But I already know subversion and it works for me…” stop thinking that, buy the peepcode and enjoy version control again.
How am I using it?
I’m now managing most of my gems on GitHub and loving it. After Jesse Newland whipped together a github-campfire post-commit, I copied him and took the credit, making a github-twitter post-commit soon after. On the topic, that was my first use of Sinatra and I definitely like it for little things like this.
In the interest of kicking my butt into posting more, I’m going to put up a sample deploy file for a merb app we launched about a month ago. Below is an example capistrano 1.4 file that I use for a merb app that runs on active record and sqlite3. It’s a small stat program that grabs analytics in the flash carousel on nd.edu.
set :application, "myapp"
set :user, "deploy"
set :domain, "yourdomain.com"
set :repository, "svn+ssh://#{user}@#{domain}/var/www/apps/repos/#{application}/trunk"
set :deploy_to, "/var/www/apps/#{application}"
role :app, domain
role :web, domain
role :db, domain, :primary => true
desc "Link in the production extras"
task :after_symlink do
# symlink log path
run "ln -nfs #{shared_path}/log #{release_path}/log"
# symlink path to database, this is only needed if you are using sqlite (which is ok for little things and is really easy)
run "ln -nfs #{shared_path}/db/#{application}_production.sqlite3 #{release_path}/db/#{application}_production.sqlite3"
end
desc "Merb it up"
task :restart do
run "cd #{current_path};./script/stop_merb"
# run "cd #{current_path};env EVENT=1 merb -e production -c 1" # for evented mongrel
run "cd #{current_path}; merb -e production -c 1" # plain old mongrel
end
Once you have everything configured, deployment is as easy as cap setup and cap deploy (you’ll probably need to migrate your db as well).
You didn’t think I’d leave you hanging forever, right? I knew everyone was waiting with bated breath so here comes part 3. You have backups. You know how to fix those pesky simple problems that plague us all. Now it’s on to bigger and better things. Next stop…the monit ho train. The whole point of this post is to give you some pointers on how to automate recovering from bad things. It’s nice to know how to fix things manually but that won’t get you anywhere. The best pets are those that mostly take care of themselves. Below are some tips for turning your servers into good pets.
Installation
If you are on a slicehost server, you can use apt-get to install monit (sudo apt-get install monit) and if you are on a Railsmachine slice, you can hookup the dag repository and then use yum to install it (sudo yum install monit).
Either route you take, you will then have to edit the config file (look for /etc/monit/monitrc or /etc/monit.conf). The settings in the config file are pretty well documented. I pretty much just use monit to monitor my mongrel instances. The annoying thing is that you have to completely duplicate the entire config block for each instance of mongrel you want to monitor. I put a simple example that I use on pastie for those interested.
Once you have everything configured, you can start it up using the monit command (monit -c /etc/monit.conf or whatever the path is to your monit conf file).
Now comes the fun part. I’m assuming you have a few instances of mongrel running and you’ve installed and configured monit. If you have those basics covered, you can play with monit to confirm that it is actually working.
# get a list of your mongrels and the command that started them
ps aux | grep mongrel
# kill one of them by pid
sudo kill 3453 # or whatever the pid number is
Now just wait and you should get an email and see the process you just killed fire back up on its own (ps aux | grep mongrel). Pretty cool, eh?
Honestly, there isn’t a whole lot more to monit. It has an optional web interface. I threw a few screenshots in below that you can check out.
Monit Start Page
Monit Individual Process Page
Other Options
The new kid on the monitoring block is God. It was created by Tom Preston-Werner and is a pure ruby solution. The sweetness of this is that you can use ruby in your config file which makes for less repitition (think looping through an array of mongrel ports). The only downside is I’m sure it uses more memory (though it’s probably so little that who cares?). Anyway, I can’t vouch for it as I haven’t used it but I talked to Tom at RubyConf and he said he uses it in production on several servers with great success. I’ll be trying it soon.
In Conclusion
Whether you go with old faithful, Monit, or the new kid, God, you need to use something to monitor your tag cloud. If you are like me, the only way you want to be up at 3AM is if you are hacking on something fun, not bringing mongrel instances back online.
Ack is a tool for programmers similar to grep but made for heterogeneous source code trees. What I was impressed with upon downloading and installing it was the highlighting it does all in the terminal.
I’ve used it a few times searching through rails trunk as TextMate seems to dog when you get a bunch of files in one project. It also prominently displays the file path which you can then copy and open with the mate command.
A few reasons you might want to use ack (from the ack site, there are more there as well):
it’s really fast
it runs on windows
the standalone version has no dependencies so you can throw it in ~/bin
searches recursively through directories by default, while ignoring .svn, CVS and other VCS directories
let’s you specify file types $ ack --perl pattern
Installation is really easy on a mac. Just grab the standalone version and throw it in ~/bin/ack. Note: make sure that it is executable (chmod u+x) and that ~/bin is in your path. Enjoy.
Pro Active Record “helps you take advantage of the full power of your database engine from within your Ruby programs and Rails applications.” It is authored by Kevin Marshall, Chad Pytel, and Jon Yurek and published by Apress. For this review, I’ll start with the bad and then finish on a positive note.
The Bad
I feel the title of this book is very misleading. It should be “Learning Active Record” or “More Documentation for Active Record” or “On Your Way to Being a Pro in Active Record.” Why is a “Pro” level book teaching you how to set up your database in chapter 3. Heck, why is it talking about setting up your database at all? I would think that a book on the pro level would literally be chocked full of advanced code samples and such, not simply above average documentation for all the the methods in the Active Record module. Teach me how to model complex things. Start right off with an example of how to do something that you don’t learn in the first few months of rails.
There were a few places in the book where I felt there were contradictions with idiomatic ruby/rails and even with other parts in the book. For example, page 43 recommends not to use upper case database columns. I agree. Rails agrees. It doesn’t cause any harm but Ruby is case sensitive so it’s better to stick with lower. Also, upper case typically denotes constants and classes. Method names in ruby (and thus column names) should always be lower case, yet several places in the book use examples with column names in mixed case (ie: Account_Username instead of account_username).
Also along these lines, typically, method names are words separated by underscores (ie: my_method_name instead of mymethodname). More than a few examples in the book were multiple word methods or column names but didn’t separate with underscores.
I realize that I am being really picky and this seems kind of harsh but I see this book as a beginner to intermediate level and would hate for someone in that range to see something generally frowned upon in the community and unknowingly accept it as an “ok” practice.
The Good
Ok. Enough ripping, now on to the good in this book. First off, it reminded me of how Active Record is capable of handling locking by use of the lock version column. I had read this a while ago but forgot. Not too long ago I spent 30 minutes at the whiteboard coming up with a solution (purely for academic purposes), not thinking to see if Active Record handled this in any way.
Second, it had a few examples of how and when to use the straight up connection methods for executing raw sql (execute, insert, update, select_on, select_values, select_all, etc.; more on this on Josh Susser’s blog). Just yesterday, I used select_all to do some sql for the news archives area of Conductor, the CMS I am building at work for our University.
Third, it mentioned the convertible to csv gem/plugin which adds a to_csv method to your active record models in a similar fashion to the existing to_xml, to_json, etc. that you can do in Rails.
Lastly, it pointed out the uuidtools gem. I had forgotten about it.
Conclusion
I would recommend this book to someone who has some grasp on Active Record but wants to learn more about all the options it has. Don’t expect any complex examples but the book does work great as a documentation source for Active Record. It has a few more examples and explanation than the AR docs themselves.
Other Reviews
Don’t take just my word for it, check out these other bloggers who have written up reviews and make an informed decision.
Overall, I had a great time at my first RubyConf. I found myself complaining about a few things so I figured I should at least attempt to offer some constructive criticism, rather than just whining. :) Below are some ideas for RubyConf 2008.
Enlarge the individual’s name on the name tags. This year the logo and design was huge and the person’s name almost disappeared. Everyone knows we are at RubyConf, but we don’t know everyone’s names. Make names huge and easy to read, you know, so stalking is a bit easier. The 40px square avatars that people use on twitter aren’t enough to recognize them at a conference.
Good wifi. Is this not possible? Can we not pair with a university or something to get sweet conference room setups and good wifi. Hotels and conference centers may be able to handle the interweb load of some suit on a blackberry but not a geek installing a gem that a presenter is talking about or checking out some code to hack on.
Start at no earlier than 10. The best part about any conference is meeting all the people. When the talks and keynotes go until 8 or 9 at night that only leaves the nightlife as a time to meet-up. This leads to some late nights and makes it pretty hard to function at 8 or 9AM (not that I function well at that hour anyway).
Less talks per day. I would say four or five talks is plenty for a day. The sessions start talk. Use them as that, not as a full day of education. Use the sessions to get everyone primed to fight over interpreters at the bar later. I found myself a bit “talked” out by afternoon of day 2. Or maybe weight the number of talks by day? Day 1 has 7 or 8, day 2 has 4 or 5 and day 3 has 2 or 3.
Don’t get any bigger. RubyConf was much easier to meet people than RailsConf. I think this is largely due to the fact that RailsConf is 3x as large. Stay small RubyConf, stay small.
Those are my suggestions. Anyone else agree or disagree with any of my points? Have some other constructive suggestions?
Again, I would like to repeat that this is not necessarily defacto or best practice, I’m just putting out what is working for me. If you think I’m stupid feel free to say so, but post an example of a better way or your comment will be deleted. Onward.
Your app is up and running. You have regularly scheduled backups. Oh crap. 503 Proxy Error. Downstream upstream. Clients and users are getting pissed. What do you do? No, calling ghostbusters is not an option. Most likely you are having memory problems or something.
Before I was using Monit (which we’ll discuss in part 3), these are steps I would take to figure out why things went down. First you have to get in your server so fire up terminal or iTerm and ssh in (be sure to check out I Can Has Command Line? for ssh shortcuts and other command line tips).
ps aux
Once you have ssh’d in, check the processes (ps aux) and see what is running (or not running) and what is pissing off your server.
From the pic above you can see the columns of information that are given to you. Most likely you will get a long list of processes but don’t worry that is normal. There are a few important things to look at. First, scan up and down the list in the CPU and MEM columns. Is anything over like 20-25? If so, there is a good chance that process is causing some issues. You can kill it if you want (and you know how to start it back up). sudo kill 206 will do the trick where 206 is replaced with the process id in the ps aux PID column. It is a good idea to check out processes every now and then when things are running fine so you have an idea of what is typical. That way when something is wrong, you’ll instantly notice what it is different and you can start your troubleshooting there.
The other column to note in this output is COMMAND. Command is typically what was called to startup this process. For example, often apache processes COMMAND output look something like this:
/usr/sbin/httpd.worker
Likewise, mongrel instance COMMAND output often looks like this:
/usr/bin/ruby /usr/bin/mongrel_rails start -d -e production -p 8000 -a 127.0.0.1 -P log/mongrel.8000.pid -c /var/www/apps/myapp/current --user username --group groupname
If your mongrel conf file has four instances set and you only see 3 when you run the following:
ps aux | grep ruby
That means that one of your instances of mongrel has crashed. If you have them running on 8000, 8001, 8002, and 8003 and 8002 is missing, you don’t need to restart all the dogs, you can just start the one. Simply copy the command of one of the running mongrels and replace the port with the port of the instance that crashed. So for example, if port 8002 is not showing up in your ps aux output, copy the port 8000 command, replace 8000 with 8002 near -p and -P and hit enter. You can ps aux | grep ruby again to make sure that it has in fact started up.
If you see a ton of apache workers, there is a good chance that it’s time to pimp your setup a bit. Most likely, in my experience, you have a ton of requests coming in and apache is overloading one or two of your mongrel instances and nearly forgetting about the others. Apache’s mod_proxy_balancer is easy to setup but it seems to not care if a mongrel instance is slow or not responding and keeps firing requests at it. If you have KeepAlive set to on in apache and a mongrel instance gets overloaded, you’ll end up with a crap load of httpd workers. Anyway…those are a few things that can go wrong.
Sometimes if things are running slow or stopped and all my instances of mongrel are still running a simple restart of them and an ensuing restart of apache will clear things up. On my machine, I setup alias in my bash profile to allow for easy restarting of the entire cluster and apache:
alias restart_web='sudo /etc/init.d/httpd restart'
alias restart_app='sudo mongrel_rails cluster::restart -C /etc/mongrel_cluster/myapp.conf'
Your config and restart commands may be different, but you get the idea. You don’t want to have to think when something goes down. Typically, a restart will fix things so make it really easy to do.
free -m
Before you go cowboy and start restarting all your dogs, you might want to check out your memory and file system usage. To see how much memory you are using, you can use free -m.
The top row used value (243) will always be very close to the top row total value (254). This is because linux uses any spare memory to cache disk blocks (or so I’ve read online). The important number is the buffers/cache used value (138). This tells how much memory you are actually using. For best performance, it should be less than your total memory (254). If your buffers/cache used value (138) is higher than your total memory + your swap memory, you are going to get out of memory errors which I’m assuming would be really bad. :)
vmstat
Another way to analyze memory is with vmstat (which will be graphing in part 4).
The first row shows your server averages. The si (swap in) and so (swap out) columns show if you have been swapping (i.e. needing to dip into ‘virtual’ memory) in order to run your server’s applications. The si/so numbers should be 0 (or close to it). Numbers in the hundreds or thousands indicate your server is swapping heavily. This consumes a lot of CPU and other server resources and you would get a very (!) significant benefit from adding more memory to your server.
Some other columns of interest: The r (runnable) b (blocked) and w (waiting) columns help see your server load. Waiting processes are swapped out. Blocked processes are typically waiting on I/O. The runnable column is the number of processes trying to something. These numbers combine to form the ‘load’ value on your server. Typically you want the load value to be one or less per CPU in your server.
The bi (bytes in) and bo (bytes out) column show disk I/O (including swapping memory to/from disk) on your server.
The us (user), sy (system) and id (idle) show the amount of CPU your server is using. The higher the idle value, the better.
df and du
You’ll also want to keep an eye on total disk usage (until we start graphing it in part 4) so get familiar with df and du. df is for checking your total file size and du is for checking particular directories. Use the -h option with either to get more human readable file sizes.
Your best friend in times of crisis is a level head. Remember…outages happen. Computers messup. You messup. The most important thing is that you stay calm and attack logically. In the year or so that I’ve been managing rails apps, I haven’t ran into to many problems that a restart of the mongrel cluster and apache wouldn’t fix.
Did I miss anything? Have a better idea? Didn’t understand something? Leave comments below.
Started slow during the coding, in my opinion, because I have already played with Camping. I enjoyed the talk of the differences between rubyconf and railsconf simply because Brandon and I were talking about the same topic yesterday at a Starbucks.
Rails is awesome. It allows you to focus on business problems and solutions. I have to be honest though…it’s getting kind of boring. The more conventions and the easier Rails gets, the more boring it gets. Go try out camping, merb, sequel, and datamapper. Start learning something new in ruby. Keep getting paid to do rails and keep solving problems quickly, but force new ideas in. Don’t just continually take orders.
“yes, these are your father’s parenthesis” – in regards to learning lisp
“FORTH runs great in 2k of memory”
Box 1 Master of Disguise
Rake – FileList is like an array except initialized as GLOB, specialized to_s, extra methods (ext, pathmap, etc.), lazy evaluation (they don’t look at file system until they are iterated or whatever)
RESOLVING_METHODS.each do |method|
class_eval %{
def #{method}(*args, &block)
resolve unless @resolved
# blah...
end
}
end
Box 2 (I didn’t see the name)
“Builder was a fun library to build. I wrote it just for the fun of it.”
Very cool stuff in builder. Install it if you haven’t and check out the BlankSlate class. It’s an object that doesn’t behave like and object.
Box 3 Parsing without Parsing
“how many people have written a ruby parser?” – everyone laughs because it’s probably just matz and his crew
Jim showed one way to begin converting ruby enumerable to sql generator. Created TableNode, MethodNode and a Node base. Node base implements methods such as which converts stuff like users.age 45 to users.age = 45 in sql. Problem ensues with literal strings (users.name == ‘jim’ converts to users.name = jim). He would add as_sql_node method to each object in ruby which means each object knows how it should be represented in sql.
Learn the corners of your laguage of choice to take full advantage.
Don’t be afraid to think outside the box of past experience…not all the time but occasionally ruby can provide very special solutions.
Conclusion
Jim’s talk was really interesting. More talks should be like this. He gave practical examples of how to solve interesting problems which really opens up your mind.
“at the risk of sounding emo” – during his explanation of why he wanted to determine what beautiful code is
“her jawline is the golden ratio” – how do you explain the fact that you think your wife is beautiful
“i realize the fact that some people are going to think i’m totally wrong, but that’s not important” – this is not deep and rigorous, it’s only what is needed to apply it to the current (software development)
Thomas Aquinas definition of beauty
proportion – economy of size (smallest size needed to do the job) and ratio of pars
integrity – thing being well suited for it’s purpose
clarity – simple and clear (there is a difference between sophistication and complication)
Showed an example of a CoercibleString which allows a string being automatically converted to string, boolean, integer, or datetime. Very cool idea. He created it for web services project. Really sweet idea. I’ve parsed a lot of XML from web services and that is always annoying. Refactored it to String.coerce which is about 6 lines instead of 20.
The first version felt cool and interesting but it turned out to be terrible.
Proportion/Integrity/Clarity – Each are necessary. None are sufficient.
Does code quality relate to beauty?
Kent Beck: Smalltalk Best Practice Patters – Recommended reading.
Beauty is a moving target. Ruby in 20 years could be like assembler now a days.
Can you imagine programming without if? If might have been beautiful when it came out.
“An expert is a person who has made al the mistakes that can be made in a very narrow field.”
Ruby is optimized for beauty. Try to imagine better modes of expression. Violations of beauty rules (proportion, integrity, clarity) reveal mistakes. Do that enough and you will innovate.
Conclusion
Very interesting. I thought the first example seemed beautiful but I was just looking at aesthetics (tabs, lining up, etc.). Simple, practical ways to analyze code beauty.
Again, I would like to repeat that this is not necessarily defacto or best practice, I’m just putting out what is working for me. If you think I’m stupid feel free to say so, but post an example of a better way or your comment will be deleted. Onward.
So your app is up and running, what is next? The most important thing now is to back everything up. This includes your database, your subversion repository and any files that are not versioned (user uploads, etc.).
First things first, prepare move offsite
Before we even get to backups, note this: backups are no good if they are only on your server and your server crashes. Push backups offsite with sftp, ssh, scp or rsync. Below is an example yaml config I use called sftp.yml. You’ll see how it comes into use later in this article.
Database backups for the size of apps we are talking about are pretty simple. I only know mysql so that is all I will show but you could take these principles to any RDBMS. What I did is create a folder in my user home directory named bin. Inside it I keep all the ruby files that do the backup work. Below is ~/bin/database_dump.rb.
#!/usr/bin/ruby
require 'yaml'
require 'logger'
require 'rubygems'
require 'net/ssh'
require 'net/sftp'
APP = '/path/to/app/current/directory'
LOG_FILE = '/home/username/log/database.log'
TIMESTAMP = '%Y%m%d%H%M%S'
log = Logger.new(LOG_FILE, 5, 10*1024)
dump = "conductor_#{Time.now.strftime(TIMESTAMP)}.sql.gz"
# get the off server sftp configuration settings
ftp_config = YAML::load(open('/home/username/bin/sftp.yml'))
# get the production database configuration
config = YAML::load(open(APP + '/config/database.yml'))['production']
cmd = "mysqldump -u #{config['username']} -p#{config['password']} -h #{config['host']} --add-drop-table --add-locks --extended-insert --lock-tables #{config['database']} | gzip -cf9 > #{dump}"
log.info 'Getting ready to create a backup'
`#{cmd}`
log.info 'Backup created, starting the transfer offsite'
Net::SSH.start(ftp_config['host'], ftp_config['username'], ftp_config['password']) do |ssh|
ssh.sftp.connect do |sftp|
sftp.open_handle("#{ftp_config['dir']}/#{dump}", 'w') do |handle|
sftp.write(handle, open("#{dump}").read)
end
end
end
log.info 'Finished transferring backup offsite'
log.info 'Removing local file'
cmd = "rm -f #{dump}"
log.debug "Executing: #{cmd}"
`#{cmd}`
log.info 'Local file removed'
So that is the quick and dirty. Basically, I use ruby to run a mysqldump command (which uses the information from our database.yml file) and sftp the backup off the server to another location. I store the sftp settings in a yaml file that I can reuse for each of the backup scripts. What would I do different? I would move this to a rake task and/or use one of the tools that is already out there. You don’t have to setup crazy tools and all kinds of things. Think inside the box. You have ruby. Ruby is fun. Just use it to get the job done. That is what the script above does. Might seem simple but it’s been working great for several months. I run it nightly using a cron entry like this:
crontab -e will open up your cron config file and you can simply paste in the above snippet and change the paths to ruby and your database dump script. The settings above will force the dump to run at 3AM nightly. To test your db dump script you can run it just like any ruby script (ruby database_dump.rb). Once you have it working like that, throw it in cron as I just mentioned.
Ok. So now you have database backups. Sweet. Next up is subversion. But isn’t subversion backup enough? I mean sometimes I commit and go home and svn up and…Stop. Please stop. Subversion is for version control. It is not a code backup solution. Use subversion for version control and do regular svn dumps for backup. I even scoot them offsite with the mysql backups. Below is a simple subversion dump script I threw together:
#!/usr/bin/ruby
require 'yaml'
require 'logger'
require 'rubygems'
require 'net/ssh'
require 'net/sftp'
REPO = '/path/to/your/repository'
LOG_FILE = '/home/username/log/svn_dump.log'
TIMESTAMP = '%Y%m%d%H%M%S'
log = Logger.new(LOG_FILE, 5, 10*1024)
dump = "yourappname_svn_#{Time.now.strftime(TIMESTAMP)}.dump.gz"
ftp_config = YAML::load(open('/home/username/bin/sftp.yml'))
log.info 'Starting subversion dump'
cmd = "svnadmin dump #{REPO} | gzip -c9 > #{dump}"
log.debug "Executing: #{cmd}"
`#{cmd}`
log.info "Dumped #{REPO} to #{dump}"
log.info 'Backup created, starting the transfer offsite'
Net::SSH.start(ftp_config['host'], ftp_config['username'], ftp_config['password']) do |ssh|
ssh.sftp.connect do |sftp|
sftp.open_handle("#{ftp_config['dir']}/#{dump}", 'w') do |handle|
sftp.write(handle, open("#{dump}").read)
end
end
end
log.info 'Finished transferring backup offsite'
log.info 'Removing local file'
cmd = "rm -f #{dump}"
log.debug "Executing: #{cmd}"
`#{cmd}`
log.info 'Local file removed'
Again. Quick and dirty. I just use ruby to run a shell command (svnadmin dump) which is piped into gzip for compression to make the file smaller and the transfer a bit speedier. That is really all there is to it. Once you have an svn dump, it’s really easy to restore. Let’s say the dump was mydump.gz. The following would unzip it and load it into a directory:
Other things you need to backup are any user uploaded files, anything in your capistrano shared directory and any custom configured files on your server. I was using ruby to tar and gzip most of these files and then sftp them offsite but I noticed a huge memory spike during this process, so large in fact that it choked out my mongrels. Instead, I would recommend using rsync for anything over a few megs. It was made to move files around and you can even get cheap backup offsite that is georedundant from rsync.net for less than $3/GB per month. Well worth the cash, IMO.
Ok. So I got an overwhelming response to my last post about some things I have learned managing an ok sized rails app. And by overwhelming I mean 14 comments. Ha. Anyway, that is a lot for this here blog so I’m betting I struck a key. I put together a basic outline and I’ll now present that and explain the title of this article. First the title.
Why the title?
It’s simple really, and kind of funny. I was watching an Adobe Flex web demo a few weeks back and the presenter opened things up for questions at the end. One of the questions was: ‘Can ColdFusion make tag clouds as easily as Rails?’ Granted that could have been a neophyte, or a jokester but either way it cracked me up. I mean all Rails does is tag clouds, right? So the question spinning through everyone’s minds is “How can I keep my tag cloud up” and ready for consumption by the web 2.0 beta croud, especially when you get techcrunch’d. Anyway, enough about the title, let’s get to the meat.
A little background
This is not a “how to scale twitter article” as there are plenty of those already. Most likely if you need to scale like that you shouldn’t be reading blog posts but should be hiring a team with previous experience. This is, however, an “I know rails, have an app or two in production and want to do whatever I can in my limited time to make sure things run smoothly” article. I keep a few rails applications running and for the most part they lay dormant. The one app that does handle some load is a multi-site content management system. It is currently serving over 400,000 page views a month (according to Google Analytics). It runs on one 512MB web/app server (with 4 instances of mongrel) and one 512MB database server (which also is the db server for a few other apps). Not huge but it’s growing quickly (a month ago it was only around 100,000 page views). That is what I’m dealing with. You could be dealing with more or less but regardless the stuff I’ll be posting I hope will be helpful.
The Outline of this series
So here are my thoughts and what I have minor knowledge about. I want this to be a legitimate resource for Rails’ers so if you have other ideas, comment and I’ll try to fit them in where possible. Also, if you are an expert about any of the topics that I mention below, let me know and maybe you can give the post a once over before I release it to the masses. In no way do I consider myself an expert or heck, even an intermediate on any of these topics, but I got them all working (cheers to me) and figured there are others in the same boat. I’m probably not the best person to write on these topics but those people have everything under control and probably think it is too easy to deserve a blog post, not realizing that the rest of us are all clamoring for it. These are in sequential order and are not currently written (just notes jotted).
Backups. The first and most important thing once your app is up and running is making sure you can recover from problems.
Short Term Problem Analysis and Resolution. Ok, something just went wrong. What? This post will provide a few tips for how to troubleshoot quickly and, most important, calmly.
Resource Reporting. Munin. Who doesn’t love an awkwardly ugly graph that shows memory consumption and such?
At this point you have backups, know how to solve easy glitches, have automation for nighttime problems and reporting so you can figure out why monit is emailing like a desperate ho. What you need now is simple fixes and some optimizations. Ever notice your 500MB log file? Yeah, it’s time for log rotation and analyzation.
Hmm. Things seem to be running smooth but now we are getting a lot of data in our app and it is beginning to dog. Indexes. Yeah, heard of them before. It’s time to turn on slow query logging and check the output every now and then, analyze (word of the day) the output and add some indexes (that you should have added at the beginning). Oh, and get some idea if they actually helped.
Things are going good. Can we squeak a bit more out of our current setup? Some thoughts on alternate setups, multiple databases and caching.
Did I miss anything? Are you an expert on any of these? Let me know before I hit the Mephisto admin hard.
